Version 2006 June 5th
newLISP 8.8.9
and after
Copyright
© 2006 Lutz Mueller, www.nuevatec.com.
All rights reserved.
Permission is granted to copy,
distribute and/or modify this document under the terms of the GNU
Free Documentation License,
Version 1.2 or any later version
published by the Free Software Foundation; with no Invariant
Sections, no Front-Cover Texts,
and no Back-Cover Texts. A copy of
the license is included in the section entitled GNU
Free Documentation License.
newLISP is a trademark of Lutz
Mueller.
Specifying command line options in script files
Scripts as pipes
File filters
Scoping and protection of symbols
Variable symbols
Context symbols
Passing parameters by reference
The default function
Creating objects
Locals in looping functions
Local symbols using let and letn
Unused parameters as local symbols
Using args as local substitute
Recursion or not?
Walking a tree
Walking a directory tree
push and pop
Accessing lists
Selecting more than one element
Changing list elements
Passing lists by reference
Loops
Blocks
Branching
Fuzzy flow
Change flow with catch and throw
Change flow with and or or
System errors
User defined errors
Error event handlers
Catching errors
Manipulate functions after definition
Function currying: functions making functions
Mapping and applying functions
Regular expressions
Scanning text
Appending strings
Growing strings in place
Rearranging strings
Dictionaries
Object oriented implementation of a dictionary
Client - server TCP/IP - open connection
Client - server TCP/IP - closed connection
Open communications with UDP
Closed transaction oriented UDP
UDP multi-cast communications
Using net-select
Using net-peek
Launch blocking with shell
Launch with output in list
Communications via STD I/O
Communications via TCP/IP
Communicate via named FIFO
Communicate via UDP
Shell execution
Execution capturing std-out in a list
Execution feeding std-in
Association lists
Symbol creation and lookup
Compile a shared library on Linux/UNIX/MacOSX
Compile a DLL on Win32
Using data structures in imported libraries
Unevenly aligned data structures
Passing parameters in library calls
Extracting return values from library calls
When programming in newLISP certain functions and usage patterns occur repeatedly. For some problems an optimal way to solve them evolves over time. The following chapters present example code and explanations for the solution of specific problems when programming in newLISP.
Some content is overlapping with material covered in the newLISP Users Manual and Reference or presented here with a different slant.
Only a subset of newLISP's total function repertoire is used here. Some functions demonstrated have additional calling patterns or applications not mentioned on these pages.
This collection of patterns and solutions is a work in progress. Over time material will be added or existing material improved.
On Linux/UNIX put the following in the first line of the script/program file:
#!/usr/bin/newlisp
specifying a bigger stack:
#!/usr/bin/newlisp -s 100000
or
#!/usr/bin/newlisp -s100000
Operating systems shells behave differently when parsing the first line and extract parameters. newLISP takes both, attached or detached parameters. Put the following lines in small script to test the behavior of the underlying OS and platform. The script changes the stack size allocated to 100,000 and limits LISP cell memory to about 10 M bytes.
#!/usr/bin/newlisp -s 100000 -m 10 (println (main-args)) (println (sys-info))
A typical output executing the script from the system shell would be:
~> ./arg-test
(308 655360 299 2 0 100000 8410 2)
("/usr/bin/newlisp" "-s" "100000" "-m" "10" "./arg-test")
~>Note that few programs in newLISP need a bigger stack configured, most programs run on the internal default of 2048. Each stack position takes an average of 80 bytes. Other options are available to start newLISP. See the Users Manual for details.
The following examples shows how a file can be piped into a newLISP script.
#!/usr/bin/newlisp # # uppercase - demo filter script as pipe # # usage: # ./uppercase < file-spec # # example: # ./uppercase < my-text # # (while (read-line) (println (upper-case (current-line)))) (exit)
The file will be printed to stdout translated to uppercase.
The following script works like a Unix grep utility iterating through files and filtering each line in a file using a regular expression pattern.
#!/usr/bin/newlisp
#
# nlgrep - grep utility on newLISP
#
# usage:
# ./nlgrep "regex-pattern" file-spec
#
# file spec can contain globbing characters
#
# example:
# ./nlgrep "this|that" *.c
#
# will print all line containing 'this' or 'that' in *.c files
#
(dolist (file-name (3 (main-args)))
(set 'file (open file-name "read"))
(println "file ---> " file-name)
(while (read-line file)
(if (find (main-args 2) (current-line) 0)
(write-line)))
(close file))
(exit)The expression:
(3 (main-args))
is a short form of writing:
(rest (rest (rest (main-args))))
It returns a list of all the filenames. This form of specifying indices for rest is called implicit indexing. See the Users Manual for implicit indexing with other functions.
The expression (main-args 2) extracts the 3rd argument from the commandline containing the regular expression pattern.
To understand the working of newLISP it is important to understand symbols and name-spaces or contexts. Contexts facilitate programming with objects and are a method of dividing code into manageable well isolated modules, important for program design and when developing newLISP applications in teams.
newLISP's symbol implementation is at the core of much of it's
capabilities. The Red Black binary tree implementation
balances a binary symbol tree for faster access and insertion of new
symbols. newLISP's symbol implementation scales well to millions of
symbols. The symbol tree in newLISP can be partitioned in name
spaces called contexts and name spaces can easily be
serialized to a disk file as newLISP source. A context is just
another symbol in the newLISP's MAIN symbol table,
serving as a root for a new symbol tree:
MAIN ->
*
+
-
MAIN
MyContext ->
my-func
foo
x
y
abs
acos
add
address
aVar
...
xml-xxx
The graphic shows part of the MAIN symbol tree. All
context symbols including MAIN are part of this tree.
Symbols are created in newLISP during loading of source code or reading code from the command line. Symbols are also created during evaluation of eval-string and when using the sym or symbols (older writing) functions.
Only built in functions and operator symbols, context symbols and
special symbols like nil, true, etc. are
global. All other symbols created by users are local to their
context, but symbols belonging to MAIN can be made
global using the global function.
Context symbols, the symbols of built-in functions and special
symbols are protected against change. User defined symbols can be
made protected using the constant keyword.
; making a symbol global (global 'myvar) ; protecting a symbol (constant 'MAX_BYTES 1024) (set 'MAX_BYTES 123) => symbol is protected in function set : MAX_BYTES ; protecting and globalizing together (constant (global 'foo) (lambda (x) (+ x x)))
The last example shows how a global and protected function can be created in one expression. When loading code multiple times this form avoids symbol protection errors.
The following paragraphs describe in detail the two forms of symbols: variables and contexts.
Variable symbols are used for variables and can contain any data type: Boolean, integer, float, string and all forms of s-expressions like lists, lambda expressions for user defined functions, etc.. Variable symbols can also hold other symbols and contexts.
A symbol can be transformed into a context symbol only via the context function.
Context symbols are handles for name-spaces or contexts. Once a context symbol is created, it is protected and cannot be used as a variable symbol. The context is a special data type in newLISP and cannot be changed. Context symbols are protected and global. They are visible from any other context.
There are two ways to create a context symbol:
;; just use a symbol as context and it gets created (set 'MyContext:var 123) (define (Foo:double x) (+ x x)) ;; or make the context and switch to it (context 'MyContext) ;; contexts evaluate to themselves MyContext => MyContext
Like any other symbol in newLISP, context symbols are created when they occur in code and are seen by the newLISP loader or the functions eval-string or sym. The context function switches the translation in functions eval-string and sym to the new context. All symbols subsequently created will go into this name-space.
; foo is created in MAIN
(set 'foo "hello")
(context 'CTX)
; this is CTX:foo not MAIN:foo
(set 'foo "hi")
; bar, x and y are all in CTX
(define (bar x y) (+ x y))
(context MAIN) ; can omit quote because MAIN already exists
(define (func x y z)
(println foo) ; this will print "hello"
(context CTX)
(println foo) ; this will still print "hello"
;but this will print "hi" because the
;translation context switched
(eval-string "(println foo)") )
(func)
=> ; will out put the following
hello
hello
hi
When trying to use a context symbol as a variable an error message will occur. This may lead to situations where the sequence of loading code modules becomes important.
(context 'FOO) (set 'ABC 123) (context MAIN) (context 'ABC) (set 'FOO 456) (context 'MAIN)
The loading of context ABC will cause an error message because FOO is already created as a global protected context symbol.
The variable symbol ABC is not affected because it is local to the name-space FOO.
The situation could be avoided by making FOO local in ABC.
(context 'ABC) (set 'ABC:FOO 456) (context 'MAIN)
But the best is to avoid overwriting globally symbols altogether.
Another situation arises when using symbols holding contexts.
(define (func value) (set 'ctx:var value)) (func 456) ctx:var => 456
When the preceding definition gets loaded, a context ctx with a symbol variable var in it will be created. Calling func will change the value of symbol var in context ctx. But imaging the following code:
(set 'MyCtx:var 123) ; a symbol variable ctx pointing to a context (set 'ctx MyCtx) (define (func value) (set 'ctx:var value)) ctx:var => 123 (func 999) MyCtx:var => 999
Now ctx is not a context by itself but a variable holding a context. While in the first example ctx was translated into a context symbol, now it is translated to a variable symbol, because it was introduced earlier as such in the expression:
(set 'ctx MyCtx) ; ctx is a variable
To avoid ambiguities like this, symbol variables serving as context variables should always be part of the function definition:
(define (func ctx value) (set 'ctx:var value))
And the function could be called like this:
(func MyCtx 999) MyCtx:var => 999
All looping functions like dolist, dotimes, dotree and for, use a variable symbol as a local. During loop execution the variable takes different values, but after leaving the looping function the variable regains its old value.
Let is the traditional way in LISP to declare symbols as local to a block.
(define (sum-sq a b)
(let ((x (* a a)) (y (* b b)))
(+ x y)))
(sum-sq 3 4) => 25
; alternative syntax
(define (sum-sq a b)
(let (x (* a a) y (* b b))
(+ x y)))The variables x and y are initialized, then the expression (+ x y) is evaluated. The let form is just an optimized version and syntactic convenience for writing:
((lambda (sym1 [sym2 ...]) exp-body ) exp-init1 [ exp-init2 ...])
When initializing several parameters, a nested let letn can be used to reference previously initialized variables in subsequent initializer expressions. See the newLISP Reference manual for details.
In newLISP all parameters in user defined functions are optional. Unused parameters are filled with nil and local to the dynamic scope of the function. Defining a user function with more parameters than required is a convenient method to create local variable symbols:
(define (sum-sq a b , x y)
(set 'x (* a a))
(set 'y (* b b))
(+ x y))The comma is not a special syntax feature but just a visual helper to separate normal parameters from local variable symbols. The comma is just another symbol.
Using the args function no parameter symbols need to be used at all and args returns a list of all parameters passed and not taken by declared parameters:
(define (foo)
(args))
(foo 1 2 3) => (1 2 3)
(define (foo a b)
(args))
(foo 1 2 3 4 5) => (3 4 5)The second example shows how args only contains the list of arguments not already bound by declared variable symbols.
Indices can be used to access members of the (args) list:
(define (foo) (+ (args 0) (args 1))) (foo 3 4) => 7
Although recursion is a powerful feature to express many algorithms in a readable form, they are also inefficient in some instances. newLISP has many iterative constructs and high level functions like flat or the xml functions, which use recursion already internally. This makes it unnecessary in many cases to define a recursive algorithm.
Some times a non-recursive solution can be much faster and lighter on system resources.
;; classic recursion - elegant but slow
;; and resource hungry
(define (fib n)
(if (< n 2) 1
(+ (fib (- n 1))
(fib (- n 2)))))
;; iteration - fast and returns the whole list
;; 70 times faster
(define (fibo n , f)
(set 'f '(1 0))
(dotimes (i n)
(push (+ (f 0) (f 1)) f) -1)
(rest f))
;; generator – uses contexts to keep state
;; the default funcion (fibo) gets called repeatedly
;;
;; (fibo) => 1
;; (fibo) => 2
;; (fibo) => 3, 5, 8, 13, 21 ...
;;
;; fibo:mem => (0 1 1 2 3 5 8 13 21 ...)
(define (fibo:fibo)
(if (not fibo:mem) (set 'fibo:mem '(0 1)))
(push (+ (fibo:mem -2) (fibo:mem -1)) fibo:mem -1))The last example using a generator uses a default function. A default function is a function with the same name as the context it belongs too.
Tree walks are a typical pattern in traditional LISP and in newLISP as well. But may times a tree walk is only used to iterate through all elements of a tree. In this case the built-in flat function is much faster:
;; classic car/cdr and recursion
;;
(define (walk-tree tree)
(cond ((= tree '()) true)
((atom? (first tree))
(println (first tree))
(walk-tree (rest tree)))
(true
(walk-tree (first tree))
(walk-tree (rest tree)))))
;; classic recursion
;; 3 times faster
;;
(define (walk-tree tree)
(dolist (elmnt tree)
(if (list? elmnt)
(walk-tree elmnt)
(println elmnt))))
;; fast and short using 'flat'
;; 30 times faster
;;
(map println (flat '(a b (c d (e f)))))
Walking a directory tree is a task where recursion works well:
; walks a disk directory and prints all path-file names
;
(define (show-tree dir)
(dolist (nde (directory dir))
(if (and (directory? (append dir "/" nde))
(!= nde ".") (!= nde ".."))
(show-tree (append dir "/" nde))
(println (append dir "/" nde)))))
newLISP has facilities for multidimensional indexing into nested lists. There are destructive functions like push, pop, set-nth, nth-set, sort and reverse and many others for non-destructive operations, like nth, first, last, rest etc.. Many of the list functions in newLISP also work on strings.
Note that any index in newLISP can be negative starting with -1 from the right side of a list:
One of the most used list functions are push and pop. There are both destructive, changing the contents of a list:
(set 'L '(b c d e f)) (push 'a L) (push 'g L -1) ;; push at the end with negative index (pop L) ; pop first (pop L -1) ; pop last (pop L -2) ; pop second to last (pop L 1) ; pop second ; multidimensional push / pop (set 'L '(a b (c d (e f) g) h i)) (push 'x L 2 1) L => (a b (c x d (e f) g) h i) (pop L 2 1) => 'x
Pushing to the end of a list repeatedly is optimized in newLISP and as fast as pushing in front of a list.
When pushing an element with index vector V it can be popped with the same index vector V:
(set 'L '(a b (c d (e f) g) h i)) (set 'V '(2 1)) (push 'x L V) L => (a b (c x d (e f) g) h i)) (ref 'x L) => (2 1) ; search for a nested member (pop L V) => 'v
Multiple indexes can be specified to access elements in a nested list structure:
(set 'L '(a b (c d (e f) g) h i)) (nth 2 2 1 L) => f (nth 2 2 L) => (e f) ; implicit indexing (after version 8.4.5) (L 2 2 1) => f (L 2 2) => (e f) ; implicit indexing with vector (set 'vec '(2 2 1)) (L vec) => f
Implicit indexing shown in the last example makes code more readable. Implicit indexing also allows an unlimited number of indices, while nth is limited to 16. Indices after a list select list elements. Indices before a list select subsections of a list, which in turn are always lists.
Implicit indexing is also available for rest and slice
(rest '(a b c d e)) => (b c d e) (rest (rest '(a b c d e) => (c d e) ; same as (1 '(a b c d e)) => (b c d e) (2 '(a b c d e)) => (c d e) ; negative indices (-2 '(a b c d e)) => (d e) ; slicing (2 2 '(a b c d e f g)) => (c d) (-5 3 '(a b c d e f g)) => (c d e)
Sometimes more than one element must be selected from a list in different places. This can be done using select:
;; pick several elements from a list ;; (set 'L '(a b c d e f g)) (select L 1 2 4 -1) => (b c e g)
The indices can be delivered in an index vector:
;; indices in vector ;; (set 'vec '(1 2 4 -1)) (select L vec) => (b c e g)
The selecting process can re-arrange or double elements at the same time:
(select L 2 2 1 1) => (c c b b)
Sometimes lists need to be filterer for a specific conditions applied to its elements:
(filter fn(x) (> x 5)) '(1 6 3 7 8)) => (6 7 8) (filter symbol? '(a b 3 c 4 "hello" g)) => (a b c g) (difference '(1 3 2 5 5 7) '(3 7)) => (1 2 5)
set-nth and nth-set have the same effect on the list they are working on but the first returns the whole list while nth-set returns the changed old element:
(set 'L '(a b (c d (e f) g) h i))
; return the changed list (old deprecated syntax) (set-nth 2 2 1 L 'x) => (a b (c d (e x) g) h i) ; return the replaced element (old deprecated syntax) (nth-set 2 2 1 L 'z) => x ; since 8.8.9 a more readable syntax is available
(set-nth (L 2 2 1) 'x) => (a b (c d (e x) g) h i)
(nth-set (L 2 2 1) 'z) => z
An internal system variable $0 in newLISP holds the old list element. This can be used to configure the new one:
(set 'L '(0 0 0)) (nth-set (L 1) (+ $0 1)) => 0 ; the old value (nth-set (L 1) (+ $0 1)) => 1 (nth-set (L 1) (+ $0 1)) => 2 L => '(0 3 0)
(set 'data:data '(a b c d e f g h))
(define (change db i value) (nth-set (db i) value))
(change data 3 999) => d
data:data => '(a b c 999 d e f g h)
In this example the list is encapsulated
in a context data
holding a variable data with
the same name.
Program flow in newLISP is mostly functional but it also has looping and branching constructs and a catch and throw to break out of normal flow.
Looping expressions as a whole behave like a function or block returning the last expression evaluated inside.
Most of the traditional looping patterns are supported. Whenever there is a looping variable, it is local in scope to the loop, behaving according the rules of dynamic scoping inside the current name-space or context:
; loop a number of times
; i goes from 0 to N - 1
(dotimes (i N)
....
)
; demonstrate locality of i
(dotimes (i 3)
(print i ":")
(dotimes (i 3) (print i))
(println))
=> ; will output
0:012
1:012
2:012
; loop through a list
; takes the value of each top level element in aList
(dolist (e aList)
...
)
; loop through the symbols of a context in
; alphabetical order of the symbol name
(dotree (s CTX)
...
)
; loop from to with optional step size
; i goes from init to <= N inclusive with step size step
; Note that the sign in step is irrelevant, N can be greater
; or less then init.
(for i init N step)
...
)
; loop while a condition is true
; first test condition then perform body
(while condition
...
)
; loop while a condition is false
; first test condition then perform body
(until condition
....
)
; loop while a condition is true
; first perform body then test
; body is performed at least once
(do-while condition
...
)
; loop while a condition is false
; first perform body then test
; body is performed at least once
(do-until condition
...
)
Blocks are collections of s-expressions evaluated sequentially. All looping constructs may have expression blocks after the condition expression as a body.
Blocks can also be constructed by enclosing them in a begin expression:
(begin
s-exp1
s-exp2
.....
s-expN)Looping constructs do not need to use an explicit begin after the looping conditions. begin is mostly used to block expressions in if and cond statements.
(if condition true-expr false-expr) ;or (if condition true-expr)
;or unary if for (filter if '(...)) (if condition)
;; more then one statement in the true or false ;; part must be blocked with (begin ...) (if (= x Y) (begin (some-func x) (some-func y) (begin (do-this x y) (do-that x y)))
Depending on condition the true-expr or false-expr part is evaluated and returned.
More then one condition true-expr pair can occur in an if expression, making it look like a cond:
(if condition-1 true-expr-1
condition-2 true-expr-2
.....
condition-n true-expr-n
false-expr)The first true-expr-i of which the condition-i is not nil is evaluated and returned or the false-expr if none of the condition-i is true.
cond works like the multiple condition form of if but each part of condition-i true-expr-i must be braced in parenthesis:
(cond
(condition-1 true-expr-1 )
(condition-2 true-expr-2 )
.....
(condition-n true-expr-n )
(true true-expr))
Using amb the program flow can be regulated in a probabilistic fashion:
(amb
expr-1
expr-2
.....
expr-n)One of the alternative expressions expr-1 to expr-n is evaluated with a probability of 1 / n and the result is returned from the amb expression.
Any loop or other expression block can be enclosed in a catch expression. The moment a throw expression is evaluated, the whole catch expression returns the value of the throw expression.
(catch
(dotimes (i 10)
(if (= i 5) (throw "The End"))
(print i " ")))
; will output
0 1 2 3 4
; and the return value will be
=> "The End"Several catch may be nested.
Loops built using dotimes or dolist can specify a break condition under which the loop is lef early:
(dotimes (x 10 (> (* x x) 9))
(println x))
=> 0
1
2
3
(dolist (i '(a b c nil d e) (not i)) (println i))
=>
a
b
c
Similar to programming in Prolog the logical and and or can be used to control program flow depending on the outcome of expressions logically connected:
(and
expr-1
expr-2
...
expr-n)
Expressions are evaluated sequentially until one expr-i
evaluates to nil or the empty list'() or until
all expr-i are exhausted. The last expression evaluated is
the return value of the whole and expression.
(or
expr-1
expr-2
...
expr-n)
Expressions are evaluated sequentially until one expr-i
evaluates to not nil and not '()
or until all expr-i are exhausted. The last expression
evaluated is the return value of the whole or expression.
Several conditions during evaluation of a newLISP expression can cause error exceptions. For a complete list of errors see the newLISP Reference Manual Appendix.
System errors are caused by wrong syntax, of function invocation, not supplying the right amount of parameters, or supplying parameters with the wrong data type. Other system errors are caused by trying to evaluate non existing functions.
; examples of system errors ; (foo foo) => invalid function : (foo foo) (+ "hello") => value expected in function + : "hello"
User errors are error exceptions thrown using the function throw-error
; user defined error
;
(define (double x)
(if (= x 99) (throw-error "illegal number"))
(+ x x))
(double 8) => 16
(double 10) => 20
(double 99)
=>
user error : illegal number
called from user defined function double
System and user defined errors can be caught using the function
error-event to define an event handler. Note that this
method will only work when newLISP is run as a library newlisp.so
(on Linux/UNIX) or newlisp.dll (on Win32), or when the
program file is loaded interactively. For newLISP script file use
error handlers written with catch, as described in the
next sub chapter.
; define an error event handler ; (define (myHandler) (println "An error #" (error-number) " has occurred")) (error-event 'MyHandler) (foo) => An error #23 has occurred
A fine grainier, more specific error exception handling can be achieved using a special syntax of the function catch.
(define (double x)
(if (= x 99) (throw-error "illegal number"))
(+ x x))catch with a second parameter can be used to catch system and user defined errors:
(catch (double 8) 'result) => true result => 16 (catch (double 99) 'result) => nil (print result) => user error : illegal number called from user defined function double (catch (double "hi") 'result) => nil (print result) => value expected in function + : x called from user defined function double
The catch expression returns true when no error exception occurred and the result of the expression is found in the symbol result specified as a second parameter.
If an error exception occurs, it is caught and the catch clause returns nil. In this case the symbol result contains the error message.
(define (double x) (+ x x)) => (lambda (x) (+ x x)) (first double) => (x) (last double) => (+ x x) ;; make a ''fuzzy'' double (nth-set 1 double '(mul (normal x (div x 10)) 2)) (double 10) => 20.31445313 (double 10) => 19.60351563
lambda in newLISP is not an operator or symbol, but rather a special s-expression or list attribute:
(first double) => (x) ; not lambda
The lambda attribute of an s-expression is right-associative in append:
(append (lambda) '((x) (+ x x))) => (lambda (x) (+ x x)) (set 'double (append (lambda) '((x) (+ x x))) (double 10) => 20
and left-associative when using cons:
(cons '(x) (lambda) => (lambda (x))
Lambda expressions in newLISP never loose their first class object property.
Functions or operators can be applied to a list of data at once and all results are returned in a list
(define (double (x) (+ x x)) (map double '(1 2 3 4 5)) => (2 4 6 8 10)
Functions can be applied to parameters occurring in a list:
(apply + (sequence 1 10)) => 55
Here and expression is passed as a parameter:
(define (raise-to power) (expand (fn (base) (pow base power)) 'power)) (define square (raise-to 2)) (define cube (raise-to 3)) (square 5) => 25 (cube 5) => 125
Regular expression in newLISP can be used together with a variety of functions:
|
directory |
Returns a list of files, can use a regex patterns for filtering |
|
find |
is used to find the position / offset of a pattern. |
|
parse |
breaks a string into token at patterns found between tokens. |
|
regex |
finds patterns and lists all sub patterns with offset and length found. |
|
replace |
replaces found patterns with a user defined function, which can take the as input the patterns itself. |
|
search |
searches for a pattern in a file. |
The functions find, regex, replace and search store pattern matching results in the system variable $0 to $15. See the newLISP Users Manual for details.
The following paragraphs show frequently used algorithms for scanning and tokenizing text.
replace together with a regular expression pattern can be used to scan text. The pattern in this case describes the tokens scanned for. As each token is found it is pushed on a list. The work is done in the replacement expression part of replace. This example saves all files linked on a web page:
; tokenize using replace with regular expressions
(set 'page (get-url "http://www.nodep.nl/newlisp/index.html"))
(replace {href="(http://.*lsp)"} page (push $1 links) 0)
(dolist (link links)
(set 'file (last (parse link "/")))
(write-file file (get-url link))
(println "->" file))Another technique for tokenizing text uses parse. While with replace the regular expression defined the token with parse the regex pattern describes the space between the tokens:
; tokenize using parse (set 'str "1 2,3,4 5, 6 7 8"
(parse str ",\\ *|\\ +,*" 0) => ("1" "2" "3" "4" "5" "6" "7" "8")
When appending strings append and join can be used to form a new string:
(set 'lstr (map string (rand 1000 100)))
=> ("976" "329" ... "692" "425")
;; the wrong slowest way
(dolist (s lstr)
(set 'bigStr (append bigStr s)))
;; smarter way - 50 times faster
;;
(apply append lstr)Sometimes strings are not readily available in a list like in the above examples. In this case push can be used to push strings on a list while they get produced. The list then can be used as an argument for join, making the fastest method for putting strings together from existing pieces:
;; smartest way - 300 times faster ;; join an existing list of strings ;; (join lstr) => "97632936869242555543 ...." ;; join can specify a string between the elements ;; to be joined (join lstr "-") => "976-329-368-692-425-555-43 ...."
Very often the best method is to grow a string in place. The functions write-buffer, write-line and push can be used not only to write to file handles but also to write/append to existing strings:
;; smartest way - 150 times faster ;; grow a string in place ;;
;; using write-buffer (set 'bigStr "") (dolist (s lstr) (write-buffer bigStr s))
;; using push
(set 'bigStr “”)
(dolist (s lstr) (push s bigStr -1))
The function select for selecting elements from lists can also be used to select and re-arrange characters from strings:
(set 'str "eilnpsw") (select str '(3 0 -1 2 1 -2 -3)) => "newlisp"
After scanning and tokenizing text frequently dictionaries are built to store tokens together with their frequencies or other statistics or attributes.
; create a symbol from a string and set it to a value ; (set 'token "hello") (set (sym token 'Words) '(0 0 0 0)) ; a shorter method since 8.7.4 (context 'Words token '(0 0 0 0)) ; read contents of a word in dictionary ; (eval (sym token 'Words)) => (0 0 0 0) ; a shorter method since 8.7.4 (context 'Words token) => (0 0 0 0) ; update a word, using the old data ; incrementing a counter by one ; (nth-set 0 (eval (sym token 'Words)) (+ $0 1)) => 0 ; the old value (nth-set 0 (eval (sym token 'Words)) (+ $0 1)) => 1 (nth-set 0 (eval (sym token 'Words)) (+ $0 1)) => 2
; same as previous, but alternative shorter syntax (nth-set ((context 'Word token) 0) (+ $0 1)) => 3 (nth-set ((context 'Word token) 0) (+ $0 1)) => 4 (eval (sym token 'Words)) => (5 0 0 0) ; same in alternative shorter syntax
(context 'Words token) => (5 0 0 0)
The dictionary can be easily saved to a file by serializing the context Words:
; save the dictionary to a file ; (save "words.db" 'Words)
Just as easy you can reload the dictionary:
; load dictionary from a file ; (load "words.db")
The complexity of dictionary manipulation can greatly be reduced using an object oriented implementation:
(context 'Words)
(define (put token properties) (set (sym token) properties))
(define (get token) (eval (sym token))
(define (update token col value) (nth-set col (eval (sym token)) (+ $0 value)))
(define (remove token) (delete (sym token)))
(context 'MAIN)
Code for creating, accessing modifying token is much easier now.
; create / initialize token (Words:put "hello" '(0 0 0 0)) ; access word (Words:get "hello") => (0 0 0 0)
; update word ; increment column 0 by 1 (Words:update "hello" 0 1) => 0 (Words:update "hello" 0 4) => 1 (Words:get "hello") => 5
; remove a word (Words:remove "hello") => true
Note that in a real world implementation words or functions would be prefixed by a special character, so they are not confused with each other.
In this pattern the server keeps the connection open until the client closes the connection, then the server loops into a new listen:
;;;;;;;;;;;;;;;;;;;;;; the server
; maximum bytes to receive (constant 'max-bytes 1024)
(if (not (set 'listen (net-listen 123))) (print (net-error)))
(while (not (net-error)) (set 'connection (net-accept listen)) ;; blocking here (while (not (net-error)) (net-receive connection 'message-from-client max-bytes) .... process message from client ... .... configure message to client ... (net-send connection message-to-client)) )
;;;;;;;;;;;;;;;;;;;;; the client
; client connect
(if (not (set 'connection (net-connect "host.com" 123))) (println (net-error)))
; maximum bytes to receive (constant 'max-bytes 1024)
; message send-receive loop (while (not (net-error)) .... configure message to server ... (net-send connection message-to-server) (net-receive connection 'message-from-server max-bytes) .... process message-from-server ... )
In this pattern the server closes the connection after each transaction exchange of messages.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; server
(while (not (net-error)) (set 'connection (net-accept listen)) ;; blocking here (net-receive connection 'message-from-client max-bytes) .... process message from client ... .... configure message to client ... (net-send connection message-to-client) (close connection))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; client
(if (not (set 'connection (net-connect "host.com" 123))) (println (net-error)))
; maximum bytes to receive (constant 'max-bytes 1024) .... configure message to server ... (net-send connection message-to-server) (net-receive connection 'message-from-server max-bytes) .... process message-from-server ...
There are many different ways to set up a client/server connection, see also the examples in the newLISP manual.
They are fast and need less setup than TCP/IP and offer ''multi casting''. UDP is also less reliable because the protocol does less checking, i.e. of correct packet sequence or if all packets are received. This is normally no problem when not working on the Internet but in a well controlled local network or when doing machine control. A simple more specific protocol could be made part of the message.
In this example the server keeps the connection open. UDP communications with net-listen, net-receive-from and net-send-to can block on receiving.
;;;;;;;;;;;;;;;;;; server
(set 'socket (net-listen 10001 "" "udp"))
(if socket (println "server listening on port " 10001) (println (net-error)))
(while (not (net-error)) (set 'msg (net-receive-from socket 255)) (println "->" msg) (net-send-to (nth 1 msg) (nth 2 msg) (upper-case (first msg)) socket))
;;;;;;;;;;;;;;;;;; client
(set 'socket (net-listen 10002 "" "udp"))
(if (not socket) (println (net-error)))
(while (not (net-error)) (print "->") (net-send-to "127.0.0.1" 10001 (read-line) socket) (net-receive socket 'buff 255) (println "=>" buff))
This form is some times used for controlling hardware or equipment. No setup is required, just one function for sending, another one for receiving:
;;;;;;;;;;;;;;;;;; server
;; wait for data gram with maximum 20 bytes
(net-receive-udp 1001 20)
;; or
(net-receive-udp 1001 20 5000000) ;; wait for max 5 seconds
;;;;;;;;;;;;;;;;;; client
(net-send-udp "host.com" 1001 "Hello")
Win32 and Linux show different behavior when sending less or more bytes then specified on the receiving end.
In this scheme the server subscribes to one of a range of multi cast addresses using the net-listen function.
;; example server
(net-listen 4096 "226.0.0.1" "multi") => 5
(net-receive-from 5 20)
;; example client
(net-connect "226.0.0.1" 4096 "multi") => 3
(net-send-to "226.0.0.1" 4096 "hello" 3)
The connection in the example is blocking on net-receive but could be de-blocked using net-select or net-peek
In all previous patterns the client blocks when in receive. The net-select call can be used to ''unblock'' communications:
;; optionally poll for arriving data with 100ms timeout
(while (not (net-select connection "r" 100000)) (do-something-while-waiting ...)) (net-receive...)
connection can be a single number for a connection socket or a list of numbers to wait on various sockets.
(while ( = (net-peek aSock) 0) (do-something-while-waiting ...))
(net-receive...)
In this chapter all external applications are launched using process. This function returns immediately after launching the other application and does not block.
In all of the following patterns the server is not independent but controlled by the client, which launches the server and then communicated via a line oriented protocol:
-> launch server
-> talk to server
<- wait for response from server
-> talk to server
<- wait for response from server
.....Often a sleep time is necessary on the client side to wait for the server to be ready loading. Most of these examples are condensed snippets from GTK-Server from www.gtk-server.org.
process allows specifying 2 pipes for communications with the launched application.
# Define communication function
(define (gtk str) (write-line str myout) (if (!= str "gtk_exit 0") (read-line myin)))
# Launch gtk-server
(map set '(myin gtkout) (pipe)) (map set '(gtkin myout) (pipe)) (process "gtk-server stdin" gtkin gtkout) (gtk "gtk_init NULL NULL") (gtk "gtk_window_new 0") .....
# Define communication function
(define (gtk str , tmp) (net-send connection str) (net-receive connection 'tmp 64) tmp) # Start the gtk-server
(process "gtk-server tcp localhost:50000") (sleep 1000)
# Connect to the GTK-server
(set 'connection (net-connect "localhost" 50000)) (set 'result (gtk "gtk_init NULL NULL")) (set 'result (gtk "gtk_window_new 0")) .....
Make a FIFO first (looks like a special file node):
(exec "mkfifo myfifo")
;; or alternatively
(import "/lib/libc.so.6" "mkfifo") (mkfifo "/tmp/myfifo" 0777)
; Define communication function
(define (gtk str) (set 'handle (open "myfifo" "write")) (write-buffer handle str) (close handle) (set 'handle (open "myfifo" "read")) (read-buffer handle 'tmp 20) (close handle) tmp)
Note that the listen function with "udp" option just binds the sockets to a address/hardware but not actually listens as in TCP/IP.
# Define communication function
(define (gtk str , tmp) (net-send-to "localhost" 50000 str socket) (net-receive socket 'tmp net-buffer) tmp)
# Start the gtk-server
(define (start) (process "gtk-server udp localhost:50000") (sleep 500) (set 'socket (net-listen 50001 "localhost" "udp")) ) (set 'result (gtk "gtk_init NULL NULL")) (set 'result (gtk "gtk_window_new 0")) .....
This is frequently used from newLISP's interactive command line to execute processes in a blocking fashion, which need a shell to run:
(! "ls -ltr")
There is an interesting variant of this form working not inside a newLISP expression, but only on the command line:
!ls -ltr
The ! should be the first character on the command line. This form works like as shell escape like in the VI editor. It is useful for invoking an editor or doing quick shell work without completely leaving the newLISP console.
(exec "ls /") => ("bin" "etc" "home" "lib")
(exec "script.cgi" cgi-input)
In this example cgi-input contains a string feeding. i.e.: a query input, normally coming from a web server. Note that output in this case is written directly to the screen, and cannot be returned to newLISP. Use process and pipe for two way std i/o communications with other applications.
Shared memory, semaphores and threads work frequently together. Semaphores can synchronize tasks in different threads and shared memory can be used to communicate between threads.
The following is a more complex example showing the working of all three mechanisms at the same time.
The producer loops through all n values from i = 0 to n - 1 and puts each value into shared memory where it is picked up by the consumer thread. Semaphores are used to signal that a data value is ready for reading.
#!/usr/bin/newlisp # prodcons.lsp - Producer/consumer # # usage of 'fork', 'wait-pid', 'semaphore' and 'share'
(if (> (& (last (sys-info)) 0xF) 4) (begin (println "this will not run on Win32") (exit)))
(constant 'wait -1 'signal 1 'release 0)
(define (consumer n) (set 'i 0) (while (< i n) (semaphore cons-sem wait) (println (set 'i (share data)) " <-") (semaphore prod-sem signal)) (exit)) (define (producer n) (for (i 1 n) (semaphore prod-sem wait) (println "-> " (share data i)) (semaphore cons-sem signal)) (exit))
(define (run n) (set 'data (share)) (share data 0) (set 'prod-sem (semaphore)) ; get semaphores (set 'cons-sem (semaphore)) (set 'prod-pid (fork (producer n))) ; start threads (set 'cons-pid (fork (consumer n))) (semaphore prod-sem signal) ; get producer started (wait-pid prod-pid) ; wait for threads to finish (wait-pid cons-pid) ; (semaphore cons-sem release) ; release semaphores (semaphore prod-sem release))
(run 10)
(exit)
While association lists are the traditional means for associative data access in LISP and newLISP, other modern scripting languages use hashing to build memory based associative data tables. A hash is a table position calculated from a association key.
newLISP uses symbols instead of hashes. Symbols in name spaces can also be serialized to a file, as will be shown in the chapter about symbol creation and lookup.
The association list is a classic LISP data structure for storing information for associative retrieval:
;; creating association lists
;; pushing at the end with -1 is optimized and
;; as fast as pushing in front
(push '("John Doe" "123-5555" 1200.00) Persons -1)
(push '("Jane Doe" "456-7777" 2000.00) Persons -1)
.....
Persons => (
("John Doe" "123-5555" 1200.00)
("Jane Doe" "456-7777" 2000.00) ...)
;; access/lookup data records
(assoc "John Doe" Persons)
=> ("John Doe" "123-5555" 1200.00 male)
(assoc "Jane Doe" Persons)
=> ("Jane Doe" "456-7777" 2000.00 female)
newLISP has a lookup function similar to what is used in
spreadsheet software. This function which works like a combination
of assoc and nth can find the association
and pick a specific member of the data record at the same time:
(lookup "John Doe" Persons 0) => "123-555" (lookup "John Doe" Persons -1) => make (lookup "Jane Doe" Persons 1) => 2000.00 (lookup "Jane Doe" Persons -2) => 2000.00
;; update data records
(replace-assoc "John Doe" Persons '("John Doe" "123-5555" 900.00 male))
;; replace as a function of existing/replaced data
(replace-assoc "John Doe" Persons (update-person $0))
;; delete data records
(replace (assoc "John Does" Persons) Persons)
lookup and assoc are fine for small tables and databases with only a few hundred data records, but are too slow for look-ups in thousands or millions of records.
newLISP has a fast and highly scalable symbol implementation. Together with name spaces and object serialization symbols can be used to maintain large data sets in memory and on disk with associative access.
;; creating the database
(set (sym "John Doe" 'Persons) '("123-5555" 1200.00 male)) (set (sym "Jane Doe" 'Persons) '("456-7777" 2000.00 female))
;; alternative shorter syntax
(context 'Persons "John Doe") '("123-5555" 1200.00 male))
(context 'Persons "Jane Doe") '("456-7777" 2000.00 female))
..... ;; save the database to a file (serialization of context Persons)
(save "persons.db" 'Persons)
;; load database from a file
(load "persons.db")
;; access data records
(eval (sym "John Doe" 'Persons)) => '("123-5555" 1200.00 male)
;; alternative shorter syntax to access data records
(context 'Persons "John Doe") => '("123-5555" 1200.00 male)
;; update data records
(set (sym "John Doe" 'Persons) '("John Doe" "123-5555" 900.00 male))
;; update using alternative syntax
(context 'Persons "John Doe") => '("John Doe" "123-5555" 900.00 male))
;; delete data records (delete (sym "John Doe" 'Persons))
newLISP has an import function, which allows importing function from DLLs (Dynamic Link Libraries) on Win32 or shared libraries on Linux/UNIX (ending in .so).
This chapter shows how to compile and use libraries on both, Win32 and Linux/UNIX platforms. We will compile a DLL and a Linux/UNIX shared library from the following 'C' program:
#include <stdio.h> #include <stdlib.h> #include <ctype.h>
int foo1(char * ptr, int number) { printf("the string: %s the number: %d\n", ptr, number); return(10 * number); }
char * foo2(char * ptr, int number) { char * upper; printf("the string: %s the number: %d\n", ptr, number); upper = ptr; while(*ptr) { *ptr = toupper(*ptr); ptr++; } return(upper); }
/* eof */
Both functions foo1 and foo2 print their arguments, but while foo1 returns the number multiplied 10 times, foo2 returns the uppercase of a string to show how to return strings from 'C' functions.
On Linux/UNIX we can compile and link testlib.so in one step:
gcc testlib.c -shared -o testlib.so
Or On Mac OSX/darwin do:
gcc -bundle -o testlib.so
The library testlib.so will be built with Linux/UNIX default cdecl conventions. Importing the library is very similar on both Linux and Win32 platforms, but on Win32 the library can be found in the current directory. You may have to specify the full path or put the library in the library path of the os:
newLISP v.8.4.3 on Linux, execute 'newlisp -h' for more info. > (import "/home/newlisp/testlib.so" "foo1") foo1 <6710118F> > (import "/home/newlisp/testlib.so" "foo2") foo2 <671011B9> > (foo1 "hello" 123) the string: hello the number: 123 1230 > (foo2 "hello" 123) the string: hello the number: 123 4054088 > (get-string (foo2 "hello" 123)) the string: hello the number: 123 "HELLO" >
Again, the number returned from foo2 is the string address pointer
and get-string can be used to access the string. When
using get-string only character up to a zero byte are
returned. When returning the addresses to binary buffers different
techniques using unpack are used to access the
information.
DLLs on Win32 can be made using the MinGW, Borland or CYGWIN compilers. This example shows, how to do it using the MinGW compiler.
Compile it:
gcc -c testlib.c -o testlib.o
Before we can transform testlib.o into a DLL we need a testlib.def declaring the exported functions:
LIBRARY testlib.dll
EXPORTS
foo1 @1 foo1
foo2 @2 foo2Now wrap the DLL:
dllwrap testlib.o --def testlib.def -o testlib.dll -lws2_32
The library testlib.dll will be built with default Win32 stdcall conventions. The following shows an interactive session, importing the library and using the functions:
newLISP v.8.4.3 on Win32 MinGW, execute 'newlisp -h' for more info. > (import "testlib.dll" "foo1") foo1 <6710118F> > (import "testlib.dll" "foo2") foo2 <671011B9> > (foo1 "hello" 123) the string: hello the number: 123 1230 > (foo2 "hello" 123) the string: hello the number: 123 4054088 > (get-string (foo2 "hello" 123)) the string: hello the number: 123 "HELLO" > ; import a library compiled for cdecl ; calling conventsions > (import "foo.dll" "func" "cdecl")
Note that the first time using foo2 the return value 4054088 is the
memory address of the string returned. Using get-string
the string belonging to it can be accessed. If the library is
compiled using cdecl calling conventions, the cdecl
keyword must be used in the import expression.
Just like 'C' strings are returned using string pointers, 'C'
structures can be returned using structure pointers and functions
like get-string, get-int or get-char
can be used to access the members. The following example illustrates
this:
typedef struct mystruc {
int number;
char * ptr;
} MYSTRUC;
MYSTRUC * foo3(char * ptr, int num )
{
MYSTRUC * astruc;
astruc = malloc(sizeof(MYSTRUC));
astruc->ptr = malloc(strlen(ptr) + 1);
strcpy(astruc->ptr, ptr);
astruc->number = num;
return(astruc);
}The newLISP program would access the structure members as follows:
> (set 'astruc (foo3 "hello world" 123)) 4054280 > (get-string (get-integer (+ astruc 4))) "hello world" > (get-integer astruc) 123 >
The return value from foo is the address to the structure astruc.
To access the string pointer, 4 must be added as the size of an
integer type in the 'C' programming language. The string in the
string pointer then gets accessed using get-string.
Sometimes data structures contain data types of different length than the normal CPU register word:
struct mystruct {
short int x;
int z;
short int y;
} data;
struct mystruct * foo(void)
{
data.x = 123;
data.y = 456;
data.z = sizeof(data);
return(&data);
}The x and y variables are 16 bit wide and only z takes 32 bit. When a compiler on a 32bit CPU packs this structure the variables x and y will each fill up 32 bits instead of the 16 bit each. This is necessary so the 32 bit variable z can be aligned properly. The following code would be necessary to access the structure members:
> (import "/usr/home/nuevatec/test.so" "foo") foo <281A1588> > (unpack "lu lu lu" (foo)) (123 12 456)
The whole structure consumes 3 by 4 = 12 bytes, because all members have to be aligned to 32 bit borders in memory.
The following data structure packs the short 16 bit variables next to each other. This time only 8 bytes are required: 2 each for x and y and 4 bytes or z. Because x and y are together in one 32 bit word, none of the variables needs to cross a 32 bit boundary:
struct mystruct {
short int x;
short int y;
int z;
} data;
struct mystruct * foo(void)
{
data.x = 123;
data.y = 456;
data.z = sizeof(data);
return(&data);
}This time the access code in newLISP reflects the size of the structure members:
> (import "/usr/home/nuevatec/test.so" "foo") foo <281A1588> > (unpack "u u lu" (foo)) (123 456 8)
|
Data Type |
newLISP call |
'C' Function call |
|---|---|---|
|
Integer |
(foo 123) |
foo(int number) |
|
Double Float |
(foo 1.234) |
foo(double number) |
|
Float |
(foo (flt 1.234)) |
foo(float number) |
|
String |
(foo "Hello World!") |
foo(char * string) |
|
Integer [] |
(foo (pack "d d d" 123 456 789)) |
foo(int numbers[]) |
|
Float [] |
(foo (pack "f f f" 1.23 4.56 7.89)) |
foo(float[]) |
|
Double [] |
(foo (pack "lf lf lf) 1.23 4.56 7.89) |
foo(double[]) |
|
String [] |
(foo (pack "ld ld ld |
foo(char * string[]) |
|
Data Type |
newLISP to extract return value |
'C' return |
|---|---|---|
|
Integer |
(set 'number (foo x y z)) |
return(int number) |
|
Double Float |
n/a - only 32bit returns |
return(double number) |
|
Float |
(set 'number (first (unpack "f" (pack "ld" (foo x y z))))) |
return(float number) |
|
String |
(set 'string (get-string (foo x y z) |
return(char * string) |
|
Integer [] |
(set 'numList (unpack "ld ld ld" (foo x y z))) |
return(int numList[]) |
|
Float [] |
(set 'numList (unpack "f f f" (foo x y z))) |
return(float numList[]) |
|
Double [] |
(set 'numList (unpack "lf lf lf") (foo x y z))) |
return(double numList[]) |
|
String [] |
(set 'stringList (map get-string (unpack "ld ld ld" (foo x y z)))) |
return(char * string[]) |
When returning array types the number of elements in the array must be known. The examples always assume 3 elements.
All pack and unpack and formats can
also be given without spaces, but are spaced in the examples for
better readability.
The formats "ld" and "lu" are interchangeable, but the 16 bit formats "u" and "d" may produce different results, because of sign expansion from 16 to 32 bits.
Flags are available for changing endian byte order during
pack and unpack.
- fin -
Version 1.2, November 2002
Copyright (C) 2000,2001,2002 Free Software Foundation, Inc. 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
0. PREAMBLE
The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.
This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.
We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.
1. APPLICABILITY AND DEFINITIONS
This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.
A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.
A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.
The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.
The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.
A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".
Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.
The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.
A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.
The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.
2. VERBATIM COPYING
You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.
You may also lend copies, under the same conditions stated above, and you may publicly display copies.
3. COPYING IN QUANTITY
If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.
If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.
If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.
It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.
4. MODIFICATIONS
You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:
A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.
B. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.
C. State on the Title page the name of the publisher of the Modified Version, as the publisher.
D. Preserve all the copyright notices of the Document.
E. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.
F. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.
G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.
H. Include an unaltered copy of this License.
I. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.
J. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.
K. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.
L. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.
M. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.
N. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.
O. Preserve any Warranty Disclaimers.
If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.
You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.
You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.
The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.
5. COMBINING DOCUMENTS
You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.
The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.
In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements."
6. COLLECTIONS OF DOCUMENTS
You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.
You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.
7. AGGREGATION WITH INDEPENDENT WORKS
A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.
If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.
8. TRANSLATION
Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.
If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.
9. TERMINATION
You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.
10. FUTURE REVISIONS OF THIS LICENSE
The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.
Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.