doc/CodePatterns.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
   2 <html>
   3 <head>
   4    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
   5    <meta name="author" content="Lutz Mueller">
   6    <meta name="keywords"
   7          content="newLISP LISP SCHEME programming language manual reference Artificial Intelligence AI NUEVATEC">
   8    <meta name="description" content="newLISP Code Patterns">
   9    <title>newLISP Code Patterns</title>
  10
  11 <style type="text/css" media="screen">
  12
  13 <!--
  14
  15 .divider {
  16         margin-top: 2em;
  17         margin-bottom: 1em;
  18         font-family: Times New Roman, Times, serif;
  19         color: #ffAA28;
  20         }
  21
  22 .title {
  23         font-family:Optima, Georgia, Times New Roman, Times, serif;
  24         font-size:300%;
  25         font-weight: 500;
  26         }
  27
  28 .trade {
  29         font-family:Optima, Georgia, Times New Roman, Times, serif;
  30         font-size:100%;
  31         padding-top:16px;
  32         }
  33
  34 span.arw {
  35         color:#666666;
  36         font-size: 100%;
  37         }
  38
  39 body, h1, h2, h3, h4, p {
  40         font-family: Georgia, Times New Roman, Times, serif;
  41         line-height: 120%;
  42         }
  43
  44 table.list {
  45         border-width: 0px;
  46         padding: 3px;
  47         }
  48
  49 th.list {
  50         font-family:Georgia, Times New Roman, Times, serif;
  51         font-size:0.8em;
  52         font-weight: 700;
  53         }
  54
  55 td.left {
  56         font-family: Andale Mono, Bitstream Vera Sans Mono, Monaco, Courier New;
  57         font-size: 100%;
  58         }
  59
  60 td.right {
  61         font-family:Georgia, Times New Roman, Times, serif;
  62         font-size: 100%;
  63         font-style: italic;
  64         }
  65
  66 pre {
  67         font-family: Andale Mono, "Bitstream Vera Sans Mono", Monaco, "Courier New";
  68         font-size: 100%;
  69         }
  70
  71 tt {
  72         font-family: Andale Mono, "Bitstream Vera Sans Mono", Monaco, "Courier New";
  73         font-size: 100%;
  74         }
  75
  76 -->
  77
  78 </style>
  79 </head>
  80 <body style="margin: 20px;" text="#000000" bgcolor="#FFFFFF" link="#376590" vlink="#551A8B" alink="#ffAA28">
  81
  82 <br><br>
  83
  84 <center>
  85 <h1 class="title">Code Patterns in newLISP<sup><font size="-2">TM</font></sup></h1>
  86 Version 2008 March 2nd<sup>th</sup><br>
  87 <a href="http://newlisp.org">newLISP</a> v.9.3 and after
  88 </center>
  89
  90 <br><br><br>
  91
  92 <center>
  93 <span style="line-height:80%;">
  94 <font size="-2">
  95 Copyright &copy; 2008 Lutz Mueller, <a href="http://www.nuevatec.com">www.nuevatec.com</a>. All rights reserved.<br>
  96 Permission is granted to copy, distribute and/or modify this document under the terms of the <a href="#GFDL">GNU Free Documentation License</a>,<br>
  97 Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts,<br>
  98 and no Back-Cover Texts. A copy of the license is included in the section entitled GNU Free Documentation License.<br>
  99 <br>
 100 newLISP is a trademark of Lutz Mueller.
 101 </font>
 102 </span>
 103 </center>
 104
 105 <br><br>
 106
 107 <blockquote>
 108 <center><h2>Contents</h2></center>
 109 <ol>
 110 <li><a href="#intro">Introduction</a></li>
 111 <li><a href="#scripts">newLISP script files</a></li>
 112 <ul>
 113         <li>Specifying command line options in script files</li>
 114         <li>Scripts as pipes</li>
 115         <li>File filters</li>
 116 </ul>
 117 <li><a href="#modules">Writing software in modules</a></li>
 118 <ul>
 119         <li>Sructuring an application</li>
 120         <li>More than one context per file</li>
 121         <li>Simple context objects and the default function</li>
 122         <li>Packaging data with contexts</li>
 123         <li>Passing context objects by reference</li>
 124 </ul>
 125 <li><a href="#locals">Local variables</a></li>
 126 <ul>
 127         <li>Locals in looping functions</li>
 128         <li>Local symbols using <tt>let</tt>, <tt>letn</tt> and <tt>local</tt></li>
 129         <li>Unused parameters as local symbols</li>
 130         <li>Using <tt>args</tt> as local substitute</li>
 131 </ul>
 132 <li><a href="#walking">Walking through lists</a></li>
 133 <ul>
 134         <li>Recursion or not?</li>
 135         <li>Generators</li>
 136         <li>Speed up with memoization</li>
 137         <li>Walking a tree</li>
 138         <li>Walking a directory tree</li>
 139 </ul>
 140 <li><a href="#creating">Creating, accessing, modifying and searching lists</a></li>
 141 <ul>
 142         <li><tt>push</tt> and <tt>pop</tt></li>
 143         <li>Accessing lists</li>
 144         <li>Selecting more than one element</li>
 145     <li>Filtering and differencing lists</li>
 146         <li>Changing list elements</li>
 147         <li>The new element as a function of the replaced</li>
 148         <li>Search and replace in simple lists</li>
 149         <li>Search and replace in nested lists</li>
 150         <li>Passing lists by reference using the default functor</li>
 151 </ul>
 152 <li><a href="#flow">Program flow</a></li>
 153 <ul>
 154         <li>Loops</li>
 155         <li>Blocks</li>
 156         <li>Branching</li>
 157         <li>Fuzzy flow</li>
 158         <li>Change flow with <tt>catch</tt> and <tt>throw</tt></li>
 159         <li>Leave loops with a break condition</li>
 160         <li>Change flow with <tt>and</tt> or <tt>or</tt></li>
 161 </ul>
 162 <li><a href="#error">Error handling</a></li>
 163 <ul>
 164         <li>System errors</li>
 165         <li>User defined errors</li>
 166         <li>Error event handlers</li>
 167         <li>Catching errors</li>
 168 </ul>
 169 <li><a href="#functions">Functions as data</a></li>
 170 <ul>
 171         <li>Manipulate functions after definition</li>
 172         <li>Mapping and applying functions</li>
 173         <li>Function currying: functions making functions</li>
 174 </ul>
 175 <li><a href="#text">Text processing</a></li>
 176 <ul>
 177         <li>Regular expressions</li>
 178         <li>Scanning text</li>
 179         <li>Appending strings</li>
 180         <li>Growing strings in place</li>
 181         <li>Rearranging strings</li>
 182 </ul>
 183 <li><a href="#dicts">Dictionaries</a></li>
 184 <ul>
 185     <li>Creating and modifying symbols</li>
 186     <li>Iterating through dictionaries</li>
 187         <li>Object oriented implementation of a dictionary</li>
 188 </ul>
 189 <li><a href="#tcpip">TCP/IP client server communications</a></li>
 190 <ul>
 191         <li>Client - server TCP/IP - open connection</li>
 192         <li>Client - server TCP/IP - closed connection</li>
 193 </ul>
 194 <li><a href="#udp">UDP communications</a></li>
 195 <ul>
 196         <li>Open communications with UDP</li>
 197         <li>Closed transaction oriented UDP</li>
 198         <li>UDP multi-cast communications</li>
 199 </ul>
 200 <li><a href="#nonblock">Non-blocking communications</a></li>
 201 <ul>
 202         <li>Using net-select</li>
 203         <li>Using net-peek</li>
 204 </ul>
 205 <li><a href="#controlling">Controlling other apps</a></li>
 206 <ul>
 207         <li>Communications via STD I/O</li>
 208         <li>Communications via TCP/IP</li>
 209         <li>Communicate via named FIFO</li>
 210         <li>Communicate via UDP</li>
 211 </ul>
 212 <li><a href="#launch">Launching applications blocking</a></li>
 213 <ul>
 214         <li>Shell execution</li>
 215         <li>Execution capturing std-out in a list</li>
 216         <li>Execution feeding std-in</li>
 217 </ul>
 218 <li><a href="#threads">Threads, semaphores and shared memory</a></li>
 219 <li><a href="#databases">Databases, lookup tables</a></li>
 220 <ul>
 221         <li>Association lists</li>
 222         <li>Nested associations</li>
 223         <li>Updating nested associations</li>
 224         <li>Symbol creation and lookup</li>
 225 </ul>
 226 <li><a href="#distributed">Distributed computing</a></li>
 227 <ul>
 228         <li>Setting up a newLISP server node</li>
 229         <li>Evaluating multiple expressions remotely</li>
 230         <li>Transferring files to and from remote nodes</li>
 231         <li>Loading and saving code from and to remote nodes</li>
 232     <li>HTTPD-only mode, newLISP as a webserver</li>
 233     <li>Media types in HTTP modes</li>
 234         <li>Environment variables set</li>
 235         <li>Local domain UNIX sockets</li>
 236 </ul>
 237 <li><a href="#extending">Extending newLISP</a></li>
 238 <ul>
 239         <li>Compile a shared library on Linux/UNIX/MacOSX</li>
 240         <li>Compile a DLL on Win32</li>
 241         <li>Using data structures in imported libraries</li>
 242         <li>Unevenly aligned data structures</li>
 243         <li>Passing parameters in library calls</li>
 244         <li>Extracting return values from library calls</li>
 245 </ul>
 246 <li><a href="#appendix">Appendix</a></li>
 247 <ul>
 248         <li>GNU Free Documentation License</li>
 249 </ul>
 250 </ol>
 251 </blockquote>
 252
 253 <br>
 254 <center>&sect;</center>
 255 <a name="intro"></a>
 256 <h2>1. Introduction</h2>
 257 <p>When programming in newLISP certain functions and usage patterns occur repeatedly. For some problems an optimal way to solve them evolves over time. The following chapters present example code and explanations for the solution of specific problems when programming in newLISP.</p>
 258
 259 <p>Some content is overlapping with material covered in the newLISP Users Manual and Reference or presented here with a different slant.</p>
 260
 261 <p>Only a subset of newLISP's total function repertoire is used here. Some functions demonstrated have additional calling patterns or applications not mentioned on these pages.</p>
 262
 263 <p>This collection of patterns and solutions is a work in progress. Over time material will be added or existing material improved.</p>
 264 <center>&sect;</center>
 265 <br>
 266
 267 <a name="scripts"></a>
 268 <h2>2. newLISP script files</h2>
 269
 270 <h3>Specifying command line options in script files</h3>
 271 <p>On Linux/UNIX put the following in the first line of the script/program file:</p>
 272
 273 <pre>
 274         #!/usr/bin/newlisp
 275 </pre>
 276
 277 <p>specifying a bigger stack: </p>
 278
 279 <pre>
 280         #!/usr/bin/newlisp -s 100000
 281 </pre>
 282
 283 <p>or</p>
 284
 285 <pre>
 286         #!/usr/bin/newlisp -s100000
 287 </pre>
 288
 289 <p>Operating systems shells behave differently when parsing the first line and extract parameters. newLISP takes both, attached or detached parameters. Put the following lines in small script to test the behavior of the underlying OS and platform. The script changes the stack size allocated to 100,000 and limits LISP cell memory to about 10 M bytes.</p>
 290
 291 <pre>
 292         #!/usr/bin/newlisp -s 100000 -m 10
 293
 294         (println (main-args))
 295         (println (sys-info))
 296 </pre>
 297
 298 <p>A typical output executing the script from the system shell would be:</p>
 299
 300 <pre>
 301         ./arg-test
 302
 303         ("/usr/bin/newlisp" "-s" "100000" "-m" "10" "./arg-test")
 304         (308 655360 299 2 0 100000 8410 2)
 305 </pre>
 306
 307 <p>Note that few programs in newLISP need a bigger stack configured, most programs run on the internal default of 2048. Each stack position takes an average of 80 bytes. Other options are available to start newLISP. See the Users Manual for details.</p>
 308
 309 <h3>Scripts as pipes</h3>
 310
 311 <pre>
 312         The following examples shows how a file can be piped into a newLISP script.
 313         #!/usr/bin/newlisp
 314         #
 315         # uppercase - demo filter script as pipe
 316         #
 317         # usage:
 318         #          ./uppercase &lt; file-spec
 319         #
 320         # example:
 321         #          ./uppercase &lt; my-text
 322         #
 323         #
 324
 325         (while (read-line) (println (upper-case (current-line))))
 326
 327         (exit)
 328 </pre>
 329
 330
 331 <p>The file will be printed to stdout translated to uppercase.</p>
 332
 333 <h3>File filters</h3>
 334
 335 <p>The following script works like a Unix grep utility iterating through files and filtering each line in a file using a regular expression pattern.</p>
 336
 337 <pre>
 338         #!/usr/bin/newlisp
 339         #
 340         # nlgrep - grep utility on newLISP
 341         #
 342         # usage:
 343         #          ./nlgrep "regex-pattern" file-spec
 344         #
 345         # file spec can contain globbing characters
 346         #
 347         # example:
 348         #          ./nlgrep "this|that" *.c
 349         #
 350         # will print all line containing 'this' or 'that' in *.c files
 351         #
 352
 353         (dolist (file-name (3 (main-args)))
 354                 (set 'file (open file-name "read"))
 355                 (println "file ---> " file-name)
 356                 (while (read-line file)
 357                         (if (find (main-args 2) (current-line) 0)
 358                         (write-line)))
 359                 (close file))
 360
 361         (exit)
 362 </pre>
 363
 364 <p>The expression:</p>
 365 <pre>
 366         (3 (main-args))
 367 </pre>
 368
 369 <p>is a short form of writing:</p>
 370
 371 <pre>
 372         (rest (rest (rest (main-args))))
 373 </pre>
 374
 375 <p>It returns a list of all the filenames. This form of specifying indices for rest is called implicit indexing. See the Users Manual for implicit indexing with other functions.
 376 The expression <tt>(main-args 2)</tt> extracts the 3rd argument from the commandline containing the regular expression pattern.</p>
 377 <center>&sect;</center>
 378
 379 <br>
 380 <a name="modules"></a>
 381 <h2>3. Writing software in modules</h2>
 382
 383
 384 <h3>Structuring an application</h3>
 385
 386 <p>When writing bigger applications or when several programmers are working on the same code base it is necessary to divide the code base into modules. Modules in newLISP are implemented using contexts, which are namespaces. Namespaces allow lexical isolation between modules. Variables of the same name in one module cannot clash with variables of the same name in another module.</p>
 387
 388 <p>Typically moudules will be organized in one context per file. One file module may contain database access routines.</p>
 389
 390 <pre>
 391         ; database.lsp
 392         ;
 393         (context 'db)
 394
 395
 396         (define (update x y z)
 397         ...
 398         )
 399
 400         (define (erase x y z)
 401         ...
 402         )
 403 </pre>
 404
 405 <p>Another module may contain various utilities</p>
 406
 407 <pre>
 408         ; auxiliary.lsp
 409         ;
 410         (context 'aux)
 411
 412         (define (getval a b)
 413         ...
 414         )
 415 </pre>
 416
 417 <p>Typically there will be one MAIN module loading and controlling all others:</p>
 418
 419 <pre>
 420         ; application.lsp
 421         ;
 422
 423         (load "auxiliary.lsp")
 424         (load "database.lsp")
 425
 426         (define (run)
 427                 (db:update ....)
 428                 (aux:putval ...)
 429                 ...
 430                 ...
 431         )
 432
 433         (run)
 434 </pre>
 435
 436 <h3>More than one context per file</h3>
 437
 438 <p>When using more than one context per file, each context section should be closed with a <tt>(context MAIN)</tt> statement:</p>
 439
 440 <pre>
 441         ; myapp.lsp
 442         ;
 443         (context 'A)
 444
 445         (define (foo ...)
 446         ...
 447         )
 448
 449         (context MAIN)
 450
 451         (context 'B)
 452
 453         (define (bar ...)
 454         ...
 455         )
 456
 457         (context MAIN)
 458
 459         (define (main-func)
 460             (A:foo ...)
 461             (B:bar ...)
 462         ...
 463         )
 464 </pre>
 465
 466 <p>Note that in the namespace statements for contexts <tt>A</tt> and <tt>B</tt> the context names are quoted because they are newly created, while <tt>MAIN</tt> already exists when newLISP starts up and can stay unquoted, although quoting it would not represent a problem.</p>
 467
 468 <h3>Simple objects and the default function</h3>
 469
 470 <p>A function in a context may have the same name as the host context itself. This function has special characteristics:</p>
 471
 472 <pre>
 473         (context 'foo)
 474
 475         (define (foo:foo a b c)
 476         ...
 477         )
 478 </pre>
 479
 480 <p>The function <tt>foo:foo</tt> is called the <em>default function</em>, because when using the context name <tt>foo</tt> like a function it will default to <tt>foo:foo</tt>:</p>
 481
 482 <pre>
 483         (foo x y z)
 484         ; same as
 485         (foo:foo x y z)
 486 </pre>
 487
 488 <p>The default function make is possible to write functions which look like normal functions but carry their own lexical namespace. We can use this to write functions which keep state:</p>
 489
 490 <pre>
 491         (context 'generator)
 492
 493         (set 'acc 0)
 494
 495         (define (generator:generator)
 496                 (inc 'acc))
 497
 498         (context MAIN)
 499
 500         (generator) &rarr; 1
 501         (generator) &rarr; 2
 502         (generator) &rarr; 3
 503 </pre>
 504
 505 <p>The following is a more complex example for a function generating a fibonacci sequence:</p>
 506
 507 <pre>
 508         (define (fibo:fibo)
 509                 (if (not fibo:mem) (set 'fibo:mem '(0 1)))
 510                 (push (+ (fibo:mem -1) (fibo:mem -2)) fibo:mem -1))
 511
 512         (fibo) &rarr; 1
 513         (fibo) &rarr; 2
 514         (fibo) &rarr; 3
 515         (fibo) &rarr; 5
 516         (fibo) &rarr; 8
 517         ...
 518 </pre>
 519
 520 <p>This example also shows how a default function is defined <i>on the fly</i> wihtout the need of explicit <tt>context</tt> statements. As an alternative the function could also have been written creating the context explicitely:</p>
 521
 522 <pre>
 523         (context 'fibo)
 524         (define (fibo:fibo)
 525                         (if (not mem) (set 'mem '(0 1)))
 526                         (push (+ (mem -1) (mem -2)) mem -1))
 527         (context MAIN)
 528
 529         (fibo) &rarr; 1
 530         (fibo) &rarr; 2
 531         (fibo) &rarr; 3
 532         (fibo) &rarr; 5
 533         (fibo) &rarr; 8
 534 </pre>
 535
 536 <p>while the first form is shorter, the second form is more readable.</p>
 537
 538 <h3>Packaging data with contexts</h3>
 539
 540 <p>The previous examples already presented functions packaged with data in a namespace. In the <tt>generator</tt> example the <tt>acc</tt> variable kept state. In the <tt>fibo</tt> example the variable <tt>mem</tt> kept a growing list. In both cases functions and data are living together in a namespace. The follwing example shows how a namespace just holds data only in a  default functor:</p>
 541
 542 <pre>
 543         (set 'db:db '(a "b" (c d) 1 2 3 x y z))
 544 </pre>
 545
 546 <p>Just like we used the default function to refer to <tt>fibo</tt> and <tt>generator</tt> we can refer to the list in <tt>db:db</tt> by only using <tt>db</tt>. This will work in all situations wher we do list indexing:</p>
 547
 548 <pre>
 549         (db 0)    &rarr; a
 550         (db 1)    &rarr; "b"
 551         (db 2 1)  &rarr; d
 552         (db -z)   &rarr; p
 553         (db -x)   &rarr; o
 554
 555         (3 db)    &rarr; (1 2 3 x y z)
 556         (2 1 db)  &rarr; ((c d))
 557         (-6 2 db) &rarr; (1 2)
 558 </pre>
 559
 560 <h3>Passing context objects by reference</h3>
 561
 562 <p>The default functor when used as an argument in a user defined function, is passed by reference. That means that no copy of the list or string is passed but a reference to the original contents. This is useful when handling large lists or strings:</p>
 563
 564 <pre>
 565         (define (update data idx expr)
 566                 (if (not (or (lambda? expr) (primitive? expr)))
 567                         (nth-set (data idx) expr)
 568                         (nth-set (data idx) (expr $0))))
 569
 570         (update db 0 99) &rarr; a
 571         db:db &rarr; (99 "b" (c d) 1 2 3 x y z)
 572
 573         (update db 1 upper-case) &rarr; "b"
 574         db:db &rarr; (99 "B" (c d) 1 2 3 x y z)
 575
 576         (update db 4 (fn (x) (mul 1.1 x))) &rarr;
 577         db:db &rarr; (99 "B" (c d) 1 2.2 3 x y z)
 578 </pre>
 579
 580 <p>The data in <tt>db:db</tt> is passed via the <tt>update</tt> function parameter <tt>data</tt>, which now holds a reference to the context <tt>db</tt>. The <tt>expr</tt> parameter passed is checked if a built-in function, operator or a user defined lambda expression and than works on <tt>$0</tt>, the system variable containing the old content referenced by <tt>(data idx)</tt>.</p>
 581
 582 <p>The function <tt>update</tt> is also a good example how to pass operators or functions as function argument. Read more about thois in chapter <a href="#functions">9. Functions as data</a>.</p>
 583
 584 <p>when using destructive functions which take the un-indexed list like <tt>replace</tt>, <tt>rotate</tt>, <tt>sort</tt> and <tt>swap</tt> the function <tt>default</tt> can be used to extract the default functor and access the list or string in it by reference:</p>
 585
 586 <pre>
 587         (set 'foo:foo '( 6 3 5 8 6 7 4 1))
 588
 589         (define (my-sort clist)
 590                 (sort (eval (default clist))))
 591
 592         (my-sort foo)
 593
 594         foo:foo  &rarr; (1 3 4 5 6 6 7 8)
 595 </pre>
 596
 597 <center>&sect;</center>
 598 <br>
 599
 600 <a name="locals"></a>
 601 <h2>4. Local variables</h2>
 602
 603 <h3>Locals in looping functions</h3>
 604
 605 <p>All looping functions like <tt>dolist</tt>, <tt>dotimes</tt>, <tt>dotree</tt> and <tt>for</tt>, use local variables. During loop execution the variable takes different values, but after leaving the looping function the variable regains its old value. <tt>let</tt>, <tt>define</tt>, and <tt>lambda</tt> expressions are another method for making variables local:</p>
 606
 607 <h3>Local symbols using <tt>let</tt>, <tt>letn</tt> and <tt>local</tt></h3>
 608
 609 <p><tt>let</tt> is the usual way in LISP to declare symbols as local to a block.</p>
 610 <pre>
 611         (define (sum-sq a b)
 612         (let ((x (* a a)) (y (* b b)))
 613                 (+ x y)))
 614
 615         (sum-sq 3 4) &rarr; 25
 616
 617         ; alternative syntax
 618         (define (sum-sq a b)
 619         (let (x (* a a) y (* b b))
 620                 (+ x y)))
 621
 622         ; using local
 623         (define (sum-sq a b)
 624                 (local (x y)
 625                         (set 'x (* a a))
 626                         (set 'y (* b b))
 627                         (+ x y)))
 628 </pre>
 629
 630 <p>The variables x and y are initialized, then the expression (+ x y) is evaluated. The <tt>let</tt> form is just an optimized version and syntactic convenience for writing:</p>
 631
 632 <pre>
 633         ((lambda (sym1 [sym2 ...]) exp-body ) exp-init1 [ exp-init2 ...])
 634 </pre>
 635
 636 <p>When initializing several parameters, a nested <tt>let</tt>, <tt>letn</tt> can be used to reference previously initialized variables in subsequent initializer expressions:</p>
 637
 638 <pre>
 639         (letn ((x 1) (y (+ x 1)))
 640             (list x y))                 &rarr; (1 2)
 641 </pre>
 642
 643 <p>The function <tt>local</tt> works like a <tt>let</tt> but initializing all variables to <tt>nil</tt>.</p>
 644
 645 <h3>Unused parameters as local symbols</h3>
 646
 647 <p>In newLISP all parameters in user defined functions are optional. Unused parameters are filled with <tt>nil</tt> and local to the dynamic scope of the function. Defining a user function with more parameters than required is a convenient method to create local variable symbols:</p>
 648
 649 <pre>
 650         (define (sum-sq a b , x y)
 651                 (set 'x (* a a))
 652                 (set 'y (* b b))
 653                 (+ x y))
 654 </pre>
 655
 656 <p>The comma is not a special syntax feature but only a visual helper to separate normal parameters from local variable symbols.</p>
 657
 658 <h3>Using <tt>args</tt> as local substitute</h3>
 659
 660 <p>Using the <tt>args</tt> function no parameter symbols need to be used at all and <tt>args</tt> returns a list of all parameters passed but not taken by declared parameters:</p>
 661
 662 <pre>
 663         (define (foo)
 664                 (args))
 665
 666         (foo 1 2 3)   &rarr; (1 2 3)
 667
 668
 669         (define (foo a b)
 670                 (args))
 671
 672         (foo 1 2 3 4 5)   &rarr; (3 4 5)
 673 </pre>
 674
 675 <p>The second example shows how <tt>args</tt> only contains the list of arguments not bound by the variable symbols <tt>a</tt> and <tt>b</tt>.</p>
 676
 677 <p>Indices can be used to access members of the <tt>(args)</tt> list:</p>
 678
 679 <pre>
 680         (define (foo)
 681                 (+ (args 0) (args 1)))
 682
 683         (foo 3 4)   &rarr; 7
 684 </pre>
 685
 686 <center>&sect;</center>
 687 <br>
 688
 689 <a name="walking"></a>
 690 <h2>5. Walking through lists</h2>
 691
 692 <h3>Recursion or iteration?</h3>
 693
 694 <p>Although recursion is a powerful feature to express many algorithms in a readable form, they are also inefficient in some instances. newLISP has many iterative constructs and high level functions like <tt>flat</tt> or the built-in XML functions, which use recursion internally. In many cases this makes defining a recursive algorithm not necessary.</p>
 695
 696
 697 <p>Some times a non-recursive solution can be much faster and lighter on system resources.</p>
 698
 699 <pre>
 700         ;; classic recursion
 701         ;; slow and resource hungry
 702          (define (fib n)
 703                 (if (< n 2) 1
 704                         (+  (fib (- n 1))
 705                                 (fib (- n 2)))))
 706 </pre>
 707
 708 <p>The recursive solution is slow because of frequent calling overhead incurred. The recursive solution uses also a lot of memory for holding intermediate results.</p>
 709
 710 <pre>
 711         ;; iteration
 712         ;; fast and also returns the whole list
 713         (define (fibo n , f)
 714                 (set 'f '(1 0))
 715                 (dotimes (i n)
 716                         (push (+ (f 0) (f 1)) f) -1)
 717                 (rest f))
 718 </pre>
 719
 720 <p>The iterative solution is fast and uses very little memory.</p>
 721
 722 <h3>Generators</h3>
 723
 724 <p>A generator is a function which keeps and changed state between invocations and returns a new value on each call:</p>
 725
 726 <pre>
 727         ;; generator uses a namespace to keep state
 728         ;; the default funcion (fibo) gets called repeatedly
 729         ;;
 730         ;; (fibo) &rarr; 1
 731         ;; (fibo) &rarr; 2
 732         ;; (fibo) &rarr; 3, 5, 8, 13, 21 ...
 733         ;;
 734         ;; fibo:mem &rarr; (0 1 1 2 3 5 8 13 21 ...)
 735         (define (fibo:fibo)
 736                 (if (not fibo:mem) (set 'fibo:mem '(0 1)))
 737                 (push (+ (fibo:mem -2) (fibo:mem -1)) fibo:mem -1))
 738 </pre>
 739
 740 <p>Namespaces in newLISP together with default functions are a convenient method to
 741 package a function with local static variables. <tt>mem</tt> is used in the <tt>fibo</tt>
 742 name space to keep the Fibonacci sequence.</p>
 743
 744 <h3>Speed up with memoization</h3>
 745
 746 <p>A memoizing function caches results for faster retrieval when called with the
 747 same parameters again. The following function makes a memoizing function
 748 from any built-in or user defined function with an arbitrary number of arguments:<p>
 749
 750 <blockquote><pre>
 751 ;; speed up a recursive function using memoization
 752 (define-macro (memoize mem-func func)
 753   (set (sym mem-func mem-func)
 754     (letex (f func  c mem-func)
 755       (lambda ()
 756         (or (context c (string (args)))
 757             (context c (string (args)) (apply f (args))))))))
 758
 759 (memoize fibo-m fibo)
 760
 761 (time (fibo-m 25)) &rarr; 148
 762 (time (fibo-m 25)) &rarr; 0
 763 </pre></blockquote>
 764
 765 <p>The function  creates a context and <em>default</em> function for the original
 766 function with a new name and stores all results in symbols in the same context.</p>
 767
 768 <p>When memoizing recursive functions, include the the raw lambda specification
 769 of the function so recursive calls are memoized too:</p>
 770
 771 <blockquote><pre>
 772 (memoize fibo
 773   (lambda (n)
 774     (if(< n 2) 1
 775       (+  (fibo (- n 1))
 776           (fibo (- n 2))))))
 777
 778 (time (fibo 100)) &rarr; 1
 779 (fibo 80)         &rarr; 37889062373143906
 780 </pre></blockquote>
 781
 782 <p>The <tt>fibo</tt> function in the last example would take hours to calculate
 783 without memoization. The memoized version takes only about a milli second for
 784 an argument of <tt>100</tt>.</p>
 785
 786 <h3>Walking a tree</h3>
 787
 788 <p>Tree walks are a typical pattern in traditional LISP and in newLISP as well for walking through a nested list. But many times a tree walk is only used to iterate through all elements of an existing tree or nested list. In this case the built-in <tt>flat</tt> function is much faster than using recursion:</p>
 789
 790 <pre>
 791         (set 'L '(a b c (d e (f g) h i) j k))
 792
 793         ;; classic car/cdr and recursion
 794         ;;
 795         (define (walk-tree tree)
 796                 (cond ((= tree '()) true)
 797                         ((atom? (first tree))
 798                                 (println (first tree))
 799                                 (walk-tree (rest tree)))
 800                         (true
 801                                 (walk-tree (first tree))
 802                                 (walk-tree (rest tree)))))
 803
 804         ;; classic recursion
 805         ;; 3 times faster
 806         ;;
 807         (define (walk-tree tree)
 808                 (dolist (elmnt tree)
 809                         (if (list? elmnt)
 810                         (walk-tree elmnt)
 811                         (println elmnt))))
 812
 813         (walk-tree L) &rarr;
 814         a
 815         b
 816         c
 817         d
 818         e
 819         ...
 820 </pre>
 821
 822 <p>Using the built-in <tt>flat</tt> in newLISP a nested list can be transformed int a flat list. Now the list can be processed with a <tt>dolist</tt> or <tt>map</tt>:</p>
 823
 824 <pre>
 825         ;; fast and short using 'flat'
 826         ;; 30 times faster with map
 827         ;;
 828         (map println (flat L))
 829
 830         (dolist (item (flat L)) (println item)
 831 </pre>
 832
 833
 834 <h3>Walking a directory tree</h3>
 835 <p>Walking a directory tree is a task where recursion works well:</p>
 836
 837 <pre>
 838         ; walks a disk directory and prints all path-file names
 839         ;
 840         (define (show-tree dir)
 841         (if (directory dir)
 842                 (dolist (nde (directory dir))
 843                    (if (and (directory? (append dir "/" nde))
 844                                         (!= nde ".") (!= nde ".."))
 845                                  (show-tree (append dir "/" nde))
 846                                  (println (append dir "/" nde))))))
 847 </pre>
 848
 849 <p>In this example recursion is the only solution, because the entire nested list off files is not available when the function is called but gets created recursively during function execution.</p>
 850
 851 <center>&sect;</center>
 852 <br>
 853 <a name="creating"></a>
 854 <h2>6. Creating, accessing, modifying and searchung lists</h2>
 855
 856 <p>newLISP has facilities for multidimensional indexing into nested lists. There are
 857 <i>destructive</i> functions like <tt>push</tt>, <tt>pop</tt>, <tt>set-nth</tt>,
 858 <tt>nth-set</tt>, <tt>ref-set</tt>, <tt>set-ref</tt>, <tt>set-ref-all</tt>,
 859 <tt>sort</tt> and <tt>reverse</tt> and many others for
 860 <i>non-destructive</i> operations, like <tt>nth</tt>, <tt>ref</tt>, <tt>ref-all</tt>,
 861 <tt>first</tt>, <tt>last</tt> and <tt>rest</tt> etc..
 862 Many of the list functions in newLISP also work on strings.</p>
 863
 864 <p>Note that any list or string index in newLISP can be negative starting with -1
 865 from the right side of a list:</p>
 866
 867 <pre>
 868         (set 'L '(a b c d))
 869         (L -1)   &rarr; d
 870         (L -2)   &rarr; c
 871         (-3 2 L) &rarr; (b c)
 872
 873         (set 'S  "abcd")
 874
 875         (S -1)   &rarr; d
 876         (S -2)   &rarr; c
 877         (-3 2 S) &rarr; "bc")
 878 </pre>
 879
 880
 881 <h3><tt>push</tt> and <tt>pop</tt></h3>
 882
 883 <p>To add to a list use <tt>push</tt>, to eliminate an element from a list use <tt>pop</tt>.
 884 Both functions are destructive, changing the contents of a list:</p>
 885
 886 <pre>
 887         (set 'L '(b c d e f))
 888
 889         (push 'a L)
 890         (push 'g L -1) ;; push at the end with negative index
 891         (pop L) ; pop first
 892         (pop L -1) ; pop last
 893         (pop L -2) ; pop second to last
 894         (pop L 1)  ; pop second
 895         ; multidimensional push / pop
 896         (set 'L '(a b (c d (e f) g) h i))
 897         (push 'x L 2 1)
 898         L &rarr; (a b (c x d (e f) g) h i)
 899         (pop L 2 1) &rarr; x
 900 </pre>
 901
 902 <p>Pushing to the end of a list repeatedly is optimized in newLISP and as fast as pushing in front of a list.</p>
 903
 904 <p>When pushing an element with index vector V it can be popped with the same index vector V:</p>
 905
 906 <pre>
 907         (set 'L '(a b (c d (e f) g) h i))
 908         (set 'V '(2 1))
 909         (push 'x L V)
 910         L &rarr; (a b (c x d (e f) g) h i))
 911         (ref 'x L) &rarr; (2 1) ; search for a nested member
 912         (pop L V) &rarr; 'v
 913 </pre>
 914
 915 <h3>Accessing lists</h3>
 916
 917 <p>Multiple indexes can be specified to access elements in a nested list structure:</p>
 918
 919 <pre>
 920         (set 'L '(a b (c d (e f) g) h i))
 921
 922         ; old syntax only for one index
 923         (nth 2 L) &rarr; (c d (e f) g)
 924
 925         ; use new syntax for multiple indices
 926         (nth (L 2 2 1)) &rarr; f
 927         (nth (L 2 2)) &rarr; (e f)
 928
 929         ; implicit indexing
 930         (L 2 2 1) &rarr; f
 931         (L 2 2)   &rarr; (e f)
 932
 933         ; implicit indexing with vector
 934         (set 'vec '(2 2 1))
 935         (L vec)   &rarr; f
 936 </pre>
 937
 938 <p>When using the <tt>(nth (L idx))</tt> syntax, <tt>L</tt> may be a context representing a
 939 default functor for passing a list by reference.</p>
 940
 941 <p>Implicit indexing shown in the last example makes code more readable. Implicit indexing also allows an unlimited number of indices, while nth is limited to 16. Indices after a list select list elements. Indices before a list select subsections of a list, which in turn are always lists.</p>
 942
 943 <p>Implicit indexing is also available for rest and slice </p>
 944
 945 <pre>
 946         (rest '(a b c d e))      &rarr; (b c d e)
 947         (rest (rest '(a b c d e) &rarr; (c d e)
 948         ; same as
 949         (1 '(a b c d e)) &rarr; (b c d e)
 950         (2 '(a b c d e)) &rarr; (c d e)
 951         ; negative indices
 952         (-2 '(a b c d e)) &rarr; (d e)
 953         ; slicing
 954         (2 2 '(a b c d e f g))   &rarr; (c d)
 955         (-5 3 '(a b c d e f g)) &rarr; (c d e)
 956 </pre>
 957
 958 <h3>Selecting more than one element</h3>
 959 <p>Sometimes more than one element must be selected from a list.
 960 This is done using select:</p>
 961
 962 <pre>
 963         ;; pick several elements from a list
 964         (set 'L '(a b c d e f g))
 965         (select L 1 2 4 -1) &rarr; (b c e g)
 966
 967         ;; The indices can be delivered in an index vector:
 968         (set 'vec '(1 2 4 -1))
 969         (select L vec) &rarr; (b c e g)
 970 </pre>
 971
 972 <p>The selecting process can re-arrange or double elements at the same time:</p>
 973
 974 <pre>
 975         (select L 2 2 1 1) &rarr; (c c b b)
 976 </pre>
 977
 978 <h3>Filtering and differencing lists</h3>
 979
 980 <p>Sometimes lists need to be filterer for a specific conditions applied to its elements:</p>
 981
 982 <pre>
 983         (filter (fn(x) (&lt; 5 x)) '(1 6 3 7 8))    &rarr; (6 7 8)
 984         (filter symbol? '(a b 3 c 4 "hello" g)) &rarr; (a b c g)
 985         (difference '(1 3 2 5 5 7) '(3 7)) &rarr; (1 2 5)
 986 </pre>
 987
 988 <p>The first example could be rewritten shorter as follows:</p>
 989
 990 <pre>
 991         (filter (curry &lt; 5) '(1 6 3 7 8))
 992 </pre>
 993
 994 <p>The <tt>curry</tt> function makes a one-argument function out of a two argument
 995 function:</p>
 996
 997 <pre>
 998         (curry &lt; 5) &rarr; (lambda (_x) (&lt; 5 _x))
 999 </pre>
1000
1001 <p>Using <tt>curry</tt> a function taking two arguments quickly is converted into
1002 a predicate taking one argument.</p>
1003
1004 <h3>Changing list elements</h3>
1005
1006 <p>set-nth and nth-set have the same effect on the list they are working on but the first returns the whole list while nth-set returns the changed old element:</p>
1007
1008 <pre>
1009         (set 'L '(a b (c d (e f) g) h i))
1010         ; return the changed list (old deprecated syntax)
1011         (set-nth (L 2 2 1) 'x) &rarr; (a b (c d (e x) g) h i)
1012
1013         ; return the old replaced element
1014         (nth-set (L 2 2 1) 'z) &rarr; z
1015 </pre>
1016
1017 <h3>The new element as a function of the replaced</h3>
1018
1019 <p>An internal system variable $0 in newLISP holds the old list element. This can be used to configure the new one:</p>
1020
1021 <pre>
1022         (set 'L '(0 0 0))
1023         (nth-set (L 1) (+ $0 1)) &rarr; 0 ; the old value
1024         (nth-set (L 1) (+ $0 1)) &rarr; 1
1025         (nth-set (L 1) (+ $0 1)) &rarr; 2
1026         L &rarr; '(0 3 0)
1027         </pre>
1028
1029 <h3>Search and replace in simple lists</h3>
1030
1031 <p>Replace, which can also be used on strings, can search for and replace
1032 multiple elements in a list at once. Together with <tt>match</tt> and <tt>unify</tt>
1033 complex search pattersn can be specified. Like with <tt>set-nth</tt> and <tt>nth-set</tt>,
1034 the replacement expression can use the old elelement contents to form the replacement.</p>
1035
1036 <pre>
1037         (set 'aList '(a b c d e a b c d))
1038
1039         (replace 'b aList 'B) &rarr; (a B c d e a B c d)
1040 </pre>
1041
1042 <p>The function <tt>replace</tt> can take a comparison function for picking
1043 list elements:</p>
1044
1045 <pre>
1046         ; replace all numbers where 10 &lt; number
1047         (set 'L '(1 4 22 5 6 89 2 3 24))
1048
1049         (replace 10 L 10 &lt;) &rarr; (1 4 10 5 6 10 2 3 10)
1050 </pre>
1051
1052 <p>Using the built-in functions <tt>match</tt> and <tt>unify</tt> more complex selection
1053 criteria can be defined:</p>
1054
1055 <pre>
1056         ; replace only sublists starting with 'mary'
1057
1058         (replace '(mary *)  AL (list 'mary (apply + (rest $0))) match)
1059         &rarr; ((john 5 6 4) (mary 14) (bob 4 2 7 9) (jane 3))
1060
1061         ; make sum in all expressions
1062
1063         (set 'AL '((john 5 6 4) (mary 3 4 7) (bob 4 2 7 9) (jane 3)))
1064
1065         (replace '(*) AL (list ($0 0) (apply + (rest $0))) match)
1066         &rarr; ((john 15) (mary 14) (bob 22) (jane 3))
1067
1068         $0 &rarr; 4  ; replacements made
1069
1070         ; change only sublists where both elements are the same
1071
1072         (replace '(X X) '((3 10) (2 5) (4 4) (6 7) (8 8)) (list ($0 0) 'double ($0 1)) unify)
1073         &rarr; ((3 10) (2 5) (4 double 4) (6 7) (8 double 8))
1074
1075         $0 &rarr; 2  ; replacements made
1076 </pre>
1077
1078 <p>After a replacement statement is executed the newLISP system variable <tt>$0</tt>
1079 contains the number of replacements made.</p>
1080
1081 <h3>Search and replace in nested lists</h3>
1082
1083 <p>Sometimes lists are nested, e.g. the SXML results fropm parsing XML. The functions
1084 <tt>ref-set</tt>, <tt>set-ref</tt> and <tt>set-ref-all</tt> can be used to find and
1085 replace a single or all elelent in a nested list and replace it or all.</p>
1086
1087 <pre>
1088         (set 'data '((monday (apples 20 30) (oranges 2 4 9)) (tuesday (apples 5) (oranges 32 1))))
1089
1090         (set-ref (data 'monday) tuesday)
1091         &rarr; ((tuesday (apples 20 30) (oranges 2 4 9)) (tuesday (apples 5) (oranges 32 1)))
1092 </pre>
1093
1094 <p>The function <tt>ref-set</tt> works just like <tt>set-ref</tt> but instead of the
1095 whole changed version of the list, only the old element is returned</p>
1096
1097 <pre>
1098         (set 'data '((monday (apples 20 30) (oranges 2 4 9)) (tuesday (apples 5) (oranges 32 1))))
1099
1100         (ref-set  (data 'monday) 'tuesday) &rarr; monday
1101 </pre>
1102
1103 <p>The function <tt>set-ref-all</tt> does a <tt>set-ref</tt> multiple times, replacing all
1104 found occurrences of and element.</p>
1105
1106 <pre>
1107         (set 'data '((monday (apples 20 30) (oranges 2 4 9)) (tuesday (apples 5) (oranges 32 1))))
1108
1109         (set-ref-all (data 'apples) "Apples")
1110         &rarr; ((monday ("Apples" 20 30) (oranges 2 4 9)) (tuesday ("Apples" 5) (oranges 32 1)))
1111 </pre>
1112
1113 <p>Like <tt>find</tt>, <tt>replace</tt>, <tt>ref</tt> and <tt>ref-all</tt>, more
1114 complex searches can be expressed using <tt>match</tt> or <tt>unify</tt>:</p>
1115
1116 <pre>
1117         (set 'data '((monday (apples 20 30) (oranges 2 4 9)) (tuesday (apples 5) (oranges 32 1))))
1118
1119         (set-ref-all (data '(oranges *)) (list (first $0) (apply + (rest $0))) match)
1120         &rarr; ((monday (apples 20 30) (oranges 15)) (tuesday (apples 5) (oranges 33)))
1121 </pre>
1122
1123 <p>The last exampls shows how <tt>$0</tt> can be used to access the old list element
1124 in the updating expression. In this case the numbers for <tt>oranges</tt> records
1125 have been summed up.</p>
1126
1127 <h3>Passing lists by reference using the default functor</h3>
1128
1129 <p>Sometimes a larger list (more than a few hundred elements) must passed to a function
1130 for changing elemements in it. Normally newLISP passes all parameters to user-defined
1131 functions by value. The following snipped shows a technique that can be used to pass a
1132 bigger list or string object by reference:</p>
1133
1134 <pre>
1135         (set 'data:data '(a b c d e f g h))
1136
1137         (define (change db i value)
1138             (nth-set (db i) value))
1139
1140         (change data 3 999) &rarr; d
1141         data:data &rarr; '(a b c 999 d e f g h)
1142 </pre>
1143
1144 <p>In this example the list is encapsulated in a context named <tt>data</tt> holding a variable <tt>data</tt> with the same name.</p>
1145
1146 <p>Wheneve the syntactic form <tt>(<em>func</em> (L x) ...)</tt> is encounterd in newLISP as in the
1147 functions: <tt>nth</tt>, <tt>nth-set</tt>, <tt>set-nth</tt>, <tt>set-ref</tt>, <tt>set-ref-all</tt>,
1148 <tt>ref-set</tt>,<tt>-set-assoc</tt> and <tt>assoc-set</tt> the same techinique of reference passing
1149 can be employed.</p>
1150
1151 <center>&sect;</center>
1152 <br>
1153 <a name="flow"></a>
1154 <h2>7. Program flow</h2>
1155
1156 <p>Program flow in newLISP is mostly functional but it also has looping and branching constructs and a catch and throw to break out of normal flow.</p>
1157
1158 <p>Looping expressions as a whole behave like a function or block returning the last expression evaluated inside. </p>
1159
1160 <h3>Loops</h3>
1161 <p>Most of the traditional looping patterns are supported. Whenever there is a looping variable, it is local in scope to the loop, behaving according the rules of dynamic scoping inside the current name-space or context:</p>
1162
1163
1164 <pre>
1165         ; loop a number of times
1166         ; i goes from 0 to N - 1
1167         (dotimes (i N)
1168            ....
1169         )
1170
1171         ; demonstrate locality of i
1172         (dotimes (i 3)
1173             (print i ":")
1174             (dotimes (i 3) (print i))
1175             (println))
1176
1177         &rarr; ; will output
1178         0:012
1179         1:012
1180         2:012
1181
1182         ; loop through a list
1183         ; takes the value of each top level element in aList
1184         (dolist (e aList)
1185            ...
1186         )
1187
1188         ; loop through a sring
1189         ; takes the ASCII or UTF-8 value of each character in aString
1190         (dostring (e aString)
1191            ...
1192         )
1193
1194         ; loop through the symbols of a context in
1195         ; alphabetical order of the symbol name
1196         (dotree (s CTX)
1197            ...
1198         )
1199
1200         ; loop from to with optional step size
1201         ; i goes from init to <= N inclusive with step size step
1202         ; Note that the sign in step is irrelevant, N can be greater
1203         ; or less then init.
1204         (for i init N step)
1205            ...
1206         )
1207
1208         ; loop while a condition is true
1209         ; first test condition then perform body
1210         (while condition
1211            ...
1212         )
1213
1214         ; loop while a condition is false
1215         ; first test condition then perform body
1216         (until condition
1217            ....
1218         )
1219
1220         ; loop while a condition is true
1221         ; first perform body then test
1222         ; body is performed at least once
1223         (do-while condition
1224           ...
1225         )
1226
1227         ; loop while a condition is false
1228         ; first perform body then test
1229         ; body is performed at least once
1230         (do-until condition
1231           ...
1232         )
1233 </pre>
1234
1235 <p>Note that the looping functions <tt>dolist</tt>, <tt>dotimes</tt> and <tt>for</tt> can also
1236 take a break condition as an additional argument. When the break condition evaluates to true
1237 the loop finishes:</p>
1238
1239 <pre>
1240         (dolist (x '(a b c d e f g) (= x 'e))
1241         (print x))
1242         &rarr; ; will output
1243         abcd
1244 </pre>
1245
1246 <h3>Blocks</H3>
1247 <p>Blocks are collections of s-expressions evaluated sequentially. All looping constructs may have expression blocks after the condition expression as a body.</p>
1248
1249 <p>Blocks can also be constructed by enclosing them in a begin expression:</p>
1250
1251  <pre>
1252         (begin
1253             s-exp1
1254             s-exp2
1255             .....
1256             s-expN)
1257 </pre>
1258
1259 <p>Looping constructs do not need to use an explicit begin after the looping conditions. <tt>begin</tt>
1260 is mostly used to block expressions in <tt>if</tt> and <tt>cond</tt> statements.</p>
1261
1262 <p>The functions <tt>and</tt>, <tt>or</tt>, <tt>let</tt>, <tt>letn</tt> and <tt>local</tt>
1263 can also be used to form blocks and do not require <tt>begin</tt> for blocking statements.</p>
1264
1265 <H3>Branching</H3>
1266
1267 <pre>
1268         (if condition true-expr false-expr)
1269
1270         ;or
1271         (if condition true-expr)
1272
1273         ;or unary if for (filter if '(...))
1274         (if condition)
1275
1276         ;; more then one statement in the true or false
1277         ;; part must be blocked with (begin ...)
1278         (if (= x Y)
1279            (begin
1280               (some-func x)
1281               (some-func y)
1282            (begin
1283               (do-this x y)
1284               (do-that x y)))
1285 </pre>
1286
1287 <p>Depending on condition the true-expr or false-expr part is evaluated and returned.</p>
1288
1289 <p>More then one condition true-expr pair can occur in an if expression, making it look like a cond:</p>
1290
1291 <pre>
1292         (if condition-1 true-expr-1
1293             condition-2 true-expr-2
1294                   .....
1295             condition-n true-expr-n
1296             false-expr)
1297 </pre>
1298
1299 <p>The first true-expr-i of which the condition-i is not nil is evaluated and returned or the false-expr if none of the condition-i is true.</p>
1300
1301 <p>cond works like the multiple condition form of if but each part of condition-i true-expr-i must be braced in parenthesis:</p>
1302
1303  <pre>
1304         (cond
1305              (condition-1 true-expr-1 )
1306              (condition-2 true-expr-2 )
1307                    .....
1308              (condition-n true-expr-n )
1309              (true true-expr))
1310 </pre>
1311
1312 <H3>Fuzzy flow</H3>
1313
1314 <p>Using amb the program flow can be regulated in a probabilistic fashion:</p>
1315
1316 <pre>
1317         (amb
1318             expr-1
1319             expr-2
1320             .....
1321             expr-n)
1322 </pre>
1323 <p>One of the alternative expressions expr-1 to expr-n is evaluated with a probability of 1 / n and the result is returned from the amb expression.</p>
1324
1325 <h3>Change flow with <tt>catch</tt> and <tt>throw</tt></h3>
1326
1327 <p>Any loop or other expression block can be enclosed in a catch expression. The moment a throw expression is evaluated, the whole catch expression returns the value of the throw expression.</p>
1328 <pre>
1329         (catch
1330            (dotimes (i 10)
1331                (if (= i 5) (throw "The End"))
1332                 (print i " ")))
1333         ; will output
1334
1335         0 1 2 3 4
1336         ; and the return value will be
1337         &rarr; "The End"
1338 </pre>
1339 <p>Several catch may be nested.</p>
1340
1341 <h3>Leave loops with a break condition</h3>
1342
1343 <p>Loops built using <tt>dotimes</tt>, <tt>dolist</tt> of <tt>for</tt> can specify a break condition
1344 under which the loop is left early:</p>
1345
1346 <pre>
1347         (dotimes (x 10 (&gt; (* x x) 9))
1348             (println x))
1349         &rarr;
1350         0
1351         1
1352         2
1353         3
1354
1355         (dolist (i '(a b c nil d e) (not i))
1356             (println i))
1357         &rarr;
1358         a
1359         b
1360         c
1361 </pre>
1362 <h3>Change flow with <tt>and</tt> or <tt>or</tt></h3>
1363
1364 <p>Similar to programming in Prolog the logical and and or can be used to control program flow depending on the outcome of expressions logically connected:</p>
1365
1366 <pre>
1367         (and
1368            expr-1
1369            expr-2
1370             ...
1371            expr-n)
1372 </pre>
1373 <p>Expressions are evaluated sequentially until one expr-i evaluates to nil or the empty list'() or until all expr-i are exhausted. The last expression evaluated is the return value of the whole and expression.</p>
1374
1375 <pre>
1376         (or
1377            expr-1
1378            expr-2
1379             ...
1380            expr-n)
1381 </pre>
1382
1383 <p>Expressions are evaluated sequentially until one expr-i evaluates to not nil and not '() or until all expr-i are exhausted. The last expression evaluated is the return value of the whole or expression.</p>
1384 <center>&sect;</center>
1385 <br>
1386
1387 <a name="error"></a>
1388 <h2>8. Error handling</h2>
1389
1390 <p>Several conditions during evaluation of a newLISP expression can cause error exceptions. For a complete list of errors see the newLISP Reference Manual Appendix.</p>
1391
1392 <h3>System errors</h3>
1393
1394 <p>System errors are caused by wrong syntax, of function invocation, not supplying the right amount of parameters, or supplying parameters with the wrong data type. Other system errors are caused by trying to evaluate non existing functions.</p>
1395
1396 <pre>
1397          ; examples of system errors
1398          ;
1399          (foo foo)   &rarr; invalid function : (foo foo)
1400          (+ "hello") &rarr; value expected in function + : "hello"
1401 </pre>
1402
1403 <h3>User defined errors</h3>
1404
1405 <p>User errors are error exceptions thrown using the function throw-error:</p>
1406
1407 <pre>
1408         ; user defined error
1409         ;
1410         (define (double x)
1411             (if (= x 99) (throw-error "illegal number"))
1412             (+ x x))
1413
1414         (double 8)   &rarr; 16
1415         (double 10)  &rarr; 20
1416         (double 99)
1417         &rarr;
1418         user error : illegal number
1419         called from user defined function double
1420 </pre>
1421
1422 <h3>Error event handlers</h3>
1423 <p>System and user defined errors can be caught using the function error-event to define an event handler.</p>
1424
1425 <pre>
1426         ; define an error event handler
1427         ;
1428         (define (MyHandler)
1429            (println "An error #" (error-number) " has occurred"))
1430
1431         (error-event 'MyHandler)
1432
1433         (foo) &rarr; An error #23 has occurred
1434 </pre>
1435
1436 <h3>Catching errors</h3>
1437 <p>A fine grainier, more specific error exception handling can be achieved using a special syntax of the function catch.</p>
1438
1439 <pre>
1440         (define (double x)
1441             (if (= x 99) (throw-error "illegal number"))
1442             (+ x x))
1443 </pre>
1444
1445 <p>catch with a second parameter can be used to catch both, system and user defined errors:</p>
1446
1447 <pre>
1448         (catch (double 8) 'result) &rarr; true
1449         result &rarr; 16
1450         (catch (double 99) 'result) &rarr; nil
1451         (print result)
1452         &rarr;
1453         user error : illegal number
1454         called from user defined function double
1455
1456         (catch (double "hi") 'result) &rarr; nil
1457         (print result)
1458         &rarr;
1459         value expected in function + : x
1460         called from user defined function double
1461 </pre>
1462
1463 <p>The catch expression returns true when no error exception occurred and the result of the expression is found in the symbol result specified as a second parameter.</p>
1464
1465 <p>If an error exception occurs, it is caught and the catch clause returns nil. In this case the symbol result contains the error message.</p>
1466 <center>&sect;</center>
1467 <br>
1468
1469 <a name="functions"></a>
1470 <h2>9. Functions as data</h2>
1471 <h3>Manipulate functions after definition</h3>
1472
1473 <pre>
1474         (define (double x) (+ x x))
1475         &rarr; (lambda (x) (+ x x))
1476
1477         (first double) &rarr; (x)
1478         (last double) &rarr; (+ x x)
1479
1480         ;; make a ''fuzzy'' double
1481         (nth-set (double 1) '(mul (normal x (div x 10)) 2))
1482
1483         (double 10) &rarr; 20.31445313
1484         (double 10) &rarr; 19.60351563
1485 </pre>
1486
1487 <p>lambda in newLISP is not an operator or symbol, but rather a special s-expression or list attribute:</p>
1488
1489 <pre>
1490         (first double) &rarr; (x)   ; not lambda
1491 </pre>
1492
1493 <p>The lambda attribute of an s-expression is right-associative in append:</p>
1494
1495 <pre>
1496         (append (lambda) '((x) (+ x x))) &rarr; (lambda (x) (+ x x))
1497         (set 'double (append (lambda) '((x) (+ x x)))
1498
1499         (double 10) &rarr; 20
1500 </pre>
1501
1502 <p>and left-associative when using cons:</p>
1503
1504 <pre>
1505         (cons '(x) (lambda) &rarr; (lambda (x))
1506 </pre>
1507
1508 <p>Lambda expressions in newLISP never loose their first class object property.</p>
1509
1510 <h3>Mapping and applying functions</h3>
1511
1512 <p>Functions or operators can be applied to a list of data at once and all results are returned in a list</p>
1513
1514 <pre>
1515         (define (double (x) (+ x x))
1516           (map double '(1 2 3 4 5)) &rarr; (2 4 6 8 10)
1517 </pre>
1518
1519
1520 <p>Functions can be applied to parameters occurring in a list:</p>
1521
1522 <pre>
1523         (apply + (sequence 1 10)) &rarr; 55
1524 </pre>
1525
1526 <a name="functions"></a>
1527
1528 <h3>Function currying: functions making functions</h3>
1529
1530 <p>Here and expression is passed as a parameter:</p>
1531
1532 <pre>
1533         (define (raise-to power)
1534            (expand (fn (base) (pow base power)) 'power))
1535
1536         (define square (raise-to 2))
1537
1538         (define cube (raise-to 3))
1539
1540         (square 5) &rarr; 25
1541         (cube 5)   &rarr; 125
1542 </pre>
1543
1544 <p>The built-in function <tt>curry</tt> can be used to make a function taking
1545 one argument from a function taking two arguments.</p>
1546
1547 <pre>
1548         (define add-one (curry add 1))  &rarr; (lambda (_x) (add 1 _x))
1549
1550         (define by-ten (curry mul 10))  &rarr; (lambda (_x) (mul 10 _x))
1551
1552         (add-one 5)  &rarr; 6
1553
1554         (by-ten 1.23)  &rarr; 12.3
1555 </pre>
1556
1557 <p>Note that the <em>curried</em> parameter is always the first (left) one.</p>
1558
1559 <center>&sect;</center>
1560 <br>
1561
1562 <a name="text"></a>
1563 <h2>10. Text processing</h2>
1564 <h3>Regular expressions</h3>
1565
1566 <p>Regular expression in newLISP can be used together with a variety of functions:</p>
1567
1568 <center>
1569 <table border="0" width="90%" cellpadding="5">
1570
1571 <tr><td><tt>directory</tt></td><td>Returns a list of files, can use a regex patterns for filtering</td></tr>
1572
1573 <tr><td><tt>find</tt></td><td>is used to find the position / offset of a pattern.</td></tr>
1574
1575 <tr><td><tt>find-all</tt></td><td>Assemble a list of all patterns found.</td></tr>
1576
1577 <tr><td><tt>parse</tt></td><td>breaks a string into token at patterns found between tokens.</td></tr>
1578
1579 <tr><td><tt>regex</tt></td><td>finds patterns and lists all sub patterns with offset and length found</td></tr>.
1580
1581 <tr><td><tt>replace</tt></td><td>replaces found patterns with a user defined function, which can take the as input the patterns itself.</td></tr>
1582
1583 <tr><td><tt>search</tt></td><td>searches for a pattern in a file.</td></tr>
1584 </table>
1585 </center>
1586
1587 <p>The functions find, regex, replace and search store pattern matching results in the system variable $0 to $15. See the newLISP Users Manual for details.</p>
1588
1589 <p>The following paragraphs show frequently used algorithms for scanning and tokenizing text.</p>
1590
1591 <h3>Scanning text</h3>
1592
1593 <p>replace together with a regular expression pattern can be used to scan text. The pattern in this case describes the tokens scanned for. As each token is found it is pushed on a list. The work is done in the replacement expression part of replace. This example saves all files linked on a web page:</p>
1594
1595 <pre>
1596         ; tokenize using replace with regular expressions
1597
1598         (set 'page (get-url "http://www.nodep.nl/newlisp/index.html"))
1599
1600         (replace {href="(http://.*lsp)"} page (push $1 links) 0)
1601
1602         (dolist (link links)
1603            (set 'file (last (parse link "/")))
1604            (write-file file (get-url link))
1605            (println "-&gt;" file))
1606 </pre>
1607
1608 <p>Curly braces <tt>{</tt>,<tt>}</tt> are used in the regular expression pattern to avoid escaping the quotes <tt>"</tt>.</p>
1609
1610 <p>The following technique is better but could not be used before version 8.9.0 of newLISP. The new
1611 <tt>find-all</tt> function puts all pattern matching strings into a list:</p>
1612
1613 <pre>
1614         (set 'links (find-all {href="(http://.*lsp)"} page))
1615
1616         (dolist (link links)
1617            (set 'file (last (parse link "/")))
1618            (write-file file (get-url link))
1619            (println "-&gt;" file))
1620 </pre>
1621
1622 <p><tt>find-all</tt> works in this example similar to <tt>replace</tt> with the <tt>push</tt> expression,
1623 but the <tt>find-all</tt> approach is more readable.</p>
1624
1625
1626 <p>In an additional expressions <tt>find-all</tt> can be directed to do additional
1627 work with subexpressions found:</p>
1628
1629 <pre>
1630         (find-all {(new)(lisp)} "newLISPisNEWLISP" (append $2 $1) 1)
1631
1632         &rarr; ("LISPnew" "LISPNEW")
1633 </pre>
1634
1635 <p>In the last example <tt>find-all</tt> appends the sub expressions found in reverse
1636 order before returning them in the result list.</p>
1637
1638 <p>Another technique for tokenizing text uses <tt>parse</tt>. While with <tt>replace</tt>
1639 and <tt>find-all</tt> the regular expression defined the token, when using <tt>parse</tt>
1640 the regex pattern describes the space between the tokens:</p>
1641
1642 <pre>
1643         ; tokenize using parse
1644         (set 'str "1 2,3,4 5, 6 7  8"
1645         (parse str {,\ *|\ +,*} 0)
1646         &rarr; ("1" "2" "3" "4" "5" "6" "7" "8")
1647 </pre>
1648
1649 <p>Without the curly braces <tt>{</tt>,<tt>}</tt> in the parse pattern the backslashes would need to be doubled.</p>
1650
1651 <h3>Appending strings</h3>
1652
1653 <p>When appending strings append and join can be used to form a new string:</p>
1654
1655 <pre>
1656         (set 'lstr (map string (rand 1000 100)))
1657         &rarr; ("976" "329" ... "692" "425")
1658
1659         ;; the wrong slowest way
1660         (dolist (s lstr)
1661             (set 'bigStr (append bigStr s)))
1662
1663         ;; smarter way - 50 times faster
1664         ;;
1665         (apply append lstr)
1666 </pre>
1667
1668 <p>Sometimes strings are not readily available in a list like in the above examples. In this case push can be used to push strings on a list while they get produced. The list then can be used as an argument for join, making the fastest method for putting strings together from existing pieces:</p>
1669
1670 <pre>
1671         ;; smartest way - 300 times faster
1672         ;; join an existing list of strings
1673         ;;
1674         (join lstr) &rarr; "97632936869242555543 ...."
1675
1676         ;; join can specify a string between the elements
1677         ;; to be joined
1678         (join lstr "-") &rarr; "976-329-368-692-425-555-43 ...."
1679 </pre>
1680
1681 <h3>Growing strings in place</h3>
1682
1683 <p>Very often the best method is to grow a string in place. The functions <tt>write-buffer</tt>,
1684 <tt>write-line</tt> and <tt>push</tt> can be used not only to write to file handles or push on
1685 lists but also to write/append to existing strings:</p>
1686
1687 <pre>
1688         ;; smartest way - 150 times faster
1689         ;; grow a string in place
1690         ;;
1691         ;; using write-buffer
1692         (set 'bigStr "")
1693         (dolist (s lstr) (write-buffer bigStr s))
1694
1695         ;; using push
1696         (set 'bigStr "")
1697         (dolist (s lstr) (push s bigStr -1))
1698 </pre>
1699
1700 <h3>Rearranging strings</h3>
1701
1702 <p>The function select for selecting elements from lists can also be used to select and re-arrange characters from strings:</p>
1703
1704 <pre>
1705         (set 'str "eilnpsw")
1706         (select str '(3 0 -1 2 1 -2 -3)) &rarr; "newlisp"
1707
1708         ; alternative syntax
1709         (select str 3 0 -1 2 1 -2 -3) &rarr; "newlisp"
1710 </pre>
1711
1712 <p>The second syntax is useful when ineces are specied not as constants but occur as variables.
1713 </p>
1714
1715 <center>&sect;</center>
1716 <br>
1717
1718 <a name="dicts"></a>
1719 <h2>11. Dictionaries</h2>
1720
1721 <h3>Creating and modifying symbols</h3>
1722
1723 <p>After scanning and tokenizing text frequently dictionaries are built to store tokens together with their frequencies or other statistics or attributes. </p>
1724
1725 <pre>
1726         ; create a symbol from a string and set it to a value
1727         ;
1728         (set 'token "hello")
1729         (set (sym token 'Words) '(0 0 0 0))
1730
1731         ; a shorter method since 8.7.4
1732         (context 'Words token '(0 0 0 0))
1733
1734         ; read contents of a word in dictionary
1735         ;
1736         (eval (sym token 'Words)) &rarr; (0 0 0 0)
1737
1738         ; a shorter method since 8.7.4
1739         (context 'Words token)    &rarr; (0 0 0 0)
1740
1741         ; update a word, using the old data
1742         ; incrementing a counter by one
1743         ;
1744         (nth-set 0 (eval (sym token 'Words)) (+ $0 1)) &rarr; 0  ; the old value
1745         (nth-set 0 (eval (sym token 'Words)) (+ $0 1)) &rarr; 1
1746         (nth-set 0 (eval (sym token 'Words)) (+ $0 1)) &rarr; 2
1747         ; same as previous, but alternative shorter syntax
1748         (nth-set ((context 'Word token) 0) (+ $0 1)) &rarr; 3
1749         (nth-set ((context 'Word token) 0) (+ $0 1)) &rarr; 4
1750
1751         (eval (sym token 'Words)) &rarr; (5 0 0 0)
1752         ; same in alternative shorter syntax
1753         (context 'Words token) &rarr; (5 0 0 0)
1754 </pre>
1755
1756 <p>The dictionary can be easily saved to a file by serializing the context Words:</p>
1757
1758 <pre>
1759         ; save the dictionary to a file
1760         ;
1761         (save "words.db" 'Words)
1762         Just as easy you can reload the dictionary:
1763         ; load dictionary from a file
1764         ;
1765         (load "words.db")
1766 </pre>
1767
1768 <p>Test if a symbol exist in a namespace:</p>
1769
1770 <pre>
1771         (context 'foo "abc" 123)
1772
1773         (context? 'foo "abc") &rarr; true
1774         (context? 'foo "def") &rarr; nil
1775 </pre>
1776
1777 <h3>Iterating through dictionaries</h3>
1778
1779 <p>Dictionaries are a collection of symbols in a namespace, constructed via a binary tree.
1780 Yhe functions <tt>symbols</tt> or <tt>dotree</tt> can be used to iterate through all
1781 symbols in a namespace in alphabbetical fashion:</p>
1782
1783 <pre>
1784         ; make a dictionary
1785         (context 'foo "sdfg" 111)
1786         (context 'foo "yete" 222)
1787         (context 'foo "asdf" 333)
1788
1789         ; display all key value pairs in alphabetical order
1790
1791         (dotree (s foo)
1792                 (println s "-&gt;" (eval s)))
1793
1794         ; or an alternative method
1795
1796         (dolist (s (symbols foo))
1797                 (println s "-&gt;" (eval s)))
1798
1799         ; generates the output
1800
1801         foo:asdf-&gt;333
1802         foo:sdfg-&gt;111
1803         foo:yete-&gt;222
1804 </pre>
1805
1806 <p>The first method using <tt>dotree</tt> is really only a faster shortcut
1807 od the second method using symbols.</p>
1808
1809
1810 <h3>Object oriented implementation of a dictionary</h3>
1811
1812 <p>The quickest shortest way to define and handle dictionaries is the folllowing function definition
1813 using a default functor for the dictionary:</p>
1814
1815 <pre>
1816         (define (myhash:myhash key value)
1817                 (if value
1818                         (context 'myhash key value)
1819                 (context 'myhash key)))
1820
1821         (myhash "hello" 123)
1822
1823         (myhash "hello")     &rarr; 123
1824 </pre>
1825
1826 <p>The default functor is the getter/setter function and namespace identifier for the
1827 dictionary at the same time.</p>
1828
1829 <center>&sect;</center>
1830 <br>
1831
1832 <a name="tcpip"></a>
1833 <h2>12. TCP/IP client server communications</h2>
1834
1835 <h3>Client - server TCP/IP - open connection</h3>
1836
1837 <p>In this pattern the server keeps the connection open until the client closes the connection, then the server loops into a new listen:</p>
1838
1839 <pre>
1840         ;;;;;;;;;;;;;;;;;;;;;; the server
1841         ; maximum bytes to receive
1842         (constant 'max-bytes 1024)
1843         (if (not (set 'listen (net-listen 123)))
1844             (print (net-error)))
1845         (while (not (net-error))
1846             (set 'connection (net-accept listen)) ;; blocking here
1847             (while (not (net-error))
1848                  (net-receive connection 'message-from-client max-bytes)
1849                  .... process message from client ...
1850                  .... configure message to client ...
1851                  (net-send connection message-to-client)) )
1852
1853         ;;;;;;;;;;;;;;;;;;;;; the client
1854         ; client connect
1855         (if (not (set 'connection (net-connect "host.com" 123)))
1856             (println (net-error)))
1857         ; maximum bytes to receive
1858         (constant 'max-bytes 1024)
1859         ; message send-receive loop
1860         (while (not (net-error))
1861           .... configure message to server ...
1862           (net-send connection message-to-server)
1863           (net-receive connection 'message-from-server max-bytes)
1864           .... process message-from-server ...
1865           )
1866 </pre>
1867
1868 <h3>Client - server TCP/IP - closed transaction</h3>
1869
1870 <p>In this pattern the server closes the connection after each transaction exchange of messages.</p>
1871
1872 <pre>
1873         ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; server
1874         (while (not (net-error))
1875             (set 'connection (net-accept listen)) ;; blocking here
1876             (net-receive connection 'message-from-client max-bytes)
1877                  .... process message from client ...
1878                  .... configure message to client ...
1879             (net-send connection message-to-client)
1880             (close connection))
1881
1882         ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; client
1883         (if (not (set 'connection (net-connect "host.com" 123)))
1884             (println (net-error)))
1885         ; maximum bytes to receive
1886         (constant 'max-bytes 1024)
1887           .... configure message to server ...
1888         (net-send connection message-to-server)
1889         (net-receive connection 'message-from-server max-bytes)
1890           .... process message-from-server ...
1891 </pre>
1892
1893 <p>There are many different ways to set up a client/server connection, see also the examples in the newLISP manual.</p>
1894
1895 <center>&sect;</center>
1896 <br>
1897 <a name="udp"></a>
1898
1899 <h2>13. UDP communications</h2>
1900
1901 <p>They are fast and need less setup than TCP/IP and offer ''multi casting''. UDP is also less reliable because the protocol does less checking, i.e. of correct packet sequence or if all packets are received. This is normally no problem when not working on the Internet but in a well controlled local network or when doing machine control. A simple more specific protocol could be made part of the message.</p>
1902
1903 <h3>Open communications with UDP</h3>
1904
1905 <p>In this example the server keeps the connection open. UDP communications with net-listen, net-receive-from and net-send-to can block on receiving.</p>
1906
1907 <pre>
1908         ;;;;;;;;;;;;;;;;;; server
1909         (set 'socket (net-listen 10001 "" "udp"))
1910         (if socket (println "server listening on port " 10001)
1911            (println (net-error)))
1912         (while (not (net-error))
1913            (set 'msg (net-receive-from socket 255))
1914            (println "->" msg)
1915            (net-send-to (nth 1 msg) (nth 2 msg)
1916            (upper-case (first msg)) socket))
1917
1918         ;;;;;;;;;;;;;;;;;; client
1919         (set 'socket (net-listen 10002 "" "udp"))
1920         (if (not socket) (println (net-error)))
1921         (while (not (net-error))
1922            (print "->")
1923            (net-send-to "127.0.0.1" 10001 (read-line) socket)
1924            (net-receive socket 'buff 255)
1925            (println "&rarr;" buff))
1926         closed transaction oriented UDP
1927 </pre>
1928
1929 <h3>Closed transaction oriented UDP</h3>
1930
1931 <p>This form is some times used for controlling hardware or equipment. No setup is required, just one function for sending, another one for receiving:</p>
1932
1933 <pre>
1934         ;;;;;;;;;;;;;;;;;; server
1935         ;; wait for data gram with maximum 20 bytes
1936         (net-receive-udp 1001 20)
1937         ;; or
1938         (net-receive-udp 1001 20 5000000)  ;; wait for max 5 seconds
1939
1940         ;;;;;;;;;;;;;;;;;; client
1941         (net-send-udp "host.com" 1001 "Hello")
1942 </pre>
1943
1944 <p>Win32 and Linux show different behavior when sending less or more bytes then specified on the receiving end.</p>
1945
1946 <h3>UDP multi-cast communications</h3>
1947
1948 <p>In this scheme the server subscribes to one of a range of multi cast addresses using the net-listen function.</p>
1949
1950 <pre>
1951         ;; example server
1952         (net-listen 4096 "226.0.0.1" "multi") &rarr; 5
1953         (net-receive-from 5 20)
1954         ;; example client
1955         (net-connect "226.0.0.1" 4096 "multi") &rarr; 3
1956         (net-send-to "226.0.0.1" 4096 "hello" 3)
1957 </pre>
1958
1959 <p>The connection in the example is blocking on <tt>net-receive</tt> but could be de-blocked using <tt>net-select</tt> or <tt>net-peek</tt></p>
1960
1961 <center>&sect;</center>
1962 <br>
1963
1964 <a name="nonblock"></a>
1965 <h2>14. Non-blocking communications</h2>
1966
1967 <h3>Using net-select</h3>
1968
1969 <p>In all previous patterns the client blocks when in receive. The net-select call can be used to unblock communications:</p>
1970
1971 <pre>
1972         ;; optionally poll for arriving data with 100ms timeout
1973         (while (not (net-select connection "r" 100000))
1974             (do-something-while-waiting ...))
1975
1976         (net-receive...)
1977 </pre>
1978
1979 <p><tt>connection</tt> can be a single number for a connection socket or a list of numbers to wait on various sockets.</p>
1980
1981
1982 <h3>Using net-peek</h3>
1983
1984
1985 <pre>
1986         (while ( = (net-peek aSock) 0)
1987            (do-something-while-waiting ...))
1988         (net-receive...)
1989 </pre>
1990
1991 <center>&sect;</center>
1992 <br>
1993
1994 <a name="controlling"></a>
1995 <h2>15. Controlling other apps.</h2>
1996
1997 <p>In this chapter all external applications are launched using process. This function returns immediately after launching the other application and does not block.</p>
1998
1999 <p>In all of the following patterns the server is not independent but controlled by the client, which launches the server and then communicated via a line oriented protocol:</p>
2000
2001 <pre>
2002         -&gt; launch server
2003         -&gt; talk to server
2004         &lt;- wait for response from server
2005         -&gt; talk to server
2006         &lt;- wait for response from server
2007             .....
2008 </pre>
2009
2010 <p>Often a sleep time is necessary on the client side to wait for the server to be ready loading. Most of these examples are condensed snippets from GTK-Server from www.gtk-server.org.</p>
2011
2012 <h3>Communications via STD I/O </h3>
2013
2014 <p>process allows specifying 2 pipes for communications with the launched application.</p>
2015
2016 <pre>
2017         # Define communication function
2018         (define (gtk str)
2019            (write-line str myout)
2020            (if (!= str "gtk_exit 0")
2021                (read-line myin)))
2022
2023         # Launch gtk-server
2024         (map set '(myin gtkout) (pipe))
2025         (map set '(gtkin myout) (pipe))
2026         (process "gtk-server stdin" gtkin gtkout)
2027         (gtk "gtk_init NULL NULL")
2028         (gtk "gtk_window_new 0")
2029                   .....
2030 </pre>
2031
2032 <h3>communications via TCP/IP</h3>
2033
2034 <pre>
2035         ; Define communication function
2036         (define (gtk str , tmp)
2037             (net-send connection str)
2038             (net-receive connection 'tmp 64)
2039             tmp)
2040
2041         ; Start the gtk-server
2042         (process "gtk-server tcp localhost:50000")
2043         (sleep 1000)
2044
2045         ; Connect to the GTK-server
2046         (set 'connection (net-connect "localhost" 50000))
2047         (set 'result (gtk "gtk_init NULL NULL"))
2048         (set 'result (gtk "gtk_window_new 0"))
2049                   .....
2050 </pre>
2051
2052
2053 <h3>Communicate via named FIFO</h3>
2054
2055 <p>Make a FIFO first (looks like a special file node):</p>
2056
2057 <pre>
2058          (exec "mkfifo myfifo")
2059
2060         ;; or alternatively
2061
2062         (import "/lib/libc.so.6" "mkfifo")
2063         (mkfifo "/tmp/myfifo" 0777)
2064
2065         ; Define communication function
2066         (define (gtk str)
2067             (set 'handle (open "myfifo" "write"))
2068             (write-buffer handle str)
2069             (close handle)
2070             (set 'handle (open "myfifo" "read"))
2071             (read-buffer handle 'tmp 20)
2072             (close handle)
2073             tmp)
2074 </pre>
2075
2076 <h3>Communicate via UDP</h3>
2077
2078 <p>Note that the listen function with "udp" option just binds the sockets to a address/hardware but not actually listens as in TCP/IP.</p>
2079
2080 <pre>
2081         ; Define communication function
2082         (define (gtk str , tmp)
2083            (net-send-to "localhost" 50000 str socket)
2084            (net-receive socket 'tmp net-buffer)
2085            tmp)
2086
2087         ; Start the gtk-server
2088         (define (start)
2089            (process "gtk-server udp localhost:50000")
2090            (sleep 500)
2091            (set 'socket (net-listen 50001 "localhost" "udp")) )
2092
2093         (set 'result (gtk "gtk_init NULL NULL"))
2094
2095         (set 'result (gtk "gtk_window_new 0"))
2096                  .....
2097 </pre>
2098
2099 <center>&sect;</center>
2100 <br>
2101
2102 <a name="launching"></a>
2103 <h2>16. Launching applications blocking</h2>
2104
2105 <h3>Shell execution</h3>
2106
2107 <p>This is frequently used from newLISP's interactive command line to execute processes in a blocking fashion, which need a shell to run:</p>
2108
2109 <pre>
2110         (! "ls -ltr")
2111 </pre>
2112
2113 <p>There is an interesting variant of this form working not inside a newLISP expression, but only on the command line:</p>
2114
2115 <pre>
2116         !ls -ltr
2117 </pre>
2118
2119 <p>The <tt>!</tt> should be the first character on the command line. This form works like as shell escape like in the VI editor. It is useful for invoking an editor or doing quick shell work without completely leaving the newLISP console.</p>
2120
2121 <h3>Execution capturing std-out in a list</h3>
2122
2123 <pre>
2124         (exec "ls /") &rarr; ("bin" "etc" "home" "lib")
2125 </pre>
2126
2127
2128 <h3>Execution feeding std-in</h3>
2129
2130 <pre>
2131         (exec "script.cgi" cgi-input)
2132 </pre>
2133
2134 <p>In this example cgi-input contains a string feeding. i.e.: a query input, normally coming from a web server. Note that output in this case is written directly to the screen, and cannot be returned to newLISP. Use process and pipe for two way std i/o communications with other applications.</p>
2135
2136 <center>&sect;</center>
2137 <br>
2138
2139 <a name="threads"></a>
2140
2141 <h2>17. Threads, semaphores and shared memory</h2>
2142
2143 <p>Shared memory, semaphores and threads work frequently together. Semaphores can synchronize tasks in different threads and shared memory can be used to communicate between threads.</p>
2144
2145 <p>The following is a more complex example showing the working of all three mechanisms at the same time.</p>
2146
2147 <p>The producer loops through all n values from i = 0 to n - 1 and puts each value into shared memory where it is picked up by the consumer thread. Semaphores are used to signal that a data value is ready for reading.</p>
2148
2149 <pre>
2150         #!/usr/bin/newlisp
2151         # prodcons.lsp -  Producer/consumer
2152         #
2153         # usage of 'fork', 'wait-pid', 'semaphore' and 'share'
2154
2155         (if (> (& (last (sys-info)) 0xF) 4)
2156            (begin
2157                (println "this will not run on Win32")
2158                (exit)))
2159
2160         (constant 'wait -1 'signal 1 'release 0)
2161
2162         (define (consumer n)
2163             (set 'i 0)
2164             (while (< i n)
2165                (semaphore cons-sem wait)
2166                (println (set 'i (share data)) " <-")
2167                (semaphore prod-sem signal))
2168                (exit))
2169
2170         (define (producer n)
2171             (for (i 1 n)
2172                (semaphore prod-sem wait)
2173                (println "-> " (share data i))
2174                (semaphore cons-sem signal))
2175                (exit))
2176
2177         (define (run n)
2178             (set 'data (share))
2179             (share data 0)
2180             (set 'prod-sem (semaphore)) ; get semaphores
2181             (set 'cons-sem (semaphore))
2182             (set 'prod-pid (fork (producer n))) ; start threads
2183             (set 'cons-pid (fork (consumer n)))
2184             (semaphore prod-sem signal) ; get producer started
2185             (wait-pid prod-pid) ; wait for threads to finish
2186             (wait-pid cons-pid) ;
2187             (semaphore cons-sem release) ; release semaphores
2188             (semaphore prod-sem release))
2189
2190         (run 10)
2191
2192         (exit)
2193 </pre>
2194
2195 <center>&sect;</center>
2196 <br>
2197
2198 <a name="databases"></a>
2199 <h2>18. Databases and lookup tables</h2>
2200
2201 <p>While association lists are the traditional means for associative data access in LISP and newLISP, other modern scripting languages use hashing to build memory based associative data tables. A hash is a table position calculated from a association key.</p>
2202
2203 <p>newLISP uses symbols instead of hashes. Symbols in name spaces can also be serialized to a file, as will be shown in the chapter about symbol creation and lookup.</p>
2204
2205 <h3>Association lists</h3>
2206
2207 <p>The association list is a classic LISP data structure for storing information for associative retrieval:</p>
2208
2209 <pre>
2210         ;; creating association lists
2211         ;; pushing at the end with -1 is optimized and
2212         ;; as fast as pushing in front
2213
2214         (push '("John Doe" "123-5555" 1200.00) Persons -1)
2215         (push '("Jane Doe" "456-7777" 2000.00) Persons -1)
2216             .....
2217
2218         Persons &rarr;  (
2219             ("John Doe" "123-5555" 1200.00)
2220             ("Jane Doe" "456-7777" 2000.00) ...)
2221
2222         ;; access/lookup data records
2223         (assoc "John Doe" Persons)
2224
2225         &rarr; ("John Doe" "123-5555" 1200.00 male)
2226
2227         (assoc "Jane Doe" Persons)
2228
2229         &rarr; ("Jane Doe" "456-7777" 2000.00 female)
2230 </pre>
2231
2232 <p>newLISP has a lookup function similar to what is used in spreadsheet software. This function which works like a combination of assoc and nth can find the association and pick a specific member of the data record at the same time:</p>
2233
2234 <pre>
2235         (lookup "John Doe" Persons 0)   &rarr; "123-555"
2236         (lookup "John Doe" Persons -1)  &rarr; make
2237         (lookup "Jane Doe" Persons 1)   &rarr; 2000.00
2238         (lookup "Jane Doe" Persons -2)  &rarr; 2000.00
2239
2240         ;; update data records
2241         (replace-assoc "John Doe" Persons
2242                        '("John Doe" "123-5555" 900.00 male))
2243
2244         ;; replace as a function of existing/replaced data
2245         (replace-assoc "John Doe" Persons (update-person $0))
2246
2247         ;; delete data records
2248         (replace (assoc "John Doe" Persons) Persons)
2249 </pre>
2250
2251 <h3>Nested associations</h3>
2252
2253 <p>If the data part of an association is itself an association llist, we have a nested association:</p>
2254
2255 <pre>
2256         (set 'persons '(
2257                 ("Anne" (address (country "USA") (city "New York")))
2258                 ("Jean" (address (country "France") (city "Paris")))
2259         ))
2260 </pre>
2261
2262 <p>A different syntax of the <tt>assoc</tt> function can be used to specify multiple ketys:</p>
2263
2264 <pre>
2265         ; one key
2266         (assoc (persons "Anne")) &rarr; ("Anne" (address (country "USA") (city "New York")))
2267
2268         ; two keys
2269         (assoc (persons "Anne" 'address)) &rarr; (address (country "USA") (city "New York"))
2270
2271         ; three keys
2272         (assoc (persons "Anne" 'address 'city)) &rarr; (city "New York")
2273 </pre>
2274
2275 <p>When all keys are symbols, as is in <tt>address</tt>, <tt>country</tt>
2276 and <tt>city</tt>, simple and nested associations in newLISP have the same format as newLISP
2277 <tt>FOOP</tt> (Functional Object Oriented Programming) objects. See the users manual chapter
2278 "17. Object Oriented Programming in newLISP" for details.</p>
2279
2280 <h3>Updating nested associations</h3>
2281
2282 <p>The functions <tt>set-assoc</tt> and <tt>assoc-set</tt> can be used to update simple
2283 or nested associations:</p>
2284
2285 <pre>
2286         (assoc-set (persons "Anne" 'address 'city) '(city "Boston")) &rarr; (city "New York")
2287 </pre>
2288
2289 <p>The function <tt>assoc-set</tt> returns the old association element. When usinf <tt>set-assoc</tt>, the while <tt>persons</tt> list would be returned.</p>
2290
2291 <h3>Symbol creation and lookup</h3>
2292
2293 <p><tt>lookup</tt> and <tt>assoc</tt> are fine for small tables and databases with only a few hundred data records, but are too slow for look-ups in thousands or millions of records.</p>
2294
2295 <p>newLISP has a fast and highly scalable symbol implementation. Together with name spaces and object serialization symbols can be used to maintain large data sets in memory and on disk with associative access.</p>
2296
2297 <pre>
2298         ;; creating the database
2299         (set (sym "John Doe" 'Persons) '("123-5555" 1200.00 male))
2300         (set (sym "Jane Doe" 'Persons) '("456-7777" 2000.00 female))
2301         ;; alternative shorter syntax
2302
2303         (context 'Persons "John Doe") '("123-5555" 1200.00 male))
2304         (context 'Persons "Jane Doe") '("456-7777" 2000.00 female))
2305              .....
2306         ;; save the database to a file (serialization of context Persons)
2307         (save "persons.db" 'Persons)
2308
2309         ;; load database from a file
2310         (load "persons.db")
2311
2312         ;; access data records
2313         (eval (sym "John Doe" 'Persons)) &rarr; '("123-5555" 1200.00 male)
2314
2315         ;; alternative shorter syntax to access data records
2316         (context 'Persons "John Doe") &rarr; '("123-5555" 1200.00 male)
2317
2318         ;; update data records
2319         (set (sym "John Doe" 'Persons) '("John Doe" "123-5555" 900.00 male))
2320
2321         ;; update using alternative syntax
2322         (context 'Persons "John Doe") &rarr; '("John Doe" "123-5555" 900.00 male))
2323
2324         ;; delete data records
2325         (delete (sym "John Doe" 'Persons))
2326 </pre>
2327
2328 <center>&sect;</center>
2329 <br>
2330
2331 <a name="distributed"></a>
2332 <h2>19. Distributed computing</h2>
2333
2334 <p>Many of todays applications are distributed on to several computers on the
2335 network or distibuted on to several processes on one CPU. Often both methods
2336 of distributing an application are used at the same time.</p>
2337
2338 <p>newLISP has facilities to evaluate many expressions in parallel
2339 on different network nodes or processes running newLISP. The <tt>net-eval</tt> function
2340 does all the work necessary to communicate to other nodes, distribute expressions
2341 for evaluation and collect results in either a blocking or event driven fashion.</p>
2342
2343 <p>The functions <tt>read-file</tt>, <tt>write-file</tt>, <tt>append-file</tt>
2344 and <tt>delete-file</tt> can also be used to access with files on remote nodes when using
2345 URLs in file specifications. In a similar way the functions <tt>load</tt> and
2346 <tt>save</tt> can be used to load and save code from and to remote nodes.</p>
2347
2348 <p>newLISP uses existing HTTP protocols and newLISP commandline behaviour to implement
2349 this functionality. This means that programs can be debugged and tested using standard
2350 UNIX applications like terminal, telnet or a web browser. This also enables
2351 easy integration of other tools and programs into distributed applications built with newLISP.
2352 For example the UNIX utility <i>netcat</i>, <tt>nc</tt> could be used to evaluate expressions
2353 remotely or a web browser could be used to retrieve webpages from nodes running a newLISP
2354 server.</p>
2355
2356
2357 <h3>Setting up a newLISP server node</h3>
2358
2359 <p>A newLISP server node is essentially a newLISP process listening to a network
2360 port and behaving like a newLISP commandline console and HTTP server for HTTP
2361 <tt>GET</tt>, <tt>PUT</tt>, <tt>POST</tt> and <tt>DELETE</tt> requests. Since version 9.1 newLISP
2362 server mode also answers <tt>CGI</tt> queries received by either <tt>GET</tt>
2363 or <tt>POST</tt> request.</p>
2364
2365 <p>Two methods are used to start a newLISP server node. One results in a stateful
2366 server, maintaining state in between communications with different clients, the other
2367 method a server with no state, reloading for every new client connection.</p>
2368
2369 <b>Start a newLISP stateful server</b>
2370
2371 <pre>
2372         newlisp -c -d 4711 &amp;
2373
2374         newlisp myprog.lsp -c -d 4711 &amp;
2375
2376         newlisp myprog.lsp -c -w /home/node25 -d 4711 &amp;
2377 </pre>
2378
2379 <p>newLISP is now listening on port 4711, the &amp; (ampersand) sign tells
2380 newLISP to run in the background (UNIX only). The <tt>-c</tt> switch suppresses command line
2381 prompts. newLISP now behaves like a newLISP console without prompts listening on port
2382 4711 for command line like input. Any other available port could have been chosen. Note that
2383 on UNIX ports below 1024 need administrator access rights.</p>
2384
2385 <p>The second example also preloads code. The third example also specifies a working
2386 directory using the <tt>-w</tt> option. If no working directory is specified using <tt>-w</tt>
2387 the startup directoruy is assumed to be the working directory.</p>
2388
2389 <b>Start a newLISP <tt>inetd</tt> stateless server</b>
2390
2391 <p>On UNIX the <i>inetd</i> or <i>xindetd</i> facility can be used to start a stateless
2392 server. In this case the TCP/IP net connections are managed by a special UNIX utility
2393 with the ability to handle multiple requests at the same time. For each connection
2394 made by a client the <i>inetd</i> or <i>xinetd</i> utility will start a fresh newLISP
2395 process. After the connection is closed the newLISP process will shut down.</p>
2396
2397 <p>When nodes are not required to keep state, this is the preferred method for a
2398 newLISP server node, for handeling multiple connections at the same time.</p>
2399
2400 <p>The <i>inetd</i> or <i>xinetd</i> process needs to be configured using configuration
2401 files found in the <tt>/etc</tt> directory of most UNIX installations.</p>
2402
2403 <p>For both the <i>inetd</i> and <i>xinetd</i> configurations add the following line
2404 to the <tt>/etc/services</tt> file:</p>
2405
2406 <pre>
2407         net-eval        4711/tcp     # newLISP net-eval requests
2408 </pre>
2409
2410 <p>Note that any other port than <tt>4711</tt> could be supplied.</p>
2411
2412 <p>When configuring <i>inetd</i> add also the following lines to the <tt>/etc/inetd.conf</tt>
2413 file:</p>
2414
2415 <pre>
2416         net-eval  stream  tcp  nowait  root  /usr/bin/newlisp -c
2417
2418         # as an alternative, a program can also be preloaded
2419
2420         net-eval  stream  tcp  nowait  root  /usr/bin/newlisp myprog.lsp -c
2421
2422         # a working directory can also be specified
2423
2424         net-eval  stream  tcp  nowait  newlisp  /usr/bin/newlisp -c -w /usr/home/newlisp
2425 </pre>
2426
2427 <p>The last line also specified a working directory and a user <tt>newlisp</tt>
2428 instead of the <tt>root</tt> user. This is a more secure mode limiting newLISP
2429 server node access to a specific user account with restricted permissions.</p>
2430
2431 <p>On Mac OS X and some other UNIX system a modern flavor of <i>inetd</i>: the
2432 <i>xinetd</i> facility is used. Add the following configuration to a file
2433 <tt>/etc/xinet.d/net-eval</tt>:</p>
2434
2435 <pre>
2436         service net-eval
2437         {
2438         socket_type = stream
2439         wait = no
2440         user = root
2441         server = /usr/bin/newlisp
2442         port = 4711
2443         server_args = -c -w /home/node
2444         }
2445 </pre>
2446
2447 <p>Note that a variety of parameter combinations are possible to restrict access from
2448 different places or limit access to certain users. Consult the manpages for <i>inetd</i>
2449 and <i>xinetd</i> for details.</p>
2450
2451 <p>After condiguring <i>inetd</i> or <i>xinetd</i> either process must be restarted
2452 to re-read the configuration files. This can be accomplished by sending the
2453 UNIX <tt>HUP</tt> signal to either the <i>inetd</i> or <i>xinetd</i> process using
2454 the UNIX <i>kill</i> or UNIX <i>nohup</i> utility.</p>
2455
2456
2457 <b>Testing the server using <i>telnet</i></b>
2458
2459 <p>A newLISP server node can be tested using the UNIX <i>telnet</i> utility:</p>
2460
2461 <pre>
2462         telnet localhost 4711
2463
2464         ; or when running on a different computer i.e. ip 192.168.1.100
2465
2466         telnet 192.168.1.100 4711
2467 </pre>
2468
2469 <p>Multi-line expressions can be entered by enclosing them in <tt>[cmd]</tt>, <tt>[/cmd]</tt>
2470 tags, each tag on a separate line. Both, the openeing and closing tags should be on separate lines.</p>
2471
2472
2473 <b>Testing the server using <i>netcat</i> (named <tt>nc</tt> on most UNIX)</b>
2474
2475 <pre>
2476         echo '(symbols) (exit)' | nc localhost 4711
2477 </pre>
2478
2479 <p>Or talking to a remote node:</p>
2480
2481 <pre>
2482         echo '(symbols) (exit)' | nc 192.168.1.100 4711
2483 </pre>
2484
2485 <p>In both examples <i>netcat</i> will echo back the result of evaluating the
2486 <tt>(symbols)</tt> expression.</p>
2487
2488 <p>Multi-line expressions can be entered by enclosing them in <tt>[cmd]</tt>, <tt>[/cmd]</tt>
2489 tags, each tag on a separate line.</p>
2490
2491 <b>Testing the server from a newLISP commandline</b>
2492
2493 <p>The <tt>net-eval</tt> function as a syntax form for connecting to only one remote
2494 server node. This mode is practical for quick testing from the newLISP command line:</p>
2495
2496 <pre>
2497         (net-eval "localhost" 4711 "(+ 3 4)" 1000) &rarr; 7
2498
2499         ; to a remote node
2500
2501         (net-eval "192.168.1.100" 4711 {(upper-case "newlisp")} 1000) &rarr; "NEWLISP"
2502 </pre>
2503
2504 <p>In the second example curly braces <tt>{,}</tt> are used to limit the program string
2505 for evaluation. This way quotes can be used to limit a string inside the expression.</p>
2506
2507 <p>No <tt>[cmd]</tt>, <tt>[/cmd]</tt> tags are required when sending multi-line expressions.
2508 <tt>net-eval</tt> supplies these tags automatically.</p>
2509
2510 <b>Testing the server HTTP mode using a web browser</b>
2511
2512 <p>A newLISP server also understands simple HTTP <tt>GET</tt> and <tt>PUT</tt>
2513 requests (currently UNIX only). Enter the full path of a file
2514 in the addressbar of the browser:</p>
2515
2516 <pre>
2517         http://localhost:4711//usr/share/newlisp/doc/newlisp_manual.html
2518 </pre>
2519
2520 <p>The manual file is almost 800 Kbyte in size and will take a few seconds to load into
2521 the browser. Specify the portnumber with a colon separated from the hostname or host IP.
2522 Note the double slash <tt>//</tt> necessary to specify a file address relative to the root
2523 directory.</p>
2524
2525 <h3>Evaluating multiple expressions remotely</h3>
2526
2527 <p>When testing the correct installation of newLISP server nodes, we were already
2528 sending expressions to remote node for evaluation. Many times remote evlaluation is
2529 used to split up a lengthy task into shorter subtasks for remote evaluation on different
2530 nodes.</p>
2531
2532 <p>The first example is trivial, because it only evaluates several very simple
2533 expressions remotely, but it shows the principles involved in a way easy to
2534 understand:</p>
2535
2536 <pre>
2537         #!/usr/bin/newlisp
2538
2539         (set 'result (net-eval '(
2540                 ("192.168.1.100" 4711 {(+ 3 4)})
2541                 ("192.168.1.101" 4711 {(+ 5 6)})
2542                 ("192.168.1.102" 4711 {(+ 7 8)})
2543                 ("192.168.1.103" 4711 {(+ 9 10)})
2544                 ("192.168.1.104" 4711 {(+ 11 12)})
2545         ) 1000))
2546
2547
2548         (println "result: " result)
2549
2550         (exit)
2551 </pre>
2552
2553 <p>Running this program will produce the following output:</p>
2554
2555 <pre>
2556         result: (7 11 15 19 23)
2557 </pre>
2558
2559 <p>When running UNIX and using an <i>inetd</i> or <i>xinetd</i> configured newLISP
2560 server, the servers and programs can be run on just one CPU replacing all IP numbers with
2561 <tt>"localhost"</tt> or the same local IP number. The <i>indetd</i> or <i>xinetd</i>
2562 daemon will then start 5 independent newLISP processes. On Win32 5 statefull
2563 newLISP servers could be started on different port numbers to accomplish the same.</p>
2564
2565 <p>Instead of collecting all results at one on the return of <tt>net-eval</tt>, a
2566 call back function can be used to receive and process results as they become available:</p>
2567
2568 <pre>
2569         #!/usr/bin/newlisp
2570
2571         (define (idle-loop p)
2572                 (if p (println p)))
2573
2574         (set 'result (net-eval '(
2575                 ("192.168.1.100" 4711 {(+ 3 4)})
2576                 ("192.168.1.101" 4711 {(+ 5 6)})
2577                 ("192.168.1.102" 4711 {(+ 7 8)})
2578                 ("192.168.1.103" 4711 {(+ 9 10)})
2579                 ("192.168.1.104" 4711 {(+ 11 12)})
2580         ) 1000 idle-loop))
2581
2582         (exit)
2583 </pre>
2584
2585 <p>While <tt>net-eval</tt> is waiting for results it calls the function <tt>idle-loop</tt>
2586 repeatedly with parameter <tt>p</tt>. The parameter <tt>p</tt> is <tt>nil</tt> when no result
2587 was received during the last 100 micro seconds or <tt>p</tt> contains a list sent back
2588 from the remote node. The list contains the remote address and port and the evaluation result.
2589 The example shown would generate the following output:</p>
2590
2591 <pre>
2592         ("192.168.1.100" 4711 7)
2593         ("192.168.1.101" 4711 11)
2594         ("192.168.1.102" 4711 15)
2595         ("192.168.1.103" 4711 19)
2596         ("192.168.1.104" 4711 23)
2597 </pre>
2598
2599 <p>For testing on just one CPU replace addresses with <tt>"localhost"</tt>
2600 the UNIX <i>inetd</i> or <i>xinetd</i> daemon will start a separate process
2601 for each connection made and all listening on port <tt>4711</tt>. When using
2602 a stateful server on the same Win32 CPU specify a different port number for
2603 each server</p>
2604
2605 <b>Setting up the <tt>net-eval</tt> parameter structure</b>
2606
2607 <p>In a networked environment where an application gets moved around, or server
2608 nodes with changing IP nubers are used, it is necessary to set up the node
2609 parameters in the <tt>net-eval</tt> parameter list as variables. The following
2610 more complex example shows how this can be done. The example also shows how
2611 a bigger piece of program text can be transferred to a remote node for evaluation
2612 and how this program piece can be customized for each node differently:</p>
2613
2614 <pre>
2615         #!/usr/bin/newlisp
2616
2617         ; node parameters
2618         (set 'nodes '(
2619                 ("192.168.1.100" 4711)
2620                 ("192.168.1.101" 4711)
2621                 ("192.168.1.102" 4711)
2622                 ("192.168.1.103" 4711)
2623                 ("192.168.1.104" 4711)
2624         ))
2625
2626         ; program template for nodes
2627         (set 'program [text]
2628         (begin
2629                 (map set '(from to node) '(%d %d %d))
2630                 (for (x from to)
2631                         (if (= 1 (length (factor x)))
2632                         (push x primes -1)))
2633                 primes)
2634         [/text])
2635
2636         ; call back routine for net-eval
2637         (define (idle-loop p)
2638                 (if p
2639                         (begin
2640                                 (println (p 0) ":" (p 1))
2641                                 (push (p 2) primes))))
2642
2643         (println "Sending request to nodes, and waiting ...")
2644
2645         ; machines could be on different IP addresses.
2646         ; For this test 5 nodes are started on localhost
2647         (set 'result (net-eval '(
2648                 ((nodes 0 0) (nodes 0 1) (format program 0 99999 1))
2649                 ((nodes 1 0) (nodes 1 1) (format program 100000 199999 2))
2650                 ((nodes 2 0) (nodes 2 1) (format program 200000 299999 3))
2651                 ((nodes 3 0) (nodes 3 1) (format program 300000 399999 4))
2652                 ((nodes 4 0) (nodes 4 1) (format program 400000 499999 5))
2653         ) 20000 idle-loop))
2654
2655         (set 'primes (sort (flat primes)))
2656         (save "primes" 'primes)
2657
2658         (exit)
2659 </pre>
2660
2661 <p>At the beginning of the program a <tt>nodes</tt> list structure
2662 contains all the relevant node information for hostname and port.</p>
2663
2664 <p>The <tt>program</tt> calculates all prime numbers in a given
2665 range. The <tt>from</tt>, <tt>to</tt> and <tt>node</tt> variables
2666 are configured into the program text using <tt>format</tt>. All
2667 instructions are placed into a <tt>begin</tt> expression block, so
2668 only one expression result will be send back from the remote node.</p>
2669
2670 <p>Many other schemes to configure a <tt>net-eval</tt> paramneter list
2671 are possible. The example shows, that <tt>net-eval</tt> evaluates
2672 the node parameter specifications inside the quoted list. The following
2673 scheme would give the same resuls:</p>
2674
2675 <pre>
2676         (set 'node-eval-list '(
2677                 ((nodes 0 0) (nodes 0 1) (format program 0 99999 1))
2678                 ((nodes 1 0) (nodes 1 1) (format program 100000 199999 2))
2679                 ((nodes 2 0) (nodes 2 1) (format program 200000 299999 3))
2680                 ((nodes 3 0) (nodes 3 1) (format program 300000 399999 4))
2681                 ((nodes 4 0) (nodes 4 1) (format program 400000 499999 5))
2682         ))
2683
2684         (set 'result (net-eval node-eval-list  20000 idle-loop))
2685 </pre>
2686
2687 <p>The function <tt>idle-loop</tt> aggregates all lists of primes
2688 received and generates the following output:</p>
2689
2690 <pre>
2691         192.168.1.100:4711
2692         192.168.1.101:4711
2693         192.168.1.102:4711
2694         192.168.1.103:4711
2695         192.168.1.104:4711
2696 </pre>
2697
2698 <p>As with the previous examples all IP numbers could be replaced
2699 with <tt>"localhost"</tt> or any other hostname or IP number to test a distributed
2700 application on a single host before deployment in a distributed environment
2701 with many networked hosts.</p>
2702
2703 <h3>Transferring files to and from remote nodes</h3>
2704
2705 <p>Files can be read from or written to remote nodes with the same functions
2706 used to read and write files to a local file system. This functionality is
2707 currently only available on UNIX systems when talking to newLISP servers. As
2708 functions are based on standard <tt>GET</tt> and <tt>PUT</tt> HTTP protocols
2709 they can also be used communicating with web servers. Note that few Apache
2710 webserver installations have enabled the <tt>PUT</tt> protocol by default.</p>
2711
2712 <p>The functions <tt>read-file</tt>, <tt>write-file</tt> and <tt>append-file</tt>
2713 can all take URLs in their filename specifications for reading from and
2714 writing to remote nodes running a newLISP server or a webserver:</p>
2715
2716 <pre>
2717         (write-file "http://127.0.0.1:4711//Users/newlisp/afile.txt" "The message - ")
2718         &rarr; "14 bytes transferred for /Users/lutz/afile.txt\r\n"
2719
2720         (append-file "http://127.0.0.1:4711//Users/newlisp/afile.txt" "more text")
2721         &rarr; "9 bytes transferred for /Users/lutz/afile.txt\r\n"
2722
2723         (read-file "http://127.0.0.1:4711//Users/newlisp/afile.txt")
2724         &rarr; "The message - more text"
2725 </pre>
2726
2727 <p>The first two function return a message starting with the numbers of bytes
2728 transferred and the name of the remote file affected. The <tt>-read-file</tt>
2729 function returns the contents received.</p>
2730
2731 <p>Under all error conditions an error message startriong with the characters
2732 <tt>ERR:</tt> would be returned:</p>
2733
2734 <pre>
2735         (read-file "http://127.0.0.1:4711//Users/newlisp/somefile.txt")
2736         &rarr; "ERR:404 File not found: /Users/newlisp/somefile.txt\r\n"
2737 </pre>
2738
2739 <p>Note the double backslash necessary to reference files relative to root on the server
2740 node.</p>
2741
2742 <p>All functions can be used to transfer binary non-ascii contents containing zero
2743 characters. Internally newLISP uses the functions <tt>get-url</tt> and <tt>put-url</tt>,
2744 which could be used instead of the functions <tt>read-file</tt>, <tt>write-file</tt> and
2745 <tt>append-file</tt>. Additional options like used with <tt>get-url</tt> and <tt>put-url</tt>
2746 could be used with the functions <tt>read-file</tt>, <tt>write-file</tt> and
2747 <tt>append-file</tt> as well. For more detail see the newLISP function reference for these
2748 functions.</p>
2749
2750
2751 <h3>Loading and saving data from and to remote nodes</h3>
2752
2753 <p>The same <tt>load</tt> and <tt>save</tt> fuctions used to load program or LISP data
2754 from a local file system can be used to load or save programs and LISP data from or
2755 to remote nodes.</p>
2756
2757 <p>By using URLs in the file specifications of <tt>load</tt> and <tt>save</tt> these
2758 functions can work over the network commuicating with a newLISP server node.:</p>
2759
2760 <pre>
2761         (load "http://192.168.1.2:4711//usr/share/newlisp/mysql5.lsp")
2762
2763         (save "http://192.168.1.2:4711//home/newlisp/data.lsp" 'db-data)
2764 </pre>
2765
2766 <p>Although the <tt>load</tt> and <tt>save</tt> functions inernally use <tt>get-url</tt> and <tt>put-url</tt> to perform its works they behave  exactly as when used on a local filesystem, but instead of a
2767 file path URLs are specified. Both function will timeout after 60 seconds if a connection could
2768 not be established. When finer control is necessary use the functions <tt>get-url</tt> and
2769 <tt>put-url</tt> together with <tt>eval-string</tt> and <tt>source</tt> to realize a similar
2770 result as when using the <tt>load</tt> and <tt>save</tt> in HTTP mode.</p>
2771
2772 <h3>HTTPD-only mode, newLISP as a webserver</h3>
2773
2774 <p>In all previous chapters the <tt>-c</tt> server mode was used. This mode can act as
2775 a <tt>net-eval</tt> server and at the same time answer <tt>HTTP</tt> requests for serving
2776 web pages or transfer of files and programs. The <tt>-c</tt> mode is the preferred mode
2777 for secure operation behind a firewall. newLISP also has a <tt>-http</tt> mode which works
2778 like a resticted <tt>-c</tt> mode. In <tt>-http</tt> mode only <tt>HTTP</tt> requests are
2779 served and commandline like formatted requests and <tt>net-eval</tt> requests are not answered.
2780 In this mode newLISP can act like a web server answering HTTP <tt>GET</tt>, <tt>PUT</tt>,
2781 <tt>POST</tt> and <tt>DELETE</tt> requests as well as <tt>CGI</tt> requests, but additional
2782 efforts should be made to restrict the access to aunauthorized files and directories to secure
2783 the server when exposed to the internet.</p>
2784
2785 <p>When newLISP server answers any <tt>HTTP</tt> request, it first calls a function
2786 <tt>httpd-conf</tt>. This function can be supplied in a startup file:</p>
2787
2788 <pre>
2789     server_args = httpd-conf.lsp -http -w /home/node
2790 </pre>
2791
2792 <p>The above snippet shows part of a <i>xinetd</i> configuration file. A startup
2793 program <tt>httpd-conf.lsp</tt> has been added which will be loaded upon invocation
2794 of newLISP. The <tt>-c</tt> options has been replaced with the <tt>-http</tt> option.
2795 Now neither <tt>net-eval</tt> nor commandline requests will be answered but only
2796 HTTP requests.</p>
2797
2798 <p>The file <tt>httpd-conf.lsp</tt> contains a user-defined function
2799 <tt>http-conf</tt> which takes as arguments the HTTP path-name string of the request and the
2800 query string part of the request string. If the <tt>http-conf</tt> function
2801 returns <tt>nil</tt> the request will be aborted, else whatever string is returned will
2802 be taken as the path-name for further processing. This can be used to transform or
2803 redirect requested file names. <tt>http-conf</tt> can also be used to analyze headers for
2804 redirection or to respond with customized error pages. The function does not need to
2805 process the query string but must either return a valid path-name
2806 for further processing or <tt>nil</tt> to abort request processing.</p>
2807
2808 <p>The following small <tt>httpd-conf.lsp</tt> file will transform the path-name
2809 into a request for <tt>errorpage.html</tt> if the original path-name ends
2810 either with <tt>.exe</tt> or starts with <tt>/</tt> trying to access the root
2811 directory of the server, or tries the access any other directory outside
2812 the current directory:</p>
2813
2814 <pre>
2815         (define (httpd-conf path-name query)
2816                 (if (or
2817                                 (ends-with path-name ".exe")
2818                                 (starts-with path-name "/")
2819                                 (find ".." path-name)
2820                         )
2821                 "errorpage.html"
2822                 ; else
2823                 path-name
2824 ))
2825 </pre>
2826
2827 <p>The following requests would all be rejected:</p>
2828
2829 <pre>
2830         http://asite.com/cgi-bin/prog.exe   ; trying to access prog.exe
2831         http://asite.com//usr/etc/passwd    ; trying to access path from / root
2832         http://asite.com/../../var/log      ; trying to hop out of web directory
2833 </pre>
2834
2835 <p>A more elaborate <tt>httpd-conf</tt> file can be found in the source
2836 distribution of newLISP in the example directory. On UNIX systems this
2837 file is also installed as <tt>/usr/share/newlisp/httpd-conf.lsp</tt>.
2838 This fille contains a <tt>httpd-conf</tt> function which also sends back a
2839 <tt>Location:</tt> response with appended forward slash (<tt>/</tt>) for directory
2840 requests without a trailing <tt>/</tt>. When a directory is requested,
2841 newLISP HTTP mode will try to access either <tt>index.html</tt> or
2842 <tt>index.cgi</tt> in that directory.</p>
2843
2844 <h3>Media types in HTTP modes</h3>
2845
2846 <p>In both the <tt>-c</tt> and <tt>-http</tt> HTTP modes the following file types are
2847 recognized and a correctly formatted <tt>Content-Type:</tt> header is sent back:</p>
2848
2849 <center>
2850 <table border="1" cellpadding="3"  width="50%">
2851 <tr align="left"><th>file extension</th><th>media type</th></tr>
2852 <tr><td><tt>.jpg</tt></td><td><tt>image/jpg</tt></td></tr>
2853 <tr><td><tt>.pgn</tt></td><td><tt>image/png</tt></td></tr>
2854 <tr><td><tt>.gif</tt></td><td><tt>image/gif</tt></td></tr>
2855 <tr><td><tt>.pdf</tt></td><td><tt>application/pdf</tt></td></tr>
2856 <tr><td><tt>.mp3</tt></td><td><tt>image/mpeg</tt></td></tr>
2857 <tr><td><tt>.mov</tt></td><td><tt>image/quicktime</tt></td></tr>
2858 <tr><td><tt>.mpg</tt></td><td><tt>image/mpeg</tt></td></tr>
2859 <tr><td><em>any other</em></td><td><tt>text/html</tt></td></tr>
2860 </table>
2861 <p><font size="-1">media types in newLISP HTTP request handling</font></p>
2862 </center>
2863
2864 <h3>Environment variables set</h3>
2865
2866 <p>When receiving HTTP requests newLISP server mode will extract information from the HTTP request header and configure <tt>HTTP_HOST</tt>, <tt>HTTP_USER_AGENT</tt> and <tt>HTTP_COOKIE</tt> (v.9.1.4) in the environment of the host operating system. If a <tt>httpd-conf</tt> functions was loaded on newLISP server startup header processing will happen afer processing <tt>httpd-conf</tt>. The environment variables can be accessed by a CGI process.</p>
2867
2868 <h3>Local domain UNIX sockets</h3>
2869
2870 <p>newLISP supports named local domain sockets in newLISP server mode and using the built-in functions <tt>net-eval</tt>, <tt>net-listen</tt>, <tt>net-connect</tt> together with the functions <tt>net-accept</tt>, <tt>net-receive</tt>, <tt>net-select</tt> and <tt>net-send</tt>.</p>
2871
2872 <p>Using local domain sockets fast communications between processes on the same file system and with newLISP servers is possible. See the Users Manual for more details.</p>
2873
2874 <br /><br />
2875
2876 <center>&sect;</center>
2877
2878 <a name="extending"></a>
2879 <h2>20. Extending newLISP</h2>
2880
2881
2882 <p>newLISP has an import function, which allows importing function from DLLs (Dynamic Link Libraries) on Win32 or shared libraries on Linux/UNIX (ending in .so).</p>
2883
2884 <p>This chapter shows how to compile and use libraries on both, Win32 and Linux/UNIX platforms. We will compile a DLL and a Linux/UNIX shared library from the following 'C' program:</p>
2885
2886 <pre>
2887         #include &lt;stdio.h&gt;
2888         #include &lt;stdlib.h&gt;
2889         #include &lt;ctype.h>&gt;
2890
2891         int foo1(char * ptr, int number)
2892         {
2893         printf("the string: %s the number: %d\n", ptr, number);
2894         return(10 * number);
2895         }
2896
2897         char * foo2(char * ptr, int number)
2898         {
2899         char * upper;
2900         printf("the string: %s the number: %d\n", ptr, number);
2901         upper = ptr;
2902         while(*ptr) { *ptr = toupper(*ptr); ptr++; }
2903         return(upper);
2904         }
2905
2906         /* eof */
2907 </pre>
2908
2909 <p>Both functions foo1 and foo2 print their arguments, but while foo1 returns the number multiplied 10 times, foo2 returns the uppercase of a string to show how to return strings from 'C' functions.</p>
2910
2911 <h3>Compile a shared library on Linux/UNIX/MacOSX</h3>
2912
2913 <p>On Linux/UNIX we can compile and link testlib.so in one step:</p>
2914
2915 <pre>
2916         gcc testlib.c -shared -o testlib.so
2917 </pre>
2918
2919 <p>Or On Mac OSX/darwin do:</p>
2920
2921 <pre>
2922         gcc -bundle -o testlib.so
2923 </pre>
2924
2925 <p>The library testlib.so will be built with Linux/UNIX default cdecl conventions. Importing the library is very similar on both Linux and Win32 platforms, but on Win32 the library can be found in the current directory. You may have to specify the full path or put the library in the library path of the os:</p>
2926
2927 <pre>
2928          newLISP v.8.4.3 on Linux, execute 'newlisp -h' for more info.
2929
2930         &gt; (import "/home/newlisp/testlib.so" "foo1")
2931         foo1 &lt;6710118F&gt;
2932
2933         &gt; (import "/home/newlisp/testlib.so" "foo2")
2934         foo2 &lt;671011B9&gt;
2935
2936         &gt; (foo1 "hello" 123)
2937         the string: hello the number: 123
2938         1230
2939
2940         &gt; (foo2 "hello" 123)
2941         the string: hello the number: 123
2942         4054088
2943
2944         &gt; (get-string (foo2 "hello" 123))
2945         the string: hello the number: 123
2946         "HELLO"
2947         &gt;
2948 </pre>
2949
2950 <p>Again, the number returned from foo2 is the string address pointer and get-string can be used to access the string. When using get-string only character up to a zero byte are returned. When returning the addresses to binary buffers different techniques using unpack are used to access the information.</p>
2951
2952 <h3>Compile a DLL on Win32</h3>
2953
2954 <p>DLLs on Win32 can be made using the MinGW, Borland or CYGWIN compilers. This example shows, how to do it using the MinGW compiler.</p>
2955
2956 <p>Compile it:</p>
2957
2958 <pre>
2959         gcc -c testlib.c -o testlib.o
2960 </pre>
2961
2962 <p>Before we can transform testlib.o into a DLL we need a testlib.def declaring the exported functions:</p>
2963
2964 <pre>
2965         LIBRARY    testlib.dll
2966         EXPORTS
2967             foo1 @1 foo1
2968             foo2 @2 foo2
2969 </pre>
2970
2971 <p>Now wrap the DLL:</p>
2972
2973 <pre>
2974         dllwrap testlib.o --def testlib.def -o testlib.dll -lws2_32
2975 </pre>
2976
2977 <p>The library testlib.dll will be built with default Win32 stdcall conventions. The following shows an interactive session, importing the library and using the functions:</p>
2978
2979 <pre>
2980          newLISP v.8.4.3 on Win32 MinGW, execute 'newlisp -h' for more info.
2981         &gt; (import "testlib.dll" "foo1")
2982         foo1 &lt;6710118F&gt;
2983
2984         &gt; (import "testlib.dll" "foo2")
2985         foo2 &lt;671011B9&gt;
2986
2987         &gt; (foo1 "hello" 123)
2988         the string: hello the number: 123
2989         1230
2990
2991         &gt; (foo2 "hello" 123)
2992         the string: hello the number: 123
2993         4054088
2994
2995         &gt; (get-string (foo2 "hello" 123))
2996         the string: hello the number: 123
2997         "HELLO"
2998
2999         &gt;
3000         ; import a library compiled for cdecl
3001         ; calling conventsions
3002         &gt; (import "foo.dll" "func" "cdecl")
3003 </pre>
3004
3005 <p>Note that the first time using foo2 the return value 4054088 is the memory address of the string returned. Using get-string the string belonging to it can be accessed. If the library is compiled using cdecl calling conventions, the cdecl keyword must be used in the import expression.</p>
3006
3007 <h3>Using data structures in imported libraries</h3>
3008
3009 <p>Just like 'C' strings are returned using string pointers, 'C' structures can be returned using structure pointers and functions like get-string, get-int or get-char can be used to access the members. The following example illustrates this:</p>
3010
3011 <pre>
3012         typedef struct mystruc
3013            {
3014            int number;
3015            char * ptr;
3016            } MYSTRUC;
3017
3018         MYSTRUC * foo3(char * ptr, int num )
3019            {
3020            MYSTRUC * astruc;
3021            astruc = malloc(sizeof(MYSTRUC));
3022            astruc-&gt;ptr = malloc(strlen(ptr) + 1);
3023            strcpy(astruc-&gt;ptr, ptr);
3024            astruc-&gt;number = num;
3025            return(astruc);
3026            }
3027 </pre>
3028
3029 <p>The newLISP program would access the structure members as follows:</p>
3030
3031 <pre>
3032         &gt; (set 'astruc (foo3 "hello world" 123))
3033         4054280
3034
3035         &gt; (get-string (get-integer (+ astruc 4)))
3036         "hello world"
3037
3038         &gt; (get-integer astruc)
3039         123
3040         &gt;
3041 </pre>
3042
3043 <p>The return value from foo is the address to the structure astruc. To access the string pointer, 4 must be added as the size of an integer type in the 'C' programming language. The string in the string pointer then gets accessed using get-string.</p>
3044
3045 <h3>Unevenly aligned data structures</h3>
3046
3047 <p>Sometimes data structures contain data types of different length than the normal CPU register word:</p>
3048
3049 <pre>
3050         sruct mystruct
3051            {
3052            short int x;
3053            int z;
3054            short int y;
3055            } data;
3056
3057         struct mystruct * foo(void)
3058            {
3059            data.x = 123;
3060            data.y = 456;
3061            data.z = sizeof(data);
3062            return(&amp;data);
3063            }
3064 </pre>
3065
3066 <p>The x and y variables are 16 bit wide and only z takes 32 bit. When a compiler on a 32bit CPU packs this structure the variables x and y will each fill up 32 bits instead of the 16 bit each. This is necessary so the 32 bit variable z can be aligned properly. The following code would be necessary to access the structure members:</p>
3067
3068 <pre>
3069         &gt; (import "/usr/home/nuevatec/test.so" "foo")
3070            foo &lt;281A1588&gt;
3071
3072         &gt; (unpack "lu lu lu" (foo))
3073            (123 12 456)
3074 </pre>
3075 <p>The whole structure consumes 3 by 4 = 12 bytes, because all members have to be aligned to 32 bit borders in memory.</p>
3076 <p>The following data structure packs the short 16 bit variables next to each other. This time only 8 bytes are required: 2 each for x and y and 4 bytes or z. Because x and y are together in one 32 bit word, none of the variables needs to cross a 32 bit boundary: </p>
3077
3078 <pre>
3079         struct mystruct
3080             {
3081             short int x;
3082             short int y;
3083             int z;
3084             } data;
3085
3086         struct mystruct * foo(void)
3087            {
3088            data.x = 123;
3089            data.y = 456;
3090            data.z = sizeof(data);
3091            return(&amp;data);
3092            }
3093 </pre>
3094
3095 <p>This time the access code in newLISP reflects the size of the structure members:</p>
3096
3097 <pre>
3098         &gt; (import "/usr/home/nuevatec/test.so" "foo")
3099            foo &lt;281A1588&gt;
3100
3101         &gt; (unpack "u u lu" (foo))
3102            (123 456 8)
3103 </pre>
3104
3105 <br>
3106
3107 <h3>Passing parameters in library calls</h3>
3108
3109 <center>
3110 <table border="1" cellpadding="3" width="95%">
3111 <tr><th>Data Type</th><th>newLISP call</th><th>C Function call</th></tr>
3112 <tr><td><em>integer</em></td><td><tt>(foo 123)</tt></td><td><tt>foo(int number)</tt></td></tr>
3113 <tr><td><em>double float</em></td><td><tt>(foo 1.234)</tt></td><td><tt>foo(double number)</tt></td></tr>
3114 <tr><td><em>float</em></td><td><tt>(foo (flt 1.234))</tt></td><td><tt>foo(float number)</tt></td></tr>
3115 <tr><td><em>string</em></td><td><tt>(foo "Hello World!")</tt></td><td><tt>foo(char * string)</tt></td></tr>
3116 <tr><td><em>integer array</em></td><td><tt>(foo (pack "d d d" 123 456 789))</tt></td><td><tt>foo(int numbers[])</tt></td></tr>
3117 <tr><td><em>float array</em></td><td><tt>(foo (pack "f f f" 1.23 4.56 7.89))</tt></td><td><tt>foo(float[])</tt></td></tr>
3118 <tr><td><em>double array</em></td><td><tt>(foo (pack "lf lf lf) 1.23 4.56 7.89)</tt></td><td><tt>foo(double[])</tt></td></tr>
3119 <tr><td><em>string array</em></td><td><tt>(foo (pack "lu lu lu" "one" "two" "three")))</tt></td><td><tt>foo(char * string[])</tt></td></tr>
3120 </table>
3121 </center>
3122 <br>
3123
3124 <p>Note that <em>floats</em> and <em>double floats</em> are only passed correctly on x86 platforms
3125 with <em>cdecl</em> calling conventions or when passed by pointer reference as in variable argument
3126 functions, i.e: <em>printf()</em>.</p>
3127
3128 <br />
3129
3130 <h3>Extracting return values from library calls</h3>
3131 <center>
3132 <table border="1" cellpadding="3" width="95%">
3133 <tr><th>Data Type</th><th>newLISP to extract return value</th><th>C return</th></tr>
3134 <tr><td><em>integer</em></td><td><tt>(set 'number (foo x y z))</tt></td><td><tt>return(int number)</tt></td></tr>
3135 <tr><td><em>double float</em></td><td>n/a - only 32bit returns, use double float pointer instead</td><td>not available</td></tr>
3136 <tr><td><em>double float ptr</em></td><td><tt>(set 'number (get-float (foo x y z)))</tt></td><td><tt>return(double * numPtr)</tt></td></tr>
3137 <tr><td><em>float</em></td><td>not available</td><td>not available</td></tr>
3138 <tr><td><em>string</em></td><td><tt>(set 'string (get-string (foo x y z)</tt></td><td><tt>return(char * string)</tt></td></tr>
3139 <tr><td><em>integer array</em></td><td><tt>(set 'numList (unpack "ld ld ld" (foo x y z)))</tt></td><td><tt>return(int numList[])</tt></td></tr>
3140 <tr><td><em>float array</em></td><td><tt>(set 'numList (unpack "f f f" (foo x y z)))</tt></td><td><tt>return(float numList[])</tt></td></tr>
3141 <tr><td><em>double array</em></td><td><tt>(set 'numList (unpack "lf lf lf") (foo x y z)))</tt></td><td><tt>return(double numList[])</tt></td></tr>
3142 <tr><td><em>string array</em></td><td><tt>(set 'stringList (map get-string (unpack "ld ld ld" (foo x y z))))</tt>
3143 </td><td><tt>return(char * string[])</tt></td></tr>
3144 </table>
3145 </center>
3146
3147 <p><em>Floats</em> and <em>doubles</em> can only be returned via address pointer references.</p>
3148
3149 <p>When returning array types the number of elements in the array must be known. The examples always assume 3 elements.</p>
3150
3151 <p>All pack and unpack and formats can also be given without spaces, but are spaced in the examples for better readability.</p>
3152
3153 <p>The formats "ld" and "lu" are interchangeable, but the 16 bit formats "u" and "d" may produce different results, because of sign expansion from 16 to 32 bits.</p>
3154
3155 <p>Flags are available for changing endian byte order during <tt>pack</tt> and <tt>unpack</tt>.</p>
3156
3157 <center>&part;</center>
3158
3159 <br><br><br><br>
3160
3161 <hr>
3162
3163 <br><br>
3164
3165 <a NAME="appendix"></a>
3166 <a NAME="GFDL"></a>
3167 <center>
3168 <h2><font color="#EE0000">GNU Free Documentation License</font></h2>
3169 <p>Version 1.2, November 2002</p>
3170
3171 <p>
3172 Copyright (C) 2000,2001,2002  Free Software Foundation, Inc.
3173 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
3174 Everyone is permitted to copy and distribute verbatim copies
3175 of this license document, but changing it is not allowed.</p>
3176 </center>
3177
3178 <br><br>
3179
3180 <b>0. PREAMBLE</b>
3181
3182 <p>The purpose of this License is to make a manual, textbook, or other
3183 functional and useful document "free" in the sense of freedom: to
3184 assure everyone the effective freedom to copy and redistribute it,
3185 with or without modifying it, either commercially or noncommercially.
3186 Secondarily, this License preserves for the author and publisher a way
3187 to get credit for their work, while not being considered responsible
3188 for modifications made by others.
3189 </p>
3190 <p>This License is a kind of "copyleft", which means that derivative
3191 works of the document must themselves be free in the same sense. It
3192 complements the GNU General Public License, which is a copyleft
3193 license designed for free software.
3194 </p>
3195 <p>We have designed this License in order to use it for manuals for
3196 free
3197 software, because free software needs free documentation: a free
3198 program should come with manuals providing the same freedoms that the
3199 software does. But this License is not limited to software manuals;
3200 it can be used for any textual work, regardless of subject matter or
3201 whether it is published as a printed book. We recommend this License
3202 principally for works whose purpose is instruction or reference.
3203 </p>
3204 <p><b>1. APPLICABILITY AND DEFINITIONS</b>
3205 </p>
3206 <p>This License applies to any manual or other work, in any medium,
3207 that
3208 contains a notice placed by the copyright holder saying it can be
3209 distributed under the terms of this License. Such a notice grants a
3210 world-wide, royalty-free license, unlimited in duration, to use that
3211 work under the conditions stated herein. The "Document", below,
3212 refers to any such manual or work. Any member of the public is a
3213 licensee, and is addressed as "you". You accept the license if you
3214 copy, modify or distribute the work in a way requiring permission
3215 under copyright law.
3216 </p>
3217 <p>A "Modified Version" of the Document means any work containing the
3218 Document or a portion of it, either copied verbatim, or with
3219 modifications and/or translated into another language.
3220 </p>
3221 <p>A "Secondary Section" is a named appendix or a front-matter section
3222 of
3223 the Document that deals exclusively with the relationship of the
3224 publishers or authors of the Document to the Document's overall subject
3225 (or to related matters) and contains nothing that could fall directly
3226 within that overall subject. (Thus, if the Document is in part a
3227 textbook of mathematics, a Secondary Section may not explain any
3228 mathematics.) The relationship could be a matter of historical
3229 connection with the subject or with related matters, or of legal,
3230 commercial, philosophical, ethical or political position regarding
3231 them.
3232 </p>
3233 <p>The "Invariant Sections" are certain Secondary Sections whose titles
3234 are designated, as being those of Invariant Sections, in the notice
3235 that says that the Document is released under this License. If a
3236 section does not fit the above definition of Secondary then it is not
3237 allowed to be designated as Invariant. The Document may contain zero
3238 Invariant Sections. If the Document does not identify any Invariant
3239 Sections then there are none.
3240 </p>
3241 <p>The "Cover Texts" are certain short passages of text that are
3242 listed,
3243 as Front-Cover Texts or Back-Cover Texts, in the notice that says that
3244 the Document is released under this License. A Front-Cover Text may
3245 be at most 5 words, and a Back-Cover Text may be at most 25 words.
3246 </p>
3247 <p>A "Transparent" copy of the Document means a machine-readable copy,
3248 represented in a format whose specification is available to the
3249 general public, that is suitable for revising the document
3250 straightforwardly with generic text editors or (for images composed of
3251 pixels) generic paint programs or (for drawings) some widely available
3252 drawing editor, and that is suitable for input to text formatters or
3253 for automatic translation to a variety of formats suitable for input
3254 to text formatters. A copy made in an otherwise Transparent file
3255 format whose markup, or absence of markup, has been arranged to thwart
3256 or discourage subsequent modification by readers is not Transparent.
3257 An image format is not Transparent if used for any substantial amount
3258 of text. A copy that is not "Transparent" is called "Opaque".
3259 </p>
3260 <p>Examples of suitable formats for Transparent copies include plain
3261 ASCII without markup, Texinfo input format, LaTeX input format, SGML
3262 or XML using a publicly available DTD, and standard-conforming simple
3263 HTML, PostScript or PDF designed for human modification. Examples of
3264 transparent image formats include PNG, XCF and JPG. Opaque formats
3265 include proprietary formats that can be read and edited only by
3266 proprietary word processors, SGML or XML for which the DTD and/or
3267 processing tools are not generally available, and the
3268 machine-generated HTML, PostScript or PDF produced by some word
3269 processors for output purposes only.
3270 </p>
3271 <p>The "Title Page" means, for a printed book, the title page itself,
3272 plus such following pages as are needed to hold, legibly, the material
3273 this License requires to appear in the title page. For works in
3274 formats which do not have any title page as such, "Title Page" means
3275 the text near the most prominent appearance of the work's title,
3276 preceding the beginning of the body of the text.
3277 </p>
3278 <p>A section "Entitled XYZ" means a named subunit of the Document whose
3279 title either is precisely XYZ or contains XYZ in parentheses following
3280 text that translates XYZ in another language. (Here XYZ stands for a
3281 specific section name mentioned below, such as "Acknowledgements",
3282 "Dedications", "Endorsements", or "History".) To "Preserve the Title"
3283 of such a section when you modify the Document means that it remains a
3284 section "Entitled XYZ" according to this definition.
3285 </p>
3286 <p>The Document may include Warranty Disclaimers next to the notice
3287 which
3288 states that this License applies to the Document. These Warranty
3289 Disclaimers are considered to be included by reference in this
3290 License, but only as regards disclaiming warranties: any other
3291 implication that these Warranty Disclaimers may have is void and has
3292 no effect on the meaning of this License.
3293 </p>
3294 <p><b>2. VERBATIM COPYING</b>
3295 </p>
3296 <p>You may copy and distribute the Document in any medium, either
3297 commercially or noncommercially, provided that this License, the
3298 copyright notices, and the license notice saying this License applies
3299 to the Document are reproduced in all copies, and that you add no other
3300 conditions whatsoever to those of this License. You may not use
3301 technical measures to obstruct or control the reading or further
3302 copying of the copies you make or distribute. However, you may accept
3303 compensation in exchange for copies. If you distribute a large enough
3304 number of copies you must also follow the conditions in section 3.
3305 </p>
3306 <p>You may also lend copies, under the same conditions stated above,
3307 and
3308 you may publicly display copies.
3309 </p>
3310 <p><b>3. COPYING IN QUANTITY</b>
3311 </p>
3312 <p>If you publish printed copies (or copies in media that commonly have
3313 printed covers) of the Document, numbering more than 100, and the
3314 Document's license notice requires Cover Texts, you must enclose the
3315 copies in covers that carry, clearly and legibly, all these Cover
3316 Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
3317 the back cover. Both covers must also clearly and legibly identify
3318 you as the publisher of these copies. The front cover must present
3319 the full title with all words of the title equally prominent and
3320 visible. You may add other material on the covers in addition.
3321 Copying with changes limited to the covers, as long as they preserve
3322 the title of the Document and satisfy these conditions, can be treated
3323 as verbatim copying in other respects.
3324 </p>
3325 <p>If the required texts for either cover are too voluminous to fit
3326 legibly, you should put the first ones listed (as many as fit
3327 reasonably) on the actual cover, and continue the rest onto adjacent
3328 pages.
3329 </p>
3330 <p>If you publish or distribute Opaque copies of the Document numbering
3331 more than 100, you must either include a machine-readable Transparent
3332 copy along with each Opaque copy, or state in or with each Opaque copy
3333 a computer-network location from which the general network-using
3334 public has access to download using public-standard network protocols
3335 a complete Transparent copy of the Document, free of added material.
3336 If you use the latter option, you must take reasonably prudent steps,
3337 when you begin distribution of Opaque copies in quantity, to ensure
3338 that this Transparent copy will remain thus accessible at the stated
3339 location until at least one year after the last time you distribute an
3340 Opaque copy (directly or through your agents or retailers) of that
3341 edition to the public.
3342 </p>
3343 <p>It is requested, but not required, that you contact the authors of
3344 the
3345 Document well before redistributing any large number of copies, to give
3346 them a chance to provide you with an updated version of the Document.
3347 </p>
3348 <p><b>4. MODIFICATIONS</b>
3349 </p>
3350 <p>You may copy and distribute a Modified Version of the Document under
3351 the conditions of sections 2 and 3 above, provided that you release
3352 the Modified Version under precisely this License, with the Modified
3353 Version filling the role of the Document, thus licensing distribution
3354 and modification of the Modified Version to whoever possesses a copy
3355 of it. In addition, you must do these things in the Modified Version:
3356 </p>
3357 <ul>
3358   <li><b>A.</b> Use in the Title Page (and on the covers, if
3359 any) a title distinct from that of the Document, and from those of
3360 previous versions (which should, if there were any, be listed in the
3361 History section of the Document). You may use the same title as a
3362 previous version if the original publisher of that version gives
3363 permission.
3364   </li>
3365   <li><b>B.</b> List on the Title Page, as authors, one or
3366 more persons or entities responsible for authorship of the
3367 modifications in the Modified Version, together with at least five of
3368 the principal authors of the Document (all of its principal authors, if
3369 it has fewer than five), unless they release you from this requirement.
3370   </li>
3371   <li><b>C.</b> State on the Title page the name of the
3372 publisher of the Modified Version, as the publisher.
3373   </li>
3374   <li><b>D.</b> Preserve all the copyright notices of the
3375 Document.
3376   </li>
3377   <li><b>E.</b> Add an appropriate copyright notice for your
3378 modifications adjacent to the other copyright notices.
3379   </li>
3380   <li><b>F.</b> Include, immediately after the copyright
3381 notices, a license notice giving the public permission to use the
3382 Modified Version under the terms of this License, in the form shown in
3383 the Addendum below.
3384   </li>
3385   <li><b>G.</b> Preserve in that license notice the full
3386 lists of Invariant Sections and required Cover Texts given in the
3387 Document's license notice.
3388   </li>
3389   <li><b>H.</b> Include an unaltered copy of this License.
3390   </li>
3391   <li><b>I.</b> Preserve the section Entitled "History",
3392 Preserve its Title, and add to it an item stating at least the title,
3393 year, new authors, and publisher of the Modified Version as given on
3394 the Title Page. If there is no section Entitled "History" in the
3395 Document, create one stating the title, year, authors, and publisher of
3396 the Document as given on its Title Page, then add an item describing
3397 the Modified Version as stated in the previous sentence.
3398   </li>
3399   <li><b>J.</b> Preserve the network location, if any, given
3400 in the Document for public access to a Transparent copy of the
3401 Document, and likewise the network locations given in the Document for
3402 previous versions it was based on. These may be placed in the "History"
3403 section. You may omit a network location for a work that was published
3404 at least four years before the Document itself, or if the original
3405 publisher of the version it refers to gives permission.
3406   </li>
3407   <li><b>K.</b> For any section Entitled "Acknowledgements"
3408 or "Dedications", Preserve the Title of the section, and preserve in
3409 the section all the substance and tone of each of the contributor
3410 acknowledgements and/or dedications given therein.
3411   </li>
3412   <li><b>L.</b> Preserve all the Invariant Sections of the
3413 Document, unaltered in their text and in their titles. Section numbers
3414 or the equivalent are not considered part of the section titles.
3415   </li>
3416   <li><b>M.</b> Delete any section Entitled "Endorsements".
3417 Such a section may not be included in the Modified Version.
3418   </li>
3419   <li><b>N.</b> Do not retitle any existing section to be
3420 Entitled "Endorsements" or to conflict in title with any Invariant
3421 Section.
3422   </li>
3423   <li><b>O.</b> Preserve any Warranty Disclaimers.
3424   </li>
3425 </ul>
3426 <p>
3427 If the Modified Version includes new front-matter sections or
3428 appendices that qualify as Secondary Sections and contain no material
3429 copied from the Document, you may at your option designate some or all
3430 of these sections as invariant. To do this, add their titles to the
3431 list of Invariant Sections in the Modified Version's license notice.
3432 These titles must be distinct from any other section titles.
3433 </p>
3434 <p>You may add a section Entitled "Endorsements", provided it contains
3435 nothing but endorsements of your Modified Version by various
3436 parties--for example, statements of peer review or that the text has
3437 been approved by an organization as the authoritative definition of a
3438 standard.
3439 </p>
3440 <p>You may add a passage of up to five words as a Front-Cover Text, and
3441 a
3442 passage of up to 25 words as a Back-Cover Text, to the end of the list
3443 of Cover Texts in the Modified Version. Only one passage of
3444 Front-Cover Text and one of Back-Cover Text may be added by (or
3445 through arrangements made by) any one entity. If the Document already
3446 includes a cover text for the same cover, previously added by you or
3447 by arrangement made by the same entity you are acting on behalf of,
3448 you may not add another; but you may replace the old one, on explicit
3449 permission from the previous publisher that added the old one.
3450 </p>
3451 <p>The author(s) and publisher(s) of the Document do not by this
3452 License
3453 give permission to use their names for publicity for or to assert or
3454 imply endorsement of any Modified Version.
3455 </p>
3456 <p><b>5. COMBINING DOCUMENTS</b>
3457 </p>
3458 <p>You may combine the Document with other documents released under
3459 this
3460 License, under the terms defined in section 4 above for modified
3461 versions, provided that you include in the combination all of the
3462 Invariant Sections of all of the original documents, unmodified, and
3463 list them all as Invariant Sections of your combined work in its
3464 license notice, and that you preserve all their Warranty Disclaimers.
3465 </p>
3466 <p>The combined work need only contain one copy of this License, and
3467 multiple identical Invariant Sections may be replaced with a single
3468 copy. If there are multiple Invariant Sections with the same name but
3469 different contents, make the title of each such section unique by
3470 adding at the end of it, in parentheses, the name of the original
3471 author or publisher of that section if known, or else a unique number.
3472 Make the same adjustment to the section titles in the list of
3473 Invariant Sections in the license notice of the combined work.
3474 </p>
3475 <p>In the combination, you must combine any sections Entitled "History"
3476 in the various original documents, forming one section Entitled
3477 "History"; likewise combine any sections Entitled "Acknowledgements",
3478 and any sections Entitled "Dedications". You must delete all sections
3479 Entitled "Endorsements."
3480 </p>
3481 <p><b>6. COLLECTIONS OF DOCUMENTS</b>
3482 </p>
3483 <p>You may make a collection consisting of the Document and other
3484 documents
3485 released under this License, and replace the individual copies of this
3486 License in the various documents with a single copy that is included in
3487 the collection, provided that you follow the rules of this License for
3488 verbatim copying of each of the documents in all other respects.
3489 </p>
3490 <p>You may extract a single document from such a collection, and
3491 distribute
3492 it individually under this License, provided you insert a copy of this
3493 License into the extracted document, and follow this License in all
3494 other respects regarding verbatim copying of that document.
3495 </p>
3496 <p><b>7. AGGREGATION WITH INDEPENDENT WORKS</b>
3497 </p>
3498 <p>A compilation of the Document or its derivatives with other separate
3499 and independent documents or works, in or on a volume of a storage or
3500 distribution medium, is called an "aggregate" if the copyright
3501 resulting from the compilation is not used to limit the legal rights
3502 of the compilation's users beyond what the individual works permit.
3503 When the Document is included in an aggregate, this License does not
3504 apply to the other works in the aggregate which are not themselves
3505 derivative works of the Document.
3506 </p>
3507 <p>If the Cover Text requirement of section 3 is applicable to these
3508 copies of the Document, then if the Document is less than one half of
3509 the entire aggregate, the Document's Cover Texts may be placed on
3510 covers that bracket the Document within the aggregate, or the
3511 electronic equivalent of covers if the Document is in electronic form.
3512 Otherwise they must appear on printed covers that bracket the whole
3513 aggregate.
3514 </p>
3515 <p><b>8. TRANSLATION</b>
3516 </p>
3517 <p>Translation is considered a kind of modification, so you may
3518 distribute translations of the Document under the terms of section 4.
3519 Replacing Invariant Sections with translations requires special
3520 permission from their copyright holders, but you may include
3521 translations of some or all Invariant Sections in addition to the
3522 original versions of these Invariant Sections. You may include a
3523 translation of this License, and all the license notices in the
3524 Document, and any Warranty Disclaimers, provided that you also include
3525 the original English version of this License and the original versions
3526 of those notices and disclaimers. In case of a disagreement between
3527 the translation and the original version of this License or a notice
3528 or disclaimer, the original version will prevail.
3529 </p>
3530 <p>If a section in the Document is Entitled "Acknowledgements",
3531 "Dedications", or "History", the requirement (section 4) to Preserve
3532 its Title (section 1) will typically require changing the actual
3533 title.
3534 </p>
3535 <p><b>9. TERMINATION</b>
3536 </p>
3537 <p>You may not copy, modify, sublicense, or distribute the Document
3538 except
3539 as expressly provided for under this License. Any other attempt to
3540 copy, modify, sublicense or distribute the Document is void, and will
3541 automatically terminate your rights under this License. However,
3542 parties who have received copies, or rights, from you under this
3543 License will not have their licenses terminated so long as such
3544 parties remain in full compliance.
3545 </p>
3546 <p><b>10. FUTURE REVISIONS OF THIS LICENSE</b>
3547 </p>
3548 <p>The Free Software Foundation may publish new, revised versions
3549 of the GNU Free Documentation License from time to time. Such new
3550 versions will be similar in spirit to the present version, but may
3551 differ in detail to address new problems or concerns. See
3552 http://www.gnu.org/copyleft/.
3553 </p>
3554 <p>Each version of the License is given a distinguishing version
3555 number.
3556 If the Document specifies that a particular numbered version of this
3557 License "or any later version" applies to it, you have the option of
3558 following the terms and conditions either of that specified version or
3559 of any later version that has been published (not as a draft) by the
3560 Free Software Foundation. If the Document does not specify a version
3561 number of this License, you may choose any version ever published (not
3562 as a draft) by the Free Software Foundation.
3563 </p>
3564 <br><br>
3565
3566
3567 <center>&part;</center>
3568
3569
3570
3571
3572 </body>
3573 </html>