Doc/lib/libshlex.tex

   1 \section{\module{shlex} ---
   2          Simple lexical analysis}
   3
   4 \declaremodule{standard}{shlex}
   5 \modulesynopsis{Simple lexical analysis for \UNIX{} shell-like languages.}
   6 \moduleauthor{Eric S. Raymond}{esr@snark.thyrsus.com}
   7 \sectionauthor{Eric S. Raymond}{esr@snark.thyrsus.com}
   8
   9 \versionadded{1.5.2}
  10
  11 The \class{shlex} class makes it easy to write lexical analyzers for
  12 simple syntaxes resembling that of the \UNIX{} shell.  This will often
  13 be useful for writing minilanguages, e.g.\ in run control files for
  14 Python applications.
  15
  16 \begin{classdesc}{shlex}{\optional{stream\optional{, file}}}
  17 A \class{shlex} instance or subclass instance is a lexical analyzer
  18 object.  The initialization argument, if present, specifies where to
  19 read characters from. It must be a file- or stream-like object with
  20 \method{read()} and \method{readline()} methods.  If no argument is given,
  21 input will be taken from \code{sys.stdin}.  The second optional
  22 argument is a filename string, which sets the initial value of the
  23 \member{infile} member.  If the stream argument is omitted or
  24 equal to \code{sys.stdin}, this second argument defaults to ``stdin''.
  25 \end{classdesc}
  26
  27
  28 \begin{seealso}
  29   \seemodule{ConfigParser}{Parser for configuration files similar to the
  30                            Windows \file{.ini} files.}
  31 \end{seealso}
  32
  33
  34 \subsection{shlex Objects \label{shlex-objects}}
  35
  36 A \class{shlex} instance has the following methods:
  37
  38
  39 \begin{methoddesc}{get_token}{}
  40 Return a token.  If tokens have been stacked using
  41 \method{push_token()}, pop a token off the stack.  Otherwise, read one
  42 from the input stream.  If reading encounters an immediate
  43 end-of-file, an empty string is returned.
  44 \end{methoddesc}
  45
  46 \begin{methoddesc}{push_token}{str}
  47 Push the argument onto the token stack.
  48 \end{methoddesc}
  49
  50 \begin{methoddesc}{read_token}{}
  51 Read a raw token.  Ignore the pushback stack, and do not interpret source
  52 requests.  (This is not ordinarily a useful entry point, and is
  53 documented here only for the sake of completeness.)
  54 \end{methoddesc}
  55
  56 \begin{methoddesc}{sourcehook}{filename}
  57 When \class{shlex} detects a source request (see
  58 \member{source} below) this method is given the following token as
  59 argument, and expected to return a tuple consisting of a filename and
  60 an open file-like object.
  61
  62 Normally, this method first strips any quotes off the argument.  If
  63 the result is an absolute pathname, or there was no previous source
  64 request in effect, or the previous source was a stream
  65 (e.g. \code{sys.stdin}), the result is left alone.  Otherwise, if the
  66 result is a relative pathname, the directory part of the name of the
  67 file immediately before it on the source inclusion stack is prepended
  68 (this behavior is like the way the C preprocessor handles
  69 \code{\#include "file.h"}).  The result of the manipulations is treated
  70 as a filename, and returned as the first component of the tuple, with
  71 \function{open()} called on it to yield the second component.
  72
  73 This hook is exposed so that you can use it to implement directory
  74 search paths, addition of file extensions, and other namespace hacks.
  75 There is no corresponding `close' hook, but a shlex instance will call
  76 the \method{close()} method of the sourced input stream when it
  77 returns \EOF.
  78 \end{methoddesc}
  79
  80 \begin{methoddesc}{error_leader}{\optional{file\optional{, line}}}
  81 This method generates an error message leader in the format of a
  82 \UNIX{} C compiler error label; the format is '"\%s", line \%d: ',
  83 where the \samp{\%s} is replaced with the name of the current source
  84 file and the \samp{\%d} with the current input line number (the
  85 optional arguments can be used to override these).
  86
  87 This convenience is provided to encourage \module{shlex} users to
  88 generate error messages in the standard, parseable format understood
  89 by Emacs and other \UNIX{} tools.
  90 \end{methoddesc}
  91
  92 Instances of \class{shlex} subclasses have some public instance
  93 variables which either control lexical analysis or can be used for
  94 debugging:
  95
  96 \begin{memberdesc}{commenters}
  97 The string of characters that are recognized as comment beginners.
  98 All characters from the comment beginner to end of line are ignored.
  99 Includes just \character{\#} by default.
 100 \end{memberdesc}
 101
 102 \begin{memberdesc}{wordchars}
 103 The string of characters that will accumulate into multi-character
 104 tokens.  By default, includes all \ASCII{} alphanumerics and
 105 underscore.
 106 \end{memberdesc}
 107
 108 \begin{memberdesc}{whitespace}
 109 Characters that will be considered whitespace and skipped.  Whitespace
 110 bounds tokens.  By default, includes space, tab, linefeed and
 111 carriage-return.
 112 \end{memberdesc}
 113
 114 \begin{memberdesc}{quotes}
 115 Characters that will be considered string quotes.  The token
 116 accumulates until the same quote is encountered again (thus, different
 117 quote types protect each other as in the shell.)  By default, includes
 118 \ASCII{} single and double quotes.
 119 \end{memberdesc}
 120
 121 \begin{memberdesc}{infile}
 122 The name of the current input file, as initially set at class
 123 instantiation time or stacked by later source requests.  It may
 124 be useful to examine this when constructing error messages.
 125 \end{memberdesc}
 126
 127 \begin{memberdesc}{instream}
 128 The input stream from which this \class{shlex} instance is reading
 129 characters.
 130 \end{memberdesc}
 131
 132 \begin{memberdesc}{source}
 133 This member is \code{None} by default.  If you assign a string to it,
 134 that string will be recognized as a lexical-level inclusion request
 135 similar to the \samp{source} keyword in various shells.  That is, the
 136 immediately following token will opened as a filename and input taken
 137 from that stream until \EOF, at which point the \method{close()}
 138 method of that stream will be called and the input source will again
 139 become the original input stream. Source requests may be stacked any
 140 number of levels deep.
 141 \end{memberdesc}
 142
 143 \begin{memberdesc}{debug}
 144 If this member is numeric and \code{1} or more, a \class{shlex}
 145 instance will print verbose progress output on its behavior.  If you
 146 need to use this, you can read the module source code to learn the
 147 details.
 148 \end{memberdesc}
 149
 150 Note that any character not declared to be a word character,
 151 whitespace, or a quote will be returned as a single-character token.
 152
 153 Quote and comment characters are not recognized within words.  Thus,
 154 the bare words \samp{ain't} and \samp{ain\#t} would be returned as single
 155 tokens by the default parser.
 156
 157 \begin{memberdesc}{lineno}
 158 Source line number (count of newlines seen so far plus one).
 159 \end{memberdesc}
 160
 161 \begin{memberdesc}{token}
 162 The token buffer.  It may be useful to examine this when catching
 163 exceptions.
 164 \end{memberdesc}