Doc/lib/libshlex.tex

   1 \section{\module{shlex} ---
   2          Simple lexical analysis}
   3
   4 \declaremodule{standard}{shlex}
   5 \modulesynopsis{Simple lexical analysis for \UNIX\ shell-like languages.}
   6 \moduleauthor{Eric S. Raymond}{esr@snark.thyrsus.com}
   7 \sectionauthor{Eric S. Raymond}{esr@snark.thyrsus.com}
   8
   9 \versionadded{1.5.2}
  10
  11 The \class{shlex} class makes it easy to write lexical analyzers for
  12 simple syntaxes resembling that of the \UNIX{} shell.  This will often
  13 be useful for writing minilanguages, e.g.\ in run control files for
  14 Python applications.
  15
  16 \begin{classdesc}{shlex}{\optional{stream\optional{, file}}}
  17 A \class{shlex} instance or subclass instance is a lexical analyzer
  18 object.  The initialization argument, if present, specifies where to
  19 read characters from. It must be a file- or stream-like object with
  20 \method{read()} and \method{readline()} methods.  If no argument is given,
  21 input will be taken from \code{sys.stdin}.  The second optional
  22 argument is a filename string, which sets the initial value of the
  23 \member{infile} member.  If the stream argument is omitted or
  24 equal to \code{sys.stdin}, this second argument defaults to ``stdin''.
  25 \end{classdesc}
  26
  27
  28 \begin{seealso}
  29   \seemodule{ConfigParser}{Parser for configuration files similar to the
  30                            Windows \file{.ini} files.}
  31 \end{seealso}
  32
  33
  34 \subsection{shlex Objects \label{shlex-objects}}
  35
  36 A \class{shlex} instance has the following methods:
  37
  38
  39 \begin{methoddesc}{get_token}{}
  40 Return a token.  If tokens have been stacked using
  41 \method{push_token()}, pop a token off the stack.  Otherwise, read one
  42 from the input stream.  If reading encounters an immediate
  43 end-of-file, an empty string is returned.
  44 \end{methoddesc}
  45
  46 \begin{methoddesc}{push_token}{str}
  47 Push the argument onto the token stack.
  48 \end{methoddesc}
  49
  50 \begin{methoddesc}{read_token}{}
  51 Read a raw token.  Ignore the pushback stack, and do not interpret source
  52 requests.  (This is not ordinarily a useful entry point, and is
  53 documented here only for the sake of completeness.)
  54 \end{methoddesc}
  55
  56 \begin{methoddesc}{sourcehook}{filename}
  57 When \class{shlex} detects a source request (see
  58 \member{source} below) this method is given the following token as
  59 argument, and expected to return a tuple consisting of a filename and
  60 an open file-like object.
  61
  62 Normally, this method first strips any quotes off the argument.  If
  63 the result is an absolute pathname, or there was no previous source
  64 request in effect, or the previous source was a stream
  65 (e.g. \code{sys.stdin}), the result is left alone.  Otherwise, if the
  66 result is a relative pathname, the directory part of the name of the
  67 file immediately before it on the source inclusion stack is prepended
  68 (this behavior is like the way the C preprocessor handles
  69 \code{\#include "file.h"}).
  70
  71 The result of the manipulations is treated as a filename, and returned
  72 as the first component of the tuple, with
  73 \function{open()} called on it to yield the second component. (Note:
  74 this is the reverse of the order of arguments in instance initialization!)
  75
  76 This hook is exposed so that you can use it to implement directory
  77 search paths, addition of file extensions, and other namespace hacks.
  78 There is no corresponding `close' hook, but a shlex instance will call
  79 the \method{close()} method of the sourced input stream when it
  80 returns \EOF.
  81
  82 For more explicit control of source stacking, use the
  83 \method{push_source()} and \method{pop_source()} methods.
  84 \end{methoddesc}
  85
  86 \begin{methoddesc}{push_source}{stream\optional{, filename}}
  87 Push an input source stream onto the input stack.  If the filename
  88 argument is specified it will later be available for use in error
  89 messages.  This is the same method used internally by the
  90 \method{sourcehook} method.
  91 \versionadded{2.1}
  92 \end{methoddesc}
  93
  94 \begin{methoddesc}{pop_source}{}
  95 Pop the last-pushed input source from the input stack.
  96 This is the same method used internally when the lexer reaches
  97 \EOF on a stacked input stream.
  98 \versionadded{2.1}
  99 \end{methoddesc}
 100
 101 \begin{methoddesc}{error_leader}{\optional{file\optional{, line}}}
 102 This method generates an error message leader in the format of a
 103 \UNIX{} C compiler error label; the format is \code{'"\%s", line \%d: '},
 104 where the \samp{\%s} is replaced with the name of the current source
 105 file and the \samp{\%d} with the current input line number (the
 106 optional arguments can be used to override these).
 107
 108 This convenience is provided to encourage \module{shlex} users to
 109 generate error messages in the standard, parseable format understood
 110 by Emacs and other \UNIX{} tools.
 111 \end{methoddesc}
 112
 113 Instances of \class{shlex} subclasses have some public instance
 114 variables which either control lexical analysis or can be used for
 115 debugging:
 116
 117 \begin{memberdesc}{commenters}
 118 The string of characters that are recognized as comment beginners.
 119 All characters from the comment beginner to end of line are ignored.
 120 Includes just \character{\#} by default.
 121 \end{memberdesc}
 122
 123 \begin{memberdesc}{wordchars}
 124 The string of characters that will accumulate into multi-character
 125 tokens.  By default, includes all \ASCII{} alphanumerics and
 126 underscore.
 127 \end{memberdesc}
 128
 129 \begin{memberdesc}{whitespace}
 130 Characters that will be considered whitespace and skipped.  Whitespace
 131 bounds tokens.  By default, includes space, tab, linefeed and
 132 carriage-return.
 133 \end{memberdesc}
 134
 135 \begin{memberdesc}{quotes}
 136 Characters that will be considered string quotes.  The token
 137 accumulates until the same quote is encountered again (thus, different
 138 quote types protect each other as in the shell.)  By default, includes
 139 \ASCII{} single and double quotes.
 140 \end{memberdesc}
 141
 142 \begin{memberdesc}{infile}
 143 The name of the current input file, as initially set at class
 144 instantiation time or stacked by later source requests.  It may
 145 be useful to examine this when constructing error messages.
 146 \end{memberdesc}
 147
 148 \begin{memberdesc}{instream}
 149 The input stream from which this \class{shlex} instance is reading
 150 characters.
 151 \end{memberdesc}
 152
 153 \begin{memberdesc}{source}
 154 This member is \code{None} by default.  If you assign a string to it,
 155 that string will be recognized as a lexical-level inclusion request
 156 similar to the \samp{source} keyword in various shells.  That is, the
 157 immediately following token will opened as a filename and input taken
 158 from that stream until \EOF, at which point the \method{close()}
 159 method of that stream will be called and the input source will again
 160 become the original input stream. Source requests may be stacked any
 161 number of levels deep.
 162 \end{memberdesc}
 163
 164 \begin{memberdesc}{debug}
 165 If this member is numeric and \code{1} or more, a \class{shlex}
 166 instance will print verbose progress output on its behavior.  If you
 167 need to use this, you can read the module source code to learn the
 168 details.
 169 \end{memberdesc}
 170
 171 Note that any character not declared to be a word character,
 172 whitespace, or a quote will be returned as a single-character token.
 173
 174 Quote and comment characters are not recognized within words.  Thus,
 175 the bare words \samp{ain't} and \samp{ain\#t} would be returned as single
 176 tokens by the default parser.
 177
 178 \begin{memberdesc}{lineno}
 179 Source line number (count of newlines seen so far plus one).
 180 \end{memberdesc}
 181
 182 \begin{memberdesc}{token}
 183 The token buffer.  It may be useful to examine this when catching
 184 exceptions.
 185 \end{memberdesc}