Doc/lib/liburllib.tex

   1 \section{\module{urllib} ---
   2          Open arbitrary resources by URL}
   3
   4 \declaremodule{standard}{urllib}
   5 \modulesynopsis{Open an arbitrary network resource by URL (requires sockets).}
   6
   7 \index{WWW}
   8 \index{World-Wide Web}
   9 \index{URL}
  10
  11
  12 This module provides a high-level interface for fetching data across
  13 the World-Wide Web.  In particular, the \function{urlopen()} function
  14 is similar to the built-in function \function{open()}, but accepts
  15 Universal Resource Locators (URLs) instead of filenames.  Some
  16 restrictions apply --- it can only open URLs for reading, and no seek
  17 operations are available.
  18
  19 It defines the following public functions:
  20
  21 \begin{funcdesc}{urlopen}{url\optional{, data}}
  22 Open a network object denoted by a URL for reading.  If the URL does
  23 not have a scheme identifier, or if it has \file{file:} as its scheme
  24 identifier, this opens a local file; otherwise it opens a socket to a
  25 server somewhere on the network.  If the connection cannot be made, or
  26 if the server returns an error code, the \exception{IOError} exception
  27 is raised.  If all went well, a file-like object is returned.  This
  28 supports the following methods: \method{read()}, \method{readline()},
  29 \method{readlines()}, \method{fileno()}, \method{close()},
  30 \method{info()} and \method{geturl()}.
  31
  32 Except for the \method{info()} and \method{geturl()} methods,
  33 these methods have the same interface as for
  34 file objects --- see section \ref{bltin-file-objects} in this
  35 manual.  (It is not a built-in file object, however, so it can't be
  36 used at those few places where a true built-in file object is
  37 required.)
  38
  39 The \method{info()} method returns an instance of the class
  40 \class{mimetools.Message} containing meta-information associated
  41 with the URL.  When the method is HTTP, these headers are those
  42 returned by the server at the head of the retrieved HTML page
  43 (including Content-Length and Content-Type).  When the method is FTP,
  44 a Content-Length header will be present if (as is now usual) the
  45 server passed back a file length in response to the FTP retrieval
  46 request.  When the method is local-file, returned headers will include
  47 a Date representing the file's last-modified time, a Content-Length
  48 giving file size, and a Content-Type containing a guess at the file's
  49 type. See also the description of the
  50 \refmodule{mimetools}\refstmodindex{mimetools} module.
  51
  52 The \method{geturl()} method returns the real URL of the page.  In
  53 some cases, the HTTP server redirects a client to another URL.  The
  54 \function{urlopen()} function handles this transparently, but in some
  55 cases the caller needs to know which URL the client was redirected
  56 to.  The \method{geturl()} method can be used to get at this
  57 redirected URL.
  58
  59 If the \var{url} uses the \file{http:} scheme identifier, the optional
  60 \var{data} argument may be given to specify a \code{POST} request
  61 (normally the request type is \code{GET}).  The \var{data} argument
  62 must in standard \file{application/x-www-form-urlencoded} format;
  63 see the \function{urlencode()} function below.
  64
  65 The \function{urlopen()} function works transparently with proxies
  66 which do not require authentication.  In a \UNIX{} or Windows
  67 environment, set the \envvar{http_proxy}, \envvar{ftp_proxy} or
  68 \envvar{gopher_proxy} environment variables to a URL that identifies
  69 the proxy server before starting the Python interpreter.  For example
  70 (the \character{\%} is the command prompt):
  71
  72 \begin{verbatim}
  73 % http_proxy="http://www.someproxy.com:3128"
  74 % export http_proxy
  75 % python
  76 ...
  77 \end{verbatim}
  78
  79 In a Macintosh environment, \function{urlopen()} will retrieve proxy
  80 information from Internet\index{Internet Config} Config.
  81
  82 Proxies which require authentication for use are not currently
  83 supported; this is considered an implementation limitation.
  84 \end{funcdesc}
  85
  86 \begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{, hook}}}
  87 Copy a network object denoted by a URL to a local file, if necessary.
  88 If the URL points to a local file, or a valid cached copy of the
  89 object exists, the object is not copied.  Return a tuple
  90 \code{(\var{filename}, \var{headers})} where \var{filename} is the
  91 local file name under which the object can be found, and \var{headers}
  92 is either \code{None} (for a local object) or whatever the
  93 \method{info()} method of the object returned by \function{urlopen()}
  94 returned (for a remote object, possibly cached).  Exceptions are the
  95 same as for \function{urlopen()}.
  96
  97 The second argument, if present, specifies the file location to copy
  98 to (if absent, the location will be a tempfile with a generated name).
  99 The third argument, if present, is a hook function that will be called
 100 once on establishment of the network connection and once after each
 101 block read thereafter.  The hook will be passed three arguments; a
 102 count of blocks transferred so far, a block size in bytes, and the
 103 total size of the file.  The third argument may be \code{-1} on older
 104 FTP servers which do not return a file size in response to a retrieval
 105 request.
 106
 107 If the \var{url} uses the \file{http:} scheme identifier, the optional
 108 \var{data} argument may be given to specify a \code{POST} request
 109 (normally the request type is \code{GET}).  The \var{data} argument
 110 must in standard \file{application/x-www-form-urlencoded} format;
 111 see the \function{urlencode()} function below.
 112 \end{funcdesc}
 113
 114 \begin{funcdesc}{urlcleanup}{}
 115 Clear the cache that may have been built up by previous calls to
 116 \function{urlretrieve()}.
 117 \end{funcdesc}
 118
 119 \begin{funcdesc}{quote}{string\optional{, safe}}
 120 Replace special characters in \var{string} using the \samp{\%xx} escape.
 121 Letters, digits, and the characters \character{_,.-} are never quoted.
 122 The optional \var{safe} parameter specifies additional characters
 123 that should not be quoted --- its default value is \code{'/'}.
 124
 125 Example: \code{quote('/\~{}connolly/')} yields \code{'/\%7econnolly/'}.
 126 \end{funcdesc}
 127
 128 \begin{funcdesc}{quote_plus}{string\optional{, safe}}
 129 Like \function{quote()}, but also replaces spaces by plus signs, as
 130 required for quoting HTML form values.  Plus signs in the original
 131 string are escaped unless they are included in \var{safe}.
 132 \end{funcdesc}
 133
 134 \begin{funcdesc}{unquote}{string}
 135 Replace \samp{\%xx} escapes by their single-character equivalent.
 136
 137 Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~{}connolly/'}.
 138 \end{funcdesc}
 139
 140 \begin{funcdesc}{unquote_plus}{string}
 141 Like \function{unquote()}, but also replaces plus signs by spaces, as
 142 required for unquoting HTML form values.
 143 \end{funcdesc}
 144
 145 \begin{funcdesc}{urlencode}{dict}
 146 Convert a dictionary to a ``url-encoded'' string, suitable to pass to
 147 \function{urlopen()} above as the optional \var{data} argument.  This
 148 is useful to pass a dictionary of form fields to a \code{POST}
 149 request.  The resulting string is a series of
 150 \code{\var{key}=\var{value}} pairs separated by \character{\&}
 151 characters, where both \var{key} and \var{value} are quoted using
 152 \function{quote_plus()} above.
 153 \end{funcdesc}
 154
 155 The public functions \function{urlopen()} and
 156 \function{urlretrieve()} create an instance of the
 157 \class{FancyURLopener} class and use it to perform their requested
 158 actions.  To override this functionality, programmers can create a
 159 subclass of \class{URLopener} or \class{FancyURLopener}, then assign
 160 that an instance of that class to the
 161 \code{urllib._urlopener} variable before calling the desired function.
 162 For example, applications may want to specify a different
 163 \code{user-agent} header than \class{URLopener} defines.  This can be
 164 accomplished with the following code:
 165
 166 \begin{verbatim}
 167 class AppURLopener(urllib.FancyURLopener):
 168     def __init__(self, *args):
 169         self.version = "App/1.7"
 170         apply(urllib.FancyURLopener.__init__, (self,) + args)
 171
 172 urllib._urlopener = AppURLopener()
 173 \end{verbatim}
 174
 175 \begin{classdesc}{URLopener}{\optional{proxies\optional{, **x509}}}
 176 Base class for opening and reading URLs.  Unless you need to support
 177 opening objects using schemes other than \file{http:}, \file{ftp:},
 178 \file{gopher:} or \file{file:}, you probably want to use
 179 \class{FancyURLopener}.
 180
 181 By default, the \class{URLopener} class sends a
 182 \code{user-agent} header of \samp{urllib/\var{VVV}}, where
 183 \var{VVV} is the \module{urllib} version number.  Applications can
 184 define their own \code{user-agent} header by subclassing
 185 \class{URLopener} or \class{FancyURLopener} and setting the instance
 186 attribute \member{version} to an appropriate string value before the
 187 \method{open()} method is called.
 188
 189 Additional keyword parameters, collected in \var{x509}, are used for
 190 authentication with the \file{https:} scheme.  The keywords
 191 \var{key_file} and \var{cert_file} are supported; both are needed to
 192 actually retrieve a resource at an \file{https:} URL.
 193 \end{classdesc}
 194
 195 \begin{classdesc}{FancyURLopener}{...}
 196 \class{FancyURLopener} subclasses \class{URLopener} providing default
 197 handling for the following HTTP response codes: 301, 302 or 401.  For
 198 301 and 302 response codes, the \code{location} header is used to
 199 fetch the actual URL.  For 401 response codes (authentication
 200 required), basic HTTP authentication is performed.
 201
 202 The parameters to the constructor are the same as those for
 203 \class{URLopener}.
 204 \end{classdesc}
 205
 206 Restrictions:
 207
 208 \begin{itemize}
 209
 210 \item
 211 Currently, only the following protocols are supported: HTTP, (versions
 212 0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files.
 213 \indexii{HTTP}{protocol}
 214 \indexii{Gopher}{protocol}
 215 \indexii{FTP}{protocol}
 216
 217 \item
 218 The caching feature of \function{urlretrieve()} has been disabled
 219 until I find the time to hack proper processing of Expiration time
 220 headers.
 221
 222 \item
 223 There should be a function to query whether a particular URL is in
 224 the cache.
 225
 226 \item
 227 For backward compatibility, if a URL appears to point to a local file
 228 but the file can't be opened, the URL is re-interpreted using the FTP
 229 protocol.  This can sometimes cause confusing error messages.
 230
 231 \item
 232 The \function{urlopen()} and \function{urlretrieve()} functions can
 233 cause arbitrarily long delays while waiting for a network connection
 234 to be set up.  This means that it is difficult to build an interactive
 235 web client using these functions without using threads.
 236
 237 \item
 238 The data returned by \function{urlopen()} or \function{urlretrieve()}
 239 is the raw data returned by the server.  This may be binary data
 240 (e.g. an image), plain text or (for example) HTML\index{HTML}.  The
 241 HTTP\indexii{HTTP}{protocol} protocol provides type information in the
 242 reply header, which can be inspected by looking at the
 243 \code{content-type} header.  For the Gopher\indexii{Gopher}{protocol}
 244 protocol, type information is encoded in the URL; there is currently
 245 no easy way to extract it.  If the returned data is HTML, you can use
 246 the module \refmodule{htmllib}\refstmodindex{htmllib} to parse it.
 247
 248 \item
 249 This module does not support the use of proxies which require
 250 authentication.  This may be implemented in the future.
 251
 252 \item
 253 Although the \module{urllib} module contains (undocumented) routines
 254 to parse and unparse URL strings, the recommended interface for URL
 255 manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}.
 256
 257 \end{itemize}
 258
 259
 260 \subsection{URLopener Objects \label{urlopener-objs}}
 261 \sectionauthor{Skip Montanaro}{skip@mojam.com}
 262
 263 \class{URLopener} and \class{FancyURLopener} objects have the
 264 following attributes.
 265
 266 \begin{methoddesc}[URLopener]{open}{fullurl\optional{, data}}
 267 Open \var{fullurl} using the appropriate protocol.  This method sets
 268 up cache and proxy information, then calls the appropriate open method with
 269 its input arguments.  If the scheme is not recognized,
 270 \method{open_unknown()} is called.  The \var{data} argument
 271 has the same meaning as the \var{data} argument of \function{urlopen()}.
 272 \end{methoddesc}
 273
 274 \begin{methoddesc}[URLopener]{open_unknown}{fullurl\optional{, data}}
 275 Overridable interface to open unknown URL types.
 276 \end{methoddesc}
 277
 278 \begin{methoddesc}[URLopener]{retrieve}{url\optional{,
 279                                         filename\optional{,
 280                                         reporthook\optional{, data}}}}
 281 Retrieves the contents of \var{url} and places it in \var{filename}.  The
 282 return value is a tuple consisting of a local filename and either a
 283 \class{mimetools.Message} object containing the response headers (for remote
 284 URLs) or None (for local URLs).  The caller must then open and read the
 285 contents of \var{filename}.  If \var{filename} is not given and the URL
 286 refers to a local file, the input filename is returned.  If the URL is
 287 non-local and \var{filename} is not given, the filename is the output of
 288 \function{tempfile.mktemp()} with a suffix that matches the suffix of the last
 289 path component of the input URL.  If \var{reporthook} is given, it must be
 290 a function accepting three numeric parameters.  It will be called after each
 291 chunk of data is read from the network.  \var{reporthook} is ignored for
 292 local URLs.
 293
 294 If the \var{url} uses the \file{http:} scheme identifier, the optional
 295 \var{data} argument may be given to specify a \code{POST} request
 296 (normally the request type is \code{GET}).  The \var{data} argument
 297 must in standard \file{application/x-www-form-urlencoded} format;
 298 see the \function{urlencode()} function below.
 299 \end{methoddesc}
 300
 301 \begin{memberdesc}[URLopener]{version}
 302 Variable that specifies the user agent of the opener object.  To get
 303 \refmodule{urllib} to tell servers that it is a particular user agent,
 304 set this in a subclass as a class variable or in the constructor
 305 before calling the base constructor.
 306 \end{memberdesc}
 307
 308
 309 \subsection{Examples}
 310 \nodename{Urllib Examples}
 311
 312 Here is an example session that uses the \samp{GET} method to retrieve
 313 a URL containing parameters:
 314
 315 \begin{verbatim}
 316 >>> import urllib
 317 >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
 318 >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
 319 >>> print f.read()
 320 \end{verbatim}
 321
 322 The following example uses the \samp{POST} method instead:
 323
 324 \begin{verbatim}
 325 >>> import urllib
 326 >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
 327 >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
 328 >>> print f.read()
 329 \end{verbatim}