Doc/lib/liburllib.tex

   1 \section{\module{urllib} ---
   2          Open arbitrary resources by URL}
   3
   4 \declaremodule{standard}{urllib}
   5 \modulesynopsis{Open an arbitrary network resource by URL (requires sockets).}
   6
   7 \index{WWW}
   8 \index{World Wide Web}
   9 \index{URL}
  10
  11
  12 This module provides a high-level interface for fetching data across
  13 the World Wide Web.  In particular, the \function{urlopen()} function
  14 is similar to the built-in function \function{open()}, but accepts
  15 Universal Resource Locators (URLs) instead of filenames.  Some
  16 restrictions apply --- it can only open URLs for reading, and no seek
  17 operations are available.
  18
  19 It defines the following public functions:
  20
  21 \begin{funcdesc}{urlopen}{url\optional{, data\optional{, proxies}}}
  22 Open a network object denoted by a URL for reading.  If the URL does
  23 not have a scheme identifier, or if it has \file{file:} as its scheme
  24 identifier, this opens a local file; otherwise it opens a socket to a
  25 server somewhere on the network.  If the connection cannot be made, or
  26 if the server returns an error code, the \exception{IOError} exception
  27 is raised.  If all went well, a file-like object is returned.  This
  28 supports the following methods: \method{read()}, \method{readline()},
  29 \method{readlines()}, \method{fileno()}, \method{close()},
  30 \method{info()} and \method{geturl()}.
  31
  32 Except for the \method{info()} and \method{geturl()} methods,
  33 these methods have the same interface as for
  34 file objects --- see section \ref{bltin-file-objects} in this
  35 manual.  (It is not a built-in file object, however, so it can't be
  36 used at those few places where a true built-in file object is
  37 required.)
  38
  39 The \method{info()} method returns an instance of the class
  40 \class{mimetools.Message} containing meta-information associated
  41 with the URL.  When the method is HTTP, these headers are those
  42 returned by the server at the head of the retrieved HTML page
  43 (including Content-Length and Content-Type).  When the method is FTP,
  44 a Content-Length header will be present if (as is now usual) the
  45 server passed back a file length in response to the FTP retrieval
  46 request. A Content-Type header will be present if the MIME type can
  47 be guessed.  When the method is local-file, returned headers will include
  48 a Date representing the file's last-modified time, a Content-Length
  49 giving file size, and a Content-Type containing a guess at the file's
  50 type. See also the description of the
  51 \refmodule{mimetools}\refstmodindex{mimetools} module.
  52
  53 The \method{geturl()} method returns the real URL of the page.  In
  54 some cases, the HTTP server redirects a client to another URL.  The
  55 \function{urlopen()} function handles this transparently, but in some
  56 cases the caller needs to know which URL the client was redirected
  57 to.  The \method{geturl()} method can be used to get at this
  58 redirected URL.
  59
  60 If the \var{url} uses the \file{http:} scheme identifier, the optional
  61 \var{data} argument may be given to specify a \code{POST} request
  62 (normally the request type is \code{GET}).  The \var{data} argument
  63 must in standard \mimetype{application/x-www-form-urlencoded} format;
  64 see the \function{urlencode()} function below.
  65
  66 The \function{urlopen()} function works transparently with proxies
  67 which do not require authentication.  In a \UNIX{} or Windows
  68 environment, set the \envvar{http_proxy}, \envvar{ftp_proxy} or
  69 \envvar{gopher_proxy} environment variables to a URL that identifies
  70 the proxy server before starting the Python interpreter.  For example
  71 (the \character{\%} is the command prompt):
  72
  73 \begin{verbatim}
  74 % http_proxy="http://www.someproxy.com:3128"
  75 % export http_proxy
  76 % python
  77 ...
  78 \end{verbatim}
  79
  80 In a Windows environment, if no proxy envvironment variables are set,
  81 proxy settings are obtained from the registry's Internet Settings
  82 section.
  83
  84 In a Macintosh environment, \function{urlopen()} will retrieve proxy
  85 information from Internet\index{Internet Config} Config.
  86
  87 Alternatively, the optional \var{proxies} argument may be used to
  88 explicitly specify proxies.  It must be a dictionary mapping scheme
  89 names to proxy URLs, where an empty dictionary causes no proxies to be
  90 used, and \code{None} (the default value) causes environmental proxy
  91 settings to be used as discussed above.  For example:
  92
  93 \begin{verbatim}
  94 # Use http://www.someproxy.com:3128 for http proxying
  95 proxies = proxies={'http': 'http://www.someproxy.com:3128'}
  96 filehandle = urllib.urlopen(some_url, proxies=proxies)
  97 # Don't use any proxies
  98 filehandle = urllib.urlopen(some_url, proxies={})
  99 # Use proxies from environment - both versions are equivalent
 100 filehandle = urllib.urlopen(some_url, proxies=None)
 101 filehandle = urllib.urlopen(some_url)
 102 \end{verbatim}
 103
 104 The \function{urlopen()} function does not support explicit proxy
 105 specification.  If you need to override environmental proxy settings,
 106 use \class{URLopener}, or a subclass such as \class{FancyURLopener}.
 107
 108 Proxies which require authentication for use are not currently
 109 supported; this is considered an implementation limitation.
 110
 111 \versionchanged[Added the \var{proxies} support]{2.3}
 112 \end{funcdesc}
 113
 114 \begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{,
 115                               reporthook\optional{, data}}}}
 116 Copy a network object denoted by a URL to a local file, if necessary.
 117 If the URL points to a local file, or a valid cached copy of the
 118 object exists, the object is not copied.  Return a tuple
 119 \code{(\var{filename}, \var{headers})} where \var{filename} is the
 120 local file name under which the object can be found, and \var{headers}
 121 is either \code{None} (for a local object) or whatever the
 122 \method{info()} method of the object returned by \function{urlopen()}
 123 returned (for a remote object, possibly cached).  Exceptions are the
 124 same as for \function{urlopen()}.
 125
 126 The second argument, if present, specifies the file location to copy
 127 to (if absent, the location will be a tempfile with a generated name).
 128 The third argument, if present, is a hook function that will be called
 129 once on establishment of the network connection and once after each
 130 block read thereafter.  The hook will be passed three arguments; a
 131 count of blocks transferred so far, a block size in bytes, and the
 132 total size of the file.  The third argument may be \code{-1} on older
 133 FTP servers which do not return a file size in response to a retrieval
 134 request.
 135
 136 If the \var{url} uses the \file{http:} scheme identifier, the optional
 137 \var{data} argument may be given to specify a \code{POST} request
 138 (normally the request type is \code{GET}).  The \var{data} argument
 139 must in standard \mimetype{application/x-www-form-urlencoded} format;
 140 see the \function{urlencode()} function below.
 141 \end{funcdesc}
 142
 143 \begin{funcdesc}{urlcleanup}{}
 144 Clear the cache that may have been built up by previous calls to
 145 \function{urlretrieve()}.
 146 \end{funcdesc}
 147
 148 \begin{funcdesc}{quote}{string\optional{, safe}}
 149 Replace special characters in \var{string} using the \samp{\%xx} escape.
 150 Letters, digits, and the characters \character{_,.-} are never quoted.
 151 The optional \var{safe} parameter specifies additional characters
 152 that should not be quoted --- its default value is \code{'/'}.
 153
 154 Example: \code{quote('/\~{}connolly/')} yields \code{'/\%7econnolly/'}.
 155 \end{funcdesc}
 156
 157 \begin{funcdesc}{quote_plus}{string\optional{, safe}}
 158 Like \function{quote()}, but also replaces spaces by plus signs, as
 159 required for quoting HTML form values.  Plus signs in the original
 160 string are escaped unless they are included in \var{safe}.
 161 \end{funcdesc}
 162
 163 \begin{funcdesc}{unquote}{string}
 164 Replace \samp{\%xx} escapes by their single-character equivalent.
 165
 166 Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~{}connolly/'}.
 167 \end{funcdesc}
 168
 169 \begin{funcdesc}{unquote_plus}{string}
 170 Like \function{unquote()}, but also replaces plus signs by spaces, as
 171 required for unquoting HTML form values.
 172 \end{funcdesc}
 173
 174 \begin{funcdesc}{urlencode}{query\optional{, doseq}}
 175 Convert a mapping object or a sequence of two-element tuples  to a
 176 ``url-encoded'' string, suitable to pass to
 177 \function{urlopen()} above as the optional \var{data} argument.  This
 178 is useful to pass a dictionary of form fields to a \code{POST}
 179 request.  The resulting string is a series of
 180 \code{\var{key}=\var{value}} pairs separated by \character{\&}
 181 characters, where both \var{key} and \var{value} are quoted using
 182 \function{quote_plus()} above.  If the optional parameter \var{doseq} is
 183 present and evaluates to true, individual \code{\var{key}=\var{value}} pairs
 184 are generated for each element of the sequence.
 185 When a sequence of two-element tuples is used as the \var{query} argument,
 186 the first element of each tuple is a key and the second is a value.  The
 187 order of parameters in the encoded string will match the order of parameter
 188 tuples in the sequence.
 189 \end{funcdesc}
 190
 191 The public functions \function{urlopen()} and
 192 \function{urlretrieve()} create an instance of the
 193 \class{FancyURLopener} class and use it to perform their requested
 194 actions.  To override this functionality, programmers can create a
 195 subclass of \class{URLopener} or \class{FancyURLopener}, then assign
 196 that an instance of that class to the
 197 \code{urllib._urlopener} variable before calling the desired function.
 198 For example, applications may want to specify a different
 199 \mailheader{User-Agent} header than \class{URLopener} defines.  This
 200 can be accomplished with the following code:
 201
 202 \begin{verbatim}
 203 class AppURLopener(urllib.FancyURLopener):
 204     def __init__(self, *args):
 205         self.version = "App/1.7"
 206         urllib.FancyURLopener.__init__(self, *args)
 207
 208 urllib._urlopener = AppURLopener()
 209 \end{verbatim}
 210
 211 \begin{classdesc}{URLopener}{\optional{proxies\optional{, **x509}}}
 212 Base class for opening and reading URLs.  Unless you need to support
 213 opening objects using schemes other than \file{http:}, \file{ftp:},
 214 \file{gopher:} or \file{file:}, you probably want to use
 215 \class{FancyURLopener}.
 216
 217 By default, the \class{URLopener} class sends a
 218 \mailheader{User-Agent} header of \samp{urllib/\var{VVV}}, where
 219 \var{VVV} is the \module{urllib} version number.  Applications can
 220 define their own \mailheader{User-Agent} header by subclassing
 221 \class{URLopener} or \class{FancyURLopener} and setting the instance
 222 attribute \member{version} to an appropriate string value before the
 223 \method{open()} method is called.
 224
 225 The optional \var{proxies} parameter should be a dictionary mapping
 226 scheme names to proxy URLs, where an empty dictionary turns proxies
 227 off completely.  Its default value is \code{None}, in which case
 228 environmental proxy settings will be used if present, as discussed in
 229 the definition of \function{urlopen()}, above.
 230
 231 Additional keyword parameters, collected in \var{x509}, are used for
 232 authentication with the \file{https:} scheme.  The keywords
 233 \var{key_file} and \var{cert_file} are supported; both are needed to
 234 actually retrieve a resource at an \file{https:} URL.
 235 \end{classdesc}
 236
 237 \begin{classdesc}{FancyURLopener}{...}
 238 \class{FancyURLopener} subclasses \class{URLopener} providing default
 239 handling for the following HTTP response codes: 301, 302 or 401.  For
 240 301 and 302 response codes, the \mailheader{Location} header is used to
 241 fetch the actual URL.  For 401 response codes (authentication
 242 required), basic HTTP authentication is performed.  For 301 and 302 response
 243 codes, recursion is bounded by the value of the \var{maxtries} attribute,
 244 which defaults 10.
 245
 246 The parameters to the constructor are the same as those for
 247 \class{URLopener}.
 248
 249 \note{When performing basic authentication, a
 250 \class{FancyURLopener} instance calls its
 251 \method{prompt_user_passwd()} method.  The default implementation asks
 252 the users for the required information on the controlling terminal.  A
 253 subclass may override this method to support more appropriate behavior
 254 if needed.}
 255 \end{classdesc}
 256
 257 Restrictions:
 258
 259 \begin{itemize}
 260
 261 \item
 262 Currently, only the following protocols are supported: HTTP, (versions
 263 0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files.
 264 \indexii{HTTP}{protocol}
 265 \indexii{Gopher}{protocol}
 266 \indexii{FTP}{protocol}
 267
 268 \item
 269 The caching feature of \function{urlretrieve()} has been disabled
 270 until I find the time to hack proper processing of Expiration time
 271 headers.
 272
 273 \item
 274 There should be a function to query whether a particular URL is in
 275 the cache.
 276
 277 \item
 278 For backward compatibility, if a URL appears to point to a local file
 279 but the file can't be opened, the URL is re-interpreted using the FTP
 280 protocol.  This can sometimes cause confusing error messages.
 281
 282 \item
 283 The \function{urlopen()} and \function{urlretrieve()} functions can
 284 cause arbitrarily long delays while waiting for a network connection
 285 to be set up.  This means that it is difficult to build an interactive
 286 Web client using these functions without using threads.
 287
 288 \item
 289 The data returned by \function{urlopen()} or \function{urlretrieve()}
 290 is the raw data returned by the server.  This may be binary data
 291 (e.g. an image), plain text or (for example) HTML\index{HTML}.  The
 292 HTTP\indexii{HTTP}{protocol} protocol provides type information in the
 293 reply header, which can be inspected by looking at the
 294 \mailheader{Content-Type} header.  For the
 295 Gopher\indexii{Gopher}{protocol} protocol, type information is encoded
 296 in the URL; there is currently no easy way to extract it.  If the
 297 returned data is HTML, you can use the module
 298 \refmodule{htmllib}\refstmodindex{htmllib} to parse it.
 299
 300 \item
 301 This module does not support the use of proxies which require
 302 authentication.  This may be implemented in the future.
 303
 304 \item
 305 Although the \module{urllib} module contains (undocumented) routines
 306 to parse and unparse URL strings, the recommended interface for URL
 307 manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}.
 308
 309 \end{itemize}
 310
 311
 312 \subsection{URLopener Objects \label{urlopener-objs}}
 313 \sectionauthor{Skip Montanaro}{skip@mojam.com}
 314
 315 \class{URLopener} and \class{FancyURLopener} objects have the
 316 following attributes.
 317
 318 \begin{methoddesc}[URLopener]{open}{fullurl\optional{, data}}
 319 Open \var{fullurl} using the appropriate protocol.  This method sets
 320 up cache and proxy information, then calls the appropriate open method with
 321 its input arguments.  If the scheme is not recognized,
 322 \method{open_unknown()} is called.  The \var{data} argument
 323 has the same meaning as the \var{data} argument of \function{urlopen()}.
 324 \end{methoddesc}
 325
 326 \begin{methoddesc}[URLopener]{open_unknown}{fullurl\optional{, data}}
 327 Overridable interface to open unknown URL types.
 328 \end{methoddesc}
 329
 330 \begin{methoddesc}[URLopener]{retrieve}{url\optional{,
 331                                         filename\optional{,
 332                                         reporthook\optional{, data}}}}
 333 Retrieves the contents of \var{url} and places it in \var{filename}.  The
 334 return value is a tuple consisting of a local filename and either a
 335 \class{mimetools.Message} object containing the response headers (for remote
 336 URLs) or \code{None} (for local URLs).  The caller must then open and read the
 337 contents of \var{filename}.  If \var{filename} is not given and the URL
 338 refers to a local file, the input filename is returned.  If the URL is
 339 non-local and \var{filename} is not given, the filename is the output of
 340 \function{tempfile.mktemp()} with a suffix that matches the suffix of the last
 341 path component of the input URL.  If \var{reporthook} is given, it must be
 342 a function accepting three numeric parameters.  It will be called after each
 343 chunk of data is read from the network.  \var{reporthook} is ignored for
 344 local URLs.
 345
 346 If the \var{url} uses the \file{http:} scheme identifier, the optional
 347 \var{data} argument may be given to specify a \code{POST} request
 348 (normally the request type is \code{GET}).  The \var{data} argument
 349 must in standard \mimetype{application/x-www-form-urlencoded} format;
 350 see the \function{urlencode()} function below.
 351 \end{methoddesc}
 352
 353 \begin{memberdesc}[URLopener]{version}
 354 Variable that specifies the user agent of the opener object.  To get
 355 \refmodule{urllib} to tell servers that it is a particular user agent,
 356 set this in a subclass as a class variable or in the constructor
 357 before calling the base constructor.
 358 \end{memberdesc}
 359
 360 The \class{FancyURLopener} class offers one additional method that
 361 should be overloaded to provide the appropriate behavior:
 362
 363 \begin{methoddesc}[FancyURLopener]{prompt_user_passwd}{host, realm}
 364 Return information needed to authenticate the user at the given host
 365 in the specified security realm.  The return value should be a tuple,
 366 \code{(\var{user}, \var{password})}, which can be used for basic
 367 authentication.
 368
 369 The implementation prompts for this information on the terminal; an
 370 application should override this method to use an appropriate
 371 interaction model in the local environment.
 372 \end{methoddesc}
 373
 374
 375 \subsection{Examples}
 376 \nodename{Urllib Examples}
 377
 378 Here is an example session that uses the \samp{GET} method to retrieve
 379 a URL containing parameters:
 380
 381 \begin{verbatim}
 382 >>> import urllib
 383 >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
 384 >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
 385 >>> print f.read()
 386 \end{verbatim}
 387
 388 The following example uses the \samp{POST} method instead:
 389
 390 \begin{verbatim}
 391 >>> import urllib
 392 >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
 393 >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
 394 >>> print f.read()
 395 \end{verbatim}
 396
 397 The following example uses an explicitly specified HTTP proxy,
 398 overriding environment settings:
 399
 400 \begin{verbatim}
 401 >>> import urllib
 402 >>> proxies = {'http': 'http://proxy.example.com:8080/'}
 403 >>> opener = urllib.FancyURLopener(proxies)
 404 >>> f = opener.open("http://www.python.org")
 405 >>> f.read()
 406 \end{verbatim}
 407
 408 The following example uses no proxies at all, overriding environment
 409 settings:
 410
 411 \begin{verbatim}
 412 >>> import urllib
 413 >>> opener = urllib.FancyURLopener({})
 414 >>> f = opener.open("http://www.python.org/")
 415 >>> f.read()
 416 \end{verbatim}