1 \section{\module{urllib
} ---
2 Open arbitrary resources by URL
}
4 \declaremodule{standard
}{urllib
}
5 \modulesynopsis{Open an arbitrary network resource by URL (requires sockets).
}
12 This module provides a high-level interface for fetching data across
13 the World-Wide Web. In particular, the
\function{urlopen()
} function
14 is similar to the built-in function
\function{open()
}, but accepts
15 Universal Resource Locators (URLs) instead of filenames. Some
16 restrictions apply --- it can only open URLs for reading, and no seek
17 operations are available.
19 It defines the following public functions:
21 \begin{funcdesc
}{urlopen
}{url
\optional{, data
}}
22 Open a network object denoted by a URL for reading. If the URL does
23 not have a scheme identifier, or if it has
\file{file:
} as its scheme
24 identifier, this opens a local file; otherwise it opens a socket to a
25 server somewhere on the network. If the connection cannot be made, or
26 if the server returns an error code, the
\exception{IOError
} exception
27 is raised. If all went well, a file-like object is returned. This
28 supports the following methods:
\method{read()
},
\method{readline()
},
29 \method{readlines()
},
\method{fileno()
},
\method{close()
},
30 \method{info()
} and
\method{geturl()
}.
32 Except for the
\method{info()
} and
\method{geturl()
} methods,
33 these methods have the same interface as for
34 file objects --- see section
\ref{bltin-file-objects
} in this
35 manual. (It is not a built-in file object, however, so it can't be
36 used at those few places where a true built-in file object is
39 The
\method{info()
} method returns an instance of the class
40 \class{mimetools.Message
} containing meta-information associated
41 with the URL. When the method is HTTP, these headers are those
42 returned by the server at the head of the retrieved HTML page
43 (including Content-Length and Content-Type). When the method is FTP,
44 a Content-Length header will be present if (as is now usual) the
45 server passed back a file length in response to the FTP retrieval
46 request. When the method is local-file, returned headers will include
47 a Date representing the file's last-modified time, a Content-Length
48 giving file size, and a Content-Type containing a guess at the file's
49 type. See also the description of the
50 \refmodule{mimetools
}\refstmodindex{mimetools
} module.
52 The
\method{geturl()
} method returns the real URL of the page. In
53 some cases, the HTTP server redirects a client to another URL. The
54 \function{urlopen()
} function handles this transparently, but in some
55 cases the caller needs to know which URL the client was redirected
56 to. The
\method{geturl()
} method can be used to get at this
59 If the
\var{url
} uses the
\file{http:
} scheme identifier, the optional
60 \var{data
} argument may be given to specify a
\code{POST
} request
61 (normally the request type is
\code{GET
}). The
\var{data
} argument
62 must in standard
\mimetype{application/x-www-form-urlencoded
} format;
63 see the
\function{urlencode()
} function below.
65 The
\function{urlopen()
} function works transparently with proxies
66 which do not require authentication. In a
\UNIX{} or Windows
67 environment, set the
\envvar{http_proxy
},
\envvar{ftp_proxy
} or
68 \envvar{gopher_proxy
} environment variables to a URL that identifies
69 the proxy server before starting the Python interpreter. For example
70 (the
\character{\%
} is the command prompt):
73 % http_proxy="http://www.someproxy.com:3128"
79 In a Macintosh environment,
\function{urlopen()
} will retrieve proxy
80 information from Internet
\index{Internet Config
} Config.
82 Proxies which require authentication for use are not currently
83 supported; this is considered an implementation limitation.
86 \begin{funcdesc
}{urlretrieve
}{url
\optional{, filename
\optional{,
87 reporthook
\optional{, data
}}}}
88 Copy a network object denoted by a URL to a local file, if necessary.
89 If the URL points to a local file, or a valid cached copy of the
90 object exists, the object is not copied. Return a tuple
91 \code{(
\var{filename
},
\var{headers
})
} where
\var{filename
} is the
92 local file name under which the object can be found, and
\var{headers
}
93 is either
\code{None
} (for a local object) or whatever the
94 \method{info()
} method of the object returned by
\function{urlopen()
}
95 returned (for a remote object, possibly cached). Exceptions are the
96 same as for
\function{urlopen()
}.
98 The second argument, if present, specifies the file location to copy
99 to (if absent, the location will be a tempfile with a generated name).
100 The third argument, if present, is a hook function that will be called
101 once on establishment of the network connection and once after each
102 block read thereafter. The hook will be passed three arguments; a
103 count of blocks transferred so far, a block size in bytes, and the
104 total size of the file. The third argument may be
\code{-
1} on older
105 FTP servers which do not return a file size in response to a retrieval
108 If the
\var{url
} uses the
\file{http:
} scheme identifier, the optional
109 \var{data
} argument may be given to specify a
\code{POST
} request
110 (normally the request type is
\code{GET
}). The
\var{data
} argument
111 must in standard
\mimetype{application/x-www-form-urlencoded
} format;
112 see the
\function{urlencode()
} function below.
115 \begin{funcdesc
}{urlcleanup
}{}
116 Clear the cache that may have been built up by previous calls to
117 \function{urlretrieve()
}.
120 \begin{funcdesc
}{quote
}{string
\optional{, safe
}}
121 Replace special characters in
\var{string
} using the
\samp{\%xx
} escape.
122 Letters, digits, and the characters
\character{_,.-
} are never quoted.
123 The optional
\var{safe
} parameter specifies additional characters
124 that should not be quoted --- its default value is
\code{'/'
}.
126 Example:
\code{quote('/\~
{}connolly/')
} yields
\code{'/\%
7econnolly/'
}.
129 \begin{funcdesc
}{quote_plus
}{string
\optional{, safe
}}
130 Like
\function{quote()
}, but also replaces spaces by plus signs, as
131 required for quoting HTML form values. Plus signs in the original
132 string are escaped unless they are included in
\var{safe
}.
135 \begin{funcdesc
}{unquote
}{string
}
136 Replace
\samp{\%xx
} escapes by their single-character equivalent.
138 Example:
\code{unquote('/\%
7Econnolly/')
} yields
\code{'/\~
{}connolly/'
}.
141 \begin{funcdesc
}{unquote_plus
}{string
}
142 Like
\function{unquote()
}, but also replaces plus signs by spaces, as
143 required for unquoting HTML form values.
146 \begin{funcdesc
}{urlencode
}{query
\optional{, doseq
}}
147 Convert a mapping object or a sequence of two-element tuples to a
148 ``url-encoded'' string, suitable to pass to
149 \function{urlopen()
} above as the optional
\var{data
} argument. This
150 is useful to pass a dictionary of form fields to a
\code{POST
}
151 request. The resulting string is a series of
152 \code{\var{key
}=
\var{value
}} pairs separated by
\character{\&
}
153 characters, where both
\var{key
} and
\var{value
} are quoted using
154 \function{quote_plus()
} above. If the optional parameter
\var{doseq
} is
155 present and evaluates to true, individual
\code{\var{key
}=
\var{value
}} pairs
156 are generated for each element of the sequence.
157 When a sequence of two-element tuples is used as the
\var{query
} argument,
158 the first element of each tuple is a key and the second is a value. The
159 order of parameters in the encoded string will match the order of parameter
160 tuples in the sequence.
163 The public functions
\function{urlopen()
} and
164 \function{urlretrieve()
} create an instance of the
165 \class{FancyURLopener
} class and use it to perform their requested
166 actions. To override this functionality, programmers can create a
167 subclass of
\class{URLopener
} or
\class{FancyURLopener
}, then assign
168 that an instance of that class to the
169 \code{urllib._urlopener
} variable before calling the desired function.
170 For example, applications may want to specify a different
171 \code{user-agent
} header than
\class{URLopener
} defines. This can be
172 accomplished with the following code:
175 class AppURLopener(urllib.FancyURLopener):
176 def __init__(self, *args):
177 self.version = "App/
1.7"
178 apply(urllib.FancyURLopener.__init__, (self,) + args)
180 urllib._urlopener = AppURLopener()
183 \begin{classdesc
}{URLopener
}{\optional{proxies
\optional{, **x509
}}}
184 Base class for opening and reading URLs. Unless you need to support
185 opening objects using schemes other than
\file{http:
},
\file{ftp:
},
186 \file{gopher:
} or
\file{file:
}, you probably want to use
187 \class{FancyURLopener
}.
189 By default, the
\class{URLopener
} class sends a
190 \code{user-agent
} header of
\samp{urllib/
\var{VVV
}}, where
191 \var{VVV
} is the
\module{urllib
} version number. Applications can
192 define their own
\code{user-agent
} header by subclassing
193 \class{URLopener
} or
\class{FancyURLopener
} and setting the instance
194 attribute
\member{version
} to an appropriate string value before the
195 \method{open()
} method is called.
197 Additional keyword parameters, collected in
\var{x509
}, are used for
198 authentication with the
\file{https:
} scheme. The keywords
199 \var{key_file
} and
\var{cert_file
} are supported; both are needed to
200 actually retrieve a resource at an
\file{https:
} URL.
203 \begin{classdesc
}{FancyURLopener
}{...
}
204 \class{FancyURLopener
} subclasses
\class{URLopener
} providing default
205 handling for the following HTTP response codes:
301,
302 or
401. For
206 301 and
302 response codes, the
\code{location
} header is used to
207 fetch the actual URL. For
401 response codes (authentication
208 required), basic HTTP authentication is performed. For
301 and
302 response
209 codes, recursion is bounded by the value of the
\var{maxtries
} attribute,
212 The parameters to the constructor are the same as those for
221 Currently, only the following protocols are supported: HTTP, (versions
222 0.9 and
1.0), Gopher (but not Gopher-+), FTP, and local files.
223 \indexii{HTTP
}{protocol
}
224 \indexii{Gopher
}{protocol
}
225 \indexii{FTP
}{protocol
}
228 The caching feature of
\function{urlretrieve()
} has been disabled
229 until I find the time to hack proper processing of Expiration time
233 There should be a function to query whether a particular URL is in
237 For backward compatibility, if a URL appears to point to a local file
238 but the file can't be opened, the URL is re-interpreted using the FTP
239 protocol. This can sometimes cause confusing error messages.
242 The
\function{urlopen()
} and
\function{urlretrieve()
} functions can
243 cause arbitrarily long delays while waiting for a network connection
244 to be set up. This means that it is difficult to build an interactive
245 web client using these functions without using threads.
248 The data returned by
\function{urlopen()
} or
\function{urlretrieve()
}
249 is the raw data returned by the server. This may be binary data
250 (e.g. an image), plain text or (for example) HTML
\index{HTML
}. The
251 HTTP
\indexii{HTTP
}{protocol
} protocol provides type information in the
252 reply header, which can be inspected by looking at the
253 \code{content-type
} header. For the Gopher
\indexii{Gopher
}{protocol
}
254 protocol, type information is encoded in the URL; there is currently
255 no easy way to extract it. If the returned data is HTML, you can use
256 the module
\refmodule{htmllib
}\refstmodindex{htmllib
} to parse it.
259 This module does not support the use of proxies which require
260 authentication. This may be implemented in the future.
263 Although the
\module{urllib
} module contains (undocumented) routines
264 to parse and unparse URL strings, the recommended interface for URL
265 manipulation is in module
\refmodule{urlparse
}\refstmodindex{urlparse
}.
270 \subsection{URLopener Objects
\label{urlopener-objs
}}
271 \sectionauthor{Skip Montanaro
}{skip@mojam.com
}
273 \class{URLopener
} and
\class{FancyURLopener
} objects have the
274 following attributes.
276 \begin{methoddesc
}[URLopener
]{open
}{fullurl
\optional{, data
}}
277 Open
\var{fullurl
} using the appropriate protocol. This method sets
278 up cache and proxy information, then calls the appropriate open method with
279 its input arguments. If the scheme is not recognized,
280 \method{open_unknown()
} is called. The
\var{data
} argument
281 has the same meaning as the
\var{data
} argument of
\function{urlopen()
}.
284 \begin{methoddesc
}[URLopener
]{open_unknown
}{fullurl
\optional{, data
}}
285 Overridable interface to open unknown URL types.
288 \begin{methoddesc
}[URLopener
]{retrieve
}{url
\optional{,
290 reporthook
\optional{, data
}}}}
291 Retrieves the contents of
\var{url
} and places it in
\var{filename
}. The
292 return value is a tuple consisting of a local filename and either a
293 \class{mimetools.Message
} object containing the response headers (for remote
294 URLs) or None (for local URLs). The caller must then open and read the
295 contents of
\var{filename
}. If
\var{filename
} is not given and the URL
296 refers to a local file, the input filename is returned. If the URL is
297 non-local and
\var{filename
} is not given, the filename is the output of
298 \function{tempfile.mktemp()
} with a suffix that matches the suffix of the last
299 path component of the input URL. If
\var{reporthook
} is given, it must be
300 a function accepting three numeric parameters. It will be called after each
301 chunk of data is read from the network.
\var{reporthook
} is ignored for
304 If the
\var{url
} uses the
\file{http:
} scheme identifier, the optional
305 \var{data
} argument may be given to specify a
\code{POST
} request
306 (normally the request type is
\code{GET
}). The
\var{data
} argument
307 must in standard
\mimetype{application/x-www-form-urlencoded
} format;
308 see the
\function{urlencode()
} function below.
311 \begin{memberdesc
}[URLopener
]{version
}
312 Variable that specifies the user agent of the opener object. To get
313 \refmodule{urllib
} to tell servers that it is a particular user agent,
314 set this in a subclass as a class variable or in the constructor
315 before calling the base constructor.
319 \subsection{Examples
}
320 \nodename{Urllib Examples
}
322 Here is an example session that uses the
\samp{GET
} method to retrieve
323 a URL containing parameters:
327 >>> params = urllib.urlencode(
{'spam':
1, 'eggs':
2, 'bacon':
0})
328 >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?
%s" % params)
332 The following example uses the
\samp{POST
} method instead:
336 >>> params = urllib.urlencode(
{'spam':
1, 'eggs':
2, 'bacon':
0})
337 >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
