Doc/lib/libpickle.tex

   1 \section{\module{pickle} --- Python object serialization}
   2
   3 \declaremodule{standard}{pickle}
   4 \modulesynopsis{Convert Python objects to streams of bytes and back.}
   5 % Substantial improvements by Jim Kerr <jbkerr@sr.hp.com>.
   6 % Rewritten by Barry Warsaw <barry@zope.com>
   7
   8 \index{persistence}
   9 \indexii{persistent}{objects}
  10 \indexii{serializing}{objects}
  11 \indexii{marshalling}{objects}
  12 \indexii{flattening}{objects}
  13 \indexii{pickling}{objects}
  14
  15 The \module{pickle} module implements a fundamental, but powerful
  16 algorithm for serializing and de-serializing a Python object
  17 structure.  ``Pickling'' is the process whereby a Python object
  18 hierarchy is converted into a byte stream, and ``unpickling'' is the
  19 inverse operation, whereby a byte stream is converted back into an
  20 object hierarchy.  Pickling (and unpickling) is alternatively known as
  21 ``serialization'', ``marshalling,''\footnote{Don't confuse this with
  22 the \refmodule{marshal} module} or ``flattening'',
  23 however the preferred term used here is ``pickling'' and
  24 ``unpickling'' to avoid confusing.
  25
  26 This documentation describes both the \module{pickle} module and the
  27 \refmodule{cPickle} module.
  28
  29 \subsection{Relationship to other Python modules}
  30
  31 The \module{pickle} module has an optimized cousin called the
  32 \module{cPickle} module.  As its name implies, \module{cPickle} is
  33 written in C, so it can be up to 1000 times faster than
  34 \module{pickle}.  However it does not support subclassing of the
  35 \function{Pickler()} and \function{Unpickler()} classes, because in
  36 \module{cPickle} these are functions, not classes.  Most applications
  37 have no need for this functionality, and can benefit from the improved
  38 performance of \module{cPickle}.  Other than that, the interfaces of
  39 the two modules are nearly identical; the common interface is
  40 described in this manual and differences are pointed out where
  41 necessary.  In the following discussions, we use the term ``pickle''
  42 to collectively describe the \module{pickle} and
  43 \module{cPickle} modules.
  44
  45 The data streams the two modules produce are guaranteed to be
  46 interchangeable.
  47
  48 Python has a more primitive serialization module called
  49 \refmodule{marshal}, but in general
  50 \module{pickle} should always be the preferred way to serialize Python
  51 objects.  \module{marshal} exists primarily to support Python's
  52 \file{.pyc} files.
  53
  54 The \module{pickle} module differs from \refmodule{marshal} several
  55 significant ways:
  56
  57 \begin{itemize}
  58
  59 \item The \module{pickle} module keeps track of the objects it has
  60       already serialized, so that later references to the same object
  61       won't be serialized again.  \module{marshal} doesn't do this.
  62
  63       This has implications both for recursive objects and object
  64       sharing.  Recursive objects are objects that contain references
  65       to themselves.  These are not handled by marshal, and in fact,
  66       attempting to marshal recursive objects will crash your Python
  67       interpreter.  Object sharing happens when there are multiple
  68       references to the same object in different places in the object
  69       hierarchy being serialized.  \module{pickle} stores such objects
  70       only once, and ensures that all other references point to the
  71       master copy.  Shared objects remain shared, which can be very
  72       important for mutable objects.
  73
  74 \item \module{marshal} cannot be used to serialize user-defined
  75       classes and their instances.  \module{pickle} can save and
  76       restore class instances transparently, however the class
  77       definition must be importable and live in the same module as
  78       when the object was stored.
  79
  80 \item The \module{marshal} serialization format is not guaranteed to
  81       be portable across Python versions.  Because its primary job in
  82       life is to support \file{.pyc} files, the Python implementers
  83       reserve the right to change the serialization format in
  84       non-backwards compatible ways should the need arise.  The
  85       \module{pickle} serialization format is guaranteed to be
  86       backwards compatible across Python releases.
  87
  88 \item The \module{pickle} module doesn't handle code objects, which
  89       the \module{marshal} module does.  This avoids the possibility
  90       of smuggling Trojan horses into a program through the
  91       \module{pickle} module\footnote{This doesn't necessarily imply
  92       that \module{pickle} is inherently secure.  See
  93       section~\ref{pickle-sec} for a more detailed discussion on
  94       \module{pickle} module security.  Besides, it's possible that
  95       \module{pickle} will eventually support serializing code
  96       objects.}.
  97
  98 \end{itemize}
  99
 100 Note that serialization is a more primitive notion than persistence;
 101 although
 102 \module{pickle} reads and writes file objects, it does not handle the
 103 issue of naming persistent objects, nor the (even more complicated)
 104 issue of concurrent access to persistent objects.  The \module{pickle}
 105 module can transform a complex object into a byte stream and it can
 106 transform the byte stream into an object with the same internal
 107 structure.  Perhaps the most obvious thing to do with these byte
 108 streams is to write them onto a file, but it is also conceivable to
 109 send them across a network or store them in a database.  The module
 110 \refmodule{shelve} provides a simple interface
 111 to pickle and unpickle objects on DBM-style database files.
 112
 113 \subsection{Data stream format}
 114
 115 The data format used by \module{pickle} is Python-specific.  This has
 116 the advantage that there are no restrictions imposed by external
 117 standards such as XDR\index{XDR}\index{External Data Representation}
 118 (which can't represent pointer sharing); however it means that
 119 non-Python programs may not be able to reconstruct pickled Python
 120 objects.
 121
 122 By default, the \module{pickle} data format uses a printable \ASCII{}
 123 representation.  This is slightly more voluminous than a binary
 124 representation.  The big advantage of using printable \ASCII{} (and of
 125 some other characteristics of \module{pickle}'s representation) is that
 126 for debugging or recovery purposes it is possible for a human to read
 127 the pickled file with a standard text editor.
 128
 129 A binary format, which is slightly more efficient, can be chosen by
 130 specifying a true value for the \var{bin} argument to the
 131 \class{Pickler} constructor or the \function{dump()} and \function{dumps()}
 132 functions.
 133
 134 \subsection{Usage}
 135
 136 To serialize an object hierarchy, you first create a pickler, then you
 137 call the pickler's \method{dump()} method.  To de-serialize a data
 138 stream, you first create an unpickler, then you call the unpickler's
 139 \method{load()} method.  The \module{pickle} module provides the
 140 following functions to make this process more convenient:
 141
 142 \begin{funcdesc}{dump}{object, file\optional{, bin}}
 143 Write a pickled representation of \var{object} to the open file object
 144 \var{file}.  This is equivalent to
 145 \code{Pickler(\var{file}, \var{bin}).dump(\var{object})}.
 146 If the optional \var{bin} argument is true, the binary pickle format
 147 is used; otherwise the (less efficient) text pickle format is used
 148 (for backwards compatibility, this is the default).
 149
 150 \var{file} must have a \method{write()} method that accepts a single
 151 string argument.  It can thus be a file object opened for writing, a
 152 \refmodule{StringIO} object, or any other custom
 153 object that meets this interface.
 154 \end{funcdesc}
 155
 156 \begin{funcdesc}{load}{file}
 157 Read a string from the open file object \var{file} and interpret it as
 158 a pickle data stream, reconstructing and returning the original object
 159 hierarchy.  This is equivalent to \code{Unpickler(\var{file}).load()}.
 160
 161 \var{file} must have two methods, a \method{read()} method that takes
 162 an integer argument, and a \method{readline()} method that requires no
 163 arguments.  Both methods should return a string.  Thus \var{file} can
 164 be a file object opened for reading, a
 165 \module{StringIO} object, or any other custom
 166 object that meets this interface.
 167
 168 This function automatically determines whether the data stream was
 169 written in binary mode or not.
 170 \end{funcdesc}
 171
 172 \begin{funcdesc}{dumps}{object\optional{, bin}}
 173 Return the pickled representation of the object as a string, instead
 174 of writing it to a file.  If the optional \var{bin} argument is
 175 true, the binary pickle format is used; otherwise the (less efficient)
 176 text pickle format is used (this is the default).
 177 \end{funcdesc}
 178
 179 \begin{funcdesc}{loads}{string}
 180 Read a pickled object hierarchy from a string.  Characters in the
 181 string past the pickled object's representation are ignored.
 182 \end{funcdesc}
 183
 184 The \module{pickle} module also defines three exceptions:
 185
 186 \begin{excdesc}{PickleError}
 187 A common base class for the other exceptions defined below.  This
 188 inherits from \exception{Exception}.
 189 \end{excdesc}
 190
 191 \begin{excdesc}{PicklingError}
 192 This exception is raised when an unpicklable object is passed to
 193 the \method{dump()} method.
 194 \end{excdesc}
 195
 196 \begin{excdesc}{UnpicklingError}
 197 This exception is raised when there is a problem unpickling an object,
 198 such as a security violation.  Note that other exceptions may also be
 199 raised during unpickling, including (but not necessarily limited to)
 200 \exception{AttributeError} and \exception{ImportError}.
 201 \end{excdesc}
 202
 203 The \module{pickle} module also exports two callables\footnote{In the
 204 \module{pickle} module these callables are classes, which you could
 205 subclass to customize the behavior.  However, in the \module{cPickle}
 206 modules these callables are factory functions and so cannot be
 207 subclassed.  One of the common reasons to subclass is to control what
 208 objects can actually be unpickled.  See section~\ref{pickle-sec} for
 209 more details on security concerns.}, \class{Pickler} and
 210 \class{Unpickler}:
 211
 212 \begin{classdesc}{Pickler}{file\optional{, bin}}
 213 This takes a file-like object to which it will write a pickle data
 214 stream.  Optional \var{bin} if true, tells the pickler to use the more
 215 efficient binary pickle format, otherwise the \ASCII{} format is used
 216 (this is the default).
 217
 218 \var{file} must have a \method{write()} method that accepts a single
 219 string argument.  It can thus be an open file object, a
 220 \module{StringIO} object, or any other custom
 221 object that meets this interface.
 222 \end{classdesc}
 223
 224 \class{Pickler} objects define one (or two) public methods:
 225
 226 \begin{methoddesc}[Pickler]{dump}{object}
 227 Write a pickled representation of \var{object} to the open file object
 228 given in the constructor.  Either the binary or \ASCII{} format will
 229 be used, depending on the value of the \var{bin} flag passed to the
 230 constructor.
 231 \end{methoddesc}
 232
 233 \begin{methoddesc}[Pickler]{clear_memo}{}
 234 Clears the pickler's ``memo''.  The memo is the data structure that
 235 remembers which objects the pickler has already seen, so that shared
 236 or recursive objects pickled by reference and not by value.  This
 237 method is useful when re-using picklers.
 238
 239 \strong{Note:} \method{clear_memo()} is only available on the picklers
 240 created by \module{cPickle}.  In the \module{pickle} module, picklers
 241 have an instance variable called \member{memo} which is a Python
 242 dictionary.  So to clear the memo for a \module{pickle} module
 243 pickler, you could do the following:
 244
 245 \begin{verbatim}
 246 mypickler.memo.clear()
 247 \end{verbatim}
 248 \end{methoddesc}
 249
 250 It is possible to make multiple calls to the \method{dump()} method of
 251 the same \class{Pickler} instance.  These must then be matched to the
 252 same number of calls to the \method{load()} method of the
 253 corresponding \class{Unpickler} instance.  If the same object is
 254 pickled by multiple \method{dump()} calls, the \method{load()} will
 255 all yield references to the same object\footnote{\emph{Warning}: this
 256 is intended for pickling multiple objects without intervening
 257 modifications to the objects or their parts.  If you modify an object
 258 and then pickle it again using the same \class{Pickler} instance, the
 259 object is not pickled again --- a reference to it is pickled and the
 260 \class{Unpickler} will return the old value, not the modified one.
 261 There are two problems here: (1) detecting changes, and (2)
 262 marshalling a minimal set of changes.  Garbage Collection may also
 263 become a problem here.}.
 264
 265 \class{Unpickler} objects are defined as:
 266
 267 \begin{classdesc}{Unpickler}{file}
 268 This takes a file-like object from which it will read a pickle data
 269 stream.  This class automatically determines whether the data stream
 270 was written in binary mode or not, so it does not need a flag as in
 271 the \class{Pickler} factory.
 272
 273 \var{file} must have two methods, a \method{read()} method that takes
 274 an integer argument, and a \method{readline()} method that requires no
 275 arguments.  Both methods should return a string.  Thus \var{file} can
 276 be a file object opened for reading, a
 277 \module{StringIO} object, or any other custom
 278 object that meets this interface.
 279 \end{classdesc}
 280
 281 \class{Unpickler} objects have one (or two) public methods:
 282
 283 \begin{methoddesc}[Unpickler]{load}{}
 284 Read a pickled object representation from the open file object given
 285 in the constructor, and return the reconstituted object hierarchy
 286 specified therein.
 287 \end{methoddesc}
 288
 289 \begin{methoddesc}[Unpickler]{noload}{}
 290 This is just like \method{load()} except that it doesn't actually
 291 create any objects.  This is useful primarily for finding what's
 292 called ``persistent ids'' that may be referenced in a pickle data
 293 stream.  See section~\ref{pickle-protocol} below for more details.
 294
 295 \strong{Note:} the \method{noload()} method is currently only
 296 available on \class{Unpickler} objects created with the
 297 \module{cPickle} module.  \module{pickle} module \class{Unpickler}s do
 298 not have the \method{noload()} method.
 299 \end{methoddesc}
 300
 301 \subsection{What can be pickled and unpickled?}
 302
 303 The following types can be pickled:
 304
 305 \begin{itemize}
 306
 307 \item \code{None}
 308
 309 \item integers, long integers, floating point numbers, complex numbers
 310
 311 \item normal and Unicode strings
 312
 313 \item tuples, lists, and dictionaries containing only picklable objects
 314
 315 \item functions defined at the top level of a module
 316
 317 \item built-in functions defined at the top level of a module
 318
 319 \item classes that are defined at the top level of a module
 320
 321 \item instances of such classes whose \member{__dict__} or
 322 \method{__setstate__()} is picklable  (see
 323 section~\ref{pickle-protocol} for details)
 324
 325 \end{itemize}
 326
 327 Attempts to pickle unpicklable objects will raise the
 328 \exception{PicklingError} exception; when this happens, an unspecified
 329 number of bytes may have already been written to the underlying file.
 330
 331 Note that functions (built-in and user-defined) are pickled by ``fully
 332 qualified'' name reference, not by value.  This means that only the
 333 function name is pickled, along with the name of module the function
 334 is defined in.  Neither the function's code, nor any of its function
 335 attributes are pickled.  Thus the defining module must be importable
 336 in the unpickling environment, and the module must contain the named
 337 object, otherwise an exception will be raised\footnote{The exception
 338 raised will likely be an \exception{ImportError} or an
 339 \exception{AttributeError} but it could be something else.}.
 340
 341 Similarly, classes are pickled by named reference, so the same
 342 restrictions in the unpickling environment apply.  Note that none of
 343 the class's code or data is pickled, so in the following example the
 344 class attribute \code{attr} is not restored in the unpickling
 345 environment:
 346
 347 \begin{verbatim}
 348 class Foo:
 349     attr = 'a class attr'
 350
 351 picklestring = pickle.dumps(Foo)
 352 \end{verbatim}
 353
 354 These restrictions are why picklable functions and classes must be
 355 defined in the top level of a module.
 356
 357 Similarly, when class instances are pickled, their class's code and
 358 data are not pickled along with them.  Only the instance data are
 359 pickled.  This is done on purpose, so you can fix bugs in a class or
 360 add methods to the class and still load objects that were created with
 361 an earlier version of the class.  If you plan to have long-lived
 362 objects that will see many versions of a class, it may be worthwhile
 363 to put a version number in the objects so that suitable conversions
 364 can be made by the class's \method{__setstate__()} method.
 365
 366 \subsection{The pickle protocol
 367 \label{pickle-protocol}}\setindexsubitem{(pickle protocol)}
 368
 369 This section describes the ``pickling protocol'' that defines the
 370 interface between the pickler/unpickler and the objects that are being
 371 serialized.  This protocol provides a standard way for you to define,
 372 customize, and control how your objects are serialized and
 373 de-serialized.  The description in this section doesn't cover specific
 374 customizations that you can employ to make the unpickling environment
 375 safer from untrusted pickle data streams; see section~\ref{pickle-sec}
 376 for more details.
 377
 378 \subsubsection{Pickling and unpickling normal class
 379     instances\label{pickle-inst}}
 380
 381 When a pickled class instance is unpickled, its \method{__init__()}
 382 method is normally \emph{not} invoked.  If it is desirable that the
 383 \method{__init__()} method be called on unpickling, a class can define
 384 a method \method{__getinitargs__()}, which should return a
 385 \emph{tuple} containing the arguments to be passed to the class
 386 constructor (i.e. \method{__init__()}).  The
 387 \method{__getinitargs__()} method is called at
 388 pickle time; the tuple it returns is incorporated in the pickle for
 389 the instance.
 390 \withsubitem{(copy protocol)}{\ttindex{__getinitargs__()}}
 391 \withsubitem{(instance constructor)}{\ttindex{__init__()}}
 392
 393 \withsubitem{(copy protocol)}{
 394   \ttindex{__getstate__()}\ttindex{__setstate__()}}
 395 \withsubitem{(instance attribute)}{
 396   \ttindex{__dict__}}
 397
 398 Classes can further influence how their instances are pickled; if the
 399 class defines the method \method{__getstate__()}, it is called and the
 400 return state is pickled as the contents for the instance, instead of
 401 the contents of the instance's dictionary.  If there is no
 402 \method{__getstate__()} method, the instance's \member{__dict__} is
 403 pickled.
 404
 405 Upon unpickling, if the class also defines the method
 406 \method{__setstate__()}, it is called with the unpickled
 407 state\footnote{These methods can also be used to implement copying
 408 class instances.}.  If there is no \method{__setstate__()} method, the
 409 pickled object must be a dictionary and its items are assigned to the
 410 new instance's dictionary.  If a class defines both
 411 \method{__getstate__()} and \method{__setstate__()}, the state object
 412 needn't be a dictionary and these methods can do what they
 413 want\footnote{This protocol is also used by the shallow and deep
 414 copying operations defined in the
 415 \refmodule{copy} module.}.
 416
 417 \subsubsection{Pickling and unpickling extension types}
 418
 419 When the \class{Pickler} encounters an object of a type it knows
 420 nothing about --- such as an extension type --- it looks in two places
 421 for a hint of how to pickle it.  One alternative is for the object to
 422 implement a \method{__reduce__()} method.  If provided, at pickling
 423 time \method{__reduce__()} will be called with no arguments, and it
 424 must return either a string or a tuple.
 425
 426 If a string is returned, it names a global variable whose contents are
 427 pickled as normal.  When a tuple is returned, it must be of length two
 428 or three, with the following semantics:
 429
 430 \begin{itemize}
 431
 432 \item A callable object, which in the unpickling environment must be
 433       either a class, a callable registered as a ``safe constructor''
 434       (see below), or it must have an attribute
 435       \member{__safe_for_unpickling__} with a true value.  Otherwise,
 436       an \exception{UnpicklingError} will be raised in the unpickling
 437       environment.  Note that as usual, the callable itself is pickled
 438       by name.
 439
 440 \item A tuple of arguments for the callable object, or \code{None}.
 441
 442 \item Optionally, the object's state, which will be passed to
 443       the object's \method{__setstate__()} method as described in
 444       section~\ref{pickle-inst}.  If the object has no
 445       \method{__setstate__()} method, then, as above, the value must
 446       be a dictionary and it will be added to the object's
 447       \member{__dict__}.
 448
 449 \end{itemize}
 450
 451 Upon unpickling, the callable will be called (provided that it meets
 452 the above criteria), passing in the tuple of arguments; it should
 453 return the unpickled object.  If the second item was \code{None}, then
 454 instead of calling the callable directly, its \method{__basicnew__()}
 455 method is called without arguments.  It should also return the
 456 unpickled object.
 457
 458 An alternative to implementing a \method{__reduce__()} method on the
 459 object to be pickled, is to register the callable with the
 460 \refmodule[copyreg]{copy_reg} module.  This module provides a way
 461 for programs to register ``reduction functions'' and constructors for
 462 user-defined types.   Reduction functions have the same semantics and
 463 interface as the \method{__reduce__()} method described above, except
 464 that they are called with a single argument, the object to be pickled.
 465
 466 The registered constructor is deemed a ``safe constructor'' for purposes
 467 of unpickling as described above.
 468
 469 \subsubsection{Pickling and unpickling external objects}
 470
 471 For the benefit of object persistence, the \module{pickle} module
 472 supports the notion of a reference to an object outside the pickled
 473 data stream.  Such objects are referenced by a ``persistent id'',
 474 which is just an arbitrary string of printable \ASCII{} characters.
 475 The resolution of such names is not defined by the \module{pickle}
 476 module; it will delegate this resolution to user defined functions on
 477 the pickler and unpickler\footnote{The actual mechanism for
 478 associating these user defined functions is slightly different for
 479 \module{pickle} and \module{cPickle}.  The description given here
 480 works the same for both implementations.  Users of the \module{pickle}
 481 module could also use subclassing to effect the same results,
 482 overriding the \method{persistent_id()} and \method{persistent_load()}
 483 methods in the derived classes.}.
 484
 485 To define external persistent id resolution, you need to set the
 486 \member{persistent_id} attribute of the pickler object and the
 487 \member{persistent_load} attribute of the unpickler object.
 488
 489 To pickle objects that have an external persistent id, the pickler
 490 must have a custom \function{persistent_id()} method that takes an
 491 object as an argument and returns either \code{None} or the persistent
 492 id for that object.  When \code{None} is returned, the pickler simply
 493 pickles the object as normal.  When a persistent id string is
 494 returned, the pickler will pickle that string, along with a marker
 495 so that the unpickler will recognize the string as a persistent id.
 496
 497 To unpickle external objects, the unpickler must have a custom
 498 \function{persistent_load()} function that takes a persistent id
 499 string and returns the referenced object.
 500
 501 Here's a silly example that \emph{might} shed more light:
 502
 503 \begin{verbatim}
 504 import pickle
 505 from cStringIO import StringIO
 506
 507 src = StringIO()
 508 p = pickle.Pickler(src)
 509
 510 def persistent_id(obj):
 511     if hasattr(obj, 'x'):
 512         return 'the value %d' % obj.x
 513     else:
 514         return None
 515
 516 p.persistent_id = persistent_id
 517
 518 class Integer:
 519     def __init__(self, x):
 520         self.x = x
 521     def __str__(self):
 522         return 'My name is integer %d' % self.x
 523
 524 i = Integer(7)
 525 print i
 526 p.dump(i)
 527
 528 datastream = src.getvalue()
 529 print repr(datastream)
 530 dst = StringIO(datastream)
 531
 532 up = pickle.Unpickler(dst)
 533
 534 class FancyInteger(Integer):
 535     def __str__(self):
 536         return 'I am the integer %d' % self.x
 537
 538 def persistent_load(persid):
 539     if persid.startswith('the value '):
 540         value = int(persid.split()[2])
 541         return FancyInteger(value)
 542     else:
 543         raise pickle.UnpicklingError, 'Invalid persistent id'
 544
 545 up.persistent_load = persistent_load
 546
 547 j = up.load()
 548 print j
 549 \end{verbatim}
 550
 551 In the \module{cPickle} module, the unpickler's
 552 \member{persistent_load} attribute can also be set to a Python
 553 list, in which case, when the unpickler reaches a persistent id, the
 554 persistent id string will simply be appended to this list.  This
 555 functionality exists so that a pickle data stream can be ``sniffed''
 556 for object references without actually instantiating all the objects
 557 in a pickle\footnote{We'll leave you with the image of Guido and Jim
 558 sitting around sniffing pickles in their living rooms.}.  Setting
 559 \member{persistent_load} to a list is usually used in conjunction with
 560 the \method{noload()} method on the Unpickler.
 561
 562 % BAW: Both pickle and cPickle support something called
 563 % inst_persistent_id() which appears to give unknown types a second
 564 % shot at producing a persistent id.  Since Jim Fulton can't remember
 565 % why it was added or what it's for, I'm leaving it undocumented.
 566
 567 \subsection{Security \label{pickle-sec}}
 568
 569 Most of the security issues surrounding the \module{pickle} and
 570 \module{cPickle} module involve unpickling.  There are no known
 571 security vulnerabilities
 572 related to pickling because you (the programmer) control the objects
 573 that \module{pickle} will interact with, and all it produces is a
 574 string.
 575
 576 However, for unpickling, it is \strong{never} a good idea to unpickle
 577 an untrusted string whose origins are dubious, for example, strings
 578 read from a socket.  This is because unpickling can create unexpected
 579 objects and even potentially run methods of those objects, such as
 580 their class constructor or destructor\footnote{A special note of
 581 caution is worth raising about the \refmodule{Cookie}
 582 module.  By default, the \class{Cookie.Cookie} class is an alias for
 583 the \class{Cookie.SmartCookie} class, which ``helpfully'' attempts to
 584 unpickle any cookie data string it is passed.  This is a huge security
 585 hole because cookie data typically comes from an untrusted source.
 586 You should either explicitly use the \class{Cookie.SimpleCookie} class
 587 --- which doesn't attempt to unpickle its string --- or you should
 588 implement the defensive programming steps described later on in this
 589 section.}.
 590
 591 You can defend against this by customizing your unpickler so that you
 592 can control exactly what gets unpickled and what gets called.
 593 Unfortunately, exactly how you do this is different depending on
 594 whether you're using \module{pickle} or \module{cPickle}.
 595
 596 One common feature that both modules implement is the
 597 \member{__safe_for_unpickling__} attribute.  Before calling a callable
 598 which is not a class, the unpickler will check to make sure that the
 599 callable has either been registered as a safe callable via the
 600 \refmodule[copyreg]{copy_reg} module, or that it has an
 601 attribute \member{__safe_for_unpickling__} with a true value.  This
 602 prevents the unpickling environment from being tricked into doing
 603 evil things like call \code{os.unlink()} with an arbitrary file name.
 604 See section~\ref{pickle-protocol} for more details.
 605
 606 For safely unpickling class instances, you need to control exactly
 607 which classes will get created.  Be aware that a class's constructor
 608 could be called (if the pickler found a \method{__getinitargs__()}
 609 method) and the the class's destructor (i.e. its \method{__del__()} method)
 610 might get called when the object is garbage collected.  Depending on
 611 the class, it isn't very heard to trick either method into doing bad
 612 things, such as removing a file.  The way to
 613 control the classes that are safe to instantiate differs in
 614 \module{pickle} and \module{cPickle}\footnote{A word of caution: the
 615 mechanisms described here use internal attributes and methods, which
 616 are subject to change in future versions of Python.  We intend to
 617 someday provide a common interface for controlling this behavior,
 618 which will work in either \module{pickle} or \module{cPickle}.}.
 619
 620 In the \module{pickle} module, you need to derive a subclass from
 621 \class{Unpickler}, overriding the \method{load_global()}
 622 method.  \method{load_global()} should read two lines from the pickle
 623 data stream where the first line will the the name of the module
 624 containing the class and the second line will be the name of the
 625 instance's class.  It then look up the class, possibly importing the
 626 module and digging out the attribute, then it appends what it finds to
 627 the unpickler's stack.  Later on, this class will be assigned to the
 628 \member{__class__} attribute of an empty class, as a way of magically
 629 creating an instance without calling its class's \method{__init__()}.
 630 You job (should you choose to accept it), would be to have
 631 \method{load_global()} push onto the unpickler's stack, a known safe
 632 version of any class you deem safe to unpickle.  It is up to you to
 633 produce such a class.  Or you could raise an error if you want to
 634 disallow all unpickling of instances.  If this sounds like a hack,
 635 you're right.  UTSL.
 636
 637 Things are a little cleaner with \module{cPickle}, but not by much.
 638 To control what gets unpickled, you can set the unpickler's
 639 \member{find_global} attribute to a function or \code{None}.  If it is
 640 \code{None} then any attempts to unpickle instances will raise an
 641 \exception{UnpicklingError}.  If it is a function,
 642 then it should accept a module name and a class name, and return the
 643 corresponding class object.  It is responsible for looking up the
 644 class, again performing any necessary imports, and it may raise an
 645 error to prevent instances of the class from being unpickled.
 646
 647 The moral of the story is that you should be really careful about the
 648 source of the strings your application unpickles.
 649
 650 \subsection{Example \label{pickle-example}}
 651
 652 Here's a simple example of how to modify pickling behavior for a
 653 class.  The \class{TextReader} class opens a text file, and returns
 654 the line number and line contents each time its \method{readline()}
 655 method is called. If a \class{TextReader} instance is pickled, all
 656 attributes \emph{except} the file object member are saved. When the
 657 instance is unpickled, the file is reopened, and reading resumes from
 658 the last location. The \method{__setstate__()} and
 659 \method{__getstate__()} methods are used to implement this behavior.
 660
 661 \begin{verbatim}
 662 class TextReader:
 663     """Print and number lines in a text file."""
 664     def __init__(self, file):
 665         self.file = file
 666         self.fh = open(file)
 667         self.lineno = 0
 668
 669     def readline(self):
 670         self.lineno = self.lineno + 1
 671         line = self.fh.readline()
 672         if not line:
 673             return None
 674         if line.endswith("\n"):
 675             line = line[:-1]
 676         return "%d: %s" % (self.lineno, line)
 677
 678     def __getstate__(self):
 679         odict = self.__dict__.copy() # copy the dict since we change it
 680         del odict['fh']              # remove filehandle entry
 681         return odict
 682
 683     def __setstate__(self,dict):
 684         fh = open(dict['file'])      # reopen file
 685         count = dict['lineno']       # read from file...
 686         while count:                 # until line count is restored
 687             fh.readline()
 688             count = count - 1
 689         self.__dict__.update(dict)   # update attributes
 690         self.fh = fh                 # save the file object
 691 \end{verbatim}
 692
 693 A sample usage might be something like this:
 694
 695 \begin{verbatim}
 696 >>> import TextReader
 697 >>> obj = TextReader.TextReader("TextReader.py")
 698 >>> obj.readline()
 699 '1: #!/usr/local/bin/python'
 700 >>> # (more invocations of obj.readline() here)
 701 ... obj.readline()
 702 '7: class TextReader:'
 703 >>> import pickle
 704 >>> pickle.dump(obj,open('save.p','w'))
 705 \end{verbatim}
 706
 707 If you want to see that \refmodule{pickle} works across Python
 708 processes, start another Python session, before continuing.  What
 709 follows can happen from either the same process or a new process.
 710
 711 \begin{verbatim}
 712 >>> import pickle
 713 >>> reader = pickle.load(open('save.p'))
 714 >>> reader.readline()
 715 '8:     "Print and number lines in a text file."'
 716 \end{verbatim}
 717
 718
 719 \begin{seealso}
 720   \seemodule[copyreg]{copy_reg}{Pickle interface constructor
 721                                 registration for extension types.}
 722
 723   \seemodule{shelve}{Indexed databases of objects; uses \module{pickle}.}
 724
 725   \seemodule{copy}{Shallow and deep object copying.}
 726
 727   \seemodule{marshal}{High-performance serialization of built-in types.}
 728 \end{seealso}
 729
 730
 731 \section{\module{cPickle} --- A faster \module{pickle}}
 732
 733 \declaremodule{builtin}{cPickle}
 734 \modulesynopsis{Faster version of \refmodule{pickle}, but not subclassable.}
 735 \moduleauthor{Jim Fulton}{jfulton@digicool.com}
 736 \sectionauthor{Fred L. Drake, Jr.}{fdrake@acm.org}
 737
 738 The \module{cPickle} module supports serialization and
 739 de-serialization of Python objects, providing an interface and
 740 functionality nearly identical to the
 741 \refmodule{pickle}\refstmodindex{pickle} module.  There are several
 742 differences, the most important being performance and subclassability.
 743
 744 First, \module{cPickle} can be up to 1000 times faster than
 745 \module{pickle} because the former is implemented in C.  Second, in
 746 the \module{cPickle} module the callables \function{Pickler()} and
 747 \function{Unpickler()} are functions, not classes.  This means that
 748 you cannot use them to derive custom pickling and unpickling
 749 subclasses.  Most applications have no need for this functionality and
 750 should benefit from the greatly improved performance of the
 751 \module{cPickle} module.
 752
 753 The pickle data stream produced by \module{pickle} and
 754 \module{cPickle} are identical, so it is possible to use
 755 \module{pickle} and \module{cPickle} interchangeably with existing
 756 pickles\footnote{Since the pickle data format is actually a tiny
 757 stack-oriented programming language, and some freedom is taken in the
 758 encodings of certain objects, it is possible that the two modules
 759 produce different data streams for the same input objects.  However it
 760 is guaranteed that they will always be able to read each other's
 761 data streams.}.
 762
 763 There are additional minor differences in API between \module{cPickle}
 764 and \module{pickle}, however for most applications, they are
 765 interchangable.  More documentation is provided in the
 766 \module{pickle} module documentation, which
 767 includes a list of the documented differences.
 768
 769