Lib/doctest.py

   1 # Module doctest.
   2 # Released to the public domain 16-Jan-2001,
   3 # by Tim Peters (tim.one@home.com).
   4
   5 # Provided as-is; use at your own risk; no warranty; no promises; enjoy!
   6
   7 """Module doctest -- a framework for running examples in docstrings.
   8
   9 NORMAL USAGE
  10
  11 In normal use, end each module M with:
  12
  13 def _test():
  14     import doctest, M           # replace M with your module's name
  15     return doctest.testmod(M)   # ditto
  16
  17 if __name__ == "__main__":
  18     _test()
  19
  20 Then running the module as a script will cause the examples in the
  21 docstrings to get executed and verified:
  22
  23 python M.py
  24
  25 This won't display anything unless an example fails, in which case the
  26 failing example(s) and the cause(s) of the failure(s) are printed to stdout
  27 (why not stderr? because stderr is a lame hack <0.2 wink>), and the final
  28 line of output is "Test failed.".
  29
  30 Run it with the -v switch instead:
  31
  32 python M.py -v
  33
  34 and a detailed report of all examples tried is printed to stdout, along
  35 with assorted summaries at the end.
  36
  37 You can force verbose mode by passing "verbose=1" to testmod, or prohibit
  38 it by passing "verbose=0".  In either of those cases, sys.argv is not
  39 examined by testmod.
  40
  41 In any case, testmod returns a 2-tuple of ints (f, t), where f is the
  42 number of docstring examples that failed and t is the total number of
  43 docstring examples attempted.
  44
  45
  46 WHICH DOCSTRINGS ARE EXAMINED?
  47
  48 + M.__doc__.
  49
  50 + f.__doc__ for all functions f in M.__dict__.values(), except those
  51   with private names and those defined in other modules.
  52
  53 + C.__doc__ for all classes C in M.__dict__.values(), except those with
  54   private names and those defined in other modules.
  55
  56 + If M.__test__ exists and "is true", it must be a dict, and
  57   each entry maps a (string) name to a function object, class object, or
  58   string.  Function and class object docstrings found from M.__test__
  59   are searched even if the name is private, and strings are searched
  60   directly as if they were docstrings.  In output, a key K in M.__test__
  61   appears with name
  62       <name of M>.__test__.K
  63
  64 Any classes found are recursively searched similarly, to test docstrings in
  65 their contained methods and nested classes.  Private names reached from M's
  66 globals are skipped, but all names reached from M.__test__ are searched.
  67
  68 By default, a name is considered to be private if it begins with an
  69 underscore (like "_my_func") but doesn't both begin and end with (at least)
  70 two underscores (like "__init__").  You can change the default by passing
  71 your own "isprivate" function to testmod.
  72
  73 If you want to test docstrings in objects with private names too, stuff
  74 them into an M.__test__ dict, or see ADVANCED USAGE below (e.g., pass your
  75 own isprivate function to Tester's constructor, or call the rundoc method
  76 of a Tester instance).
  77
  78 WHAT'S THE EXECUTION CONTEXT?
  79
  80 By default, each time testmod finds a docstring to test, it uses a *copy*
  81 of M's globals (so that running tests on a module doesn't change the
  82 module's real globals, and so that one test in M can't leave behind crumbs
  83 that accidentally allow another test to work).  This means examples can
  84 freely use any names defined at top-level in M.  It also means that sloppy
  85 imports (see above) can cause examples in external docstrings to use
  86 globals inappropriate for them.
  87
  88 You can force use of your own dict as the execution context by passing
  89 "globs=your_dict" to testmod instead.  Presumably this would be a copy of
  90 M.__dict__ merged with the globals from other imported modules.
  91
  92
  93 WHAT IF I WANT TO TEST A WHOLE PACKAGE?
  94
  95 Piece o' cake, provided the modules do their testing from docstrings.
  96 Here's the test.py I use for the world's most elaborate Rational/
  97 floating-base-conversion pkg (which I'll distribute some day):
  98
  99 from Rational import Cvt
 100 from Rational import Format
 101 from Rational import machprec
 102 from Rational import Rat
 103 from Rational import Round
 104 from Rational import utils
 105
 106 modules = (Cvt,
 107            Format,
 108            machprec,
 109            Rat,
 110            Round,
 111            utils)
 112
 113 def _test():
 114     import doctest
 115     import sys
 116     verbose = "-v" in sys.argv
 117     for mod in modules:
 118         doctest.testmod(mod, verbose=verbose, report=0)
 119     doctest.master.summarize()
 120
 121 if __name__ == "__main__":
 122     _test()
 123
 124 IOW, it just runs testmod on all the pkg modules.  testmod remembers the
 125 names and outcomes (# of failures, # of tries) for each item it's seen, and
 126 passing "report=0" prevents it from printing a summary in verbose mode.
 127 Instead, the summary is delayed until all modules have been tested, and
 128 then "doctest.master.summarize()" forces the summary at the end.
 129
 130 So this is very nice in practice:  each module can be tested individually
 131 with almost no work beyond writing up docstring examples, and collections
 132 of modules can be tested too as a unit with no more work than the above.
 133
 134
 135 WHAT ABOUT EXCEPTIONS?
 136
 137 No problem, as long as the only output generated by the example is the
 138 traceback itself.  For example:
 139
 140     >>> [1, 2, 3].remove(42)
 141     Traceback (most recent call last):
 142       File "<stdin>", line 1, in ?
 143     ValueError: list.remove(x): x not in list
 144     >>>
 145
 146 Note that only the exception type and value are compared (specifically,
 147 only the last line in the traceback).
 148
 149
 150 ADVANCED USAGE
 151
 152 doctest.testmod() captures the testing policy I find most useful most
 153 often.  You may want other policies.
 154
 155 testmod() actually creates a local instance of class doctest.Tester, runs
 156 appropriate methods of that class, and merges the results into global
 157 Tester instance doctest.master.
 158
 159 You can create your own instances of doctest.Tester, and so build your own
 160 policies, or even run methods of doctest.master directly.  See
 161 doctest.Tester.__doc__ for details.
 162
 163
 164 SO WHAT DOES A DOCSTRING EXAMPLE LOOK LIKE ALREADY!?
 165
 166 Oh ya.  It's easy!  In most cases a copy-and-paste of an interactive
 167 console session works fine -- just make sure the leading whitespace is
 168 rigidly consistent (you can mix tabs and spaces if you're too lazy to do it
 169 right, but doctest is not in the business of guessing what you think a tab
 170 means).
 171
 172     >>> # comments are ignored
 173     >>> x = 12
 174     >>> x
 175     12
 176     >>> if x == 13:
 177     ...     print "yes"
 178     ... else:
 179     ...     print "no"
 180     ...     print "NO"
 181     ...     print "NO!!!"
 182     ...
 183     no
 184     NO
 185     NO!!!
 186     >>>
 187
 188 Any expected output must immediately follow the final ">>>" or "..." line
 189 containing the code, and the expected output (if any) extends to the next
 190 ">>>" or all-whitespace line.  That's it.
 191
 192 Bummers:
 193
 194 + Expected output cannot contain an all-whitespace line, since such a line
 195   is taken to signal the end of expected output.
 196
 197 + Output to stdout is captured, but not output to stderr (exception
 198   tracebacks are captured via a different means).
 199
 200 + If you continue a line via backslashing in an interactive session, or for
 201   any other reason use a backslash, you need to double the backslash in the
 202   docstring version.  This is simply because you're in a string, and so the
 203   backslash must be escaped for it to survive intact.  Like:
 204
 205 >>> if "yes" == \\
 206 ...     "y" +   \\
 207 ...     "es":   # in the source code you'll see the doubled backslashes
 208 ...     print 'yes'
 209 yes
 210
 211 The starting column doesn't matter:
 212
 213 >>> assert "Easy!"
 214      >>> import math
 215             >>> math.floor(1.9)
 216             1.0
 217
 218 and as many leading whitespace characters are stripped from the expected
 219 output as appeared in the initial ">>>" line that triggered it.
 220
 221 If you execute this very file, the examples above will be found and
 222 executed, leading to this output in verbose mode:
 223
 224 Running doctest.__doc__
 225 Trying: [1, 2, 3].remove(42)
 226 Expecting:
 227 Traceback (most recent call last):
 228   File "<stdin>", line 1, in ?
 229 ValueError: list.remove(x): x not in list
 230 ok
 231 Trying: x = 12
 232 Expecting: nothing
 233 ok
 234 Trying: x
 235 Expecting: 12
 236 ok
 237 Trying:
 238 if x == 13:
 239     print "yes"
 240 else:
 241     print "no"
 242     print "NO"
 243     print "NO!!!"
 244 Expecting:
 245 no
 246 NO
 247 NO!!!
 248 ok
 249 ... and a bunch more like that, with this summary at the end:
 250
 251 5 items had no tests:
 252     doctest.Tester.__init__
 253     doctest.Tester.run__test__
 254     doctest.Tester.summarize
 255     doctest.run_docstring_examples
 256     doctest.testmod
 257 12 items passed all tests:
 258    8 tests in doctest
 259    6 tests in doctest.Tester
 260   10 tests in doctest.Tester.merge
 261   14 tests in doctest.Tester.rundict
 262    3 tests in doctest.Tester.rundoc
 263    3 tests in doctest.Tester.runstring
 264    2 tests in doctest.__test__._TestClass
 265    2 tests in doctest.__test__._TestClass.__init__
 266    2 tests in doctest.__test__._TestClass.get
 267    1 tests in doctest.__test__._TestClass.square
 268    2 tests in doctest.__test__.string
 269    7 tests in doctest.is_private
 270 60 tests in 17 items.
 271 60 passed and 0 failed.
 272 Test passed.
 273 """
 274
 275 __all__ = [
 276     'testmod',
 277     'run_docstring_examples',
 278     'is_private',
 279     'Tester',
 280 ]
 281
 282 import __future__
 283
 284 import re
 285 PS1 = ">>>"
 286 PS2 = "..."
 287 _isPS1 = re.compile(r"(\s*)" + re.escape(PS1)).match
 288 _isPS2 = re.compile(r"(\s*)" + re.escape(PS2)).match
 289 _isEmpty = re.compile(r"\s*$").match
 290 _isComment = re.compile(r"\s*#").match
 291 del re
 292
 293 from types import StringTypes as _StringTypes
 294
 295 from inspect import isclass    as _isclass
 296 from inspect import isfunction as _isfunction
 297 from inspect import ismodule   as _ismodule
 298 from inspect import classify_class_attrs as _classify_class_attrs
 299
 300 # Extract interactive examples from a string.  Return a list of triples,
 301 # (source, outcome, lineno).  "source" is the source code, and ends
 302 # with a newline iff the source spans more than one line.  "outcome" is
 303 # the expected output if any, else an empty string.  When not empty,
 304 # outcome always ends with a newline.  "lineno" is the line number,
 305 # 0-based wrt the start of the string, of the first source line.
 306
 307 def _extract_examples(s):
 308     isPS1, isPS2 = _isPS1, _isPS2
 309     isEmpty, isComment = _isEmpty, _isComment
 310     examples = []
 311     lines = s.split("\n")
 312     i, n = 0, len(lines)
 313     while i < n:
 314         line = lines[i]
 315         i = i + 1
 316         m = isPS1(line)
 317         if m is None:
 318             continue
 319         j = m.end(0)  # beyond the prompt
 320         if isEmpty(line, j) or isComment(line, j):
 321             # a bare prompt or comment -- not interesting
 322             continue
 323         lineno = i - 1
 324         if line[j] != " ":
 325             raise ValueError("line " + `lineno` + " of docstring lacks "
 326                 "blank after " + PS1 + ": " + line)
 327         j = j + 1
 328         blanks = m.group(1)
 329         nblanks = len(blanks)
 330         # suck up this and following PS2 lines
 331         source = []
 332         while 1:
 333             source.append(line[j:])
 334             line = lines[i]
 335             m = isPS2(line)
 336             if m:
 337                 if m.group(1) != blanks:
 338                     raise ValueError("inconsistent leading whitespace "
 339                         "in line " + `i` + " of docstring: " + line)
 340                 i = i + 1
 341             else:
 342                 break
 343         if len(source) == 1:
 344             source = source[0]
 345         else:
 346             # get rid of useless null line from trailing empty "..."
 347             if source[-1] == "":
 348                 del source[-1]
 349             source = "\n".join(source) + "\n"
 350         # suck up response
 351         if isPS1(line) or isEmpty(line):
 352             expect = ""
 353         else:
 354             expect = []
 355             while 1:
 356                 if line[:nblanks] != blanks:
 357                     raise ValueError("inconsistent leading whitespace "
 358                         "in line " + `i` + " of docstring: " + line)
 359                 expect.append(line[nblanks:])
 360                 i = i + 1
 361                 line = lines[i]
 362                 if isPS1(line) or isEmpty(line):
 363                     break
 364             expect = "\n".join(expect) + "\n"
 365         examples.append( (source, expect, lineno) )
 366     return examples
 367
 368 # Capture stdout when running examples.
 369
 370 class _SpoofOut:
 371     def __init__(self):
 372         self.clear()
 373     def write(self, s):
 374         self.buf.append(s)
 375     def get(self):
 376         guts = "".join(self.buf)
 377         # If anything at all was written, make sure there's a trailing
 378         # newline.  There's no way for the expected output to indicate
 379         # that a trailing newline is missing.
 380         if guts and not guts.endswith("\n"):
 381             guts = guts + "\n"
 382         # Prevent softspace from screwing up the next test case, in
 383         # case they used print with a trailing comma in an example.
 384         if hasattr(self, "softspace"):
 385             del self.softspace
 386         return guts
 387     def clear(self):
 388         self.buf = []
 389         if hasattr(self, "softspace"):
 390             del self.softspace
 391     def flush(self):
 392         # JPython calls flush
 393         pass
 394
 395 # Display some tag-and-msg pairs nicely, keeping the tag and its msg
 396 # on the same line when that makes sense.
 397
 398 def _tag_out(printer, *tag_msg_pairs):
 399     for tag, msg in tag_msg_pairs:
 400         printer(tag + ":")
 401         msg_has_nl = msg[-1:] == "\n"
 402         msg_has_two_nl = msg_has_nl and \
 403                         msg.find("\n") < len(msg) - 1
 404         if len(tag) + len(msg) < 76 and not msg_has_two_nl:
 405             printer(" ")
 406         else:
 407             printer("\n")
 408         printer(msg)
 409         if not msg_has_nl:
 410             printer("\n")
 411
 412 # Run list of examples, in context globs.  "out" can be used to display
 413 # stuff to "the real" stdout, and fakeout is an instance of _SpoofOut
 414 # that captures the examples' std output.  Return (#failures, #tries).
 415
 416 def _run_examples_inner(out, fakeout, examples, globs, verbose, name,
 417                         compileflags):
 418     import sys, traceback
 419     OK, BOOM, FAIL = range(3)
 420     NADA = "nothing"
 421     stderr = _SpoofOut()
 422     failures = 0
 423     for source, want, lineno in examples:
 424         if verbose:
 425             _tag_out(out, ("Trying", source),
 426                           ("Expecting", want or NADA))
 427         fakeout.clear()
 428         try:
 429             exec compile(source, "<string>", "single",
 430                          compileflags, 1) in globs
 431             got = fakeout.get()
 432             state = OK
 433         except KeyboardInterrupt:
 434             raise
 435         except:
 436             # See whether the exception was expected.
 437             if want.find("Traceback (innermost last):\n") == 0 or \
 438                want.find("Traceback (most recent call last):\n") == 0:
 439                 # Only compare exception type and value - the rest of
 440                 # the traceback isn't necessary.
 441                 want = want.split('\n')[-2] + '\n'
 442                 exc_type, exc_val = sys.exc_info()[:2]
 443                 got = traceback.format_exception_only(exc_type, exc_val)[-1]
 444                 state = OK
 445             else:
 446                 # unexpected exception
 447                 stderr.clear()
 448                 traceback.print_exc(file=stderr)
 449                 state = BOOM
 450
 451         if state == OK:
 452             if got == want:
 453                 if verbose:
 454                     out("ok\n")
 455                 continue
 456             state = FAIL
 457
 458         assert state in (FAIL, BOOM)
 459         failures = failures + 1
 460         out("*" * 65 + "\n")
 461         _tag_out(out, ("Failure in example", source))
 462         out("from line #" + `lineno` + " of " + name + "\n")
 463         if state == FAIL:
 464             _tag_out(out, ("Expected", want or NADA), ("Got", got))
 465         else:
 466             assert state == BOOM
 467             _tag_out(out, ("Exception raised", stderr.get()))
 468
 469     return failures, len(examples)
 470
 471 # Get the future-flags associated with the future features that have been
 472 # imported into globs.
 473
 474 def _extract_future_flags(globs):
 475     flags = 0
 476     for fname in __future__.all_feature_names:
 477         feature = globs.get(fname, None)
 478         if feature is getattr(__future__, fname):
 479             flags |= feature.compiler_flag
 480     return flags
 481
 482 # Run list of examples, in a shallow copy of context (dict) globs.
 483 # Return (#failures, #tries).
 484
 485 def _run_examples(examples, globs, verbose, name, compileflags):
 486     import sys
 487     saveout = sys.stdout
 488     globs = globs.copy()
 489     try:
 490         sys.stdout = fakeout = _SpoofOut()
 491         x = _run_examples_inner(saveout.write, fakeout, examples,
 492                                 globs, verbose, name, compileflags)
 493     finally:
 494         sys.stdout = saveout
 495         # While Python gc can clean up most cycles on its own, it doesn't
 496         # chase frame objects.  This is especially irksome when running
 497         # generator tests that raise exceptions, because a named generator-
 498         # iterator gets an entry in globs, and the generator-iterator
 499         # object's frame's traceback info points back to globs.  This is
 500         # easy to break just by clearing the namespace.  This can also
 501         # help to break other kinds of cycles, and even for cycles that
 502         # gc can break itself it's better to break them ASAP.
 503         globs.clear()
 504     return x
 505
 506 def run_docstring_examples(f, globs, verbose=0, name="NoName",
 507                            compileflags=None):
 508     """f, globs, verbose=0, name="NoName" -> run examples from f.__doc__.
 509
 510     Use (a shallow copy of) dict globs as the globals for execution.
 511     Return (#failures, #tries).
 512
 513     If optional arg verbose is true, print stuff even if there are no
 514     failures.
 515     Use string name in failure msgs.
 516     """
 517
 518     try:
 519         doc = f.__doc__
 520         if not doc:
 521             # docstring empty or None
 522             return 0, 0
 523         # just in case CT invents a doc object that has to be forced
 524         # to look like a string <0.9 wink>
 525         doc = str(doc)
 526     except KeyboardInterrupt:
 527         raise
 528     except:
 529         return 0, 0
 530
 531     e = _extract_examples(doc)
 532     if not e:
 533         return 0, 0
 534     if compileflags is None:
 535         compileflags = _extract_future_flags(globs)
 536     return _run_examples(e, globs, verbose, name, compileflags)
 537
 538 def is_private(prefix, base):
 539     """prefix, base -> true iff name prefix + "." + base is "private".
 540
 541     Prefix may be an empty string, and base does not contain a period.
 542     Prefix is ignored (although functions you write conforming to this
 543     protocol may make use of it).
 544     Return true iff base begins with an (at least one) underscore, but
 545     does not both begin and end with (at least) two underscores.
 546
 547     >>> is_private("a.b", "my_func")
 548     False
 549     >>> is_private("____", "_my_func")
 550     True
 551     >>> is_private("someclass", "__init__")
 552     False
 553     >>> is_private("sometypo", "__init_")
 554     True
 555     >>> is_private("x.y.z", "_")
 556     True
 557     >>> is_private("_x.y.z", "__")
 558     False
 559     >>> is_private("", "")  # senseless but consistent
 560     False
 561     """
 562
 563     return base[:1] == "_" and not base[:2] == "__" == base[-2:]
 564
 565 # Determine if a class of function was defined in the given module.
 566
 567 def _from_module(module, object):
 568     if _isfunction(object):
 569         return module.__dict__ is object.func_globals
 570     if _isclass(object):
 571         return module.__name__ == object.__module__
 572     raise ValueError("object must be a class or function")
 573
 574 class Tester:
 575     """Class Tester -- runs docstring examples and accumulates stats.
 576
 577 In normal use, function doctest.testmod() hides all this from you,
 578 so use that if you can.  Create your own instances of Tester to do
 579 fancier things.
 580
 581 Methods:
 582     runstring(s, name)
 583         Search string s for examples to run; use name for logging.
 584         Return (#failures, #tries).
 585
 586     rundoc(object, name=None)
 587         Search object.__doc__ for examples to run; use name (or
 588         object.__name__) for logging.  Return (#failures, #tries).
 589
 590     rundict(d, name, module=None)
 591         Search for examples in docstrings in all of d.values(); use name
 592         for logging.  Exclude functions and classes not defined in module
 593         if specified.  Return (#failures, #tries).
 594
 595     run__test__(d, name)
 596         Treat dict d like module.__test__.  Return (#failures, #tries).
 597
 598     summarize(verbose=None)
 599         Display summary of testing results, to stdout.  Return
 600         (#failures, #tries).
 601
 602     merge(other)
 603         Merge in the test results from Tester instance "other".
 604
 605 >>> from doctest import Tester
 606 >>> t = Tester(globs={'x': 42}, verbose=0)
 607 >>> t.runstring(r'''
 608 ...      >>> x = x * 2
 609 ...      >>> print x
 610 ...      42
 611 ... ''', 'XYZ')
 612 *****************************************************************
 613 Failure in example: print x
 614 from line #2 of XYZ
 615 Expected: 42
 616 Got: 84
 617 (1, 2)
 618 >>> t.runstring(">>> x = x * 2\\n>>> print x\\n84\\n", 'example2')
 619 (0, 2)
 620 >>> t.summarize()
 621 *****************************************************************
 622 1 items had failures:
 623    1 of   2 in XYZ
 624 ***Test Failed*** 1 failures.
 625 (1, 4)
 626 >>> t.summarize(verbose=1)
 627 1 items passed all tests:
 628    2 tests in example2
 629 *****************************************************************
 630 1 items had failures:
 631    1 of   2 in XYZ
 632 4 tests in 2 items.
 633 3 passed and 1 failed.
 634 ***Test Failed*** 1 failures.
 635 (1, 4)
 636 >>>
 637 """
 638
 639     def __init__(self, mod=None, globs=None, verbose=None,
 640                  isprivate=None):
 641         """mod=None, globs=None, verbose=None, isprivate=None
 642
 643 See doctest.__doc__ for an overview.
 644
 645 Optional keyword arg "mod" is a module, whose globals are used for
 646 executing examples.  If not specified, globs must be specified.
 647
 648 Optional keyword arg "globs" gives a dict to be used as the globals
 649 when executing examples; if not specified, use the globals from
 650 module mod.
 651
 652 In either case, a copy of the dict is used for each docstring
 653 examined.
 654
 655 Optional keyword arg "verbose" prints lots of stuff if true, only
 656 failures if false; by default, it's true iff "-v" is in sys.argv.
 657
 658 Optional keyword arg "isprivate" specifies a function used to determine
 659 whether a name is private.  The default function is doctest.is_private;
 660 see its docs for details.
 661 """
 662
 663         if mod is None and globs is None:
 664             raise TypeError("Tester.__init__: must specify mod or globs")
 665         if mod is not None and not _ismodule(mod):
 666             raise TypeError("Tester.__init__: mod must be a module; " +
 667                             `mod`)
 668         if globs is None:
 669             globs = mod.__dict__
 670         self.globs = globs
 671
 672         if verbose is None:
 673             import sys
 674             verbose = "-v" in sys.argv
 675         self.verbose = verbose
 676
 677         if isprivate is None:
 678             isprivate = is_private
 679         self.isprivate = isprivate
 680
 681         self.name2ft = {}   # map name to (#failures, #trials) pair
 682
 683         self.compileflags = _extract_future_flags(globs)
 684
 685     def runstring(self, s, name):
 686         """
 687         s, name -> search string s for examples to run, logging as name.
 688
 689         Use string name as the key for logging the outcome.
 690         Return (#failures, #examples).
 691
 692         >>> t = Tester(globs={}, verbose=1)
 693         >>> test = r'''
 694         ...    # just an example
 695         ...    >>> x = 1 + 2
 696         ...    >>> x
 697         ...    3
 698         ... '''
 699         >>> t.runstring(test, "Example")
 700         Running string Example
 701         Trying: x = 1 + 2
 702         Expecting: nothing
 703         ok
 704         Trying: x
 705         Expecting: 3
 706         ok
 707         0 of 2 examples failed in string Example
 708         (0, 2)
 709         """
 710
 711         if self.verbose:
 712             print "Running string", name
 713         f = t = 0
 714         e = _extract_examples(s)
 715         if e:
 716             f, t = _run_examples(e, self.globs, self.verbose, name,
 717                                  self.compileflags)
 718         if self.verbose:
 719             print f, "of", t, "examples failed in string", name
 720         self.__record_outcome(name, f, t)
 721         return f, t
 722
 723     def rundoc(self, object, name=None):
 724         """
 725         object, name=None -> search object.__doc__ for examples to run.
 726
 727         Use optional string name as the key for logging the outcome;
 728         by default use object.__name__.
 729         Return (#failures, #examples).
 730         If object is a class object, search recursively for method
 731         docstrings too.
 732         object.__doc__ is examined regardless of name, but if object is
 733         a class, whether private names reached from object are searched
 734         depends on the constructor's "isprivate" argument.
 735
 736         >>> t = Tester(globs={}, verbose=0)
 737         >>> def _f():
 738         ...     '''Trivial docstring example.
 739         ...     >>> assert 2 == 2
 740         ...     '''
 741         ...     return 32
 742         ...
 743         >>> t.rundoc(_f)  # expect 0 failures in 1 example
 744         (0, 1)
 745         """
 746
 747         if name is None:
 748             try:
 749                 name = object.__name__
 750             except AttributeError:
 751                 raise ValueError("Tester.rundoc: name must be given "
 752                     "when object.__name__ doesn't exist; " + `object`)
 753         if self.verbose:
 754             print "Running", name + ".__doc__"
 755         f, t = run_docstring_examples(object, self.globs, self.verbose, name,
 756                                       self.compileflags)
 757         if self.verbose:
 758             print f, "of", t, "examples failed in", name + ".__doc__"
 759         self.__record_outcome(name, f, t)
 760         if _isclass(object):
 761             # In 2.2, class and static methods complicate life.  Build
 762             # a dict "that works", by hook or by crook.
 763             d = {}
 764             for tag, kind, homecls, value in _classify_class_attrs(object):
 765
 766                 if homecls is not object:
 767                     # Only look at names defined immediately by the class.
 768                     continue
 769
 770                 elif self.isprivate(name, tag):
 771                     continue
 772
 773                 elif kind == "method":
 774                     # value is already a function
 775                     d[tag] = value
 776
 777                 elif kind == "static method":
 778                     # value isn't a function, but getattr reveals one
 779                     d[tag] = getattr(object, tag)
 780
 781                 elif kind == "class method":
 782                     # Hmm.  A classmethod object doesn't seem to reveal
 783                     # enough.  But getattr turns it into a bound method,
 784                     # and from there .im_func retrieves the underlying
 785                     # function.
 786                     d[tag] = getattr(object, tag).im_func
 787
 788                 elif kind == "property":
 789                     # The methods implementing the property have their
 790                     # own docstrings -- but the property may have one too.
 791                     if value.__doc__ is not None:
 792                         d[tag] = str(value.__doc__)
 793
 794                 elif kind == "data":
 795                     # Grab nested classes.
 796                     if _isclass(value):
 797                         d[tag] = value
 798
 799                 else:
 800                     raise ValueError("teach doctest about %r" % kind)
 801
 802             f2, t2 = self.run__test__(d, name)
 803             f += f2
 804             t += t2
 805
 806         return f, t
 807
 808     def rundict(self, d, name, module=None):
 809         """
 810         d, name, module=None -> search for docstring examples in d.values().
 811
 812         For k, v in d.items() such that v is a function or class,
 813         do self.rundoc(v, name + "." + k).  Whether this includes
 814         objects with private names depends on the constructor's
 815         "isprivate" argument.  If module is specified, functions and
 816         classes that are not defined in module are excluded.
 817         Return aggregate (#failures, #examples).
 818
 819         Build and populate two modules with sample functions to test that
 820         exclusion of external functions and classes works.
 821
 822         >>> import new
 823         >>> m1 = new.module('_m1')
 824         >>> m2 = new.module('_m2')
 825         >>> test_data = \"""
 826         ... def _f():
 827         ...     '''>>> assert 1 == 1
 828         ...     '''
 829         ... def g():
 830         ...    '''>>> assert 2 != 1
 831         ...    '''
 832         ... class H:
 833         ...    '''>>> assert 2 > 1
 834         ...    '''
 835         ...    def bar(self):
 836         ...        '''>>> assert 1 < 2
 837         ...        '''
 838         ... \"""
 839         >>> exec test_data in m1.__dict__
 840         >>> exec test_data in m2.__dict__
 841         >>> m1.__dict__.update({"f2": m2._f, "g2": m2.g, "h2": m2.H})
 842
 843         Tests that objects outside m1 are excluded:
 844
 845         >>> t = Tester(globs={}, verbose=0)
 846         >>> t.rundict(m1.__dict__, "rundict_test", m1)  # _f, f2 and g2 and h2 skipped
 847         (0, 3)
 848
 849         Again, but with a custom isprivate function allowing _f:
 850
 851         >>> t = Tester(globs={}, verbose=0, isprivate=lambda x,y: 0)
 852         >>> t.rundict(m1.__dict__, "rundict_test_pvt", m1)  # Only f2, g2 and h2 skipped
 853         (0, 4)
 854
 855         And once more, not excluding stuff outside m1:
 856
 857         >>> t = Tester(globs={}, verbose=0, isprivate=lambda x,y: 0)
 858         >>> t.rundict(m1.__dict__, "rundict_test_pvt")  # None are skipped.
 859         (0, 8)
 860
 861         The exclusion of objects from outside the designated module is
 862         meant to be invoked automagically by testmod.
 863
 864         >>> testmod(m1)
 865         (0, 3)
 866
 867         """
 868
 869         if not hasattr(d, "items"):
 870             raise TypeError("Tester.rundict: d must support .items(); " +
 871                             `d`)
 872         f = t = 0
 873         # Run the tests by alpha order of names, for consistency in
 874         # verbose-mode output.
 875         names = d.keys()
 876         names.sort()
 877         for thisname in names:
 878             value = d[thisname]
 879             if _isfunction(value) or _isclass(value):
 880                 if module and not _from_module(module, value):
 881                     continue
 882                 f2, t2 = self.__runone(value, name + "." + thisname)
 883                 f = f + f2
 884                 t = t + t2
 885         return f, t
 886
 887     def run__test__(self, d, name):
 888         """d, name -> Treat dict d like module.__test__.
 889
 890         Return (#failures, #tries).
 891         See testmod.__doc__ for details.
 892         """
 893
 894         failures = tries = 0
 895         prefix = name + "."
 896         savepvt = self.isprivate
 897         try:
 898             self.isprivate = lambda *args: 0
 899             # Run the tests by alpha order of names, for consistency in
 900             # verbose-mode output.
 901             keys = d.keys()
 902             keys.sort()
 903             for k in keys:
 904                 v = d[k]
 905                 thisname = prefix + k
 906                 if type(v) in _StringTypes:
 907                     f, t = self.runstring(v, thisname)
 908                 elif _isfunction(v) or _isclass(v):
 909                     f, t = self.rundoc(v, thisname)
 910                 else:
 911                     raise TypeError("Tester.run__test__: values in "
 912                             "dict must be strings, functions "
 913                             "or classes; " + `v`)
 914                 failures = failures + f
 915                 tries = tries + t
 916         finally:
 917             self.isprivate = savepvt
 918         return failures, tries
 919
 920     def summarize(self, verbose=None):
 921         """
 922         verbose=None -> summarize results, return (#failures, #tests).
 923
 924         Print summary of test results to stdout.
 925         Optional arg 'verbose' controls how wordy this is.  By
 926         default, use the verbose setting established by the
 927         constructor.
 928         """
 929
 930         if verbose is None:
 931             verbose = self.verbose
 932         notests = []
 933         passed = []
 934         failed = []
 935         totalt = totalf = 0
 936         for x in self.name2ft.items():
 937             name, (f, t) = x
 938             assert f <= t
 939             totalt = totalt + t
 940             totalf = totalf + f
 941             if t == 0:
 942                 notests.append(name)
 943             elif f == 0:
 944                 passed.append( (name, t) )
 945             else:
 946                 failed.append(x)
 947         if verbose:
 948             if notests:
 949                 print len(notests), "items had no tests:"
 950                 notests.sort()
 951                 for thing in notests:
 952                     print "   ", thing
 953             if passed:
 954                 print len(passed), "items passed all tests:"
 955                 passed.sort()
 956                 for thing, count in passed:
 957                     print " %3d tests in %s" % (count, thing)
 958         if failed:
 959             print "*" * 65
 960             print len(failed), "items had failures:"
 961             failed.sort()
 962             for thing, (f, t) in failed:
 963                 print " %3d of %3d in %s" % (f, t, thing)
 964         if verbose:
 965             print totalt, "tests in", len(self.name2ft), "items."
 966             print totalt - totalf, "passed and", totalf, "failed."
 967         if totalf:
 968             print "***Test Failed***", totalf, "failures."
 969         elif verbose:
 970             print "Test passed."
 971         return totalf, totalt
 972
 973     def merge(self, other):
 974         """
 975         other -> merge in test results from the other Tester instance.
 976
 977         If self and other both have a test result for something
 978         with the same name, the (#failures, #tests) results are
 979         summed, and a warning is printed to stdout.
 980
 981         >>> from doctest import Tester
 982         >>> t1 = Tester(globs={}, verbose=0)
 983         >>> t1.runstring('''
 984         ... >>> x = 12
 985         ... >>> print x
 986         ... 12
 987         ... ''', "t1example")
 988         (0, 2)
 989         >>>
 990         >>> t2 = Tester(globs={}, verbose=0)
 991         >>> t2.runstring('''
 992         ... >>> x = 13
 993         ... >>> print x
 994         ... 13
 995         ... ''', "t2example")
 996         (0, 2)
 997         >>> common = ">>> assert 1 + 2 == 3\\n"
 998         >>> t1.runstring(common, "common")
 999         (0, 1)
1000         >>> t2.runstring(common, "common")
1001         (0, 1)
1002         >>> t1.merge(t2)
1003         *** Tester.merge: 'common' in both testers; summing outcomes.
1004         >>> t1.summarize(1)
1005         3 items passed all tests:
1006            2 tests in common
1007            2 tests in t1example
1008            2 tests in t2example
1009         6 tests in 3 items.
1010         6 passed and 0 failed.
1011         Test passed.
1012         (0, 6)
1013         >>>
1014         """
1015
1016         d = self.name2ft
1017         for name, (f, t) in other.name2ft.items():
1018             if name in d:
1019                 print "*** Tester.merge: '" + name + "' in both" \
1020                     " testers; summing outcomes."
1021                 f2, t2 = d[name]
1022                 f = f + f2
1023                 t = t + t2
1024             d[name] = f, t
1025
1026     def __record_outcome(self, name, f, t):
1027         if name in self.name2ft:
1028             print "*** Warning: '" + name + "' was tested before;", \
1029                 "summing outcomes."
1030             f2, t2 = self.name2ft[name]
1031             f = f + f2
1032             t = t + t2
1033         self.name2ft[name] = f, t
1034
1035     def __runone(self, target, name):
1036         if "." in name:
1037             i = name.rindex(".")
1038             prefix, base = name[:i], name[i+1:]
1039         else:
1040             prefix, base = "", base
1041         if self.isprivate(prefix, base):
1042             return 0, 0
1043         return self.rundoc(target, name)
1044
1045 master = None
1046
1047 def testmod(m=None, name=None, globs=None, verbose=None, isprivate=None,
1048                report=1):
1049     """m=None, name=None, globs=None, verbose=None, isprivate=None, report=1
1050
1051     Test examples in docstrings in functions and classes reachable
1052     from module m (or the current module if m is not supplied), starting
1053     with m.__doc__.  Private names are skipped.
1054
1055     Also test examples reachable from dict m.__test__ if it exists and is
1056     not None.  m.__dict__ maps names to functions, classes and strings;
1057     function and class docstrings are tested even if the name is private;
1058     strings are tested directly, as if they were docstrings.
1059
1060     Return (#failures, #tests).
1061
1062     See doctest.__doc__ for an overview.
1063
1064     Optional keyword arg "name" gives the name of the module; by default
1065     use m.__name__.
1066
1067     Optional keyword arg "globs" gives a dict to be used as the globals
1068     when executing examples; by default, use m.__dict__.  A copy of this
1069     dict is actually used for each docstring, so that each docstring's
1070     examples start with a clean slate.
1071
1072     Optional keyword arg "verbose" prints lots of stuff if true, prints
1073     only failures if false; by default, it's true iff "-v" is in sys.argv.
1074
1075     Optional keyword arg "isprivate" specifies a function used to
1076     determine whether a name is private.  The default function is
1077     doctest.is_private; see its docs for details.
1078
1079     Optional keyword arg "report" prints a summary at the end when true,
1080     else prints nothing at the end.  In verbose mode, the summary is
1081     detailed, else very brief (in fact, empty if all tests passed).
1082
1083     Advanced tomfoolery:  testmod runs methods of a local instance of
1084     class doctest.Tester, then merges the results into (or creates)
1085     global Tester instance doctest.master.  Methods of doctest.master
1086     can be called directly too, if you want to do something unusual.
1087     Passing report=0 to testmod is especially useful then, to delay
1088     displaying a summary.  Invoke doctest.master.summarize(verbose)
1089     when you're done fiddling.
1090     """
1091
1092     global master
1093
1094     if m is None:
1095         import sys
1096         # DWA - m will still be None if this wasn't invoked from the command
1097         # line, in which case the following TypeError is about as good an error
1098         # as we should expect
1099         m = sys.modules.get('__main__')
1100
1101     if not _ismodule(m):
1102         raise TypeError("testmod: module required; " + `m`)
1103     if name is None:
1104         name = m.__name__
1105     tester = Tester(m, globs=globs, verbose=verbose, isprivate=isprivate)
1106     failures, tries = tester.rundoc(m, name)
1107     f, t = tester.rundict(m.__dict__, name, m)
1108     failures = failures + f
1109     tries = tries + t
1110     if hasattr(m, "__test__"):
1111         testdict = m.__test__
1112         if testdict:
1113             if not hasattr(testdict, "items"):
1114                 raise TypeError("testmod: module.__test__ must support "
1115                                 ".items(); " + `testdict`)
1116             f, t = tester.run__test__(testdict, name + ".__test__")
1117             failures = failures + f
1118             tries = tries + t
1119     if report:
1120         tester.summarize()
1121     if master is None:
1122         master = tester
1123     else:
1124         master.merge(tester)
1125     return failures, tries
1126
1127 class _TestClass:
1128     """
1129     A pointless class, for sanity-checking of docstring testing.
1130
1131     Methods:
1132         square()
1133         get()
1134
1135     >>> _TestClass(13).get() + _TestClass(-12).get()
1136     1
1137     >>> hex(_TestClass(13).square().get())
1138     '0xa9'
1139     """
1140
1141     def __init__(self, val):
1142         """val -> _TestClass object with associated value val.
1143
1144         >>> t = _TestClass(123)
1145         >>> print t.get()
1146         123
1147         """
1148
1149         self.val = val
1150
1151     def square(self):
1152         """square() -> square TestClass's associated value
1153
1154         >>> _TestClass(13).square().get()
1155         169
1156         """
1157
1158         self.val = self.val ** 2
1159         return self
1160
1161     def get(self):
1162         """get() -> return TestClass's associated value.
1163
1164         >>> x = _TestClass(-42)
1165         >>> print x.get()
1166         -42
1167         """
1168
1169         return self.val
1170
1171 __test__ = {"_TestClass": _TestClass,
1172             "string": r"""
1173                       Example of a string object, searched as-is.
1174                       >>> x = 1; y = 2
1175                       >>> x + y, x * y
1176                       (3, 2)
1177                       """
1178            }
1179
1180 def _test():
1181     import doctest
1182     return doctest.testmod(doctest)
1183
1184 if __name__ == "__main__":
1185     _test()