Release 0.2pre2 metadata/documentation sync

[nobug.git] / README

NOBUG Documentation

  * "Everyone makes errors, but with NoBug you won't make them twice!"

Nobug is a simple debugging library for instrumenting C and C++ programs
inspired from Design-by-Contract ideas.

Overview

Nobug provides you with the following:

  * Three different levels for checks - in depth to final no-overhead
  * Scope tags - tell whenever a function or loop is considered to be bug free
  * Preconditional, Postcondition and Invariant checks, generic Assertions
  * Debugger support (actions are only executed while running under a
    debugger), currently only valgrind
  * Dumping of your datastructures
  * Logging your application's activities
  * Runtime customizable logging via an enviromnet variable
  * Different logging targets (stderr, syslog, debugger...)
  * Annotation of your sourcecode about known bugs, things to do, etc.
  * Tracking Resources (Files, Locks, etc.) used by your programm; help in
    detecting misuse.

Building and Installing

Release Tarballs

Release tarballs are attached to this wiki at:

  * http://pipawiki/NoBug?action=AttachFile

I am using gpg signed tarballs for distribution. As first step one has to check
the signature

$ gpg nobug-X.Y.tar.gz.gpg

This will produce a nobug-X.Y.tar.gz and report if the signature could be
validated.

Since they are built with gnu autotools, the usual build and install procedure
works:

$ tar xzvf nobug-X.Y.tar.gz
$ cd nobug-X.Y
$ ./configure
$ make
$ make install

Development Version via git

The devlopment version is available via git from 'git://git.pipapo.org/nobug'
or mirrored at repo.or.cz 'git://repo.or.cz/nobug.git'.

After you cloned the repository you need to bootstap the autotools first

$ autoreconf -i

Then the usual ./configure && make && make install will work.

There is a special makefile target make meta to bring serveral files (README,
AUTHORS, NEWS) in sync with the NoBug Documentation wiki and update the
ChangeLog.

What gets installed

Currently, NoBug installs the following:

  * A single nobug.h headerfile which your code will use
  * Two libs for static linking:
      + 'libnobug.a' for singlethreaded programs
      + 'libnobugmt.a' for multithreaded programs.

Using NoBug

You can make Nobug features available either by installing it as described
above or by shipping the nobug sources along with your project.

To use NoBug, some controling preprocessor macros have to be defined. The
nobug.h header should then be included.

A project using NoBug should use autoconf to check for execinfo.h and valgrind/
valgrind.h

AC_CHECK_HEADERS([execinfo.h valgrind/valgrind.h])

For Multithreaded programs, you should also check for pthread.h.

When the resulting HAVE_PTHREAD_H, HAVE_EXECINFO_H and HAVE_VALGRIND_VALGRIND_H
are defined by the configure script, the corresponding features become
available.

NoBug then defines 'NOBUG_USE_PTHREAD', 'NOBUG_USE_VALGRIND' and
'NOBUG_USE_EXECINFO' to 1. If you do not want to use any of these features in
NoBug, you can define these macros to 0 before including nobug.h.

If NVALGRIND is defined, valgrind support will not be used.

There are many other macros which can be set and overridden by the user to
control the behavior. Please edit the Documentation wiki when you find them
useful.

A program using NoBug should be linked against 'libnobug.a' if it is
singlethreaded or against 'libnobugmt.a' if it is multithreaded. The Library
must be initialized with NOBUG_INIT before using any other features or creating
threads. This is discussed in more detail at NoBug/Documentation/Macros/
MultiThreading.

Debug Control

Build Levels

Nobug uses the following different levels of checks:

  * ALPHA
      + for expensive testing and logging while developing the software
    BETA
      + for testers and users who want to try the software
    RELEASE
      + finished version for end users

Release builds remove all assertions, but some logging is kept. We make the
assumption that bugs which where not covered in alpha and beta builds will not
easily show up in releases because the assertions there where not sufficent.
Further, end users are not test bunnies and will not provide good bug reports
anyway. If there is a problem in a release build, try to investigate the cause
with a beta build from the same source.

To define a debug level for compilation, just use '-DEBUG_ALPHA' or
'-DEBUG_BETA' for bebug builds and use the standard '-DNDEBUG' for a release
build. Nobug will complain if the debug level has not been defined.

Scope Checks

The programmer can tag any Scope as UNCHECKED or CHECKED. In ALPHA and BETA
builds, a global UNCHECKED is implied. In RELEASE builds, UNCHECKED Scopes are
not allowed.

Test Matrix

Here is a table of which basic assertions gets checked on each level/scope
combination:

          ALPHA                          BETA                   RELEASE
UNCHECKED Preconditions, Postconditions, Preconditions,         compiling will
          Invariants                     Postconditions         abort
CHECKED   Preconditions, Postconditions  Preconditions

Macros

The NoBug interface is almost completely implemented in preprocessor macros.
This is needed because it uses the __FILE__ and __LINE__ macros for logging.
All of the flat namespace uppercase identifiers make it easily recognizeable
within source code too.

All macros are unconditionally available with NOBUG_ prefixed. For convenience,
there are also macros without this prefix as long as such a macro was not
already defined. For each Assertion and Logging macro, there is a form with the
suffix _DBG which will be only active under a debugger. The _DBG versions are
only enabled in alpha builds.

When NOBUG_DISABLE_SHORTNAMES is defined by the user, then only the NOBUG_
prefixed macros are available and the short ones will never be defined.

There are also assertions with an _IF suffix taking a 'when' parameter as first
argument. The following is an example:

  * REQUIRE_IF(foo!=NULL, foo->something == constrained)

The assertion will only be performed when when is true. The debugger versions
are available as _IF_DBG prefixed macros.

Parameters types:

when    Assertion is only performed if expression 'when' is true at runtime
expr    Test without side effects
fmt     printf like formatstring
...     If not preceeded by 'fmt', then printf like formatstring. Otherwise,
        only its arguments
flag    Flag for enabling custom logging groups
type    Type of the data to be checked as a single identifier name
pointer Pointer to type
lvl     Log level
depth   Depth for invariants and dumps

Initialization

Global init

You have to call

  * NOBUG_INIT

before using any other NoBug feature. Probably in main or any other library
initialization routine. Calling NOBUG_INIT more than once is supported and each
subsequent call will be a no-op, thus initialization in main and in libraries
won't interfere.

Control Flags

If you want to use environment variable controlled debuging, then you have to
initialize each flag with

  * NOBUG_INIT_FLAG(flagname)

or

  * NOBUG_INIT_FLAG_LIMIT(flagname, default)

This is documented later in NoBug/Documentation/Macros/LoggingConfiguration.

  * <!> These two macros call NOBUG_INIT for backward compatibility.

Threads

In Multithreaded programs you should assign an identifier to each thread after
it is created with

  * NOBUG_THREAD_ID_SET(name)

If you don't call it, then NoBug will assign a automatic identifier. This is
documented in NoBug/Documentation/Macros/MultiThreading.

Assertions

  * REQUIRE(expr, ...)
      + Precondition (input) check. Use these macros to validate input a
        function recieves. The checks are enabled in ALPHA and BETA builds and
        optimized out in RELEASE builds.

    ENSURE(expr, ...)
      + Postcondition (progress/output) check. Use these macros to validate the
        data a function produces (example: return value). The checks enabled
        unconditionally in ALPHA builds and optimized out in BETA builds for
        scopes which are tagged as CHECKED. In RELEASE builds this checks are
        always optimized out, but scopes tagged as UNCHECKED are not permitted.

    ASSERT(expr, ...)
      + Generic check. Use these macros when you want to validate something
        which doesn't fall into one of the above categories. A example is when
        a library function can return a unexpected result (scanf with syntax
        error in the formatstring, when a constant/literal formatstring is
        expected). The checks are enabled in ALPHA and BETA builds and
        optimized out in RELEASE builds.

    assert(expr)
      + Nobug overrides the standard assert macro in ALPHA and BETA builds.
        This is just a compatibility feature, its use is not suggested.

    INVARIANT(type, pointer, depth)
      + Checking invariants. You can provide more complex checking functions
        which test the validity of datastructures. Invariants are only enabled
        in ALPHA builds for scopes which are not tagged as CHECKED and
        otherwise optimized out. (TODO: document how to write invariants)

Logging

Nearly all NoBug Macros emit some log message. NoBug gives the user fine
grained control over these log messages to display only interesting information
without loosing details.

Log messages are routed to different destinations which are:

  * RINGBUFFER
      + The underlying storage backend. Messages are appended to the end of the
        buffer, overwriting older messages at the front of the buffer. NoBug
        comes with a highly efficent ringbuffer implementation.
  * CONSOLE
      + This is either just stderr or if running under valgrind then valgrind
        facilities to include messages into its log will be used.
  * FILE
      + The user can open Files for log messages.
  * SYSLOG
      + Messages are send to the standard system logging daemon.
  * APPLICATION
      + There is a interface which allows the programmer to catch logmessages
        and display them in a application defined way.

Each logmessage has a priority descibing its severity in the same way as syslog
messages do.

All non-fatal messages are associated with a programmer defined flag describing
the source of the message (subsystem, module, ...).

Putting it all together: A user can define which source/flag shall be logged at
what priority level and to which destination. To make this all easier NoBug
tries to give reasonable defaults.

Configuration

Log Levels

Each Log macro has a explicit or implicit Log-Level which correspondends to
syslog levels. Logging is only emitted when the messages is more servere than a
defined limit.

The defaults look like:

            ALPHA BETA    RELEASE
ringbuffer  TRACE INFO    NOTICE  ringbuffer must always be most verbose
console     INFO  NOTICE  -1      no log to console in release
file        TRACE NOTICE  WARNING
syslog      -1    NOTICE  WARNING no syslog for test runs
application INFO  WARNING ERROR

Depending on the build level there is a default logging target and a default
limit which is choosen when the user doesn't specify one.

The default limits are:

  * In ALPHA builds, NOBUG_LOG_LIMIT_ALPHA is used which defaults to LOG_INFO
  * In BETA builds, NOBUG_LOG_LIMIT_BETA is used and defaults to LOG_WARNING
  * In RELEASE builds, NOBUG_LOG_LIMIT_RELEASE is used and defaults to LOG_CRIT

The default targets are:

  * In ALPHA builds, NOBUG_LOG_TARGET_ALPHA is used which defaults to
    NOBUG_TARGET_CONSOLE
  * In BETA builds, NOBUG_LOG_TARGET_BETA is used and defaults to
    NOBUG_TARGET_FILE
  * In RELEASE builds, NOBUG_LOG_TARGET_RELEASE is used and defaults to
    NOBUG_TARGET_SYSLOG

You can override all of those values with your own preference. Alternatively
NOBUG_LOG_LIMIT and NOBUG_LOG_TARGET can be defined by the user to override all
defaults.

Log Flags

Flags are used to tell NoBug about subsystems/modules or even finer grained
parts of the code.

A Flag should be declared with

  * NOBUG_DECLARE_FLAG(flagname)

preferably in one of your headers

Further it must be defined with

  * NOBUG_DEFINE_FLAG(flagname)

or

  * NOBUG_DEFINE_FLAG_LIMIT(flagname, limit)

in one of your source files

Next you should call

  * NOBUG_INIT_FLAG(flagname)

or

  * NOBUG_INIT_FLAG_LIMIT(flagname, default)

once at the start of your program for every flag.

For flags defined with NOBUG_DEFINE_FLAG(flagname) the defaults are initialized
as in the table above, while NOBUG_DEFINE_FLAG_LIMIT(flagname, level) is used
to initialize the default target (depending on build level) to limit. Calling
NOBUG_INIT_FLAG(flagname) is optional for limit initialized flags.

NOBUG_INIT_FLAG(flagname) parses the environment variable '$NOBUG_LOG' for
flagname, the syntax is as following:

logdecl_list --> logdecl, any( ',' logdecl_list).

logdecl --> flag, opt(limitdecl, any(targetdecl))

flag --> "identifier of a flag"

limitdecl --> ':', "LIMITNAME"

targetdecl --> '@', "targetname", opt(targetopts)

targetopts --> '(', "options for target", ')', opt(targetopts)

Roughly speaking, NOBUG_LOG contains a comma separated list of declarations for
flags which are the name of the flag followed by a limit which is written in
all uppercase letters and preceeded by a colon, followed by target declarations
which are names of the targets, introduced by a at sign. Target declarations
can have options in future which is not yet implemented. Limit and target
declarations are optional and then choosen from the defaults table above. These
defaults are currently just an guess what should be useable and might be
redefined in future.

<!> NOTE: '(options)' on targets are not yet implemented!

Examples:

NOBUG_LOG='flag,other'                        # set the limit of the default target a default limit (see table above)
NOBUG_LOG='flag:DEBUG'                        # set the limit of the default target to DEBUG
NOBUG_LOG='flag:DEBUG@console@syslog'         # set console and syslog limits for flag to DEBUG
NOBUG_LOG='flag:DEBUG,other:TRACE@ringbuffer'

There is a predefined flag NOBUG_ON which is always enabled.

Example code:

   1 #include "nobug.h"
   2 NOBUG_DEFINE_FLAG (test);
   3
   4 int
   5 main()
   6 {
   7   NOBUG_INIT_FLAG (test);
   8
   9   INFO (test, "Logging enabled");
  10   INFO (NOBUG_ON, "Always on");
  11 }

test it:

$ tcc -DEBUG_ALPHA -run example.c
example.c:10: debug: INFO: main: Always on

$ NOBUG_LOG=test tcc -DEBUG_ALPHA -run example.c
example.c:9: debug: INFO: main: Logging enabled
example.c:10: debug: INFO: main: Always on

The Logging Macros

These macros log with implicit Log Level, note that there are no more servere
levels than LOG_ERROR since these should be handled by Assertions in a
debugging library.

  * ERROR(flag, fmt, ...)
    ERROR_IF(expr, rflag, fmt, ...)
      + Application takes a error handling brach

    WARN(flag, fmt, ...)
    WARN_IF(expr, flag, fmt, ...)
      + Rare, handled but unexpected branch

    INFO(flag, fmt, ...)
    INFO_IF(expr, flag, fmt, ...)
      + Message about program progress

    NOTICE(flag, fmt, ...)
    NOTICE_IF(expr, flag, fmt, ...)
      + More detailed

    TRACE(flag, fmt, ...)
    TRACE_IF(expr, flag, fmt, ...)
      + Very fine grained messages

Note that TRACE correspondends to LOG_DEBUG, using 'DEBUG' could be ambiguous.

There is one generic LOG macro which takes the level explicitly:

  * LOG(flag, lvl, fmt, ...)
    LOG_IF(expr, flag, lvl, fmt, ...)

Dumping Datastructures

  * DUMP(flag, type, pointer, depth)
      + Dump a datastructure
    DUMP_IF(expr, flag, type, pointer, depth)
      + Dump datastructure if expr is true

How to write DUMP handlers

if you have a

   1 struct STRUCTNAME
   2 {
   3   int INTEGER_MEMBER;
   4   char * STRING_MEMBER;
   5   struct STRUCTNAME* next;
   6 }

then you define a function like:

   1 void
   2 nobug_STRUCTNAME_dump (const struct STRUCTNAME* self,
   3                        const int depth,
   4                        const char* file,
   5                        const int line,
   6                        const char* func)
   7 {
   8   // check for self != NULL and that the depth
   9   // limit did not exceed in recursive datastructures
  10   if (self && depth)
  11   {
  12     // use DUMP_LOG not LOG to print the data
  13     DUMP_LOG("STRUCTNAME %p: int is %d, string is %s", self,
  14                              self->INTEGER_MEMBER,
  15                              self->STRING_MEMBER);
  16     // now recurse with decremented depth
  17     nobug_STRUCTNAME_dump (self->next, depth-1, file, line, func);
  18   }
  19 }

now you can use the DUMP() macros within the code

   1 example()
   2 {
   3   struct STRUCTNAME foo;
   4   init(&foo);
   5   DUMP (my_flag, STRUCTNAME, &foo, 2);
   6 }

Dumping is by default done on level LOG_DEBUG, this can be overridden by
defining NOBUG_DUMP_LEVEL to some other level, further DUMP_IF is the only
enabled dumping macro for the RELEASE build level.

Source Annotations

One can tagging features as:

  * DEPRECATED(...)
      + Something which shouldn't be used in future.
    UNIMPLEMENTED(...)
      + important, not yet finished feature
    PLANNED(...)
      + planned for future feature
    FIXME(...)
      + known bug to be fixed later
    TODO(...)
      + enhancement to be done soon
    NOTREACHED
      + used to tag code-path which shall be never executed (defaults in switch
        statements, else NOTREACHED after series of if's)

The advantage of this tagging over plain source comments is that we can take
some actions if we run in such a tag at compile or runtime:

the action to be taken when such a macro is hit depends on the build level:

              ALPHA BETA         RELEASE
DEPRECATED    log   nothing      wont compile
UNIMPLEMENTED abort abort        wont compile
FIXME         log   wont compile wont compile
TODO          log   log          wont compile
PLANNED       log   nothing      nothing
NOTREACHED    abort abort        removed

Legend:

  * abort means first log and then abort
  * log will only log once for each sourceline (not on each hit)
  * wont compile will abort compilation with a error message
  * nothing optimized out, sane way
  * removed optimized out for performance reasons

Resource Tracking

<!> This part is work in progress, interfaces may change

With little effort, NoBug can watch all kinds of resources a program uses. This
becomes useful for resources which are distributed over a potentially
multithreaded program. Resource tracking is only active in ALPHA builds and
optimized out in BETA and RELEASE builds.

Concepts

Resources are abstracted, NoBug doesn't know anything about the Semantics of a
resource, it only keeps records of resources and the code using it and ensures
basic constraints. Detailed use checks of resource have to be done with other
NoBug facilities.

Resources are identified by c-strings, there are some optimizations to make
this very efficent. They can be announced by a 'type' and 'name' pair from
which NoBug creates (copies) a 'type:name' identifier or by supplying a string
which must be available through the entire lifetime of the resource, NoBug will
then only reference this string.

Code which wants to use a resource calls a enter macro with its own identifier
and a leave macro when finished with it.

The macros

Certain operations could use a 'handle' which will improve performance
considerably. These handles are defined with

  * RESOURCE_HANDLE(name)

This macro takes care that the declaration is optimized out in the same manner
the rest of the resource tracker would be disabled. You can still instantiate
handles as struct nobug_resource_record* in structures which must have a
constant size unconditional of the build level. It is important to note that
macros which take the handle as last optional parameter must be given as
pointer to pointer with &handle, while when using it as target for a macro as
first parameter its used just normally as pointer.

Resources are published by

  * RESOURCE_ANNOUNCE(type, name[, &handle])

or

  * RESOURCE_ANNOUNCE_LITERAL(identifier[, &handle])

Resources must be unique, it is a fatal error when a resource it tried to be
announced more than one time.

  * 'type' is a cstring which should denote the domain of the resource, the
    type "nobug" is reserved for internal use. Other strings are freely
    chooseable, examples are "file" "mutex" "lock" "database" and so on.
  * 'name' is the actual name of a named resource this is a cstring which
    together with type forms a unique identifier of the resource. 'type' and
    'name' are assembled to a new string seperated by a colon and thus can be
    volatile strings. Example 'type' = "file" and 'name' = "/etc/passwd" will
    create a identifier "file:/etc/passwd" for this resource.
  * 'identifier' must be a static string which is available throughout the
    lifetime of the resource, it will only be referenced, the implementation is
    little more efficent this way, but you are more constrained about the name.
  * 'handle' is a pointer to a nobug_resource_record* which will be initialized
    to point to the newly created resource. If it is feasible to store this
    pointer somewhere, where clients can access it then the enter/leave below
    can be done more efficent because the resource record doesnt need to be
    looked up.

When a resource becomes unavailable it is removed from the registry by

  * RESOURCE_FORGET(type, name[, &handle])

or

  * RESOURCE_FORGET_LITERAL(identifier[, &handle])

The resource must still exist and no users must be attacched to it, else a
fatal error is raised. The parameters are the same here as for
RESOURCE_ANNOUNCE above. 'handle' will be set to NULL and should be provided
here when used while announcing because the nulling has to be done in a lock
when using multithreading.

Code which wants to use a resource attaches to it by

  * RESOURCE_ENTER(type, name, client[, handle])

or

  * RESOURCE_ENTER_LITERAL(identifier, clientid[, handle])

or

  * RESOURCE_ENTER_HANDLE(handle, user[, handle])

When finished with using the resource you dissconnect from it with

  * RESOURCE_LEAVE(type, name, user[, handle])

or

  * RESOURCE_LEAVE_LITERAL(name, user[, handle])

or

  * RESOURCE_LEAVE_HANDLE(handle, user[, handle])

There is special support for resources which might block during acquisition
(blocking IO, locks, ...). Such resources are entered just before you actually
do the call to the resource and after you aquired it you call

  * RESOURCE_ACQUIRED_HANDLE(handle)

This just attaches the string "acquired" to the resource record which will be
shown on dumps.

Just a simple example:

   1 void example()
   2 {
   3   // define a mutex and announce it
   4   pthread_mutex_t my_mutex = PTHREAD_MUTEX_INITIALIZER;
   5   RESOURCE_ANNOUNCE_NAME("mutex", "my_mutex");
   6
   7   // the following would be actually done in a different thread in a real program
   8   RESOURCE_HANDLE(e);                                      // define a handle
   9   RESOURCE_ENTER_NAME("mutex", "my_mutex", "example", &e); // announce that we want to use the resource
  10   pthread_mutex_lock (&my_mutex);                          // this might block
  11   RESOURCE_ACQUIRED_HANDLE(e1);                            // we got it, now announce that
  12
  13   // program does something useful here
  14
  15   RESOURCE_LEAVE_NAME("mutex", "my_mutex", "example");     // announce that we dont need it anymore
  16   pthread_mutex_lock (&my_mutex);                          // and instantly unlock
  17
  18
  19   // back in the main thread
  20   RESOURCE_FORGET_NAME("mutex", "my_mutex");               // remove the resource from the public registry
  21 }

Notes

The Resource Tracker is neither bullet-proof nor exact in the current state.
There are small race conditions in the time we announce/forget/enter/remove
resources and doing the actual call to a resource. These race conditions affect
the reporting exactness in certain corner cases and are a design decision, for
practical use they should be no danger. More important is that the Resource
Tracker relies on that announce/forget and enter/leave are properly paired. The
programmer should ensure that this is done right, else it will become
unuseable. This will be fixed in future NoBug versions.

The underlying resource-tracking library may report errors, all these errors
are fatal for NoBug and trigger an abort(). When such an error occurs the
Resource Tracker is left in a locked state, which is harmless for generating
reports prior the abort.

Multithreading

It is important that NoBug protects certain operations with locks in
multithreaded programs. You have to ensure that 'HAVE_PTHREAD_H' is defined by
the configuration system and use the 'libnobugmt.a' library for linking. It is
particular important that libraries using NoBug are compiled with
'HAVE_PTHREAD_H' enabled when they are intended to be used in multithreaded
programs.

When Multithreading is used, log messages contain a identifier of the
orginating thread. This identifier should be set by

  * NOBUG_THREAD_ID_SET(name)

right after thread creation. name is a string describing the purpose of the
thread. Nobug will assemble a unique identifier by appending a underscore and a
number to name, for example NOBUG_THREAD_ID_SET("gui") will result in a
identifier like "gui_5". When you don't set a thread identifier, then NoBug
assigns one automatically with the name 'thread' preprended if needed. Thread
identifiers are immutable once set.

For querying the thread identifier you use

  * NOBUG_THREAD_ID_GET

which will return a const char* to the thread id in multithreaded programms and
a pointer to a literal "" in singlethreaded programs.

Tool Macros

  * BACKTRACE(...)
      + Log a stacktrace
    ABORT
      + just calls abort()

Best Practices

<!> this section is very work in progress

Workflow

 1. Development
      + Write a testsuite, build your program with -O0 -g -DEBUG_ALPHA and run
        the testsuite under valgrind control. Hack until the program fits the
        requirements defined by the testsuite.
 2. Beta Test
      + Build with desired optimization level and -g -DEBUG_BETA and give the
        program to your beta testers.
 3. Release
      + Just build it with optimization and without -g -DEBUG_*

What and when to check

  * Add REQUIRE checks on your interfaces (incoming parameters). Especially if
    a argument might not cover the whole range of the underlying type.
  * Don't waste your and your CPU's time with unessesary checks. The testsuite
    should validate your program. NoBug aids in debugging. You can add
    Postconditions (ENSURE) and Invariants when you have a bug somewhere and
    want to nail it down.
  * Added checks don't need to be removed.
  * When you use the CHECKED/UNCHECKED features then don't forget C scoping
    rules, tag things as CHECKED from the leaves to the root.

Tips & Tricks

  * TRACE(flagname) or TRACE_DBG(flagname) at the begin of every nontrivial
    function will easily log the progress of your application.
  * Trying a RELEASE build will abort on certain conditions (known BUG, TODO's,
    UNCHECKED code), you can use this to find these spots.

This Documentation is maintained at:

  * http://www.pipapo.org/pipawiki/NoBug/Documentation

NoBug/Documentation (last edited 2007-03-26 14:51:39 by ct)