Add doc/testrunner.md, for documenting the testrunner.tcl script.

[sqlite.git] / doc / testrunner.md

# The testrunner.tcl Script

# 1. Overview

testrunner.tcl is a Tcl script used to run multiple SQLite tests using 
multiple jobs. It supports the following types of tests:

  *  Tcl test scripts.

  *  Tests run with [make] commands. Specifically, at time of writing, 
     [make fuzztest], [make mptest], [make sourcetest] and [make threadtest].

testrunner.tcl pipes the output of all tests and builds run into log file
**testrunner.log**, created in the cwd directory. Searching this file for
"failed" is a good way to find the output of a failed test.

testrunner.tcl also populates SQLite database **testrunner.db**. This database
contains details of all tests run, running and to be run. A useful query
might be:

```
  SELECT * FROM script WHERE state='failed'
```

Running the command:

```
  ./testfixture $(TESTDIR)/testrunner.tcl status
```

in the directory containing the testrunner.db database runs various queries
to produce a succinct report on the state of a running testrunner.tcl script.
Running:

```
  watch ./testfixture $(TESTDIR)/testrunner.tcl status
```

in another terminal is a good way to keep an eye on a long running test.

Sometimes testrunner.tcl uses the [testfixture] binary that it is run with
to run tests (see "Binary Tests" below). Sometimes it builds testfixture and
other binaries in specific configurations to test (see "Source Tests").

# 2. Binary Tests

The commands described in this section all run various combinations of the Tcl
test scripts using the [testfixture] binary used to run the testrunner.tcl
script (i.e. they do not invoke the compiler to build new binaries, or the
[make] command to run tests that are not Tcl scripts). The procedure to run
these tests is therefore:

  1. Build the "testfixture" (or "testfixture.exe" for windows) binary using
     whatever method seems convenient.

  2. Test the binary built in step 1 by running testrunner.tcl with it, 
     perhaps with various options.

The following sub-sections describe the various options that can be
passed to testrunner.tcl to test binary testfixture builds.

## 2.1. Organization of Tcl Tests

Tcl tests are stored in files that match the pattern *\*.test*. They are
found in both the $TOP/test/ directory, and in the various sub-directories
of the $TOP/ext/ directory of the source tree. Not all *\*.test* files
contain Tcl tests - a handful are Tcl scripts designed to invoke other
*\*.test* files.

The **veryquick** set of tests is a subset of all Tcl test scripts in the
source tree. In includes most tests, but excludes some that are very slow.
Almost all fault-injection tests (those that test the response of the library
to OOM or IO errors) are excluded. It is defined in source file 
*test/permutations.test*.

The **full** set of tests includes all Tcl test scripts in the source tree.
To run a "full" test is to run all Tcl test scripts that can be found in the
source tree.

File *permutations.test* defines various test "permutations". A permutation
consists of:

  *  A subset of Tcl test scripts, and 

  *  Runtime configuration to apply before running each test script 
     (e.g. enabling auto-vacuum, or disable lookaside).

Running **all** tests is to run all tests in the full test set, plus a dozen
or so permutations. The specific permutations that are run as part of "all"
are defined in file *testrunner_data.tcl*.

## 2.2. Commands to Run Tests

To run the "veryquick" test set, use either of the following:

```
  ./testfixture $TESTDIR/testrunner.tcl
  ./testfixture $TESTDIR/testrunner.tcl veryquick
```

To run the "full" test suite:

```
  ./testfixture $TESTDIR/testrunner.tcl full
```

To run the subset of the "full" test suite for which the test file name matches
a specified pattern (e.g. all tests that start with "fts5"), either of:

```
  ./testfixture $TESTDIR/testrunner.tcl fts5%
  ./testfixture $TESTDIR/testrunner.tcl 'fts5*'
```

To run "all" tests (full + permutations):

```
  ./testfixture $TESTDIR/testrunner.tcl all
```

<a name=binary_test_failures></a>
## 2.3. Investigating Binary Test Failures

If a test fails, testrunner.tcl reports name of the Tcl test script and, if
applicable, the name of the permutation, to stdout. This information can also
be retrieved from either *testrunner.log* or *testrunner.db*.

If there is no permutation, the individual test script may be run with:

```
  ./testfixture $PATH_TO_SCRIPT
```

Or, if the failure occured as part of a permutation:

```
  ./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
```

TODO: An example instead of "$PERMUTATION" and $PATH\_TO\_SCRIPT?

# 3. Source Code Tests

The commands described in this section invoke the C compiler to build 
binaries from the source tree, then use those binaries to run Tcl and
other tests. The advantages of this are that:

  *  it is possible to test multiple build configurations with a single
     command, and 

  *  it ensures that tests are always run using binaries created with the
     same set of compiler options.

The testrunner.tcl commands described in this section may be run using
either a *testfixture* (or testfixture.exe) build, or with any other Tcl
shell that supports SQLite 3.31.1 or newer via "package require sqlite3".

TODO: ./configure + Makefile.msc build systems.

## Commands to Run SQLite Tests

The **mdevtest** command is equivalent to running the veryquick tests and
the [make fuzztest] target once for each of two --enable-all builds - one 
with debugging enabled and one without:

```
  tclsh $TESTDIR/testrunner.tcl mdevtest
```

In other words, it is equivalent to running:

```
  $TOP/configure --enable-all --enable-debug
  make fuzztest
  make testfixture
  ./testfixture $TOP/test/testrunner.tcl veryquick

  # Then, after removing files created by the tests above:
  $TOP/configure --enable-all OPTS="-O0"
  make fuzztest
  make testfixture
  ./testfixture $TOP/test/testrunner.tcl veryquick
```

The **sdevtest** command is identical to the mdevtest command, except that the
second of the two builds is a sanitizer build. Specifically, this means that
OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":

```
  tclsh $TESTDIR/testrunner.tcl sdevtest
```

The **release** command runs lots of tests under lots of builds. It runs
different combinations of builds and tests depending on whether it is run
on Linux, Windows or OSX. Refer to *testrunner\_data.tcl* for the details
of the specific tests run.

```
  tclsh $TESTDIR/testrunner.tcl release
```

## Running ZipVFS Tests

testrunner.tcl can build a zipvfs-enabled testfixture and use it to run
tests from the Zipvfs project with the following command:

```
  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
```

This can be combined with any of "mdevtest", "sdevtest" or "release" to
test both SQLite and Zipvfs with a single command:

```
  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
```

## Investigating Source Code Test Failures

Investigating a test failure that occurs during source code testing is a
two step process:

  1. Recreating the build configuration in which the test failed, and

  2. Re-running the actual test.

To recreate a build configuration, use the testrunner.tcl **script** command
to create a build script. A build script is a bash script on Linux or OSX, or
a dos \*.bat file on windows. For example:

```
  # Create a script that recreates build configuration "Device-One" on 
  # Linux or OSX:
  tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh 

  # Create a script that recreates build configuration "Have-Not" on Windows:
  tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat 
```

The generated bash or \*.bat file script accepts a single argument - a makefile
target to build. This may be used either to run a [make] command test directly,
or else to build a testfixture (or testfixture.exe) binary with which to
run a Tcl test script, as <a href=#binary_test_failures>described above</a>.



# 4. Controlling CPU Core Utilization

When running either binary or source code tests, testrunner.tcl reports the
number of jobs it intends to use to stdout. e.g.

```
  $ ./testfixture $TESTDIR/testrunner.tcl
  splitting work across 16 jobs
  ... more output ...
```

By default, testfixture.tcl attempts to set the number of jobs to the number 
of real cores on the machine. This can be overridden using the "--jobs" (or -j)
switch:

```
  $ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
  splitting work across 8 jobs
  ... more output ...
```

The number of jobs may also be changed while an instance of testrunner.tcl is
running by exucuting the following command from the directory containing the
testrunner.log and testrunner.db files:

```
  $ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
```