ctest_harness

ctest_harness is a small and portable unit test framework for C.

Features

• Easy to integrate

  Having only one header file and source file, integrating it into your build system is trivial.

• Small, Portable
  • C99 compliant
  • Self-contained, no external dependencies
  • Easy to compile and uses POSIX make
  • Should compile without warnings using -Wall -Wextra -Wpedantic
  • No dynamic allocation.

• Timing information

  Provides wall clock timing information to help diagnose performance regressions.

• Test suites

  Organize your tests into test suites.

• Handy assertion macros

  Assertions are provided for testing equality and inequality of integral data types, strings and double floating point.

• Pseudo-random number generator

  32-bit PRNG based on xoroshiro64** used for seed generation and to aid the creation of randomized tests.

Getting Started

ctest_harness is designed to be easy to use. Just add harness.c to your sources, include harness.h, and you're good to go. Here's a basic example

#include "harness.h"

static int x_should_equal_1(void *user_data) {
    int x = 1;

    ASSERT_EQ(1, x);

    return HARNESS_PASS;
}

int main(void) {
    struct harness_cfg cfg = {0};
    int ret = EXIT_SUCCESS;

    if (harness_init(&cfg) == HARNESS_FAIL) {
        return EXIT_FAILURE;
    }

    (void)harness_add_test("my_suite", "x_should_equal_1",x_should_equal_1, NULL, NULL, NULL);
    ret = harness_run(NULL);
    harness_destroy(&cfg);

    return ret;
}

Assertions

Assertions are a fundamental part of testing. ctest_harness provides two levels of assertions: (1) fatal and (2) non-fatal. Fatal assertions stop the execution of the test resulting in a FAIL verdict, while non-fatal assertions as the name implies don't stop the execution.

Let's say you want to test two values for equality:

int your_test_function(void *user_data) {
    (void)user_data;

    int a = 5;
    int b = 10;

    ASSERT_EQ(a, b);

    return HARNESS_PASS;
}

ERROR> srcfile.c:6: assertion failed: a == b

The non-fatal variants are prefixed with EXPECT_ instead of ASSERT_.

Pseudo-Random numbers

Being able to randomize tests is a great way to increase coverage of your tests. The drawback of randomized tests is reproducibility: a test failed, but if it's randomized then how do I reproduce it? That's where seeding comes into play, by initializing the PRNG with the same seed you can reproduce the exact same scenario. Every time the test harness runs an unsigned 32-bit seed is generated and printed in hexadecimal notation so that you can later use it to re-run the tests with the same output. If you want to set a specific seed you can do that by setting the seed field in the struct harness_cfg.

The API is the following:

void random_init(void);

Initialize the state with the value of time(0) is used.

uint32_t random_next(void);

Generate a 32-bit random value

uint32_t random_bounded(uint32_t range);

Generate a 32-bit random value in the interval [0, range)

float random_float(void);

Generate a random float in the interval [0, 1)

Setting up the test harness

Before adding and executing test cases you need to initilize the test harness. This is done by first defining a variable of type struct harness_cfg and calling harness_init:

struct harness_cfg cfg = {
    .setup = setup,
    .teardown = teardown,
    .iterations = 1, /* if not set or set to 0, it defaults to 1 */
};

harness_init(&cfg);

This is also the place where we have to define a global fixture setup and teardown, they must follow the following prototype (of course the name can be changed):

int setup_function(void *user_data);
void teardown_function(void *user_data);

If they are not needed they can be set to NULL.

Once we are done with our tests we have to destroy the test harness:

harness_destroy(&cfg);

Tests and Suites

Let's look at how to structure testcases for ctest_harness. Each testcase should be a separate function with the following prototype:

int my_test(void *user_data);

Of course the name of the test does not matter it's only for your internal use and convenience.

The possible return values of a test case are four:

It is recommended that each test scenario should be implemented in a different function.

Adding a test case

Once you have defined a test case, or more than one, it's time to add them to a test suite. There are two ways you can do that: you can either add them one by one using the harness_add_test function:

harness_add_test(
    "my_suite", /* suite name */
    "my_test",  /* test name */
    my_test,    /* test function */
    NULL,       /* fixture setup */
    NULL,       /* fixture teardown */
    NULL        /* user data */
    );

or you can add them by declaring an array of struct harness_test and then feeding that array to harness_add_suite:

struct harness_test tests[] = {
    {"my_test", my_test, NULL, NULL, NULL},
    {NULL, NULL, NULL, NULL, NULL}
};

harness_add_suite("my_suite", tests);

Note that the NULL entry at the end is used to tell the test runner when the array is over.

Fixture setup and teardown

ctest_harness has two levels of fixtures: global and test specific fixtures. If global setup and teardown functions are provided to the harness_cfg structure, the setup function will be executed before all tests are run. Similarly if a global teardown function is provided it will be executed at the end of all the test cases. Test specific fixtures will be executed before (setup) and after (teardown) each test case.

These functions are commonly used to initialize and destroy resources used by all the test cases or by a specific test. For example:

static int setup(void *user_data) {
    char *buf = (char *)user_data;

    buf = malloc(1024);
    if (buf == NULL) {
        return (HARNESS_ERROR);
    }

    return (HARNESS_PASS);
}

static void teardown(void *user_data) {
    char *buf = (char *)user_data;
    free(buf);
}

static int test_setup(void *user_data) {
    char *buf = (char *)user_data;
    FILE *fp = fopen("file.txt", "r");
    if (fp == NULL) {
        return (HARNESS_ERROR);
    }

    fgets(buf, 1024, fp);
    fclose(fp);

    return (HARNESS_SUCCESS);
}

static void test_teardown(void *user_data) {
    (void)user_data;
}

static int my_test(void *user_data) {
    char* buf = (char *)user_data;
    ASSERT_STREQ(buf, "Hello, world!");

    return HARNESS_PASS;
}

int main(void) {
    struct harness_cfg cfg = {
        .setup = setup,
        .teardown = teardown,
    };
    char *buf = 0;

    harness_init(&cfg);
    harness_add_test("my_suite", "my_test", my_test, test_setup, test_teardown, buf);
}

The return value of the test specific setup function will be checked before executing the test case itself, if a value different from HARNESS_SUCCESS is returned then the execution will stop without calling the actual test function.

Running the tests

Once you've added your tests and suites, it's time to call harness_run, its prototype is

int harness_run(void *user_data);

The return value will be EXIT_SUCCESS if all tests pass,or EXIT_FAILURE if any test has failed. This makes it particularly suitable for returning directly from your main() function.

User data

Up until now we've skipped talking about the user_data parameters, but it's time to address it.

If a global setup and/or teardown function has been provided, the user_data parameter passed to harness_run is passed to the setup and/or teardown fixtures.

If a test specific setup and/or teardown function has been provided, the user_data field of the struct harness_test is passed to the setup and/or teardown fixtures as well as the test function.

Static analysis

Static analysis on the code base is done by using clang's static analyzer run through scan-build.sh which wraps the scan-build utility. The checkers used are part of the Experimental Checkers (aka alpha checkers):

• alpha.security
• alpha.core.CastSize
• alpha.core.CastToStruct
• alpha.core.IdenticalExpr
• alpha.core.PointerArithm
• alpha.core.PointerSub
• alpha.core.SizeofPtr
• alpha.core.TestAfterDivZero
• alpha.unix

License

BSD 2-Clause FreeBSD License, see LICENSE.