1 Rearchitecting ATF: Road trip to 1.0
3 By Julio Merino, The NetBSD Foundation
13 4. The plan to 1.0 prereleases
15 5. The plan to 1.0 release candidates
17 6. The plan to releases after 1.0
21 This document describes the major steps to be taken before getting to the
22 first stable release of ATF, 1.0. Some of these steps describe the
23 transition from the old code base to the new ideas documented in the
24 specification (TODO: Add link somehow).
28 * Add atf-sh interface to build-time tests.
30 * Properly document the libraries: i.e. one page per module, detailed
31 information of each function and type, etc. At the very least atf-c.
33 * Add a tool to collect multiple outputs of atf-run (from different
34 machines) and generate a single (XML-only?) log with everything. Must
35 allow easy conversion to HTML for online publishing.
37 * Allow grouping of test programs in tiers in an Atffile. This is to
38 permit the user specify "dependencies" between test programs: e.g. do
39 not run a specific test program if its dependencies have failed,
40 because it will certainly fail also.
42 * Provide a kernel-level unit testing API (for NetBSD only, at the
43 moment). This should come in the form of an atf.ko module that
44 provides functions to define and register test cases, functions for
45 results reporting and an interface (a trivial file system?) that
46 transports the application/X-atf-tcs output to user-space, provides
47 information to user-space about the available test cases (a list) and
48 allows user-space to launch the execution of test cases.
52 * Add a module to atf-c to manage dynamic memory. Should provide a "mem
53 chunk" object that can only be managed through functions (i.e. not
54 directly) so that access to buffers can be safely controlled. Dealing
55 with strdup and similar functions, for example, might be complex.
57 See these old revisions for a start, but these did not work correctly
58 because the use of (void **) casts brought aliasing problems:
60 78eeacf3b9284493e5e96b7866fc1d163a13bc02
61 8e200683a2b70eadca728c2e0347d4790777b3fc
62 872393ed0177cbcc30ffacede228eb3986c42ab7
64 The plan to 1.0 prereleases
66 * Fix all occurrences of XXX, TODO and FIXME.
68 * Split the distfile into multiple components. We should have a
69 component for each language binding and a component providing the ATF
70 tools, at the very least. If we had this, external programs using ATF
71 wouldn't need to depend on the tools and/or the C++ binding, because
72 they could just require the user to build the atf-c binding.
74 * Think of a way to properly add tests for (almost?) all error paths.
75 Most of them are probably broken already.
77 * Improve error reporting: aside from clarifying error messages, this
78 also implies adding more error cases to give them more semantic
79 meaning at the source code level..
81 * Make the shell library work with 'set -e'?
83 * Shell test programs dynamically locate where the shell library is by
84 calling atf-config (done by atf.init.subr). Contrarywise, binary test
85 programs are directly linked against the final location of libatf. It
86 may be nice if the latter loaded the library dynamically based on what
87 atf-config said, so the user could switch atf installations by simply
88 changing its PATH (and effectively making atf relocatable on the file
89 system). Why could this be nice? To painlessly run an older atf test
90 suite against a more recent version of the code base to ensure there
91 are no regressions even with older tests. Just a crazy idea, as maybe
92 what the shell test programs currently do is stupid.
94 * Allow users to customize the build of atf by defining additional
95 meta-data for test cases. At the moment this is possible because the
96 meta-data is not sanity-checked, but I think it should be. Following
97 the previous item, NetBSD could add a 'netbsd.pr' variable and then
98 use this data when generating reports to add direct links to the
101 * Make sure that the tests in tests/atf have, at the very least, the
102 same coverage as the ones in tests/bootstrap.
106 * Possibly add a way to automatically gain or drop privileges when
109 * Add a way to specify which bug/issue/whatever a given test case is
110 stress-testing. This information is useful when detecting regressions.
112 The plan to 1.0 release candidates
114 * Build libatf as a shared library and set -version-info accordingly.
116 * Set the DTDs' versions to 1.0.
118 The plan to releases after 1.0
120 * Allow the parallel execution of tests. Achieving this with a test
121 program granularity is easy: only need to change atf-run. Lowering it
122 to a finer granularity (test cases) is harder and maybe not worth it.