| blob | 2e8596c2817d7b570a9d942cf5d25aa816f0ae59 |
1 .\"
2 .\" Automated Testing Framework (atf)
3 .\"
4 .\" Copyright (c) 2007, 2008 The NetBSD Foundation, Inc.
5 .\" All rights reserved.
6 .\"
7 .\" Redistribution and use in source and binary forms, with or without
8 .\" modification, are permitted provided that the following conditions
9 .\" are met:
10 .\" 1. Redistributions of source code must retain the above copyright
11 .\" notice, this list of conditions and the following disclaimer.
12 .\" 2. Redistributions in binary form must reproduce the above copyright
13 .\" notice, this list of conditions and the following disclaimer in the
14 .\" documentation and/or other materials provided with the distribution.
15 .\"
16 .\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
17 .\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
18 .\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
19 .\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
20 .\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
21 .\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
22 .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
23 .\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
24 .\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
25 .\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
26 .\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
27 .\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
28 .\"
29 .Dd November 4, 2007
30 .Dt ATF-FORMATS 5
31 .Os
32 .Sh NAME
33 .Nm atf-formats
34 .Nd machine-parseable data formats used by ATF
35 .Sh DESCRIPTION
36 This manual page describes the multiple data formats used in ATF.
37 These formats affect configuration files, control files and any data that
38 is externalized or internalized by the tools.
39 .Pp
40 Data files are always organized as follows:
41 .Bd -literal -offset indent
42 Header1: Value1 \\
43 ... | head
44 HeaderN: ValueN /
45 mandatory blank line
46 Free-form text contents \\
47 ... | body
48 ... /
49 .Ed
50 .Pp
51 A file must always contain a
52 .Sq Content-Type
53 header and must always separate that header from the body with a blank
54 line, even if the body is empty.
55 .Pp
56 The
57 .Sq Content-Type
58 is always of the form:
59 .Bd -literal -offset indent
60 Content-Type: application/X-atf-<subtype>; version="<version>"
61 .Ed
62 .Pp
63 where
64 .Sq subtype
65 indicates the specific file format and
66 .Sq version
67 its format version.
68 This header must be the first one of the file.
69 .Pp
70 The main purpose of the
71 .Sq Content-Type
72 header, aside from determining the format used in the file, is to allow
73 future changes to a given format.
74 Whenever an incompatible change is made, the version is bumped by one.
75 By keeping the header in the first line, future versions may even remove
76 the need for such a header -- e.g. if some format was replaced by XML
77 files, which have their own mandatory header.
78 .Pp
79 The rest of this document details the different format types.
80 .Ss Format: application/X-atf-atffile, version: 1
81 Atffiles are logically divided into three sections:
82 .Bl -bullet
83 .It
84 Test programs: the list of test programs that define the test suite
85 described by the Atffile.
86 .It
87 Meta-data properties: these define some constant values applicable to
88 all the test programs defined in the file.
89 In some sense they define the properties that describe the test suite.
90 .It
91 Configuration variables: defaults for configuration variables that
92 can be overridden through configuration files or the command line.
93 .El
94 .Pp
95 The grammar for Atffiles is the following:
96 .Bd -literal -offset indent
97 DATA ::= ( ( CONF | PROP | TP )? COMMENT? NEWLINE )* EOF
98 CONF ::= 'conf:' WORD EQUAL STRING
99 PROP ::= 'prop:' WORD EQUAL STRING
100 TP ::= TPFILE | TPGLOB
101 TPFILE ::= 'tp: ' STRING
102 TPGLOB ::= 'tp-glob: ' STRING
103 STRING ::= WORD | '"' WORD* '"'
104 .Ed
105 .Pp
106 The meaning of the constructions above is:
107 .Bl -tag -width TPGLOBXX
108 .It CONF
109 Definition of a configuration variable.
110 .It PROP
111 Definition of a meta-data property.
112 .It TPFILE
113 Addition of a test program into the test suite.
114 The string is taken literally as the program's name, and this program
115 must exist.
116 .It TPGLOB
117 Addition of multiple test programs into the test suite.
118 The string is taken as a glob pattern, which may have or not have any
119 matches in the current directory.
120 .El
121 .Pp
122 An example:
123 .Bd -literal -offset indent
124 prop: test-suite = utilities
126 conf: unprivileged-user = nobody
128 tp: t_cp
129 tp: t_mv
130 tp: t_df
131 tp-glob: t_dir_*
132 .Ed
133 .Ss Format: application/X-atf-config, version: 1
134 Configuration files are very simple: they only contain a list of variable
135 name/variable value pairs.
136 Their grammar is:
137 .Bd -literal -offset indent
138 DATA ::= ( VAR? COMMENT? NEWLINE )* EOF
139 VAR ::= WORD EQUAL STRING
140 COMMENT ::= HASH WORD*
141 STRING ::= WORD | '"' WORD* '"'
142 .Ed
143 .Pp
144 An example:
145 .Bd -literal -offset indent
146 # This is the system-wide configuration file for ATF.
147 # The above and this line are comments placed on their own line.
149 var1 = this is a variable value
150 var2 = this is another one # Optional comment at the end.
151 .Ed
152 .Ss Format: application/X-atf-tcs, version: 1
153 The
154 .Sq application/X-atf-tcs
155 format is used to describe the results of a collection of test cases;
156 in other words, it represents
157 .Em the output of a test program .
158 Unfortunately, it is not easy to control, from inside a test program, what
159 it prints to both its standard output and standard error streams.
160 This is specially the case of test programs written in the POSIX shell
161 language, because they are constantly executing external tools that may
162 print unexpected messages at all times.
163 Due to this, ATF imposes no restrictions on what a test program can send to
164 these two channels; in fact, they are encouraged to print as much useful
165 information as possible to aid in the debugging of test failures.
166 .Pp
167 Because we have no control over the two standard streams, the
168 .Sq application/X-atf-tcs
169 format describes the structure of a third stream, known as the
170 .Em results output ,
171 that test programs must generate.
172 (Note that test programs send, by default, the results output to the
173 standard output; use their
174 .Fl r
175 flag to change this whenever you need to parse the data.)
176 This stream is decoupled from the other two and has the following grammar:
177 .Bd -literal -offset indent
178 DATA ::= TCS-COUNT TC-STANZA* EOF
179 TCS-COUNT ::= 'tcs-count' COLON POSITIVE-NUMBER NEWLINE
180 TC-STANZA ::= TC-START TC-END
181 TC-START ::= 'tc-start' COLON STRING NEWLINE
182 TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
183 TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
184 .Ed
185 .Pp
186 The meaning of the constructions above is:
187 .Bl -tag -width TCSXCOUNTXX
188 .It TCS-COUNT
189 Indicates the number of test cases that will be executed.
190 There will be this exact amount of
191 .Sq TC-STANZA
192 constructions following it.
193 .It TC-START
194 Indicates the beginning of a test case.
195 This is accompanied by the test case's name.
196 .It TC-END
197 Indicates the completion of a test case.
198 This is accompanied by the test case's name, its result and the reason
199 associated with this result (if applicable).
200 .El
201 .Pp
202 There are multiple reasons behind this design:
203 .Bl -bullet
204 .It
205 The reader of this format must be able to show real-time progress to the
206 user as the test cases are processed.
207 Therefore, the
208 .Sq TC-START
209 construction tells the reader
210 .Em when
211 a test case has started to process data.
212 .It
213 The reader of this format has to be able to provide useful statistics to
214 the user without having to wait for the end of the file.
215 Hence, the existence of the
216 .Sq TCS-COUNT
217 construction located at the beginning of the file.
218 .It
219 Text-based tools have to be able to easily look for the results of a given
220 test case.
221 This is why the
222 .Sq TC-END
223 construction duplicate the test case name already provided in
224 .Sq TC-START .
225 .El
226 .Pp
227 An example:
228 .Bd -literal -offset indent
229 tcs-count: 2
230 tc-start: add
231 tc-end: add, passed
232 tc-start: subtract
233 tc-end: subtract, failed, Calculated an unexpected value
234 .Ed
235 .Pp
236 Going back to the standard output and standard error streams, the reader
237 has to be able to match the messages in those two streams to the test cases
238 they belong to.
239 To do this, these two streams must print a magic string that separates the
240 output of test cases from each other, which is enough to synchronize their
241 contents with the results output.
242 This string is
243 .Sq __atf_tc_separator__
244 and it must printed on a line of its own.
245 The last test case should not be followed by this line because the end of
246 file marker takes its role.
247 .Ss Format: application/X-atf-tps, version: 2
248 The
249 .Sq application/X-atf-tps
250 format multiplexes the standard output, standard error and results output
251 streams from multiple test programs into a single data file.
252 This format is used by
253 .Xr atf-run 1
254 to report the execution of several test programs and is later parsed by
255 .Xr atf-report 1
256 to inform the user of this process.
257 It has the following grammar:
258 .Bd -literal -offset indent
259 DATA ::= INFO* TPS-COUNT TP-STANZA* INFO* EOF
260 INFO ::= 'info' COLON STRING COMMA STRING NEWLINE
261 TPS-COUNT ::= 'tps-count' COLON POSITIVE-NUMBER NEWLINE
262 TP-STANZA ::= TP-START TC-STANZA* TC-END
263 TP-START ::= 'tp-start' COLON STRING COMMA POSITIVE-NUMBER NEWLINE
264 TP-END ::= 'tc-end' COLON STRING (COMMA STRING)?
265 TC-STANZA ::= TC-START (TC-SO | TC-SE)* TC-END
266 TC-START ::= 'tc-start' COLON STRING NEWLINE
267 TC-SO ::= 'tc-so' COLON STRING NEWLINE
268 TC-SE ::= 'tc-se' COLON STRING NEWLINE
269 TC-END ::= 'tc-end' COLON STRING COMMA TCR NEWLINE
270 TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
271 .Ed
272 .Pp
273 The meaning of the constructions above is:
274 .Bl -tag -width TPSXCOUNTXX
275 .It TPS-COUNT
276 Indicates the number of test programs that will be executed.
277 There will be this exact amount of
278 .Sq TP-STANZA
279 constructions following it.
280 .It TP-START
281 Indicates the beginning of a test program.
282 This includes the program's name and the amount of test cases that
283 will follow.
284 .It TP-END
285 Indicates the completion of a test program.
286 This is followed by the program's name and, if the program ended
287 prematurely, an error message indicating the reason of its failure.
288 A successful execution of a test program (regardless of the status of its
289 test cases) must not be accompanied by any reason.
290 .It TC-START
291 Indicates the beginning of a test case.
292 This is accompanied by the test case's name.
293 .It TC-SO
294 Contains a text line sent to the standard output stream during the
295 execution of the test case.
296 Leading and trailing space is preserved.
297 .It TC-SE
298 Contains a text line sent to the standard error stream during the
299 execution of the test case.
300 Leading and trailing space is preserved.
301 .It TC-END
302 Indicates the completion of a test case.
303 This is accompanied by the test case's name, its result and the reason
304 associated with this result (if applicable).
305 .El
306 .Pp
307 An example:
308 .Bd -literal -offset indent
309 tps-count: 2
310 tp-start: calculator, 2
311 tc-start: add
312 tc-end: add, passed
313 tc-start: subtract
314 tc-so: 3-2 expected to return 1 but got 0
315 tc-end: subtract, failed, Calculated an unexpected value
316 tp-end: calculator
317 tp-start: files, 1
318 tc-start: copy
319 tc-se: could not find the cp(1) utility
320 tc-end: copy, skipped
321 tp-end: files
322 .Ed
323 .Sh SEE ALSO
324 .Xr atf 7