| blob | 6b7f1b04a2ac09c1fa413fc089d17e0308dbd6c9 |
1 <section xmlns="http://docbook.org/ns/docbook" version="5.0"
2 xml:id="manual.intro.setup.test" xreflabel="Testing">
3 <?dbhtml filename="test.html"?>
5 <info><title>Testing</title>
6 <keywordset>
7 <keyword>ISO C++</keyword>
8 <keyword>test</keyword>
9 <keyword>testsuite</keyword>
10 <keyword>performance</keyword>
11 <keyword>conformance</keyword>
12 <keyword>ABI</keyword>
13 <keyword>exception safety</keyword>
14 </keywordset>
15 </info>
17 <para>
18 The libstdc++ testsuite includes testing for standard conformance,
19 regressions, ABI, and performance.
20 </para>
22 <section xml:id="test.organization" xreflabel="Test Organization"><info><title>Test Organization</title></info>
25 <section xml:id="test.organization.layout" xreflabel="Directory Layout"><info><title>Directory Layout</title></info>
28 <para>
29 The directory
30 <filename class="directory"><replaceable>gccsrcdir</replaceable>/libstdc++-v3/testsuite</filename>
31 contains the individual test cases organized in sub-directories
32 corresponding to clauses of the C++ standard (detailed below),
33 the DejaGnu test harness support files, and sources to various
34 testsuite utilities that are packaged in a separate testing library.
35 </para>
37 <para>
38 All test cases for functionality required by the runtime components
39 of the C++ standard (ISO 14882) are files within the following
40 directories:
42 <programlisting>
43 17_intro
44 18_support
45 19_diagnostics
46 20_util
47 21_strings
48 22_locale
49 23_containers
50 24_iterators
51 25_algorithms
52 26_numerics
53 27_io
54 28_regex
55 29_atomics
56 30_threads
57 </programlisting>
58 </para>
60 <para>
61 In addition, the following directories include test files:
63 <variablelist spacing="compact">
64 <varlistentry>
65 <term><filename class="directory">tr1</filename></term>
66 <listitem>Tests for components as described by the Technical Report
67 on Standard Library Extensions (<link linked="status.iso.tr1">TR1</link>).
68 </listitem>
69 </varlistentry>
70 <varlistentry>
71 <term><filename class="directory">backward</filename></term>
72 <listitem>Tests for backwards compatibility and deprecated features.
73 </listitem>
74 </varlistentry>
75 <varlistentry>
76 <term><filename class="directory">demangle</filename></term>
77 <listitem>Tests for <function>__cxa_demangle</function>, the IA-64 C++ ABI
78 demangler.
79 </listitem>
80 </varlistentry>
81 <varlistentry>
82 <term><filename class="directory">ext</filename></term>
83 <listitem>Tests for extensions.</listitem>
84 </varlistentry>
85 <varlistentry>
86 <term><filename class="directory">performance</filename></term>
87 <listitem>Tests for performance analysis, and performance regressions.
88 </listitem>
89 </varlistentry>
90 </variablelist>
91 </para>
93 <para>
94 Some directories don't have test files, but instead contain
95 auxiliary information:
97 <variablelist spacing="compact">
98 <varlistentry>
99 <term><filename class="directory">config</filename></term>
100 <listitem>Files for the DejaGnu test harness.</listitem>
101 </varlistentry>
102 <varlistentry>
103 <term><filename class="directory">lib</filename></term>
104 <listitem>Files for the DejaGnu test harness.</listitem>
105 </varlistentry>
106 <varlistentry>
107 <term><filename class="directory">libstdc++*</filename></term>
108 <listitem>Files for the DejaGnu test harness.</listitem>
109 </varlistentry>
110 <varlistentry>
111 <term><filename class="directory">data</filename></term>
112 <listitem>Sample text files for testing input and output.</listitem>
113 </varlistentry>
114 <varlistentry>
115 <term><filename class="directory">util</filename></term>
116 <listitem>Files for libtestc++, utilities and testing routines.</listitem>
117 </varlistentry>
118 </variablelist>
119 </para>
121 <para>
122 Within a directory that includes test files, there may be
123 additional subdirectories, or files. Originally, test cases
124 were appended to one file that represented a particular section
125 of the chapter under test, and was named accordingly. For
126 instance, to test items related to <code> 21.3.6.1 -
127 <function>basic_string::find</function> [lib.string::find]</code>
128 in the standard, the following was used:
129 <programlisting> 21_strings/find.cc </programlisting>
130 However, that practice soon became a liability as the test cases
131 became huge and unwieldy, and testing new or extended
132 functionality (like wide characters or named locales) became
133 frustrating, leading to aggressive pruning of test cases on some
134 platforms that covered up implementation errors. Now, the test
135 suite has a policy of one file, one test case, which solves the
136 above issues and gives finer grained results and more manageable
137 error debugging. As an example, the test case quoted above
138 becomes:
139 <programlisting> 21_strings/basic_string/find/char/1.cc
140 21_strings/basic_string/find/char/2.cc
141 21_strings/basic_string/find/char/3.cc
142 21_strings/basic_string/find/wchar_t/1.cc
143 21_strings/basic_string/find/wchar_t/2.cc
144 21_strings/basic_string/find/wchar_t/3.cc</programlisting>
145 </para>
147 <para>
148 All new tests should be written with the policy of "one test
149 case, one file" in mind.
150 </para>
151 </section>
154 <section xml:id="test.organization.naming" xreflabel="Naming Conventions"><info><title>Naming Conventions</title></info>
157 <para>
158 In addition, there are some special names and suffixes that are
159 used within the testsuite to designate particular kinds of
160 tests.
161 </para>
163 <variablelist>
164 <varlistentry>
165 <term><filename class="extension">_xin.cc</filename></term>
166 <listitem>
167 This test case expects some kind of interactive input in order
168 to finish or pass. At the moment, the interactive tests are not
169 run by default. Instead, they are run by hand, like:
170 <programlisting>
171 g++ 27_io/objects/char/3_xin.cc
172 cat 27_io/objects/char/3_xin.in | a.out</programlisting>
173 </listitem>
174 </varlistentry>
175 <varlistentry>
176 <term><filename class="extension">.in</filename></term>
177 <listitem>
178 This file contains the expected input for the corresponding <emphasis>
179 _xin.cc</emphasis> test case.
180 </listitem>
181 </varlistentry>
182 <varlistentry>
183 <term><filename class="extension">_neg.cc</filename></term>
184 <listitem>
185 This test case is expected to fail: it's a negative test. At the
186 moment, these are almost always compile time errors.
187 </listitem>
188 </varlistentry>
189 <varlistentry>
190 <term><filename class="directory">char</filename></term>
191 <listitem>
192 This can either be a directory name or part of a longer file
193 name, and indicates that this file, or the files within this
194 directory are testing the <code>char</code> instantiation of a
195 template.
196 </listitem>
197 </varlistentry>
198 <varlistentry>
199 <term><filename class="directory">wchar_t</filename></term>
200 <listitem>
201 This can either be a directory name or part of a longer file
202 name, and indicates that this file, or the files within this
203 directory are testing the <code>wchar_t</code> instantiation of
204 a template. Some hosts do not support <code>wchar_t</code>
205 functionality, so for these targets, all of these tests will not
206 be run.
207 </listitem>
208 </varlistentry>
209 <varlistentry>
210 <term><filename class="directory">thread</filename></term>
211 <listitem>
212 This can either be a directory name or part of a longer file
213 name, and indicates that this file, or the files within this
214 directory are testing situations where multiple threads are
215 being used.
216 </listitem>
217 </varlistentry>
218 <varlistentry>
219 <term><filename class="directory">performance</filename></term>
220 <listitem>
221 This can either be an enclosing directory name or part of a
222 specific file name. This indicates a test that is used to
223 analyze runtime performance, for performance regression testing,
224 or for other optimization related analysis. At the moment, these
225 test cases are not run by default.
226 </listitem>
227 </varlistentry>
228 </variablelist>
230 </section>
231 </section>
234 <section xml:id="test.run" xreflabel="Running the Testsuite"><info><title>Running the Testsuite</title></info>
237 <section xml:id="test.run.basic"><info><title>Basic</title></info>
240 <para>
241 You can check the status of the build without installing it
242 using the DejaGnu harness, much like the rest of the gcc
243 tools, i.e.
244 <userinput>make check</userinput>
245 in the
246 <filename class="directory"><replaceable>libbuilddir</replaceable></filename>
247 directory, or
248 <userinput>make check-target-libstdc++-v3</userinput>
249 in the
250 <filename class="directory"><replaceable>gccbuilddir</replaceable></filename>
251 directory.
252 </para>
254 <para>
255 These commands are functionally equivalent and will create a
256 '<filename class="directory">testsuite</filename>' directory underneath
257 <filename class="directory"><replaceable>libbuilddir</replaceable></filename>
258 containing the results of the
259 tests. Two results files will be generated:
260 <filename>libstdc++.sum</filename>, which is a PASS/FAIL summary
261 for each test, and
262 <filename>libstdc++.log</filename> which is a log of
263 the exact command-line passed to the compiler, the compiler
264 output, and the executable output (if any) for each test.
265 </para>
267 <para>
268 Archives of test results for various versions and platforms are
269 archived on a daily basis on the <link xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://gcc.gnu.org/ml/gcc-testresults/current">gcc-testresults</link>
270 mailing list. Please check there for a similar
271 combination of source version, operating system, and host CPU.
272 </para>
273 </section>
275 <section xml:id="test.run.variations"><info><title>Variations</title></info>
277 <para>
278 There are several options for running tests, including testing
279 the regression tests, testing a subset of the regression tests,
280 testing the performance tests, testing just compilation, testing
281 installed tools, etc. In addition, there is a special rule for
282 checking the exported symbols of the shared library.
283 </para>
284 <para>
285 To debug the DejaGnu test harness during runs, try invoking with a
286 specific argument to the variable <varname>RUNTESTFLAGS</varname>,
287 like so:
288 <programlisting>
289 make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
290 </programlisting>
291 or
292 <programlisting>
293 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
294 </programlisting>
295 </para>
297 <para>
298 To run a subset of the library tests, you can either generate the
299 <filename>testsuite_files</filename> file (described below) by running
300 <userinput>make testsuite_files</userinput> in the
301 <filename class="directory"><replaceable>libbuilddir</replaceable>/testsuite</filename>
302 directory, then edit the
303 file to remove the tests you don't want and then run the testsuite as
304 normal, or you can specify a testsuite and a subset of tests in the
305 <varname>RUNTESTFLAGS</varname> variable.
306 </para>
308 <para>
309 For example, to run only the tests for containers you could use:
311 <programlisting>
312 make check-target-libstdc++-v3 RUNTESTFLAGS="conformance.exp=23_containers/*"
313 </programlisting>
314 </para>
316 <para>
317 When combining this with other options in <varname>RUNTESTFLAGS</varname>
318 the <option>testsuite.exp=testfiles</option> options must come first.
319 </para>
321 <para>
322 There are two ways to run on a simulator: set up <envar>DEJAGNU</envar>
323 to point to a specially crafted <filename>site.exp</filename>,
324 or pass down <option>--target_board</option> flags.
325 </para>
327 <para>
328 Example flags to pass down for various embedded builds are as follows:
330 <programlisting>
331 --target=powerpc-eabisim <emphasis>(libgloss/sim)</emphasis>
332 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
334 --target=calmrisc32 <emphasis>(libgloss/sid)</emphasis>
335 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
337 --target=xscale-elf <emphasis>(newlib/sim)</emphasis>
338 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
339 </programlisting>
340 </para>
342 <para>
343 Also, here is an example of how to run the libstdc++ testsuite
344 for a multilibed build directory with different ABI settings:
346 <programlisting>
347 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
348 </programlisting>
349 </para>
351 <para>
352 If you wish to run the testsuite in a freestanding configuration, you can
353 pass the <code>-ffreestanding</code> flag. Doing so will run the tests
354 that do not require hosted features, and emit a
355 <literal>UNSUPPORTED</literal> for those that do. To run tests in the
356 freestanding configuration, you still need to build for a target you can
357 run programs on, e.g. <code>x86_64-pc-linux-gnu</code>, as a few tests
358 still execute the code they build. Here's an example of how to run the
359 testsuite with libstdc++ in freestanding mode:
360 <programlisting>
361 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board=unix/-ffreestanding'
362 </programlisting>
363 </para>
365 <para>
366 You can run the tests with a compiler and library that have
367 already been installed. Make sure that the compiler (e.g.,
368 <command>g++</command>) is in your <envar>PATH</envar>. If you are
369 using shared libraries, then you must also ensure that the
370 directory containing the shared version of libstdc++ is in your
371 <envar>LD_LIBRARY_PATH</envar>, or
372 <link linkend="manual.intro.using.linkage.dynamic">equivalent</link>.
373 If your GCC source tree is at
374 <filename class="directory">/path/to/gcc</filename>,
375 then you can run the tests as follows:
377 <programlisting>
378 runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
379 </programlisting>
380 </para>
382 <para>
383 The testsuite will create a number of files in the directory in
384 which you run this command,. Some of those files might use the
385 same name as files created by other testsuites (like the ones
386 for GCC and G++), so you should not try to run all the
387 testsuites in parallel from the same directory.
388 </para>
390 <para>
391 In addition, there are some testing options that are mostly of
392 interest to library maintainers and system integrators. As such,
393 these tests may not work on all CPU and host combinations, and
394 may need to be executed in the
395 <filename class="directory"><replaceable>libbuilddir</replaceable>/testsuite</filename>
396 directory. These
397 options include, but are not necessarily limited to, the
398 following:
399 </para>
401 <variablelist>
402 <varlistentry>
403 <term><userinput>
404 make testsuite_files
405 </userinput></term>>
407 <listitem>
408 <para>
409 Five files are generated that determine what test files
410 are run. These files are:
412 <variablelist>
413 <varlistentry>
414 <term> <filename>testsuite_files</filename> </term>
415 <listitem>
416 This is a list of all the test cases that will be run. Each
417 test case is on a separate line, given with an absolute path
418 from the
419 <filename class="directory"><replaceable>libsrcdir</replaceable>/testsuite</filename>
420 directory.
421 </listitem>
422 </varlistentry>
424 <varlistentry>
425 <term> <filename>testsuite_files_interactive</filename> </term>
426 <listitem>
427 This is a list of all the interactive test cases, using the
428 same format as the file list above. These tests are not run
429 by default.
430 </listitem>
431 </varlistentry>
433 <varlistentry>
434 <term> <filename>testsuite_files_performance</filename> </term>
435 <listitem>
436 This is a list of all the performance test cases, using the
437 same format as the file list above. These tests are not run
438 by default.
439 </listitem>
440 </varlistentry>
442 <varlistentry>
443 <term> <filename>testsuite_thread</filename> </term>
444 <listitem>
445 This file indicates that the host system can run tests which
446 involved multiple threads.
447 </listitem>
448 </varlistentry>
450 <varlistentry>
451 <term> <filename>testsuite_wchar_t</filename> </term>
452 <listitem>
453 This file indicates that the host system can run the
454 <code>wchar_t</code> tests, and corresponds to the macro
455 definition <literal>_GLIBCXX_USE_WCHAR_T</literal> in the
456 file <filename>c++config.h</filename>.
457 </listitem>
458 </varlistentry>
459 </variablelist>
460 </para>
461 </listitem>
462 </varlistentry>
464 <varlistentry>
465 <term><userinput>
466 make check-abi
467 </userinput></term>>
469 <listitem>
470 <para>
471 The library ABI can be tested. This involves testing the shared
472 library against a baseline list of symbol exports that defines the
473 previous version of the ABI. The tests require that no exported
474 symbols are removed, no new symbols are added to the old symbol
475 versions, and any new symbols have the latest symbol version.
476 See <link linkend="abi.versioning">Versioning</link> for more details
477 of the ABI version history.
478 </para>
479 </listitem>
480 </varlistentry>
482 <varlistentry>
483 <term><userinput>
484 make new-abi-baseline
485 </userinput></term>>
487 <listitem>
488 <para>
489 Generate a new baseline set of symbols exported from the library
490 (written to a file under
491 <filename class="directory"><replaceable>libsrcdir</replaceable>/config/abi/post/<replaceable>target</replaceable>/</filename>).
492 A different baseline symbols file is needed for each architecture and
493 is used by the <literal>check-abi</literal> target described above.
494 The files are usually re-generated by target maintainers for releases.
495 </para>
496 </listitem>
497 </varlistentry>
499 <varlistentry>
500 <term><userinput>
501 make check-compile
502 </userinput></term>>
504 <listitem>
505 <para>
506 This rule compiles, but does not link or execute, the
507 <filename>testsuite_files</filename> test cases and displays the
508 output on stdout.
509 </para>
510 </listitem>
511 </varlistentry>
513 <varlistentry>
514 <term><userinput>
515 make check-performance
516 </userinput></term>>
518 <listitem>
519 <para>
520 This rule runs through the
521 <filename>testsuite_files_performance</filename> test cases and
522 collects information for performance analysis and can be used to
523 spot performance regressions. Various timing information is
524 collected, as well as number of hard page faults, and memory
525 used. This is not run by default, and the implementation is in
526 flux.
527 </para>
528 </listitem>
529 </varlistentry>
531 <varlistentry>
532 <term><userinput>
533 make check-debug
534 </userinput></term>>
536 <listitem>
537 <para>
538 This rule runs through the test suite under the
539 <link linkend="manual.ext.debug_mode">debug mode</link>.
540 </para>
541 </listitem>
542 </varlistentry>
544 <varlistentry>
545 <term><userinput>
546 make check-parallel
547 </userinput></term>>
549 <listitem>
550 <para>
551 This rule runs through the test suite under the
552 <link linkend="manual.ext.parallel_mode">parallel mode</link>.
553 </para>
554 </listitem>
555 </varlistentry>
557 </variablelist>
559 <para>
560 We are interested in any strange failures of the testsuite;
561 please email the main libstdc++ mailing list if you see
562 something odd or have questions.
563 </para>
564 </section>
566 <section xml:id="test.run.permutations"><info><title>Permutations</title></info>
568 <para>
569 The tests will be compiled with a set of default compiler flags defined
570 by the
571 <filename><replaceable>libbuilddir</replaceable>/scripts/testsuite_flags</filename>
572 file, as well as options specified in individual tests. You can run
573 the tests with different options by adding them to the output of
574 the <option>--cxxflags</option> option of that script, or by setting
575 the <varname>CXXFLAGS</varname> variable when running
576 <command>make</command>, or via options for the DejaGnu test framework
577 (described below). The latter approach uses the
578 <option>--target_board</option> option that was shown earlier,
579 but requires DejaGnu version 1.5.3 or newer to work reliably, so that the
580 <literal>dg-options</literal> in the test aren't overridden.
581 For example, to run the tests with
582 <option>-O1 -D_GLIBCXX_ASSERTIONS</option>
583 you could use:
584 <programlisting> make check RUNTESTFLAGS=--target_board=unix/-O1/-D_GLIBCXX_ASSERTIONS</programlisting>
585 </para>
587 <para>
588 The <option>--target_board</option> option can also be used to run the
589 tests multiple times in different variations. For example, to run the
590 entire testsuite three times using <option>-O3</option> but with
591 different <option>-std</option> options:
592 <programlisting> make check 'RUNTESTFLAGS=--target_board=unix/-O3\"{-std=gnu++98,-std=gnu++11,-std=gnu++14}\"'</programlisting>
593 N.B. that set of variations could also be written as
594 <literal>unix/-O3\"{-std=gnu++98,-std=gnu++11,}\"</literal> so that
595 the third variation would use the default for <option>-std</option>
596 (which is <option>-std=gnu++14</option> as of GCC 6).
597 </para>
599 <para>
600 Since GCC 14, the libstdc++ testsuite has built-in support for running
601 tests with more than one <option>-std</option>, similar to the G++ tests.
602 Adding <code>set v3_std_list { 11 17 23 }</code> to
603 <filename>~/.dejagnurc</filename> or to a file named by the
604 <envar>DEJAGNU</envar> environment variable will cause every test to
605 be run three times, using a different <option>-std</option> each time.
606 Alternatively, a list of standard versions to test with can be specified
607 as a comma-separated list in the <envar>GLIBCXX_TESTSUITE_STDS</envar>
608 environment variable, e.g. <envar>GLIBCXX_TESTSUITE_STDS=11,17,23</envar>
609 is equivalent to the <code>v3_std_list</code> value above.
610 </para>
612 <para>
613 To run the libstdc++ test suite under the
614 <link linkend="manual.ext.debug_mode">debug mode</link>, use
615 <userinput>make check-debug</userinput>. Alternatively, edit
616 <filename><replaceable>libbuilddir</replaceable>/scripts/testsuite_flags</filename>
617 to add the compile-time flag <option>-D_GLIBCXX_DEBUG</option> to the
618 result printed by the <option>--cxxflags</option>
619 option. Additionally, add the
620 <option>-D_GLIBCXX_DEBUG_PEDANTIC</option> flag to turn on
621 pedantic checking. The libstdc++ test suite should produce
622 the same results under debug mode that it does under release mode:
623 any deviation indicates an error in either the library or the test suite.
624 Note, however, that the number of tests that PASS may change, because
625 some test cases are skipped in normal mode, and some are skipped in
626 debug mode, as determined by the
627 <literal>dg-require-<replaceable>support</replaceable></literal>
628 directives described below.
629 </para>
631 <para>
632 The <link linkend="manual.ext.parallel_mode">parallel
633 mode</link> can be tested using
634 <userinput>make check-parallel</userinput>, or in much the same manner
635 as the debug mode, substituting
636 <option>-D_GLIBCXX_PARALLEL</option> for
637 <option>-D_GLIBCXX_DEBUG</option> in the previous paragraph.
638 </para>
640 <para>
641 Or, just run the testsuite
642 <option>-D_GLIBCXX_DEBUG</option> or <option>-D_GLIBCXX_PARALLEL</option>
643 in <varname>CXXFLAGS</varname> or <varname>RUNTESTFLAGS</varname>.
644 </para>
645 </section>
646 </section>
648 <section xml:id="test.new_tests"><info><title>Writing a new test case</title></info>
651 <para>
652 The first step in making a new test case is to choose the correct
653 directory and file name, given the organization as previously
654 described.
655 </para>
657 <para>
658 Historically all test files were copyright the FSF, and GPL licensed.
659 We no longer require that, because most tests are uninteresting
660 and contain no "original authorship", and so would not be protected
661 by copyright anyway.
662 If you do want to add the FSF copyright notice and GPL licence text,
663 then the first copyright year should correspond to the date
664 the file was checked in to version control. If a test is copied from
665 an existing file it should retain the copyright years from the
666 original file.
667 </para>
669 <para>
670 The DejaGnu instructions say to always return <literal>0</literal>
671 from <function>main</function> to indicate success. Strictly speaking
672 this is redundant in C++, since returning from <function>main</function>
673 is defined to return <literal>0</literal>. Most tests still have an
674 explicit return.
675 </para>
677 <para>
678 A bunch of utility functions and classes have already been
679 abstracted out into the testsuite utility library, <code>
680 libtestc++</code>. To use this functionality, just include the
681 appropriate header file: the library or specific object files will
682 automatically be linked in as part of the testsuite run.
683 </para>
685 <para>
686 Tests that need to perform runtime checks should use the
687 <literal>VERIFY</literal> macro, defined in the
688 <filename class="headerfile"><testsuite_hooks.h></filename> header.
689 This expands to a custom assertion using
690 <function>__builtin_printf</function> and
691 <function>__builtin_abort</function>
692 (to avoid using <literal>assert</literal> and being affected by
693 <literal>NDEBUG</literal>).
694 </para>
696 <para>
697 Prior to GCC 7.1, <literal>VERIFY</literal> was defined differently.
698 It usually expanded to the standard <literal>assert</literal> macro, but
699 allowed targets to define it to something different. In order to support
700 the alternative expansions of <literal>VERIFY</literal>, before any use
701 of the macro there needed to be a variable called <varname>test</varname>
702 in scope, which was usually defined like so (the attribute avoids
703 warnings about an unused variable):
704 <programlisting>
705 bool test __attribute__((unused)) = true;
706 </programlisting>
707 This is no longer needed, and should not be added to new tests.
708 </para>
710 <para>
711 The testsuite uses the DejaGnu framework to compile and run the tests.
712 Test cases are normal C++ files which contain special directives in
713 comments. These directives look like <literal>{ dg-* ... }</literal>
714 and tell DejaGnu what to do and what kinds of behavior are to be expected
715 for a test. The core DejaGnu directives are documented in the
716 <filename>dg.exp</filename> file installed by DejaGnu.
717 The GCC testsuites support additional directives
718 as described in the GCC internals documentation, see <link
719 xmlns:xlink="http://www.w3.org/1999/xlink"
720 xlink:href="https://gcc.gnu.org/onlinedocs/gccint/Directives.html">Syntax
721 and Descriptions of test directives</link>. GCC also defines many <link
722 xmlns:xlink="http://www.w3.org/1999/xlink"
723 xlink:href="https://gcc.gnu.org/onlinedocs/gccint/Effective-Target-Keywords.html">
724 Keywords describing target attributes</link> (a.k.a effective targets)
725 which can be used where a target <replaceable>selector</replaceable> can
726 appear.
727 </para>
729 <para>
730 Some directives commonly used in the libstdc++ testsuite are:
732 <variablelist>
733 <varlistentry>
734 <term><literal>{ dg-do <replaceable>do-what-keyword</replaceable> [{ target/xfail <replaceable>selector</replaceable> }] }</literal></term>
735 <listitem>Where <replaceable>do-what-keyword</replaceable> is usually
736 one of <literal>run</literal> (which is the default),
737 <literal>compile</literal>, or <literal>link</literal>,
738 and typical selectors are targets such as <literal>*-*-gnu*</literal>
739 or an effective target such as <literal>c++11</literal>.
740 </listitem>
741 </varlistentry>
742 <varlistentry>
743 <term><literal>{ dg-require-<replaceable>support</replaceable> args }</literal></term>
744 <listitem>Skip the test if the target does not provide the required support.
745 See below for values of <replaceable>support</replaceable>.
746 </listitem>
747 </varlistentry>
748 <varlistentry>
749 <term><literal>{ dg-options <replaceable>options</replaceable> [{ target <replaceable>selector</replaceable> }] }</literal></term>
750 </varlistentry>
751 <varlistentry>
752 <term><literal>{ dg-error <replaceable>regexp</replaceable> [ <replaceable>comment</replaceable> [{ target/xfail <replaceable>selector</replaceable> } [<replaceable>line</replaceable>] ]] }</literal></term>
753 </varlistentry>
754 <varlistentry>
755 <term><literal>{ dg-excess-errors <replaceable>comment</replaceable> [{ target/xfail <replaceable>selector</replaceable> }] }</literal></term>
756 </varlistentry>
757 </variablelist>
758 For full details of these and other directives see the main GCC DejaGnu
759 documentation in the internals manual.
760 </para>
762 <para>
763 Test cases that use features of a particular C++ standard should specify
764 the minimum required standard as an effective target:
765 <programlisting> // { dg-do run { target c++11 } }</programlisting>
766 or
767 <programlisting> // { dg-require-effective-target c++11 }</programlisting>
768 Specifying the minimum required standard for a test allows it to be run
769 using later standards, so that we can verify that C++11 components still
770 work correctly when compiled as C++14 or later. Specifying a minimum also
771 means the test will be skipped if the test is compiled using
772 an older standard, e.g. using
773 <option>RUNTESTFLAGS=--target_board=unix/-std=gnu++98</option>.
774 </para>
776 <para>
777 It is possible to indicate that a test should <emphasis>only</emphasis>
778 be run for a specific standard (and not later standards) using an
779 effective target like <literal>c++11_only</literal>. However, this means
780 the test will be skipped by default unless <option>-std=gnu++11</option>
781 or <option>-std=c++11</option> is explicitly specified, either via a
782 target board, the <varname>v3_std_list</varname> dejagnu variable,
783 or the <envar>GLIBCXX_TESTSUITE_STDS</envar> environment variable.
784 For tests that require a specific standard it is useful to also add a
785 <literal>dg-options</literal> directive:
786 <programlisting> // { dg-options "-std=gnu++11" }</programlisting>
787 This means the test will not get skipped by default, and will always use
788 the specific standard dialect that the test requires. This isn't needed
789 often, and most tests should use an effective target to specify a
790 minimum standard instead, to allow them to be tested for all
791 possible variations.
792 </para>
794 <para>
795 N.B. when a <literal>dg-options</literal> directive is used, it must come
796 first so dejagnu will include those options when checking against any
797 effective targets in <literal>dg-do</literal> and
798 <literal>dg-require-effective-target</literal> directives.
799 </para>
801 <para>
802 Since GCC 14, tests which depend on a newer standard than the default
803 do not need to specify that standard in a <literal>dg-options</literal>
804 directive. The testsuite will detect when a test requires a newer standard
805 and will automatically add a suitable <option>-std</option> flag.
806 </para>
808 <para>
809 If a testcase requires the use of a strict language dialect, e.g.
810 <option>-std=c++11</option> rather than <option>-std=gnu++11</option>,
811 the following directive will cause that to be used when the testsuite
812 decides which <option>-std</option> options to use for the test:
813 <programlisting> // { dg-add-options strict_std }</programlisting>
814 </para>
816 <section xml:id="tests.dg.examples"><info><title>Examples of Test Directives</title></info>
818 <para>
819 Example 1: Testing compilation only:
820 <programlisting>
821 // { dg-do compile }
822 </programlisting>
824 Example 2: Testing for expected warnings on line 36, which all targets fail:
825 <programlisting>
826 // { dg-warning "string literals" "" { xfail *-*-* } 36 }
827 </programlisting>
829 Example 3: Testing for expected warnings on line 36:
830 <programlisting>
831 // { dg-warning "string literals" "" { target *-*-* } 36 }
832 </programlisting>
834 Example 4: Testing for compilation errors on line 41:
835 <programlisting>
836 // { dg-do compile }
837 // { dg-error "no match for" "" { target *-*-* } 41 }
838 </programlisting>
840 Example 5: Testing with special command line settings, or without the
841 use of pre-compiled headers, in particular the
842 <filename class="headerfile">stdc++.h.gch</filename> file. Any
843 options here will override the <varname>DEFAULT_CXXFLAGS</varname> and
844 <varname>PCH_CXXFLAGS</varname> set up in the <filename>normal.exp</filename>
845 file:
846 <programlisting>
847 // { dg-options "-O0" { target *-*-* } }
848 </programlisting>
850 Example 6: Compiling and linking a test only for C++14 and later, and only
851 if Debug Mode is active:
852 <programlisting>
853 // { dg-do link { target c++14 } }
854 // { dg-require-debug-mode "" }
855 </programlisting>
857 Example 7: Running a test only on x86 targets, and only for C++11 and later,
858 with specific options, and additional options for 32-bit x86:
859 <programlisting>
860 // { dg-options "-fstrict-enums" }
861 // { dg-additional-options "-march=i486" { target ia32 } }
862 // { dg-do run { target { ia32 || x86_64-*-* } } }
863 // { dg-require-effective-target "c++11" }
864 </programlisting>
865 </para>
867 <para>
868 More examples can be found in the
869 <filename>libstdc++-v3/testsuite/*/*.cc</filename> files.
870 </para>
871 </section>
873 <section xml:id="tests.dg.directives"><info><title>Directives Specific to Libstdc++ Tests</title></info>
875 <para>
876 In addition to the usual <link
877 xmlns:xlink="http://www.w3.org/1999/xlink"
878 xlink:href="https://gcc.gnu.org/onlinedocs/gccint/Require-Support.html">Variants
879 of <literal>dg-require-<replaceable>support</replaceable></literal></link>
880 several more directives are available for use in libstdc++ tests,
881 including the following:
882 </para>
884 <variablelist>
885 <varlistentry><term><literal>dg-require-namedlocale</literal> <replaceable>name</replaceable></term>
886 <listitem><para>The named locale must be available.
887 </para></listitem>
888 </varlistentry>
889 <varlistentry><term><literal>dg-require-debug-mode ""</literal></term>
890 <listitem><para>Skip the test if the Debug Mode is not active
891 (as determined by the <literal>_GLIBCXX_DEBUG</literal> macro).
892 </para></listitem>
893 </varlistentry>
894 <varlistentry><term><literal>dg-require-parallel-mode ""</literal></term>
895 <listitem><para>Skip the test if the Parallel Mode is not active
896 (as determined by the <literal>_GLIBCXX_PARALLEL</literal> macro).
897 </para></listitem>
898 </varlistentry>
899 <varlistentry><term><literal>dg-require-normal-mode ""</literal></term>
900 <listitem><para>Skip the test if Debug or Parallel Mode is active.
901 </para></listitem>
902 </varlistentry>
903 <varlistentry><term><literal>dg-require-atomic-builtins ""</literal></term>
904 <listitem><para>Skip the test if atomic operations on <type>bool</type>
905 and <type>int</type> are not lock-free.
906 </para></listitem>
907 </varlistentry>
908 <varlistentry><term><literal>dg-require-gthreads ""</literal></term>
909 <listitem><para>Skip the test if the C++11 thread library is not
910 supported, as determined by the <literal>_GLIBCXX_HAS_GTHREADS</literal>
911 macro.
912 </para></listitem>
913 </varlistentry>
914 <varlistentry><term><literal>dg-require-gthreads-timed ""</literal></term>
915 <listitem><para>Skip the test if C++11 timed mutexes are not supported,
916 as determined by the <literal>_GLIBCXX_HAS_GTHREADS</literal> and
917 <literal>_GTHREAD_USE_MUTEX_TIMEDLOCK</literal> macros.
918 </para></listitem>
919 </varlistentry>
920 <varlistentry><term><literal>dg-require-string-conversions ""</literal></term>
921 <listitem><para>Skip the test if the C++11 <function>to_string</function>
922 and <function>stoi</function>, <function>stod</function> etc. functions
923 are not fully supported (including wide character versions).
924 </para></listitem>
925 </varlistentry>
926 <varlistentry><term><literal>dg-require-filesystem-ts ""</literal></term>
927 <listitem><para>Skip the test if the Filesystem TS is not supported.
928 </para></listitem>
929 </varlistentry>
930 </variablelist>
931 </section>
933 </section>
936 <section xml:id="test.harness" xreflabel="Test Harness and Utilities"><info><title>Test Harness and Utilities</title></info>
939 <section xml:id="test.harness.dejagnu"><info><title>DejaGnu Harness Details</title></info>
941 <para>
942 Underlying details of testing for conformance and regressions are
943 abstracted via the GNU DejaGnu package. This is similar to the
944 rest of GCC.
945 </para>
948 <para>This is information for those looking at making changes to the testsuite
949 structure, and/or needing to trace DejaGnu's actions with
950 <option>--verbose</option>.
951 This will not be useful to people who are "merely" adding new tests
952 to the existing structure.
953 </para>
955 <para>The first key point when working with DejaGnu is the idea of a "tool".
956 Files, directories, and functions are all implicitly used when they are
957 named after the tool in use. Here, the tool will always be "libstdc++".
958 </para>
960 <para>The <code>lib</code> subdir contains support routines. The
961 <code>lib/libstdc++.exp</code> file ("support library") is loaded
962 automagically, and must explicitly load the others. For example, files can
963 be copied from the core compiler's support directory into <code>lib</code>.
964 </para>
966 <para>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are
967 our own. Callbacks must be prefixed with the name of the tool. To easily
968 distinguish the others, by convention our own routines are named "v3-*".
969 </para>
971 <para>The next key point when working with DejaGnu is "test files". Any
972 directory whose name starts with the tool name will be searched for test files.
973 (We have only one.) In those directories, any <code>.exp</code> file is
974 considered a test file, and will be run in turn. Our main test file is called
975 <code>normal.exp</code>; it runs all the tests in testsuite_files using the
976 callbacks loaded from the support library.
977 </para>
979 <para>The <code>config</code> directory is searched for any particular "target
980 board" information unique to this library. This is currently unused and sets
981 only default variables.
982 </para>
984 </section>
986 <section xml:id="test.harness.utils"><info><title>Utilities</title></info>
988 <para>
989 </para>
990 <para>
991 The testsuite directory also contains some files that implement
992 functionality that is intended to make writing test cases easier,
993 or to avoid duplication, or to provide error checking in a way that
994 is consistent across platforms and test harnesses. A stand-alone
995 executable, called <emphasis>abi_check</emphasis>, and a static
996 library called <emphasis>libtestc++</emphasis> are
997 constructed. Both of these items are not installed, and only used
998 during testing.
999 </para>
1001 <para>
1002 These files include the following functionality:
1003 </para>
1005 <itemizedlist>
1006 <listitem>
1007 <para>
1008 <emphasis>testsuite_abi.h</emphasis>,
1009 <emphasis>testsuite_abi.cc</emphasis>,
1010 <emphasis>testsuite_abi_check.cc</emphasis>
1011 </para>
1012 <para>
1013 Creates the executable <emphasis>abi_check</emphasis>.
1014 Used to check correctness of symbol versioning, visibility of
1015 exported symbols, and compatibility on symbols in the shared
1016 library, for hosts that support this feature. More information
1017 can be found in the ABI documentation <link linkend="appendix.porting.abi">here</link>
1018 </para>
1019 </listitem>
1020 <listitem>
1021 <para>
1022 <emphasis>testsuite_allocator.h</emphasis>,
1023 <emphasis>testsuite_allocator.cc</emphasis>
1024 </para>
1025 <para>
1026 Contains specialized allocators that keep track of construction
1027 and destruction. Also, support for overriding global new and
1028 delete operators, including verification that new and delete
1029 are called during execution, and that allocation over max_size
1030 fails.
1031 </para>
1032 </listitem>
1033 <listitem>
1034 <para>
1035 <emphasis>testsuite_character.h</emphasis>
1036 </para>
1037 <para>
1038 Contains <code>std::char_traits</code> and
1039 <code>std::codecvt</code> specializations for a user-defined
1040 POD.
1041 </para>
1042 </listitem>
1043 <listitem>
1044 <para>
1045 <emphasis>testsuite_hooks.h</emphasis>,
1046 <emphasis>testsuite_hooks.cc</emphasis>
1047 </para>
1048 <para>
1049 A large number of utilities, including:
1050 </para>
1051 <itemizedlist>
1052 <listitem><para>VERIFY</para></listitem>
1053 <listitem><para>set_memory_limits</para></listitem>
1054 <listitem><para>verify_demangle</para></listitem>
1055 <listitem><para>run_tests_wrapped_locale</para></listitem>
1056 <listitem><para>run_tests_wrapped_env</para></listitem>
1057 <listitem><para>try_named_locale</para></listitem>
1058 <listitem><para>try_mkfifo</para></listitem>
1059 <listitem><para>func_callback</para></listitem>
1060 <listitem><para>counter</para></listitem>
1061 <listitem><para>copy_tracker</para></listitem>
1062 <listitem><para>copy_constructor</para></listitem>
1063 <listitem><para>assignment_operator</para></listitem>
1064 <listitem><para>destructor</para></listitem>
1065 <listitem>
1066 <para>pod_char, pod_int and associated char_traits specializations</para>
1067 </listitem>
1068 </itemizedlist>
1069 </listitem>
1070 <listitem>
1071 <para>
1072 <emphasis>testsuite_io.h</emphasis>
1073 </para>
1074 <para>
1075 Error, exception, and constraint checking for
1076 <code>std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.
1077 </para>
1078 </listitem>
1079 <listitem>
1080 <para>
1081 <emphasis>testsuite_iterators.h</emphasis>
1082 </para>
1083 <para>
1084 Wrappers for various iterators.
1085 </para>
1086 </listitem>
1087 <listitem>
1088 <para>
1089 <emphasis>testsuite_performance.h</emphasis>
1090 </para>
1091 <para>
1092 A number of class abstractions for performance counters, and
1093 reporting functions including:
1094 </para>
1095 <itemizedlist>
1096 <listitem><para>time_counter</para></listitem>
1097 <listitem><para>resource_counter</para></listitem>
1098 <listitem><para>report_performance</para></listitem>
1099 </itemizedlist>
1100 </listitem>
1101 </itemizedlist>
1102 </section>
1104 </section>
1106 <section xml:id="test.special"><info><title>Special Topics</title></info>
1109 <section xml:id="test.exception.safety"><info><title>
1110 Qualifying Exception Safety Guarantees
1111 <indexterm>
1112 <primary>Test</primary>
1113 <secondary>Exception Safety</secondary>
1114 </indexterm>
1115 </title></info>
1118 <section xml:id="test.exception.safety.overview"><info><title>Overview</title></info>
1121 <para>
1122 Testing is composed of running a particular test sequence,
1123 and looking at what happens to the surrounding code when
1124 exceptions are thrown. Each test is composed of measuring
1125 initial state, executing a particular sequence of code under
1126 some instrumented conditions, measuring a final state, and
1127 then examining the differences between the two states.
1128 </para>
1130 <para>
1131 Test sequences are composed of constructed code sequences
1132 that exercise a particular function or member function, and
1133 either confirm no exceptions were generated, or confirm the
1134 consistency/coherency of the test subject in the event of a
1135 thrown exception.
1136 </para>
1138 <para>
1139 Random code paths can be constructed using the basic test
1140 sequences and instrumentation as above, only combined in a
1141 random or pseudo-random way.
1142 </para>
1144 <para> To compute the code paths that throw, test instruments
1145 are used that throw on allocation events
1146 (<classname>__gnu_cxx::throw_allocator_random</classname>
1147 and <classname>__gnu_cxx::throw_allocator_limit</classname>)
1148 and copy, assignment, comparison, increment, swap, and
1149 various operators
1150 (<classname>__gnu_cxx::throw_type_random</classname>
1151 and <classname>__gnu_cxx::throw_type_limit</classname>). Looping
1152 through a given test sequence and conditionally throwing in
1153 all instrumented places. Then, when the test sequence
1154 completes without an exception being thrown, assume all
1155 potential error paths have been exercised in a sequential
1156 manner.
1157 </para>
1158 </section>
1161 <section xml:id="test.exception.safety.status"><info><title>
1162 Existing tests
1163 </title></info>
1166 <itemizedlist>
1167 <listitem>
1168 <para>
1169 Ad Hoc
1170 </para>
1171 <para>
1172 For example,
1173 <filename>testsuite/23_containers/list/modifiers/3.cc</filename>.
1174 </para>
1175 </listitem>
1177 <listitem>
1178 <para>
1179 Policy Based Data Structures
1180 </para>
1181 <para>
1182 For example, take the test
1183 functor <classname>rand_reg_test</classname> in
1184 in <filename>testsuite/ext/pb_ds/regression/tree_no_data_map_rand.cc</filename>. This uses <classname>container_rand_regression_test</classname> in
1185 <filename>testsuite/util/regression/rand/assoc/container_rand_regression_test.h</filename>.
1187 </para>
1189 <para>
1190 Which has several tests for container member functions,
1191 Includes control and test container objects. Configuration includes
1192 random seed, iterations, number of distinct values, and the
1193 probability that an exception will be thrown. Assumes instantiating
1194 container uses an extension
1195 allocator, <classname>__gnu_cxx::throw_allocator_random</classname>,
1196 as the allocator type.
1197 </para>
1198 </listitem>
1200 <listitem>
1201 <para>
1202 C++11 Container Requirements.
1203 </para>
1205 <para>
1206 Coverage is currently limited to testing container
1207 requirements for exception safety,
1208 although <classname>__gnu_cxx::throw_type</classname> meets
1209 the additional type requirements for testing numeric data
1210 structures and instantiating algorithms.
1211 </para>
1213 <para>
1214 Of particular interest is extending testing to algorithms and
1215 then to parallel algorithms. Also io and locales.
1216 </para>
1218 <para>
1219 The test instrumentation should also be extended to add
1220 instrumentation to <classname>iterator</classname>
1221 and <classname>const_iterator</classname> types that throw
1222 conditionally on iterator operations.
1223 </para>
1224 </listitem>
1225 </itemizedlist>
1226 </section>
1229 <section xml:id="test.exception.safety.containers"><info><title>
1230 C++11 Requirements Test Sequence Descriptions
1231 </title></info>
1234 <itemizedlist>
1235 <listitem>
1236 <para>
1237 Basic
1238 </para>
1240 <para>
1241 Basic consistency on exception propagation tests. For
1242 each container, an object of that container is constructed,
1243 a specific member function is exercised in
1244 a <literal>try</literal> block, and then any thrown
1245 exceptions lead to error checking in the appropriate
1246 <literal>catch</literal> block. The container's use of
1247 resources is compared to the container's use prior to the
1248 test block. Resource monitoring is limited to allocations
1249 made through the container's <type>allocator_type</type>,
1250 which should be sufficient for container data
1251 structures. Included in these tests are member functions
1252 are <type>iterator</type> and <type>const_iterator</type>
1253 operations, <function>pop_front</function>, <function>pop_back</function>, <function>push_front</function>, <function>push_back</function>, <function>insert</function>, <function>erase</function>, <function>swap</function>, <function>clear</function>,
1254 and <function>rehash</function>. The container in question is
1255 instantiated with two instrumented template arguments,
1256 with <classname>__gnu_cxx::throw_allocator_limit</classname>
1257 as the allocator type, and
1258 with <classname>__gnu_cxx::throw_type_limit</classname> as
1259 the value type. This allows the test to loop through
1260 conditional throw points.
1261 </para>
1263 <para>
1264 The general form is demonstrated in
1265 <filename>testsuite/23_containers/list/requirements/exception/basic.cc
1266 </filename>. The instantiating test object is <classname>__gnu_test::basic_safety</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>.
1267 </para>
1268 </listitem>
1271 <listitem>
1272 <para>
1273 Generation Prohibited
1274 </para>
1276 <para>
1277 Exception generation tests. For each container, an object of
1278 that container is constructed and all member functions
1279 required to not throw exceptions are exercised. Included in
1280 these tests are member functions
1281 are <type>iterator</type> and <type>const_iterator</type> operations, <function>erase</function>, <function>pop_front</function>, <function>pop_back</function>, <function>swap</function>,
1282 and <function>clear</function>. The container in question is
1283 instantiated with two instrumented template arguments,
1284 with <classname>__gnu_cxx::throw_allocator_random</classname>
1285 as the allocator type, and
1286 with <classname>__gnu_cxx::throw_type_random</classname> as
1287 the value type. This test does not loop, an instead is sudden
1288 death: first error fails.
1289 </para>
1290 <para>
1291 The general form is demonstrated in
1292 <filename>testsuite/23_containers/list/requirements/exception/generation_prohibited.cc
1293 </filename>. The instantiating test object is <classname>__gnu_test::generation_prohibited</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>.
1294 </para>
1295 </listitem>
1298 <listitem>
1299 <para>
1300 Propagation Consistent
1301 </para>
1303 <para>
1304 Container rollback on exception propagation tests. For
1305 each container, an object of that container is constructed,
1306 a specific member function that requires rollback to a previous
1307 known good state is exercised in
1308 a <literal>try</literal> block, and then any thrown
1309 exceptions lead to error checking in the appropriate
1310 <literal>catch</literal> block. The container is compared to
1311 the container's last known good state using such parameters
1312 as size, contents, and iterator references. Included in these
1313 tests are member functions
1314 are <function>push_front</function>, <function>push_back</function>, <function>insert</function>,
1315 and <function>rehash</function>. The container in question is
1316 instantiated with two instrumented template arguments,
1317 with <classname>__gnu_cxx::throw_allocator_limit</classname>
1318 as the allocator type, and
1319 with <classname>__gnu_cxx::throw_type_limit</classname> as
1320 the value type. This allows the test to loop through
1321 conditional throw points.
1322 </para>
1324 <para>
1325 The general form demonstrated in
1326 <filename>testsuite/23_containers/list/requirements/exception/propagation_coherent.cc
1327 </filename>. The instantiating test object is <classname>__gnu_test::propagation_coherent</classname> and is detailed in <filename>testsuite/util/exception/safety.h</filename>.
1328 </para>
1329 </listitem>
1330 </itemizedlist>
1332 </section>
1334 </section>
1336 </section>
1338 </section>