toolchain/gcc/INSTALL/test.html

   1 <html lang="en">
   2 <head>
   3 <title>Installing GCC: Testing</title>
   4 <meta http-equiv="Content-Type" content="text/html">
   5 <meta name="description" content="Installing GCC: Testing">
   6 <meta name="generator" content="makeinfo 4.7">
   7 <link title="Top" rel="top" href="#Top">
   8 <link href="http://www.gnu.org/software/texinfo/" rel="generator-home" title="Texinfo Homepage">
   9 <!--
  10 Copyright (C) 1988, 1989, 1992, 1993, 1994, 1995, 1996, 1997, 1998,
  11 1999, 2000, 2001, 2002, 2003, 2004 Free Software Foundation, Inc.
  12
  13    Permission is granted to copy, distribute and/or modify this document
  14 under the terms of the GNU Free Documentation License, Version 1.2 or
  15 any later version published by the Free Software Foundation; with no
  16 Invariant Sections, the Front-Cover texts being (a) (see below), and
  17 with the Back-Cover Texts being (b) (see below).  A copy of the
  18 license is included in the section entitled "GNU Free Documentation License".
  19
  20 (a) The FSF's Front-Cover Text is:
  21
  22      A GNU Manual
  23
  24 (b) The FSF's Back-Cover Text is:
  25
  26      You have freedom to copy and modify this GNU Manual, like GNU
  27      software.  Copies published by the Free Software Foundation raise
  28      funds for GNU development.-->
  29 <meta http-equiv="Content-Style-Type" content="text/css">
  30 <style type="text/css"><!--
  31   pre.display { font-family:inherit }
  32   pre.format  { font-family:inherit }
  33   pre.smalldisplay { font-family:inherit; font-size:smaller }
  34   pre.smallformat  { font-family:inherit; font-size:smaller }
  35   pre.smallexample { font-size:smaller }
  36   pre.smalllisp    { font-size:smaller }
  37   span.sc { font-variant:small-caps }
  38   span.roman { font-family: serif; font-weight: normal; }
  39 --></style>
  40 </head>
  41 <body>
  42 <h1 class="settitle">Installing GCC: Testing</h1>
  43 <a name="index-Testing-1"></a><a name="index-Installing-GCC_003a-Testing-2"></a><a name="index-Testsuite-3"></a>
  44 Before you install GCC, we encourage you to run the testsuites and to
  45 compare your results with results from a similar configuration that have
  46 been submitted to the
  47 <a href="http://gcc.gnu.org/ml/gcc-testresults/">gcc-testresults mailing list</a>.
  48 Some of these archived results are linked from the build status lists
  49 at <a href="http://gcc.gnu.org/buildstat.html">http://gcc.gnu.org/buildstat.html</a>, although not everyone who
  50 reports a successful build runs the testsuites and submits the results.
  51 This step is optional and may require you to download additional software,
  52 but it can give you confidence in your new GCC installation or point out
  53 problems before you install and start using your new GCC.
  54
  55    <p>First, you must have <a href="download.html">downloaded the testsuites</a>.
  56 These are part of the full distribution, but if you downloaded the
  57 &ldquo;core&rdquo; compiler plus any front ends, you must download the testsuites
  58 separately.
  59
  60    <p>Second, you must have the testing tools installed.  This includes
  61 <a href="http://www.gnu.org/software/dejagnu/">DejaGnu</a> 1.4.1 or 1.4.3
  62 and later, Tcl, and Expect; the DejaGnu site has links to these.
  63
  64    <p>If the directories where <span class="command">runtest</span> and <span class="command">expect</span> were
  65 installed are not in the <span class="env">PATH</span>, you may need to set the following
  66 environment variables appropriately, as in the following example (which
  67 assumes that DejaGnu has been installed under <span class="file">/usr/local</span>):
  68
  69 <pre class="smallexample">          TCL_LIBRARY = /usr/local/share/tcl8.0
  70           DEJAGNULIBS = /usr/local/share/dejagnu
  71 </pre>
  72    <p>(On systems such as Cygwin, these paths are required to be actual
  73 paths, not mounts or links; presumably this is due to some lack of
  74 portability in the DejaGnu code.)
  75
  76    <p>Finally, you can run the testsuite (which may take a long time):
  77 <pre class="smallexample">          cd <var>objdir</var>; make -k check
  78 </pre>
  79    <p>This will test various components of GCC, such as compiler
  80 front ends and runtime libraries.  While running the testsuite, DejaGnu
  81 might emit some harmless messages resembling
  82 <span class="samp">WARNING: Couldn't find the global config file.</span> or
  83 <span class="samp">WARNING: Couldn't find tool init file</span> that can be ignored.
  84
  85 <h3 class="section"><a name="TOC0"></a>0.1 How can you run the testsuite on selected tests?</h3>
  86
  87 <p>In order to run sets of tests selectively, there are targets
  88 <span class="samp">make check-gcc</span> and <span class="samp">make check-g++</span>
  89 in the <span class="file">gcc</span> subdirectory of the object directory.  You can also
  90 just run <span class="samp">make check</span> in a subdirectory of the object directory.
  91
  92    <p>A more selective way to just run all <span class="command">gcc</span> execute tests in the
  93 testsuite is to use
  94
  95 <pre class="smallexample">         make check-gcc RUNTESTFLAGS="execute.exp <var>other-options</var>"
  96 </pre>
  97    <p>Likewise, in order to run only the <span class="command">g++</span> &ldquo;old-deja&rdquo; tests in
  98 the testsuite with filenames matching <span class="samp">9805*</span>, you would use
  99
 100 <pre class="smallexample">         make check-g++ RUNTESTFLAGS="old-deja.exp=9805* <var>other-options</var>"
 101 </pre>
 102    <p>The <span class="file">*.exp</span> files are located in the testsuite directories of the GCC
 103 source, the most important ones being <span class="file">compile.exp</span>,
 104 <span class="file">execute.exp</span>, <span class="file">dg.exp</span> and <span class="file">old-deja.exp</span>.
 105 To get a list of the possible <span class="file">*.exp</span> files, pipe the
 106 output of <span class="samp">make check</span> into a file and look at the
 107 <span class="samp">Running ...  .exp</span> lines.
 108
 109 <h3 class="section"><a name="TOC1"></a>0.2 Passing options and running multiple testsuites</h3>
 110
 111 <p>You can pass multiple options to the testsuite using the
 112 <span class="samp">--target_board</span> option of DejaGNU, either passed as part of
 113 <span class="samp">RUNTESTFLAGS</span>, or directly to <span class="command">runtest</span> if you prefer to
 114 work outside the makefiles.  For example,
 115
 116 <pre class="smallexample">         make check-g++ RUNTESTFLAGS="--target_board=unix/-O3/-fno-strength-reduce"
 117 </pre>
 118    <p>will run the standard <span class="command">g++</span> testsuites (&ldquo;unix&rdquo; is the target name
 119 for a standard native testsuite situation), passing
 120 <span class="samp">-O3 -fno-strength-reduce</span> to the compiler on every test, i.e.,
 121 slashes separate options.
 122
 123    <p>You can run the testsuites multiple times using combinations of options
 124 with a syntax similar to the brace expansion of popular shells:
 125
 126 <pre class="smallexample">         ..."--target_board=arm-sim{-mhard-float,-msoft-float}{-O1,-O2,-O3,}"
 127 </pre>
 128    <p>(Note the empty option caused by the trailing comma in the final group.)
 129 The following will run each testsuite eight times using the <span class="samp">arm-sim</span>
 130 target, as if you had specified all possible combinations yourself:
 131
 132 <pre class="smallexample">         --target_board=arm-sim/-mhard-float/-O1
 133          --target_board=arm-sim/-mhard-float/-O2
 134          --target_board=arm-sim/-mhard-float/-O3
 135          --target_board=arm-sim/-mhard-float
 136          --target_board=arm-sim/-msoft-float/-O1
 137          --target_board=arm-sim/-msoft-float/-O2
 138          --target_board=arm-sim/-msoft-float/-O3
 139          --target_board=arm-sim/-msoft-float
 140 </pre>
 141    <p>They can be combined as many times as you wish, in arbitrary ways.  This
 142 list:
 143
 144 <pre class="smallexample">         ..."--target_board=unix/-Wextra{-O3,-fno-strength-reduce}{-fomit-frame-pointer,}"
 145 </pre>
 146    <p>will generate four combinations, all involving <span class="samp">-Wextra</span>.
 147
 148    <p>The disadvantage to this method is that the testsuites are run in serial,
 149 which is a waste on multiprocessor systems.  For users with GNU Make and
 150 a shell which performs brace expansion, you can run the testsuites in
 151 parallel by having the shell perform the combinations and <span class="command">make</span>
 152 do the parallel runs.  Instead of using <span class="samp">--target_board</span>, use a
 153 special makefile target:
 154
 155 <pre class="smallexample">         make -j<var>N</var> check-<var>testsuite</var>//<var>test-target</var>/<var>option1</var>/<var>option2</var>/...
 156 </pre>
 157    <p>For example,
 158
 159 <pre class="smallexample">         make -j3 check-gcc//sh-hms-sim/{-m1,-m2,-m3,-m3e,-m4}/{,-nofpu}
 160 </pre>
 161    <p>will run three concurrent &ldquo;make-gcc&rdquo; testsuites, eventually testing all
 162 ten combinations as described above.  Note that this is currently only
 163 supported in the <span class="file">gcc</span> subdirectory.  (To see how this works, try
 164 typing <span class="command">echo</span> before the example given here.)
 165
 166 <h3 class="section"><a name="TOC2"></a>0.3 Additional testing for Java Class Libraries</h3>
 167
 168 <p>The Java runtime tests can be executed via <span class="samp">make check</span>
 169 in the <var>target</var><span class="file">/libjava/testsuite</span> directory in
 170 the build tree.
 171
 172    <p>The <a href="http://sources.redhat.com/mauve/">Mauve Project</a> provides
 173 a suite of tests for the Java Class Libraries.  This suite can be run
 174 as part of libgcj testing by placing the Mauve tree within the libjava
 175 testsuite at <span class="file">libjava/testsuite/libjava.mauve/mauve</span>, or by
 176 specifying the location of that tree when invoking <span class="samp">make</span>, as in
 177 <span class="samp">make MAUVEDIR=~/mauve check</span>.
 178
 179    <p><a href="http://www-124.ibm.com/developerworks/oss/cvs/jikes/~checkout~/jacks/jacks.html">Jacks</a>
 180 is a free testsuite that tests Java compiler front ends.  This suite
 181 can be run as part of libgcj testing by placing the Jacks tree within
 182 the libjava testsuite at <span class="file">libjava/testsuite/libjava.jacks/jacks</span>.
 183
 184 <h3 class="section"><a name="TOC3"></a>0.4 How to interpret test results</h3>
 185
 186 <p>The result of running the testsuite are various <span class="file">*.sum</span> and <span class="file">*.log</span>
 187 files in the testsuite subdirectories.  The <span class="file">*.log</span> files contain a
 188 detailed log of the compiler invocations and the corresponding
 189 results, the <span class="file">*.sum</span> files summarize the results.  These summaries
 190 contain status codes for all tests:
 191
 192      <ul>
 193 <li>PASS: the test passed as expected
 194 <li>XPASS: the test unexpectedly passed
 195 <li>FAIL: the test unexpectedly failed
 196 <li>XFAIL: the test failed as expected
 197 <li>UNSUPPORTED: the test is not supported on this platform
 198 <li>ERROR: the testsuite detected an error
 199 <li>WARNING: the testsuite detected a possible problem
 200 </ul>
 201
 202    <p>It is normal for some tests to report unexpected failures.  At the
 203 current time the testing harness does not allow fine grained control
 204 over whether or not a test is expected to fail.  This problem should
 205 be fixed in future releases.
 206
 207 <h3 class="section"><a name="TOC4"></a>0.5 Submitting test results</h3>
 208
 209 <p>If you want to report the results to the GCC project, use the
 210 <span class="file">contrib/test_summary</span> shell script.  Start it in the <var>objdir</var> with
 211
 212 <pre class="smallexample">         <var>srcdir</var>/contrib/test_summary -p your_commentary.txt \
 213              -m gcc-testresults@gcc.gnu.org |sh
 214 </pre>
 215    <p>This script uses the <span class="command">Mail</span> program to send the results, so
 216 make sure it is in your <span class="env">PATH</span>.  The file <span class="file">your_commentary.txt</span> is
 217 prepended to the testsuite summary and should contain any special
 218 remarks you have on your results or your build environment.  Please
 219 do not edit the testsuite result block or the subject line, as these
 220 messages may be automatically processed.
 221
 222    <p><hr />
 223 <p><a href="./index.html">Return to the GCC Installation page</a>
 224
 225 <!-- ***Final install*********************************************************** -->
 226 <!-- ***Binaries**************************************************************** -->
 227 <!-- ***Specific**************************************************************** -->
 228 <!-- ***Old documentation****************************************************** -->
 229 <!-- ***GFDL******************************************************************** -->
 230 <!-- *************************************************************************** -->
 231 <!-- Part 6 The End of the Document -->
 232 </body></html>
 233