docs/TestingGuide.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
   2                       "http://www.w3.org/TR/html4/strict.dtd">
   3 <html>
   4 <head>
   5   <title>LLVM Test Suite Guide</title>
   6   <link rel="stylesheet" href="llvm.css" type="text/css">
   7 </head>
   8 <body>
   9
  10 <div class="doc_title">
  11   LLVM Test Suite Guide
  12 </div>
  13
  14 <ol>
  15   <li><a href="#overview">Overview</a></li>
  16   <li><a href="#Requirements">Requirements</a></li>
  17   <li><a href="#quick">Quick Start</a></li>
  18   <li><a href="#org">LLVM Test Suite Organization</a>
  19     <ul>
  20       <li><a href="#codefragments">Code Fragments</a></li>
  21       <li><a href="#wholeprograms">Whole Programs</a></li>
  22     </ul>
  23   </li>
  24   <li><a href="#tree">LLVM Test Suite Tree</a></li>
  25   <li><a href="#dgstructure">DejaGNU Structure</a></li>
  26   <li><a href="#progstructure"><tt>llvm-test</tt> Structure</a></li>
  27   <li><a href="#run">Running the LLVM Tests</a>
  28     <ul>
  29       <li><a href="#customtest">Writing custom tests for llvm-test</a></li>
  30     </ul>
  31   </li>
  32   <li><a href="#nightly">Running the nightly tester</a></li>
  33 </ol>
  34
  35 <div class="doc_author">
  36   <p>Written by John T. Criswell, <a
  37   href="http://llvm.x10sys.com/rspencer">Reid Spencer</a>, and Tanya Lattner</p>
  38 </div>
  39
  40 <!--=========================================================================-->
  41 <div class="doc_section"><a name="overview">Overview</a></div>
  42 <!--=========================================================================-->
  43
  44 <div class="doc_text">
  45
  46 <p>This document is the reference manual for the LLVM test suite.  It documents
  47 the structure of the LLVM test suite, the tools needed to use it, and how to add
  48 and run tests.</p>
  49
  50 </div>
  51
  52 <!--=========================================================================-->
  53 <div class="doc_section"><a name="Requirements">Requirements</a></div>
  54 <!--=========================================================================-->
  55
  56 <div class="doc_text">
  57
  58 <p>In order to use the LLVM test suite, you will need all of the software
  59 required to build LLVM, plus the following:</p>
  60
  61 <dl>
  62 <dt><a href="http://www.gnu.org/software/dejagnu/">DejaGNU</a></dt>
  63 <dd>The Feature and Regressions tests are organized and run by DejaGNU.</dd>
  64 <dt><a href="http://expect.nist.gov/">Expect</a></dt>
  65 <dd>Expect is required by DejaGNU.</dd>
  66 <dt><a href="http://www.tcl.tk/software/tcltk/">tcl</a></dt>
  67 <dd>Tcl is required by DejaGNU. </dd>
  68
  69 <dt><a href="http://www.netlib.org/f2c">F2C</a></dt>
  70 <dd>For now, LLVM does not have a Fortran front-end, but using F2C, we can run
  71 Fortran benchmarks.  F2C support must be enabled via <tt>configure</tt> if not
  72 installed in a standard place.  F2C requires three items: the <tt>f2c</tt>
  73 executable, <tt>f2c.h</tt> to compile the generated code, and <tt>libf2c.a</tt>
  74 to link generated code.  By default, given an F2C directory <tt>$DIR</tt>, the
  75 configure script will search <tt>$DIR/bin</tt> for <tt>f2c</tt>,
  76 <tt>$DIR/include</tt> for <tt>f2c.h</tt>, and <tt>$DIR/lib</tt> for
  77 <tt>libf2c.a</tt>.  The default <tt>$DIR</tt> values are: <tt>/usr</tt>,
  78 <tt>/usr/local</tt>, <tt>/sw</tt>, and <tt>/opt</tt>.  If you installed F2C in a
  79 different location, you must tell <tt>configure</tt>:
  80
  81 <ul>
  82 <li><tt>./configure --with-f2c=$DIR</tt><br>
  83 This will specify a new <tt>$DIR</tt> for the above-described search
  84 process.  This will only work if the binary, header, and library are in their
  85 respective subdirectories of <tt>$DIR</tt>.</li>
  86
  87 <li><tt>./configure --with-f2c-bin=/binary/path --with-f2c-inc=/include/path
  88 --with-f2c-lib=/lib/path</tt><br>
  89 This allows you to specify the F2C components separately.  Note: if you choose
  90 this route, you MUST specify all three components, and you need to only specify
  91 <em>directories</em> where the files are located; do NOT include the
  92 filenames themselves on the <tt>configure</tt> line.</li>
  93 </ul></dd>
  94 </dl>
  95
  96 <p>Darwin (Mac OS X) developers can simplify the installation of Expect and tcl
  97 by using fink.  <tt>fink install expect</tt> will install both. Alternatively,
  98 Darwinports users can use <tt>sudo port install expect</tt> to install Expect
  99 and tcl.</p>
 100
 101 </div>
 102
 103 <!--=========================================================================-->
 104 <div class="doc_section"><a name="quick">Quick Start</a></div>
 105 <!--=========================================================================-->
 106
 107 <div class="doc_text">
 108
 109   <p>The tests are located in two separate Subversion modules. The basic feature
 110   and regression tests are in the main "llvm" module under the directory
 111   <tt>llvm/test</tt>. A more comprehensive test suite that includes whole
 112 programs in C and C++ is in the <tt>test-suite</tt> module. This module should
 113 be checked out to the <tt>llvm/projects</tt> directory as llvm-test (for
 114 historical purpose).  When you <tt>configure</tt> the <tt>llvm</tt> module,
 115 the <tt>llvm-test</tt> directory will be automatically configured.
 116 Alternatively, you can configure the <tt>test-suite</tt> module manually.</p>
 117 <p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile
 118  in the <tt>llvm/test</tt> directory:</p>
 119
 120 <div class="doc_code">
 121 <pre>
 122 % gmake -C llvm/test
 123 </pre>
 124 </div>
 125
 126 <p>or</p>
 127
 128 <div class="doc_code">
 129 <pre>
 130 % gmake check
 131 </pre>
 132 </div>
 133
 134 <p>To run only a subdirectory of tests in llvm/test using DejaGNU (ie.
 135 Regression/Transforms), just set the TESTSUITE variable to the path of the
 136 subdirectory (relative to <tt>llvm/test</tt>):</p>
 137
 138 <div class="doc_code">
 139 <pre>
 140 % gmake -C llvm/test TESTSUITE=Regression/Transforms
 141 </pre>
 142 </div>
 143
 144 <p><b>Note: If you are running the tests with <tt>objdir != subdir</tt>, you
 145 must have run the complete testsuite before you can specify a
 146 subdirectory.</b></p>
 147
 148 <p>To run the comprehensive test suite (tests that compile and execute whole
 149 programs), run the <tt>llvm-test</tt> tests:</p>
 150
 151 <div class="doc_code">
 152 <pre>
 153 % cd llvm/projects
 154 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
 155 % cd llvm-test
 156 % ./configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT
 157 % gmake
 158 </pre>
 159 </div>
 160
 161 </div>
 162
 163 <!--=========================================================================-->
 164 <div class="doc_section"><a name="org">LLVM Test Suite Organization</a></div>
 165 <!--=========================================================================-->
 166
 167 <div class="doc_text">
 168
 169 <p>The LLVM test suite contains two major categories of tests: code
 170 fragments and whole programs. Code fragments are in the <tt>llvm</tt> module
 171 under the <tt>llvm/test</tt> directory. The whole programs
 172 test suite is in the <tt>llvm-test</tt> module under the main directory.</p>
 173
 174 </div>
 175
 176 <!-- _______________________________________________________________________ -->
 177 <div class="doc_subsection"><a name="codefragments">Code Fragments</a></div>
 178 <!-- _______________________________________________________________________ -->
 179
 180 <div class="doc_text">
 181
 182 <p>Code fragments are small pieces of code that test a specific feature of LLVM
 183 or trigger a specific bug in LLVM.  They are usually written in LLVM assembly
 184 language, but can be written in other languages if the test targets a particular
 185 language front end.</p>
 186
 187 <p>Code fragments are not complete programs, and they are never executed to
 188 determine correct behavior.</p>
 189
 190 <p>These code fragment tests are located in the <tt>llvm/test/Features</tt> and
 191 <tt>llvm/test/Regression</tt> directories.</p>
 192
 193 </div>
 194
 195 <!-- _______________________________________________________________________ -->
 196 <div class="doc_subsection"><a name="wholeprograms">Whole Programs</a></div>
 197 <!-- _______________________________________________________________________ -->
 198
 199 <div class="doc_text">
 200
 201 <p>Whole Programs are pieces of code which can be compiled and linked into a
 202 stand-alone program that can be executed.  These programs are generally written
 203 in high level languages such as C or C++, but sometimes they are written
 204 straight in LLVM assembly.</p>
 205
 206 <p>These programs are compiled and then executed using several different
 207 methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation,
 208 etc).  The output of these programs is compared to ensure that LLVM is compiling
 209 the program correctly.</p>
 210
 211 <p>In addition to compiling and executing programs, whole program tests serve as
 212 a way of benchmarking LLVM performance, both in terms of the efficiency of the
 213 programs generated as well as the speed with which LLVM compiles, optimizes, and
 214 generates code.</p>
 215
 216 <p>All "whole program" tests are located in the <tt>test-suite</tt> Subversion
 217 module.</p>
 218
 219 </div>
 220
 221 <!--=========================================================================-->
 222 <div class="doc_section"><a name="tree">LLVM Test Suite Tree</a></div>
 223 <!--=========================================================================-->
 224
 225 <div class="doc_text">
 226
 227 <p>Each type of test in the LLVM test suite has its own directory. The major
 228 subtrees of the test suite directory tree are as follows:</p>
 229
 230 <ul>
 231   <li><tt>llvm/test</tt>
 232   <p>This directory contains a large array of small tests
 233   that exercise various features of LLVM and to ensure that regressions do not
 234   occur. The directory is broken into several sub-directories, each focused on
 235   a particular area of LLVM. A few of the important ones are:</p>
 236   <ul>
 237     <li><tt>Analysis</tt>: checks Analysis passes.</li>
 238     <li><tt>Archive</tt>: checks the Archive library.</li>
 239     <li><tt>Assembler</tt>: checks Assembly reader/writer functionality.</li>
 240     <li><tt>Bitcode</tt>: checks Bitcode reader/writer functionality.</li>
 241     <li><tt>CodeGen</tt>: checks code generation and each target.</li>
 242     <li><tt>Features</tt>: checks various features of the LLVM language.</li>
 243     <li><tt>Linker</tt>: tests bitcode linking.</li>
 244     <li><tt>Transforms</tt>: tests each of the scalar, IPO, and utility
 245     transforms to ensure they make the right transformations.</li>
 246     <li><tt>Verifier</tt>: tests the IR verifier.</li>
 247   </ul>
 248   <p>Typically when a bug is found in LLVM, a regression test containing
 249   just enough code to reproduce the problem should be written and placed
 250   somewhere underneath this directory.  In most cases, this will be a small
 251   piece of LLVM assembly language code, often distilled from an actual
 252   application or benchmark.</p></li>
 253
 254 <li><tt>test-suite</tt>
 255 <p>The <tt>test-suite</tt> module contains programs that can be compiled
 256 with LLVM and executed.  These programs are compiled using the native compiler
 257 and various LLVM backends.  The output from the program compiled with the
 258 native compiler is assumed correct; the results from the other programs are
 259 compared to the native program output and pass if they match.</p>
 260
 261 <p>In addition for testing correctness, the <tt>llvm-test</tt> directory also
 262 performs timing tests of various LLVM optimizations.  It also records
 263 compilation times for the compilers and the JIT.  This information can be
 264 used to compare the effectiveness of LLVM's optimizations and code
 265 generation.</p></li>
 266
 267 <li><tt>llvm-test/SingleSource</tt>
 268 <p>The SingleSource directory contains test programs that are only a single
 269 source file in size.  These are usually small benchmark programs or small
 270 programs that calculate a particular value.  Several such programs are grouped
 271 together in each directory.</p></li>
 272
 273 <li><tt>llvm-test/MultiSource</tt>
 274 <p>The MultiSource directory contains subdirectories which contain entire
 275 programs with multiple source files.  Large benchmarks and whole applications
 276 go here.</p></li>
 277
 278 <li><tt>llvm-test/External</tt>
 279 <p>The External directory contains Makefiles for building code that is external
 280 to (i.e., not distributed with) LLVM.  The most prominent members of this
 281 directory are the SPEC 95 and SPEC 2000 benchmark suites.  The presence and
 282 location of these external programs is configured by the llvm-test
 283 <tt>configure</tt> script.</p></li>
 284
 285 </ul>
 286
 287 </div>
 288 <!--=========================================================================-->
 289 <div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div>
 290 <!--=========================================================================-->
 291 <div class="doc_text">
 292   <p>The LLVM test suite is partially driven by DejaGNU and partially driven by
 293   GNU Make. Specifically, the Features and Regression tests are all driven by
 294   DejaGNU. The <tt>llvm-test</tt> module is currently driven by a set of
 295   Makefiles.</p>
 296
 297   <p>The DejaGNU structure is very simple, but does require some information to
 298   be set. This information is gathered via <tt>configure</tt> and is written
 299   to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt>
 300   Makefile does this work for you.</p>
 301
 302   <p>In order for DejaGNU to work, each directory of tests must have a
 303   <tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the
 304   tests. This file is just a Tcl script and it can do anything you want, but
 305   we've standardized it for the LLVM regression tests. It simply loads a Tcl
 306   library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt>
 307   function defined in that library with a list of file names to run. The names
 308   are obtained by using Tcl's glob command.  Any directory that contains only
 309   directories does not need the <tt>dg.exp</tt> file.</p>
 310
 311   <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to
 312   it and gathers any lines together that match "RUN:". This are the "RUN" lines
 313   that specify how the test is to be run. So, each test script must contain
 314   RUN lines if it is to do anything. If there are no RUN lines, the
 315   <tt>llvm-runtests</tt> function will issue an error and the test will
 316   fail.</p>
 317
 318   <p>RUN lines are specified in the comments of the test program using the
 319   keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline)
 320   to execute.  Together, these lines form the "script" that
 321   <tt>llvm-runtests</tt> executes to run the test case.  The syntax of the
 322   RUN lines is similar to a shell's syntax for pipelines including I/O
 323   redirection and variable substitution.  However, even though these lines
 324   may <i>look</i> like a shell script, they are not. RUN lines are interpreted
 325   directly by the Tcl <tt>exec</tt> command. They are never executed by a
 326   shell. Consequently the syntax differs from normal shell script syntax in a
 327   few ways.  You can specify as many RUN lines as needed.</p>
 328
 329   <p>Each RUN line is executed on its own, distinct from other lines unless
 330   its last character is <tt>\</tt>. This continuation character causes the RUN
 331   line to be concatenated with the next one. In this way you can build up long
 332   pipelines of commands without making huge line lengths. The lines ending in
 333   <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is
 334   found. This concatenated set or RUN lines then constitutes one execution.
 335   Tcl will substitute variables and arrange for the pipeline to be executed. If
 336   any process in the pipeline fails, the entire line (and test case) fails too.
 337   </p>
 338
 339   <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p>
 340
 341 <div class="doc_code">
 342 <pre>
 343 ; RUN: llvm-as &lt; %s | llvm-dis &gt; %t1
 344 ; RUN: llvm-dis &lt; %s.bc-13 &gt; %t2
 345 ; RUN: diff %t1 %t2
 346 </pre>
 347 </div>
 348
 349   <p>As with a Unix shell, the RUN: lines permit pipelines and I/O redirection
 350   to be used. However, the usage is slightly different than for Bash. To check
 351   what's legal, see the documentation for the
 352   <a href="http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2">Tcl exec</a>
 353   command and the
 354   <a href="http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html">tutorial</a>.
 355   The major differences are:</p>
 356   <ul>
 357     <li>You can't do <tt>2&gt;&amp;1</tt>. That will cause Tcl to write to a
 358     file named <tt>&amp;1</tt>. Usually this is done to get stderr to go through
 359     a pipe. You can do that in tcl with <tt>|&amp;</tt> so replace this idiom:
 360     <tt>... 2&gt;&amp;1 | grep</tt> with <tt>... |&amp; grep</tt></li>
 361     <li>You can only redirect to a file, not to another descriptor and not from
 362     a here document.</li>
 363     <li>tcl supports redirecting to open files with the @ syntax but you
 364     shouldn't use that here.</li>
 365   </ul>
 366
 367   <p>There are some quoting rules that you must pay attention to when writing
 368   your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any
 369   ' or " so they will get passed to the invoked program. For example:</p>
 370
 371 <div class="doc_code">
 372 <pre>
 373 ... | grep 'find this string'
 374 </pre>
 375 </div>
 376
 377   <p>This will fail because the ' characters are passed to grep. This would
 378   instruction grep to look for <tt>'find</tt> in the files <tt>this</tt> and
 379   <tt>string'</tt>. To avoid this use curly braces to tell Tcl that it should
 380   treat everything enclosed as one value. So our example would become:</p>
 381
 382 <div class="doc_code">
 383 <pre>
 384 ... | grep {find this string}
 385 </pre>
 386 </div>
 387
 388   <p>Additionally, the characters <tt>[</tt> and <tt>]</tt> are treated
 389   specially by Tcl. They tell Tcl to interpret the content as a command to
 390   execute. Since these characters are often used in regular expressions this can
 391   have disastrous results and cause the entire test run in a directory to fail.
 392   For example, a common idiom is to look for some basicblock number:</p>
 393
 394 <div class="doc_code">
 395 <pre>
 396 ... | grep bb[2-8]
 397 </pre>
 398 </div>
 399
 400   <p>This, however, will cause Tcl to fail because its going to try to execute
 401   a program named "2-8". Instead, what you want is this:</p>
 402
 403 <div class="doc_code">
 404 <pre>
 405 ... | grep {bb\[2-8\]}
 406 </pre>
 407 </div>
 408
 409   <p>Finally, if you need to pass the <tt>\</tt> character down to a program,
 410   then it must be doubled. This is another Tcl special character. So, suppose
 411   you had:
 412
 413 <div class="doc_code">
 414 <pre>
 415 ... | grep 'i32\*'
 416 </pre>
 417 </div>
 418
 419   <p>This will fail to match what you want (a pointer to i32). First, the
 420   <tt>'</tt> do not get stripped off. Second, the <tt>\</tt> gets stripped off
 421   by Tcl so what grep sees is: <tt>'i32*'</tt>. That's not likely to match
 422   anything. To resolve this you must use <tt>\\</tt> and the <tt>{}</tt>, like
 423   this:</p>
 424
 425 <div class="doc_code">
 426 <pre>
 427 ... | grep {i32\\*}
 428 </pre>
 429 </div>
 430
 431 </div>
 432
 433 <!-- _______________________________________________________________________ -->
 434 <div class="doc_subsection"><a name="dgvars">Vars And Substitutions</a></div>
 435 <div class="doc_text">
 436   <p>With a RUN line there are a number of substitutions that are permitted. In
 437   general, any Tcl variable that is available in the <tt>substitute</tt>
 438   function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
 439   To make a substitution just write the variable's name preceded by a $.
 440   Additionally, for compatibility reasons with previous versions of the test
 441   library, certain names can be accessed with an alternate syntax: a % prefix.
 442   These alternates are deprecated and may go away in a future version.
 443   </p>
 444   <p>Here are the available variable names. The alternate syntax is listed in
 445   parentheses.</p>
 446
 447   <dl style="margin-left: 25px">
 448     <dt><b>$test</b> (%s)</dt>
 449     <dd>The full path to the test case's source. This is suitable for passing
 450     on the command line as the input to an llvm tool.</dd>
 451
 452     <dt><b>$srcdir</b></dt>
 453     <dd>The source directory from where the "<tt>make check</tt>" was run.</dd>
 454
 455     <dt><b>objdir</b></dt>
 456     <dd>The object directory that corresponds to the <tt>$srcdir</tt>.</dd>
 457
 458     <dt><b>subdir</b></dt>
 459     <dd>A partial path from the <tt>test</tt> directory that contains the
 460     sub-directory that contains the test source being executed.</dd>
 461
 462     <dt><b>srcroot</b></dt>
 463     <dd>The root directory of the LLVM src tree.</dd>
 464
 465     <dt><b>objroot</b></dt>
 466     <dd>The root directory of the LLVM object tree. This could be the same
 467     as the srcroot.</dd>
 468
 469     <dt><b>path</b><dt>
 470     <dd>The path to the directory that contains the test case source.  This is
 471     for locating any supporting files that are not generated by the test, but
 472     used by the test.</dd>
 473
 474     <dt><b>tmp</b></dt>
 475     <dd>The path to a temporary file name that could be used for this test case.
 476     The file name won't conflict with other test cases. You can append to it if
 477     you need multiple temporaries. This is useful as the destination of some
 478     redirected output.</dd>
 479
 480     <dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt>
 481     <dd>The directory where the LLVM libraries are located.</dd>
 482
 483     <dt><b>target_triplet</b> (%target_triplet)</dt>
 484     <dd>The target triplet that corresponds to the current host machine (the one
 485     running the test cases). This should probably be called "host".<dd>
 486
 487     <dt><b>prcontext</b> (%prcontext)</dt>
 488     <dd>Path to the prcontext tcl script that prints some context around a
 489     line that matches a pattern. This isn't strictly necessary as the test suite
 490     is run with its PATH altered to include the test/Scripts directory where
 491     the prcontext script is located. Note that this script is similar to
 492     <tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because
 493     not all platforms support <tt>grep -C</tt>.</dd>
 494
 495     <dt><b>llvmgcc</b> (%llvmgcc)</dt>
 496     <dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the
 497     configured LLVM environment</dd>
 498
 499     <dt><b>llvmgxx</b> (%llvmgxx)</dt>
 500     <dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the
 501     configured LLVM environment</dd>
 502
 503     <dt><b>llvmgcc_version</b> (%llvmgcc_version)</dt>
 504     <dd>The full version number of the <tt>llvm-gcc</tt> executable.</dd>
 505
 506     <dt><b>llvmgccmajvers</b> (%llvmgccmajvers)</dt>
 507     <dd>The major version number of the <tt>llvm-gcc</tt> executable.</dd>
 508
 509     <dt><b>gccpath</b></dt>
 510     <dd>The full path to the C compiler used to <i>build </i> LLVM. Note that
 511     this might not be gcc.</dd>
 512
 513     <dt><b>gxxpath</b></dt>
 514     <dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that
 515     this might not be g++.</dd>
 516
 517     <dt><b>compile_c</b> (%compile_c)</dt>
 518     <dd>The full command line used to compile LLVM C source  code. This has all
 519     the configured -I, -D and optimization options.</dd>
 520
 521     <dt><b>compile_cxx</b> (%compile_cxx)</dt>
 522     <dd>The full command used to compile LLVM C++ source  code. This has
 523     all the configured -I, -D and optimization options.</dd>
 524
 525     <dt><b>link</b> (%link)</dt>
 526     <dd>This full link command used to link LLVM executables. This has all the
 527     configured -I, -L and -l options.</dd>
 528
 529     <dt><b>shlibext</b> (%shlibext)</dt>
 530     <dd>The suffix for the host platforms share library (dll) files. This
 531     includes the period as the first character.</dd>
 532   </dl>
 533   <p>To add more variables, two things need to be changed. First, add a line in
 534   the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will
 535   "set" the variable as a global in the site.exp file. Second, in the
 536   <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name
 537   to the list of "global" declarations at the beginning of the proc. That's it,
 538   the variable can then be used in test scripts.</p>
 539 </div>
 540
 541 <!-- _______________________________________________________________________ -->
 542 <div class="doc_subsection"><a name="dgfeatures">Other Features</a></div>
 543 <div class="doc_text">
 544   <p>To make RUN line writing easier, there are several shell scripts located
 545   in the <tt>llvm/test/Scripts</tt> directory. For example:</p>
 546   <dl>
 547     <dt><b>ignore</b></dt>
 548     <dd>This script runs its arguments and then always returns 0. This is useful
 549     in cases where the test needs to cause a tool to generate an error (e.g. to
 550     check the error output). However, any program in a pipeline that returns a
 551     non-zero result will cause the test to fail. This script overcomes that
 552     issue and nicely documents that the test case is purposefully ignoring the
 553     result code of the tool</dd>
 554
 555     <dt><b>not</b></dt>
 556     <dd>This script runs its arguments and then inverts the result code from
 557     it. Zero result codes become 1. Non-zero result codes become 0. This is
 558     useful to invert the result of a grep. For example "not grep X" means
 559     succeed only if you don't find X in the input.</dd>
 560   </dl>
 561
 562   <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
 563   You can easily mark a test as XFAIL just by including  <tt>XFAIL: </tt> on a
 564   line near the top of the file. This signals that the test case should succeed
 565   if the test fails. Such test cases are counted separately by DejaGnu. To
 566   specify an expected fail, use the XFAIL keyword in the comments of the test
 567   program followed by a colon and one or more regular expressions (separated by
 568   a comma). The regular expressions allow you to XFAIL the test conditionally
 569   by host platform. The regular expressions following the : are matched against
 570   the target triplet or llvmgcc version number for the host machine. If there is
 571   a match, the test is expected to fail. If not, the test is expected to
 572   succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. When matching
 573   the llvm-gcc version, you can specify the major (e.g. 3) or full version
 574   (i.e. 3.4) number. Here is an example of an <tt>XFAIL</tt> line:</p>
 575
 576 <div class="doc_code">
 577 <pre>
 578 ; XFAIL: darwin,sun,llvmgcc4
 579 </pre>
 580 </div>
 581
 582   <p>To make the output more useful, the <tt>llvm_runtest</tt> function wil
 583   scan the lines of the test case for ones that contain a pattern that matches
 584   PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
 585   is related to the test case. The numer after "PR" specifies the LLVM bugzilla
 586   number. When a PR number is specified, it will be used in the pass/fail
 587   reporting. This is useful to quickly get some context when a test fails.</p>
 588
 589   <p>Finally, any line that contains "END." will cause the special
 590   interpretation of lines to terminate. This is generally done right after the
 591   last RUN: line. This has two side effects: (a) it prevents special
 592   interpretation of lines that are part of the test program, not the
 593   instructions to the test case, and (b) it speeds things up for really big test
 594   cases by avoiding interpretation of the remainder of the file.</p>
 595
 596 </div>
 597
 598 <!--=========================================================================-->
 599 <div class="doc_section"><a name="progstructure"><tt>llvm-test</tt>
 600 Structure</a></div>
 601 <!--=========================================================================-->
 602
 603 <div class="doc_text">
 604
 605 <p>As mentioned previously, the <tt>llvm-test</tt> module  provides three types
 606 of tests: MultiSource, SingleSource, and External.  Each tree is then subdivided
 607 into several categories, including applications, benchmarks, regression tests,
 608 code that is strange grammatically, etc.  These organizations should be
 609 relatively self explanatory.</p>
 610
 611 <p>In addition to the regular "whole program"  tests, the <tt>llvm-test</tt>
 612 module also provides a mechanism for compiling the programs in different ways.
 613 If the variable TEST is defined on the gmake command line, the test system will
 614 include a Makefile named <tt>TEST.&lt;value of TEST variable&gt;.Makefile</tt>.
 615 This Makefile can modify build rules to yield different results.</p>
 616
 617 <p>For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt> to
 618 create the nightly test reports.  To run the nightly tests, run <tt>gmake
 619 TEST=nightly</tt>.</p>
 620
 621 <p>There are several TEST Makefiles available in the tree.  Some of them are
 622 designed for internal LLVM research and will not work outside of the LLVM
 623 research group.  They may still be valuable, however, as a guide to writing your
 624 own TEST Makefile for any optimization or analysis passes that you develop with
 625 LLVM.</p>
 626
 627 <p>Note, when configuring the <tt>llvm-test</tt> module, you might want to
 628 specify the following configuration options:</p>
 629 <dl>
 630   <dt><i>--enable-spec2000</i>
 631   <dt><i>--enable-spec2000=&lt;<tt>directory</tt>&gt;</i>
 632   <dd>
 633     Enable the use of SPEC2000 when testing LLVM.  This is disabled by default
 634     (unless <tt>configure</tt> finds SPEC2000 installed).  By specifying
 635     <tt>directory</tt>, you can tell configure where to find the SPEC2000
 636     benchmarks.  If <tt>directory</tt> is left unspecified, <tt>configure</tt>
 637     uses the default value
 638     <tt>/home/vadve/shared/benchmarks/speccpu2000/benchspec</tt>.
 639     <p>
 640
 641   <dt><i>--enable-spec95</i>
 642   <dt><i>--enable-spec95=&lt;<tt>directory</tt>&gt;</i>
 643   <dd>
 644     Enable the use of SPEC95 when testing LLVM.  It is similar to the
 645     <i>--enable-spec2000</i> option.
 646     <p>
 647
 648   <dt><i>--enable-povray</i>
 649   <dt><i>--enable-povray=&lt;<tt>directory</tt>&gt;</i>
 650   <dd>
 651     Enable the use of Povray as an external test.  Versions of Povray written
 652     in C should work.  This option is similar to the <i>--enable-spec2000</i>
 653     option.
 654 </dl>
 655 </div>
 656
 657 <!--=========================================================================-->
 658 <div class="doc_section"><a name="run">Running the LLVM Tests</a></div>
 659 <!--=========================================================================-->
 660
 661 <div class="doc_text">
 662
 663 <p>First, all tests are executed within the LLVM object directory tree.  They
 664 <i>are not</i> executed inside of the LLVM source tree. This is because the
 665 test suite creates temporary files during execution.</p>
 666
 667 <p>The master Makefile in <tt>llvm/test</tt> is capable of running only the
 668 DejaGNU driven tests. By default, it will run all of these tests.</p>
 669
 670 <p>To run only the DejaGNU driven tests, run <tt>gmake</tt> at the
 671 command line in <tt>llvm/test</tt>.  To run a specific directory of tests, use
 672 the <tt>TESTSUITE</tt> variable.
 673 </p>
 674
 675 <p>For example, to run the Regression tests, type
 676 <tt>gmake TESTSUITE=Regression</tt> in <tt>llvm/tests</tt>.</p>
 677
 678 <p>Note that there are no Makefiles in <tt>llvm/test/Features</tt> and
 679 <tt>llvm/test/Regression</tt>. You must use DejaGNU from the <tt>llvm/test</tt>
 680 directory to run them.</p>
 681
 682 <p>To run the <tt>llvm-test</tt> suite, you need to use the following steps:</p>
 683
 684 <ol>
 685   <li><tt>cd</tt> into the <tt>llvm/projects</tt> directory</li>
 686
 687   <li><p>Check out the <tt>test-suite</tt> module with:</p>
 688
 689 <div class="doc_code">
 690 <pre>
 691 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk llvm-test
 692 </pre>
 693 </div>
 694
 695       <p>This will get the test suite into <tt>llvm/projects/llvm-test</tt></p>
 696
 697   <li><p>Configure the test suite. You can do this one of two ways:</p>
 698
 699   <ol>
 700     <li>Use the regular llvm configure:<br/><br/>
 701
 702 <div class="doc_code">
 703 <pre>
 704 % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure
 705 </pre>
 706 </div>
 707
 708         <p>This will ensure that the <tt>projects/llvm-test</tt> directory is
 709            also properly configured.</p></li>
 710
 711     <li><p>Use the <tt>configure</tt> script found in the <tt>llvm-test</tt>
 712         source directory:</p>
 713
 714 <div class="doc_code">
 715 <pre>
 716 % $LLVM_SRC_ROOT/projects/llvm-test/configure \
 717   --with-llvmsrc=$LLVM_SRC_ROOT               \
 718   --with-llvmobj=$LLVM_OBJ_ROOT
 719 </pre>
 720 </div>
 721     </li>
 722   </ol>
 723   <li><tt>gmake</tt></li>
 724 </ol>
 725 <p>Note that the second and third steps only need to be done once. After you
 726 have the suite checked out and configured, you don't need to do it again (unless
 727 the test code or configure script changes).</p>
 728
 729 <p>To make a specialized test (use one of the
 730 <tt>llvm-test/TEST.&lt;type&gt;.Makefile</tt>s), just run:</p>
 731
 732 <div class="doc_code">
 733 <pre>
 734 % gmake TEST=&lt;type&gt; test
 735 </pre>
 736 </div>
 737
 738 <p>For example, you could run the nightly tester tests using the following
 739 commands:</p>
 740
 741 <div class="doc_code">
 742 <pre>
 743 % cd llvm/projects/llvm-test
 744 % gmake TEST=nightly test
 745 </pre>
 746 </div>
 747
 748 <p>Regardless of which test you're running, the results are printed on standard
 749 output and standard error.  You can redirect these results to a file if you
 750 choose.</p>
 751
 752 <p>Some tests are known to fail.  Some are bugs that we have not fixed yet;
 753 others are features that we haven't added yet (or may never add).  In DejaGNU,
 754 the result for such tests will be XFAIL (eXpected FAILure).  In this way, you
 755 can tell the difference between an expected and unexpected failure.</p>
 756
 757 <p>The tests in <tt>llvm-test</tt> have no such feature at this time. If the
 758 test passes, only warnings and other miscellaneous output will be generated.  If
 759 a test fails, a large &lt;program&gt; FAILED message will be displayed.  This
 760 will help you separate benign warnings from actual test failures.</p>
 761
 762 </div>
 763
 764 <!-- _______________________________________________________________________ -->
 765 <div class="doc_subsection">
 766 <a name="customtest">Writing custom tests for llvm-test</a></div>
 767 <!-- _______________________________________________________________________ -->
 768
 769 <div class="doc_text">
 770
 771 <p>Assuming you can run llvm-test, (e.g. "<tt>gmake TEST=nightly report</tt>"
 772 should work), it is really easy to run optimizations or code generator
 773 components against every program in the tree, collecting statistics or running
 774 custom checks for correctness.  At base, this is how the nightly tester works,
 775 it's just one example of a general framework.</p>
 776
 777 <p>Lets say that you have an LLVM optimization pass, and you want to see how
 778 many times it triggers.  First thing you should do is add an LLVM
 779 <a href="ProgrammersManual.html#Statistic">statistic</a> to your pass, which
 780 will tally counts of things you care about.</p>
 781
 782 <p>Following this, you can set up a test and a report that collects these and
 783 formats them for easy viewing.  This consists of two files, an
 784 "<tt>llvm-test/TEST.XXX.Makefile</tt>" fragment (where XXX is the name of your
 785 test) and an "<tt>llvm-test/TEST.XXX.report</tt>" file that indicates how to
 786 format the output into a table.  There are many example reports of various
 787 levels of sophistication included with llvm-test, and the framework is very
 788 general.</p>
 789
 790 <p>If you are interested in testing an optimization pass, check out the
 791 "libcalls" test as an example.  It can be run like this:<p>
 792
 793 <div class="doc_code">
 794 <pre>
 795 % cd llvm/projects/llvm-test/MultiSource/Benchmarks  # or some other level
 796 % make TEST=libcalls report
 797 </pre>
 798 </div>
 799
 800 <p>This will do a bunch of stuff, then eventually print a table like this:</p>
 801
 802 <div class="doc_code">
 803 <pre>
 804 Name                                  | total | #exit |
 805 ...
 806 FreeBench/analyzer/analyzer           | 51    | 6     |
 807 FreeBench/fourinarow/fourinarow       | 1     | 1     |
 808 FreeBench/neural/neural               | 19    | 9     |
 809 FreeBench/pifft/pifft                 | 5     | 3     |
 810 MallocBench/cfrac/cfrac               | 1     | *     |
 811 MallocBench/espresso/espresso         | 52    | 12    |
 812 MallocBench/gs/gs                     | 4     | *     |
 813 Prolangs-C/TimberWolfMC/timberwolfmc  | 302   | *     |
 814 Prolangs-C/agrep/agrep                | 33    | 12    |
 815 Prolangs-C/allroots/allroots          | *     | *     |
 816 Prolangs-C/assembler/assembler        | 47    | *     |
 817 Prolangs-C/bison/mybison              | 74    | *     |
 818 ...
 819 </pre>
 820 </div>
 821
 822 <p>This basically is grepping the -stats output and displaying it in a table.
 823 You can also use the "TEST=libcalls report.html" target to get the table in HTML
 824 form, similarly for report.csv and report.tex.</p>
 825
 826 <p>The source for this is in llvm-test/TEST.libcalls.*.  The format is pretty
 827 simple: the Makefile indicates how to run the test (in this case,
 828 "<tt>opt -simplify-libcalls -stats</tt>"), and the report contains one line for
 829 each column of the output.  The first value is the header for the column and the
 830 second is the regex to grep the output of the command for.  There are lots of
 831 example reports that can do fancy stuff.</p>
 832
 833 </div>
 834
 835
 836 <!--=========================================================================-->
 837 <div class="doc_section"><a name="nightly">Running the nightly tester</a></div>
 838 <!--=========================================================================-->
 839
 840 <div class="doc_text">
 841
 842 <p>
 843 The <a href="http://llvm.org/nightlytest/">LLVM Nightly Testers</a>
 844 automatically check out an LLVM tree, build it, run the "nightly"
 845 program test (described above), run all of the feature and regression tests,
 846 delete the checked out tree, and then submit the results to
 847 <a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>.
 848 After test results are submitted to
 849 <a href="http://llvm.org/nightlytest/">http://llvm.org/nightlytest/</a>,
 850 they are processed and displayed on the tests page. An email to
 851 <a href="http://lists.cs.uiuc.edu/pipermail/llvm-testresults/">
 852 llvm-testresults@cs.uiuc.edu</a> summarizing the results is also generated.
 853 This testing scheme is designed to ensure that programs don't break as well
 854 as keep track of LLVM's progress over time.</p>
 855
 856 <p>If you'd like to set up an instance of the nightly tester to run on your
 857 machine, take a look at the comments at the top of the
 858 <tt>utils/NewNightlyTest.pl</tt> file. If you decide to set up a nightly tester
 859 please choose a unique nickname and invoke <tt>utils/NewNightlyTest.pl</tt>
 860 with the "-nickname [yournickname]" command line option.
 861
 862 <p>You can create a shell script to encapsulate the running of the script.
 863 The optimized x86 Linux nightly test is run from just such a script:</p>
 864
 865 <div class="doc_code">
 866 <pre>
 867 #!/bin/bash
 868 BASE=/proj/work/llvm/nightlytest
 869 export BUILDDIR=$BASE/build
 870 export WEBDIR=$BASE/testresults
 871 export LLVMGCCDIR=/proj/work/llvm/cfrontend/install
 872 export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH
 873 export LD_LIBRARY_PATH=/proj/install/lib
 874 cd $BASE
 875 cp /proj/work/llvm/llvm/utils/NewNightlyTest.pl .
 876 nice ./NewNightlyTest.pl -nice -release -verbose -parallel -enable-linscan \
 877    -nickname NightlyTester -noexternals &gt; output.log 2&gt;&amp;1
 878 </pre>
 879 </div>
 880
 881 <p>It is also possible to specify the the location your nightly test results
 882 are submitted. You can do this by passing the command line option
 883 "-submit-server [server_address]" and "-submit-script [script_on_server]" to
 884 <tt>utils/NewNightlyTest.pl</tt>. For example, to submit to the llvm.org
 885 nightly test results page, you would invoke the nightly test script with
 886 "-submit-server llvm.org -submit-script /nightlytest/NightlyTestAccept.cgi".
 887 If these options are not specified, the nightly test script sends the results
 888 to the llvm.org nightly test results page.</p>
 889
 890 <p>Take a look at the <tt>NewNightlyTest.pl</tt> file to see what all of the
 891 flags and strings do.  If you start running the nightly tests, please let us
 892 know. Thanks!</p>
 893
 894 </div>
 895
 896 <!-- *********************************************************************** -->
 897
 898 <hr>
 899 <address>
 900   <a href="http://jigsaw.w3.org/css-validator/check/referer"><img
 901   src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
 902   <a href="http://validator.w3.org/check/referer"><img
 903   src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" /></a>
 904
 905   John T. Criswell, Reid Spencer, and Tanya Lattner<br>
 906   <a href="http://llvm.org">The LLVM Compiler Infrastructure</a><br/>
 907   Last modified: $Date$
 908 </address>
 909 </body>
 910 </html>