layout/doc/regression_tests.html

   1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
   2 <!-- ***** BEGIN LICENSE BLOCK *****
   3    - Version: MPL 1.1/GPL 2.0/LGPL 2.1
   4    -
   5    - The contents of this file are subject to the Mozilla Public License Version
   6    - 1.1 (the "License"); you may not use this file except in compliance with
   7    - the License. You may obtain a copy of the License at
   8    - http://www.mozilla.org/MPL/
   9    -
  10    - Software distributed under the License is distributed on an "AS IS" basis,
  11    - WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
  12    - for the specific language governing rights and limitations under the
  13    - License.
  14    -
  15    - The Original Code is a description how to run the layout regression tests.
  16    -
  17    - The Initial Developer of the Original Code is
  18    - Bernd Mielke.
  19    - Portions created by the Initial Developer are Copyright (C) 2___
  20    - the Initial Developer. All Rights Reserved.
  21    -
  22    - Contributor(s):
  23    -
  24    - Alternatively, the contents of this file may be used under the terms of
  25    - either the GNU General Public License Version 2 or later (the "GPL"), or
  26    - the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
  27    - in which case the provisions of the GPL or the LGPL are applicable instead
  28    - of those above. If you wish to allow use of your version of this file only
  29    - under the terms of either the GPL or the LGPL, and not to allow others to
  30    - use your version of this file under the terms of the MPL, indicate your
  31    - decision by deleting the provisions above and replace them with the notice
  32    - and other provisions required by the LGPL or the GPL. If you do not delete
  33    - the provisions above, a recipient may use your version of this file under
  34    - the terms of any one of the MPL, the GPL or the LGPL.
  35    -
  36    - ***** END LICENSE BLOCK ***** -->
  37
  38 <html>
  39 <head>
  40    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
  41      <title>Layout Regression tests</title>
  42 <style type="text/css">
  43   body {font-family:arial}
  44 .screen {background-color:silver; color:blue}
  45 pre.screen {display:table-cell}
  46  </style>
  47
  48 </head>
  49 <body>
  50
  51 <h1>Table Regression Tests</h1>
  52 <h6>Bernd Mielke</h6>
  53 <h2>About This Document</h2>
  54 <p>
  55 This document describes my guerrilla approach to successfully  running the table layout regression tests. It should encourage more people to run those tests.
  56
  57 <h2>Subject Overview</h2>
  58 <p>
  59 The table element and its children are one of the major layout components that can create some structure on the screen. They are widely (ab)used  by page authors. Every small change in the table code can generate a bunch of bugzilla entries and a lot of seemingly unrelated dupes. With the high frequency of checkins in to the source tree  it is difficult and time consuming to track the regression down to a single checkin. The risk of becoming very infamous can be reduced by running the layout regression tests.
  60
  61 <h2>Prerequisites</h2>
  62 <p>
  63 In order to run these tests, you will need to have:
  64 <ol>
  65 <li>a debug build with a working viewer, an optimized build will not work!
  66 <li>a mozilla tree with the testfiles at <code>%MOZ_SRC%/layout/html/tests</code>,
  67 <li> <code>DISABLE_TESTS</code> should not be defined,
  68 <li>patience and time.
  69 </ol>
  70
  71 <h2>Test Runs</h2>
  72
  73 <ul>
  74 <li>Go to the <code>%MOZ_SRC%/layout/html/tests/block</code>  directory.
  75 <li> Avoid the nasty assertions to pop up by <code>set XPCOM_DEBUG_BREAK=warn</code>, the world would be a much better place if we could <a href="http://bugzilla.mozilla.org/show_bug.cgi?id=152015">first fix those assertions</a>.
  76 <li>The following shell scripts need a <code>sh</code> before the command under WinXX.
  77 <li>Execute  <code>rtest.sh baseline</code>
  78 <li>Execute  <code>rtest.sh verify&gt;outputfile.txt</code>
  79 <li>Make a note which tests have failed (these are  false positives. If you are in bad mood file a bug against Chris Waterson about them)
  80 <li>you can grep the failures by:<br>
  81 <code>grep 'rgd failed' &lt;outputfile.txt &gt;false_positive.txt</code>
  82 <li> Make a copy of <code>gklayout.dll</code> and <code>gkcontent.dll</code> in the <code>dist</code> directory. If you would like to switch back, you will need them.
  83 <li>Make your changes to the source, compile and install.
  84 <li>execute in the <code>layout/html/tests/block</code>  directory <code>rtest.sh verify &gt;regression.txt</code>
  85 <li>Check how many tests have failed and analyse your results:
  86 <ul>
  87 <li>If the regression tests have failed at the same place as the previous diagnosed false positives =&gt; ignore
  88 <ol>
  89 <li>copy  <code>rtest.sh</code>  to <code>my_rtest.sh</code>
  90 <li>create a new directory <code>my_files</code>.
  91 <li>create in this directory a file <code>file_list.txt</code>, where you put the url's of your remaining test failures
  92 <li>change <code>my_rtest.sh</code> to use only directory <code>my_files</code>
  93 <li>go back to the original gklayout.dll and gkcontent.dll and repeat the regression test procedure on a much smaller set now.
  94 </ol>
  95 <li>Try to figure out what the other regressions are: Are they
  96      improving the picture or are they regressions?
  97 <li>Note the difficult to explain testfailures.
  98 </ul>
  99 <li> Change the source if necessary until the regression tests indicate that your patch does not hork the block and table layout
 100 <li> Submit your patch and lists the remaining differences in the bug
 101 </ul>
 102
 103 <p>Be prepared to cycle.
 104
 105 <h2>How the layout regression tests work</h2>
 106
 107 <p>First the URL's to test are red from <code>file_list.txt</code>.<p>
 108 <table><caption>snippet from file_list.txt</caption><tr><td>
 109 <pre>file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html
 110 file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html
 111 file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html
 112 file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html
 113 file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html
 114 file:///s|/mozilla/layout/html/tests/table/dom/appendRowsExpand1.html</pre>
 115 </td></tr></table>
 116 <p>The regression tests compare the frame tree dumps recorded at two different times. The URL's to test are red from <code>file_list.txt</code>
 117 For every file in <code>file_list.txt</code>viewer takes the root frame and dumps the whole tree via
 118 <a href="http://lxr.mozilla.org/seamonkey/search?string=nsFrame%3A%3ADumpRegression">
 119 nsFrame::DumpRegression</a>.
 120 <p>
 121  The frame tree dumps are  invoked by the <code>-o</code> switch to the <code>viewer</code> (see <a href="http://lxr.mozilla.org/seamonkey/ident?i=PrintHelpInfo">commandline arguments </a>).
 122 When <code>-o</code> is specified  together with a output directory for instance (<code>-o s:\mozilla\layout\html\tests\table\dom\</code>),
 123 the frame trees are dumped to separate files in that directory for all URL's in <code>file_list.txt</code> (see:
 124 <a href="http://lxr.mozilla.org/seamonkey/search?string=nsWebCrawler%3A%3ADumpRegression">
 125 nsWebCrawler::DumpRegressionData</a>) .
 126 <p>Their name is generated by replacing the old extension <code>.html</code> or <code>.xml</code> with <code>.rgd</code>.  For <code>file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html</code> a file <code>appendCells1.rgd</code> will be created in the <code>s:\mozilla\layout\html\tests\table\dom\</code> directory.
 127
 128 <p>
 129  There is a special type of frame dumps - the printing regression tests, they are invoked by the <code>-Prt</code> command line argument to the <code>viewer</code> and include first a display then a frame dump taking into account the printer pages. A typical beginning of a dump (<code>*.rgd</code> file) looks like:<p>
 130 <pre class="screen">
 131 &lt;frame va="15022440" type="Viewport(-1)" state="270340" parent="0"&gt;
 132   &lt;view va="47171904"&gt;
 133   &lt;/view&gt;
 134   &lt;stylecontext va="15022232"&gt;
 135     &lt;font serif 240 240 0 /&gt;
 136     &lt;color data="-16777216"/&gt;
 137     &lt;background data="0 2 3 -1 0 0 "/&gt;
 138     &lt;spacing data="left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0" /&gt;
 139     &lt;list data="100 100 " /&gt;
 140     &lt;position data="left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  " /&gt;
 141     &lt;text data="0 0 0 Normal  Normal  0[0x0]tw  Normal  " /&gt;
 142     &lt;textreset data="0 10[0xa]enum  " /&gt;
 143     &lt;display data="0 1 0 0 0 0 0 0 0 0 0 0 " /&gt;
 144     &lt;visibility data="0 1 1.000000" /&gt;
 145     &lt;table data="0 0 4 -1 1 " /&gt;
 146     &lt;tableborder data="1 Null  Null  0 2 " /&gt;
 147     &lt;content data="0 0 0 Null  " /&gt;
 148     &lt;quotes data="0 " /&gt;
 149     &lt;ui data="3 0 0 1 " /&gt;
 150     &lt;uireset data="7 0 4" /&gt;
 151     &lt;xul data="0 0 0 0 0 1    &lt;svg data="0 1.000000 1.000000 0 1.000000" /&gt;
 152   &lt;/stylecontext&gt;</pre>
 153 <p>The first run of the regression tests is started with the argument <code>baseline</code> and it creates the reference frame dumps.
 154 <p>The <code>baseline</code> log will look like:
 155 <pre class="screen">
 156 Type Manifest File: C:\MOZ_SOUR\MOZILLA\MOZILLA\DIST\WIN32_D.OBJ\BIN\components\xpti.dat
 157 nsNativeComponentLoader: autoregistering begins.
 158 nsNativeComponentLoader: autoregistering succeeded
 159 Going to create the event queue
 160 WEBSHELL+ = 1
 161 Note: verifyreflow is disabled
 162 Note: styleverifytree is disabled
 163 Note: frameverifytree is disabled
 164 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html: done loading (11810 msec)
 165 appendCells1.rgd - being written
 166 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html: done loading (1260 msec)
 167 appendCellsRebuild1.rgd - being written
 168 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html: done loading (270 msec)
 169 appendCol1.rgd - being written
 170 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html: done loading (220 msec)
 171 appendCol2.rgd - being written
 172 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html: done loading (390 msec)
 173 appendColGroup1.rgd - being written
 174 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendRowsExpand1.html: done loading (220 msec)
 175 appendRowsExpand1.rgd - being written
 176 </pre>
 177 <p>
 178 The second run with the argument <code>verify</code> creates first the frame dumps in a subdirectory (<code>verify</code>) and <a href="http://lxr.mozilla.org/seamonkey/search?string=nsFrameUtil%3A%3ACompareTrees">compares</a> each frame dump with the corresponding reference frame dump. If they differ in critical points the test fails.
 179 <p>A  typical part of the <code>verify</code> log would look like:
 180 <pre class="screen">
 181 Type Manifest File: C:\MOZ_SOUR\MOZILLA\MOZILLA\DIST\WIN32_D.OBJ\BIN\components\xpti.dat
 182 nsNativeComponentLoader: autoregistering begins.
 183 nsNativeComponentLoader: autoregistering succeeded
 184 Going to create the event queue
 185 WEBSHELL+ = 1
 186 Note: verifyreflow is disabled
 187 Note: styleverifytree is disabled
 188 Note: frameverifytree is disabled
 189 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCells1.html: done loading (2200 msec)
 190 frame bbox mismatch: 0,285,930,435 vs. 0,285,1305,435
 191 Node 1:
 192   TableOuter(table)(1) 0x10004 0,285,930,435, |null attr|-16777216|left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Inherit top: Inherit right: Inherit bottom: Inherit  0[0x0]tw  0[0x0]tw  Null  0[0x0]tw  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 1 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 193 Node 2:
 194   TableOuter(table)(1) 0x10004 0,285,1305,435, |null attr|-16777216|left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 0[0x0]tw top: 0[0x0]tw right: 0[0x0]tw bottom: 0[0x0]tw  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Inherit top: Inherit right: Inherit bottom: Inherit  0[0x0]tw  0[0x0]tw  Null  0[0x0]tw  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 1 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 195 frame bbox mismatch: 0,0,930,435 vs. 0,0,1305,435
 196 Node 1:
 197   Table(table)(1) 0x10004 0,0,930,435, |null attr|-16777216|left: Null top: 0[0x0]tw right: Null bottom: 0[0x0]tw  left: Null top: Null right: Null bottom: Null  left: 15[0xf]tw top: 15[0xf]tw right: 15[0xf]tw bottom: 15[0xf]tw  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  2 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 0 0 0 0 0 |0 8 4 -1 1 |0 0 0 Null
 198 Node 2:
 199   Table(table)(1) 0x10004 0,0,1305,435, |null attr|-16777216|left: Null top: 0[0x0]tw right: Null bottom: 0[0x0]tw  left: Null top: Null right: Null bottom: Null  left: 15[0xf]tw top: 15[0xf]tw right: 15[0xf]tw bottom: 15[0xf]tw  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  2 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 8 0 0 0 0 0 0 0 0 0 0 |0 8 4 -1 1 |0 0 0 Null
 200 frame bbox mismatch: 45,15,840,405 vs. 45,15,1215,405
 201 Node 1:
 202   TableColGroup(table)(1) 0x80010004 45,15,840,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 12 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 203 Node 2:
 204   TableColGroup(table)(1) 0x80010004 45,15,1215,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 12 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 205 frame state mismatch: 0x30010406 vs. 0x30010004
 206 Node 1:
 207   TableCol(table)(1) 0x30010406 0,0,0,0, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 208 Node 2:
 209   TableCol(table)(1) 0x30010004 870,0,345,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 210 frame bbox mismatch: 0,0,0,0 vs. 870,0,345,405
 211 Node 1:
 212   TableCol(table)(1) 0x30010406 0,0,0,0, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 213 Node 2:
 214   TableCol(table)(1) 0x30010004 870,0,345,405, |null attr|-16777216|left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null left: 1[0x1]enum top: 1[0x1]enum right: 1[0x1]enum bottom: 1[0x1]enum  left: Null top: Null right: Null bottom: Null  left: Null top: Null right: Null bottom: Null  1[0x1]enum 0|100 100 |left: Auto top: Auto right: Auto bottom: Auto  Auto  0[0x0]tw  Null  Auto  0[0x0]tw  Null  0 Auto  |0 0 0 Normal  Normal  0[0x0]tw  Normal  |0 11 0 0 0 0 0 0 0 0 0 0 |0 0 4 -1 1 |0 0 0 Null
 215 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCells1.rgd failed
 216 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCellsRebuild1.html: done loading (390 msec)
 217 044regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCellsRebuild1.rgd passed
 218 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol1.html: done loading (280 msec)
 219 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCol1.rgd passed
 220 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendCol2.html: done loading (220 msec)
 221 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendCol2.rgd passed
 222 +++ file:///s|/mozilla/layout/html/tests/table/dom/appendColGroup1.html: done loading (220 msec)
 223 regression test s:\mozilla\layout\html\tests\table\dom\verify\appendColGroup1.rgd passed
 224 </pre>
 225 <p>
 226 The first regression test has failed and it is marked so.
 227
 228
 229 <h2>Adding new regression tests</h2>
 230 <p>
 231 Once you have checked in the code:
 232 <ul>
 233 <li> please add your testcase for the bug to the regression tests.
 234 <li>A testcase should only contain local files.
 235 <li>Place the images in <code>table/images</code> (don't forget <code> cvs commit -kb </code>)
 236 <li>and the testfile in the <code>table/bugs</code> directory.
 237 <li>Update the <code>file_list<b>x</b>.txt</code> in order to include your file.
 238 </ul>
 239 <p>I would like to thank Chris Karnaze for his guidance and fantasai for his language support.
 240 <div style="text-align:right"><i>&lt;regression.html&gt;&nbsp; &lt;Last updated:
 241 2002-06-16 <a href="mailto:bernd.mielke@snafu.de">Bernd Mielke</a></i></div>
 242
 243 </body>
 244 </html>