ACE/docs/bczar/bczar.html

   1 <!--  -->
   2 <body>
   3   <html>
   4     <head>
   5       <title>The realm of the build czar</title>
   6     </head>
   7     <h2>Build Czar Duties</h2>
   8     <p>
   9       The main duties of the Build Czar are summarized as follows
  10       <li>
  11         Continuously monitor the builds using the <a href="http://www.dre.vanderbilt.edu/scoreboard">
  12           Scoreboard </a>
  13       as one of the primary source of information.
  14       <li>
  15       Notify developers who broke compilation to fix the errors as soon as possible,
  16       ideally by the next day. A red color in the "Compile" column is not at all
  17       acceptable - the Build Czar needs to ensure that these problems are identified
  18       and fixed in a timely manner. If possible, the Build Czar should let developers
  19       know what the source of problems might be. It is quite possible that developers
  20       who checked in the code or users who provided the patch may not have resources
  21       to investigate the issues, so the Builds Czar's help is essential to keep
  22       things moving ahead.
  23       <li>
  24       Keep an eye on the tests that are run in every build. Anything abnormal needs
  25       to be notified to the right developer. The Build Czar should try helping the
  26       developer by providing stack traces (in case of crashes) or other details like
  27       printouts with debugging level turned on.
  28       <li>
  29         Some tests fail in the daily builds for many reasons like known bugs, transient
  30         timeouts etc. Make sure that no new test failures show up. This <a href="mailto:jwillemsen@remedy.nl">
  31           guy</a>
  32       knows most of the information. Ask him to help you out with known problems.
  33       Keep the builds ticking. Any red on the "Last Finished" column in the
  34       Scoreboard should be fixed. The link to the "Build Name" indicates the machine
  35       where the build is being run.
  36       <li>
  37       The builds don't cover all the possible configurations. If you get a bug report
  38       about a compile error in a particular configuration, try setting up a build to
  39       make sure that it doesn't show up again if it has been fixed.
  40     </p>
  41     <P>
  42       The document <a href="./privileges.html">here </a>talks about the powers of a
  43       build Czar.
  44     </P>
  45     <P>
  46       The Build Czar is empowered to set up more builds on his own for his
  47       convenience. This <a href="https://github.com/DOCGroup/autobuild/blob/master/README.md">
  48         page </a>has a step by step instructions on how to do that.
  49     </P>
  50     <P>
  51       The build czar can get the build configuration by looking at the config portion
  52       of the scoreboard.
  53     </P>
  54     <p>Pro-active involvement by the build czar is necessary. Being a pro-active
  55       build czar requires monitoring the git repository carefully and responding
  56       quickly to suspected changes to keep the repo stays stable.</p>
  57     <p>After an email from the Coverity scan project check the Coverity project and
  58        check that there are no new errors reported. If so, contact the person who
  59        introduced the error.</p>
  60     <p>Check the daily test statistics email to determine any drop in test failures.
  61        Do note that when there are a lot of compile errors the test results are really
  62        unusable.</p>
  63     <hr>
  64     <h2>Recipe for Cutting a Minor Kit</h2>
  65     <P>
  66       The build czar is also in charge for the release of the micro. Cutting a micro is
  67       as simple as cutting butter if things go well. Here is the procedure followed
  68       while cutting a micro:
  69       <ol>
  70         <li>
  71           The whole process takes somewhere between 2-3 hours.</li>
  72         <li>
  73           We suggest you take advantage of GNU Screen so that even if your SSH session is
  74           interrupted, the cutting process can continue. This command must be installed on
  75           the machines we use to cut the release.
  76           <ul>
  77             <li>
  78               type <code>screen</code> to start screen.</li>
  79             <li>
  80               execute commands as normal. Note that Ctrl-A is special in screen, so you need
  81               to type Ctrl-A-A to send a Ctrl-A to the shell</li>
  82             <li>
  83               should your session be interrupted, reconnect and type <code>screen -x</code></li>
  84             <li>
  85               when finished, just type exit twice</li>
  86           </ul>
  87           <li>
  88             After login check that you can,
  89             <br>
  90             <code>ssh bczar@download.dre.vanderbilt.edu</code><br>
  91             to ensure that this succeeds. If not fix the problem, if ok exit again back to
  92         <li>
  93           Prior to starting this, gather aggregate release notes from all developers.
  94           This is usually in the form of an email plea asking to update all NEWS files in
  95           the archive. These NEWS files are used as part of the release notes for the release.</li>
  96         <li>
  97           Make sure your release system has all the needed tools. This can be achieved on Fedora
  98           using:
  99           <ul>
 100             <li><code>yum install perl screen automake doxygen bzip2 tar gzip openssh graphviz zip libtool GitPython python3</code></li>
 101             <li><code>yum update</code></li>
 102           </ul>
 103           or on OpenSuSE
 104           <ul>
 105             <li><code>zypper install perl screen automake doxygen bzip2 tar gzip openssh graphviz zip libtool python-gitpython python3</code></li>
 106             <li><code>zypper update</code></li>
 107           </ul>
 108           If you want to perform a full build with qt support, than run:
 109           <ul>
 110             <li><code>yum install deltarpm ntp rubygem-rmagick bison xerces-c-devel psmisc yum-utils gdb unzip glibc-devel libasan bison redhat-lsb perl-Pod-Usage rubygems clang make patch libcgroup-devel ant setuptool system-config-network-tui system-config-firewall-tui lcov gnuplot java-1.8.0-openjdk perl screen automake doxygen bzip2 tar gzip openssh graphviz zip libtool gcc-c++ boost-devel valgrind openssl-devel gcc qt4 fltk-devel bzip2-devel rsync openssl lzo-devel zziplib-devel acpid acpi nfs-utils java xerces-c xerces-c-devel mc qt qt-devel icecream ruby ruby-devel lksctp-tools-devel git telnet GitPython NetworkManager wget mailx python3</code></li>
 111           </ul>
 112           For some optional i686 packages run
 113           <ul>
 114             <li><code>yum install openssl-devel.i686 xerces-c-devel.i686 glibc-devel.i686 glibc.i686 libstdc++.i686 ncurses-libs.i686 zlib.i686</code></li>
 115           </ul>
 116         </li>
 117         <li>
 118           Setup your ssh access to github according to the github <a href="https://help.github.com/articles/generating-ssh-keys/">ssh documentation</a>
 119         </li>
 120         <li>
 121           Checkout a new workspace on a Fedora system with the last public release and with
 122           all patches installed.
 123         </li>
 124         <ul>
 125           <li>
 126             The best place to create the workspace is under <code>/export/anduriltmp/bczar</code> (if you are on anduril). Don't
 127             use the home directory itself, it is an NFS share and not really fast.
 128           </li>
 129           <li>
 130             Checkout like this:
 131             <ul>
 132               <li>
 133                 <code>git clone https://github.com/DOCGroup/ACE_TAO.git ACE_TAO</code></li>
 134               <li>
 135                 <code>git clone https://github.com/DOCGroup/MPC.git MPC</code></li>
 136             </ul>
 137         </ul>
 138         <li>
 139           Set $DOC_ROOT to point to the new workspace you checked out.</li>
 140         <li>
 141           Change directories to <tt>$DOC_ROOT</tt>
 142         </li>
 143         <li>
 144           Checkout the correct branch you want to release on, the default is <code>master</code>
 145           but you can also check out an different branch <code>ace6tao2</code> for the ACE 6.x/TAO 2.x
 146           releases.
 147         </li>
 148         <li>
 149           Tag the release by executing<br>
 150           <code>ACE/bin/make_release.py --micro --update --tag --push</code><br>
 151           This will only take a couple minutes to complete and once done successfully,
 152           you can carry on with BOTH creating the kits and generating the doxygen
 153           documentation in parallel. NOTE that <code>--micro</code> should be replaced
 154           with <code>--minor</code> or <code>--major</code> as appropriate.</li>
 155         <br>
 156         After the repository has been tagged check each file listed below to make
 157         sure version numbers are updated as expected.<br>
 158         <br>
 159         In the <em>EXTREMELY</em> unlikely event that something goes wrong during the <em>
 160           tagging</em> of the repo, the following files must be returned to the state
 161         they were in before the release process started and then checked back into git:<br>
 162         <ul>
 163           <code>
 164             <li>ACE/PROBLEM-REPORT-FORM</li>
 165             <li>ACE/VERSION.txt</li>
 166             <li>TAO/PROBLEM-REPORT-FORM</li>
 167             <li>TAO/VERSION.txt</li>
 168             <li>TAO/tao/Version.h</li>
 169             <li>ace/Version.h</li>
 170           </code>
 171         </ul>
 172     The tag will also need to be removed (both in Middleware and MPC):
 173     ACE+TAO-X_Y_Z (where X is the ACE Major version number, and Y & Z are the
 174     Minor and Micro release numbers of the release that is to be restarted).<p>
 175       E.g.:<br>
 176       <code>cd ACE_TAO && git tag -d ACE+TAO-X_Y_Z<br />
 177         cd MPC && git tag -d ACE+TAO-X_Y_Z<br />
 178       </code>
 179       <br>
 180       Note that this <em>only</em> needs to be done if the <em>tagging</em> fails. If
 181       kit creation fails, simply restart that process.
 182       <li>
 183         Create the kits by executing<br>
 184         <code>ACE/bin/make_release.py --kit</code><br>
 185         This will take somewhere around 1 hours to complete.
 186         <ul>
 187           <li>
 188             These commands only tags and creates the kits for the software itself, not
 189             documentation, this can be started in parallel with this activity.
 190           </li>
 191           <li>
 192             The kits end up in <tt>$DOC_ROOT/packages</tt></li>
 193         </ul>
 194     <p>
 195     To summarize, the following is a transcript of the steps up to this point
 196     executing successfully:
 197     <p><code>$ ssh ..<br>
 198         No default printer<br>
 199         screen<br>
 200         cd $HOME<br>
 201         rm -rf DOC_ROOT<br>
 202         mkdir DOC_ROOT<br>
 203         cd DOC_ROOT<br>
 204         export DOC_ROOT=$PWD<br>
 205         git clone https://github.com/DOCGroup/ACE_TAO.git ACE_TAO<br>
 206         git clone https://github.com/DOCGroup/MPC.git MPC<br>
 207         cd ACE_TAO<br>
 208         git checkout master</br>
 209         ACE/bin/make_release.py --micro --update --verbose<br>
 210         ACE/bin/make_release.py --micro --tag --verbose<br>
 211         ACE/bin/make_release.py --micro --push --verbose<br>
 212         ACE/bin/make_release.py --kit<br>
 213       </code>
 214     <p>
 215       Feel free to cut and paste with suitable edits.
 216       <li>
 217         The packages end up by default under $DOC_ROOT/package-&lt;PID&gt;, you can
 218         copy them to the webserver using the following commands. (Note that &lt;PID&gt;
 219         needs to be the numerical pid of the process that created the kit, use<br>
 220         <code>ls -ald</code>
 221         to determine the correct filename.) At the moment you execute these commands
 222         all users can download these packages.</li>
 223       <code>scp $DOC_ROOT/package-&lt;PID&gt;/ACE*
 224         bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution<br>
 225       </code>
 226       When releasing from a branch of master the resulting packages shouldn't be uploaded to
 227       <code>ACE+TAO-distribution</code> but <b>only</b> the versioned packages should be
 228       uploaded to <code>bczar@download.dre.vanderbilt.edu:/export/www/download.dre/previous_versions</code>
 229       using
 230       <code>scp $DOC_ROOT/package-&lt;PID&gt;/ACE*6*
 231         bczar@download.dre.vanderbilt.edu:/export/www/download.dre/previous_versions</code>
 232       <li>
 233         After the repository is tagged you can also start generating the doxygen
 234         documentation in parallel with the kit generation above.<br>
 235         <ul>
 236           <li>
 237             Login to a release system you prepared with the same packages as above:<br>
 238             <code>screen</code></li>
 239           <li>
 240             After login check that you can,
 241             <br>
 242             <code>ssh bczar@download.dre.vanderbilt.edu</code><br>
 243             to ensure that this succeeds. If not fix the problem, if ok exit again back to
 244             your release system. </li>
 245           <li>
 246             <code>cd /tmp</code><br>
 247             and remove the contents of the doxygen directory and recreate it again with</li><br>
 248           <code>rm -rf doxygen</code><br>
 249           <code>mkdir doxygen</code><br>
 250           <code>cd doxygen</code><br>
 251           If you create the doxygen documentation on <code>naboo.dre.vanderbilt.edu</code>
 252           than make sure you use <code>/web/users/isisbuilds/tmp/ACE_wrappers</code>
 253           as working directory
 254           <li>
 255             Update the workspace with the right version tag (replace the X_Y_Z with the ACE
 256             version number being released e.g. 7_0_5)
 257             <br>
 258             <code>git clone https://github.com/DOCGroup/ACE_TAO.git --depth 1 --branch ACE+TAO-7_0_5 ACE_TAO</code><br>
 259           </li>
 260           <li>Change to the ACE_TAO directory using <br>
 261           cd ACE_TAO</li>
 262           <li>
 263             Set the needed environment variables using<br>
 264             <code>export ACE_ROOT=$PWD/ACE</code><br>
 265         <code>export TAO_ROOT=$PWD/TAO</code><br>
 266         <code>export CIAO_ROOT=</code><br>
 267         <code>export DANCE_ROOT=</code><br>
 268         <code>export DDS_ROOT=</code><br></li>
 269           <li>
 270             Check the doxygen version at the shell by executing the command:<br>
 271             <code>doxygen --version</code>
 272             This should be at least 1.6.2
 273           </li>
 274           <br>
 275         Now you are ready to create documentation
 276       </li>
 277       <li>
 278         <code>cd $ACE_ROOT</code><br>
 279         <code>$ACE_ROOT/bin/generate_rel_manpages</code><br>
 280         When this is ready copy the resulting files using<br>
 281         <code>scp ACE-html.tar.gz ACE-html.tar.bz2 ACE-html.zip ACE-html.tar.gz.md5 ACE-html.tar.bz2.md5 ACE-html.zip.md5 bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution</code>
 282       </li></ul>
 283             <code>
 284         screen<br>
 285         cd $HOME<br>
 286         rm -rf doxygen<br>
 287         mkdir doxygen<br>
 288         cd doxygen<br>
 289         git clone https://github.com/DOCGroup/ACE_TAO.git --depth 1 --branch ACE+TAO-7_0_9 ACE_TAO<br>
 290         cd ACE_TAO<br>
 291         export ACE_ROOT=$PWD/ACE<br>
 292         export TAO_ROOT=$PWD/TAO<br>
 293         export CIAO_ROOT=<br>
 294         export DANCE_ROOT=<br>
 295         export DDS_ROOT=<br>
 296         cd ACE<br>
 297         $ACE_ROOT/bin/generate_rel_manpages<br>
 298         scp ACE-html.tar.gz ACE-html.tar.bz2 ACE-html.zip ACE-html.tar.gz.md5 ACE-html.tar.bz2.md5 ACE-html.zip.md5 bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution
 299      </code>
 300 <br>
 301       <li>
 302         While doxygen churns, format a release announcement, including the release
 303         notes gathered from developers.
 304       <li>
 305         On <b>naboo.dre.vanderbilt.edu</b> once the doxygen generation has finished, update the documentation at www.dre.vanderbilt.edu/Doxygen.</li>
 306       <ul>
 307         <li>
 308           Create a temporary directory. eg. tmp and change Directory to this- </li>
 309         <li>
 310           scp file from the download server
 311           </li>
 312         <li>
 313           Unzip and untar the files</li>
 314         <li>back out of the temporary directory
 315           </li>
 316         <li>
 317           Create directory 'Current Version No' for example 6.3.4 and change directory into this new one
 318         <li>
 319           Move contents of the temporary directory's html to this directory -</li>
 320         <li>
 321           Now back our of this directory and remove the already existing softlink to the "Micro" directory -</li>
 322         <li>
 323           Create softlink "Micro" linking it to new Documentation using -. If this is a minor release also update the <code>Stable</code> link.</li>
 324         <li>
 325         Remove the directory tmp</li>
 326         <code>
 327         cd /web/www/Doxygen<br>
 328         mkdir tmp<br>
 329         cd tmp<br>
 330         scp bczar@download.dre.vanderbilt.edu:/export/www/download.dre/ACE+TAO-distribution/ACE-html.tar.bz2 .<br>
 331         bunzip2 ACE-html.tar.bz2;tar -xvf ACE-html.tar<br>
 332         cd ..<br>
 333         mkdir 7.0.8<br>
 334         cd 7.0.8<br>
 335         mv ../tmp/html .<br>
 336         cd ..<br>
 337         rm Micro<br>
 338         ln -s 7.0.8/html Micro<br>
 339         rm -rf tmp<br>
 340         </code>
 341
 342       </ul><br>
 343       <li>
 344         On <b>download.dre.vanderbilt.edu</b> where the kit was uploaded to and once <b>BOTH</b> the kit
 345         and doxygen generation have finished their work, you should also move the packages to the
 346         previous versions directory with the appropriate decorators.
 347         <ul>
 348           <li>
 349             <code>cd /export/www/download.dre/ACE+TAO-distribution</code></li>
 350           <li>
 351             Check that the packages in this directory are all from today</li>
 352           <li>
 353             Modify <b><code>/export/anduriltmp/bczar/copy_script.sh</code></b>
 354           to use the correct ACE version X.Y.Z and run it.
 355           <li>
 356             Update the copy_script.sh file for the new micro release</li>
 357         </ul><br>
 358       <li>
 359         Update ACE_wrappers/bin/copy-local-script.sh for use the latest version number and use this script to rename all packages
 360         locally to include the version number. After this is done upload the versioned packages to the github release page
 361       </li>
 362       <li>
 363         Validate the packages on the webserver whether they are really containing the new release. Make at least
 364         one build where you run the TAO Hello world test and check if the libraries are having the
 365         correct version number.</li>
 366       <li>
 367         Update the ACE_wrappers repo:<ul>
 368         <li>docs/Download.html to show the new release.</li>
 369       <li>
 370         etc/index.hml to show the new doxygen package you installed</li>
 371         <li>bin/diff-builds-and-group-fixed-tests-only.sh to give the correct default old_date for this release.</li></ul>
 372       <li>
 373         Update the ACE/NEWS and TAO/NEWS files to have a new section for the next release.</li>
 374       <li>
 375         Put the release details also on the github release page for this latest release, see https://github.com/DOCGroup/ACE_TAO/releases</li>
 376       <li>
 377         Update openSUSE Build service using<br>
 378         <code>
 379         osc checkout devel:libraries:ACE:micro ACE<br>
 380         cd &quot;devel:libraries:ACE:micro/ACE&quot;<br>
 381         osc rm ACE+TAO-src-*.tar.gz<br>
 382         cp $DOC_ROOT/package-&lt;PID&gt;/ACE+TAO-src-*.tar.gz .<br>
 383         osc commit<br>
 384         </code>
 385       </li>
 386       <li>
 387         Make a new pull request for homebrew-core to update ACE to the new release, see https://github.com/Homebrew/homebrew-core/blob/master/Formula/ace.rb
 388       </li>
 389       <li>
 390         Make a new pull request to Microsoft vcpkg to update ACE to the new release, see https://github.com/Microsoft/vcpkg/tree/master/ports/ace
 391       </li>
 392       <li>
 393         Make a new pull request to OpenDDS to update the configure script to use the new release, see https://github.com/OpenDDS/OpenDDS/blob/master/configure
 394       </li>
 395       <li>
 396         Mail the approved release announcement out to, at minimum the following:
 397         <tt>tao-users@list.isis.vanderbilt.edu</tt>, <tt>tao-announce@list.isis.vanderbilt.edu</tt>,
 398         <tt>ace-users@list.isis.vanderbilt.edu</tt>, <tt>ace-announce@list.isis.vanderbilt.edu</tt>.
 399         Do this as yourself (not as bugzilla). <b>N.B.</b>
 400       You will not be able to post to the users' lists unless you are subscribed to
 401       them. Odds are you will not be able to post to the announce lists at all. Ask
 402       someone else (like Doug or Johnny) to do this step.<br>
 403       </ol>
 404     </p>
 405     <hr>
 406     <h2>
 407       Tips to being a Build Czar</h2>
 408     <p><ol>
 409       <li>Trust no one.</li>
 410       <li>Be careful with <a href="http://www.dre.vanderbilt.edu/~schmidt/">this guy</a>, he
 411       is notorious in breaking builds (and fixing them as well...rumour has it that
 412       it's actually a super-scalar, super-pipelined processor capable of out-of-order
 413       execution, in human incarnation).</li>
 414       <li>Don't forgive people who break ACE :-)</li>
 415       <li>If a build hasn't run in a long time (symptoms are a "red" in the Last Run
 416       column of the build scoreboard), delete the .disable file in
 417       /path/to/build/directory/BUILD_NAME/ by hand.</li>
 418       <li>Think of the group who wrote the scoreboard update script, every time you
 419       catch an otherwise not so obvious error with the help of the scoreboard. Tell <a href="mailto:devo-group@list.isis.vanderbilt.edu">
 420         DEVO group</a> about it.</li>
 421       <li>Compile once on Win32 and Linux before cutting a micro.</li>
 422       <li>Trust the release script when making a release. Don't make tar balls by
 423       hand.</li>
 424       <li>When all hell breaks loose, don't wait for the nightly builds to monitor
 425       improvement. Instead manually start the builds.</li>
 426       <li>Don't hesitate to ask for help.</li>
 427       <li>Install your public key to the different machines you have frequent access
 428       to avoid typing password.</li>
 429       <li>Update this page if you have any more tips for future build czars :-). This
 430       page is in git under <code>ACE/docs/bczar/bczar.html</code></li>
 431     </p></ol>
 432     <hr>
 433     <Center>
 434       <h1>The Realm of the Build Czar</h1>
 435     </Center>
 436     <hr>
 437     <h2>Build Czar Arthur</h2>
 438     <p>Many years have passes since the days of the legendary Build Czar Arthur. His
 439       duties were given to him by the mystical Lady of the Lake, who outlined the
 440       first responsibilities of the Build Czar.</p>
 441     <tt>
 442       <br>
 443       Then bespake the Lady of the Lake,<br>
 444       And these were the words said shee:<br>
 445       &quot;I knoweth of thy deeds, thou noble Arthur,<br>
 446       but thy task hath not finished for thee&quot;<br>
 447       <br>
 448       &quot;Thou shalt feitch thy trusty steed,<br>
 449       And cleanse thy builds againe;<br>
 450       Then shallt thy ryde hath finnished,<br>
 451       When new kits released thee cann.&quot;<br>
 452       <br>
 453       Then bespake him noble Arthur<br>
 454       And these were the words said he:<br>
 455       &quot;With what weapons shallt I use,<br>
 456       To asure these from the devil free?&quot;<br>
 457       <br>
 458       Then appeered before noble Arthur,<br>
 459       Uppon the ground a sacred scroll<br>
 460       Conjurred by the Lady of the Lake<br>
 461       Borne of the earth in a roll.<br>
 462       <br>
 463       She saies, &quot;Clasp this to thine selfe<br>
 464       For thee shallt find need for it.<br>
 465       It shall keep others in the cold,<br>
 466       Only to be ressurected when thee sees fit.&quot;<br>
 467       <br>
 468       &quot;Others shall join thy person,<br>
 469       To ryde with thee in thy quest;<br>
 470       Thee shallt be thankful of theire help,<br>
 471       And to alsoe hold them steadfast.&quot;<br>
 472       <br>
 473       &quot;But if theire talke too lodly rise,<br>
 474       And causeth much damage to thine cuntry,<br>
 475       He must come forth, and make proclamation,<br>
 476       For the next one he shall be.&quot;<br>
 477       <br>
 478       So hath Arthur to the Lady spoke:<br>
 479       &quot;For I sweare, and save my othe,<br>
 480       While enimes and evils I seeke,<br>
 481       I shall fight against them bothe.<br>
 482       <br>
 483     </tt>
 484     <hr>
 485   </html>
 486 </body>