1============================= 2How To Validate a New Release 3============================= 4 5.. contents:: 6 :local: 7 :depth: 1 8 9Introduction 10============ 11 12This document contains information about testing the release candidates that will 13ultimately be the next LLVM release. For more information on how to manage the 14actual release, please refer to :doc:`HowToReleaseLLVM`. 15 16Overview of the Release Process 17------------------------------- 18 19Once the release process starts, the Release Manager will ask for volunteers, 20and it'll be the role of each volunteer to: 21 22* Test and benchmark the previous release 23 24* Test and benchmark each release candidate, comparing to the previous release and candidates 25 26* Identify, reduce and report every regression found during tests and benchmarks 27 28* Make sure the critical bugs get fixed and merged to the next release candidate 29 30Not all bugs or regressions are show-stoppers and it's a bit of a grey area what 31should be fixed before the next candidate and what can wait until the next release. 32 33It'll depend on: 34 35* The severity of the bug, how many people it affects and if it's a regression or a 36 known bug. Known bugs are "unsupported features" and some bugs can be disabled if 37 they have been implemented recently. 38 39* The stage in the release. Less critical bugs should be considered to be fixed between 40 RC1 and RC2, but not so much at the end of it. 41 42* If it's a correctness or a performance regression. Performance regression tends to be 43 taken more lightly than correctness. 44 45.. _scripts: 46 47Scripts 48======= 49 50The scripts are in the ``utils/release`` directory. 51 52test-release.sh 53--------------- 54 55This script will check-out, configure and compile LLVM+Clang (+ most add-ons, 56like ``compiler-rt``, ``libcxx`` and ``clang-extra-tools``) in three stages, and 57will test the final stage. It'll have installed the final binaries on the 58Phase3/Releasei(+Asserts) directory, and that's the one you should use for the 59test-suite and other external tests. 60 61To run the script on a specific release candidate run:: 62 63 ./test-release.sh \ 64 -release 3.4 \ 65 -rc 1 \ 66 -triple x86_64-apple-darwin \ 67 -no-64bit \ 68 -test-asserts \ 69 -no-compare-files 70 71Each system will require different options. For instance, x86_64 will obviously not need 72``-no-64bit`` while 32-bit systems will, or the script will fail. 73 74The important flags to get right are: 75 76* On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2, change it to ``-rc 2`` and so on. 77 78* On non-release testing, you can use ``-final`` in conjunction with ``-no-checkout``, but you'll have to 79 create the ``final`` directory by hand and link the correct source dir to ``final/llvm.src``. 80 81* For release candidates, you need ``-test-asserts``, or it won't create a "Release+Asserts" directory, 82 which is needed for release testing and benchmarking. This will take twice as long. 83 84* On the final candidate you just need Release builds, and that's the binary directory you'll have to pack. 85 86This script builds three phases of Clang+LLVM twice each (Release and Release+Asserts), so use 87screen or nohup to avoid headaches, since it'll take a long time. 88 89Use the ``--help`` option to see all the options and chose it according to your needs. 90 91 92findRegressions-nightly.py 93-------------------------- 94 95TODO 96 97.. _test-suite: 98 99Test Suite 100========== 101 102.. contents:: 103 :local: 104 105Follow the `LNT Quick Start Guide <http://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the test-suite 106 107The binary location you'll have to use for testing is inside the ``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``. 108Link that directory to an easier location and run the test-suite. 109 110An example on the run command line, assuming you created a link from the correct 111install directory to ``~/devel/llvm/install``:: 112 113 ./sandbox/bin/python sandbox/bin/lnt runtest \ 114 nt \ 115 -j4 \ 116 --sandbox sandbox \ 117 --test-suite ~/devel/llvm/test/test-suite \ 118 --cc ~/devel/llvm/install/bin/clang \ 119 --cxx ~/devel/llvm/install/bin/clang++ 120 121It should have no new regressions, compared to the previous release or release candidate. You don't need to fix 122all the bugs in the test-suite, since they're not necessarily meant to pass on all architectures all the time. This is 123due to the nature of the result checking, which relies on direct comparison, and most of the time, the failures are 124related to bad output checking, rather than bad code generation. 125 126If the errors are in LLVM itself, please report every single regression found as blocker, and all the other bugs 127as important, but not necessarily blocking the release to proceed. They can be set as "known failures" and to be 128fix on a future date. 129 130.. _pre-release-process: 131 132Pre-Release Process 133=================== 134 135.. contents:: 136 :local: 137 138When the release process is announced on the mailing list, you should prepare 139for the testing, by applying the same testing you'll do on the release candidates, 140on the previous release. 141 142You should: 143 144* Download the previous release sources from http://llvm.org/releases/download.html. 145 146* Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to ``-final``). 147 148* Once all three stages are done, it'll test the final stage. 149 150* Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base, run the test-suite. 151 152If the final phase's ``make check-all`` failed, it's a good idea to also test the 153intermediate stages by going on the obj directory and running ``make check-all`` to find 154if there's at least one stage that passes (helps when reducing the error for bug report 155purposes). 156 157.. _release-process: 158 159Release Process 160=============== 161 162.. contents:: 163 :local: 164 165When the Release Manager sends you the release candidate, download all sources, 166unzip on the same directory (there will be sym-links from the appropriate places 167to them), and run the release test as above. 168 169You should: 170 171* Download the current candidate sources from where the release manager points you 172 (ex. http://llvm.org/pre-releases/3.3/rc1/). 173 174* Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the test-suite 175 the same way. 176 177* Compare the results, report all errors on Bugzilla and publish the binary blob 178 where the release manager can grab it. 179 180Once the release manages announces that the latest candidate is the good one, you 181have to pack the ``Release`` (no Asserts) install directory on ``Phase3`` and that 182will be the official binary. 183 184* Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory 185 186* Tar that into the same name with ``.tar.gz`` extensioan from outside the directory 187 188* Make it available for the release manager to download 189 190.. _bug-reporting: 191 192Bug Reporting Process 193===================== 194 195.. contents:: 196 :local: 197 198If you found regressions or failures when comparing a release candidate with the 199previous release, follow the rules below: 200 201* Critical bugs on compilation should be fixed as soon as possible, possibly before 202 releasing the binary blobs. 203 204* Check-all tests should be fixed before the next release candidate, but can wait 205 until the test-suite run is finished. 206 207* Bugs in the test suite or unimportant check-all tests can be fixed in between 208 release candidates. 209 210* New features or recent big changes, when close to the release, should have done 211 in a way that it's easy to disable. If they misbehave, prefer disabling them than 212 releasing an unstable (but untested) binary package. 213