1============================= 2How To Validate a New Release 3============================= 4 5.. contents:: 6 :local: 7 :depth: 1 8 9Introduction 10============ 11 12This document contains information about testing the release candidates that 13will ultimately be the next LLVM release. For more information on how to 14manage the actual release, please refer to :doc:`HowToReleaseLLVM`. 15 16Overview of the Release Process 17------------------------------- 18 19Once the release process starts, the Release Manager will ask for volunteers, 20and it'll be the role of each volunteer to: 21 22* Test and benchmark the previous release 23 24* Test and benchmark each release candidate, comparing to the previous release 25 and candidates 26 27* Identify, reduce and report every regression found during tests and benchmarks 28 29* Make sure the critical bugs get fixed and merged to the next release candidate 30 31Not all bugs or regressions are show-stoppers and it's a bit of a grey area what 32should be fixed before the next candidate and what can wait until the next 33release. 34 35It'll depend on: 36 37* The severity of the bug, how many people it affects and if it's a regression 38 or a known bug. Known bugs are "unsupported features" and some bugs can be 39 disabled if they have been implemented recently. 40 41* The stage in the release. Less critical bugs should be considered to be 42 fixed between RC1 and RC2, but not so much at the end of it. 43 44* If it's a correctness or a performance regression. Performance regression 45 tends to be taken more lightly than correctness. 46 47.. _scripts: 48 49Scripts 50======= 51 52The scripts are in the ``utils/release`` directory. 53 54test-release.sh 55--------------- 56 57This script will check-out, configure and compile LLVM+Clang (+ most add-ons, 58like ``compiler-rt``, ``libcxx``, ``libomp`` and ``clang-extra-tools``) in 59three stages, and will test the final stage. 60It'll have installed the final binaries on the Phase3/Releasei(+Asserts) 61directory, and that's the one you should use for the test-suite and other 62external tests. 63 64To run the script on a specific release candidate run:: 65 66 ./test-release.sh \ 67 -release 3.3 \ 68 -rc 1 \ 69 -no-64bit \ 70 -test-asserts \ 71 -no-compare-files 72 73Each system will require different options. For instance, x86_64 will 74obviously not need ``-no-64bit`` while 32-bit systems will, or the script will 75fail. 76 77The important flags to get right are: 78 79* On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2, 80 change it to ``-rc 2`` and so on. 81 82* On non-release testing, you can use ``-final`` in conjunction with 83 ``-no-checkout``, but you'll have to create the ``final`` directory by hand 84 and link the correct source dir to ``final/llvm.src``. 85 86* For release candidates, you need ``-test-asserts``, or it won't create a 87 "Release+Asserts" directory, which is needed for release testing and 88 benchmarking. This will take twice as long. 89 90* On the final candidate you just need Release builds, and that's the binary 91 directory you'll have to pack. 92 93* On macOS, you must export ``MACOSX_DEPLOYMENT_TARGET=10.9`` before running 94 the script. 95 96This script builds three phases of Clang+LLVM twice each (Release and 97Release+Asserts), so use screen or nohup to avoid headaches, since it'll take 98a long time. 99 100Use the ``--help`` option to see all the options and chose it according to 101your needs. 102 103 104findRegressions-nightly.py 105-------------------------- 106 107TODO 108 109.. _test-suite: 110 111Test Suite 112========== 113 114.. contents:: 115 :local: 116 117Follow the `LNT Quick Start Guide 118<https://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the 119test-suite 120 121The binary location you'll have to use for testing is inside the 122``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``. 123Link that directory to an easier location and run the test-suite. 124 125An example on the run command line, assuming you created a link from the correct 126install directory to ``~/devel/llvm/install``:: 127 128 ./sandbox/bin/python sandbox/bin/lnt runtest \ 129 nt \ 130 -j4 \ 131 --sandbox sandbox \ 132 --test-suite ~/devel/llvm/test/test-suite \ 133 --cc ~/devel/llvm/install/bin/clang \ 134 --cxx ~/devel/llvm/install/bin/clang++ 135 136It should have no new regressions, compared to the previous release or release 137candidate. You don't need to fix all the bugs in the test-suite, since they're 138not necessarily meant to pass on all architectures all the time. This is 139due to the nature of the result checking, which relies on direct comparison, 140and most of the time, the failures are related to bad output checking, rather 141than bad code generation. 142 143If the errors are in LLVM itself, please report every single regression found 144as blocker, and all the other bugs as important, but not necessarily blocking 145the release to proceed. They can be set as "known failures" and to be 146fix on a future date. 147 148.. _pre-release-process: 149 150Pre-Release Process 151=================== 152 153.. contents:: 154 :local: 155 156When the release process is announced on the mailing list, you should prepare 157for the testing, by applying the same testing you'll do on the release 158candidates, on the previous release. 159 160You should: 161 162* Download the previous release sources from 163 https://llvm.org/releases/download.html. 164 165* Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to 166 ``-final``). 167 168* Once all three stages are done, it'll test the final stage. 169 170* Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base, 171 run the test-suite. 172 173If the final phase's ``make check-all`` failed, it's a good idea to also test 174the intermediate stages by going on the obj directory and running 175``make check-all`` to find if there's at least one stage that passes (helps 176when reducing the error for bug report purposes). 177 178.. _release-process: 179 180Release Process 181=============== 182 183.. contents:: 184 :local: 185 186When the Release Manager sends you the release candidate, download all sources, 187unzip on the same directory (there will be sym-links from the appropriate places 188to them), and run the release test as above. 189 190You should: 191 192* Download the current candidate sources from where the release manager points 193 you (ex. https://llvm.org/pre-releases/3.3/rc1/). 194 195* Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the 196 test-suite the same way. 197 198* Compare the results, report all errors on Bugzilla and publish the binary blob 199 where the release manager can grab it. 200 201Once the release manages announces that the latest candidate is the good one, 202you have to pack the ``Release`` (no Asserts) install directory on ``Phase3`` 203and that will be the official binary. 204 205* Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory 206 207* Tar that into the same name with ``.tar.gz`` extension from outside the 208 directory 209 210* Make it available for the release manager to download 211 212.. _bug-reporting: 213 214Bug Reporting Process 215===================== 216 217.. contents:: 218 :local: 219 220If you found regressions or failures when comparing a release candidate with the 221previous release, follow the rules below: 222 223* Critical bugs on compilation should be fixed as soon as possible, possibly 224 before releasing the binary blobs. 225 226* Check-all tests should be fixed before the next release candidate, but can 227 wait until the test-suite run is finished. 228 229* Bugs in the test suite or unimportant check-all tests can be fixed in between 230 release candidates. 231 232* New features or recent big changes, when close to the release, should have 233 done in a way that it's easy to disable. If they misbehave, prefer disabling 234 them than releasing an unstable (but untested) binary package. 235