1.. _build_toolchains: 2 3=========================== 4Creating Toolchain Archives 5=========================== 6 7There are various scripts in the repository for producing archives 8of the build tools (e.g. compilers and linkers) required to build. 9 10Clang and Rust 11============== 12 13To modify the toolchains used for a particular task, you may need several 14things: 15 161. A `build task`_ 17 182. Which uses a toolchain task 19 20 - `clang toolchain`_ 21 - `rust toolchain`_ 22 233. Which uses a git fetch 24 25 - `clang fetch`_ 26 - (from-source ``dev`` builds only) `rust fetch`_ 27 284. (clang only) Which uses a `config json`_ 29 305. Which takes patches_ you may want to apply. 31 32For the most part, you should be able to accomplish what you want by 33copying/editing the existing examples in those files. 34 35.. _build task: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/build/linux.yml#5-45 36.. _clang toolchain: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/clang.yml#51-72 37.. _rust toolchain: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/rust.yml#57-74 38.. _clang fetch: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/fetch/toolchains.yml#413-418 39.. _rust fetch: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/fetch/toolchains.yml#434-439 40.. _config json: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/build/build-clang/clang-linux64.json 41.. _patches: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/build/build-clang/static-llvm-symbolizer.patch 42 43Clang 44----- 45 46Building clang is handled by `build-clang.py`_, which uses several resources 47in the `build-clang`_ directory. Read the `build-clang README`_ for more 48details. 49 50Note for local builds: build-clang.py can be run on developer machines but its 51lengthy multi-stage build process is unnecessary for most local development. The 52upstream `LLVM Getting Started Guide`_ has instructions on how to build 53clang more directly. 54 55.. _build-clang.py: https://searchfox.org/mozilla-central/source/build/build-clang/build-clang.py 56.. _build-clang README: https://searchfox.org/mozilla-central/source/build/build-clang/README 57.. _build-clang: https://searchfox.org/mozilla-central/source/build/build-clang/ 58.. _LLVM Getting Started Guide: https://llvm.org/docs/GettingStarted.html 59 60Rust 61---- 62 63Rust builds are handled by `repack_rust.py`_. The primary purpose of 64that script is to download prebuilt tarballs from the Rust project. 65 66It uses the same basic format as `rustup` for specifying the toolchain 67(via ``--channel``): 68 69- request a stable build with ``1.xx.y`` (e.g. ``1.47.0``) 70- request a beta build with ``beta-yyyy-mm-dd`` (e.g. ``beta-2020-08-26``) 71- request a nightly build with ``nightly-yyyy-mm-dd`` (e.g. ``nightly-2020-08-26``) 72- request a build from `Rust's ci`_ with ``bors-$sha`` (e.g. ``bors-796a2a9bbe7614610bd67d4cd0cf0dfff0468778``) 73- request a from-source build with ``dev`` 74 75Rust From Source 76---------------- 77 78As of this writing, from-source builds for Rust are a new feature, and not 79used anywhere by default. The feature was added so that we can test patches 80to rustc against the tree. Expect things to be a bit hacky and limited. 81 82Most importantly, building from source requires your toolchain to have a 83`fetch of the rust tree`_ as well as `clang and binutils toolchains`_. It is also 84recommended to upgrade the worker-type to e.g. ``b-linux-large``. 85 86Rust's build dependencies are fairly minimal, and it has a sanity check 87that should catch any missing or too-old dependencies. See the `Rust README`_ 88for more details. 89 90Patches are set via `the --patch flag`_ (passed via ``toolchain/rust.yml``). 91Patch paths are assumed to be relative to ``/build/build-rust/``, and may be 92optionally prefixed with ``module-path:`` to specify they apply to that git 93submodule in the Rust source. e.g. ``--patch src/llvm-project:mypatch.diff`` 94patches rust's llvm with ``/build/build-rust/mypatch.diff``. There are no 95currently checked in rust patches to use as an example, but they should be 96the same format as `the clang ones`_. 97 98Rust builds are not currently configurable, and uses a `hardcoded config.toml`_, 99which you may need to edit for your purposes. See Rust's `example config`_ for 100details/defaults. Note that these options do occasionally change, so be sure 101you're using options for the version you're targeting. For instance, there was 102a large change around Rust ~1.48, and the currently checked in config was for 1031.47, so it may not work properly when building the latest version of Rust. 104 105Rust builds are currently limited to targeting only the host platform. 106Although the machinery is in place to request additional targets, the 107cross-compilation fails for some unknown reason. We have not yet investigated 108what needs to be done to get this working. 109 110While Rust generally maintains a clean tree for building ``rustc`` and 111``cargo``, other tools like ``rustfmt`` or ``miri`` are allowed to be 112transiently broken. This means not every commit in the Rust tree will be 113able to build the `tools we require`_. 114 115Although ``repack_rust`` considers ``rustfmt`` an optional package, Rust builds 116do not currently implement this and will fail if ``rustfmt`` is busted. Some 117attempt was made to work around it, but `more work is needed`_. 118 119.. _Rust's ci: https://github.com/rust-lang/rust/pull/77875#issuecomment-736092083 120.. _repack_rust.py: https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/repack_rust.py 121.. _fetch of the rust tree: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/rust.yml#69-71 122.. _clang and binutils toolchains: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/ci/toolchain/rust.yml#72-74 123.. _the --patch flag: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/scripts/misc/repack_rust.py#667-675 124.. _the clang ones: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/build/build-clang/static-llvm-symbolizer.patch 125.. _Rust README: https://github.com/rust-lang/rust/#building-on-a-unix-like-system 126.. _hardcoded config.toml: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/scripts/misc/repack_rust.py#384-421 127.. _example config: https://github.com/rust-lang/rust/blob/b7ebc6b0c1ba3c27ebb17c0b496ece778ef11e18/config.toml.example 128.. _tools we require: https://searchfox.org/mozilla-central/rev/168c45a7acc44e9904cfd4eebcb9eb080e05699c/taskcluster/scripts/misc/repack_rust.py#398 129.. _more work is needed: https://github.com/rust-lang/rust/issues/79249 130 131 132Windows 133======= 134 135The ``build/windows_toolchain.py`` script is used to build and manage 136Windows toolchain archives containing Visual Studio executables, SDKs, 137etc. 138 139The way Firefox build automation works is an archive containing the 140toolchain is produced and uploaded to an internal Mozilla server. The 141build automation will download, verify, and extract this archive before 142building. The archive is self-contained so machines don't need to install 143Visual Studio, SDKs, or various other dependencies. Unfortunately, 144Microsoft's terms don't allow Mozilla to distribute this archive 145publicly. However, the same tool can be used to create your own copy. 146 147Configuring Your System 148----------------------- 149 150It is **highly** recommended to perform this process on a fresh installation 151of Windows 7 or 10 (such as in a VM). Installing all updates through 152Windows Update is not only acceptable - it is encouraged. Although it 153shouldn't matter. 154 155Next, install Visual Studio 2017 Community. The download link can be found 156at https://www.visualstudio.com/vs/community/. 157Be sure to follow these install instructions: 158 1591. Choose a ``Custom`` installation and click ``Next`` 1602. Select ``Programming Languages`` -> ``Visual C++`` (make sure all sub items are 161 selected) 1623. Under ``Windows and Web Development`` uncheck everything except 163 ``Universal Windows App Development Tools`` and the items under it 164 (should be ``Tools (1.3.1)...`` and the ``Windows 10 SDK``). 165 166Once Visual Studio 2017 Community has been installed, from a checkout 167of mozilla-central, run something like the following to produce a ZIP 168archive:: 169 170 $ ./mach python build/windows_toolchain.py create-zip vs2017_15.8.4 171 172The produced archive will be the argument to ``create-zip`` + ``.zip``. 173 174Firefox for Android with Gradle 175=============================== 176 177To build Firefox for Android with Gradle in automation, archives 178containing both the Gradle executable and a Maven repository 179comprising the exact build dependencies are produced and uploaded to 180an internal Mozilla server. The build automation will download, 181verify, and extract these archive before building. These archives 182provide a self-contained Gradle and Maven repository so that machines 183don't need to fetch additional Maven dependencies at build time. 184(Gradle and the downloaded Maven dependencies can be both 185redistributed publicly.) 186 187Archiving the Gradle executable is straight-forward, but archiving a 188local Maven repository is not. Therefore a toolchain job exists for 189producing the required archives, `android-gradle-dependencies`. The 190job runs in a container based on a custom Docker image and spawns a 191Sonatype Nexus proxying Maven repository process in the background. 192The job builds Firefox for Android using Gradle and the in-tree Gradle 193configuration rooted at ``build.gradle``. The spawned proxying Maven 194repository downloads external dependencies and collects them. After 195the Gradle build completes, the job archives the Gradle version used 196to build, and the downloaded Maven repository, and exposes them as 197Task Cluster artifacts. 198 199To update the version of Gradle in the archive produced, update 200``gradle/wrapper/gradle-wrapper.properties``. Be sure to also update 201the SHA256 checksum to prevent poisoning the build machines! 202 203To update the versions of Gradle dependencies used, update 204``dependencies`` sections in the in-tree Gradle configuration rooted 205at ``build.gradle``. Once you are confident your changes build 206locally, push a fresh build to try. The `android-gradle-dependencies` 207toolchain should run automatically, fetching your new dependencies and 208wiring them into the appropriate try build jobs. 209 210To update the version of Sonatype Nexus, update the `sonatype-nexus` 211`fetch` task definition. 212 213To modify the Sonatype Nexus configuration, typically to proxy a new 214remote Maven repository, modify 215`taskcluster/scripts/misc/android-gradle-dependencies/nexus.xml`. 216 217There is also a toolchain job that fetches the Android SDK and related 218packages. To update the versions of packaged fetched, modify 219`python/mozboot/mozboot/android-packages.txt` and update the various 220in-tree versions accordingly. 221