1Partner repacks
2===============
3
4We create slightly-modified Firefox releases for some extra audiences
5
6* EME-free builds, which disable DRM plugins by default
7* Funnelcake builds, which are used for Mozilla experiments
8* partner builds, which customize Firefox for external partners
9
10We use the phrase "partner repacks" to refer to all these builds because they
11use the same process of repacking regular Firefox releases with additional files.
12The specific differences depend on the type of build.
13
14We produce partner repacks for some beta builds, and for release builds, as part of the release
15automation. We don't produce any files to update these builds as they are handled automatically
16(see updates_).
17
18
19Parameters & Scheduling
20-----------------------
21
22Partner repacks have a number of parameters which control how they work:
23
24* ``release_enable_emefree``
25* ``release_enable_partners``
26* ``release_partner_config``
27* ``release_partner_build_number``
28* ``release_partners``
29
30We split the repacks into two 'paths', EME-free and everything else, to retain some
31flexibility over enabling/disabling them separately. This costs us some duplication of the kinds
32in the repacking stack. The two enable parameters are booleans to turn these two paths
33on/off. We set them in release-runner3's `is_partner_enabled() <https://dxr.mozilla
34.org/build-central/search?q=function%3Ais_partner_enabled&redirect=true>`_ when starting a
35release. They're both true for Firefox betas >= b8 and releases, but otherwise disabled.
36
37``release_partner_config`` is a dictionary of configuration data which drives the task generation
38logic. It's usually looked up during the release promotion action task, using the Github
39GraphQL API in the `get_partner_config_by_url()
40<python/taskgraph.util.html#taskgraph.util.partners.get_partner_config_by_url>`_ function, with the
41url defined in `taskcluster/ci/config.yml <https://dxr.mozilla
42.org/mozilla-release/search?q=regexp%3A^partner+path%3Aconfig.yml&redirect=true>`_.
43
44``release_partner_build_number`` is an integer used to create unique upload paths in the firefox
45candidates directory, while ``release_partners`` is a list of partners that should be
46repacked (i.e. a subset of the whole config). Both are intended for use when respinning a few partners after
47the regular Firefox has shipped. More information on that can be found in the `release-warrior docs
48<https://github.com/mozilla-releng/releasewarrior-2
49.0/blob/master/docs/misc-operations/off-cycle-partner-repacks -and-funnelcake.md>`_.
50
51Most of the machine time for generating partner repacks takes place in the `promote` phase of the
52automation, or `promote_rc` in the case of X.0 release candidates. The EME-free builds are copied into the
53Firefox releases directory in the `push` phase, along with the regular bits.
54
55
56Configuration
57-------------
58
59We need some configuration to know *what* to repack, and *how* to do that. The *what* is defined by
60default.xml manifests, as used with the `repo <https://gerrit.googlesource.com/git-repo>`_ tool
61for git. The `default.xml for EME-free <https://github
62.com/mozilla-partners/mozilla-EME-free-manifest/blob/master/default.xml>`_ illustrates this::
63
64    <?xml version="1.0" ?>
65    <manifest>
66      <remote fetch="git@github.com:mozilla-partners/" name="mozilla-partners"/>
67      <remote fetch="git@github.com:mozilla/" name="mozilla"/>
68
69      <project name="repack-scripts" path="scripts" remote="mozilla-partners" revision="master"/>
70      <project name="build-tools" path="scripts/tools" remote="mozilla" revision="master"/>
71      <project name="mozilla-EME-free" path="partners/mozilla-EME-free" remote="mozilla-partners" revision="master"/>
72    </manifest>
73
74The repack-scripts and build-tools repos are found in all manifests, and then there is a list of
75partner repositories which contain the *how* configuration. Some of these repos are not publicly
76visible.
77
78A partner repository may contain multiple configurations inside the ``desktop`` directory. Each
79subdirectory must contain a ``repack.cfg`` and a ``distribution`` directory, the latter
80containing the customizations needed. Here's `EME-free's repack.cfg <https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/repack.cfg>`_::
81
82    aus="mozilla-EMEfree"
83    dist_id="mozilla-EMEfree"
84    dist_version="1.0"
85    linux-i686=false
86    linux-x86_64=false
87    locales="ach af an ar"  # truncated for display here
88    mac=true
89    win32=true
90    win64=true
91    output_dir="%(platform)s-EME-free/%(locale)s"
92
93    # Upload params
94    upload_to_candidates=true
95
96Note the list of locales and boolean toggles for enabling platforms. The ``output_dir`` and
97``upload_to_candidates`` parameters are only present for repacks which are uploaded into the
98`candidates directory <https://archive.mozilla.org/pub/firefox/candidates/>`_.
99
100All customizations will be placed in the ``distribution`` directory at the root of the Firefox
101install directory, or in the case of OS X in ``Firefox.app/Contents/Resources/distribution/``. A
102``distribution.ini`` file is the minimal requirement, here's an example from `EME-free
103<https://github.com/mozilla-partners/mozilla-EME-free/blob/master/desktop/mozilla-EME-free/distribution
104/distribution.ini>`_::
105
106    # Partner Distribution Configuration File
107    # Author: Mozilla
108    # Date: 2015-03-27
109
110    [Global]
111    id=mozilla-EMEfree
112    version=1.0
113    about=Mozilla Firefox EME-free
114
115    [Preferences]
116    media.eme.enabled=false
117    app.partner.mozilla-EMEfree="mozilla-EMEfree"
118
119Extensions and other customizations might also be included in repacks.
120
121
122Repacking process
123-----------------
124
125The stack of tasks to create partner repacks is broadly similar to localised nightlies and
126regular releases. The basic form is
127
128* partner repack - insert the customisations into the the regular builds
129* signing - sign the internals which will become the installer (Mac only)
130* repackage - create the "installer" (Mac and Windows)
131* chunking dummy - a linux only bridge to ...
132* repackage signing - sign the "installers" (mainly Windows)
133* beetmover - move the files to a partner-specific destination
134* beetmover checksums - possibly beetmove the checksums from previous step
135
136Some key divergences are:
137
138* all intermediate artifacts are uploaded with a ``releng/partner`` prefix
139* we don't insert any binaries on Windows so no need for internal signing
140* there's no need to create any complete mar files at the repackage step
141* we support both public and private destinations in beetmover
142* we only need beetmover checksums for EME-free builds
143
144
145Partner repack
146^^^^^^^^^^^^^^
147
148* kinds: ``release-partner-repack`` ``release-eme-free-repack``
149* platforms: Typically all (but depends on what's enabled by partner configuration)
150* upstreams: ``build-signing`` ``l10n-signing``
151
152There is one task per platform in this step, calling out to `scripts/desktop_partner_repacks.py
153<https://hg.mozilla.org/releases/mozilla-release/file/default/testing/mozharness/scripts
154/desktop_partner_repacks.py>`_ in mozharness to prepare an environment and then perform the repacks.
155
156It takes as input the build-signing and l10n-signing artifacts, which are all zip/tar.gz/tar.bz2
157archives, simplifying the repack process by avoiding dmg and exe. Windows produces ``target.zip``
158& ``setup.exe``, Mac is ``target.tar.gz``, Linux is the final product ``target.tar.bz2``
159(beetmover handles pretty naming as usual).
160
161Signing
162^^^^^^^
163
164* kinds: ``release-partner-repack-signing`` ``release-eme-free-repack-signing``
165* platforms: Mac
166* upstreams: ``release-partner-repack`` ``release-eme-free-repack``
167
168We chunk the single partner repack task out to a signing task per artifact at this point. For
169example, EME-free will become ~95 tasks, one for each locale. We collect the target.tar.gz from the
170upstream, and return a signed target.tar.gz. We use a ``target.dmg`` artifact for
171nightlies/regular releases, but this is converted to ``target.tar.gz`` by the signing
172scriptworker before sending it to the signing server, so partners are equivalent.
173
174Repackage
175^^^^^^^^^
176
177* kinds: ``release-partner-repack-repackage`` ``release-eme-free-repack-repackage``
178* platforms: Mac & Windows
179* upstreams:
180
181    * Mac: ``release-partner-signing`` ``release-eme-free-signing``
182    * Windows: ``release-partner-repack`` ``release-eme-free-repack``
183
184Mac has a repackage job for each of the signing tasks. Windows repackages are chunked here to
185the same granularity as mac. Takes ``target.zip`` & ``setup.exe`` to produce ``target.exe`` on
186Windows, and ``target.tar.gz`` to produce ``target.dmg`` on Mac. There's no need to produce any
187complete.mar files here like regular release bits do because we can reuse those.
188
189Chunking dummy
190^^^^^^^^^^^^^^
191
192* kinds: ``release-partner-repack-chunking-dummy``
193* platforms: Linux
194* upstreams: ``release-partner-repack``
195
196We're need Linux chunked at the next step so this dummy takes care of that for the relatively simple path
197Linux follows. One task per sub config+locale combination, the same as Windows and Mac. This doesn't need to
198exist for EME-free because we don't need to create Linux builds there.
199
200Repackage Signing
201^^^^^^^^^^^^^^^^^
202
203* kinds: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing``
204* platforms: All
205* upstreams:
206
207   * Mac & Windows: ``release-partner-repackage`` ``release-eme-free-repackage``
208   * Linux: ``release-partner-repack-chunking-dummy``
209
210This step GPG signs all platforms, and authenticode signs the Windows installer.
211
212Beetmover
213^^^^^^^^^
214
215* kinds: ``release-partner-repack-beetmover`` ``release-eme-free-repack-beetmover``
216* platforms: All
217* upstreams: ``release-partner-repack-repackage-signing`` ``release-eme-free-repack-repackage-signing``
218
219Moves and renames the artifacts to their public location in the `candidates directory
220<https://archive.mozilla.org/pub/firefox/candidates/>`_, or a private S3 bucket. Each task will
221have the ``project:releng:beetmover:action:push-to-partner`` scope, with public uploads having
222``project:releng:beetmover:bucket:release`` and private uploads using
223``project:releng:beetmover:bucket:partner``. The ``upload_to_candidates`` key in the partner config
224controls the second scope. There's a separate partner code path in `beetmoverscript <https://github
225.com/mozilla-releng/beetmoverscript>`_.
226
227Beetmover checksums
228^^^^^^^^^^^^^^^^^^^
229
230* kinds: ``release-eme-free-repack-beetmover-checksums``
231* platforms: Mac & Windows
232* upstreams: ``release-eme-free-repack-repackage-beetmover``
233
234The EME-free builds should be present in our SHA256SUMS file and friends (`e.g. <https://archive
235.mozilla.org/pub/firefox/releases/61.0/SHA256SUMS>`_) so we beetmove the target.checksums from
236the beetmover tasks into the candidates directory. They get picked up by the
237``release-generate-checksums`` kind.
238
239.. _updates:
240
241Updates
242-------
243
244It's very rare to need to update a partner repack differently from the original
245release build but we retain that capability. A partner build with distribution name ``foo``,
246based on a release Firefox build, will query for an update on the ``release-cck-foo`` channel. If
247the update server `Balrog <http://mozilla-balrog.readthedocs.io/en/latest/>`_ finds no rule for
248that channel it will fallback to the ``release`` channel. The update files for the regular releases do not
249modify the ``distribution/`` directory, so the customizations are not modified.
250
251`Bug 1430254 <https://bugzilla.mozilla.org/show_bug.cgi?id=1430254>`_ is an example of an exception to this
252logic.
253