1.. _localization: 2 3================ 4Localized Builds 5================ 6 7Localization repacks 8==================== 9 10To save on build time, the build system and automation collaborate to allow 11downloading a packaged en-US Firefox, performing some locale-specific 12post-processing, and re-packaging a locale-specific Firefox. Such artifacts 13are termed "single-locale language repacks". There is another concept of a 14"multi-locale language build", which is more like a regular build and less 15like a re-packaging post-processing step. 16 17.. note:: 18 19 These builds rely on make targets that don't work for 20 `artifact builds <https://bugzilla.mozilla.org/show_bug.cgi?id=1387485>`_. 21 22Instructions for single-locale repacks for developers 23----------------------------------------------------- 24 25This assumes that ``$AB_CD`` is the locale you want to repack with; you 26find the available localizations on `l10n-central <https://hg.mozilla.org/l10n-central/>`_. 27 28#. You must have a built and packaged object directory, or a pre-built 29 ``en-US`` package. 30 31 .. code-block:: shell 32 33 ./mach build 34 ./mach package 35 36#. Repackage using the locale-specific changes. 37 38 .. code-block:: shell 39 40 ./mach build installers-$AB_CD 41 42You should find a re-packaged build at ``OBJDIR/dist/``, and a 43runnable binary in ``OBJDIR/dist/l10n-stage/``. 44The ``installers`` target runs quite a few things for you, including getting 45the repository for the requested locale from 46https://hg.mozilla.org/l10n-central/. It will clone them into 47``~/.mozbuild/l10n-central``. If you have an existing repository there, you 48may want to occasionally update that via ``hg pull -u``. If you prefer 49to have the l10n repositories at a different location on your disk, you 50can point to the directory via 51 52 .. code-block:: shell 53 54 ac_add_options --with-l10n-base=/make/this/a/absolute/path 55 56This build also packages a language pack. 57 58Instructions for language packs 59------------------------------- 60 61Language packs are extensions that contain just the localized resources. Building 62them doesn't require an actual build, but they're only compatible with the 63``mozilla-central`` source they're built with. 64 65 66.. code-block:: shell 67 68 ./mach build langpack-$AB_CD 69 70This target shares much of the logic of the ``installers-$AB_CD`` target above, 71and does the check-out of the localization repository etc. It doesn't require 72a package or a build, though. The generated language pack is in 73``OBJDIR/dist/$(MOZ_PKG_PLATFORM)/xpi/``. 74 75.. note:: 76 77 Despite the platform-dependent location in the build directory, language packs 78 are platform independent, and the content that goes into them needs to be 79 built in a platform-independent way. 80 81Instructions for multi-locale builds 82------------------------------------ 83 84If you want to create a single build with multiple locales, you will do 85 86#. Create a build and package 87 88 .. code-block:: shell 89 90 ./mach build 91 ./mach package 92 93#. For each locale you want to include in the build: 94 95 .. code-block:: shell 96 97 export MOZ_CHROME_MULTILOCALE="de it zh-TW" 98 for AB_CD in $MOZ_CHROME_MULTILOCALE; do 99 ./mach build chrome-$AB_CD 100 done 101 102#. Create the multilingual package: 103 104 .. code-block:: shell 105 106 AB_CD=multi ./mach package 107 108General flow of repacks 109----------------------- 110 111The general flow of the locale repacks is controlled by 112``$MOZ_BUILD_APP/locales/Makefile.in`` and ``toolkit/locales/l10n.mk``, plus 113the packaging build system. The three main entry points above all trigger 114related build flows: 115 116#. Get the localization repository, if needed 117#. Run l10n-merge with a prior clobber of the merge dir 118#. Copy l10n files to ``dist``, with minor differences here between ``l10n-%`` and ``chrome-%`` 119#. Repackage and package 120 121Details on l10n-merge are described in its own section below. 122The copying of files is mainly controlled by ``jar.mn``, in the few source 123directories that include localizable files. ``l10n-%`` is used for repacks, 124``chrome-%`` for multi-locale packages. The repackaging is dedicated 125Python code in ``toolkit/mozapps/installer/l10n-repack.py``, using an existing 126package. It strips existing ``chrome`` l10n resources, and adds localizations 127and metadata. 128 129Language packs don't require repackaging. The windows installers are generated 130by merely packaging an existing repackaged zip into to an installer. 131 132Exposing strings 133================ 134 135The localization flow handles a few file formats in well-known locations in the 136source tree. 137 138Alongside being built by including the directory in ``$MOZ_BUILD_APP/locales/Makefile.in`` 139and respective entries in a ``jar.mn``, we also have configuration files tailored 140to localization tools and infrastructure. They're also controlling which 141files l10n-merge handles, and how. 142 143These configurations are TOML files. They're part of the bigger 144localization ecosystem at Mozilla, and `the documentation about the 145file format <http://moz-l10n-config.readthedocs.io/en/latest/fileformat.html>`_ 146explains how to set them up, and what the entries mean. In short, you find 147 148.. code-block:: toml 149 150 [[paths]] 151 reference = browser/locales/en-US/** 152 l10n = {l}browser/** 153 154to add a directory for all localizations. Changes to these files are best 155submitted for review by :Pike or :flod. 156 157These configuration files are the future, and right now, we still have 158support for the previous way to configuring l10n, which is described below. 159 160The locations are commonly in directories like 161 162 :file:`browser/`\ ``locales/en-US/``\ :file:`subdir/file.ext` 163 164The first thing to note is that only files beneath :file:`locales/en-US` are 165exposed to localizers. The second thing to note is that only a few directories 166are exposed. Which directories are exposed is defined in files called 167``l10n.ini``, which are at a 168`few places <https://searchfox.org/mozilla-central/search?q=path%3Al10n.ini&redirect=true>`_ 169in the source code. 170 171An example looks like this 172 173.. code-block:: ini 174 175 [general] 176 depth = ../.. 177 178 [compare] 179 dirs = browser 180 browser/branding/official 181 182 [includes] 183 toolkit = toolkit/locales/l10n.ini 184 185This tells the l10n infrastructure three things: 186 187* resolve the paths against the directory two levels up 188* include files in :file:`browser/locales/en-US` and 189 :file:`browser/branding/official/locales/en-US` 190* load more data from :file:`toolkit/locales/l10n.ini` 191 192For projects like Thunderbird and SeaMonkey in ``comm-central``, additional 193data needs to be provided when including an ``l10n.ini`` from a different 194repository: 195 196.. code-block:: ini 197 198 [include_toolkit] 199 type = hg 200 mozilla = mozilla-central 201 repo = https://hg.mozilla.org/ 202 l10n.ini = toolkit/locales/l10n.ini 203 204This tells the l10n infrastructure where to find the repository, and where inside 205that repository the ``l10n.ini`` file is. This is needed because for local 206builds, :file:`mail/locales/l10n.ini` references 207:file:`mozilla/toolkit/locales/l10n.ini`, which is where the comm-central 208build setup expects toolkit to be. 209 210Now that the directories exposed to l10n are known, we can talk about the 211supported file formats. 212 213File formats 214------------ 215 216The following file formats are known to the l10n tool chains: 217 218Fluent 219 Used in Firefox UI, both declarative and programmatically. 220DTD 221 Deprecated. Used in XUL and XHTML. 222Properties 223 Used from JavaScript and C++. When used from js, also comes with 224 `plural support <https://developer.mozilla.org/docs/Mozilla/Localization/Localization_and_Plurals>`_. 225ini 226 Used by the crashreporter and updater, avoid if possible. 227inc 228 Used during builds, for example to create metadata for 229 language packs or bookmarks. 230 231Adding new formats involves changing various different tools, and is strongly 232discouraged. 233 234Exceptions 235---------- 236Generally, anything that exists in ``en-US`` needs a one-to-one mapping in 237all localizations. There are a few cases where that's not wanted, notably 238around locale configuration and locale-dependent metadata. 239 240For optional strings and files, l10n-merge won't add ``en-US`` content if 241the localization doesn't have that content. 242 243For the TOML files, the 244`[[filters]] documentation <https://moz-l10n-config.readthedocs.io/en/latest/fileformat.html#filters>`_ 245is a good reference. In short, filters match the localized source code, optionally 246a ``key``, and an action. An example like 247 248.. code-block:: toml 249 250 [[filters]] 251 path = "{l}browser/defines.inc" 252 key = "MOZ_LANGPACK_CONTRIBUTORS" 253 action = "ignore" 254 255indicates that the ``MOZ_LANGPACK_CONTRIBUTORS`` in ``browser/defines.inc`` 256is optional. 257 258For the legacy ini configuration files, there's a Python module 259``filter.py`` next to the main ``l10n.ini``, implementing :py:func:`test`, with the following 260signature 261 262.. code-block:: python 263 264 def test(mod, path, entity = None): 265 if does_not_matter: 266 return "ignore" 267 if show_but_do_not_merge: 268 return "report" 269 # default behavior, localizer or build need to do something 270 return "error" 271 272For any missing file, this function is called with ``mod`` being 273the *module*, and ``path`` being the relative path inside 274:file:`locales/en-US`. The module is the top-level dir as referenced in 275:file:`l10n.ini`. 276 277For missing strings, the :py:data:`entity` parameter is the key of the string 278in the en-US file. 279 280l10n-merge 281========== 282 283The chrome registry in Gecko doesn't support fallback from a localization to ``en-US`` at runtime. 284Thus, the build needs to ensure that the localization as it's built into 285the package has all required strings, and that the strings don't contain 286errors. To ensure that, we're *merging* the localization and ``en-US`` 287at build time, nick-named l10n-merge. 288 289For Fluent, we're also removing erroneous messages. For many errors in Fluent, 290that's cosmetic, but when a localization has different values or attributes 291on a message, that's actually important so that the DOM bindings of Fluent 292can apply the translation without having to load the ``en-US`` source to 293compare against. 294 295The process can be manually triggered via 296 297.. code-block:: bash 298 299 $> ./mach build merge-$AB_CD 300 301It creates another directory in the object dir, :file:`browser/locales/merge-dir/$AB_CD`, in 302which the sanitized files are stored. The actual repackaging process only looks 303in the merged directory, so the preparation steps of l10n-merge need to ensure 304that all files are generated or copied. 305 306l10n-merge modifies a file if it supports the particular file type, and there 307are missing strings which are not filtered out, or if an existing string 308shows an error. See the Checks section below for details. If the files are 309not modified, l10n-merge copies them over to the respective location in the 310merge dir. 311 312Checks 313------ 314 315As part of the build and other localization tool chains, we run a variety 316of source-based checks. Think of them as linters. 317 318The suite of checks is usually determined by file type, i.e., there's a 319suite of checks for DTD files and one for properties files, etc. 320 321Localizations 322------------- 323 324Now that we talked in-depth about how to expose content to localizers, 325where are the localizations? 326 327We host a mercurial repository per locale. All of our 328localizations can be found on https://hg.mozilla.org/l10n-central/. 329 330You can search inside our localized files on 331`Transvision <https://transvision.mozfr.org/>`_. 332