1.. _mozbuild-files: 2 3=============== 4moz.build Files 5=============== 6 7``moz.build`` files are the mechanism by which tree metadata (notably 8the build configuration) is defined. 9 10Directories in the tree contain ``moz.build`` files which declare 11functionality for their respective part of the tree. This includes 12things such as the list of C++ files to compile, where to find tests, 13etc. 14 15``moz.build`` files are actually Python scripts. However, their 16execution is governed by special rules. This is explained below. 17 18moz.build Python Sandbox 19======================== 20 21As mentioned above, ``moz.build`` files are Python scripts. However, 22they are executed in a special Python *sandbox* that significantly 23changes and limits the execution environment. The environment is so 24different, it's doubtful most ``moz.build`` files would execute without 25error if executed by a vanilla Python interpreter (e.g. ``python 26moz.build``. 27 28The following properties make execution of ``moz.build`` files special: 29 301. The execution environment exposes a limited subset of Python. 312. There is a special set of global symbols and an enforced naming 32 convention of symbols. 333. Some symbols are inherited from previously-executed ``moz.build`` 34 files. 35 36The limited subset of Python is actually an extremely limited subset. 37Only a few symbols from ``__builtin__`` are exposed. These include 38``True``, ``False``, ``None``, ``sorted``, ``int``, and ``set``. Global 39functions like ``import``, ``print``, and ``open`` aren't available. 40Without these, ``moz.build`` files can do very little. *This is by design*. 41 42The execution sandbox treats all ``UPPERCASE`` variables specially. Any 43``UPPERCASE`` variable must be known to the sandbox before the script 44executes. Any attempt to read or write to an unknown ``UPPERCASE`` 45variable will result in an exception being raised. Furthermore, the 46types of all ``UPPERCASE`` variables is strictly enforced. Attempts to 47assign an incompatible type to an ``UPPERCASE`` variable will result in 48an exception being raised. 49 50The strictness of behavior with ``UPPERCASE`` variables is a very 51intentional design decision. By ensuring strict behavior, any operation 52involving an ``UPPERCASE`` variable is guaranteed to have well-defined 53side-effects. Previously, when the build configuration was defined in 54``Makefiles``, assignments to variables that did nothing would go 55unnoticed. ``moz.build`` files fix this problem by eliminating the 56potential for false promises. 57 58After a ``moz.build`` file has completed execution, only the 59``UPPERCASE`` variables are used to retrieve state. 60 61The set of variables and functions available to the Python sandbox is 62defined by the :py:mod:`mozbuild.frontend.context` module. The 63data structures in this module are consumed by the 64:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct 65the sandbox. There are tests to ensure that the set of symbols exposed 66to an empty sandbox are all defined in the ``context`` module. 67This module also contains documentation for each symbol, so nothing can 68sneak into the sandbox without being explicitly defined and documented. 69 70Reading and Traversing moz.build Files 71====================================== 72 73The process for reading ``moz.build`` files roughly consists of: 74 751. Start at the root ``moz.build`` (``<topsrcdir>/moz.build``). 762. Evaluate the ``moz.build`` file in a new sandbox. 773. Emit the main *context* and any *sub-contexts* from the executed 78 sandbox. 794. Extract a set of ``moz.build`` files to execute next. 805. For each additional ``moz.build`` file, goto #2 and repeat until all 81 referenced files have executed. 82 83From the perspective of the consumer, the output of reading is a stream 84of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each 85``Context`` defines a particular aspect of data. Consumers iterate over 86these objects and do something with the data inside. Each object is 87essentially a dictionary of all the ``UPPERCASE`` variables populated 88during its execution. 89 90.. note:: 91 92 Historically, there was only one ``context`` per ``moz.build`` file. 93 As the number of things tracked by ``moz.build`` files grew and more 94 and more complex processing was desired, it was necessary to split these 95 contexts into multiple logical parts. It is now common to emit 96 multiple contexts per ``moz.build`` file. 97 98Build System Reading Mode 99------------------------- 100 101The traditional mode of evaluation of ``moz.build`` files is what's 102called *build system traversal mode.* In this mode, the ``CONFIG`` 103variable in each ``moz.build`` sandbox is populated from data coming 104from ``config.status``, which is produced by ``configure``. 105 106During evaluation, ``moz.build`` files often make decisions conditional 107on the state of the build configuration. e.g. *only compile foo.cpp if 108feature X is enabled*. 109 110In this mode, traversal of ``moz.build`` files is governed by variables 111like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child 112directory, ``foo``, you would add ``DIRS += ['foo']`` to a ``moz.build`` 113file and ``foo/moz.build`` would be evaluated. 114 115.. _mozbuild_fs_reading_mode: 116 117Filesystem Reading Mode 118----------------------- 119 120There is an alternative reading mode that doesn't involve the build 121system and doesn't use ``DIRS`` variables to control traversal into 122child directories. This mode is called *filesystem reading mode*. 123 124In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty 125object. Accessing all but a few special variables will return an empty 126value. This means that nearly all ``if CONFIG['FOO']:`` branches will 127not be taken. 128 129Instead of using content from within the evaluated ``moz.build`` 130file to drive traversal into subsequent ``moz.build`` files, the set 131of files to evaluate is controlled by the thing doing the reading. 132 133A single ``moz.build`` file is not guaranteed to be executable in 134isolation. Instead, we must evaluate all *parent* ``moz.build`` files 135first. For example, in order to evaluate ``/foo/moz.build``, one must 136execute ``/moz.build`` and have its state influence the execution of 137``/foo/moz.build``. 138 139Filesystem reading mode is utilized to power the 140:ref:`mozbuild_files_metadata` feature. 141 142Technical Details 143----------------- 144 145The code for reading ``moz.build`` files lives in 146:py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results 147(:py:class:`mozbuild.frontend.context.Context`) are passed into 148:py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined 149in :py:mod:`mozbuild.frontend.data`. Each class in this module defines a 150domain-specific component of tree metadata. e.g. there will be separate 151classes that represent a JavaScript file vs a compiled C++ file or test 152manifests. This means downstream consumers of this data can filter on class 153types to only consume what they are interested in. 154 155There is no well-defined mapping between ``moz.build`` file instances 156and the number of :py:mod:`mozbuild.frontend.data` classes derived from 157each. Depending on the content of the ``moz.build`` file, there may be 1 158object derived or 100. 159 160The purpose of the ``emitter`` layer between low-level sandbox execution 161and metadata representation is to facilitate a unified normalization and 162verification step. There are multiple downstream consumers of the 163``moz.build``-derived data and many will perform the same actions. This 164logic can be complicated, so we have a component dedicated to it. 165 166:py:class:`mozbuild.frontend.reader.BuildReader`` and 167:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a 168stream-based API courtesy of generators. When you hook them up properly, 169the :py:mod:`mozbuild.frontend.data` classes are emitted before all 170``moz.build`` files have been read. This means that downstream errors 171are raised soon after sandbox execution. 172 173Lots of the code for evaluating Python sandboxes is applicable to 174non-Mozilla systems. In theory, it could be extracted into a standalone 175and generic package. However, until there is a need, there will 176likely be some tightly coupled bits. 177