1Optimization and SCHEDULES
2==========================
3
4Most optimization of builds and tests is handled with ``SCHEDULES``.
5The concept is this: we allocate tasks into named components, and associate a set of such components to each file in the source tree.
6Given a set of files changed in a push, we then calculate the union of components affected by each file, and remove tasks that are not tagged with any of them.
7
8This optimization system is intended to be *conservative*.
9It represents what could *possibly* be affected, rather than any intuitive notion of what tasks would be useful to run for changes to a particular file.
10For example:
11
12* ``dom/url/URL.cpp`` schedules tasks on all platform and could potentially cause failures in any test suite
13
14* ``dom/system/mac/CoreLocationLocationProvider.mm`` could not possibly affect any platform but ``macosx``, but potentially any test suite
15
16* ``python/mozbuild/mozbuild/preprocessor.py`` could possibly affect any platform, and should also schedule Python lint tasks
17
18Exclusive and Inclusive
19-----------------------
20
21The first wrinkle in this "simple" plan is that there are a lot of files, and for the most part they all affect most components.
22But there are some components which are only affected by a well-defined set of files.
23For example, a Python lint component need only be scheduled when Python files are changed.
24
25We divide the components into "exclusive" and "inclusive" components.
26Absent any other configuration, any file in the repository is assumed to affect all of the exclusive components and none of the inclusive components.
27
28Exclusive components can be thought of as a series of families.
29For example, the platform (linux, windows, macosx, android) is a component family.
30The test suite (mochitest, reftest, xpcshell, etc.) is another.
31By default, source files are associated with every component in every family.
32This means tasks tagged with an exclusive component will *always* run, unless none of the modified source files are associated with that component.
33
34But what if we only want to run a particular task when a pre-determined file is modified?
35This is where inclusive components are used.
36Any task tagged with an inclusive component will *only* be run when a source file associated with that component is modified.
37Lint tasks and well separated unittest tasks are good examples of things you might want to schedule inclusively.
38
39A good way to keep this straight is to think of exclusive platform-family components (``macosx``, ``android``, ``windows``, ``linux``) and inclusive linting components (``py-lint``, ``js-lint``).
40An arbitrary file in the repository affects all platform families, but does not necessarily require a lint run.
41But we can configure mac-only files such as ``CoreLocationLocationProvider.mm`` to affect exclusively ``macosx``, and Python files like ``preprocessor.py`` to affect ``py-lint`` in addition to the exclusive components.
42
43It is also possible to define a file as affecting an inclusive component and nothing else.
44For example, the source code and configuration for the Python linting tasks does not affect any tasks other than linting.
45
46.. note::
47
48    Most unit test suite tasks are allocated to components for their platform family and for the test suite.
49    This indicates that if a platform family is affected (for example, ``android``) then the builds for that platform should execute as well as the full test suite.
50    If only a single suite is affected (for example, by a change to a reftest source file), then the reftests should execute for all platforms.
51
52    However, some test suites, for which the set of contributing files are well-defined, are represented as inclusive components.
53    These components will not be executed by default for any platform families, but only when one or more of the contributing files are changed.
54
55Specification
56-------------
57
58Components are defined as either inclusive or exclusive in :py:mod:`mozbuild.schedules`.
59
60File Annotation
61:::::::::::::::
62
63Files are annotated with their affected components in ``moz.build`` files with stanzas like ::
64
65    with Files('**/*.py'):
66        SCHEDULES.inclusive += ['py-lint']
67
68for inclusive components and ::
69
70    with Files('*gradle*'):
71        SCHEDULES.exclusive = ['android']
72
73for exclusive components.
74Note the use of ``+=`` for inclusive compoenents (as this is adding to the existing set of affected components) but ``=`` for exclusive components (as this is resetting the affected set to something smaller).
75For cases where an inclusive component is affected exclusively (such as the python-lint configuration in the example above), that component can be assigned to ``SCHEDULES.exclusive``::
76
77    with Files('**/pep8rc'):
78        SCHEDULES.exclusive = ['py-lint']
79
80If multiple stanzas set ``SCHEDULES.exclusive``, the last one will take precedence.  Thus the following
81will set ``SCHEDULES.exclusive`` to ``hpux`` for all files except those under ``docs/``. ::
82
83    with Files('**'):
84        SCHEDULES.exclusive = ['hpux']
85
86    with Files('**/docs'):
87        SCHEDULES.exclusive = ['docs']
88
89Task Annotation
90:::::::::::::::
91
92Tasks are annotated with the components they belong to using the ``"skip-unless-schedules"`` optimization, which takes a list of components for this task::
93
94    task['optimization'] = {'skip-unless-schedules': ['windows', 'gtest']}
95
96For tests, this value is set automatically by the test transform based on the suite name and the platform family, doing the correct thing for inclusive test suites.
97Tests can also use a variety of other optimizers, such as ``relevant_tests``, ``bugbug`` (uses machine learning) or ``backstop`` (ensures regressions aren't missed).
98