1.. _test_manifests:
2
3==============
4Test Manifests
5==============
6
7Many test suites have their test metadata defined in files called
8**test manifests**.
9
10Test manifests are divided into two flavors: :ref:`manifestparser_manifests`
11and :ref:`reftest_manifests`.
12
13Naming Convention
14=================
15
16The build system does not enforce file naming for test manifest files.
17However, the following convention is used.
18
19mochitest.ini
20   For the *plain* flavor of mochitests.
21
22chrome.ini
23   For the *chrome* flavor of mochitests.
24
25browser.ini
26   For the *browser chrome* flavor of mochitests.
27
28a11y.ini
29   For the *a11y* flavor of mochitests.
30
31xpcshell.ini
32   For *xpcshell* tests.
33
34.. _manifestparser_manifests:
35
36ManifestParser Manifests
37==========================
38
39ManifestParser manifests are essentially ini files that conform to a basic
40set of assumptions.
41
42The :doc:`reference documentation </mozbase/manifestparser>`
43for manifestparser manifests describes the basic format of test manifests.
44
45In summary, manifests are ini files with section names describing test files::
46
47    [test_foo.js]
48    [test_bar.js]
49
50Keys under sections can hold metadata about each test::
51
52    [test_foo.js]
53    skip-if = os == "win"
54    [test_foo.js]
55    skip-if = os == "linux" && debug
56    [test_baz.js]
57    fail-if = os == "mac" || os == "android"
58
59There is a special **DEFAULT** section whose keys/metadata apply to all
60sections/tests::
61
62    [DEFAULT]
63    property = value
64
65    [test_foo.js]
66
67In the above example, **test_foo.js** inherits the metadata **property = value**
68from the **DEFAULT** section.
69
70Recognized Metadata
71-------------------
72
73Test manifests can define some common keys/metadata to influence behavior.
74Those keys are as follows:
75
76head
77   List of files that will be executed before the test file. (Used in
78   xpcshell tests.)
79
80tail
81   List of files that will be executed after the test file. (Used in
82   xpcshell tests.)
83
84support-files
85   List of additional files required to run tests. This is typically
86   defined in the **DEFAULT** section.
87
88   Unlike other file lists, *support-files* supports a globbing mechanism
89   to facilitate pulling in many files with minimal typing. This globbing
90   mechanism is activated if an entry in this value contains a ``*``
91   character. A single ``*`` will wildcard match all files in a directory.
92   A double ``**`` will descend into child directories. For example,
93   ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where
94   ``data/**`` will match ``data/foo`` and ``data/subdir/bar``.
95
96   Support files starting with ``/`` are placed in a root directory, rather
97   than a location determined by the manifest location. For mochitests,
98   this allows for the placement of files at the server root. The source
99   file is selected from the base name (e.g., ``foo`` for ``/path/foo``).
100   Files starting with ``/`` cannot be selected using globbing.
101
102   Some support files are used by tests across multiple directories. In
103   this case, a test depending on a support file from another directory
104   must note that dependency with the path to the required support file
105   in its own **support-files** entry. These use a syntax where paths
106   starting with ``!/`` will indicate the beginning of the path to a
107   shared support file starting from the root of the srcdir. For example,
108   if a manifest at ``dom/base/test/mochitest.ini`` has a support file,
109   ``dom/base/test/server-script.sjs``, and a mochitest in
110   ``dom/workers/test`` depends on that support file, the test manifest
111   at ``dom/workers/test/mochitest.ini`` must include
112   ``!/dom/base/test/server-script.sjs`` in its **support-files** entry.
113
114generated-files
115   List of files that are generated as part of the build and don't exist in
116   the source tree.
117
118   The build system assumes that each manifest file, test file, and file
119   listed in **head**, **tail**, and **support-files** is static and
120   provided by the source tree (and not automatically generated as part
121   of the build). This variable tells the build system not to make this
122   assumption.
123
124   This variable will likely go away sometime once all generated files are
125   accounted for in the build config.
126
127   If a generated file is not listed in this key, a clobber build will
128   likely fail.
129
130dupe-manifest
131   Record that this manifest duplicates another manifest.
132
133   The common scenario is two manifest files will include a shared
134   manifest file via the ``[include:file]`` special section. The build
135   system enforces that each test file is only provided by a single
136   manifest. Having this key present bypasses that check.
137
138   The value of this key is ignored.
139
140skip-if
141   Skip this test if the specified condition is true.
142   See :ref:`manifest_filter_language`.
143
144   Conditions can be specified on multiple lines, where each line is implicitly
145   joined by a logical OR (``||``). This makes it easier to add comments to
146   distinct failures. For example:
147
148   .. parsed-literal::
149
150      [test_foo.js]
151      skip-if =
152          os == "mac" && fission  # bug 123 - fails on fission
153          os == "windows" && debug  # bug 456 - hits an assertion
154
155fail-if
156   Expect test failure if the specified condition is true.
157   See :ref:`manifest_filter_language`.
158
159   Conditions can be specified on multiple lines (see ``skip-if``).
160
161run-sequentially
162   If present, the test should not be run in parallel with other tests.
163
164   Some test harnesses support parallel test execution on separate processes
165   and/or threads (behavior varies by test harness). If this key is present,
166   the test harness should not attempt to run this test in parallel with any
167   other test.
168
169   By convention, the value of this key is a string describing why the test
170   can't be run in parallel.
171
172scheme
173   Changes the scheme and domain from which the test runs. (Only used in mochitest suites)
174
175   There are two possible values:
176      - ``http`` (default): The test will run from http://mochi.test:8888
177      - ``https``: The test will run from https://example.com:443
178
179.. _manifest_filter_language:
180
181Manifest Filter Language
182------------------------
183
184Some manifest keys accept a special filter syntax as their values. These
185values are essentially boolean expressions that are evaluated at test
186execution time.
187
188The expressions can reference a well-defined set of variables, such as
189``os`` and ``debug``. These variables are populated from the
190``mozinfo.json`` file. For the full list of available variables, see
191the :ref:`mozinfo documentation <mozinfo_attributes>`.
192
193See
194`the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestparser/manifestparser/manifestparser.py>`_ for the full documentation of the
195expression syntax until it is documented here.
196
197.. todo::
198
199   Document manifest filter language.
200
201.. _manifest_file_installation:
202
203File Installation
204-----------------
205
206Files referenced by manifests are automatically installed into the object
207directory into paths defined in
208:py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`.
209
210Relative paths resolving to parent directory (e.g.
211``support-files = ../foo.txt`` have special behavior.
212
213For ``support-files``, the file will be installed to the default destination
214for that manifest. Only the file's base name is used to construct the final
215path: directories are irrelevant.  Files starting with ``/`` are an exception,
216these are installed relative to the root of the destination; the base name is
217instead used to select the file..
218
219For all other entry types, the file installation is skipped.
220
221.. _reftest_manifests:
222
223Reftest Manifests
224=================
225
226See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_.
227