1.. _test_manifests: 2 3============== 4Test Manifests 5============== 6 7Many test suites have their test metadata defined in files called 8**test manifests**. 9 10Test manifests are divided into two flavors: :ref:`manifestparser_manifests` 11and :ref:`reftest_manifests`. 12 13Naming Convention 14================= 15 16The build system does not enforce file naming for test manifest files. 17However, the following convention is used. 18 19mochitest.ini 20 For the *plain* flavor of mochitests. 21 22chrome.ini 23 For the *chrome* flavor of mochitests. 24 25browser.ini 26 For the *browser chrome* flavor of mochitests. 27 28a11y.ini 29 For the *a11y* flavor of mochitests. 30 31xpcshell.ini 32 For *xpcshell* tests. 33 34.. _manifestparser_manifests: 35 36ManifestParser Manifests 37========================== 38 39ManifestParser manifests are essentially ini files that conform to a basic 40set of assumptions. 41 42The :doc:`reference documentation </mozbase/manifestparser>` 43for manifestparser manifests describes the basic format of test manifests. 44 45In summary, manifests are ini files with section names describing test files:: 46 47 [test_foo.js] 48 [test_bar.js] 49 50Keys under sections can hold metadata about each test:: 51 52 [test_foo.js] 53 skip-if = os == "win" 54 [test_foo.js] 55 skip-if = os == "linux" && debug 56 [test_baz.js] 57 fail-if = os == "mac" || os == "android" 58 59There is a special **DEFAULT** section whose keys/metadata apply to all 60sections/tests:: 61 62 [DEFAULT] 63 property = value 64 65 [test_foo.js] 66 67In the above example, **test_foo.js** inherits the metadata **property = value** 68from the **DEFAULT** section. 69 70Recognized Metadata 71------------------- 72 73Test manifests can define some common keys/metadata to influence behavior. 74Those keys are as follows: 75 76head 77 List of files that will be executed before the test file. (Used in 78 xpcshell tests.) 79 80tail 81 List of files that will be executed after the test file. (Used in 82 xpcshell tests.) 83 84support-files 85 List of additional files required to run tests. This is typically 86 defined in the **DEFAULT** section. 87 88 Unlike other file lists, *support-files* supports a globbing mechanism 89 to facilitate pulling in many files with minimal typing. This globbing 90 mechanism is activated if an entry in this value contains a ``*`` 91 character. A single ``*`` will wildcard match all files in a directory. 92 A double ``**`` will descend into child directories. For example, 93 ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where 94 ``data/**`` will match ``data/foo`` and ``data/subdir/bar``. 95 96 Support files starting with ``/`` are placed in a root directory, rather 97 than a location determined by the manifest location. For mochitests, 98 this allows for the placement of files at the server root. The source 99 file is selected from the base name (e.g., ``foo`` for ``/path/foo``). 100 Files starting with ``/`` cannot be selected using globbing. 101 102 Some support files are used by tests across multiple directories. In 103 this case, a test depending on a support file from another directory 104 must note that dependency with the path to the required support file 105 in its own **support-files** entry. These use a syntax where paths 106 starting with ``!/`` will indicate the beginning of the path to a 107 shared support file starting from the root of the srcdir. For example, 108 if a manifest at ``dom/base/test/mochitest.ini`` has a support file, 109 ``dom/base/test/server-script.sjs``, and a mochitest in 110 ``dom/workers/test`` depends on that support file, the test manifest 111 at ``dom/workers/test/mochitest.ini`` must include 112 ``!/dom/base/test/server-script.sjs`` in its **support-files** entry. 113 114generated-files 115 List of files that are generated as part of the build and don't exist in 116 the source tree. 117 118 The build system assumes that each manifest file, test file, and file 119 listed in **head**, **tail**, and **support-files** is static and 120 provided by the source tree (and not automatically generated as part 121 of the build). This variable tells the build system not to make this 122 assumption. 123 124 This variable will likely go away sometime once all generated files are 125 accounted for in the build config. 126 127 If a generated file is not listed in this key, a clobber build will 128 likely fail. 129 130dupe-manifest 131 Record that this manifest duplicates another manifest. 132 133 The common scenario is two manifest files will include a shared 134 manifest file via the ``[include:file]`` special section. The build 135 system enforces that each test file is only provided by a single 136 manifest. Having this key present bypasses that check. 137 138 The value of this key is ignored. 139 140skip-if 141 Skip this test if the specified condition is true. 142 See :ref:`manifest_filter_language`. 143 144 Conditions can be specified on multiple lines, where each line is implicitly 145 joined by a logical OR (``||``). This makes it easier to add comments to 146 distinct failures. For example: 147 148 .. parsed-literal:: 149 150 [test_foo.js] 151 skip-if = 152 os == "mac" && fission # bug 123 - fails on fission 153 os == "windows" && debug # bug 456 - hits an assertion 154 155fail-if 156 Expect test failure if the specified condition is true. 157 See :ref:`manifest_filter_language`. 158 159 Conditions can be specified on multiple lines (see ``skip-if``). 160 161run-sequentially 162 If present, the test should not be run in parallel with other tests. 163 164 Some test harnesses support parallel test execution on separate processes 165 and/or threads (behavior varies by test harness). If this key is present, 166 the test harness should not attempt to run this test in parallel with any 167 other test. 168 169 By convention, the value of this key is a string describing why the test 170 can't be run in parallel. 171 172scheme 173 Changes the scheme and domain from which the test runs. (Only used in mochitest suites) 174 175 There are two possible values: 176 - ``http`` (default): The test will run from http://mochi.test:8888 177 - ``https``: The test will run from https://example.com:443 178 179.. _manifest_filter_language: 180 181Manifest Filter Language 182------------------------ 183 184Some manifest keys accept a special filter syntax as their values. These 185values are essentially boolean expressions that are evaluated at test 186execution time. 187 188The expressions can reference a well-defined set of variables, such as 189``os`` and ``debug``. These variables are populated from the 190``mozinfo.json`` file. For the full list of available variables, see 191the :ref:`mozinfo documentation <mozinfo_attributes>`. 192 193See 194`the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestparser/manifestparser/manifestparser.py>`_ for the full documentation of the 195expression syntax until it is documented here. 196 197.. todo:: 198 199 Document manifest filter language. 200 201.. _manifest_file_installation: 202 203File Installation 204----------------- 205 206Files referenced by manifests are automatically installed into the object 207directory into paths defined in 208:py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`. 209 210Relative paths resolving to parent directory (e.g. 211``support-files = ../foo.txt`` have special behavior. 212 213For ``support-files``, the file will be installed to the default destination 214for that manifest. Only the file's base name is used to construct the final 215path: directories are irrelevant. Files starting with ``/`` are an exception, 216these are installed relative to the root of the destination; the base name is 217instead used to select the file.. 218 219For all other entry types, the file installation is skipped. 220 221.. _reftest_manifests: 222 223Reftest Manifests 224================= 225 226See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_. 227