1# General Test Guidelines 2 3### File Paths and Names 4 5When choosing where in the directory structure to put any new tests, 6try to follow the structure of existing tests for that specification; 7if there are no existing tests, it is generally recommended to create 8subdirectories for each section. 9 10Due to path length limitations on Windows, test paths must be less 11that 150 characters relative to the test root directory (this gives 12vendors just over 100 characters for their own paths when running in 13automation). 14 15File names should generally be somewhat descriptive of what is being 16tested; very generic names like `001.html` are discouraged. A common 17format is `test-topic-001.html`, where `test-topic` is a short 18identifier that describes the test. It should avoid conjunctions, 19articles, and prepositions as it should be as concise as possible. The 20integer that follows is normally just increased incrementally, and 21padded to three digits. (If you'd end up with more than 999 tests, 22your `test-topic` is probably too broad!) 23 24The test filename is significant in enabling specific optional features, such as HTTPS 25or server-side substitution. See the documentation on [file names flags][file-name-flags] 26for more details. 27 28In the css directory, the file names should be unique within the whole 29css/ directory, regardless of where they are in the directory structure. 30 31### HTTPS 32 33By default, tests are served over plain HTTP. If a test requires HTTPS 34it must be given a filename containing `.https.` e.g., 35`test-secure.https.html`, or be the generated service worker test of a 36`.https`-less `.any` test. For more details see the documentation on 37[file names][file-name-flags]. 38 39### HTTP2 40 41If a test must be served from an HTTP/2 server, it must be given a 42filename containing `.h2`. 43 44#### Support Files 45 46Various support files are available in in the directories named `/common/`, 47`/media/`, and `/css/support/`. Reusing existing resources is encouraged where 48possible, as is adding generally-useful files to these common areas rather than 49to specific test suites. 50 51 52#### Tools 53 54Sometimes you may want to add a script to the repository that's meant 55to be used from the command line, not from a browser (e.g., a script 56for generating test files). If you want to ensure (e.g., for security 57reasons) that such scripts will only be usable from the command line 58but won't be handled by the HTTP server then place them in a `tools` 59subdirectory at the appropriate level—the server will then return a 60404 if they are requested. 61 62For example, if you wanted to add a script for use with tests in the 63`notifications` directory, create the `notifications/tools` 64subdirectory and put your script there. 65 66 67### File Formats 68 69Tests are generally formatted as HTML (including XHTML) or XML (including SVG). 70Some test types support other formats: 71 72- [testharness.js tests](testharness) may be expressed as JavaScript files 73 ([the WPT server automatically generates the HTML documents for these][server 74 features]) 75- [WebDriver specification tests](wdspec) are expressed as Python files 76 77The best way to determine how to format a new test is to look at how 78similar tests have been formatted. You can also ask for advice in [the 79project's matrix channel][matrix]. 80 81 82### Character Encoding 83 84Except when specifically testing encoding, files must be encoded in 85UTF-8. In file formats where UTF-8 is not the default encoding, they 86must contain metadata to mark them as such (e.g., `<meta 87charset=utf-8>` in HTML files) or be pure ASCII. 88 89 90### Server Side Support 91 92The custom web server 93supports [a variety of features][server features] useful for testing 94browsers, including (but not limited to!) support for writing out 95appropriate domains and custom (per-file and per-directory) HTTP 96headers. 97 98 99### Be Short 100 101Tests should be as short as possible. For reftests in particular 102scrollbars at 800×600px window size must be avoided unless scrolling 103behavior is specifically being tested. For all tests extraneous 104elements on the page should be avoided so it is clear what is part of 105the test (for a typical testharness test, the only content on the page 106will be rendered by the harness itself). 107 108 109### Be Conservative 110 111Tests should generally avoid depending on edge case behavior of 112features that they don't explicitly intend on testing. For example, 113except where testing parsing, tests should contain 114no [parse errors](https://validator.nu). 115 116This is not, however, to discourage testing of edge cases or 117interactions between multiple features; such tests are an essential 118part of ensuring interoperability of the web platform. When possible, use the 119canonical support libraries provided by features; for more information, see the documentation on [testing interactions between features][interacting-features]. 120 121Tests should pass when the feature under test exposes the expected behavior, 122and they should fail when the feature under test is not implemented or is 123implemented incorrectly. Tests should not rely on unrelated features if doing 124so causes failures in the latest stable release of [Apple 125Safari][apple-safari], [Google Chrome][google-chrome], or [Mozilla 126Firefox][mozilla-firefox]. They should, therefore, not rely on any features 127aside from the one under test unless they are supported in all three browsers. 128 129Existing tests can be used as a guide to identify acceptable features. For 130language features that are not used in existing tests, community-maintained 131projects such as [the ECMAScript compatibility tables][es-compat] and 132[caniuse.com][caniuse] provide an overview of basic feature support across the 133browsers listed above. 134 135For JavaScript code that is re-used across many tests (e.g. `testharness.js` 136and the files located in the directory named `common`), only use language 137features that have been supported by each of the major browser engines above 138for over a year. This practice avoids introducing test failures for consumers 139maintaining older JavaScript runtimes. 140 141Patches to make tests run on older versions or other browsers will be accepted 142provided they are relatively simple and do not add undue complexity to the 143test. 144 145 146### Be Cross-Platform 147 148Tests should be as cross-platform as reasonably possible, working 149across different devices, screen resolutions, paper sizes, etc. The 150assumptions that can be relied on are documented [here][assumptions]; 151tests that rely on anything else should be manual tests that document 152their assumptions. 153 154Fonts cannot be relied on to be either installed or to have specific 155metrics. As such, in most cases when a known font is needed, [Ahem][ahem] 156should be used and loaded as a web font. In other cases, `@font-face` 157should be used. 158 159 160### Be Self-Contained 161 162Tests must not depend on external network resources, including 163w3c-test.org. When these tests are run on CI systems they are 164typically configured with access to external resources disabled, so 165tests that try to access them will fail. Where tests want to use 166multiple hosts this is possible through a known set of subdomains and 167the [text substitution features of wptserve](server-features). 168 169 170### Be Self-Describing 171 172Tests should make it obvious when they pass and when they fail. It 173shouldn't be necessary to consult the specification to figure out 174whether a test has passed of failed. 175 176 177### Style Rules 178 179A number of style rules should be applied to the test file. These are 180not uniformly enforced throughout the existing tests, but will be for 181new tests. Any of these rules may be broken if the test demands it: 182 183 * No trailing whitespace 184 * Use spaces rather than tabs for indentation 185 * Use UNIX-style line endings (i.e. no CR characters at EOL) 186 187We have a lint tool for catching these and other common mistakes. You 188can run it manually by starting the `wpt` executable from the root of 189your local web-platform-tests working directory, and invoking the 190`lint` subcommand, like this: 191 192``` 193./wpt lint 194``` 195 196The lint tool is also run automatically for every submitted pull request, 197and reviewers will not merge branches with tests that have lint errors, so 198you must fix any errors the lint tool reports. For details on doing that, 199see the [lint-tool documentation][lint-tool]. 200 201But in the unusual case of error reports for things essential to a certain 202test or that for other exceptional reasons shouldn't prevent a merge of a 203test, update and commit the `lint.ignore` file in the web-platform-tests 204root directory to suppress the error reports. For details on doing that, 205see the [lint-tool documentation][lint-tool]. 206 207 208## CSS-Specific Requirements 209 210In order to be included in an official specification test suite, tests 211for CSS have some additional requirements for: 212 213* [Metadata][css-metadata], and 214* [User style sheets][css-user-styles]. 215 216 217[server features]: server-features 218[assumptions]: assumptions 219[ahem]: ahem 220[matrix]: https://app.element.io/#/room/#wpt:matrix.org 221[lint-tool]: lint-tool 222[css-metadata]: css-metadata 223[css-user-styles]: css-user-styles 224[file-name-flags]: file-names 225[interacting-features]: interacting-features 226[mozilla-firefox]: https://mozilla.org/firefox 227[google-chrome]: https://google.com/chrome/browser/desktop/ 228[apple-safari]: https://apple.com/safari 229[es-compat]: https://kangax.github.io/compat-table/ 230[caniuse]: https://caniuse.com/ 231