1# web-platform-tests 2 3Interoperability between browsers is 4[critical](https://www.chromium.org/blink/platform-predictability) to Chromium's 5mission of improving the web. We believe that leveraging and contributing to a 6shared test suite is one of the most important tools in achieving 7interoperability between browsers. The [web-platform-tests 8repository](https://github.com/web-platform-tests/wpt) is the primary shared 9test suite where all browser engines are collaborating. 10 11Chromium has a 2-way import/export process with the upstream web-platform-tests 12repository, where tests are imported into 13[web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt) 14and any changes to the imported tests are also exported to web-platform-tests. 15 16See https://web-platform-tests.org/ for general documentation on 17web-platform-tests, including tips for writing and reviewing tests. 18 19[TOC] 20 21## Writing tests 22 23To contribute changes to web-platform-tests, just commit your changes directly 24to [web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt) 25and the changes will be automatically upstreamed within 24 hours. 26 27Changes involving adding, removing or modifying tests can all be upstreamed. 28Any changes outside of 29[external/wpt](../../third_party/blink/web_tests/external/wpt) will not be 30upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, 31will also not be upstreamed. 32 33Running the web tests will automatically regenerate MANIFEST.json to pick up 34any local modifications. 35 36Most tests are written using testharness.js, see 37[Writing Web Tests](./writing_web_tests.md) and 38[Web Tests Tips](./web_tests_tips.md) for general guidelines. 39 40### Write tests against specifications 41 42Tests in web-platform-tests are expected to match behavior defined by the 43relevant specification. In other words, all assertions that a test makes 44should be derived from a specification's normative requirements, and not go 45beyond them. It is often necessary to change the specification to clarify what 46is and isn't required. 47 48When implementation experience is needed to inform the specification work, 49[tentative tests](https://web-platform-tests.org/writing-tests/file-names.html) 50can be appropriate. It should be apparent in context why the test is tentative 51and what needs to be resolved to make it non-tentative. 52 53### Tests that require testing APIs 54 55#### `testdriver.js` 56 57[testdriver.js](https://web-platform-tests.org/writing-tests/testdriver.html) 58provides a means to automate tests that cannot be written purely using web 59platform APIs, similar to `internals.*` and `eventSender.*` in regular Blink 60web tests. 61 62If no testdriver.js API exists, check if it's a 63[known issue](https://github.com/web-platform-tests/wpt/labels/testdriver.js) 64and otherwise consider filing a new issue. For instructions on how to add a new 65testing API, see [WPT Test Automation for 66Chromium](https://docs.google.com/document/d/18BpD41vyX1cFZ77CE0a_DJYlGpdvyLlx3pwXVRxUzvI/preview#) 67 68#### MojoJS 69 70Some specs may define testing APIs (e.g. 71[WebUSB](https://wicg.github.io/webusb/test/)), which may be polyfilled with 72internal API like [MojoJS](../../mojo/public/js/README.md). MojoJS is only 73allowed in WPT for this purpose. Please reach out to 74ecosystem-infra@chromium.org before following the process below for adding a new 75test-only API: 76 77 1. Create a full list of `*.mojom.js` files that you need, including all 78 dependencies. `mojo_bindings.js` loads dependencies recursively by default, 79 so you can check the network panel of DevTools to see the full list of 80 dependencies it loads. 81 2. Check [FILES.cfg](../../chrome/tools/build/linux/FILES.cfg) and add any 82 missing `*.mojom.js` files to the `mojojs.zip` archive. Globs are supported 83 in `filename`. Do not copy Mojom bindings into WPT. 84 3. Meanwhile in Chromium, you can create a helper for your WPT tests to do 85 browser-specific setup using 86 [test-only-api.js](../../third_party/blink/web_tests/external/wpt/resources/test-only-api.js). 87 See 88 [webxr_util.js](../../third_party/blink/web_tests/external/wpt/webxr/resources/webxr_util.js) 89 as an example. You can write tests using this helper right away, but they 90 will not work upstream (i.e. on https://wpt.fyi ) until your change in step 91 2 is included in official channels, as `mojojs.zip` is built alongside with 92 Chrome. 93 94#### `wpt_automation` 95 96An alternative to the above options is to write manual tests that are automated 97with scripts from 98[wpt_automation](../../third_party/blink/web_tests/external/wpt_automation). 99Injection of JS in manual tests is determined by `loadAutomationScript` in 100[testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js). 101 102Such tests still require case-by-case automation to run for other browser 103engines, but are more valuable than purely manual tests. 104 105Manual tests that have no automation are still imported, but skipped in 106[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests); see 107[issue 738489](https://crbug.com/738489). 108 109### Contribution process 110 111Changes made in 112[web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt) are 113[automatically exported to GitHub](#exporting-tests). 114 115It's still possible to make direct pull requests to web-platform-tests, see 116https://web-platform-tests.org/writing-tests/github-intro.html. 117 118### Adding new top-level directories 119 120Entirely new top-level directories should generally be added upstream, since 121that's the only way to add an OWNERS file upstream. After adding a new top-level 122directory upstream, you should add a line for it in `W3CImportExpectations`. 123 124Adding the new directory (and `W3CImportExpectations` entry) in Chromium and 125later adding an OWNERS file upstream also works. 126 127### `wpt_internal` 128 129It is sometimes desirable to write WPT tests that either test Chromium-specific 130behaviors, or that cannot yet be upstreamed to WPT (e.g. because the spec is 131very nascent). For these cases, we maintain a separate directory, 132[wpt_internal](../third_party/blink/web_tests/wpt_internal) that runs under the 133WPT testing infrastructure (e.g. uses wptserve, etc), but which is not 134upstreamed to WPT. 135 136Please see the `wpt_internal` 137[README](../third_party/blink/web_tests/wpt_internal/README) for more details. 138 139**Note**: A significant downside of `wpt_internal` is that your tests may be 140broken by upstream changes to the resources scripts (e.g. `testharness.js`), as 141`wpt_internal` does not use the forked version of `testharness.js` used by all 142other non-`external/wpt` tests. Use of [WPT-NOTIFY](#wpt_notify) is recommended 143to ensure you are notified of breakages. 144 145## Running tests 146 147Same as Blink web tests, you can use 148[`run_web_tests.py`](web_tests.md#running-the-tests) to run any WPT test. 149 150One thing to note is that glob patterns for WPT tests are not yet supported. 151 152See [Running WPT tests in Content Shell](web_tests_in_content_shell.md#Running-WPT-Tests-in-Content-Shell) 153for debugging etc. 154 155## Reviewing tests 156 157Anyone who can review code and tests in Chromium can also review changes in 158[external/wpt](../../third_party/blink/web_tests/external/wpt) 159that will be automatically upstreamed. There will be no additional review in 160web-platform-tests as part of the export process. 161 162If upstream reviewers have feedback on the changes, discuss on the pull request 163created during export, and if necessary work on a new pull request to iterate 164until everyone is satisfied. 165 166When reviewing tests, check that they match the relevant specification, which 167may not fully match the implementation. See also 168[Write tests against specifications](#Write-tests-against-specifications). 169 170## Importing tests 171 172Chromium has a [mirror](https://chromium.googlesource.com/external/w3c/web-platform-tests/) 173of the GitHub repo and periodically imports a subset of the tests to 174run as part of the regular Blink web test testing process. 175 176The goals of this process are to be able to run web-platform-tests unmodified 177locally just as easily as we can run the Blink tests, and ensure that we are 178tracking tip-of-tree in the web-platform-tests repository as closely as 179possible, and running as many of the tests as possible. 180 181### Automatic import process 182 183There is an automatic process for updating the Chromium copy of 184web-platform-tests. The import is done by the builder [wpt-importer 185builder][wpt-importer]. 186 187The easiest way to check the status of recent imports is to look at: 188 189- Recent logs on LUCI for [wpt-importer builder][wpt-importer] 190- Recent CLs created by [WPT 191 Autoroller](https://chromium-review.googlesource.com/q/owner:wpt-autoroller%2540chops-service-accounts.iam.gserviceaccount.com). 192 193The import jobs will generally be green if either there was nothing to do, 194or a CL was successfully submitted. 195 196For maintainers: 197 198- The source lives in 199 [third_party/blink/tools/wpt_import.py](../../third_party/blink/tools/wpt_import.py). 200- If the importer starts misbehaving, it can be disabled by landing a 201 [CL to skip the update step](https://crrev.com/c/1961906/). 202 203### WPT-NOTIFY 204 205Test owners can elect to have the importer automatically file bugs against a 206component when imported changes introduce failures. This includes new tests that 207fail in Chromium, as well as new failures introduced to an existing test. To 208opt-in to this functionality, create an `OWNERS` file in the appropriate 209`external/wpt/` subdirectory that contains the `WPT-NOTIFY` tag. For example, 210`external/wpt/css/css-grid/OWNERS` looks like: 211 212``` 213# TEAM: layout-dev@chromium.org 214# COMPONENT: Blink>Layout>Grid 215# WPT-NOTIFY: true 216``` 217 218When a test under `external/wpt/css/css-grid/` newly fails in a WPT import, the 219importer will automatically file a bug against the Blink>Layout>Grid component 220in [crbug.com][https://crbug.com], with details of which test failed and the 221output. 222 223Note that we are considering making WPT-NOTIFY opt-out instead of opt-in: see 224https://crbug.com/845232 225 226### Skipped tests (and how to re-enable them) 227 228We control which tests are imported via a file called 229[W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations), 230which has a list of directories to skip while importing. 231 232In addition to the directories and tests explicitly skipped there, tests may 233also be skipped for a couple other reasons, e.g. if the file path is too long 234for Windows. To check what files are skipped in import, check the recent logs 235for [wpt-importer builder][wpt-importer]. 236 237If you wish to un-skip some of the directories currently skipped in 238`W3CImportExpectations`, you can modify that file locally and commit it, and on 239the next auto-import, the new tests should be imported. 240 241If you want to import immediately (in order to try the tests out locally, etc) 242you can also run `wpt-import`, but this is not required. 243 244Remember your import might fail due to GitHub's limit for unauthenticated 245requests, so consider [passing your GitHub credentials](#GitHub-credentials) to 246the script. 247 248### Waterfall failures caused by automatic imports. 249 250If there are new test failures that start after an auto-import, 251there are several possible causes, including: 252 253 1. New baselines for flaky tests were added (https://crbug.com/701234). 254 2. Modified tests should have new results for non-Release builds but they weren't added (https://crbug.com/725160). 255 3. New baselines were added for tests with non-deterministic test results (https://crbug.com/705125). 256 257Because these tests are imported from the Web Platform tests, it is better 258to have them in the repository (and marked failing) than not, so prefer to 259[add test expectations](web_test_expectations.md) rather than reverting. 260However, if a huge number of tests are failing, please revert the CL so we 261can fix it manually. 262 263[wpt-importer]: https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-importer 264 265## Exporting tests 266 267If you upload a CL with any changes in 268[third_party/blink/web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt), 269once you add reviewers the exporter will create a provisional pull request with 270those changes in the [upstream WPT GitHub repository](https://github.com/web-platform-tests/wpt/). 271The exporter runs on [wpt-exporter builder][wpt-exporter]. 272 273Once you're ready to land your CL, please follow the link posted by the bot and 274check the status of the required checks of the GitHub PR. If it's green, go 275ahead landing your CL and the exporter will automatically merge the PR. 276 277If GitHub status is red on the PR, please try to resolve the failures before 278merging. If you run into any issues, or if you have a CL with WPT changes that 279the exporter did not pick up, please reach out to ecosystem-infra@chromium.org. 280 281Additional things to note: 282 283- CLs that change over 1000 files will not be exported. 284- All PRs use the 285 [`chromium-export`](https://github.com/web-platform-tests/wpt/pulls?utf8=%E2%9C%93&q=is%3Apr%20label%3Achromium-export) label. 286- All PRs for CLs that haven't yet been landed in Chromium also use the 287 [`do not merge yet`](https://github.com/web-platform-tests/wpt/pulls?q=is%3Apr+is%3Aopen+label%3A%22do+not+merge+yet%22) label. 288- The exporter cannot create upstream PRs for in-flight CLs with binary files 289 (e.g. webm files). An export PR will still be made after the CL lands. 290 291### Will the exported commits be linked to my GitHub profile? 292 293The email you commit with in Chromium will be the author of the commit on 294GitHub. You can [add it as a secondary address on your GitHub 295account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/) 296to link your exported commits to your GitHub profile. 297 298If you are a Googler, you can also register your GitHub account at go/github, 299making it easier for other Googlers to find you. 300 301### What if there are conflicts? 302 303This cannot be avoided entirely as the two repositories are independent, but 304should be rare with frequent imports and exports. When it does happen, manual 305intervention will be needed and in non-trivial cases you may be asked to help 306resolve the conflict. 307 308[wpt-exporter]: https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-exporter 309 310## Notes for WPT infra maintainers 311 312### Manual import 313 314To pull the latest versions of the tests that are currently being imported, you 315can also directly invoke the 316[wpt-import](../../third_party/blink/tools/wpt_import.py) script. 317 318That script will pull the latest version of the tests from our mirrors of the 319upstream repositories. If any new versions of tests are found, they will be 320committed locally to your local repository. You may then upload the changes. 321 322Remember your import might fail due to GitHub's limit for unauthenticated 323requests, so consider [passing your GitHub credentials](#GitHub-credentials) to 324the script. 325 326### Exporter 327 328- The source lives in 329 [third_party/blink/tools/wpt_export.py](../../third_party/blink/tools/wpt_export.py). 330- If the exporter starts misbehaving (for example, creating the same PR over 331 and over again), put it in "dry run" mode by landing [this 332 CL](https://crrev.com/c/462381/). 333 334### GitHub credentials 335 336When manually running the `wpt-import` and `wpt-export` scripts, several 337requests are made to GitHub to query the status of pull requests, look for 338existing exported commits etc. GitHub has a [fairly 339low](https://developer.github.com/v3/#rate-limiting) request limit for 340unauthenticated requests, so it is recommended that you let `wpt-export` and 341`wpt-import` use your GitHub credentials when sending requests: 342 343 1. Generate a new [personal access token](https://github.com/settings/tokens) 344 1. Set up your credentials by either: 345 * Setting the `GH_USER` environment variable to your GitHub user name 346 and the `GH_TOKEN` environment variable to the access token you have 347 just created **or** 348 * Creating a JSON file with two keys: `GH_USER`, your GitHub user name, 349 and `GH_TOKEN`, the access token you have just generated. After that, 350 pass `--credentials-json <path-to-json>` to `wpt-export` and 351 `wpt-import`. 352