1# web-platform-tests 2 3Interoperability between browsers is 4[critical](https://www.chromium.org/blink/platform-predictability) to Chromium's 5mission of improving the web. We believe that leveraging and contributing to a 6shared test suite is one of the most important tools in achieving 7interoperability between browsers. The [web-platform-tests 8repository](https://github.com/web-platform-tests/wpt) is the primary shared 9test suite where all browser engines are collaborating. 10 11Chromium has a 2-way import/export process with the upstream web-platform-tests 12repository, where tests are imported into 13[web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt) 14and any changes to the imported tests are also exported to web-platform-tests. 15 16See https://web-platform-tests.org/ for general documentation on 17web-platform-tests, including tips for writing and reviewing tests. 18 19[TOC] 20 21## Importing tests 22 23Chromium has a [mirror](https://chromium.googlesource.com/external/w3c/web-platform-tests/) 24of the GitHub repo and periodically imports a subset of the tests to 25run as part of the regular Blink web test testing process. 26 27The goals of this process are to be able to run web-platform-tests unmodified 28locally just as easily as we can run the Blink tests, and ensure that we are 29tracking tip-of-tree in the web-platform-tests repository as closely as 30possible, and running as many of the tests as possible. 31 32### Automatic import process 33 34There is an automatic process for updating the Chromium copy of 35web-platform-tests. The import is done by the builder [wpt-importer 36builder][wpt-importer]. 37 38The easiest way to check the status of recent imports is to look at: 39 40- Recent logs on LUCI for [wpt-importer builder][wpt-importer] 41- Recent CLs created by [WPT 42 Autoroller](https://chromium-review.googlesource.com/q/owner:wpt-autoroller%2540chops-service-accounts.iam.gserviceaccount.com). 43 44The import jobs will generally be green if either there was nothing to do, 45or a CL was successfully submitted. 46 47For maintainers: 48 49- The source lives in 50 [third_party/blink/tools/wpt_import.py](../../third_party/blink/tools/wpt_import.py). 51- If the importer starts misbehaving, it can be disabled by landing a 52 [CL to skip the update step](https://crrev.com/c/1961906/). 53 54### Failures caused by automatic imports. 55 56If there are new test failures that start after an auto-import, 57there are several possible causes, including: 58 59 1. New baselines for flaky tests were added (http://crbug.com/701234). 60 2. Modified tests should have new results for non-Release builds but they weren't added (http://crbug.com/725160). 61 3. New baselines were added for tests with non-deterministic test results (http://crbug.com/705125). 62 63Because these tests are imported from the Web Platform tests, it is better 64to have them in the repository (and marked failing) than not, so prefer to 65[add test expectations](web_test_expectations.md) rather than reverting. 66However, if a huge number of tests are failing, please revert the CL so we 67can fix it manually. 68 69[wpt-importer]: https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-importer 70 71### Automatic export process 72 73If you upload a CL with any changes in 74[third_party/blink/web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt), 75once you add reviewers the exporter will create a provisional pull request with 76those changes in the [upstream WPT GitHub repository](https://github.com/web-platform-tests/wpt/). 77The exporter runs on [wpt-exporter builder][wpt-exporter]. 78 79Once you're ready to land your CL, please follow the link posted by the bot and 80check the status of the required checks of the GitHub PR. If it's green, go 81ahead landing your CL and the exporter will automatically merge the PR. 82 83If GitHub status is red on the PR, please try to resolve the failures before 84merging. If you run into any issues, or if you have a CL with WPT changes that 85the exporter did not pick up, please reach out to ecosystem-infra@chromium.org. 86 87Additional things to note: 88 89- CLs that change over 1000 files will not be exported. 90- All PRs use the 91 [`chromium-export`](https://github.com/web-platform-tests/wpt/pulls?utf8=%E2%9C%93&q=is%3Apr%20label%3Achromium-export) label. 92- All PRs for CLs that haven't yet been landed in Chromium also use the 93 [`do not merge yet`](https://github.com/web-platform-tests/wpt/pulls?q=is%3Apr+is%3Aopen+label%3A%22do+not+merge+yet%22) label. 94- The exporter cannot create upstream PRs for in-flight CLs with binary files 95 (e.g. webm files). An export PR will still be made after the CL lands. 96 97For maintainers: 98 99- The source lives in 100 [third_party/blink/tools/wpt_export.py](../../third_party/blink/tools/wpt_export.py). 101- If the exporter starts misbehaving (for example, creating the same PR over 102 and over again), put it in "dry run" mode by landing [this CL](https://crrev.com/c/462381/). 103 104[wpt-exporter]: https://ci.chromium.org/p/infra/builders/luci.infra.cron/wpt-exporter 105 106### GitHub credentials 107 108When manually running the `wpt-import` and `wpt-export` scripts, several 109requests are made to GitHub to query the status of pull requests, look for 110existing exported commits etc. GitHub has a [fairly 111low](https://developer.github.com/v3/#rate-limiting) request limit for 112unauthenticated requests, so it is recommended that you let `wpt-export` and 113`wpt-import` use your GitHub credentials when sending requests: 114 115 1. Generate a new [personal access token](https://github.com/settings/tokens) 116 1. Set up your credentials by either: 117 * Setting the `GH_USER` environment variable to your GitHub user name 118 and the `GH_TOKEN` environment variable to the access token you have 119 just created **or** 120 * Creating a JSON file with two keys: `GH_USER`, your GitHub user name, 121 and `GH_TOKEN`, the access token you have just generated. After that, 122 pass `--credentials-json <path-to-json>` to `wpt-export` and 123 `wpt-import`. 124 125### Manual import 126 127To pull the latest versions of the tests that are currently being imported, you 128can also directly invoke the 129[wpt-import](../../third_party/blink/tools/wpt_import.py) script. 130 131That script will pull the latest version of the tests from our mirrors of the 132upstream repositories. If any new versions of tests are found, they will be 133committed locally to your local repository. You may then upload the changes. 134 135Remember your import might fail due to GitHub's limit for unauthenticated 136requests, so consider [passing your GitHub credentials](#GitHub-credentials) to 137the script. 138 139### Skipped tests 140 141We control which tests are imported via a file called 142[W3CImportExpectations](../../third_party/blink/web_tests/W3CImportExpectations), 143which has a list of directories to skip while importing. 144 145In addition to the directories and tests explicitly skipped there, tests may 146also be skipped for a couple other reasons, e.g. if the file path is too long 147for Windows. To check what files are skipped in import, check the recent logs 148for [wpt-importer builder][wpt-importer]. 149 150### Enabling import for a new directory 151 152If you wish to add more tests (by un-skipping some of the directories currently 153skipped in `W3CImportExpectations`), you can modify that file locally and commit 154it, and on the next auto-import, the new tests should be imported. 155 156If you want to import immediately (in order to try the tests out locally, etc) 157you can also run `wpt-import`, but this is not required. 158 159Remember your import might fail due to GitHub's limit for unauthenticated 160requests, so consider [passing your GitHub credentials](#GitHub-credentials) to 161the script. 162 163## Writing tests 164 165To contribute changes to web-platform-tests, just commit your changes directly 166to [web_tests/external/wpt](../../third_party/blink/web_tests/external/wpt) 167and the changes will be automatically upstreamed within 24 hours. 168 169Changes involving adding, removing or modifying tests can all be upstreamed. 170Any changes outside of 171[external/wpt](../../third_party/blink/web_tests/external/wpt) will not be 172upstreamed, and any changes `*-expected.txt`, `OWNERS`, and `MANIFEST.json`, 173will also not be upstreamed. 174 175Running the web tests will automatically regenerate MANIFEST.json to pick up 176any local modifications. 177 178Most tests are written using testharness.js, see 179[Writing Web Tests](./writing_web_tests.md) and 180[Web Tests Tips](./web_tests_tips.md) for general guidelines. 181 182### Write tests against specifications 183 184Tests in web-platform-tests are expected to match behavior defined by the 185relevant specification. In other words, all assertions that a test makes 186should be derived from a specification's normative requirements, and not go 187beyond them. It is often necessary to change the specification to clarify what 188is and isn't required. 189 190When implementation experience is needed to inform the specification work, 191[tentative tests](https://web-platform-tests.org/writing-tests/file-names.html) 192can be appropriate. It should be apparent in context why the test is tentative 193and what needs to be resolved to make it non-tentative. 194 195### Tests that require testing APIs 196 197[testdriver.js](https://web-platform-tests.org/writing-tests/testdriver.html) 198provides a means to automate tests that cannot be written purely using web 199platform APIs, similar to `internals.*` and `eventSender.*` in regular Blink 200web tests. 201 202If no testdriver.js API exists, check if it's a 203[known issue](https://github.com/web-platform-tests/wpt/labels/testdriver.js) 204and otherwise consider filing a new issue. For instructions on how to add a new 205testing API, see [WPT Test Automation for 206Chromium](https://docs.google.com/document/d/18BpD41vyX1cFZ77CE0a_DJYlGpdvyLlx3pwXVRxUzvI/preview#) 207 208An alternative is to write manual tests that are automated with scripts from 209[wpt_automation](../../third_party/blink/web_tests/external/wpt_automation). 210Injection of JS in manual tests is determined by `loadAutomationScript` in 211[testharnessreport.js](../../third_party/blink/web_tests/resources/testharnessreport.js). 212 213Such tests still require case-by-case automation to run for other browser 214engines, but are more valuable than purely manual tests. 215 216Manual tests that have no automation are still imported, but skipped in 217[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests); see 218[issue 738489](https://crbug.com/738489). 219 220### Adding new top-level directories 221 222Entirely new top-level directories should generally be added upstream, since 223that's the only way to add an OWNERS file upstream. After adding a new top-level 224directory upstream, you should add a line for it in `W3CImportExpectations`. 225 226Adding the new directory (and `W3CImportExpectations` entry) in Chromium and 227later adding an OWNERS file upstream also works. 228 229### Will the exported commits be linked to my GitHub profile? 230 231The email you commit with in Chromium will be the author of the commit on 232GitHub. You can [add it as a secondary address on your GitHub 233account](https://help.github.com/articles/adding-an-email-address-to-your-github-account/) 234to link your exported commits to your GitHub profile. 235 236If you are a Googler, you can also register your GitHub account at go/github, 237making it easier for other Googlers to find you. 238 239### What if there are conflicts? 240 241This cannot be avoided entirely as the two repositories are independent, but 242should be rare with frequent imports and exports. When it does happen, manual 243intervention will be needed and in non-trivial cases you may be asked to help 244resolve the conflict. 245 246### Direct pull requests 247 248It's still possible to make direct pull requests to web-platform-tests, see 249https://web-platform-tests.org/appendix/github-intro.html. 250 251## Running tests 252 253Same as Blink web tests, you can use 254[`run_web_tests.py`](web_tests.md#running-the-tests) to run any WPT test. 255 256One thing to note is that glob patterns for WPT tests are not yet supported. 257 258See [Running WPT tests in Content Shell](web_tests_in_content_shell.md#Running-WPT-Tests-in-Content-Shell) 259for debugging etc. 260 261## Reviewing tests 262 263Anyone who can review code and tests in Chromium can also review changes in 264[external/wpt](../../third_party/blink/web_tests/external/wpt) 265that will be automatically upstreamed. There will be no additional review in 266web-platform-tests as part of the export process. 267 268If upstream reviewers have feedback on the changes, discuss on the pull request 269created during export, and if necessary work on a new pull request to iterate 270until everyone is satisfied. 271 272When reviewing tests, check that they match the relevant specification, which 273may not fully match the implementation. See also 274[Write tests against specifications](#Write-tests-against-specifications). 275