1# testharness.js tutorial 2 3<!-- 4Note to maintainers: 5 6This tutorial is designed to be an authentic depiction of the WPT contribution 7experience. It is not intended to be comprehensive; its scope is intentionally 8limited in order to demonstrate authoring a complete test without overwhelming 9the reader with features. Because typical WPT usage patterns change over time, 10this should be updated periodically; please weigh extensions against the 11demotivating effect that a lengthy guide can have on new contributors. 12--> 13 14Let's say you've discovered that WPT doesn't have any tests for how [the Fetch 15API](https://fetch.spec.whatwg.org/) sets cookies from an HTTP response. This 16tutorial will guide you through the process of writing a test for the 17web-platform, verifying it, and submitting it back to WPT. Although it includes 18some very brief instructions on using git, you can find more guidance in [the 19tutorial for git and GitHub](github-intro). 20 21WPT's testharness.js is a framework designed to help people write tests for the 22web platform's JavaScript APIs. [The testharness.js reference 23page](testharness) describes the framework in the abstract, but for the 24purposes of this guide, we'll only consider the features we need to test the 25behavior of `fetch`. 26 27```eval_rst 28.. contents:: Table of Contents 29 :depth: 3 30 :local: 31 :backlinks: none 32``` 33 34## Setting up your workspace 35 36To make sure you have the latest code, first type the following into a terminal 37located in the root of the WPT git repository: 38 39 $ git fetch git@github.com:web-platform-tests/wpt.git 40 41Next, we need a place to store the change set we're about to author. Here's how 42to create a new git branch named `fetch-cookie` from the revision of WPT we 43just downloaded: 44 45 $ git checkout -b fetch-cookie FETCH_HEAD 46 47The tests we're going to write will rely on special abilities of the WPT 48server, so you'll also need to [configure your system to run 49WPT](../running-tests/from-local-system) before you continue. 50 51With that out of the way, you're ready to create your patch. 52 53## Writing a subtest 54 55<!-- 56Goals of this section: 57 58- demonstrate asynchronous testing with Promises 59- motivate non-trivial integration with WPT server 60- use web technology likely to be familiar to web developers 61- use web technology likely to be supported in the reader's browser 62--> 63 64The first thing we'll do is configure the server to respond to a certain request 65by setting a cookie. Once that's done, we'll be able to make the request with 66`fetch` and verify that it interpreted the response correctly. 67 68We'll configure the server with an "asis" file. That's the WPT convention for 69controlling the contents of an HTTP response. [You can read more about it 70here](server-features), but for now, we'll save the following text into a file 71named `set-cookie.asis` in the `fetch/api/basic/` directory of WPT: 72 73``` 74HTTP/1.1 204 No Content 75Set-Cookie: test1=t1 76``` 77 78With this in place, any requests to `/fetch/api/basic/set-cookie.asis` will 79receive an HTTP 204 response that sets the cookie named `test1`. When writing 80more tests in the future, you may want the server to behave more dynamically. 81In that case, [you can write Python code to control how the server 82responds](python-handlers/index). 83 84Now, we can write the test! Create a new file named `set-cookie.html` in the 85same directory and insert the following text: 86 87```html 88<!DOCTYPE html> 89<meta charset="utf-8"> 90<title>fetch: setting cookies</title> 91<script src="/resources/testharness.js"></script> 92<script src="/resources/testharnessreport.js"></script> 93 94<script> 95promise_test(function() { 96 return fetch('set-cookie.asis') 97 .then(function() { 98 assert_equals(document.cookie, 'test1=t1'); 99 }); 100}); 101</script> 102``` 103 104Let's step through each part of this file. 105 106- ```html 107 <!DOCTYPE html> 108 <meta charset="utf-8"> 109 ``` 110 111 We explicitly set the DOCTYPE and character set to be sure that browsers 112 don't infer them to be something we aren't expecting. We're omitting the 113 `<html>` and `<head>` tags. That's a common practice in WPT, preferred 114 because it makes tests more concise. 115 116- ```html 117 <title>fetch: setting cookies</title> 118 ``` 119 The document's title should succinctly describe the feature under test. 120 121- ```html 122 <script src="/resources/testharness.js"></script> 123 <script src="/resources/testharnessreport.js"></script> 124 ``` 125 126 These two `<script>` tags retrieve the code that powers testharness.js. A 127 testharness.js test can't run without them! 128 129- ```html 130 <script> 131 promise_test(function() { 132 return fetch('thing.asis') 133 .then(function() { 134 assert_equals(document.cookie, 'test1=t1'); 135 }); 136 }); 137 </script> 138 ``` 139 140 This script uses the testharness.js function `promise_test` to define a 141 "subtest". We're using that because the behavior we're testing is 142 asynchronous. By returning a Promise value, we tell the harness to wait until 143 that Promise settles. The harness will report that the test has passed if 144 the Promise is fulfilled, and it will report that the test has failed if the 145 Promise is rejected. 146 147 We invoke the global `fetch` function to exercise the "behavior under test," 148 and in the fulfillment handler, we verify that the expected cookie is set. 149 We're using the testharness.js `assert_equals` function to verify that the 150 value is correct; the function will throw an error otherwise. That will cause 151 the Promise to be rejected, and *that* will cause the harness to report a 152 failure. 153 154If you run the server according to the instructions in [the guide for local 155configuration](../running-tests/from-local-system), you can access the test at 156[http://web-platform.test:8000/fetch/api/basic/set-cookie.html](http://web-platform.test:8000/fetch/api/basic/set-cookie.html.). 157You should see something like this: 158 159![](../assets/testharness-tutorial-test-screenshot-1.png "screen shot of testharness.js reporting the test results") 160 161## Refining the subtest 162 163<!-- 164Goals of this section: 165 166- explain the motivation for "clean up" logic and demonstrate its usage 167- motivate explicit test naming 168--> 169 170We'd like to test a little more about `fetch` and cookies, but before we do, 171there are some improvements we can make to what we've written so far. 172 173For instance, we should remove the cookie after the subtest is complete. This 174ensures a consistent state for any additional subtests we may add and also for 175any tests that follow. We'll use the `add_cleanup` method to ensure that the 176cookie is deleted even if the test fails. 177 178```diff 179-promise_test(function() { 180+promise_test(function(t) { 181+ t.add_cleanup(function() { 182+ document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;'; 183+ }); 184+ 185 return fetch('thing.asis') 186 .then(function() { 187 assert_equals(document.cookie, 'test1=t1'); 188 }); 189 }); 190``` 191 192Although we'd prefer it if there were no other cookies defined during our test, 193we shouldn't take that for granted. As written, the test will fail if the 194`document.cookie` includes additional cookies. We'll use slightly more 195complicated logic to test for the presence of the expected cookie. 196 197 198```diff 199 promise_test(function(t) { 200 t.add_cleanup(function() { 201 document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;'; 202 }); 203 204 return fetch('thing.asis') 205 .then(function() { 206- assert_equals(document.cookie, 'test1=t1'); 207+ assert_true(/(^|; )test1=t1($|;)/.test(document.cookie); 208 }); 209 }); 210``` 211 212In the screen shot above, the subtest's result was reported using the 213document's title, "fetch: setting cookies". Since we expect to add another 214subtest, we should give this one a more specific name: 215 216```diff 217 promise_test(function(t) { 218 t.add_cleanup(function() { 219 document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;'; 220 }); 221 222 return fetch('thing.asis') 223 .then(function() { 224 assert_true(/(^|; )test1=t1($|;)/.test(document.cookie)); 225 }); 226-}); 227+}, 'cookie set for successful request'); 228``` 229 230## Writing a second subtest 231 232<!-- 233Goals of this section: 234 235- introduce the concept of cross-domain testing and the associated tooling 236- demonstrate how to verify promise rejection 237- demonstrate additional assertion functions 238--> 239 240There are many things we might want to verify about how `fetch` sets cookies. 241For instance, it should *not* set a cookie if the request fails due to 242cross-origin security restrictions. Let's write a subtest which verifies that. 243 244We'll add another `<script>` tag for a JavaScript support file: 245 246```diff 247 <!DOCTYPE html> 248 <meta charset="utf-8"> 249 <title>fetch: setting cookies</title> 250 <script src="/resources/testharness.js"></script> 251 <script src="/resources/testharnessreport.js"></script> 252+<script src="/common/get-host-info.sub.js"></script> 253``` 254 255`get-host-info.sub.js` is a general-purpose script provided by WPT. It's 256designed to help with testing cross-domain functionality. Since it's stored in 257WPT's `common/` directory, tests from all sorts of specifications rely on it. 258 259Next, we'll define the new subtest inside the same `<script>` tag that holds 260our first subtest. 261 262```js 263promise_test(function(t) { 264 t.add_cleanup(function() { 265 document.cookie = 'test1=;expires=Thu, 01 Jan 1970 00:00:01 GMT;'; 266 }); 267 const url = get_host_info().HTTP_NOTSAMESITE_ORIGIN + 268 '/fetch/api/basic/set-cookie.asis'; 269 270 return fetch(url) 271 .then(function() { 272 assert_unreached('The promise for the aborted fetch operation should reject.'); 273 }, function() { 274 assert_false(/(^|; )test1=t1($|;)/.test(document.cookie)); 275 }); 276}, 'no cookie is set for cross-domain fetch operations'); 277``` 278 279This may look familiar from the previous subtest, but there are some important 280differences. 281 282- ```js 283 const url = get_host_info().HTTP_NOTSAMESITE_ORIGIN + 284 '/fetch/api/basic/set-cookie.asis'; 285 ``` 286 287 We're requesting the same resource, but we're referring to it with an 288 alternate host name. The name of the host depends on how the WPT server has 289 been configured, so we rely on the helper to provide an appropriate value. 290 291- ```js 292 return fetch(url) 293 .then(function() { 294 assert_unreached('The promise for the aborted fetch operation should reject.'); 295 }, function() { 296 assert_false(/(^|; )test1=t1($|;)/.test(document.cookie)); 297 }); 298 ``` 299 300 We're returning a Promise value, just like the first subtest. This time, we 301 expect the operation to fail, so the Promise should be rejected. To express 302 this, we've used `assert_unreached` *in the fulfillment handler*. 303 `assert_unreached` is a testharness.js utility function which always throws 304 an error. With this in place, if fetch does *not* produce an error, then this 305 subtest will fail. 306 307 We've moved the assertion about the cookie to the rejection handler. We also 308 switched from `assert_true` to `assert_false` because the test should only 309 pass if the cookie is *not* set. It's a good thing we have the cleanup logic 310 in the previous subtest, right? 311 312If you run the test in your browser now, you can expect to see both tests 313reported as passing with their distinct names. 314 315![](../assets/testharness-tutorial-test-screenshot-2.png "screen shot of testharness.js reporting the test results") 316 317## Verifying our work 318 319We're done writing the test, but we should make sure it fits in with the rest 320of WPT before we submit it. 321 322[The lint tool](lint-tool) can detect some of the common mistakes people make 323when contributing to WPT. You enabled it when you [configured your system to 324work with WPT](../running-tests/from-local-system). To run it, open a 325command-line terminal, navigate to the root of the WPT repository, and enter 326the following command: 327 328 python ./wpt lint fetch/api/basic 329 330If this recognizes any of those common mistakes in the new files, it will tell 331you where they are and how to fix them. If you do have changes to make, you can 332run the command again to make sure you got them right. 333 334Now, we'll run the test using the automated test runner. This is important for 335testharness.js tests because there are subtleties of the automated test runner 336which can influence how the test behaves. That's not to say your test has to 337pass in all browsers (or even in *any* browser). But if we expect the test to 338pass, then running it this way will help us catch other kinds of mistakes. 339 340The tools support running the tests in many different browsers. We'll use 341Firefox this time: 342 343 python ./wpt run firefox fetch/api/basic/set-cookie.html 344 345We expect this test to pass, so if it does, we're ready to submit it. If we 346were testing a web-platform feature that Firefox didn't support, we would 347expect the test to fail instead. 348 349There are a few problems to look out for in addition to passing/failing status. 350The report will describe fewer tests than we expect if the test isn't run at 351all. That's usually a sign of a formatting mistake, so you'll want to make sure 352you've used the right file names and metadata. Separately, the web browser 353might crash. That's often a sign of a browser bug, so you should consider 354[reporting it to the browser's 355maintainers](https://rachelandrew.co.uk/archives/2017/01/30/reporting-browser-bugs/)! 356 357## Submitting the test 358 359First, let's stage the new files for committing: 360 361 $ git add fetch/api/basic/set-cookie.asis 362 $ git add fetch/api/basic/set-cookie.html 363 364We can make sure the commit has everything we want to submit (and nothing we 365don't) by using `git diff`: 366 367 $ git diff --staged 368 369On most systems, you can use the arrow keys to navigate through the changes, 370and you can press the `q` key when you're done reviewing. 371 372Next, we'll create a commit with the staged changes: 373 374 $ git commit -m '[fetch] Add test for setting cookies' 375 376And now we can push the commit to our fork of WPT: 377 378 $ git push origin fetch-cookie 379 380The last step is to submit the test for review. WPT doesn't actually need the 381test we wrote in this tutorial, but if we wanted to submit it for inclusion in 382the repository, we would create a pull request on GitHub. [The guide on git and 383GitHub](github-intro) has all the details on how to do that. 384 385## More practice 386 387Here are some ways you can keep experimenting with WPT using this test: 388 389- Improve the test's readability by defining helper functions like 390 `cookieIsSet` and `deleteCookie` 391- Improve the test's coverage by refactoring it into [a "multi-global" 392 test](testharness) 393- Improve the test's coverage by writing more subtests (e.g. the behavior when 394 the fetch operation is aborted by `window.stop`, or the behavior when the 395 HTTP response sets multiple cookies) 396