1# Writing a reftest 2 3<!-- 4Note to maintainers: 5 6This tutorial is designed to be an authentic depiction of the WPT contribution 7experience. It is not intended to be comprehensive; its scope is intentionally 8limited in order to demonstrate authoring a complete test without overwhelming 9the reader with features. Because typical WPT usage patterns change over time, 10this should be updated periodically; please weigh extensions against the 11demotivating effect that a lengthy guide can have on new contributors. 12--> 13 14Let's say you've discovered that WPT doesn't have any tests for the `dir` 15attribute of [the `<bdo>` 16element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/bdo). This 17tutorial will guide you through the process of writing and submitting a test. 18You'll need to [configure your system to use WPT's 19tools](../running-tests/from-local-system), but you won't need them until 20towards the end of this tutorial. Although it includes some very brief 21instructions on using git, you can find more guidance in [the tutorial for git 22and GitHub](../writing-tests/github-intro). 23 24WPT's reftests are great for testing web-platform features that have some 25visual effect. [The reftests reference page](reftests) describes them in the 26abstract, but for the purposes of this guide, we'll only consider the features 27we need to test the `<bdo>` element. 28 29```eval_rst 30.. contents:: 31 :local: 32``` 33 34## Setting up your workspace 35 36To make sure you have the latest code, first type the following into a terminal 37located in the root of the WPT git repository: 38 39 $ git fetch git@github.com:web-platform-tests/wpt.git 40 41Next, we need a place to store the change set we're about to author. Here's how 42to create a new git branch named `reftest-for-bdo` from the revision of WPT we 43just downloaded: 44 45 $ git checkout -b reftest-for-bdo FETCH_HEAD 46 47Now you're ready to create your patch. 48 49## Writing the test file 50 51First, we'll create a file that demonstrates the "feature under test." That is: 52we'll write an HTML document that displays some text using a `<bdo>` element. 53 54WPT has thousands of tests, so it can be daunting to decide where to put a new 55one. Generally speaking, [test files should be placed in directories 56corresponding to the specification text they are 57verifying](../test-suite-design). `<bdo>` is defined in [the "text-level 58semantics" chapter of the HTML 59specification](https://html.spec.whatwg.org/multipage/text-level-semantics.html), 60so we'll want to create our new test in the directory 61`html/semantics/text-level-semantics/the-bdo-element/`. Create a file named 62`rtl.html` and open it in your text editor. 63 64Here's one way to demonstrate the feature: 65 66```html 67<!DOCTYPE html> 68<meta charset="utf-8"> 69<title>BDO element dir=rtl</title> 70<link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element"> 71<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'."> 72 73<p>Test passes if WAS is displayed below.</p> 74<bdo dir="rtl">SAW</bdo> 75``` 76 77That's pretty dense! Let's break it down: 78 79- ```html 80 <!DOCTYPE html> 81 <meta charset="utf-8"> 82 ``` 83 84 We explicitly set the DOCTYPE and character set to be sure that browsers 85 don't infer them to be something we aren't expecting. We're omitting the 86 `<html>` and `<head>` tags. That's a common practice in WPT, preferred 87 because it makes tests more concise. 88 89- ```html 90 <title>BDO element dir=rtl</title> 91 ``` 92 The document's title should succinctly describe the feature under test. 93 94- ```html 95 <link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element"> 96 ``` 97 98 The "help" metadata should reference the specification under test so that 99 everyone understands the motivation. This is so helpful that [the CSS Working 100 Group requires it for CSS tests](css-metadata)! If you're writing a reftest 101 for a feature outside of CSS, feel free to omit this tag. 102 103- ```html 104 <meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'."> 105 ``` 106 107 The "assert" metadata is a structured way for you to describe exactly what 108 you want your reftest to verify. For a direct test like the one we're writing 109 here, it might seem a little superfluous. It's much more helpful for 110 more-involved tests where reviewers might need some help understanding your 111 intentions. 112 113 This tag is optional, so you can skip it if you think it's unnecessary. We 114 recommend using it for your first few tests since it may let reviewers give 115 you more helpful feedback. As you get more familiar with WPT and the 116 specifications, you'll get a sense for when and where it's better to leave it 117 out. 118 119- ```html 120 <p>Test passes if WAS is displayed below.</p> 121 ``` 122 123 We're communicating the "pass" condition in plain English to make the test 124 self-describing. 125 126- ```html 127 <bdo dir="rtl">SAW</bdo> 128 ``` 129 130 This is the real focus of the test. We're including some text inside a 131 `<bdo>` element in order to demonstrate the feature under test. 132 133Since this page doesn't rely on any [special WPT server 134features](server-features), we can view it by loading the HTML file directly. 135There are a bunch of ways to do this; one is to navigate to the 136`html/semantics/text-level-semantics/the-bdo-element/` directory in a file 137browser and drag the new `rtl.html` file into an open web browser window. 138 139![](/assets/reftest-tutorial-test-screenshot.png "screen shot of the new test") 140 141Sighted people can open that document and verify whether or not the stated 142expectation is satisfied. If we were writing a [manual test](manual), we'd be 143done. However, it's time-consuming for a human to run tests, so we should 144prefer making tests automatic whenever possible. Remember that we set out to 145write a "reference test." Now it's time to write the reference file. 146 147## Writing a "match" reference 148 149The "match" reference file describes what the test file is supposed to look 150like. Critically, it *must not* use the technology that we are testing. The 151reference file is what allows the test to be run by a computer--the computer 152can verify that each pixel in the test document exactly matches the 153corresponding pixel in the reference document. 154 155Make a new file in the same 156`html/semantics/text-level-semantics/the-bdo-element/` directory named 157`rtl-ref.html`, and save the following markup into it: 158 159```html 160<!DOCTYPE html> 161<meta charset="utf-8"> 162<title>BDO element dir=rtl reference</title> 163 164<p>Test passes if WAS is displayed below.</p> 165<p>WAS</p> 166``` 167 168This is like a stripped-down version of the test file. In order to produce a 169visual rendering which is the same as the expected rendering, it uses a `<p>` 170element whose contents is the characters in right-to-left order. That way, if 171the browser doesn't support the `<bdo>` element, this file will still show text 172in the correct sequence. 173 174This file is also completely functional without the WPT server, so you can open 175it in a browser directly from your hard drive. 176 177Currently, there's no way for a human operator or an automated script to know 178that the two files we've created are supposed to match visually. We'll need to 179add one more piece of metadata to the test file we created earlier. Open 180`html/semantics/text-level-semantics/the-bdo-element/rtl.html` in your text 181editor and add another `<link>` tag as described by the following change 182summary: 183 184```diff 185 <!DOCTYPE html> 186 <meta charset="utf-8"> 187 <title>BDO element dir=rtl</title> 188 <link rel="author" title="Sam Smith" href="mailto:sam@example.com"> 189 <link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element"> 190+<link rel="match" href="rtl-ref.html"> 191 <meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'."> 192 193 <p>Test passes if WAS is displayed below.</p> 194 <bdo dir="rtl">SAW</bdo> 195``` 196 197Now, anyone (human or computer) reviewing the test file will know where to find 198the associated reference file. 199 200## Verifying our work 201 202We're done writing the test, but we should make sure it fits in with the rest 203of WPT before we submit it. This involves using some of the project's tools, so 204this is the point you'll need to [configure your system to run 205WPT](../running-tests/from-local-system). 206 207[The lint tool](lint-tool) can detect some of the common mistakes people make 208when contributing to WPT. To run it, open a command-line terminal, navigate to 209the root of the WPT repository, and enter the following command: 210 211 python ./wpt lint html/semantics/text-level-semantics/the-bdo-element 212 213If this recognizes any of those common mistakes in the new files, it will tell 214you where they are and how to fix them. If you do have changes to make, you can 215run the command again to make sure you got them right. 216 217Now, we'll run the test using the automated pixel-by-pixel comparison approach 218mentioned earlier. This is important for reftests because the test and the 219reference may differ in very subtle ways that are hard to catch with the naked 220eye. That's not to say your test has to pass in all browsers (or even in *any* 221browser). But if we expect the test to pass, then running it this way will help 222us catch other kinds of mistakes. 223 224The tools support running the tests in many different browsers. We'll use 225Firefox this time: 226 227 python ./wpt run firefox html/semantics/text-level-semantics/the-bdo-element/rtl.html 228 229We expect this test to pass, so if it does, we're ready to submit it. If we 230were testing a web platform feature that Firefox didn't support, we would 231expect the test to fail instead. 232 233There are a few problems to look out for in addition to passing/failing status. 234The report will describe fewer tests than we expect if the test isn't run at 235all. That's usually a sign of a formatting mistake, so you'll want to make sure 236you've used the right file names and metadata. Separately, the web browser 237might crash. That's often a sign of a browser bug, so you should consider 238[reporting it to the browser's 239maintainers](https://rachelandrew.co.uk/archives/2017/01/30/reporting-browser-bugs/)! 240 241## Submitting the test 242 243First, let's stage the new files for committing: 244 245 $ git add html/semantics/text-level-semantics/the-bdo-element/rtl.html 246 $ git add html/semantics/text-level-semantics/the-bdo-element/rtl-ref.html 247 248We can make sure the commit has everything we want to submit (and nothing we 249don't) by using `git diff`: 250 251 $ git diff --staged 252 253On most systems, you can use the arrow keys to navigate through the changes, 254and you can press the `q` key when you're done reviewing. 255 256Next, we'll create a commit with the staged changes: 257 258 $ git commit -m '[html] Add test for the `<bdo>` element' 259 260And now we can push the commit to our fork of WPT: 261 262 $ git push origin reftest-for-bdo 263 264The last step is to submit the test for review. WPT doesn't actually need the 265test we wrote in this tutorial, but if we wanted to submit it for inclusion in 266the repository, we would create a pull request on GitHub. [The guide on git and 267GitHub](../writing-tests/github-intro) has all the details on how to do that. 268 269## More practice 270 271Here are some ways you can keep experimenting with WPT using this test: 272 273- Improve coverage by adding more tests for related behaviors (e.g. nested 274 `<bdo>` elements) 275- Add another reference document which describes what the test should *not* 276 look like using [`rel=mismatch`](reftests) 277