1# Reftests 2 3Reftests are one of the primary tools for testing things relating to 4rendering; they are made up of the test and one or more other pages 5("references") with assertions as to whether they render identically 6or not. This page describes their aspects exhaustively; [the tutorial 7on writing a reftest](reftest-tutorial) offers a more limited but 8grounded guide to the process. 9 10## How to Run Reftests 11 12Reftests can be run manually simply by opening the test and the 13reference file in multiple windows or tabs and flipping between the 14two. In automation the comparison is done in an automated fashion, 15which can lead to differences hard for the human eye to notice to 16cause the test to fail. 17 18## Components of a Reftest 19 20In the simplest case, a reftest consists of a pair of files called the 21*test* and the *reference*. 22 23The *test* file is the one that makes use of the technology being 24tested. It also contains a `link` element with `rel="match"` or 25`rel="mismatch"` and `href` attribute pointing to the *reference* 26file, e.g. `<link rel=match href=references/green-box-ref.html>`. A 27`match` test only passes if the two files render pixel-for-pixel 28identically within a 800x600 window *including* scroll-bars if 29present; a `mismatch` test only passes if they *don't* render 30identically. 31 32The *reference* file is typically written to be as simple as possible, 33and does not use the technology under test. It is desirable that the 34reference be rendered correctly even in UAs with relatively poor 35support for CSS and no support for the technology under test. 36 37## Writing a Good Reftest 38 39In general the files used in a reftest should follow 40the [general guidelines][] and 41the [rendering test guidelines][rendering]. They should also be 42self-describing, to allow a human to determine whether the the 43rendering is as expected. 44 45References can be shared between tests; this is strongly encouraged as 46it makes it easier to tell at a glance whether a test passes (through 47familiarity) and enables some optimizations in automated test 48runners. Shared references are typically placed in `references` 49directories, either alongside the tests they are expected to be useful 50for or at the top level if expected to be generally applicable (e.g., 51many layout tests can be written such that the correct rendering is a 52100x100 green square!). For references that are applicable only to a 53single test, it is recommended to use the test name with a suffix of 54`-ref` as their filename; e.g., `test.html` would have `test-ref.html` 55as a reference. 56 57## Multiple References 58 59Sometimes, a test's pass condition cannot be captured in a single 60reference. 61 62If a test has multiple links, then the test passes if: 63 64 * If there are any match references, at least one must match, and 65 * If there are any mismatch references, all must mismatch. 66 67 If you need multiple matches to succeed, these can be turned into 68 multiple tests (for example, by just having a reference be a test 69 itself!). If this seems like an unreasonable restriction, please file 70 a bug and let us know! 71 72## Controlling When Comparison Occurs 73 74By default, reftest screenshots are taken after the following 75conditions are met: 76 77* The `load` event has fired 78* Web fonts (if any) are loaded 79* Pending paints have completed 80 81In some cases it is necessary to delay the screenshot later than this, 82for example because some DOM manipulation is required to set up the 83desired test conditions. To enable this, the test may have a 84`class="reftest-wait"` attribute specified on the root element. In 85this case the harness will run the following sequence of steps: 86 87* Wait for the `load` event to fire and fonts to load. 88* Wait for pending paints to complete. 89* Fire an event named `TestRendered` at the root element, with the 90 `bubbles` attribute set to true. 91* Wait for the `reftest-wait` class to be removed from the root 92 element. 93* Wait for pending paints to complete. 94* Screenshot the viewport. 95 96The `TestRendered` event provides a hook for tests to make 97modifications to the test document that are not batched into the 98initial layout/paint. 99 100## Fuzzy Matching 101 102In some situations a test may have subtle differences in rendering 103compared to the reference due to, e.g., anti-aliasing. To allow for 104these small differences, we allow tests to specify a fuzziness 105characterised by two parameters, both of which must be specified: 106 107 * A maximum difference in the per-channel color value for any pixel. 108 * A number of total pixels that may be different. 109 110The maximum difference in the per pixel color value is formally 111defined as follows: let <code>T<sub>x,y,c</sub></code> be the value of 112colour channel `c` at pixel coordinates `x`, `y` in the test image and 113<code>R<sub>x,y,c</sub></code> be the corresponding value in the 114reference image, and let <code>width</code> and <code>height</code> be 115the dimensions of the image in pixels. Then <code>maxDifference = 116max<sub>x=[0,width) y=[0,height), c={r,g,b}</sub>(|T<sub>x,y,c</sub> - 117R<sub>x,y,c</sub>|)</code>. 118 119To specify the fuzziness in the test file one may add a `<meta 120name=fuzzy>` element (or, in the case of more complex tests, to any 121page containing the `<link rel=[mis]match>` elements). In the simplest 122case this has a `content` attribute containing the parameters above, 123separated by a semicolon e.g. 124 125``` 126<meta name=fuzzy content="maxDifference=15;totalPixels=300"> 127``` 128 129would allow for a difference of exactly 15 / 255 on any color channel 130and 300 exactly pixels total difference. The argument names are optional 131and may be elided; the above is the same as: 132 133``` 134<meta name=fuzzy content="15;300"> 135``` 136 137The values may also be given as ranges e.g. 138 139``` 140<meta name=fuzzy content="maxDifference=10-15;totalPixels=200-300"> 141``` 142 143or 144 145``` 146<meta name=fuzzy content="10-15;200-300"> 147``` 148 149In this case the maximum pixel difference must be in the range 150`10-15` and the total number of different pixels must be in the range 151`200-300`. These range checks are inclusive. 152 153In cases where a single test has multiple possible refs and the 154fuzziness is not the same for all refs, a ref may be specified by 155prefixing the `content` value with the relative url for the ref e.g. 156 157``` 158<meta name=fuzzy content="option1-ref.html:10-15;200-300"> 159``` 160 161One meta element is required per reference requiring a unique 162fuzziness value, but any unprefixed value will automatically be 163applied to any ref that doesn't have a more specific value. 164 165### Debugging fuzzy reftests 166 167When debugging a fuzzy reftest via `wpt run`, it can be useful to know what the 168allowed and detected differences were. Many of the output logger options will 169provide this information. For example, by passing `--log-mach=-` for a run of a 170hypothetical failing test, one might get: 171 172``` 173 0:08.15 TEST_START: /foo/bar.html 174 0:09.70 INFO Found 250 pixels different, maximum difference per channel 6 on page 1 175 0:09.70 INFO Allowed 0-100 pixels different, maximum difference per channel 0-0 176 0:09.70 TEST_END: FAIL, expected PASS - /foo/bar.html ['f83385ed9c9bea168108b8c448366678c7941627'] 177``` 178 179For other logging flags, see the output of `wpt run --help`. 180 181## Limitations 182 183In some cases, a test cannot be a reftest. For example, there is no 184way to create a reference for underlining, since the position and 185thickness of the underline depends on the UA, the font, and/or the 186platform. However, once it's established that underlining an inline 187element works, it's possible to construct a reftest for underlining 188a block element, by constructing a reference using underlines on a 189```<span>``` that wraps all the content inside the block. 190 191[general guidelines]: general-guidelines 192[rendering]: rendering 193