1A reftest is a test that compares the visual output of one file (the
2test case) with the output of one or more other files (the
3references). The test and the reference must be carefully written so
4that when the test passes they have identical rendering, but different
5rendering when the test fails.
6
7## How to Run Reftests
8
9Reftests can be run manually simply by opening the test and the
10reference file in multiple windows or tabs and either placing them
11side-by side or flipping between the two. In automation the comparison
12is done in an automated fashion, which can lead to differences too
13small for the human eye to notice causing tests to fail.
14
15## Components of a Reftest
16
17In the simplest case, a reftest consists of a pair of files called the
18*test* and the *reference*.
19
20The *test* file is the one that makes use of the technology being
21tested. It also contains a `link` element with `rel="match"` or
22`rel="mismatch"` and `href` attribute pointing to the *reference* file
23e.g. `<link rel=match href=references/green-box-ref.html>`.
24
25The *reference* file is typically written to be as simple as possible,
26and does not use the technology under test. It is desirable that the
27reference be rendered correctly even in UAs with relatively poor
28support for CSS and no support for the technology under test.
29
30When the `<link>` element in the *test* has `rel="match"`, the test
31only passes if the *test* and *reference* have pixel-perfect identical
32rendering. `rel="mismatch"` inverts this so the test only passes when
33the renderings differ.
34
35In general the files used in a reftest should follow the
36[format][format] and [style][style] guidelines. The *test* should also
37be [self-describing][selfdesc], to allow a human to determine whether
38the the rendering is as expected.
39
40Note that references can be shared between tests; this is strongly
41encouraged since it permits optimizations when running tests.
42
43## Controlling When Comparison Occurs
44
45By default reftest screenshots are taken in response to the `load`
46event firing. In some cases it is necessary to delay the screenshot
47later than this, for example becase some DOM manipulation is
48required to set up the desired test conditions. To enable this, the
49test may have a `class="reftest-wait"` attribute specified on the root
50element. This will cause the screenshot to be delayed until the `load`
51event has fired and the `reftest-wait` class has been removed from the
52root element (technical note: the implementation in wptrunner uses
53mutation observers so the screenshot will be triggered in the
54microtask checkpoint after the class is removed. Because the harness
55isn't synchronized with the browser event loop it is dangerous to rely
56on precise timing here).
57
58## Matching Multiple References
59
60Sometimes it is desirable for a file to match multiple references or,
61in rare cases, to allow it to match more than one possible
62reference. Note: *this is not currently supported by test runners and
63so best avoided if possible until that support improves*.
64
65Multiple references linked from a single file are interpreted as
66multiple possible renderings for that file. `<link rel=[mis]match>`
67elements in a reference create further conditions that must be met in
68order for the test to pass. For example, consider a situation where
69`a.html` has `<link rel=match href=b.html>` and `<link rel=match
70href=c.html>`, `b.html` has `<link rel=match href=b1.html>` and `c.html`
71has `<link rel=mismatch href=c1.html>`. In this case, to pass we must
72either have `a.html`, `b.html` and `b1.html` all rendering identically, or
73`a.html` and `c.html` rendering identically, but `c.html` rendering
74differently from `c1.html`.
75
76## Fuzzy Matching
77
78In some situations a test may have subtle differences in rendering
79compared to the reference due to e.g. antialiasing. This may cause the
80test to pass on some platforms but fail on others. In this case some
81affordance for subtle discrepancies is desirable. However no mechanism
82to allow this has yet been standardized.
83
84## Limitations
85
86In some cases, a test cannot be a reftest. For example, there is no
87way to create a reference for underlining, since the position and
88thickness of the underline depends on the UA, the font, and/or the
89platform. However, once it's established that underlining an inline
90element works, it's possible to construct a reftest for underlining
91a block element, by constructing a reference using underlines on a
92```<span>``` that wraps all the content inside the block.
93
94## Example Reftests
95
96These examples are all [self-describing][selfdesc] tests as they
97each have a simple statement on the page describing how it should
98render to pass the tests.
99
100### HTML example
101
102### Test File
103
104This test verifies that a right-to-left rendering of **SAW** within a
105```<bdo>``` element displays as **WAS**.
106
107([view page rendering][html-reftest-example])
108
109```html
110<!DOCTYPE html>
111<meta charset="utf-8">
112<title>BDO element dir=rtl</title>
113<link rel="help" href="http://www.whatwg.org/specs/web-apps/current-work/#the-bdo-element">
114<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
115<link rel="match" href="test-bdo-001.html">
116<p>Pass if you see WAS displayed below.</p>
117<bdo dir="rtl">SAW</bdo>
118```
119
120### Reference File
121
122The reference file must look exactly like the test file,
123except that the code behind it is different.
124
125* All metadata is removed.
126* The ```title``` need not match.
127* The markup that created the actual test data is
128  different: here, the same effect is created with
129  very mundane, dependable technology.
130
131([view page rendering][html-reffile-example])
132
133```html
134<!DOCTYPE html>
135<meta charset="utf-8">
136<title>HTML Reference File</title>
137<p>Pass if you see WAS displayed below.</p>
138<p>WAS</p>
139```
140
141[testharness]: ./testharness-documentation.html
142[format]: ./test-format-guidelines.html
143[style]: ./test-style-guidelines.html
144[selfdesc]: ./test-style-guidelines.html#self-describing-tests
145[reference-links]: ./test-templates.html#reference-links
146[html-reftest-example]: ./html-reftest-example.html
147[html-reffile-example]: ./html-reffile-example.html
148[css-reftest-example]: http://test.csswg.org/source/css21/borders/border-bottom-applies-to-009.xht
149[css-reffile-example]: http://test.csswg.org/source/css21/borders/border-bottom-applies-to-001-ref.xht
150[svg-reftest-example]: http://test.csswg.org/source/css-transforms-1/translate/svg-translate-001.html
151[svg-reffile-example]: http://test.csswg.org/source/css-transforms-1/translate/reference/svg-translate-ref.html
152[indicating-failure]: ./test-style-guidelines.html#failure
153