1# Writing Tests
2
3Each test suite is organized as a tree, both in the filesystem and further within each file.
4
5- _Suites_, e.g. `src/webgpu/`.
6  - _READMEs_, e.g. `src/webgpu/README.txt`.
7  - _Test Spec Files_, e.g. `src/webgpu/examples.spec.ts`.
8    Identified by their file path.
9    Each test spec file provides a description and a _Test Group_.
10    A _Test Group_ defines a test fixture, and contains multiple:
11    - _Tests_.
12      Identified by a comma-separated list of parts (e.g. `basic,async`)
13      which define a path through a filesystem-like tree (analogy: `basic/async.txt`).
14      Defines a _test function_ and contains multiple:
15      - _Test Cases_.
16        Identified by a list of _Public Parameters_ (e.g. `x` = `1`, `y` = `2`).
17        Each Test Case has the same test function but different Public Parameters.
18
19## Test Tree
20
21A _Test Tree_ is a tree whose leaves are individual Test Cases.
22
23A Test Tree can be thought of as follows:
24
25- Suite, which is the root of a tree with "leaves" which are:
26  - Test Spec Files, each of which is a tree with "leaves" which are:
27    - Tests, each of which is a tree with leaves which are:
28      - Test Cases.
29
30(In the implementation, this conceptual tree of trees is decomposed into one big tree
31whose leaves are Test Cases.)
32
33**Type:** `TestTree`
34
35## Suite
36
37A suite of tests.
38A single suite has a directory structure, and many _test spec files_
39(`.spec.ts` files containing tests) and _READMEs_.
40Each member of a suite is identified by its path within the suite.
41
42**Example:** `src/webgpu/`
43
44### README
45
46**Example:** `src/webgpu/README.txt`
47
48Describes (in prose) the contents of a subdirectory in a suite.
49
50READMEs are only processed at build time, when generating the _Listing_ for a suite.
51
52**Type:** `TestSuiteListingEntryReadme`
53
54## Queries
55
56A _Query_ is a structured object which specifies a subset of cases in exactly one Suite.
57A Query can be represented uniquely as a string.
58Queries are used to:
59
60- Identify a subtree of a suite (by identifying the root node of that subtree).
61- Identify individual cases.
62- Represent the list of tests that a test runner (standalone, wpt, or cmdline) should run.
63- Identify subtrees which should not be "collapsed" during WPT `cts.html` generation,
64  so that that cts.html "variants" can have individual test expectations
65  (i.e. marked as "expected to fail", "skip", etc.).
66
67There are four types of `TestQuery`:
68
69- `TestQueryMultiFile` represents any subtree of the file hierarchy:
70  - `suite:*`
71  - `suite:path,to,*`
72  - `suite:path,to,file,*`
73- `TestQueryMultiTest` represents any subtree of the test hierarchy:
74  - `suite:path,to,file:*`
75  - `suite:path,to,file:path,to,*`
76  - `suite:path,to,file:path,to,test,*`
77- `TestQueryMultiCase` represents any subtree of the case hierarchy:
78  - `suite:path,to,file:path,to,test:*`
79  - `suite:path,to,file:path,to,test:my=0;*`
80  - `suite:path,to,file:path,to,test:my=0;params="here";*`
81- `TestQuerySingleCase` represents as single case:
82  - `suite:path,to,file:path,to,test:my=0;params="here"`
83
84Test Queries are a **weakly ordered set**: any query is
85_Unordered_, _Equal_, _StrictSuperset_, or _StrictSubset_ relative to any other.
86This property is used to construct the complete tree of test cases.
87In the examples above, every example query is a StrictSubset of the previous one
88(note: even `:*` is a subset of `,*`).
89
90In the WPT and standalone harnesses, the query is stored in the URL, e.g.
91`index.html?q=q:u,e:r,y:*`.
92
93Queries are selectively URL-encoded for readability and compatibility with browsers
94(see `encodeURIComponentSelectively`).
95
96**Type:** `TestQuery`
97
98## Listing
99
100A listing of the **test spec files** in a suite.
101
102This can be generated only in Node, which has filesystem access (see `src/tools/crawl.ts`).
103As part of the build step, a _listing file_ is generated (see `src/tools/gen.ts`) so that the
104Test Spec Files can be discovered by the web runner (since it does not have filesystem access).
105
106**Type:** `TestSuiteListing`
107
108### Listing File
109
110Each Suite has one Listing File (`suite/listing.[tj]s`), containing a list of the files
111in the suite.
112
113In `src/suite/listing.ts`, this is computed dynamically.
114In `out/suite/listing.js`, the listing has been pre-baked (by `tools/gen_listings`).
115
116**Type:** Once `import`ed, `ListingFile`
117
118**Example:** `out/webgpu/listing.js`
119
120## Test Spec File
121
122A Test Spec File has a `description` and a Test Group (under which tests and cases are defined).
123
124**Type:** Once `import`ed, `SpecFile`
125
126**Example:** `src/webgpu/**/*.spec.ts`
127
128## Test Group
129
130A subtree of tests. There is one Test Group per Test Spec File.
131
132The Test Fixture used for tests is defined at TestGroup creation.
133
134**Type:** `TestGroup`
135
136## Test
137
138One test. It has a single _test function_.
139
140It may represent multiple _test cases_, each of which runs the same Test Function with different
141Parameters.
142
143A test is named using `TestGroup.test()`, which returns a `TestBuilder`.
144`TestBuilder.params()` can optionally be used to parameterize the test.
145Then, `TestBuilder.fn()` provides the Test Function.
146
147### Test Function
148
149When a test case is run, the Test Function receives an instance of the
150Test Fixture provided to the Test Group, producing test results.
151
152**Type:** `TestFn`
153
154## Test Case / Case
155
156A single case of a test. It is identified by a `TestCaseID`: a test name, and its parameters.
157
158**Type:** During test run time, a case is encapsulated as a `RunCase`.
159
160## Parameters / Params
161
162Each Test Case has a (possibly empty) set of Parameters.
163The parameters are available to the Test Function `f(t)` via `t.params`.
164
165A set of Public Parameters identifies a Test Case within a Test.
166
167There are also Private Paremeters: any parameter name beginning with an underscore (`_`).
168These parameters are not part of the Test Case identification, but are still passed into
169the Test Function. They can be used to manually specify expected results.
170
171**Type:** `CaseParams`
172
173## Test Fixture / Fixture
174
175_Test Fixtures_ provide helpers for tests to use.
176A new instance of the fixture is created for every run of every test case.
177
178There is always one fixture class for a whole test group (though this may change).
179
180The fixture is also how a test gets access to the _case recorder_,
181which allows it to produce test results.
182
183They are also how tests produce results: `.skip()`, `.fail()`, etc.
184
185**Type:** `Fixture`
186
187### `UnitTest` Fixture
188
189Provides basic fixture utilities most useful in the `unittests` suite.
190
191### `GPUTest` Fixture
192
193Provides utilities useful in WebGPU CTS tests.
194
195# Test Results
196
197## Logger
198
199A logger logs the results of a whole test run.
200
201It saves an empty `LiveTestSpecResult` into its results map, then creates a
202_test spec recorder_, which records the results for a group into the `LiveTestSpecResult`.
203
204**Type:** `Logger`
205
206### Test Case Recorder
207
208Refers to a `LiveTestCaseResult` created by the logger.
209Records the results of running a test case (its pass-status, run time, and logs) into it.
210
211**Types:** `TestCaseRecorder`, `LiveTestCaseResult`
212
213#### Test Case Status
214
215The `status` of a `LiveTestCaseResult` can be one of:
216
217- `'running'` (only while still running)
218- `'pass'`
219- `'skip'`
220- `'warn'`
221- `'fail'`
222
223The "worst" result from running a case is always reported (fail > warn > skip > pass).
224Note this means a test can still fail if it's "skipped", if it failed before
225`.skip()` was called.
226
227**Type:** `Status`
228
229## Results Format
230
231The results are returned in JSON format.
232
233They are designed to be easily merged in JavaScript:
234the `"results"` can be passed into the constructor of `Map` and merged from there.
235
236(TODO: Write a merge tool, if needed.)
237
238```js
239{
240  "version": "bf472c5698138cdf801006cd400f587e9b1910a5-dirty",
241  "results": [
242    [
243      "unittests:async_mutex:basic:",
244      { "status": "pass", "timems": 0.286, "logs": [] }
245    ],
246    [
247      "unittests:async_mutex:serial:",
248      { "status": "pass", "timems": 0.415, "logs": [] }
249    ]
250  ]
251}
252```
253