1# Writing Tests 2 3Each test suite is organized as a tree, both in the filesystem and further within each file. 4 5- _Suites_, e.g. `src/webgpu/`. 6 - _READMEs_, e.g. `src/webgpu/README.txt`. 7 - _Test Spec Files_, e.g. `src/webgpu/examples.spec.ts`. 8 Identified by their file path. 9 Each test spec file provides a description and a _Test Group_. 10 A _Test Group_ defines a test fixture, and contains multiple: 11 - _Tests_. 12 Identified by a comma-separated list of parts (e.g. `basic,async`) 13 which define a path through a filesystem-like tree (analogy: `basic/async.txt`). 14 Defines a _test function_ and contains multiple: 15 - _Test Cases_. 16 Identified by a list of _Public Parameters_ (e.g. `x` = `1`, `y` = `2`). 17 Each Test Case has the same test function but different Public Parameters. 18 19## Test Tree 20 21A _Test Tree_ is a tree whose leaves are individual Test Cases. 22 23A Test Tree can be thought of as follows: 24 25- Suite, which is the root of a tree with "leaves" which are: 26 - Test Spec Files, each of which is a tree with "leaves" which are: 27 - Tests, each of which is a tree with leaves which are: 28 - Test Cases. 29 30(In the implementation, this conceptual tree of trees is decomposed into one big tree 31whose leaves are Test Cases.) 32 33**Type:** `TestTree` 34 35## Suite 36 37A suite of tests. 38A single suite has a directory structure, and many _test spec files_ 39(`.spec.ts` files containing tests) and _READMEs_. 40Each member of a suite is identified by its path within the suite. 41 42**Example:** `src/webgpu/` 43 44### README 45 46**Example:** `src/webgpu/README.txt` 47 48Describes (in prose) the contents of a subdirectory in a suite. 49 50READMEs are only processed at build time, when generating the _Listing_ for a suite. 51 52**Type:** `TestSuiteListingEntryReadme` 53 54## Queries 55 56A _Query_ is a structured object which specifies a subset of cases in exactly one Suite. 57A Query can be represented uniquely as a string. 58Queries are used to: 59 60- Identify a subtree of a suite (by identifying the root node of that subtree). 61- Identify individual cases. 62- Represent the list of tests that a test runner (standalone, wpt, or cmdline) should run. 63- Identify subtrees which should not be "collapsed" during WPT `cts.html` generation, 64 so that that cts.html "variants" can have individual test expectations 65 (i.e. marked as "expected to fail", "skip", etc.). 66 67There are four types of `TestQuery`: 68 69- `TestQueryMultiFile` represents any subtree of the file hierarchy: 70 - `suite:*` 71 - `suite:path,to,*` 72 - `suite:path,to,file,*` 73- `TestQueryMultiTest` represents any subtree of the test hierarchy: 74 - `suite:path,to,file:*` 75 - `suite:path,to,file:path,to,*` 76 - `suite:path,to,file:path,to,test,*` 77- `TestQueryMultiCase` represents any subtree of the case hierarchy: 78 - `suite:path,to,file:path,to,test:*` 79 - `suite:path,to,file:path,to,test:my=0;*` 80 - `suite:path,to,file:path,to,test:my=0;params="here";*` 81- `TestQuerySingleCase` represents as single case: 82 - `suite:path,to,file:path,to,test:my=0;params="here"` 83 84Test Queries are a **weakly ordered set**: any query is 85_Unordered_, _Equal_, _StrictSuperset_, or _StrictSubset_ relative to any other. 86This property is used to construct the complete tree of test cases. 87In the examples above, every example query is a StrictSubset of the previous one 88(note: even `:*` is a subset of `,*`). 89 90In the WPT and standalone harnesses, the query is stored in the URL, e.g. 91`index.html?q=q:u,e:r,y:*`. 92 93Queries are selectively URL-encoded for readability and compatibility with browsers 94(see `encodeURIComponentSelectively`). 95 96**Type:** `TestQuery` 97 98## Listing 99 100A listing of the **test spec files** in a suite. 101 102This can be generated only in Node, which has filesystem access (see `src/tools/crawl.ts`). 103As part of the build step, a _listing file_ is generated (see `src/tools/gen.ts`) so that the 104Test Spec Files can be discovered by the web runner (since it does not have filesystem access). 105 106**Type:** `TestSuiteListing` 107 108### Listing File 109 110Each Suite has one Listing File (`suite/listing.[tj]s`), containing a list of the files 111in the suite. 112 113In `src/suite/listing.ts`, this is computed dynamically. 114In `out/suite/listing.js`, the listing has been pre-baked (by `tools/gen_listings`). 115 116**Type:** Once `import`ed, `ListingFile` 117 118**Example:** `out/webgpu/listing.js` 119 120## Test Spec File 121 122A Test Spec File has a `description` and a Test Group (under which tests and cases are defined). 123 124**Type:** Once `import`ed, `SpecFile` 125 126**Example:** `src/webgpu/**/*.spec.ts` 127 128## Test Group 129 130A subtree of tests. There is one Test Group per Test Spec File. 131 132The Test Fixture used for tests is defined at TestGroup creation. 133 134**Type:** `TestGroup` 135 136## Test 137 138One test. It has a single _test function_. 139 140It may represent multiple _test cases_, each of which runs the same Test Function with different 141Parameters. 142 143A test is named using `TestGroup.test()`, which returns a `TestBuilder`. 144`TestBuilder.params()` can optionally be used to parameterize the test. 145Then, `TestBuilder.fn()` provides the Test Function. 146 147### Test Function 148 149When a test case is run, the Test Function receives an instance of the 150Test Fixture provided to the Test Group, producing test results. 151 152**Type:** `TestFn` 153 154## Test Case / Case 155 156A single case of a test. It is identified by a `TestCaseID`: a test name, and its parameters. 157 158**Type:** During test run time, a case is encapsulated as a `RunCase`. 159 160## Parameters / Params 161 162Each Test Case has a (possibly empty) set of Parameters. 163The parameters are available to the Test Function `f(t)` via `t.params`. 164 165A set of Public Parameters identifies a Test Case within a Test. 166 167There are also Private Paremeters: any parameter name beginning with an underscore (`_`). 168These parameters are not part of the Test Case identification, but are still passed into 169the Test Function. They can be used to manually specify expected results. 170 171**Type:** `CaseParams` 172 173## Test Fixture / Fixture 174 175_Test Fixtures_ provide helpers for tests to use. 176A new instance of the fixture is created for every run of every test case. 177 178There is always one fixture class for a whole test group (though this may change). 179 180The fixture is also how a test gets access to the _case recorder_, 181which allows it to produce test results. 182 183They are also how tests produce results: `.skip()`, `.fail()`, etc. 184 185**Type:** `Fixture` 186 187### `UnitTest` Fixture 188 189Provides basic fixture utilities most useful in the `unittests` suite. 190 191### `GPUTest` Fixture 192 193Provides utilities useful in WebGPU CTS tests. 194 195# Test Results 196 197## Logger 198 199A logger logs the results of a whole test run. 200 201It saves an empty `LiveTestSpecResult` into its results map, then creates a 202_test spec recorder_, which records the results for a group into the `LiveTestSpecResult`. 203 204**Type:** `Logger` 205 206### Test Case Recorder 207 208Refers to a `LiveTestCaseResult` created by the logger. 209Records the results of running a test case (its pass-status, run time, and logs) into it. 210 211**Types:** `TestCaseRecorder`, `LiveTestCaseResult` 212 213#### Test Case Status 214 215The `status` of a `LiveTestCaseResult` can be one of: 216 217- `'running'` (only while still running) 218- `'pass'` 219- `'skip'` 220- `'warn'` 221- `'fail'` 222 223The "worst" result from running a case is always reported (fail > warn > skip > pass). 224Note this means a test can still fail if it's "skipped", if it failed before 225`.skip()` was called. 226 227**Type:** `Status` 228 229## Results Format 230 231The results are returned in JSON format. 232 233They are designed to be easily merged in JavaScript: 234the `"results"` can be passed into the constructor of `Map` and merged from there. 235 236(TODO: Write a merge tool, if needed.) 237 238```js 239{ 240 "version": "bf472c5698138cdf801006cd400f587e9b1910a5-dirty", 241 "results": [ 242 [ 243 "unittests:async_mutex:basic:", 244 { "status": "pass", "timems": 0.286, "logs": [] } 245 ], 246 [ 247 "unittests:async_mutex:serial:", 248 { "status": "pass", "timems": 0.415, "logs": [] } 249 ] 250 ] 251} 252``` 253