1# Testing Cranelift 2 3Cranelift is tested at multiple levels of abstraction and integration. When 4possible, Rust unit tests are used to verify single functions and types. When 5testing the interaction between compiler passes, file-level tests are 6appropriate. 7 8## Rust tests 9 10Rust and Cargo have good support for testing. Cranelift uses unit tests, doc 11tests, and integration tests where appropriate. The 12[Rust By Example page on Testing] is a great illustration on how to write 13each of these forms of test. 14 15[Rust By Example page on Testing]: https://doc.rust-lang.org/rust-by-example/testing.html 16 17## File tests 18 19Compilers work with large data structures representing programs, and it quickly 20gets unwieldy to generate test data programmatically. File-level tests make it 21easier to provide substantial input functions for the compiler tests. 22 23File tests are `*.clif` files in the `filetests/` directory 24hierarchy. Each file has a header describing what to test followed by a number 25of input functions in the :doc:`Cranelift textual intermediate representation 26<ir>`: 27 28.. productionlist:: 29 test_file : test_header `function_list` 30 test_header : test_commands (`isa_specs` | `settings`) 31 test_commands : test_command { test_command } 32 test_command : "test" test_name { option } "\n" 33 34The available test commands are described below. 35 36Many test commands only make sense in the context of a target instruction set 37architecture. These tests require one or more ISA specifications in the test 38header: 39 40.. productionlist:: 41 isa_specs : { [`settings`] isa_spec } 42 isa_spec : "isa" isa_name { `option` } "\n" 43 44The options given on the `isa` line modify the ISA-specific settings defined in 45`cranelift-codegen/meta-python/isa/*/settings.py`. 46 47All types of tests allow shared Cranelift settings to be modified: 48 49.. productionlist:: 50 settings : { setting } 51 setting : "set" { option } "\n" 52 option : flag | setting "=" value 53 54The shared settings available for all target ISAs are defined in 55`cranelift-codegen/meta-python/base/settings.py`. 56 57The `set` lines apply settings cumulatively: 58 59``` 60 test legalizer 61 set opt_level=best 62 set is_pic=1 63 isa riscv64 64 set is_pic=0 65 isa riscv32 supports_m=false 66 67 function %foo() {} 68``` 69 70This example will run the legalizer test twice. Both runs will have 71`opt_level=best`, but they will have different `is_pic` settings. The 32-bit 72run will also have the RISC-V specific flag `supports_m` disabled. 73 74The filetests are run automatically as part of `cargo test`, and they can 75also be run manually with the `clif-util test` command. 76 77By default, the test runner will spawn a thread pool with as many threads as 78there are logical CPUs. You can explicitly control how many threads are spawned 79via the `CRANELIFT_FILETESTS_THREADS` environment variable. For example, to 80limit the test runner to a single thread, use: 81 82``` 83$ CRANELIFT_FILETESTS_THREADS=1 clif-util test path/to/file.clif 84``` 85 86### Filecheck 87 88Many of the test commands described below use *filecheck* to verify their 89output. Filecheck is a Rust implementation of the LLVM tool of the same name. 90See the `documentation <https://docs.rs/filecheck/>`_ for details of its syntax. 91 92Comments in `.clif` files are associated with the entity they follow. 93This typically means an instruction or the whole function. Those tests that 94use filecheck will extract comments associated with each function (or its 95entities) and scan them for filecheck directives. The test output for each 96function is then matched against the filecheck directives for that function. 97 98Comments appearing before the first function in a file apply to every function. 99This is useful for defining common regular expression variables with the 100`regex:` directive, for example. 101 102Note that LLVM's file tests don't separate filecheck directives by their 103associated function. It verifies the concatenated output against all filecheck 104directives in the test file. LLVM's :command:`FileCheck` command has a 105`CHECK-LABEL:` directive to help separate the output from different functions. 106Cranelift's tests don't need this. 107 108### `test cat` 109 110This is one of the simplest file tests, used for testing the conversion to and 111from textual IR. The `test cat` command simply parses each function and 112converts it back to text again. The text of each function is then matched 113against the associated filecheck directives. 114 115Example: 116 117``` 118 function %r1() -> i32, f32 { 119 ebb1: 120 v10 = iconst.i32 3 121 v20 = f32const 0.0 122 return v10, v20 123 } 124 ; sameln: function %r1() -> i32, f32 { 125 ; nextln: ebb0: 126 ; nextln: v10 = iconst.i32 3 127 ; nextln: v20 = f32const 0.0 128 ; nextln: return v10, v20 129 ; nextln: } 130``` 131 132### `test verifier` 133 134Run each function through the IR verifier and check that it produces the 135expected error messages. 136 137Expected error messages are indicated with an `error:` directive *on the 138instruction that produces the verifier error*. Both the error message and 139reported location of the error is verified: 140 141``` 142 test verifier 143 144 function %test(i32) { 145 ebb0(v0: i32): 146 jump ebb1 ; error: terminator 147 return 148 } 149``` 150 151This example test passes if the verifier fails with an error message containing 152the sub-string `"terminator"` *and* the error is reported for the `jump` 153instruction. 154 155If a function contains no `error:` annotations, the test passes if the 156function verifies correctly. 157 158### `test print-cfg` 159 160Print the control flow graph of each function as a Graphviz graph, and run 161filecheck over the result. See also the :command:`clif-util print-cfg` 162command: 163 164``` 165 ; For testing cfg generation. This code is nonsense. 166 test print-cfg 167 test verifier 168 169 function %nonsense(i32, i32) -> f32 { 170 ; check: digraph %nonsense { 171 ; regex: I=\binst\d+\b 172 ; check: label="{ebb0 | <$(BRZ=$I)>brz ebb2 | <$(JUMP=$I)>jump ebb1}"] 173 174 ebb0(v0: i32, v1: i32): 175 brz v1, ebb2 ; unordered: ebb0:$BRZ -> ebb2 176 v2 = iconst.i32 0 177 jump ebb1(v2) ; unordered: ebb0:$JUMP -> ebb1 178 179 ebb1(v5: i32): 180 return v0 181 182 ebb2: 183 v100 = f32const 0.0 184 return v100 185 } 186``` 187 188### `test domtree` 189 190Compute the dominator tree of each function and validate it against the 191`dominates:` annotations:: 192 193``` 194 test domtree 195 196 function %test(i32) { 197 ebb0(v0: i32): 198 jump ebb1 ; dominates: ebb1 199 ebb1: 200 brz v0, ebb3 ; dominates: ebb3 201 jump ebb2 ; dominates: ebb2 202 ebb2: 203 jump ebb3 204 ebb3: 205 return 206 } 207``` 208 209Every reachable extended basic block except for the entry block has an 210*immediate dominator* which is a jump or branch instruction. This test passes 211if the `dominates:` annotations on the immediate dominator instructions are 212both correct and complete. 213 214This test also sends the computed CFG post-order through filecheck. 215 216### `test legalizer` 217 218Legalize each function for the specified target ISA and run the resulting 219function through filecheck. This test command can be used to validate the 220encodings selected for legal instructions as well as the instruction 221transformations performed by the legalizer. 222 223### `test regalloc` 224 225Test the register allocator. 226 227First, each function is legalized for the specified target ISA. This is 228required for register allocation since the instruction encodings provide 229register class constraints to the register allocator. 230 231Second, the register allocator is run on the function, inserting spill code and 232assigning registers and stack slots to all values. 233 234The resulting function is then run through filecheck. 235 236### `test binemit` 237 238Test the emission of binary machine code. 239 240The functions must contains instructions that are annotated with both encodings 241and value locations (registers or stack slots). For instructions that are 242annotated with a `bin:` directive, the emitted hexadecimal machine code for 243that instruction is compared to the directive: 244 245``` 246 test binemit 247 isa riscv 248 249 function %int32() { 250 ebb0: 251 [-,%x5] v0 = iconst.i32 1 252 [-,%x6] v1 = iconst.i32 2 253 [R#0c,%x7] v10 = iadd v0, v1 ; bin: 006283b3 254 [R#200c,%x8] v11 = isub v0, v1 ; bin: 40628433 255 return 256 } 257``` 258 259If any instructions are unencoded (indicated with a `[-]` encoding field), they 260will be encoded using the same mechanism as the legalizer uses. However, 261illegal instructions for the ISA won't be expanded into other instruction 262sequences. Instead the test will fail. 263 264Value locations must be present if they are required to compute the binary 265bits. Missing value locations will cause the test to crash. 266 267### `test simple-gvn` 268 269Test the simple GVN pass. 270 271The simple GVN pass is run on each function, and then results are run 272through filecheck. 273 274### `test licm` 275 276Test the LICM pass. 277 278The LICM pass is run on each function, and then results are run 279through filecheck. 280 281### `test dce` 282 283Test the DCE pass. 284 285The DCE pass is run on each function, and then results are run 286through filecheck. 287 288### `test shrink` 289 290Test the instruction shrinking pass. 291 292The shrink pass is run on each function, and then results are run 293through filecheck. 294 295### `test preopt` 296 297Test the preopt pass. 298 299The preopt pass is run on each function, and then results are run 300through filecheck. 301 302### `test postopt` 303 304Test the postopt pass. 305 306The postopt pass is run on each function, and then results are run 307through filecheck. 308 309### `test compile` 310 311Test the whole code generation pipeline. 312 313Each function is passed through the full `Context::compile()` function 314which is normally used to compile code. This type of test often depends 315on assertions or verifier errors, but it is also possible to use 316filecheck directives which will be matched against the final form of the 317Cranelift IR right before binary machine code emission. 318 319### `test run` 320 321Compile and execute a function. 322 323This test command allows several directives: 324 - to print the result of running a function to stdout, add a `print` 325 directive and call the preceding function with arguments (see `%foo` in 326 the example below); remember to enable `--nocapture` if running these 327 tests through Cargo 328 - to check the result of a function, add a `run` directive and call the 329 preceding function with a comparison (`==` or `!=`) (see `%bar` below) 330 - for backwards compatibility, to check the result of a function with a 331 `() -> b*` signature, only the `run` directive is required, with no 332 invocation or comparison (see `%baz` below); a `true` value is 333 interpreted as a successful test execution, whereas a `false` value is 334 interpreted as a failed test. 335 336Currently a `target` is required but is only used to indicate whether the host 337platform can run the test and currently only the architecture is filtered. The 338host platform's native target will be used to actually compile the test. 339 340Example: 341 342``` 343 test run 344 target x86_64 345 346 ; how to print the results of a function 347 function %foo() -> i32 { 348 block0: 349 v0 = iconst.i32 42 350 return v0 351 } 352 ; print: %foo() 353 354 ; how to check the results of a function 355 function %bar(i32) -> i32 { 356 block0(v0:i32): 357 v1 = iadd_imm v0, 1 358 return v1 359 } 360 ; run: %bar(1) == 2 361 362 ; legacy method of checking the results of a function 363 function %baz() -> b1 { 364 block0: 365 v0 = bconst.b1 true 366 return v0 367 } 368 ; run 369``` 370