1# SIMD Tests 2 3To execute the simd testsuite, call `make check-simd`, typically with `-j N` 4argument. 5 6For more control over verbosity, compiler flags, and use of a simulator, use 7the environment variables documented below. 8 9## Environment variables 10 11### `target_list` 12 13Similar to dejagnu target lists: E.g. 14`target_list="unix{-march=sandybridge,-march=native/-ffast-math,-march=native/-ffinite-math-only}"` 15would create three subdirs in `testsuite/simd/` to run the complete simd 16testsuite first with `-march=sandybridge`, then with `-march=native 17-ffast-math`, and finally with `-march=native -ffinite-math-only`. 18 19 20### `CHECK_SIMD_CONFIG` 21 22This variable can be set to a path to a file which is equivalent to a dejagnu 23board. The file needs to be a valid `sh` script since it is sourced from the 24`scripts/check_simd` script. Its purpose is to set the `target_list` variable 25depending on `$target_triplet` (or whatever else makes sense for you). Example: 26 27```sh 28case "$target_triplet" in 29x86_64-*) 30 target_list="unix{-march=sandybridge,-march=skylake-avx512,-march=native/-ffast-math,-march=athlon64,-march=core2,-march=nehalem,-march=skylake,-march=native/-ffinite-math-only,-march=knl}" 31 ;; 32 33powerpc64le-*) 34 define_target power7 "-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc112" 35 define_target power8 "-mcpu=power8 -static" "$HOME/bin/run_on_gccfarm gcc112" 36 define_target power9 "-mcpu=power9 -static" "$HOME/bin/run_on_gccfarm gcc135" 37 target_list="power7 power8 power9{,-ffast-math}" 38 ;; 39 40powerpc64-*) 41 define_target power7 "-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc110" 42 define_target power8 "-mcpu=power8 -static" "$HOME/bin/run_on_gccfarm gcc110" 43 target_list="power7 power8{,-ffast-math}" 44 ;; 45esac 46``` 47 48The `unix` target is pre-defined to have no initial flags and no simulator. Use 49the `define_target(name, flags, sim)` function to define your own targets for 50the `target_list` variable. In the example above `define_target power7 51"-mcpu=power7 -static" "$HOME/bin/run_on_gccfarm gcc112"` defines the target 52`power7` which always uses the flags `-mcpu=power7` and `-static` when 53compiling tests and prepends `$HOME/bin/run_on_gccfarm gcc112` to test 54executables. In `target_list` you can now use the name `power7`. E.g. 55`target_list="power7 power7/-ffast-math"` or its shorthand 56`target_list="power7{,-ffast-math}"`. 57 58 59### `DRIVEROPTS` 60 61This variable affects the `Makefile`s generated per target (as defined above). 62It's a string of flags that are prepended to the `driver.sh` invocation which 63builds and runs the tests. You `cd` into a simd test subdir and use `make help` 64to see possible options and a list of all valid targets. 65 66``` 67use DRIVEROPTS=<options> to pass the following options: 68-q, --quiet Disable same-line progress output (default if stdout is 69 not a tty). 70-p, --percentage Add percentage to default same-line progress output. 71-v, --verbose Print one line per test and minimal extra information on 72 failure. 73-vv Print all compiler and test output. 74-k, --keep-failed Keep executables of failed tests. 75--sim <executable> Path to an executable that is prepended to the test 76 execution binary (default: the value of 77 GCC_TEST_SIMULATOR). 78--timeout-factor <x> 79 Multiply the default timeout with x. 80-x, --run-expensive Compile and run tests marked as expensive (default: 81 true if GCC_TEST_RUN_EXPENSIVE is set, false otherwise). 82-o <pattern>, --only <pattern> 83 Compile and run only tests matching the given pattern. 84``` 85 86 87### `TESTFLAGS` 88 89This variable also affects the `Makefile`s generated per target. It's a list of 90compiler flags that are appended to `CXXFLAGS`. 91 92 93### `GCC_TEST_SIMULATOR` 94 95If `--sim` is not passed via `DRIVEROPTS`, then this variable is prepended to 96test invocations. If a simulator was defined via the `CHECK_SIMD_CONFIG` 97script, then then generated `Makefile` sets the `GCC_TEST_SIMULATOR` variable. 98 99 100### `GCC_TEST_RUN_EXPENSIVE` 101 102If set to any non-empty string, run tests marked as expensive, otherwise treat 103these tests as `UNSUPPORTED`. 104 105 106## Writing new tests 107 108A test starts with the copyright header, directly followed by directives 109influencing the set of tests to generate and whether the test driver should 110expect a failure. 111 112Then the test must at least `#include "bits/verify.h"`, which provides `main` 113and declares a `template <typename V> void test()` function, which the test has 114to define. The template parameter is set to `simd<T, Abi>` type where `T` and 115`Abi` are determined by the type and ABI subset dimensions. 116 117The `test()` functions are typically implemented using the `COMPARE(x, 118reference)`, `VERIFY(boolean)`, and `ULP_COMPARE(x, reference, 119allowed_distance)` macros. 120 121### Directives 122 123* `// skip: <type pattern> <ABI subset pattern> <target triplet pattern> 124 <CXXFLAGS pattern>` 125 If all patterns match, the test is silently skipped. 126 127* `// only: <type pattern> <ABI subset pattern> <target triplet pattern> 128 <CXXFLAGS pattern>` 129 If any pattern doesn't match, the test is silently skipped. 130 131* `// expensive: <type pattern> <ABI subset pattern> <target triplet pattern> 132 <CXXFLAGS pattern>` 133 If all patterns match, the test is `UNSUPPORTED` unless expensive tests are 134 enabled. 135 136* `// xfail: run|compile <type pattern> <ABI subset pattern> <target triplet 137 pattern> <CXXFLAGS pattern>` 138 If all patterns match, test compilation or execution is expected to fail. The 139 test then shows as "XFAIL: ...". If the test passes, the test shows "XPASS: 140 ...". 141 142All patterns are matched via 143```sh 144case '<test context>' in 145 <pattern>) 146 # treat as match 147 ;; 148esac 149``` 150The `<CXXFLAGS pattern>` is implicitly adds a `*` wildcard before and after the 151pattern. Thus, the `CXXFLAGS` pattern matches a substring and all other 152patterns require a full match. 153 154Examples: 155```cpp 156// The test is only valid for floating-point types: 157// only: float|double|ldouble * * * 158 159// Skip the test for long double for all powerpc64* targets: 160// skip: ldouble * powerpc64* * 161 162// The test is expected to unconditionally fail on execution: 163// xfail: run * * * * 164 165// ABI subsets 1-9 are considered expensive: 166// expensive: * [1-9] * * 167``` 168 169 170## Implementation sketch 171 172* `scripts/create_testsuite_files` collects all `*.c` and `*.cc` files with 173 `simd/tests/` in their path into the file `testsuite_file_simd` (and at the 174 same time removes them from `testsuite_files`. 175 176* The `check-simd` target in `testsuite/Makefile.am` calls 177 `scripts/check_simd`. This script calls 178 `testsuite/experimental/simd/generate_makefile.sh` to generate `Makefile`s in 179 all requested subdirectories. The subdirectories are communicated back to the 180 make target via a `stdout` pipe. The `check-simd` rule then spawns sub-make 181 in these subdirectories. Finally it collects all summaries 182 (`simd_testsuite.sum`) to present them at the end of the rule. 183 184* The generated Makefiles define targets for each file in `testsuite_file_simd` 185 (you can edit this file after it was generated, though that's not 186 recommended) while adding two test dimensions: type and ABI subset. The type 187 is a list of all arithmetic types, potentially reduced via `only` and/or 188 `skip` directives in the test's source file. The ABI subset is a number 189 between 0 and 9 (inclusive) mapping to a set of `simd_abi`s in 190 `testsuite/experimental/simd/tests/bits/verify.h` (`iterate_abis()`). The 191 tests are thus potentially compiled 170 (17 arithmetic types * 10 ABI 192 subsets) times. This is necessary to limit the memory usage of GCC to 193 reasonable numbers and keep the compile time below 1 minute (per compiler 194 invocation). 195 196* When `make` executes in the generated subdir, the `all` target depends on 197 building and running all tests via `testsuite/experimental/simd/driver.sh` 198 and collecting their logs into a `simd_testsuite.log` and then extracting 199 `simd_testsuite.sum` from it. 200 201* The `driver.sh` script builds and runs the test, parses the compiler and test 202 output, and prints progress information to the terminal. 203 204## Appendix 205 206### `run_on_gccfarm` script 207 208```sh 209#!/bin/sh 210usage() { 211 cat <<EOF 212Usage $0 <hostname> <executable> [arguments] 213 214Copies <executable> to $host, executes it and cleans up again. 215EOF 216} 217 218[ $# -lt 2 ] && usage && exit 1 219case "$1" in 220 -h|--help) 221 usage 222 exit 223 ;; 224esac 225 226host="$1" 227exe="$2" 228shift 2 229 230# Copy executable locally to strip it before scp to remote host 231local_tmpdir=$(mktemp -d) 232cp "$exe" $local_tmpdir 233cd $local_tmpdir 234exe="${exe##*/}" 235powerpc64le-linux-gnu-strip "$exe" 236 237ssh_controlpath=~/.local/run_on_gccfarm/$host 238if [ ! -S $ssh_controlpath ]; then 239 mkdir -p ~/.local/run_on_gccfarm 240 ( 241 flock -n 9 242 if [ ! -S $ssh_controlpath ]; then 243 ssh -o ControlMaster=yes -o ControlPath=$ssh_controlpath -o ControlPersist=10m $host.fsffrance.org true 244 fi 245 ) 9> ~/.local/run_on_gccfarm/lockfile 246fi 247opts="-o ControlPath=$ssh_controlpath" 248 249remote_tmpdir=$(ssh $opts $host.fsffrance.org mktemp -d -p .) 250scp $opts -C -q "$exe" $host.fsffrance.org:$remote_tmpdir/ 251cd 252rm -r "$local_tmpdir" & 253ssh $opts $host.fsffrance.org $remote_tmpdir/$exe "$@" 254ret=$? 255ssh $opts $host.fsffrance.org rm -r $remote_tmpdir & 256exit $ret 257``` 258