1Test suite 2########## 3 4Sparse has a number of test cases in its validation directory. The test-suite 5script aims at making automated checking of these tests possible. It works by 6embedding tags in C comments in the test cases. 7 8Tag's syntax 9============ 10 11``check-name:`` *name* 12 13 Name of the test. This is the only mandatory tag. 14 15``check-description:`` *description ...* 16 17 A description of what the test checks. 18 19``check-command:`` *command arg ...* 20 21 There are different kinds of tests. Some can validate the sparse 22 preprocessor, while others will use sparse, cgcc, or even other backends 23 of the library. check-command allows you to give a custom command to 24 run the test-case. 25 The ``$file`` string is special. It will be expanded to the file name at 26 run time. 27 It defaults to ``sparse $file``. 28 29``check-arch-ignore:`` *arch[|...]* 30 31``check-arch-only:`` *arch[|...]* 32 33 Ignore the test if the current architecture (as returned by ``uname -m``) 34 matches or not one of the archs given in the pattern. 35 36``check-assert:`` *condition* 37 38 Ignore the test if the given condition is false when evaluated as a 39 static assertion (``_Static_assert``). 40 41``check-cpp-if:`` *condition* 42 43 Ignore the test if the given condition is false when evaluated 44 by sparse's pre-processor. 45 46``check-exit-value:`` *value* 47 48 The expected exit value of check-command. It defaults to 0. 49 50``check-timeout:`` *timeout* 51 52 The maximum expected duration of check-command, in seconds. 53 It defaults to 1. 54 55``check-output-start`` / ``check-output-end`` 56 57 The expected output (stdout and stderr) of check-command lies between 58 those two tags. It defaults to no output. 59 60``check-output-ignore`` / ``check-error-ignore`` 61 62 Don't check the expected output (stdout or stderr) of check-command 63 (useful when this output is not comparable or if you're only interested 64 in the exit value). By default this check is done. 65 66``check-known-to-fail`` 67 68 Mark the test as being known to fail. 69 70``check-output-contains:`` *pattern* 71 72 Check that the output (stdout) contains the given pattern. 73 Several such tags can be given, in which case the output 74 must contains all the patterns. 75 76``check-output-excludes:`` *pattern* 77 78 Similar than the above one, but with opposite logic. 79 Check that the output (stdout) doesn't contain the given pattern. 80 Several such tags can be given, in which case the output 81 must contains none of the patterns. 82 83``check-output-pattern(``\ *nbr*\ ``):`` *pattern* 84 85``check-output-pattern(``\ *min*\ ``,``\ *max*\ ``):`` *pattern* 86 87 Similar to the contains/excludes above, but with full control 88 of the number of times the pattern should occur in the output. 89 If *min* or *max* is ``-`` the corresponding check is ignored. 90 91Using test-suite 92================ 93 94The test-suite script is called through the check target of the Makefile. It 95will try to check every test case it finds (``find validation -name '*.c'``). 96It can be called to check a single test with:: 97 98 $ cd validation 99 $ ./test-suite single preprocessor/preprocessor1.c 100 TEST Preprocessor #1 (preprocessor/preprocessor1.c) 101 preprocessor/preprocessor1.c passed ! 102 103 104Writing a test 105============== 106 107The test-suite comes with a format command to make a test easier to write:: 108 109 test-suite format [-a] [-l] [-f] file [name [cmd]] 110 111`name:` check-name value 112 If no name is provided, it defaults to the file name. 113 114`cmd:` check-command value 115 If no cmd is provided, it defaults to ``sparse $file``. 116 117The output of the test-suite format command can be redirected into the 118test case to create a test-suite formatted file.:: 119 120 $ ./test-suite format bad-assignment.c Assignment >> bad-assignment.c 121 $ cat !$ 122 cat bad-assignment.c 123 /* 124 * check-name: bad assignment 125 * 126 * check-command: sparse $file 127 * check-exit-value: 1 128 * 129 * check-output-start 130 bad-assignment.c:3:6: error: Expected ; at end of statement 131 bad-assignment.c:3:6: error: got \ 132 * check-output-end 133 */ 134 135The same effect without the redirection can be achieved by using the ``-a`` 136option. 137 138You can define the check-command you want to use for the test.:: 139 140 $ ./test-suite format -a validation/preprocessor2.c "Preprocessor #2" \ 141 "sparse -E \$file" 142 $ cat !$ 143 cat validation/preprocessor2.c 144 /* 145 * This one we happen to get right. 146 * 147 * It should result in a simple 148 * 149 * a + b 150 * 151 * for a proper preprocessor. 152 */ 153 #define TWO a, b 154 155 #define UNARY(x) BINARY(x) 156 #define BINARY(x, y) x + y 157 158 UNARY(TWO) 159 /* 160 * check-name: Preprocessor #2 161 * 162 * check-command: sparse -E $file 163 * check-exit-value: 0 164 * 165 * check-output-start 166 167 a + b 168 * check-output-end 169 */ 170