1 Writing TAP Tests 2 3Introduction 4 5 This is a guide for users of the C TAP Harness package or similar 6 TAP-based test harnesses explaining how to write tests. If your 7 package uses C TAP Harness as the test suite driver, you may want to 8 copy this document to an appropriate file name in your test suite as 9 documentation for contributors. 10 11About TAP 12 13 TAP is the Test Anything Protocol, a protocol for communication 14 between test cases and a test harness. This is the protocol used by 15 Perl for its internal test suite and for nearly all Perl modules, 16 since it's the format used by the build tools for Perl modules to run 17 tests and report their results. 18 19 A TAP-based test suite works with a somewhat different set of 20 assumptions than an xUnit test suite. In TAP, each test case is a 21 separate program. That program, when run, must produce output in the 22 following format: 23 24 1..4 25 ok 1 - the first test 26 ok 2 27 # a diagnostic, ignored by the harness 28 not ok 3 - a failing test 29 ok 4 # skip a skipped test 30 31 The output should all go to standard output. The first line specifies 32 the number of tests to be run, and then each test produces output that 33 looks like either "ok <n>" or "not ok <n>" depending on whether the 34 test succeeded or failed. Additional information about the test can 35 be provided after the "ok <n>" or "not ok <n>", but is optional. 36 Additional diagnostics and information can be provided in lines 37 beginning with a "#". 38 39 Processing directives are supported after the "ok <n>" or "not ok <n>" 40 and start with a "#". The main one of interest is "# skip" which says 41 that the test was skipped rather than successful and optionally gives 42 the reason. Also supported is "# todo", which normally annotates a 43 failing test and indicates that test is expected to fail, optionally 44 providing a reason for why. 45 46 There are three more special cases. First, the initial line stating 47 the number of tests to run, called the plan, may appear at the end of 48 the output instead of the beginning. This can be useful if the number 49 of tests to run is not known in advance. Second, a plan in the form: 50 51 1..0 # skip entire test case skipped 52 53 can be given instead, which indicates that this entire test case has 54 been skipped (generally because it depends on facilities or optional 55 configuration which is not present). Finally, if the test case 56 encounters a fatal error, it should print the text: 57 58 Bail out! 59 60 on standard output, optionally followed by an error message, and then 61 exit. This tells the harness that the test aborted unexpectedly. 62 63 The exit status of a successful test case should always be 0. The 64 harness will report the test as "dubious" if all the tests appeared to 65 succeed but it exited with a non-zero status. 66 67Writing TAP Tests 68 69 Environment 70 71 One of the special features of C TAP Harness is the environment that 72 it sets up for your test cases. If your test program is called under 73 the runtests driver, the environment variables C_TAP_SOURCE and 74 C_TAP_BUILD will be set to the top of the test directory in the source 75 tree and the top of the build tree, respectively. You can use those 76 environment variables to locate additional test data, programs and 77 libraries built as part of your software build, and other supporting 78 information needed by tests. 79 80 The C and shell TAP libraries support a test_file_path() function, 81 which looks for a file under the build tree and then under the source 82 tree, using the C_TAP_BUILD and C_TAP_SOURCE environment variables, 83 and return the full path to the file. This can be used to locate 84 supporting data files. They also support a test_tmpdir() function 85 that returns a directory that can be used for temporary files during 86 tests. 87 88 Perl 89 90 Since TAP is the native test framework for Perl, writing TAP tests in 91 Perl is very easy and extremely well-supported. If you've never 92 written tests in Perl before, start by reading the documentation for 93 Test::Tutorial and Test::Simple, which walks you through the basics, 94 including the TAP output syntax. Then, the best Perl module to use 95 for serious testing is Test::More, which provides a lot of additional 96 functions over Test::Simple including support for skipping tests, 97 bailing out, and not planning tests in advance. See the documentation 98 of Test::More for all the details and lots of examples. 99 100 C TAP Harness can run Perl test scripts directly and interpret the 101 results correctly, and similarly the Perl Test::Harness module and 102 prove command can run TAP tests written in other languages using, for 103 example, the TAP library that comes with C TAP Harness. You can, if 104 you wish, use the library that comes with C TAP Harness but use prove 105 instead of runtests for running the test suite. 106 107 C 108 109 C TAP Harness provides a basic TAP library that takes away most of the 110 pain of writing TAP test cases in C. A C test case should start with 111 a call to plan(), passing in the number of tests to run. Then, each 112 test should use is_int(), is_string(), is_double(), or is_hex() as 113 appropriate to compare expected and seen values, or ok() to do a 114 simpler boolean test. The is_*() functions take expected and seen 115 values and then a printf-style format string explaining the test 116 (which may be NULL). ok() takes a boolean and then the printf-style 117 string. 118 119 Here's a complete example test program that uses the C TAP library: 120 121 #include <stddef.h> 122 #include <tap/basic.h> 123 124 int 125 main(void) 126 { 127 plan(4); 128 129 ok(1, "the first test"); 130 is_int(42, 42, NULL); 131 diag("a diagnostic, ignored by the harness"); 132 ok(0, "a failing test"); 133 skip("a skipped test"); 134 135 return 0; 136 } 137 138 This test program produces the output shown above in the section on 139 TAP and demonstrates most of the functions. The other functions of 140 interest are sysdiag() (like diag() but adds strerror() results), 141 bail() and sysbail() for fatal errors, skip_block() to skip a whole 142 block of tests, and skip_all() which is called instead of plan() to 143 skip an entire test case. 144 145 The C TAP library also provides plan_lazy(), which can be called 146 instead of plan(). If plan_lazy() is called, the library will keep 147 track of how many test results are reported and will print out the 148 plan at the end of execution of the program. This should normally be 149 avoided since the test may appear to be successful even if it exits 150 prematurely, but it can make writing tests easier in some 151 circumstances. 152 153 Complete API documentation for the basic C TAP library that comes with 154 C TAP Harness is available at: 155 156 <https://www.eyrie.org/~eagle/software/c-tap-harness/> 157 158 It's common to need additional test functions and utility functions 159 for your C tests, particularly if you have to set up and tear down a 160 test environment for your test programs, and it's useful to have them 161 all in the libtap library so that you only have to link your test 162 programs with one library. Rather than editing tap/basic.c and 163 tap/basic.h to add those additional functions, add additional *.c and 164 *.h files into the tap directory with the function implementations and 165 prototypes, and then add those additional objects to the library. 166 That way, you can update tap/basic.c and tap/basic.h from subsequent 167 releases of C TAP Harness without having to merge changes with your 168 own code. 169 170 Libraries of additional useful TAP test functions are available in 171 rra-c-util at: 172 173 <https://www.eyrie.org/~eagle/software/rra-c-util/> 174 175 Some of the code there is particularly useful when testing programs 176 that require Kerberos keys. 177 178 If you implement new test functions that compare an expected and seen 179 value, it's best to name them is_<something> and take the expected 180 value, the seen value, and then a printf-style format string and 181 possible arguments to match the calling convention of the functions 182 provided by C TAP Harness. 183 184 Shell 185 186 C TAP Harness provides a library of shell functions to make it easier 187 to write TAP tests in shell. That library includes much of the same 188 functionality as the C TAP library, but takes its parameters in a 189 somewhat different order to make better use of shell features. 190 191 The libtap.sh file should be installed in a directory named tap in 192 your test suite area. It can then be loaded by tests written in shell 193 using the environment set up by runtests with: 194 195 . "$C_TAP_SOURCE"/tap/libtap.sh 196 197 Here is a complete test case written in shell which produces the same 198 output as the TAP sample above: 199 200 #!/bin/sh 201 202 . "$C_TAP_SOURCE"/tap/libtap.sh 203 cd "$C_TAP_BUILD" 204 205 plan 4 206 ok 'the first test' true 207 ok '' [ 42 -eq 42 ] 208 diag a diagnostic, ignored by the harness 209 ok '' false 210 skip 'a skipped test' 211 212 The shell framework doesn't provide the is_* functions, so you'll use 213 the ok function more. It takes a string describing the text and then 214 treats all of its remaining arguments as a condition, evaluated the 215 same way as the arguments to the "if" statement. If that condition 216 evaluates to true, the test passes; otherwise, the test fails. 217 218 The plan, plan_lazy, diag, and bail functions work the same as with 219 the C library. skip takes a string and skips the next test with that 220 explanation. skip_block takes a count and a string and skips that 221 many tests with that explanation. skip_all takes an optional reason 222 and skips the entire test case. 223 224 Since it's common for shell programs to want to test the output of 225 commands, there's an additional function ok_program provided by the 226 shell test library. It takes the test description string, the 227 expected exit status, the expected program output, and then treats the 228 rest of its arguments as the program to run. That program is run with 229 standard error and standard output combined, and then its exit status 230 and output are tested against the provided values. 231 232 A utility function, strip_colon_error, is provided that runs the 233 command given as its arguments and strips text following a colon and a 234 space from the output (unless there is no whitespace on the line 235 before the colon and the space, normally indicating a prefix of the 236 program name). This function can be used to wrap commands that are 237 expected to fail with output that has a system- or locale-specific 238 error message appended, such as the output of strerror(). 239 240License 241 242 This file is part of the documentation of C TAP Harness, which can be 243 found at <https://www.eyrie.org/~eagle/software/c-tap-harness/>. 244 245 Copyright 2010, 2016 Russ Allbery <eagle@eyrie.org> 246 247 Copying and distribution of this file, with or without modification, 248 are permitted in any medium without royalty provided the copyright 249 notice and this notice are preserved. This file is offered as-is, 250 without any warranty. 251 252 SPDX-License-Identifier: FSFAP 253