1.Dd December 20, 2004 2.Os 3.Dt TAP 3 4.Sh NAME 5.Nm tap 6.Nd write tests that implement the Test Anything Protocol 7.Sh SYNOPSIS 8.In tap.h 9.Sh DESCRIPTION 10The 11.Nm 12library provides functions for writing test scripts that produce output 13consistent with the Test Anything Protocol. A test harness that parses 14this protocol can run these tests and produce useful reports indicating 15their success or failure. 16.Ss PRINTF STRINGS 17In the descriptions that follow, for any function that takes as the 18last two parameters 19.Dq Fa char * , Fa ... 20it can be assumed that the 21.Fa char * 22is a 23.Fn printf 24-like format string, and the optional arguments are values to be placed 25in that string. 26.Ss TEST PLANS 27.Bl -tag -width indent 28.It Xo 29.Ft int 30.Fn plan_tests "unsigned int" 31.Xc 32.It Xo 33.Ft int 34.Fn plan_no_plan "void" 35.Xc 36.It Xo 37.Ft int 38.Fn plan_skip_all "char *" "..." 39.Xc 40.El 41.Pp 42You must first specify a test plan. This indicates how many tests you 43intend to run, and allows the test harness to notice if any tests were 44missed, or if the test program exited prematurely. 45.Pp 46To do this, use 47.Fn plan_tests , 48which always returns 0. The function will cause your program to exit 49prematurely if you specify 0 tests. 50.Pp 51In some situations you may not know how many tests you will be running, or 52you are developing your test program, and do not want to update the 53.Fn plan_tests 54parameter every time you make a change. For those situations use 55.Fn plan_no_plan . 56It returns 0, and indicates to the test harness that an indeterminate number 57of tests will be run. 58.Pp 59Both 60.Fn plan_tests 61and 62.Fn plan_no_plan 63will cause your test program to exit prematurely with a diagnostic 64message if they are called more than once. 65.Pp 66If your test program detects at run time that some required functionality 67is missing (for example, it relies on a database connection which is not 68present, or a particular configuration option that has not been included 69in the running kernel) use 70.Fn plan_skip_all , 71passing as parameters a string to display indicating the reason for skipping 72the tests. 73.Ss SIMPLE TESTS 74.Bl -tag -width indent 75.It Xo 76.Ft unsigned int 77.Fn ok "expression" "char *" "..." 78.Xc 79.It Xo 80.Ft unsigned int 81.Fn ok1 "expression" 82.Xc 83.It Xo 84.Ft unsigned int 85.Fn pass "char *" "..." 86.Xc 87.It Xo 88.Ft unsigned int 89.Fn fail "char *" "..." 90.Xc 91.El 92.Pp 93Tests are implemented as expressions checked by calls to the 94.Fn ok 95and 96.Fn ok1 97macros. In both cases 98.Fa expression 99should evaluate to true if the test succeeded. 100.Pp 101.Fn ok 102allows you to specify a name, or comment, describing the test which will 103be included in the output. 104.Fn ok1 105is for those times when the expression to be tested is self 106explanatory and does not need an associated comment. In those cases 107the test expression becomes the comment. 108.Pp 109These four calls are equivalent: 110.Bd -literal -offset indent 111int i = 5; 112 113ok(i == 5, "i equals 5"); /* Overly verbose */ 114ok(i == 5, "i equals %d", i); /* Just to demonstrate printf-like 115 behaviour of the test name */ 116ok(i == 5, "i == 5"); /* Needless repetition */ 117ok1(i == 5); /* Just right */ 118.Ed 119.Pp 120It is good practice to ensure that the test name describes the meaning 121behind the test rather than what you are testing. Viz 122.Bd -literal -offset indent 123ok(db != NULL, "db is not NULL"); /* Not bad, but */ 124ok(db != NULL, "Database conn. succeeded"); /* this is better */ 125.Ed 126.Pp 127.Fn ok 128and 129.Fn ok1 130return 1 if the expression evaluated to true, and 0 if it evaluated to 131false. This lets you chain calls from 132.Fn ok 133to 134.Fn diag 135to only produce diagnostic output if the test failed. For example, this 136code will include diagnostic information about why the database connection 137failed, but only if the test failed. 138.Bd -literal -offset indent 139ok(db != NULL, "Database conn. succeeded") || 140 diag("Database error code: %d", dberrno); 141.Ed 142.Pp 143You also have 144.Fn pass 145and 146.Fn fail . 147From the Test::More documentation: 148.Bd -literal -offset indent 149Sometimes you just want to say that the tests have passed. 150Usually the case is you've got some complicated condition 151that is difficult to wedge into an ok(). In this case, 152you can simply use pass() (to declare the test ok) or fail 153(for not ok). 154 155Use these very, very, very sparingly. 156.Ed 157.Pp 158These are synonyms for ok(1, ...) and ok(0, ...). 159.Ss SKIPPING TESTS 160.Bl -tag -width indent 161.It Xo 162.Ft int 163.Fn skip "unsigned int" "char *" "..." 164.Xc 165.It Xo 166.Fn skip_start "expression" "unsigned int" "char *" "..." 167.Xc 168.It Xo 169.Sy skip_end 170.Xc 171.El 172.Pp 173Sets of tests can be skipped. Ordinarily you would do this because 174the test can't be run in this particular testing environment. 175.Pp 176For example, suppose some tests should be run as root. If the test is 177not being run as root then the tests should be skipped. In this 178implementation, skipped tests are flagged as being ok, with a special 179message indicating that they were skipped. It is your responsibility 180to ensure that the number of tests skipped (the first parameter to 181.Fn skip ) 182is correct for the number of tests to skip. 183.Pp 184One way of implementing this is with a 185.Dq do { } while(0); 186loop, or an 187.Dq if( ) { } else { } 188construct, to ensure that there are no additional side effects from the 189skipped tests. 190.Bd -literal -offset indent 191if(getuid() != 0) { 192 skip(1, "because test only works as root"); 193} else { 194 ok(do_something_as_root() == 0, "Did something as root"); 195} 196.Ed 197.Pp 198Two macros are provided to assist with this. The previous example could 199be re-written as follows. 200.Bd -literal -offset indent 201skip_start(getuid() != 0, 1, "because test only works as root"); 202 203ok(do_something_as_root() == 0, "Did something as root"); 204 205skip_end; /* It's a macro, no parentheses */ 206.Ed 207.Ss MARKING TESTS AS Dq TODO 208.Bl -tag -width indent 209.It Xo 210.Ft void 211.Fn todo_start "char *" "..." 212.Xc 213.It Xo 214.Ft void 215.Fn todo_end "void" 216.Xc 217.El 218.Pp 219Sets of tests can be flagged as being 220.Dq TODO . 221These are tests that you expect to fail, probably because you haven't 222fixed a bug, or finished a new feature yet. These tests will still be 223run, but with additional output that indicates that they are expected 224to fail. Should a test start to succeed unexpectedly, tools like 225.Xr prove 1 226will indicate this, and you can move the test out of the todo 227block. This is much more useful than simply commenting out (or 228.Dq #ifdef 0 ... #endif ) 229the tests. 230.Bd -literal -offset indent 231todo_start("dwim() not returning true yet"); 232 233ok(dwim(), "Did what the user wanted"); 234 235todo_end(); 236.Ed 237.Pp 238Should 239.Fn dwim 240ever start succeeding you will know about it as soon as you run the 241tests. Note that 242.Em unlike 243the 244.Fn skip_* 245family, additional code between 246.Fn todo_start 247and 248.Fn todo_end 249.Em is 250executed. 251.Ss SKIP vs. TODO 252From the Test::More documentation; 253.Bd -literal -offset indent 254If it's something the user might not be able to do, use SKIP. 255This includes optional modules that aren't installed, running 256under an OS that doesn't have some feature (like fork() or 257symlinks), or maybe you need an Internet connection and one 258isn't available. 259 260If it's something the programmer hasn't done yet, use TODO. 261This is for any code you haven't written yet, or bugs you have 262yet to fix, but want to put tests in your testing script 263(always a good idea). 264.Ed 265.Ss DIAGNOSTIC OUTPUT 266.Bl -tag -width indent 267.It Xo 268.Fr unsigned int 269.Fn diag "char *" "..." 270.Xc 271.El 272.Pp 273If your tests need to produce diagnostic output, use 274.Fn diag . 275It ensures that the output will not be considered by the TAP test harness. 276.Fn diag 277adds the necessary trailing 278.Dq \en 279for you. 280.Bd -literal -offset indent 281diag("Expected return code 0, got return code %d", rcode); 282.Ed 283.Pp 284.Fn diag 285always returns 0. 286.Ss EXIT STATUS 287.Bl -tag -width indent 288.It Xo 289.Fr int 290.Fn exit_status void 291.Xc 292.El 293.Pp 294For maximum compatability your test program should return a particular 295exit code. This is calculated by 296.Fn exit_status 297so it is sufficient to always return from 298.Fn main 299with either 300.Dq return exit_status(); 301or 302.Dq exit(exit_status()); 303as appropriate. 304.Sh EXAMPLES 305The 306.Pa tests 307directory in the source distribution contains numerous tests of 308.Nm 309functionality, written using 310.Nm . 311Examine them for examples of how to construct test suites. 312.Sh COMPATABILITY 313.Nm 314strives to be compatible with the Perl Test::More and Test::Harness 315modules. The test suite verifies that 316.Nm 317is bug-for-bug compatible with their behaviour. This is why some 318functions which would more naturally return nothing return constant 319values. 320.Pp 321If the 322.Lb libpthread 323is found at compile time, 324.Nm 325.Em should 326be thread safe. Indications to the contrary (and test cases that expose 327incorrect behaviour) are very welcome. 328.Sh SEE ALSO 329.Xr Test::More 1 , 330.Xr Test::Harness 1 , 331.Xr prove 1 332.Sh STANDARDS 333.Nm 334requires a 335.St -isoC-99 336compiler. Some of the 337.Nm 338functionality is implemented as variadic macros, and that functionality 339was not formally codified until C99. Patches to use 340.Nm 341with earlier compilers that have their own implementation of variadic 342macros will be gratefully received. 343.Sh HISTORY 344.Nm 345was written to help improve the quality and coverage of the FreeBSD 346regression test suite, and released in the hope that others find it 347a useful tool to help improve the quality of their code. 348.Sh AUTHORS 349.An "Nik Clayton" Aq nik@ngo.org.uk , 350.Aq nik@FreeBSD.org 351.Pp 352.Nm 353would not exist without the efforts of 354.An "Michael G Schwern" Aq schqern@pobox.com , 355.An "Andy Lester" Aq andy@petdance.com , 356and the countless others who have worked on the Perl QA programme. 357.Sh BUGS 358Ideally, running the tests would have no side effects on the behaviour 359of the application you are testing. However, it is not always possible 360to avoid them. The following side effects of using 361.Nm 362are known. 363.Bl -bullet -offset indent 364.It 365stdout is set to unbuffered mode after calling any of the 366.Fn plan_* 367functions. 368.El 369