1.\" 2.\" Copyright (c) 2011 3.\" The DragonFly Project. All rights reserved. 4.\" 5.\" Redistribution and use in source and binary forms, with or without 6.\" modification, are permitted provided that the following conditions 7.\" are met: 8.\" 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in 13.\" the documentation and/or other materials provided with the 14.\" distribution. 15.\" 3. Neither the name of The DragonFly Project nor the names of its 16.\" contributors may be used to endorse or promote products derived 17.\" from this software without specific, prior written permission. 18.\" 19.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 22.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 23.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 24.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 25.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 26.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 27.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 29.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 30.\" SUCH DAMAGE. 31.\" 32.Dd November 17, 2011 33.Dt DFREGRESS 8 34.Os 35.Sh NAME 36.Nm dfregress 37.Nd an automation test driver and framework 38.Sh SYNOPSIS 39.Nm 40.Op Fl o Ar output_plist 41.Op Fl t Ar testcase_dir 42.Op Fl p Ar prepost_dir 43.Ar runlist_file 44.Nm 45.Fl h 46.Sh DESCRIPTION 47.Nm 48is a regression test framework and automation driver. 49It executes a series of testcases as specified in the 50.Ar runlist_file 51and collects the results. 52.Pp 53The path to the testcase collection is specified via the 54.Ar testcase_dir 55argument. 56If this argument is not specified, the default is assumed to be the 57same directory as that of the runlist. 58For example if the used runlist is 59.Pa /usr/src/test/testcases/sample.run 60the default testcase directory, unless otherwise specified, will be 61.Pa /usr/src/test/testcases . 62.Pp 63Similarly the path to the pre- and post commands is 64specified via 65.Ar prepost_dir . 66The 67.Ar prepost_dir 68only needs to be specified if the runlist contains custom pre or 69post commands. 70.Pp 71The output is saved in plist format to the 72.Ar output_plist 73file, or if none is specified, to a file with the same base name as 74the runlist, but in the current working directory and with a 75.Pa .plist 76extension. 77For example if the used runlist is 78.Pa /usr/src/test/testcases/sample.run 79the default output, unless otherwise specified, will be 80.Pa ./sample.plist . 81Other tools, known as frontends, can parse the plist output into 82an easier to read form. 83.Pp 84The following is a summary of the optional parameters: 85.Bl -tag -width indent 86.It Fl o Ar output_plist 87Specifies the file to which to write the results. 88The resulting file, 89.Ar output_plist , 90will be in plist format. 91.It Fl t Ar testcase_dir 92Specifies the directory in which to find the testcases specified in the runlist. 93.It Fl p Ar prepost_dir 94Specifies the directory in which to find the pre- and post commands used 95in the runlist. 96This argument is only required when the runlist uses custom pre- or post 97commands. 98.It Fl h 99Prints a short synopsis. 100.El 101.Sh RUNLIST SYNTAX 102Testcases are specified one testcase per line, with whitespace separated 103values. 104Empty lines and lines beginning with a 105.Dq # 106are ignored. 107.Pp 108Runlist lines are of the following format: 109.Bd -literal -offset indent 110.Ic testcase type options Cm arguments ... 111.Ed 112.Pp 113The 114.Ic testcase 115part needs to be a relative path from the testcase base directory specified 116by the 117.Fl t 118argument to the resulting (after building the testcase) testcase executable. 119The testcase will be executed with the 120.Cm arguments 121passed as command line arguments to it. 122.Pp 123Valid types are 124.Ic userland , 125.Ic kernel 126and 127.Ic buildonly : 128.Bl -tag -width indent -offset indent 129.It Ic userland 130A userland testcase is a normal userland executable that returns a non-zero 131exit value on test failure. 132.It Ic kernel 133A kernel testcase is run with the kernel test bridge inside the kernel. 134.It Ic buildonly 135A buildonly test passes when the build for the given testcase succeeds. 136.El 137.Pp 138Valid options are 139.Ic defaults , 140.Ic make , 141.Ic timeout , 142.Ic nobuild , 143.Ic runas , 144.Ic intpre , 145.Ic intpost , 146.Ic pre , 147and 148.Ic post : 149.Bl -tag -width indent -offset indent 150.It Ic defaults 151This option does nothing. 152All default options are applied. 153.It Ic make Ar make_command 154Uses 155.Ar make_command 156instead of 157.Xr make 1 158to build the testcase. 159.It Ic timeout Ar timeout 160Sets the timeout for the testcase after which the testcase will be aborted to 161.Ar timeout 162seconds. 163.It Ic nobuild 164If this option is set, the build stage and cleanup stage of the test case 165are not run. 166.It Ic runas Ar uid 167Runs the testcase as the user identified by the given 168.Ar uid . 169.It Ic intpre 170Executes the testcase command with the argument 171.Dq pre 172during the pre-run command stage. 173.It Ic intpost 174Executes the testcase command with the argument 175.Dq post 176during the post-run command stage. 177.It Ic pre Ar precmd 178Executes the command specified by 179.Ar precmd 180during the pre-run command stage. 181.It Ic pre Ar postcmd 182Executes the command specified by 183.Ar postcmd 184during the post-run command stage. 185.El 186.Sh TESTCASE EXECUTION 187.Bl -enum -width 3n -offset indent 188.It 189.Tn "CHDIR" 190to the testcase directory. 191If it fails, the result will be set to 192.Dv RESULT_PREFAIL 193and the 194.Ar sysbuf 195buffer will provide further details. 196.It 197.Tn "BUILD" 198the testcase using the command specified by the 199.Ic make 200option or, if not specified, 201.Xr make 1 , 202unless the 203.Ic nobuild 204option was specified. 205If there is an internal driver error, the result will be set to 206.Dv RESULT_PREFAIL , 207the next step to be performed will be 208.Tn "CLEANUP" 209and the 210.Ar sysbuf 211buffer will provide further details. 212If no internal error occurs, the 213.Ar buildbuf 214will contain the build log. 215.It 216.Tn "RUN PRE COMMAND" 217if 218.Ic intpre 219or 220.Ic pre 221are set. 222If there is an internal driver error, the result will be set to 223.Dv RESULT_PREFAIL , 224the next step to be performed will be 225.Tn "CLEANUP" 226and the 227.Ar sysbuf 228buffer will provide further details. 229If the pre command has a non-zero exit value, the result will be set to 230.Dv RESULT_PREFAIL 231and the 232next step to be performed will be 233.Tn "CLEANUP" . 234In this case and in the case where the command succeeds, the 235.Ar precmd_buf 236will contain the execution log. 237.It 238.Tn "RUN TESTCASE" 239depending on type: 240.Bl -tag -width 2n -compact 241.It "buildonly" 242testcases will get their result set to 243.Dv RESULT_PASS 244at this point, since the build must have succeeded already. 245.It "userland" 246testcases will get executed in a separate child process, possibly as a 247different user, depending on whether the 248.Ic runas 249option was specified. 250The result will be set to 251.Dv RESULT_TIMEOUT 252if the timeout is reached before the testcase finishes, 253.Dv RESULT_SIGNALLED 254if the testcase dies because of an unhandled signal (often due to crashing), 255.Dv RESULT_NOTRUN 256if the testcase could not be executed, 257.Dv RESULT_FAIL 258if the testcase completes but returns failure or 259.Dv RESULT_PASS 260if the testcase completes and returns success. 261.It "kernel" 262testcases will be executed in kernel space by loading a given module and 263running the testcase entry point function. 264The result will be set to 265.Dv RESULT_NOTRUN 266if the testcase could not be executed. 267Otherwise the result will be set according to the kernel test case to 268one of 269.Dv RESULT_TIMEOUT , 270.Dv RESULT_FAIL , 271or 272.Dv RESULT_PASS . 273.El 274The output will be logged separately for stdout and stderr to the 275.Ar stdout_buf 276and 277.Ar stderr_buf 278respectively. 279If the result was 280.Dv RESULT_NOTRUN 281the 282.Ar sysbuf 283will contain more information. 284.It 285.Tn "RUN POST COMMAND" 286if 287.Ic intpost 288or 289.Ic post 290are set. 291If there is an internal driver error, the result will be set to 292.Dv RESULT_POSTFAIL , 293the next step to be performed will be 294.Tn "CLEANUP" 295and the 296.Ar sysbuf 297buffer will provide further details. 298If the post command has a non-zero exit value, the result will be set to 299.Dv RESULT_POSTFAIL 300and the 301next step to be performed will be 302.Tn "CLEANUP" . 303In this case and in the case where the command succeeds, the 304.Ar postcmd_buf 305will contain the execution log. 306.It 307.Tn "CLEANUP" 308the testcase execution using the command specified by the 309.Ic make 310option or, if not specified, 311.Xr make 1 312with the parameter 313.Dq clean , 314unless the 315.Ic nobuild 316option was specified. 317If there is an internal driver error the 318.Ar sysbuf 319buffer will contain more information. 320If no internal error occurs, the 321.Ar cleanu_pbuf 322will contain the cleanup log. 323.El 324.Sh FRONTENDS 325The output of 326.Nm 327is in the Apple plist serialized object format. 328This format can be easily parsed by using 329.Xr proplib 3 330when using C. 331Ruby and Python also have parsers for the plist format. 332.Pp 333A frontend for 334.Nm 335parses the intermediate output plist into a more easily readable format 336such as plain text or websites. 337.Pp 338By default 339.Nm 340ships only with the 341.Xr dfr2text 8 342text-based frontend. 343.Sh HOW TO WRITE A TESTCASE 344A userland testcase is a simple program that prints some relevant 345information to stdout and stderr, both of which are captured by the test 346driver, and returns an exit value of 0 if the test passed, or any other 347non-zero exit value to signal a failure. 348The exact exit value is recorded by 349.Nm . 350All signals/exceptions not explicitly caught by the testcase will abort 351the execution of the testcase and the result will be set appropriately and 352the signal number will be recorded. 353.Pp 354A kernel testcase is a simple kernel module that defines two methods: 355a 356.Fn run 357method as well as an optional 358.Fn abort 359method. 360The 361.Fn run 362method will be run from a separate kernel thread. 363The testcase will need to call 364.Xr tbridge 9 365functions to record output and to notify of testcase completion. 366Refer to the 367.Xr tbridge 9 368manual page for more details. 369.Pp 370For all testcase types the build stage is common. 371Every testcase should either have the 372.Ic nobuild 373option set, or have a Makefile or similar in its directory. 374By default 375.Nm 376assumes it is a standard 377.Xr make 1 378Makefile. 379If that is not the case, the 380.Ic build 381option needs to be adjusted accordingly. 382.Sh GENERAL ADVICE ON WRITING TESTCASES 383Test only one thing at a time, it is not good practice to test multiple 384things in the same testcase as it makes it less obvious what's going on. 385.Pp 386Keep it short, simple and well documented on what the requirements are, 387what the preconditions need to be, what exactly is being tested, ideally 388with a reference to a particular bug if that exists, and most importantly 389what the expected outcomes are. 390.Pp 391Make sure your testcase doesn't depend on any other being run previously 392as well as that it won't hinder any other testcase from running. 393This effectively means that your testcase should make no assumptions as 394to what has been run previously unless it has a registered pre-command 395and that the system should be left as found before your testcase. 396.Sh EXAMPLES 397An example runlist can be found under 398.Pa test/testcases/sample.run . 399.Pp 400Several example testcases for both userland and kernel are under 401.Pa test/testcases/sample . 402.Sh SEE ALSO 403.Xr dfr2text 8 , 404.Xr tbridge 9 405.Sh HISTORY 406The 407.Nm 408utility appeared in 409.Dx 2.13 . 410.Sh AUTHORS 411.An Alex Hornung 412