1.\" 2.\" Copyright (c) 2011 3.\" The DragonFly Project. All rights reserved. 4.\" 5.\" Redistribution and use in source and binary forms, with or without 6.\" modification, are permitted provided that the following conditions 7.\" are met: 8.\" 9.\" 1. Redistributions of source code must retain the above copyright 10.\" notice, this list of conditions and the following disclaimer. 11.\" 2. Redistributions in binary form must reproduce the above copyright 12.\" notice, this list of conditions and the following disclaimer in 13.\" the documentation and/or other materials provided with the 14.\" distribution. 15.\" 3. Neither the name of The DragonFly Project nor the names of its 16.\" contributors may be used to endorse or promote products derived 17.\" from this software without specific, prior written permission. 18.\" 19.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 22.\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 23.\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 24.\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, 25.\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 26.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED 27.\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28.\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT 29.\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 30.\" SUCH DAMAGE. 31.\" 32.Dd November 27, 2020 33.Dt DFREGRESS 8 34.Os 35.Sh NAME 36.Nm dfregress 37.Nd an automation test driver and framework 38.Sh SYNOPSIS 39.Nm 40.Op Fl o Ar output_plist 41.Op Fl t Ar testcase_dir 42.Op Fl p Ar prepost_dir 43.Ar runlist_file 44.Nm 45.Fl h 46.Sh DESCRIPTION 47.Nm 48is a regression test framework and automation driver. 49It executes a series of testcases as specified in the 50.Ar runlist_file 51and collects the results. 52.Pp 53The path to the testcase collection is specified via the 54.Ar testcase_dir 55argument. 56If this argument is not specified, the default is assumed to be the 57same directory as that of the runlist. 58For example if the used runlist is 59.Pa /usr/src/test/testcases/sample.run 60the default testcase directory, unless otherwise specified, will be 61.Pa /usr/src/test/testcases . 62.Pp 63Similarly the path to the pre- and post commands is 64specified via 65.Ar prepost_dir . 66The 67.Ar prepost_dir 68only needs to be specified if the runlist contains custom pre or 69post commands. 70.Pp 71The output is saved in plist format to the 72.Ar output_plist 73file, or if none is specified, to a file with the same base name as 74the runlist, but in the current working directory and with a 75.Pa .plist 76extension. 77For example if the used runlist is 78.Pa /usr/src/test/testcases/sample.run 79the default output, unless otherwise specified, will be 80.Pa ./sample.plist . 81Other tools, known as frontends, can parse the plist output into 82an easier to read form. 83.Pp 84The following is a summary of the optional parameters: 85.Bl -tag -width indent 86.It Fl o Ar output_plist 87Specifies the file to which to write the results. 88The resulting file, 89.Ar output_plist , 90will be in plist format. 91.It Fl t Ar testcase_dir 92Specifies the directory in which to find the testcases specified in the runlist. 93.It Fl p Ar prepost_dir 94Specifies the directory in which to find the pre- and post commands used 95in the runlist. 96This argument is only required when the runlist uses custom pre- or post 97commands. 98.It Fl h 99Prints a short synopsis. 100.El 101.Sh RUNLIST SYNTAX 102Testcases are specified one testcase per line, with whitespace separated 103values. 104Empty lines and lines beginning with a 105.Dq # 106are ignored. 107.Pp 108Runlist lines are of the following format: 109.Bd -literal -offset indent 110.Ic testcase type options Cm arguments ... 111.Ed 112.Pp 113The 114.Ic testcase 115part needs to be a relative path from the testcase base directory specified 116by the 117.Fl t 118argument to the resulting (after building the testcase) testcase executable. 119The testcase will be executed with the 120.Cm arguments 121passed as command line arguments to it. 122.Pp 123Valid types are 124.Ic userland , 125.Ic kernel 126and 127.Ic buildonly : 128.Bl -tag -width indent -offset indent 129.It Ic userland 130A userland testcase is a normal userland executable that returns a non-zero 131exit value on test failure. 132.It Ic kernel 133A kernel testcase is run with the kernel test bridge inside the kernel. 134.It Ic buildonly 135A buildonly test passes when the build for the given testcase succeeds. 136.El 137.Pp 138Valid options are 139.Ic defaults , 140.Ic interpreter , 141.Ic make , 142.Ic rc , 143.Ic timeout , 144.Ic nobuild , 145.Ic runas , 146.Ic intpre , 147.Ic intpost , 148.Ic pre , 149and 150.Ic post : 151.Bl -tag -width indent -offset indent 152.It Ic defaults 153This option does nothing. 154All default options are applied. 155.It Ic interpreter Ar command 156Uses 157.Ar command 158for running the test case instead of calling it directly. 159This is useful when you have tests that you want to call with a specific shell 160like 161.Xr sh 1 , 162and there is no shebang in the testcase file, there are no 163execution permissions and you don't want or cannot modify the testcase. 164.It Ic make Ar make_command 165Uses 166.Ar make_command 167instead of 168.Xr make 1 169to build the testcase. 170.It Ic rc Ar code 171Sets the return code that will be considered successful. 172By default 0 is considered success. 173.It Ic timeout Ar timeout 174Sets the timeout for the testcase after which the testcase will be aborted to 175.Ar timeout 176seconds. 177.It Ic nobuild 178If this option is set, the build stage and cleanup stage of the test case 179are not run. 180.It Ic runas Ar uid 181Runs the testcase as the user identified by the given 182.Ar uid . 183.It Ic intpre 184Executes the testcase command with the argument 185.Dq pre 186during the pre-run command stage. 187.It Ic intpost 188Executes the testcase command with the argument 189.Dq post 190during the post-run command stage. 191.It Ic pre Ar precmd 192Executes the command specified by 193.Ar precmd 194during the pre-run command stage. 195.It Ic pre Ar postcmd 196Executes the command specified by 197.Ar postcmd 198during the post-run command stage. 199.El 200.Sh TESTCASE EXECUTION 201.Bl -enum -width 3n -offset indent 202.It 203.Tn "CHDIR" 204to the testcase directory. 205If it fails, the result will be set to 206.Dv RESULT_PREFAIL 207and the 208.Ar sysbuf 209buffer will provide further details. 210.It 211.Tn "BUILD" 212the testcase using the command specified by the 213.Ic make 214option or, if not specified, 215.Xr make 1 , 216unless the 217.Ic nobuild 218option was specified. 219If there is an internal driver error, the result will be set to 220.Dv RESULT_PREFAIL , 221the next step to be performed will be 222.Tn "CLEANUP" 223and the 224.Ar sysbuf 225buffer will provide further details. 226If no internal error occurs, the 227.Ar buildbuf 228will contain the build log. 229.It 230.Tn "RUN PRE COMMAND" 231if 232.Ic intpre 233or 234.Ic pre 235are set. 236If there is an internal driver error, the result will be set to 237.Dv RESULT_PREFAIL , 238the next step to be performed will be 239.Tn "CLEANUP" 240and the 241.Ar sysbuf 242buffer will provide further details. 243If the pre command has a non-zero exit value, the result will be set to 244.Dv RESULT_PREFAIL 245and the 246next step to be performed will be 247.Tn "CLEANUP" . 248In this case and in the case where the command succeeds, the 249.Ar precmd_buf 250will contain the execution log. 251.It 252.Tn "RUN TESTCASE" 253depending on type: 254.Bl -tag -width 2n -compact 255.It "buildonly" 256testcases will get their result set to 257.Dv RESULT_PASS 258at this point, since the build must have succeeded already. 259.It "userland" 260testcases will get executed in a separate child process, possibly as a 261different user, depending on whether the 262.Ic runas 263option was specified. 264The result will be set to 265.Dv RESULT_TIMEOUT 266if the timeout is reached before the testcase finishes, 267.Dv RESULT_SIGNALLED 268if the testcase dies because of an unhandled signal (often due to crashing), 269.Dv RESULT_NOTRUN 270if the testcase could not be executed, 271.Dv RESULT_FAIL 272if the testcase completes but returns failure or 273.Dv RESULT_PASS 274if the testcase completes and returns success. 275.It "kernel" 276testcases will be executed in kernel space by loading a given module and 277running the testcase entry point function. 278The result will be set to 279.Dv RESULT_NOTRUN 280if the testcase could not be executed. 281Otherwise the result will be set according to the kernel test case to 282one of 283.Dv RESULT_TIMEOUT , 284.Dv RESULT_FAIL , 285or 286.Dv RESULT_PASS . 287.El 288The output will be logged separately for stdout and stderr to the 289.Ar stdout_buf 290and 291.Ar stderr_buf 292respectively. 293If the result was 294.Dv RESULT_NOTRUN 295the 296.Ar sysbuf 297will contain more information. 298.It 299.Tn "RUN POST COMMAND" 300if 301.Ic intpost 302or 303.Ic post 304are set. 305If there is an internal driver error, the result will be set to 306.Dv RESULT_POSTFAIL , 307the next step to be performed will be 308.Tn "CLEANUP" 309and the 310.Ar sysbuf 311buffer will provide further details. 312If the post command has a non-zero exit value, the result will be set to 313.Dv RESULT_POSTFAIL 314and the 315next step to be performed will be 316.Tn "CLEANUP" . 317In this case and in the case where the command succeeds, the 318.Ar postcmd_buf 319will contain the execution log. 320.It 321.Tn "CLEANUP" 322the testcase execution using the command specified by the 323.Ic make 324option or, if not specified, 325.Xr make 1 326with the parameter 327.Dq clean , 328unless the 329.Ic nobuild 330option was specified. 331If there is an internal driver error the 332.Ar sysbuf 333buffer will contain more information. 334If no internal error occurs, the 335.Ar cleanu_pbuf 336will contain the cleanup log. 337.El 338.Sh FRONTENDS 339The output of 340.Nm 341is in the Apple plist serialized object format. 342This format can be easily parsed by using 343.Xr proplib 3 344when using C. 345Ruby and Python also have parsers for the plist format. 346.Pp 347A frontend for 348.Nm 349parses the intermediate output plist into a more easily readable format 350such as plain text or websites. 351.Pp 352By default 353.Nm 354ships only with the 355.Xr dfr2text 8 356text-based frontend. 357.Sh HOW TO WRITE A TESTCASE 358A userland testcase is a simple program that prints some relevant 359information to stdout and stderr, both of which are captured by the test 360driver, and returns an exit value of 0 if the test passed, or any other 361non-zero exit value to signal a failure. 362The exact exit value is recorded by 363.Nm . 364All signals/exceptions not explicitly caught by the testcase will abort 365the execution of the testcase and the result will be set appropriately and 366the signal number will be recorded. 367.Pp 368A kernel testcase is a simple kernel module that defines two methods: 369a 370.Fn run 371method as well as an optional 372.Fn abort 373method. 374The 375.Fn run 376method will be run from a separate kernel thread. 377The testcase will need to call 378.Xr tbridge 9 379functions to record output and to notify of testcase completion. 380Refer to the 381.Xr tbridge 9 382manual page for more details. 383.Pp 384For all testcase types the build stage is common. 385Every testcase should either have the 386.Ic nobuild 387option set, or have a Makefile or similar in its directory. 388By default 389.Nm 390assumes it is a standard 391.Xr make 1 392Makefile. 393If that is not the case, the 394.Ic build 395option needs to be adjusted accordingly. 396.Sh GENERAL ADVICE ON WRITING TESTCASES 397Test only one thing at a time, it is not good practice to test multiple 398things in the same testcase as it makes it less obvious what's going on. 399.Pp 400Keep it short, simple and well documented on what the requirements are, 401what the preconditions need to be, what exactly is being tested, ideally 402with a reference to a particular bug if that exists, and most importantly 403what the expected outcomes are. 404.Pp 405Make sure your testcase doesn't depend on any other being run previously 406as well as that it won't hinder any other testcase from running. 407This effectively means that your testcase should make no assumptions as 408to what has been run previously unless it has a registered pre-command 409and that the system should be left as found before your testcase. 410.Sh EXAMPLES 411An example runlist can be found under 412.Pa test/testcases/sample.run . 413.Pp 414Several example testcases for both userland and kernel are under 415.Pa test/testcases/sample . 416.Sh SEE ALSO 417.Xr dfr2text 8 , 418.Xr tbridge 9 419.Sh HISTORY 420The 421.Nm 422utility appeared in 423.Dx 2.13 . 424.Sh AUTHORS 425.An Alex Hornung 426