1.\" $OpenBSD: bsd.regress.mk.5,v 1.26 2023/11/13 10:11:41 op Exp $ 2.\" 3.\" Copyright (c) 2002 Anil Madhavapeddy 4.\" Copyright (c) 2000 Marc Espie 5.\" 6.\" All rights reserved. 7.\" 8.\" Redistribution and use in source and binary forms, with or without 9.\" modification, are permitted provided that the following conditions 10.\" are met: 11.\" 1. Redistributions of source code must retain the above copyright 12.\" notice, this list of conditions and the following disclaimer. 13.\" 2. Redistributions in binary form must reproduce the above copyright 14.\" notice, this list of conditions and the following disclaimer in the 15.\" documentation and/or other materials provided with the distribution. 16.\" 17.\" THIS SOFTWARE IS PROVIDED BY THE DEVELOPERS ``AS IS'' AND ANY EXPRESS OR 18.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 19.\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 20.\" IN NO EVENT SHALL THE DEVELOPERS BE LIABLE FOR ANY DIRECT, INDIRECT, 21.\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 22.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 23.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 24.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 25.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 26.\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27.\" 28.Dd $Mdocdate: November 13 2023 $ 29.Dt BSD.REGRESS.MK 5 30.Os 31.Sh NAME 32.Nm bsd.regress.mk 33.Nd regression test master Makefile fragment 34.Sh SYNOPSIS 35.Fd .include <bsd.regress.mk> 36.Sh DESCRIPTION 37.Nm 38holds the standard routines used by the source regression tests. 39Some variables and targets are for its internal use only. 40The rest are documented here. 41.Pp 42Since this file also includes 43.Nm bsd.prog.mk , 44all of the usual 45.Ox 46Makefile variables may be used to build the regression 47test programs. 48.Sh TARGETS 49.Bl -tag -width regress 50.It Cm depend 51Build any dependencies required to carry out the current set 52of regression tests. 53.It Cm regress 54Executes the regression targets defined in the Makefile. 55If a tests fails, the line 56.Qq FAILED 57is printed to the standard output. 58By default, execution stops after the first test failure and 59.Sy make Cm regress 60exits with a non-zero status. 61.It Cm run-regress-* 62Runs an individual regression test. 63If the exit status of the program indicates an error or timeout, 64then a failure is logged, otherwise the test is marked as a success. 65.El 66.Sh VARIABLES 67.Bl -tag -width Ds 68.It Ev REGRESS_CLEANUP 69If this variable is set, it contains a target that is executed 70after all regress targets. 71.It Ev REGRESS_EXPECTED_FAILURES 72Targets which are expected to fail. 73Specifically: sometimes tests are added for features that should be 74supported eventually, but are somewhat complicated to fix right away. 75.It Ev REGRESS_FAIL_EARLY 76If this variable is set to anything but 77.Dq no , 78the 79.Cm regress 80target will abort as soon as a test fails. 81Defaults to 82.Dq yes 83unless 84.Ev REGRESS_LOG 85is set. 86.It Ev REGRESS_LOG 87Points to the fully-qualified path of a file to which regression 88results are appended. 89Defaults to 90.Pa /dev/null . 91If set to any other path, 92.Ev REGRESS_FAIL_EARLY 93defaults to 94.Dq no . 95.It Ev REGRESS_ROOT_TARGETS 96Targets for which root access is required to run the test. 97The 98.Ev SUDO 99variable should be used in the test parts that require root. 100See also 101.Ev SUDO . 102.It Ev REGRESS_SETUP 103If this variable is set, it contains a target that is executed 104before each regress target. 105Regress targets depend on this one to enforce setup. 106.It Ev REGRESS_SETUP_ONCE 107If this variable is set, it contains a target that is executed 108once before all regress targets. 109Regress targets depend on a stamp file generated during setup. 110.It Ev REGRESS_SKIP_SLOW 111If this variable is not empty, skip over all the regression 112tests which have been marked as being 'slow' using the 113.Ev REGRESS_SLOW_TARGETS 114variable. 115.It Ev REGRESS_SKIP_TARGETS 116Targets which are unconditionally skipped. 117.It Ev REGRESS_SLOW_TARGETS 118Targets which are defined as 'slow'. 119All of these tests can be skipped by setting the 120.Ev REGRESS_SKIP_SLOW 121variable. 122.It Ev REGRESS_TARGETS 123Targets which are invoked to run the set of regression tests 124for this Makefile. 125Defaults to 126.Cm run-regress-${PROG} . 127.It Ev SUDO 128Location of a command used to switch to root for certain 129test targets which require it. 130See 131.Xr doas 1 . 132.El 133.Pp 134Some variables are intended to be set at runtime in the environment 135or in 136.Xr mk.conf 5 , 137but not in the regress Makefile itself. 138.Sh GUIDELINES 139If an individual test passes, 140.Sy make Ar testname 141should return with an exit status of 0. 142If it fails, it should return with a non-zero exit status. 143.Pp 144If a test cannot be executed because a package is not installed or 145some environment variable is not set, 146.Sy make Ar testname 147should print 148.Qq SKIPPED 149to stdout and exit with status 0. 150To skip everything, implement the 151.Cm regress 152target with a command that prints 153.Qq SKIPPED . 154.Pp 155Some tests may require a special setup on the test machine that has 156to be done manually before testing. 157This requirement has to be documented in the Makefile or in a 158.Pa README 159file. 160The test should find out whether the setup exists before running 161and print 162.Qq SKIPPED 163and exit if it is missing. 164.Pp 165Tests should not fail because an intended feature has not been 166implemented yet. 167To avoid such false failures, a test should show the reason, print 168.Qq DISABLED 169to stdout and exit with status 0. 170.Pp 171Tests must be able to run with an 172.Pa obj 173directory. 174In case the test is not linked to the build or the tester forgot 175to run 176.Sy make Cm obj 177before, this directory or symlink may not exist. 178Then the test should run anyway. 179.Pp 180Tests are executed with 181.Sy make Cm regress , 182but running 183.Sy make Cm all 184or 185.Sy make 186should have the same effect. 187Tests must be runnable by root, and may also succeed when run as a 188regular user. 189Tests must not assume they have a controlling tty, 190to allow them to be run by 191.Xr cron 8 . 192An individual regress test may create a pseudo tty if it needs one. 193.Pp 194Tests should use the binaries installed and the kernel running on 195the local system. 196They may use environment variables to test alternative binaries or 197remote kernels running on other machines. 198In some cases a test may need binaries or libraries or object files 199to be present in 200.Pa /usr/obj/ 201that exist only after 202.Sy make Cm build 203was run in 204.Pa /usr/src/ . 205The test must not assume that they have already been built, but 206should run 207.Sy make 208in the appropriate source directory as a dependency. 209For missing generated source or header files a target called 210.Sy make Cm generated 211is common. 212The 213.Pa /usr/src/ 214tree can be found with a relative path or with the 215.Ev BSDSRCDIR 216variable. 217.Pp 218Tests should generally not set or modify 219.Ev MALLOC_OPTIONS 220unless it is testing specific behaviour that depends on a particular 221flag or combination of flags. 222When running tests, the appropriate sysctl 223.Pq vm.malloc_conf 224could be used instead. 225.Sh SEE ALSO 226.Xr bsd.port.mk 5 227.Sh HISTORY 228The regression system originally came from 229.Nx , 230with many tests added by 231.Ox 232since. 233The current Makefile framework was written by Artur Grabowski 234and Marc Espie for 235.Ox 3.1 . 236