1To run the tests: 2 3 $ make check 4 5Note that if your /bin/sh doesn't support shell functions, you'll 6have to try something like this, where "/bin/sh5" is replaced by the 7pathname of a shell which handles normal shell functions: 8 9 $ make SHELL=/bin/sh5 check 10 11Also note that you must be logged in as a regular user, not root. 12 13WARNING: This test can take quite a while to run, esp. if your 14disks are slow or over-loaded. 15 16The tests work in /tmp/cvs-sanity (which the tests create) by default. 17If for some reason you want them to work in a different directory, you 18can set the TESTDIR environment variable to the desired location 19before running them. 20 21The tests use a number of tools (awk, expr, id, tr, etc.) that are not 22required for running CVS itself. In most cases, the standard vendor- 23supplied versions of these tools work just fine, but there are some 24exceptions -- expr in particular is heavily used and many vendor 25versions are deficient in one way or another. Note that some vendors 26provide multiple versions of tools (typically an ancient, traditional 27version and a new, standards-conforming version), so you may already 28have a usable version even if the default version isn't. If you don't 29have a suitable tool, you can probably get one from the GNU Project (see 30http://www.gnu.org). At this writting, expr and id are both part of the 31GNU shellutils package, tr is part of the GNU textutils package, and awk 32is part of the GNU gawk package. The test script tries to verify that 33the tools exist and are usable; if not, it tries to find the GNU 34versions and use them instead. If it can't find the GNU versions 35either, it will print an error message and, depending on the severity of 36the deficiency, it may exit. There are environment variables you can 37set to use a particular version of a tool -- see the test script 38(src/sanity.sh) for details. 39 40Some of the tests use fairly long command lines -- this usually isn't a 41problem, but if you have a very short command line length limit (or a 42lot of environment variables), you may run into trouble. Also, some of 43the tests expect your local timezone to be an integral number of hours 44from UTC -- if you usually use a fractional timezone, use a different 45(integral) timezone when running the tests to avoid spurious failures. 46 47If running the tests produces the output "FAIL:" followed by the name 48of the test that failed, then the details on the failure are in the 49file check.log. If it says "exit status is " followed by a number, 50then the exit status of the command under test was not what the test 51expected. If it says "** expected:" followed by a regular expression 52followed by "** got:" followed by some text, then the regular 53expression is the output which the test expected, and the text is the 54output which the command under test actually produced. In some cases 55you'll have to look closely to see how they differ; the debug_check_log 56script in the contrib directory can assist in this process. 57 58If output from "make remotecheck" is out of order compared to what is 59expected (for example, 60 61 a 62 b 63 cvs foo: this is a demo 64 65is expected and 66 67 a 68 cvs foo: this is a demo 69 b 70 71is output), this is probably a well-known bug in the CVS server 72(search for "out-of-order" in src/server.c for a comment explaining 73the cause). It is a real pain in running the testsuite, but if you 74are lucky and/or your machine is fast and/or lightly loaded, you won't 75run into it. Running the tests again might succeed if the first run 76failed in this manner. 77 78For more information on what goes in check.log, and how the tests are 79run in general, you'll have to read sanity.sh. Depending on just what 80you are looking for, and how familiar you are with the Bourne shell 81and regular expressions, it will range from relatively straightforward 82to obscure. 83 84If you choose to submit a bug report based on tests failing, be 85aware that, as with all bug reports, you may or may not get a 86response, and your odds might be better if you include enough 87information to reproduce the bug, an analysis of what is going 88wrong (if you have the time to provide one), etc. The check.log 89file is the first place to look. 90 91ABOUT STDOUT AND STDERR 92*********************** 93 94The sanity.sh test framework combines stdout and stderr and for tests 95to pass requires that output appear in the given order. Some people 96suggest that ordering between stdout and stderr should not be 97required, or to put it another way, that the out-of-order bug referred 98to above, and similar behaviors, should be considered features, or at 99least tolerable. The reasoning behind the current behavior is that 100having the output appear in a certain order is the correct behavior 101for users using CVS interactively--that users get confused if the 102order is unpredictable. 103 104ABOUT TEST FRAMEWORKS 105********************* 106 107People periodically suggest using dejagnu or some other test 108framework. A quick look at sanity.sh should make it clear that there 109are indeed reasons to be dissatisfied with the status quo. Ideally a 110replacement framework would achieve the following: 111 1121. Widely portable, including to a wide variety of unices, NT, Win95, 113OS/2, VMS, probably DOS and Win3, etc. 114 1152. Nicely match extended regular expressions of unlimited length. 116 1173. Be freely redistributable, and if possible already the kind of 118thing people might have already installed. The harder it is to get 119and install the framework, the less people will run the tests. 120 121The various contenders are: 122 123* Bourne shell and GNU expr (the status quo). Falls short on #1 124(we've only tried unix and NT, although MKS might help with other DOS 125mutants). #3 is pretty good (the main dependency is GNU expr which is 126fairly widely available). 127 128* Bourne shell with a new regexp matcher we would distribute with 129CVS. This means maintaining a regexp matcher and the makefiles which 130go with it. Not clearly a win over Bourne shell and GNU expr. 131 132* Bourne shell, and use sed to remove variable portions of output, and 133thus produce a form that can be compared with cmp or diff (this 134sidesteps the need for a full regular expression matcher as mentioned 135in #2 above). The C News tests are said to work this way. This would 136appear to rely on variable portions of output having a certain syntax 137and might spuriously recognize them out of context (this issue needs 138more investigation; it isn't clear how big a problem it is in 139practice). Same portability issues as the other choices based on the 140Bourne shell. 141 142* Dejagnu. This is overkill; most of dejagnu is either unnecessary 143(e.g. libraries for communicating with target boards) or undesirable 144(e.g. the code which stats every file in sight to find the tests). On 145the plus side, dejagnu is probably closer than any of the other 146choices to having everything which is needed already there. 147 148* Write our own small framework directly in tcl and distribute with 149CVS. The tests would look much like dejagnu tests, but we'd avoid the 150unnecessary baggage. The only dependency would be on tcl (that is, 151wish). 152 153* perl or python or <any other serious contenders here?> 154 155It is worth thinking about how to: 156 157a. include spaces in arguments which we pass to the program under 158test (sanity.sh dotest cannot do this; see test rcs-9 for a 159workaround). 160 161b. pass stdin to the program under test (sanity.sh, again, handles 162this by bypassing dotest). 163 164c. have a send-expect type dialog with the program under test 165 (e.g. see server-7 or pserver-4 which want to talk the CVS 166 protocol, or the many tests which need to answer the prompt of "cvs 167 release", e.g. deep-5). 168 169ABOUT ADDING YOUR OWN TESTS 170*************************** 171 172As stated in the HACKING file, patches are not accepted without documentation 173and tests. Many people seem to be scared off by the large size of the 174sanity.sh script, but it is not really very complicated. 175 176You can probably ignore most of the begining of the script. This section 177just sets some environment variables and finds the tools the script needs to 178run. 179 180There is one main loop you can find by grepping for "The big loop". This loop 181repeatedly calls a case statement where the individual cases are of the form: 182 183 testname) 184 ... 185 ;; 186 187If you add a complete new test be sure to add it into the default list of tests 188(grep for 'tests=' near the begining of the script) as well as the case 189statement. During debugging, be aware that the sanity.sh usage allows for a '-f 190testname' option to continue through the default list "from" a particular test 191as well as interpreting everything in argv past the required options as test 192names to run individual tests. 193 194Within each major test section, individual tests usually look like: 195 196 dotest testname-subtestname "shell command" "optionally multiline regexp" 197 198Tests should always start in $testdir and create a subdirectory to operate in 199and remove their cruft and end back in $testdir. The dotest functions output 200failure messages and exit if the shell command exits with the wrong exit code or 201its stdin/stderr output doesn't match the regexp. There are a few dotest 202variations, most notably dotest_fail for expected non-zero exit codes. 203 204Other than that the script is mostly vanilla Bourne shell. There are a few 205common constructs used for versatility and portability. You can grep for the 206ones I miss, but here are a few important ones. I'm leaving off long 207explanations after the first few since it probably gives you the idea and the 208data is in sanity.sh. 209 210Several variables contain things intended to make a test writer's job easier. 211Note that the boolean variables contain shell commands which return true or 212false when executed and are intended to be used like, 213"if $remote; then ... ; else ... ; fi" 214 215 216 * $testdir = the directory this test is taking place in 217 (CVSROOT=$testdir/cvsroot or CVSROOT=:fork:$testdir/cvsroot) 218 * $testcvs = full path to the cvs executable we are testing 219 * $PLUS = expr dependant uninterpreted '+' since this can vary 220 * $DOTSTAR = expr dependant _interpreted_ .* since some exprs don't match 221 EOL 222 * $username = the username of the user running the tests 223 * $username8 = the first 8 characters of $username, output by some system 224 and CVS commands 225 * $anyusername 226 = regexp to match any valid system or CVS username 227 * $hostname = regexp to match a hostname 228 * $SPROG = regexp to match server progname in CVS error messages. For 229 tests run in local and remote mode, this is usually the 230 string to test for, since some messages can be printed either 231 by the CVS client or CVS server, dependant on the mode. In 232 local mode it will = $CPROG. 233 * $CPROG = regexp to match client progname in CVS error messages. Use 234 this to match error messages known to be printed only from 235 the CVS client. 236 * $remote = ':' (true) or 'false', depending on whether the script is 237 running with a remote CVSROOT 238 * $keep = ':' (true) or 'false'. When set, the first test run will 239 leave any files and directories it created in $testdir and 240 exit when complete. 241 * $servercvs = cvs executable to be run as CVS_SERVER in remote testing 242 * $testcvs_server_support 243 = ':' (true) or 'false', depending whether server support was 244 detected in the ${testcvs} executable. 245 * $tempfile = a regex to match a temporary file name, as generated by 246 the cvs_temp_file function. 247 * $tempname = a regex to match the full path to a temporary file generated 248 by cvs_temp_file (always set to `$TMPDIR/$tempfile'). 249 250And, of course, some characters like '.' in regexps need to be '\' escaped when 251you mean them literally. Some characters may be interpreted by the shell, 252e.g. backquotes and '$', are usually either escaped or replaced with '.'. 253dotest adds the final '$' anchor to the regexp itself and all the expr 254implementations I know of implicitly supply the start anchor ('^'). 255 256If you only make a few mistakes, the work is, of course, still usable, though we 257may send the patch back to you for repair. :) 258