1To run the tests: 2 3 $ make check 4 5Note that if your /bin/sh doesn't support shell functions, you'll 6have to try something like this, where "/bin/sh5" is replaced by the 7pathname of a shell which handles normal shell functions: 8 9 $ make SHELL=/bin/sh5 check 10 11WARNING: This test can take quite a while to run, esp. if your 12disks are slow or over-loaded. 13 14The tests work in /tmp/cvs-sanity (which the tests create) by default. 15If for some reason you want them to work in a different directory, you 16can set the TESTDIR environment variable to the desired location 17before running them. 18 19You will probably need GNU expr, which is part of the GNU sh-utils 20package. This is just for running the tests; CVS itself doesn't 21require expr. 22 23With CVS 1.10 people also had trouble with the Solaris sort program 24not behaving the way that the testsuite expects (with Solaris, lines 25starting with tabs sort before blank lines). 26 27If there is some unexpected output, that is a failure which can be 28somewhat hard to track down. Finding out which test is producing the 29output is not always easy. The newer tests (that is, ones using 30dotest*) will not have this problem, but there are many old tests 31which have not been converted. 32 33If running the tests produces the output "FAIL:" followed by the name 34of the test that failed, then the details on the failure are in the 35file check.log. If it says "exit status is " followed by a number, 36then the exit status of the command under test was not what the test 37expected. If it says "** expected:" followed by a regular expression 38followed by "** got:" followed by some text, then the regular 39expression is the output which the test expected, and the text is the 40output which the command under test actually produced. In some cases 41you'll have to look closely to see how they differ. 42 43If output from "make remotecheck" is out of order compared to what is 44expected (for example, 45 46 a 47 b 48 cvs foo: this is a demo 49 50is expected and 51 52 a 53 cvs foo: this is a demo 54 b 55 56is output), this is probably a well-known bug in the CVS server 57(search for "out-of-order" in src/server.c for a comment explaining 58the cause). It is a real pain in running the testsuite, but if you 59are lucky and/or your machine is fast and/or lightly loaded, you won't 60run into it. Running the tests again might succeed if the first run 61failed in this manner. 62 63For more information on what goes in check.log, and how the tests are 64run in general, you'll have to read sanity.sh. Depending on just what 65you are looking for, and how familiar you are with the Bourne shell 66and regular expressions, it will range from relatively straightforward 67to obscure. 68 69If you choose to submit a bug report based on tests failing, be 70aware that, as with all bug reports, you may or may not get a 71response, and your odds might be better if you include enough 72information to reproduce the bug, an analysis of what is going 73wrong (if you have the time to provide one), etc. The check.log 74file is the first place to look. 75 76ABOUT STDOUT AND STDERR 77*********************** 78 79The sanity.sh test framework combines stdout and stderr and for tests 80to pass requires that output appear in the given order. Some people 81suggest that ordering between stdout and stderr should not be 82required, or to put it another way, that the out-of-order bug referred 83to above, and similar behaviors, should be considered features, or at 84least tolerable. The reasoning behind the current behavior is that 85having the output appear in a certain order is the correct behavior 86for users using CVS interactively--that users get confused if the 87order is unpredictable. 88 89ABOUT TEST FRAMEWORKS 90********************* 91 92People periodically suggest using dejagnu or some other test 93framework. A quick look at sanity.sh should make it clear that there 94are indeed reasons to be dissatisfied with the status quo. Ideally a 95replacement framework would achieve the following: 96 971. Widely portable, including to a wide variety of unices, NT, Win95, 98OS/2, VMS, probably DOS and Win3, etc. 99 1002. Nicely match extended regular expressions of unlimited length. 101 1023. Be freely redistributable, and if possible already the kind of 103thing people might have already installed. The harder it is to get 104and install the framework, the less people will run the tests. 105 106The various contenders are: 107 108* Bourne shell and GNU expr (the status quo). Falls short on #1 109(we've only tried unix and NT, although MKS might help with other DOS 110mutants). #3 is pretty good (the main dependency is GNU expr which is 111fairly widely available). 112 113* Bourne shell with a new regexp matcher we would distribute with 114CVS. This means maintaining a regexp matcher and the makefiles which 115go with it. Not clearly a win over Bourne shell and GNU expr. 116 117* Bourne shell, and use sed to remove variable portions of output, and 118thus produce a form that can be compared with cmp or diff (this 119sidesteps the need for a full regular expression matcher as mentioned 120in #2 above). The C News tests are said to work this way. This would 121appear to rely on variable portions of output having a certain syntax 122and might spuriously recognize them out of context (this issue needs 123more investigation; it isn't clear how big a problem it is in 124practice). Same portability issues as the other choices based on the 125Bourne shell. 126 127* Dejagnu. This is overkill; most of dejagnu is either unnecessary 128(e.g. libraries for communicating with target boards) or undesirable 129(e.g. the code which stats every file in sight to find the tests). On 130the plus side, dejagnu is probably closer than any of the other 131choices to having everything which is needed already there. 132 133* Write our own small framework directly in tcl and distribute with 134CVS. The tests would look much like dejagnu tests, but we'd avoid the 135unnecessary baggage. The only dependency would be on tcl (that is, 136wish). 137 138* perl or python or <any other serious contenders here?> 139 140It is worth thinking about how to: 141 142a. include spaces in arguments which we pass to the program under 143test (sanity.sh dotest cannot do this; see test rcs-9 for a 144workaround). 145 146b. pass stdin to the program under test (sanity.sh, again, handles 147this by bypassing dotest). 148 149c. have a send-expect type dialog with the program under test 150 (e.g. see server-7 or pserver-4 which want to talk the CVS 151 protocol, or the many tests which need to answer the prompt of "cvs 152 release", e.g. deep-5). 153