1To run the tests: 2 3 $ make check 4 5Note that if your /bin/sh doesn't support shell functions, you'll 6have to try something like this, where "/bin/sh5" is replaced by the 7pathname of a shell which handles normal shell functions: 8 9 $ make SHELL=/bin/sh5 check 10 11Also note that you must be logged in as a regular user, not root. 12 13WARNING: This test can take quite a while to run, esp. if your 14disks are slow or over-loaded. 15 16The tests work in /tmp/cvs-sanity (which the tests create) by default. 17If for some reason you want them to work in a different directory, you 18can set the TESTDIR environment variable to the desired location 19before running them. 20 21The tests use a number of tools (awk, expr, id, tr, etc.) that are not 22required for running CVS itself. In most cases, the standard vendor- 23supplied versions of these tools work just fine, but there are some 24exceptions -- expr in particular is heavily used and many vendor 25versions are deficient in one way or another. Note that some vendors 26provide multiple versions of tools (typically an ancient, traditional 27version and a new, standards-conforming version), so you may already 28have a usable version even if the default version isn't. If you don't 29have a suitable tool, you can probably get one from the GNU Project (see 30http://www.gnu.org). expr and id are both part of the GNU shellutils 31package, tr is part of the GNU textutils package, and awk is part of the 32GNU gawk package. The test script tries to verify that the tools exist 33and are usable; if not, it tries to find the GNU versions and use them 34instead. If it can't find the GNU versions either, it will print an 35error message and, depending on the severity of the deficiency, it may 36exit. 37 38If there is some unexpected output, that is a failure which can be 39somewhat hard to track down. Finding out which test is producing the 40output is not always easy. The newer tests (that is, ones using 41dotest*) will not have this problem, but there are many old tests 42which have not been converted. 43 44If running the tests produces the output "FAIL:" followed by the name 45of the test that failed, then the details on the failure are in the 46file check.log. If it says "exit status is " followed by a number, 47then the exit status of the command under test was not what the test 48expected. If it says "** expected:" followed by a regular expression 49followed by "** got:" followed by some text, then the regular 50expression is the output which the test expected, and the text is the 51output which the command under test actually produced. In some cases 52you'll have to look closely to see how they differ. 53 54If output from "make remotecheck" is out of order compared to what is 55expected (for example, 56 57 a 58 b 59 cvs foo: this is a demo 60 61is expected and 62 63 a 64 cvs foo: this is a demo 65 b 66 67is output), this is probably a well-known bug in the CVS server 68(search for "out-of-order" in src/server.c for a comment explaining 69the cause). It is a real pain in running the testsuite, but if you 70are lucky and/or your machine is fast and/or lightly loaded, you won't 71run into it. Running the tests again might succeed if the first run 72failed in this manner. 73 74For more information on what goes in check.log, and how the tests are 75run in general, you'll have to read sanity.sh. Depending on just what 76you are looking for, and how familiar you are with the Bourne shell 77and regular expressions, it will range from relatively straightforward 78to obscure. 79 80If you choose to submit a bug report based on tests failing, be 81aware that, as with all bug reports, you may or may not get a 82response, and your odds might be better if you include enough 83information to reproduce the bug, an analysis of what is going 84wrong (if you have the time to provide one), etc. The check.log 85file is the first place to look. 86 87ABOUT STDOUT AND STDERR 88*********************** 89 90The sanity.sh test framework combines stdout and stderr and for tests 91to pass requires that output appear in the given order. Some people 92suggest that ordering between stdout and stderr should not be 93required, or to put it another way, that the out-of-order bug referred 94to above, and similar behaviors, should be considered features, or at 95least tolerable. The reasoning behind the current behavior is that 96having the output appear in a certain order is the correct behavior 97for users using CVS interactively--that users get confused if the 98order is unpredictable. 99 100ABOUT TEST FRAMEWORKS 101********************* 102 103People periodically suggest using dejagnu or some other test 104framework. A quick look at sanity.sh should make it clear that there 105are indeed reasons to be dissatisfied with the status quo. Ideally a 106replacement framework would achieve the following: 107 1081. Widely portable, including to a wide variety of unices, NT, Win95, 109OS/2, VMS, probably DOS and Win3, etc. 110 1112. Nicely match extended regular expressions of unlimited length. 112 1133. Be freely redistributable, and if possible already the kind of 114thing people might have already installed. The harder it is to get 115and install the framework, the less people will run the tests. 116 117The various contenders are: 118 119* Bourne shell and GNU expr (the status quo). Falls short on #1 120(we've only tried unix and NT, although MKS might help with other DOS 121mutants). #3 is pretty good (the main dependency is GNU expr which is 122fairly widely available). 123 124* Bourne shell with a new regexp matcher we would distribute with 125CVS. This means maintaining a regexp matcher and the makefiles which 126go with it. Not clearly a win over Bourne shell and GNU expr. 127 128* Bourne shell, and use sed to remove variable portions of output, and 129thus produce a form that can be compared with cmp or diff (this 130sidesteps the need for a full regular expression matcher as mentioned 131in #2 above). The C News tests are said to work this way. This would 132appear to rely on variable portions of output having a certain syntax 133and might spuriously recognize them out of context (this issue needs 134more investigation; it isn't clear how big a problem it is in 135practice). Same portability issues as the other choices based on the 136Bourne shell. 137 138* Dejagnu. This is overkill; most of dejagnu is either unnecessary 139(e.g. libraries for communicating with target boards) or undesirable 140(e.g. the code which stats every file in sight to find the tests). On 141the plus side, dejagnu is probably closer than any of the other 142choices to having everything which is needed already there. 143 144* Write our own small framework directly in tcl and distribute with 145CVS. The tests would look much like dejagnu tests, but we'd avoid the 146unnecessary baggage. The only dependency would be on tcl (that is, 147wish). 148 149* perl or python or <any other serious contenders here?> 150 151It is worth thinking about how to: 152 153a. include spaces in arguments which we pass to the program under 154test (sanity.sh dotest cannot do this; see test rcs-9 for a 155workaround). 156 157b. pass stdin to the program under test (sanity.sh, again, handles 158this by bypassing dotest). 159 160c. have a send-expect type dialog with the program under test 161 (e.g. see server-7 or pserver-4 which want to talk the CVS 162 protocol, or the many tests which need to answer the prompt of "cvs 163 release", e.g. deep-5). 164