README_regress
1Background:
2
3This is to perform regression testing, which is to say, make sure the
4software works the same way that it used to. This is not to test new
5features (see "modify an existing tree" for that).
6
7When code has been changed for a certain program that is involved in a
8regression testing tree, go to that tree, run the test, and make sure
9that the results have not changed (in any bad way). This is meant to
10be fast and easy.
11
12The test consists of command scripts that were written before. The
13scripts are executed, and the output is compared with the output of
14an older execution. The should ususally be identical.
15
16---------------------------------------------------------------------------
17
18To perform regression testing, one needs the data (via 'make regress_data').
19Given that, only the @test and @show_diffs are needed (sample commands are
20shown with a preceeding prompt, '> ').
21
22 1. Run the regression testing script. It goes through the steps:
23
24 a. Create a "results" directory, including a timestamp.
25 b. Copy all "data" files into it.
26 c. 'cd' into the "results" directory.
27 d. For each command script in the commands directory, execute it,
28 storing all output text into a file called e."command name".
29
30 > ./@test
31
32
33 2. To compare against an older execution, run the @show_diffs script.
34 It goes through the steps:
35
36 a. Make a list of the "results" directories.
37 b. Create a new "diffs" directory.
38 c. Compare files between the newest and oldest "results" dirs:
39 - cd into newest "results" directory
40 - foreach file (*)
41 diff $file ../other.result/$file >& ../diffs/d.$file
42
43 > ./@show_diffs
44
45
46 3. Look at the sizes of the diff files, or the contents (only
47 expected differences should exist).
48
49 > ls -l diffs
50 > cat diffs/*
51