README
1crontab executes run_nightly
2
3This addition is also helpful
4
5 export PATH=.:/home/regression/bin:$PATH
6
7
8To rebuild the HEAD executable without running any tests
9
10./run_nightly.py --noregression --notestregression
11
12To run regression tests without rebuilding HEAD
13./run_nightly.py --noupdate
14
15
16The structure of running nightly regressions is:
17
18run_nightly (bash script to provide python environment)
19 run_nightly.py
20 build_revision.py build the requested revision (default is HEAD)
21 run_regression.py run that revision of gs on all testfiles with all paramsets
22 test_diff.py analyze checksums: compare this run with recent and with baseline
23 get_baseline_log.py summarize recent changes to the baseline update log
24
25These script use the environment provided by gsconf.py, which reads the file testing.cfg;
26as well as using the command line.
27 run_nightly.py
28 build_revision.py
29 run_regression.py
30
31The data flow:
32
33All directories are relative to gsconf.root (usually /home/regression)
34
35build_revision.py
36 input
37 from svn gs repository
38 output
39 gs build tree, located at
40 gsroot (if revsion==HEAD) -
41 root/gs.<revision> (for specific revisions)
42 gs installation tree (only if revision==HEAD)
43 installtree
44 logs/<revision>.update*,make*,config*,
45
46
47run_regression
48 input
49 comparefiles (usually /home/regression/comparefiles) - run all "appropriate" files (tested by file extension) using gs
50 gsparamsets.py - sets of output devices for gs testing
51 output
52 dailydir/<revision>.db (usually /home/regression/regression/daily) - store db with (testfile-device,checksum)
53 logs/<revision>.regression
54 logs/<revision>.gs*
55
56<recent date> is determined by the existence of the most recent checksum database, before today
57
58test_diff <recent date>
59 input
60 baselinedb (usually /home/regression/regression/baseline/baseline.db)
61 dailydir (usually /home/regression/regression/daily)
62 <previous revision>.db
63 <revision>.db
64 output
65 text report of differences. For all testfile-device entries in this revision's checksum db,
66 if the checksum does not match the previous revision, report as a new difference
67 if the checksum does not match the baseline, report as a difference
68 logs/<revision>.differences
69
70get_baseline_log <recent date>
71 input
72 baseline_log (usually /home/regression/regression/baseline/baseline.log)
73 output
74 text report of updates to the baseline since the specified date
75
76
77Details of checksums
78
79 all checksum db are in "dailydir" (usually /home/regression/regression/daily)
80 checksums are calculated using the python md5 module.
81 these dbs are files of anydb format, with key=testfile-device, value=checksum
82 New differences result from the comparison with a recent db.
83 All differences result from the comparison with the baseline.db
84 Each day (or revision) a new checksum db is created.
85 Each day, the new checksum db is compared with the most recent day's checksum db.
86 Each day the checksum db is also compared with the baseline checksum db (baseline.db).
87 A checksum log can contain 15,000 or more entries (number of test files * number of devices in gsparamsets)
88
89 bugs:
90 checksum logs that are used for comparison are still named by date, and it should be revision only.
91 each day's log is also copied to a db named by revision (but these are not yet used)
92
93 There are tools to examine the checksum dbs
94 dump_checksum.py - normal use. Be aware: you get LOTS of output
95 dump_checksum_raw.py - for debugging
96 dump_checksum_plus.py - specialized
97 collate.py - gather results of several dbs and sort
98
99
100Details of testfiles:
101 all testfiles in the gsconf.comparefiledir are tested. Smaller subsets (in other directories) can be used for testing.
102 The normal situation:
103 For each testfile, there is a set of compressed raster files (one for each device).
104 For each raster file there is an entry in the baseline checksum db.
105 To make the raster file and baseline checksum db up-to-date, use update_baseline.py (--help for command line options)
106
107
108Under the covers:
109 gsconf.py - provide an object with information about the testing environment
110 gstestgs.py - run gs
111 gsparamsets.py - provide sets of output devices (one set for "normal" and one for "pdfwrite")
112 gstestutils.py - utilities
113 gsutil.py - utilities
114 gssum.py - calculate checksums and insert into a checksum db
115 rasterdb.py - create and get compressed raster files
116
117 These scripts do the work of running gs and adding checksums to the daily checksum db:
118
119 gscheck_all.py - run the following - creating test cases for later execution
120
121 gscheck_raster.py - normal
122 for each test file, for each device create a normal test case
123 gscheck_pdfwrite.py - pdfwrite
124 for each test file, for each device create a pdfwrite test case
125
126 gscheck_testfiles.py - test only the files in a specified list
127
128 check_* - scripts to test the other aspects of the code base (dirs, comments, docrefs, source)
129
130
131Repair:
132 regen_baseline.py
133 regenerate all checksums from the raster files (>15,000 files - takes a long time)
134
135 regen_filelist.py
136 regenerate checksums specified in a file list from the raster files (>15,000 files - takes a long time)
137
138
139Tests with larger scope than nightly:
140 run_series.py - run_nightly.py with a specified range of revisions.
141