1crontab executes run_nightly 2 3This addition is also helpful 4 5 export PATH=.:/home/regression/bin:$PATH 6 7 8To rebuild the HEAD executable without running any tests 9 10./run_nightly.py --noregression --notestregression 11 12To run regression tests without rebuilding HEAD 13./run_nightly.py --noupdate 14 15 16The structure of running nightly regressions is: 17 18run_nightly (bash script to provide python environment) 19 run_nightly.py 20 build_revision.py build the requested revision (default is HEAD) 21 run_regression.py run that revision of gs on all testfiles with all paramsets 22 test_diff.py analyze checksums: compare this run with recent and with baseline 23 get_baseline_log.py summarize recent changes to the baseline update log 24 25These script use the environment provided by gsconf.py, which reads the file testing.cfg; 26as well as using the command line. 27 run_nightly.py 28 build_revision.py 29 run_regression.py 30 31The data flow: 32 33All directories are relative to gsconf.root (usually /home/regression) 34 35build_revision.py 36 input 37 from svn gs repository 38 output 39 gs build tree, located at 40 gsroot (if revsion==HEAD) - 41 root/gs.<revision> (for specific revisions) 42 gs installation tree (only if revision==HEAD) 43 installtree 44 logs/<revision>.update*,make*,config*, 45 46 47run_regression 48 input 49 comparefiles (usually /home/regression/comparefiles) - run all "appropriate" files (tested by file extension) using gs 50 gsparamsets.py - sets of output devices for gs testing 51 output 52 dailydir/<revision>.db (usually /home/regression/regression/daily) - store db with (testfile-device,checksum) 53 logs/<revision>.regression 54 logs/<revision>.gs* 55 56<recent date> is determined by the existence of the most recent checksum database, before today 57 58test_diff <recent date> 59 input 60 baselinedb (usually /home/regression/regression/baseline/baseline.db) 61 dailydir (usually /home/regression/regression/daily) 62 <previous revision>.db 63 <revision>.db 64 output 65 text report of differences. For all testfile-device entries in this revision's checksum db, 66 if the checksum does not match the previous revision, report as a new difference 67 if the checksum does not match the baseline, report as a difference 68 logs/<revision>.differences 69 70get_baseline_log <recent date> 71 input 72 baseline_log (usually /home/regression/regression/baseline/baseline.log) 73 output 74 text report of updates to the baseline since the specified date 75 76 77Details of checksums 78 79 all checksum db are in "dailydir" (usually /home/regression/regression/daily) 80 checksums are calculated using the python md5 module. 81 these dbs are files of anydb format, with key=testfile-device, value=checksum 82 New differences result from the comparison with a recent db. 83 All differences result from the comparison with the baseline.db 84 Each day (or revision) a new checksum db is created. 85 Each day, the new checksum db is compared with the most recent day's checksum db. 86 Each day the checksum db is also compared with the baseline checksum db (baseline.db). 87 A checksum log can contain 15,000 or more entries (number of test files * number of devices in gsparamsets) 88 89 bugs: 90 checksum logs that are used for comparison are still named by date, and it should be revision only. 91 each day's log is also copied to a db named by revision (but these are not yet used) 92 93 There are tools to examine the checksum dbs 94 dump_checksum.py - normal use. Be aware: you get LOTS of output 95 dump_checksum_raw.py - for debugging 96 dump_checksum_plus.py - specialized 97 collate.py - gather results of several dbs and sort 98 99 100Details of testfiles: 101 all testfiles in the gsconf.comparefiledir are tested. Smaller subsets (in other directories) can be used for testing. 102 The normal situation: 103 For each testfile, there is a set of compressed raster files (one for each device). 104 For each raster file there is an entry in the baseline checksum db. 105 To make the raster file and baseline checksum db up-to-date, use update_baseline.py (--help for command line options) 106 107 108Under the covers: 109 gsconf.py - provide an object with information about the testing environment 110 gstestgs.py - run gs 111 gsparamsets.py - provide sets of output devices (one set for "normal" and one for "pdfwrite") 112 gstestutils.py - utilities 113 gsutil.py - utilities 114 gssum.py - calculate checksums and insert into a checksum db 115 rasterdb.py - create and get compressed raster files 116 117 These scripts do the work of running gs and adding checksums to the daily checksum db: 118 119 gscheck_all.py - run the following - creating test cases for later execution 120 121 gscheck_raster.py - normal 122 for each test file, for each device create a normal test case 123 gscheck_pdfwrite.py - pdfwrite 124 for each test file, for each device create a pdfwrite test case 125 126 gscheck_testfiles.py - test only the files in a specified list 127 128 check_* - scripts to test the other aspects of the code base (dirs, comments, docrefs, source) 129 130 131Repair: 132 regen_baseline.py 133 regenerate all checksums from the raster files (>15,000 files - takes a long time) 134 135 regen_filelist.py 136 regenerate checksums specified in a file list from the raster files (>15,000 files - takes a long time) 137 138 139Tests with larger scope than nightly: 140 run_series.py - run_nightly.py with a specified range of revisions. 141