README
1This is a simple regression testing framework for Arthur.
2
3There are three parts to it.
4
51) datagenerator . which is the main part. it's used to
6 run all the tests, create output png files,
7 measure rendering performance and output it in
8 data.xml file. datagenerator reads in
9 framework.ini to figure out where to look for tests.
10 if the framework.ini file isn't in the current directory
11 you have to specify its location with the -framework
12 parameter. by default datagenerator generates output in the
13 current directory - to specify output directory pass in
14 -output parameter followed by a directory name. if the given
15 directory doesn't exist it will be created.
16
17 it's recommended to always specify -iterations argument (followed
18 by the number). if the argument has been specified datagenerator
19 will try to render all testcases the given amount of times yielding
20 a lot more precise results.
21
22 if one wants to test just a specified engine -engine argument
23 should be specified (followed by the desired engine name).
24
25 individual testsuites can be run with -suite paramenter.
26 individual testcases from testcases can also be reran with
27 -testcase paramenter.
28
29 given files can also be tested individually. if the option -file
30 is specified with a SVG file engines will be tested against just
31 that file and output will be written directly to stdout (not files).
32 it's useful when optimizing one engine with particular case in
33 mind.
34
35 Example command line:
36 ./bin/datagenerator -framework data/framework.ini -output sampleout -iterations 5 -engine [ENGINE]
37
382) htmlgenerator. which is used to generate html pages out of
39 generated output (generated by datagenerator). htmlgenerator also
40 reads in the framework.ini file. so the option -framework also
41 applies. but htmlgenerator takes a mandatory directory name as the
42 second argument. the mandatory directory name is the directory to
43 which you output the files generated by datagenerator (either
44 whatever followed the -output or the current directory)
45
46 htmlgenerator will generate html pages that can be viewed to
47 manually inspect the results.
48
49 Example command line:
50 ./bin/htmlgenerator -framework data/framework.ini sampleout
51
523) performancediff. diffs the performance data between two output
53 directories. so assuming you already have data for all the engines,
54 you change something in an engine. to see the performance
55 difference run
56 ./bin/datagenerator -framework data/framework.ini -output newoutput -iterations 5
57 and then
58 ./bin/performancediff sampleout newoutput
59
60 The performance results will be printed out.
61
62 The two first columns are in ms. The last one is a
63 percentage of difference.
64
65 After the percentage a '+' or a '-' sign followed by a
66 number will be sometimes printed.
67
68 '+' signifies the new results were mathematically better. '-' is of
69 course just the opposite.
70 the number following '+' is the minimum rendering time the engine
71 spent rendering the given file up till this moment.
72 the number following '-' is the maximum rendering time the engine
73 spent rendering the given file up till this moment.
74
75 (if the current results are out of the scope from within maximum and
76 minimum number then the respective sign is printed out.)
77-------------------------------------------------------------------
78
79Note that the generated output directories can be copied from one
80machine to the other. htmlgenerator will generate webpages for all
81engines with valid data.xml files. So if you want add NativeWin32 and
82NativeMac engines you'll have to have someone with a windows and mac
83run datagenerator with "-engine NativeWin32" or "-engine NativeMac"
84options and then sending you the respective output directories.
85