1Testing with QMTest 2=================== 3 4You can use QMTest to test G++. (In the future, it may be possible to 5test other parts of GCC with QMTest as well, but it is not possible 6yet.) 7 8The use of QMTest to run the G++ tests has not been approved as an 9officially supported testing procedure. Therefore, you must run the 10tests using DejaGNU (with "make check-g++") before committing changes 11that affect G++. 12 13QMTest emulates DejaGNU behavior very closely when running the tests. 14 15QMTest has two output modes: a DejaGNU emulation mode and a native 16QMTest mode. 17 18In the DejaGNU mode, you should receive output that is almost exactly 19the same as the DejaGNU output; in particular, you should see the same 20number of passes, failures, etc. When using the DejaGNU-style output, 21QMTest uses the "xfail" indications in the test cases to determine 22which tests are expected to pass and which are expected to fail, and 23presents that information in the same way as DejaGNU. 24 25In the QMTest mode, the number of passes and failures will be 26different from that obtained when using DejaGNU. The reason is that a 27single source file may contain multiple DejaGNU tests. In DejaGNU, 28each line where a diagnostic is expected is considered a separate 29test. Testing for successful compilation and testing for successful 30execution of the generated program are considered separate tests. So, 31a single source file "test.C" could contain, say, seven tests; some of 32which might pass and some of which might fail. 33 34In the QMTest mode, each source file is considered a single test. If 35any of the seven sub-tests fail, the entire test is considered to 36fail. However, QMTest does present information about *why* the test 37failed, so the same information is effectively available. 38 39In the QMTest mode, whether or not a test is expected to fail is 40determined not by an indication in the test, but rather by comparing 41the new results to the results of a previous run. Testing for whether 42a change caused a regression is very simple: run the tests before 43making the change, run them again after making the change, and let 44QMTest compare the results. 45 46The mode chosen only affects the output from QMTest, not how it runs 47the tests or how it stores the data. Therefore, if you choose to run 48in the QMTest mode and later want to get the DejaGNU style output, or 49vice versa, you can do that as described below. 50 51Setting Up 52========== 53 54You must download and install the following software: 55 56- Python 2.2 (or greater) 57 58 See http://www.python.org. 59 60 You may already have Python on your system; in particular, many 61 GNU/Linux systems ship with Python installed. 62 63 Installation instructions are available on the web-site. 64 65- A current version of QMTest. No released version provides all of 66 the functionality required, so you must obtain QMTest from CVS. 67 68 To do that, follow the instructions at: 69 70 http://www.codesourcery.com/qm/qmtest_development 71 72 Installation instructions are available in the file called README 73 after you check out QMTest. 74 75- The "qmtc" and "qmtest_gcc" QMTest support packages. These are 76 available from the same CVS repository as QMTest. For example, to 77 check out "qmtc", do: 78 79 cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \ 80 co qmtc 81 82 You do not have to install these packages; you need only check them 83 out. 84 85Running the Tests 86================= 87 88First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and 89qmtest_gcc support packages: 90 91 export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc 92 93The, run "make qmtest-g++" in the gcc directory of your build tree. 94 95Here are some more advanced usage instructions: 96 971. To run a particular set of tests (rather than all of the tests), 98 use the make variable "QMTEST_GPP_TESTS". For example, 99 100 make QMTEST_GPP_TESTS="g++.dg" qmtest-g++ 101 102 will run only the tests in the g++.dg subdirectory, and: 103 104 make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \ 105 g++.old-deja/g++.other/access2.C" 106 qmtest-g++ 107 108 will run only the two tests indicated. 109 1102. To run qmtest with particular flags, use the make variables 111 "QMTESTFLAGS" and "QMTESTRUNFLAGS". For example: 112 113 make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++ 114 115 will run qmtest like this: 116 117 qmtest -v run -f full ... 118 119 (The "-f full" mode will provide detailed information about each 120 test as it runs.) 121 1223. To run the compiler with particular flags, use QMTESTRUNFLAGS to 123 set the QMTest context variable "CompilerTable.cplusplus_options", 124 like this: 125 126 make \ 127 QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \ 128 qmtest-g++ 129 130 The compiler will then use the "-funroll-loops" switch when 131 compiling. 132 1334. If qmtest is not in your path, you can indicate the full path to 134 QMTest by using the make variable "QMTEST_PATH", like this: 135 136 make QMTEST_PATH=/path/to/qmtest qmtest-g++ 137 1385. To start the QMTest GUI, use: 139 140 make qmtest-gui 141 142 (Note that this will run the program called "mozilla" in your path. 143 If you want to use another browser, you must configure qmtest as 144 described in its manual.) 145 146 Bear in mind that the QMTest GUI is insecure; malicious users with 147 access to your machine may be able to run commands as if they were 148 you. The QMTest GUI only binds to the loopback IP addresss, which 149 provides a measure of security, but not enough for use in untrusted 150 environments. 151 1526. If you have a multiprocessor, you can run the tests in parallel by 153 passing the "-j" option to qmtest: 154 155 make QMTESTRUNFLAGS="-j 4" qmtest-g++ 156 157 will run tests in four threads. (It is also possible to run tests 158 across multiple machines; for more information see the QMTest 159 manual.) 160 1617. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get 162 more detailed information, you can do: 163 164 cd qmtestsuite 165 qmtest summarize g++.qmr g++.dg/abi/bitfield1.C 166 167 to get more information about the commands that were run and the 168 output produced. 169