1Testing Erlang/OTP 2================== 3 4Before you start testing you need to have the Erlang release which you 5are going to test in your path. See [$ERL_TOP/HOWTO/INSTALL.md][] for 6instructions on how to build an Erlang release. 7 8Short version 9------------- 10Move to the top directory of the Erlang release you want to test, i.e. 11cd /ldisk/work/otp 12 13 export ERL_TOP=`pwd` 14 ./otp_build setup -a 15 export PATH=`pwd`/bin:$PATH 16 ./otp_build tests 17 cd release/tests/test_server 18 erl -s ts install -s ts run all_tests -s init stop 19 20Where are the tests 21------------------- 22 23There are a lot of tests which test Erlang/OTP (as of 2012 about 12000) and 24they are located in different places throughout the source tree. Below is a list 25of the places where you can expect to find tests: 26 27* $ERL_TOP/lib/AppName/test/ 28* $ERL_TOP/erts/test/ 29* $ERL_TOP/erts/emulator/test/ 30* $ERL_TOP/erts/epmd/test/ 31 32Writing tests 33------------- 34 35All tests are [common_test][] suites and follow the same pattern as all 36[common_test][] suites. However, a couple of corner cases are 37handled by a test wrapper called `ts`. `ts` allows the test writer to put 38`Makefile.src` and `Makefile.first` files in the [data_dir][] and a special 39directory called `all_SUITE_data`. 40 41`Makefile.first` is run before any other Makefile and is typically used to 42generate .hrl files which are needed by other test suites. At the moment only 43the erl_interface tests use this feature and it should remain that way. 44 45`Makefile.src` is configured to a `Makefile` using the `variables` created when 46[configuring the tests][]. These `Makefile`s are later run by the test suite 47to compile whatever platform specific code the tests need to run. 48 49Releasing tests 50--------------- 51 52If you cannot use [ct_run][] in the source tree you have to release the tests 53into a common test directory. The easiest way to do this is to use `otp_build` 54like this: 55 56 export ERL_TOP=`pwd`; ./otp_build tests 57 58This will release all tests in Erlang/OTP to `$ERL_TOP/release/tests/`. If you 59want to change the directory where the tests are released to use the `TESTROOT` 60environmental variable. 61 62In the `$TESTROOT` you should now see *_test folders. These folders contain 63everything needed to test Erlang/OTP and are platform independent; if you are 64testing Erlang on multiple platforms you just have to release on one and copy 65the tests to all other platforms. 66 67### Releasing cross tests 68 69For releasing tests in a cross compilation environment see [$ERL_TOP/HOWTO/INSTALL-CROSS.md][]. 70 71Configuring and Running tests 72----------------------------- 73 74Running tests is done by first navigating to the `$TESTROOT/test_server` folder 75created when you released the tests and then start `erl` in that directory. The 76emulator flags specified will be used in the test runs. For example, if you want 77to test using async threads you have to supply `+A 10` to `erl` when you start it. 78 79To configure and run the tests `ts` is used. `ts` is a wrapper module to 80[common_test][] which takes care of configuration and build issues before 81[common_test][] is started. 82 83`ts` has a lot of special options and functions which can be useful when 84testing Erlang/OTP. For a full listing issue `ts:help()` in the erlang shell. 85 86### Configuring the test environment 87 88Before running released tests you have to install them on the target system. 89Installing the tests is done by calling `ts:install().` in the Erlang shell 90which you intend to test from. `ts:install()` is basically a wrapper to a 91configure script and some Erlang code which figures out what your system looks 92like and what kind of emulator you are testing with. `ts:install()` can also 93take some arguments if necessary, see `ts:help()` for details. 94 95All variables created by `ts:install()` are found in 96`$TESTROOT/test_server/variables`. 97 98### Running the tests 99 100To run all test suites go to `$TESTROOT/test_server` fire up an Erlang shell and type: 101 102 ts:run(). 103 104Note that running all tests will require several hours, so you may want to run 105the test cases for a single application 106 107 ts:run(Application, [batch]). 108 109or even part of the test suite for an application, for example 110 111 ts:run(emulator, bs, [batch]). 112 113to run all test suite modules starting with `bs` (i.e. all modules that test 114the bit syntax). 115 116To run a specific test case in a module, the full name of the module and test 117case must be specified: 118 119 ts:run(emulator, bs_bincomp_SUITE, byte_aligned, [batch]). 120 121Run `ts:help().` for more information. 122 123As of R14B02 it is also possibly to start all tests but the erl_interface tests 124by invoking Common Test directly from the released applications test directory, 125i.e. 126 127 cd $TESTROOT/test_server 128 $ERL_TOP/bin/ct_run -suite ../compiler_test/andor_SUITE -case t_orelse 129 130Running [ct_run][] from the command line still requires you to do the 131`ts:install()` step above. 132 133### Convenience for running tests without the release and configuration steps 134 135It can be convenient to run tests with a single command. This way, one 136do not need to worry about missing to run `make release_tests` after 137changing a test suite. The `make test` command can be used for this 138purpose. The `make test` command works when the current directory 139contains a directory called test and in the root directory of the 140source code tree. 141 142*(Waring)* Some test cases do not run correctly or cannot be run at 143all through the `make test` command (typically test cases that require 144test specific C code to be compiled) because `make test` runs tests 145directly by invoking the `ct_run` command instead of using the `ts` 146wrapper. One has to follow the procedure described above to run test 147cases that do not work with `make test`. 148 149Below are some examples that illustrate how `make test` can be 150used: 151 152 # ERL_TOP needs to be set correctly 153 cd /path/to/otp 154 export ERL_TOP=`pwd` 155 156 # Build Erlang/OTP 157 # 158 # Note that make test will only compile test code except when 159 # make test is executed from $ERL_TOP. 160 ./otp_build setup -a 161 162 # Run a test case (The ARGS variable is passed to ct_run) 163 (cd $ERL_TOP/erts/emulator && make ARGS="-suite binary_SUITE -case deep_bitstr_lists" test) 164 165 # Run a test suite 166 (cd $ERL_TOP/lib/stdlib && make ARGS="-suite ets_SUITE" test) 167 168 # Run all test suites for an application 169 (cd $ERL_TOP/lib/asn1 && make test) 170 171 # Run all tests 172 # 173 # When executed from $ERL_TOP, "make test" will first release and 174 # configure all tests and then attempt to run all tests with `ts:run`. 175 # This will take several hours. 176 (cd $ERL_TOP && make test) 177 178 179Examining the results 180--------------------- 181 182Open the file `release/tests/test_server/index.html` in a web browser. Or open 183`release/tests/test_server/last_test.html` when a test suite is running to 184examine the results so far for the currently executing test suite (in R14B02 and 185later you want to open the `release/tests/test_server/all_runs.html` file to 186get to the currently running test) 187 188 189Run tests with Address Sanitizer 190-------------------------------- 191 192First build emulator with `asan` build target. 193See [$ERL_TOP/HOWTO/INSTALL.md][]. 194 195Set environment variable `ASAN_LOG_DIR` to the directory 196where the error logs will be generated. 197 198 export ASAN_LOG_DIR=$TESTROOT/test_server/asan_logs 199 mkdir $ASAN_LOG_DIR 200 201Set environment variable `TS_RUN_EMU` to `asan`. 202 203 export TS_RUN_EMU=asan 204 205Then run the tests you want with `ts:run` as described above. Either 206inspect the log files directly or use the script at 207`$ERL_TOP/erts/emulator/asan/asan_logs_to_html` to read all log files 208in `$ASAN_LOG_DIR` and distill them into one html page 209`asan_summary.html`. Repeated reports from the same memory leak will 210for example be ignored by the script and make it easier to analyze. 211 212 213Run tests with Valgrind 214----------------------- 215 216First make sure [valgrind][] is installed, then build OTP from source 217and build the emulator with `valgrind` build target. See 218[$ERL_TOP/HOWTO/INSTALL.md][]. 219 220Set environment variable `VALGRIND_LOG_DIR` to the directory 221where the valgrind error logs will be generated. 222 223 export VALGRIND_LOG_DIR=$TESTROOT/test_server/vg_logs 224 mkdir $VALGRIND_LOG_DIR 225 226Set environment variable `TS_RUN_EMU` to `valgrind`. 227 228 export TS_RUN_EMU=valgrind 229 230Then run the tests you want with `ts:run` as described above and 231inspect the log file(s) in `$VALGRIND_LOG_DIR`. 232 233 234 [ct_run]: http://www.erlang.org/doc/man/ct_run.html 235 [ct hook]: http://www.erlang.org/doc/apps/common_test/ct_hooks_chapter.html 236 [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md 237 [$ERL_TOP/HOWTO/INSTALL-CROSS.md]: INSTALL-CROSS.md#testing-the-cross-compiled-system 238 [common_test]: http://www.erlang.org/doc/man/ct.html 239 [data_dir]: http://www.erlang.org/doc/apps/common_test/write_test_chapter.html#data_priv_dir 240 [configuring the tests]: #configuring-the-test-environment 241 [valgrind]: https://valgrind.org 242 243 [?TOC]: true 244