1# SCIP Unit Tests 2 3Write and run unit tests for SCIP. 4 5- [Overview](#overview) 6- [Write](#write) 7 - [Examples](#examples) 8- [Compile](#compile) 9- [Run](#run) 10- [Debug](#debug) 11 12## Overview 13 14A unit test is an automated piece of code that invokes a unit of work in the system and then checks a single assumption about the behavior of that unit of work. The SCIP Unit Test Suite leverages [Criterion](http://criterion.readthedocs.io/en/master/) as the testing framework and [ctest](https://cmake.org/cmake/help/v2.8.8/ctest.html) as the runner. The SCIP Unit Test Suite is very much in a state of development. Check out the [unit test suite milestone](https://git.zib.de/integer/scip/-/milestones/2) for more information. 15 16## Write 17 18Tests are organized into topic-specific directories in `src`. When writing new tests, find the directory that best suites your test, or create one if it doesn't already exist. For example, if a test is meant to illustrate a bug, place is in `src/bugs/`. Use `#include "include/scip_test.h"` to access Criterion and the `SCIP_CALL` macro. Ensure that this is the **last** included header. 19 20**NOTE** If your test needs `SCIP` code (eg, you are implementing a constraint handler in your test, see `src/cons/cons.c`), place `#include "include/scip_test.h"` after the SCIP code. 21 22Criterion comes with [fixtures](http://criterion.readthedocs.io/en/master/starter.html?highlight=fixture#fixtures) and [asserts](http://criterion.readthedocs.io/en/master/assert.html) built-in, and also supports [parameterized tests](http://criterion.readthedocs.io/en/master/parameterized.html). 23 24### Examples 25 26Here are some test examples that can help you get started writing unit tests. 27 28| Example Type| Location | 29| ------ | ------ | 30| catch a signal | `src/bugs/depthlevel.c` | 31| parameterized test | unittest_framework_tmp branch, `src/cons/expr/simplify.c` | 32| check stdout | unittest_framework_tmp branch, `src/cons/expr/walk.c` | 33 34## Compile 35 36Smart test discovery is already built into the `Makefile`, so anything in `src` (at any level of nesting) will be detected and compiled into the equivalent path in the `bin` directory. Also, the `Makefile` generates the test "makefile" for `ctest`. There should never be a reason to directly modify any Makefile unless you are hacking on the SCIP Unit Test Suite. 37 38The easiest way to compile and run the tests is: 39 40``` 41make 42``` 43 44**NOTE** `make` will read the options used for building `SCIP` from the binary. It uses `scip --version` to find out the options. 45If `SHARED=true`, or `OPT=dbg` were not used when compiling `SCIP`, `make` will end with a proper error. If the binary is not found, it will 46assume `SHARED=true` and `OPT=dbg`. 47 48This command will check for [Criterion](http://criterion.readthedocs.io/en/master/) in ./Criterion, download and install it if not found, and compile and run all tests in `src/`. 49If you already have installed Criterion on you system, execute `touch Criterion` or `mkdir Criterion` before calling make. 50 51**NOTE** Some tests might need to include c files from SCIP. For tests to be recompilied the included c file gets recompiled, run `make depend`. 52 53## Run 54 55See above for the easiest way to compile and run tests. For simply running tests: 56 57``` 58make 59``` 60 61This creates `CTestTestfile.cmake` with a list of the test to run and then calls `ctest --output-on-failure`. By default, tests in `src/bugs/` are not compiled or run since they take a long time. To compile and run them: 62 63``` 64make BUGS=true 65``` 66 67You can also run a single test, e.g. ` 68``` 69 >> ./bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 70``` 71 72Note, that parameterized tests will not work on systems that have address 73space layout randomization (ASLR) enabled. One can disable ASLR for a 74specific process (and its children) by calling it in a modified environment, e.g., 75``` 76 >> setarch `uname -m` -R ./bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 77``` 78 79This is the approach that is also followed by the Makefile when running 80the whole test suite. 81 82Alternatively, one can disable ASLR system-wide (requires root access): 83``` 84 >> sudo echo 0 > /proc/sys/kernel/randomize_va_space 85``` 86 87TODO: Define a policy for moving/removing tests in `src/bugs` once the bugs are fixed. 88 89## Debug (up to Criterion 2.2.2) 90 91If a test fails, use `gdb` to debug. For example: 92 93``` 94 >> ./bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 95 [----] src/cons/quadratic/gauge.c:112: Assertion failed: gauge unavailable, pointless to continue 96 [FAIL] separation::gauge: (0.00s) 97 [====] Synthesis: Tested: 1 | Passing: 0 | Failing: 1 | Crashing: 0 98``` 99 100The test suite is `separation` and the test name is `gauge`. To debug: 101 102``` 103>> gdb --args bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 --single separation::gauge 104(gdb) br src/cons/quadratic/gauge.c:112 105``` 106 107Criterion by default prints all of the critical debugging information (test_suite::test_name, file and line number were to break). When a test crashes, there is no need to `break` in `gdb`. 108 109## Debug (with Criterion 2.3 and later) 110 111If a test fails, one can use `gdb` or `undodb-gdb` to debug. For example: 112 113``` 114 >> ./bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 115 [----] src/cons/quadratic/gauge.c:112: Assertion failed: gauge unavailable, pointless to continue 116 [FAIL] separation::gauge: (0.00s) 117 [====] Synthesis: Tested: 1 | Passing: 0 | Failing: 1 | Crashing: 0 118``` 119 120The test suite is `separation` and the test name is `gauge`. To debug with `gdb` write in one terminal: 121 122``` 123>> bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 --filter *gauge* --debug 124``` 125This will start a `gdbserver`. To connect, in another terminal use 126``` 127>> gdb bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 -ex "target remote localhost:1234" 128``` 129 130If one doesn't want to use a `gdbserver` use: 131``` 132>> bin/cons/quadratic/gauge.linux.x86_64.gnu.dbg.spx2 --filter *gauge* --debug=idle 133``` 134This will give the PID of the process which can then be attached to a `undodb-gdb` or `gdb` session with 135``` 136>> gdb --pid <pid-number> 137``` 138 139After this, execute `continue` twice (or more, until you find the right place) in gdb. 140Use `bt` to see the backtrace. 141