1# Test system 2 3## openage computer-enriched testing automation 4 5### Tests 6 7There are various supported kinds of tests: 8 9 - cpp tests 10 - py tests 11 - py doctests 12 13Tests run without user interaction to check for errors automatically. 14 15All tests are run automatically by [Kevin](https://github.com/SFTtech/kevin/) for pullrequests. 16 17 18You can invoke them with `./run test -a` or `make test` 19 20Have a look at `./run test --help` for further options. 21 22 23You are encouraged to write tests for all your contributions, as well as other components that currently lack testing. 24 25 26### Demos 27 28In addition to testing, openage supports _demos_: 29 30 - cpp demos 31 - py demos 32 33As opposed to tests, demos are run manually and individually by the user. 34They usually produce lots of output on stdout or may even be interactive. Python demos even accept an `argv` parameter. 35 36All tests must be registered in `openage/testing/testlist.py` (else the game won't know about them). 37 38## Adding new tests 39 40### C++ tests 41 42C++ tests are simple `void()` functions somewhere in the `openage` namespace. 43 44They shall return on success, and raise `openage::testing::TestError` on failure. 45 46They shall not be declared in a header file; instead, add them to `openage/testing/testlist.py`. 47 48The header `libopenage/testing/testing.h` provides `TestError` and some convenience macros: 49 50 - `TESTFAIL` 51 throws `TestError` 52 - `TESTFAILMSG("a" << "b")` 53 throws `TestError("ab")` 54 - `TESTEQUALS(left, right)` 55 evaluates `left` and `right` 56 throws `TestError(left)` if `left != right` 57 throws `TestError(exc)` if a non-`TestError` exception `exc` is raised. 58 - `TESTTHROWS(expr)` 59 evaluates `expr`, catching any exception, including `TestError`. 60 raises `TestError` if no exception was caught. 61 62Example test function: 63 64``` cpp 65void test_prime() { 66 is_prime(23) or TESTFAIL; 67 is_prime(42) and TESTFAIL; 68} 69``` 70 71 72### Python tests 73 74Python tests are simple argument-less functions somewhere in the `openage` package. 75 76They shall return `None` on success, and raise `openage.testing.TestError` on failure. 77 78Add their names to `openage/testing/testlist.py`. 79 80The module `openage.testing.testing` provides `TestError` and some convenience functions: 81 82 - `assert_value(<expr>, expected)` 83 checks whether expr == expected, and raises `TestError` if not. 84 - `assert_raises(expected_exception_type)` 85 a context guard that verifies that the named exception occurs inside; 86 consult the example in `openage/testing/testing.py`. 87 88You may define tests in `.pyx` files. 89 90Example test function: 91 92``` python 93def test_prime(): 94 assert_value(is_prime(23), True) 95 assert_value(is_prime(42), False) 96 97 with assert_raises(ValueError): 98 result(is_prime(-1337)) 99``` 100 101 102### Python doctests 103 104[Doctests](https://docs.python.org/3/library/doctest.html) are an integrated feature of Python. 105 106They defined in function and module docstrings, are extremely lightweight and also serve as documentation. 107 108Simply add the name of a Python module to `openage/testing/testlist.py`, and all doctests in that module will run. 109 110Example doctest for a function: 111 112``` python 113def is_prime(p): 114 """ 115 High-performance, state-of-the-art primality tester. 116 117 >>> is_prime(23) 118 True 119 >>> is_prime(42) 120 False 121 """ 122 return not any(p % x == 0 for x in range(2, p)) 123``` 124 125### C++ demos 126 127Technically, those are very much like `C++` tests. In fact, the only difference to tests is the section in `openage/testing/testlist.py` where they are declared. 128 129C++ demos don't support `argv`; if you want that, make it a Python demo in a `.pyx` file and do the argparsing in Python; the Python demo function can then easily call any C++ function using the Python interface. 130 131 132### Python demos 133 134Similar to Python tests, but have one argument, `argv`. Pass arguments in the invocation: 135 136 ./run test -d prime_demo 100 137 138Example demo: 139 140``` python 141def prime_demo(argv): 142 import argparse 143 cli = argparse.ArgumentParser() 144 cli.add_argument('max_number', type=int) 145 args = cli.parse_args(argv) 146 147 for p in range(2, args.max_number): 148 if is_prime(p): 149 print(p) 150``` 151 152 153## Why? 154 155### Demos 156 157Demos should be used to implement and develop new features. 158You can directly call your code without having to launch up the whole engine. 159 160Use demos while developing new things or improvements. 161 162 163### Tests 164 165All tests are run for each pull requests, so we can detect your change broke something. 166Please try to write tests for everything you do. 167 168This is our only way of automated regression detection. 169