1# Test system
2
3## openage computer-enriched testing automation
4
5### Tests
6
7There are various supported kinds of tests:
8
9 - cpp tests
10 - py tests
11 - py doctests
12
13Tests run without user interaction to check for errors automatically.
14
15All tests are run automatically by [Kevin](https://github.com/SFTtech/kevin/) for pullrequests.
16
17
18You can invoke them with `./run test -a` or `make test`
19
20Have a look at `./run test --help` for further options.
21
22
23You are encouraged to write tests for all your contributions, as well as other components that currently lack testing.
24
25
26### Demos
27
28In addition to testing, openage supports _demos_:
29
30 - cpp demos
31 - py demos
32
33As opposed to tests, demos are run manually and individually by the user.
34They usually produce lots of output on stdout or may even be interactive. Python demos even accept an `argv` parameter.
35
36All tests must be registered in `openage/testing/testlist.py` (else the game won't know about them).
37
38## Adding new tests
39
40### C++ tests
41
42C++ tests are simple `void()` functions somewhere in the `openage` namespace.
43
44They shall return on success, and raise `openage::testing::TestError` on failure.
45
46They shall not be declared in a header file; instead, add them to `openage/testing/testlist.py`.
47
48The header `libopenage/testing/testing.h` provides `TestError` and some convenience macros:
49
50 - `TESTFAIL`
51    throws `TestError`
52 - `TESTFAILMSG("a" << "b")`
53    throws `TestError("ab")`
54 - `TESTEQUALS(left, right)`
55    evaluates `left` and `right`
56    throws `TestError(left)` if `left != right`
57    throws `TestError(exc)` if a non-`TestError` exception `exc` is raised.
58 - `TESTTHROWS(expr)`
59    evaluates `expr`, catching any exception, including `TestError`.
60    raises `TestError` if no exception was caught.
61
62Example test function:
63
64``` cpp
65void test_prime() {
66    is_prime(23) or TESTFAIL;
67    is_prime(42) and TESTFAIL;
68}
69```
70
71
72### Python tests
73
74Python tests are simple argument-less functions somewhere in the `openage` package.
75
76They shall return `None` on success, and raise `openage.testing.TestError` on failure.
77
78Add their names to `openage/testing/testlist.py`.
79
80The module `openage.testing.testing` provides `TestError` and some convenience functions:
81
82 - `assert_value(<expr>, expected)`
83    checks whether expr == expected, and raises `TestError` if not.
84 - `assert_raises(expected_exception_type)`
85    a context guard that verifies that the named exception occurs inside;
86    consult the example in `openage/testing/testing.py`.
87
88You may define tests in `.pyx` files.
89
90Example test function:
91
92``` python
93def test_prime():
94    assert_value(is_prime(23), True)
95    assert_value(is_prime(42), False)
96
97    with assert_raises(ValueError):
98        result(is_prime(-1337))
99```
100
101
102### Python doctests
103
104[Doctests](https://docs.python.org/3/library/doctest.html) are an integrated feature of Python.
105
106They defined in function and module docstrings, are extremely lightweight and also serve as documentation.
107
108Simply add the name of a Python module to `openage/testing/testlist.py`, and all doctests in that module will run.
109
110Example doctest for a function:
111
112``` python
113def is_prime(p):
114    """
115    High-performance, state-of-the-art primality tester.
116
117    >>> is_prime(23)
118    True
119    >>> is_prime(42)
120    False
121    """
122    return not any(p % x == 0 for x in range(2, p))
123```
124
125### C++ demos
126
127Technically, those are very much like `C++` tests. In fact, the only difference to tests is the section in `openage/testing/testlist.py` where they are declared.
128
129C++ demos don't support `argv`; if you want that, make it a Python demo in a `.pyx` file and do the argparsing in Python; the Python demo function can then easily call any C++ function using the Python interface.
130
131
132### Python demos
133
134Similar to Python tests, but have one argument, `argv`. Pass arguments in the invocation:
135
136    ./run test -d prime_demo 100
137
138Example demo:
139
140``` python
141def prime_demo(argv):
142    import argparse
143    cli = argparse.ArgumentParser()
144    cli.add_argument('max_number', type=int)
145    args = cli.parse_args(argv)
146
147    for p in range(2, args.max_number):
148        if is_prime(p):
149            print(p)
150```
151
152
153## Why?
154
155### Demos
156
157Demos should be used to implement and develop new features.
158You can directly call your code without having to launch up the whole engine.
159
160Use demos while developing new things or improvements.
161
162
163### Tests
164
165All tests are run for each pull requests, so we can detect your change broke something.
166Please try to write tests for everything you do.
167
168This is our only way of automated regression detection.
169