1# Functional tests 2 3### Writing Functional Tests 4 5#### Example test 6 7The file [test/functional/example_test.py](example_test.py) is a heavily commented example 8of a test case that uses both the RPC and P2P interfaces. If you are writing your first test, copy 9that file and modify to fit your needs. 10 11#### Coverage 12 13Running `test/functional/test_runner.py` with the `--coverage` argument tracks which RPCs are 14called by the tests and prints a report of uncovered RPCs in the summary. This 15can be used (along with the `--extended` argument) to find out which RPCs we 16don't have test cases for. 17 18#### Style guidelines 19 20- Where possible, try to adhere to [PEP-8 guidelines](https://www.python.org/dev/peps/pep-0008/) 21- Use a python linter like flake8 before submitting PRs to catch common style 22 nits (eg trailing whitespace, unused imports, etc) 23- The oldest supported Python version is specified in [doc/dependencies.md](/doc/dependencies.md). 24 Consider using [pyenv](https://github.com/pyenv/pyenv), which checks [.python-version](/.python-version), 25 to prevent accidentally introducing modern syntax from an unsupported Python version. 26 The CI linter job also checks this, but [possibly not in all cases](https://github.com/bitcoin/bitcoin/pull/14884#discussion_r239585126). 27- See [the python lint script](/test/lint/lint-python.sh) that checks for violations that 28 could lead to bugs and issues in the test code. 29- Use [type hints](https://docs.python.org/3/library/typing.html) in your code to improve code readability 30 and to detect possible bugs earlier. 31- Avoid wildcard imports 32- Use a module-level docstring to describe what the test is testing, and how it 33 is testing it. 34- When subclassing the BitcoinTestFramework, place overrides for the 35 `set_test_params()`, `add_options()` and `setup_xxxx()` methods at the top of 36 the subclass, then locally-defined helper methods, then the `run_test()` method. 37- Use `f'{x}'` for string formatting in preference to `'{}'.format(x)` or `'%s' % x`. 38 39#### Naming guidelines 40 41- Name the test `<area>_test.py`, where area can be one of the following: 42 - `feature` for tests for full features that aren't wallet/mining/mempool, eg `feature_rbf.py` 43 - `interface` for tests for other interfaces (REST, ZMQ, etc), eg `interface_rest.py` 44 - `mempool` for tests for mempool behaviour, eg `mempool_reorg.py` 45 - `mining` for tests for mining features, eg `mining_prioritisetransaction.py` 46 - `p2p` for tests that explicitly test the p2p interface, eg `p2p_disconnect_ban.py` 47 - `rpc` for tests for individual RPC methods or features, eg `rpc_listtransactions.py` 48 - `tool` for tests for tools, eg `tool_wallet.py` 49 - `wallet` for tests for wallet features, eg `wallet_keypool.py` 50- Use an underscore to separate words 51 - exception: for tests for specific RPCs or command line options which don't include underscores, name the test after the exact RPC or argument name, eg `rpc_decodescript.py`, not `rpc_decode_script.py` 52- Don't use the redundant word `test` in the name, eg `interface_zmq.py`, not `interface_zmq_test.py` 53 54#### General test-writing advice 55 56- Instead of inline comments or no test documentation at all, log the comments to the test log, e.g. 57 `self.log.info('Create enough transactions to fill a block')`. Logs make the test code easier to read and the test 58 logic easier [to debug](/test/README.md#test-logging). 59- Set `self.num_nodes` to the minimum number of nodes necessary for the test. 60 Having additional unrequired nodes adds to the execution time of the test as 61 well as memory/CPU/disk requirements (which is important when running tests in 62 parallel). 63- Avoid stop-starting the nodes multiple times during the test if possible. A 64 stop-start takes several seconds, so doing it several times blows up the 65 runtime of the test. 66- Set the `self.setup_clean_chain` variable in `set_test_params()` to `True` to 67 initialize an empty blockchain and start from the Genesis block, rather than 68 load a premined blockchain from cache with the default value of `False`. The 69 cached data directories contain a 200-block pre-mined blockchain with the 70 spendable mining rewards being split between four nodes. Each node has 25 71 mature block subsidies (25x50=1250 BTC) in its wallet. Using them is much more 72 efficient than mining blocks in your test. 73- When calling RPCs with lots of arguments, consider using named keyword 74 arguments instead of positional arguments to make the intent of the call 75 clear to readers. 76- Many of the core test framework classes such as `CBlock` and `CTransaction` 77 don't allow new attributes to be added to their objects at runtime like 78 typical Python objects allow. This helps prevent unpredictable side effects 79 from typographical errors or usage of the objects outside of their intended 80 purpose. 81 82#### RPC and P2P definitions 83 84Test writers may find it helpful to refer to the definitions for the RPC and 85P2P messages. These can be found in the following source files: 86 87- `/src/rpc/*` for RPCs 88- `/src/wallet/rpc*` for wallet RPCs 89- `ProcessMessage()` in `/src/net_processing.cpp` for parsing P2P messages 90 91#### Using the P2P interface 92 93- `P2P`s can be used to test specific P2P protocol behavior. 94[p2p.py](test_framework/p2p.py) contains test framework p2p objects and 95[messages.py](test_framework/messages.py) contains all the definitions for objects passed 96over the network (`CBlock`, `CTransaction`, etc, along with the network-level 97wrappers for them, `msg_block`, `msg_tx`, etc). 98 99- P2P tests have two threads. One thread handles all network communication 100with the bitcoind(s) being tested in a callback-based event loop; the other 101implements the test logic. 102 103- `P2PConnection` is the class used to connect to a bitcoind. `P2PInterface` 104contains the higher level logic for processing P2P payloads and connecting to 105the Bitcoin Core node application logic. For custom behaviour, subclass the 106P2PInterface object and override the callback methods. 107 108`P2PConnection`s can be used as such: 109 110```python 111p2p_conn = node.add_p2p_connection(P2PInterface()) 112p2p_conn.send_and_ping(msg) 113``` 114 115They can also be referenced by indexing into a `TestNode`'s `p2ps` list, which 116contains the list of test framework `p2p` objects connected to itself 117(it does not include any `TestNode`s): 118 119```python 120node.p2ps[0].sync_with_ping() 121``` 122 123More examples can be found in [p2p_unrequested_blocks.py](p2p_unrequested_blocks.py), 124[p2p_compactblocks.py](p2p_compactblocks.py). 125 126#### Prototyping tests 127 128The [`TestShell`](test-shell.md) class exposes the BitcoinTestFramework 129functionality to interactive Python3 environments and can be used to prototype 130tests. This may be especially useful in a REPL environment with session logging 131utilities, such as 132[IPython](https://ipython.readthedocs.io/en/stable/interactive/reference.html#session-logging-and-restoring). 133The logs of such interactive sessions can later be adapted into permanent test 134cases. 135 136### Test framework modules 137The following are useful modules for test developers. They are located in 138[test/functional/test_framework/](test_framework). 139 140#### [authproxy.py](test_framework/authproxy.py) 141Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc). 142 143#### [test_framework.py](test_framework/test_framework.py) 144Base class for functional tests. 145 146#### [util.py](test_framework/util.py) 147Generally useful functions. 148 149#### [p2p.py](test_framework/p2p.py) 150Test objects for interacting with a bitcoind node over the p2p interface. 151 152#### [script.py](test_framework/script.py) 153Utilities for manipulating transaction scripts (originally from python-bitcoinlib) 154 155#### [key.py](test_framework/key.py) 156Test-only secp256k1 elliptic curve implementation 157 158#### [blocktools.py](test_framework/blocktools.py) 159Helper functions for creating blocks and transactions. 160 161### Benchmarking with perf 162 163An easy way to profile node performance during functional tests is provided 164for Linux platforms using `perf`. 165 166Perf will sample the running node and will generate profile data in the node's 167datadir. The profile data can then be presented using `perf report` or a graphical 168tool like [hotspot](https://github.com/KDAB/hotspot). 169 170There are two ways of invoking perf: one is to use the `--perf` flag when 171running tests, which will profile each node during the entire test run: perf 172begins to profile when the node starts and ends when it shuts down. The other 173way is the use the `profile_with_perf` context manager, e.g. 174 175```python 176with node.profile_with_perf("send-big-msgs"): 177 # Perform activity on the node you're interested in profiling, e.g.: 178 for _ in range(10000): 179 node.p2ps[0].send_message(some_large_message) 180``` 181 182To see useful textual output, run 183 184```sh 185perf report -i /path/to/datadir/send-big-msgs.perf.data.xxxx --stdio | c++filt | less 186``` 187 188#### See also: 189 190- [Installing perf](https://askubuntu.com/q/50145) 191- [Perf examples](http://www.brendangregg.com/perf.html) 192- [Hotspot](https://github.com/KDAB/hotspot): a GUI for perf output analysis 193