1
2*****************************************************
3Using Testeth
4*****************************************************
5
6Ethereum cpp-client testeth tool for creation and execution of ethereum tests.
7
8To run tests you should open folder (see also `Installing and building <https://github.com/ethereum/cpp-ethereum#building-from-source>`_)
9
10   ``/build/test``
11
12and execute a command:
13
14   ``./testeth``
15
16This will run all test cases automatically.
17By default `testeth` will look for the test repository cloned in cpp-ethereum submodule ``cpp-ethereum/test/jsontests`` assuming that the build folder is ``cpp-ethereum/build``
18
19If environment variable ``ETHEREUM_TEST_PATH`` is set in /etc/environment file, ``testeth`` will use path to the test repo from that variable. Example:
20
21|    ``nano /etc/environment``
22|    ``ETHEREUM_TEST_PATH="/home/user/ethereum/tests"``
23
24You could always override the test path for testeth using an option:
25
26   ``./testeth -- --testpath "/path/to/the/tests"``
27
28Note that --testpath option argument should be an absolute path.
29For a brief help on testeth command options make sure to run
30
31   ``./testeth --help``
32
33
34Running a specific test case
35--------------------------------------------------------------------------------
36
37To run a specific test case you could use parameter ``-t`` in the command line option:
38
39    ``./testeth -t <TEST_SUITE>/<TEST_CASE>``
40
41Or just the test suite:
42
43   ``./testeth -t <TEST_SUITE>``
44
45To run a specific test from a test case (reference by name):
46
47   ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --singletest <TEST_NAME>``
48
49To run a specific test from a test case (reference by test case file path):
50   ``./testeth -t <TEST_TYPE> -- --singletest <TEST_FILE_PATH> <TEST_NAME>``
51
52   where the valid choices for <TEST_TYPE> are same as for <TEST_SUITE>: `GeneralStateTests`, `BlockchainTests`, `TransitionTests`, `TransactionTests`, `VMTests`
53
54
55Tests has cases designed for different network rules. Such as initial frontier rules, homestead rules and other fork updates. That is to make sure that your client could sync up from the very begining to the recent top block. Block fork numbers are declared in genesis config in the file:
56
57https://github.com/ethereum/cpp-ethereum/blob/develop/libethashseal/genesis/mainNetwork.cpp
58
59If you need to debug a specific test on a specific network rules use this command:
60
61   ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --singletest <TEST_NAME> --singlenet <NET_NAME>``
62
63Currently network names <NET_NAME> are following: Frontier, Homestead, EIP150, EIP158, Byzantine, Constantinople
64
65The main test suites are <TEST_SUITE>: GeneralStateTests, BlockchainTests, TransitionTests, TransactionTests, VMTests
66
67<TEST_CASE> correspond to a folder name in the tests repo. And <TEST_NAME> correspond to the filename in that folder describing a specific test or an absolute path to the test file.
68
69GeneralStateTests has a single transaction being executed on a given pre state to produce a post state.
70This transaction has arrays <data>, <value>, <gasLimit>. So a single test execute transaction with different arguments taken from those arrays to test different conditions on the same pre state. To run a transaction from the GeneralStateTests with the specified arguments use:
71
72   ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --singletest <TEST_NAME> --singlenet <NET_NAME> -d <DATA_INDEX> -g <GASLIMIT_INDEX> -v <VALUE_INDEX>``
73
74This will run a transaction with given data, gasLimit, and value as described in the test on a given network rules. Note that parameters here are indexes. The actual values described in the test file itself. This is only valid when <TEST_SUITE> is GeneralStateTests.
75
76Debugging and tracing a state test
77--------------------------------------------------------------------------------
78
79``testeth`` has debugging options for getting a step by step execution log from the EVM.
80Use following options:
81
82   ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --vmtrace --verbosity 5``
83
84``--vmtrace`` prints a step by step execution log from the EVM.
85
86   ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --jsontrace <CONFIG>``
87
88An rpc method like, providing step by step debug in json format. The <CONFIG> is in json format like following:
89
90   ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --jsontrace '{ "disableStorage" : false, "disableMemory" : false, "disableStack" : false, "fullStorage" : true }' ``
91
92Or just empty string to load default options.
93
94    ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --jsontrace '' ``
95
96You could specify some of the options in this json line or use an empty argument to load default options. Sometimes you might want to disable just the memory logs or the storage logs or both cause it could provide a lot lines to the final log.
97
98    ``./testeth -t <TEST_SUITE>/<TEST_CASE> -- --statediff``
99
100Get a statediff of a pre -> post state in general state test. Use this to see what accounts has changed after executing a transaction. This options should better be used in combination with ``--singletest <>`` ``--singlenet <>`` and ``-d -v -g`` (if any)
101
102The option --exectimelog will print the stats on how much time was spend on a specific test suite/case. It will also sort the most time consuming test at the end of the execution.
103
104   ``./testeth -- --exectimelog``
105
106Note that some tests are disabled by default. Such as Frontier, Homestead rules tests, some time consuming tests. If you want to run a full test suite with all tests available use option --all:
107
108   ``./testeth -- --all``
109
110
111Generating (filling) the tests
112--------------------------------------------------------------------------------
113
114Tests are generated from test filler files located in the src folder of the test repo. Testeth will run the execution of a ``Filler.json`` file and produce a final test in the repo.
115
116Generating a test case and creating new tests is a whole new topic and it's described in more detail here:
117
118   https://ethereum-tests.readthedocs.io/en/latest/generating-tests.html
119