README.md
1# The curl Test Suite
2
3# Running
4
5## Requires to run
6
7 - perl (and a unix-style shell)
8 - python (and a unix-style shell, for SMB and TELNET tests)
9 - python-impacket (for SMB tests)
10 - diff (when a test fails, a diff is shown)
11 - stunnel (for HTTPS and FTPS tests)
12 - OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
13 - nghttpx (for HTTP/2 tests)
14 - nroff (for --manual tests)
15 - An available `en_US.UTF-8` locale
16
17### Installation of python-impacket
18
19 The Python-based test servers support both recent Python 2 and 3.
20 You can figure out your default Python interpreter with python -V
21
22 Please install python-impacket in the correct Python environment.
23 You can use pip or your OS' package manager to install 'impacket'.
24
25 On Debian/Ubuntu the package names are:
26
27 - Python 2: 'python-impacket'
28 - Python 3: 'python3-impacket'
29
30 On FreeBSD the package names are:
31
32 - Python 2: 'py27-impacket'
33 - Python 3: 'py37-impacket'
34
35 On any system where pip is available:
36
37 - Python 2: 'pip2 install impacket'
38 - Python 3: 'pip3 install impacket'
39
40 You may also need to manually install the Python package 'six'
41 as that may be a missing requirement for impacket on Python 3.
42
43### Port numbers used by test servers
44
45 All test servers run on "random" port numbers. All tests should be written
46 to use suitable variables instead of fixed port numbers so that test cases
47 continue to work independent on what port numbers the test servers actually
48 use.
49
50 See [FILEFORMAT](FILEFORMAT.md) for the port number variables.
51
52### Test servers
53
54 The test suite runs stand-alone servers on random ports to which it makes
55 requests. For SSL tests, it runs stunnel to handle encryption to the regular
56 servers. For SSH, it runs a standard OpenSSH server. For SOCKS4/5 tests SSH
57 is used to perform the SOCKS functionality and requires a SSH client and
58 server.
59
60 The listen port numbers for the test servers are picked randomly to allow
61 users to run multiple test cases concurrently and to not collide with other
62 existing services that might listen to ports on the machine.
63
64 The HTTP server supports listening on a Unix domain socket, the default
65 location is 'http.sock'.
66
67### Run
68
69 `./configure && make && make test`. This builds the test suite support code
70 and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
71 variables of that script in case you have some specific needs, or run the
72 script manually (after the support code has been built).
73
74 The script breaks on the first test that doesn't do OK. Use `-a` to prevent
75 the script from aborting on the first error. Run the script with `-v` for
76 more verbose output. Use `-d` to run the test servers with debug output
77 enabled as well. Specifying `-k` keeps all the log files generated by the
78 test intact.
79
80 Use `-s` for shorter output, or pass test numbers to run specific tests only
81 (like `./runtests.pl 3 4` to test 3 and 4 only). It also supports test case
82 ranges with 'to', as in `./runtests.pl 3 to 9` which runs the seven tests
83 from 3 to 9. Any test numbers starting with ! are disabled, as are any test
84 numbers found in the files `data/DISABLED` or `data/DISABLED.local` (one per
85 line). The latter is meant for local temporary disables and will be ignored
86 by git.
87
88 Test cases mentioned in `DISABLED` can still be run if `-f` is provided.
89
90 When `-s` is not present, each successful test will display on one line the
91 test number and description and on the next line a set of flags, the test
92 result, current test sequence, total number of tests to be run and an
93 estimated amount of time to complete the test run. The flags consist of
94 these letters describing what is checked in this test:
95
96 s stdout
97 d data
98 u upload
99 p protocol
100 o output
101 e exit code
102 m memory
103 v valgrind
104
105### Shell startup scripts
106
107 Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
108 influenced by the output of system wide or user specific shell startup
109 scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
110 output text messages or escape sequences on user login. When these shell
111 startup messages or escape sequences are output they might corrupt the
112 expected stream of data which flows to the sftp-server or from the ssh
113 client which can result in bad test behavior or even prevent the test
114 server from running.
115
116 If the test suite ssh or sftp server fails to start up and logs the message
117 'Received message too long' then you are certainly suffering the unwanted
118 output of a shell startup script. Locate, cleanup or adjust the shell
119 script.
120
121### Memory test
122
123 The test script will check that all allocated memory is freed properly IF
124 curl has been built with the `CURLDEBUG` define set. The script will
125 automatically detect if that is the case, and it will use the
126 'memanalyze.pl' script to analyze the memory debugging output.
127
128 Also, if you run tests on a machine where valgrind is found, the script will
129 use valgrind to run the test with (unless you use `-n`) to further verify
130 correctness.
131
132 runtests.pl's `-t` option will enable torture testing mode, which runs each
133 test many times and makes each different memory allocation fail on each
134 successive run. This tests the out of memory error handling code to ensure
135 that memory leaks do not occur even in those situations. It can help to
136 compile curl with `CPPFLAGS=-DMEMDEBUG_LOG_SYNC` when using this option, to
137 ensure that the memory log file is properly written even if curl crashes.
138
139### Debug
140
141 If a test case fails, you can conveniently get the script to invoke the
142 debugger (gdb) for you with the server running and the exact same command
143 line parameters that failed. Just invoke `runtests.pl <test number> -g` and
144 then just type 'run' in the debugger to perform the command through the
145 debugger.
146
147### Logs
148
149 All logs are generated in the log/ subdirectory (it is emptied first in the
150 runtests.pl script). They remain in there after a test run.
151
152### Test input files
153
154 All test cases are put in the `data/` subdirectory. Each test is stored in
155 the file named according to the test number.
156
157 See [FILEFORMAT.md](FILEFORMAT.md) for a description of the test case file
158 format.
159
160### Code coverage
161
162 gcc provides a tool that can determine the code coverage figures for the
163 test suite. To use it, configure curl with `CFLAGS='-fprofile-arcs
164 -ftest-coverage -g -O0'`. Make sure you run the normal and torture tests to
165 get more full coverage, i.e. do:
166
167 make test
168 make test-torture
169
170 The graphical tool ggcov can be used to browse the source and create
171 coverage reports on *NIX hosts:
172
173 ggcov -r lib src
174
175 The text mode tool gcov may also be used, but it doesn't handle object files
176 in more than one directory very well.
177
178### Remote testing
179
180 The runtests.pl script provides some hooks to allow curl to be tested on a
181 machine where perl can not be run. The test framework in this case runs on
182 a workstation where perl is available, while curl itself is run on a remote
183 system using ssh or some other remote execution method. See the comments at
184 the beginning of runtests.pl for details.
185
186## Test case numbering
187
188 Test cases used to be numbered by category ranges, but the ranges filled
189 up. Subsets of tests can now be selected by passing keywords to the
190 runtests.pl script via the make `TFLAGS` variable.
191
192 New tests are added by finding a free number in `tests/data/Makefile.inc`.
193
194## Write tests
195
196 Here's a quick description on writing test cases. We basically have three
197 kinds of tests: the ones that test the curl tool, the ones that build small
198 applications and test libcurl directly and the unit tests that test
199 individual (possibly internal) functions.
200
201### test data
202
203 Each test has a master file that controls all the test data. What to read,
204 what the protocol exchange should look like, what exit code to expect and
205 what command line arguments to use etc.
206
207 These files are `tests/data/test[num]` where `[num]` is just a unique
208 identifier described above, and the XML-like file format of them is
209 described in the separate [FILEFORMAT.md](FILEFORMAT.md) document.
210
211### curl tests
212
213 A test case that runs the curl tool and verifies that it gets the correct
214 data, it sends the correct data, it uses the correct protocol primitives
215 etc.
216
217### libcurl tests
218
219 The libcurl tests are identical to the curl ones, except that they use a
220 specific and dedicated custom-built program to run instead of "curl". This
221 tool is built from source code placed in `tests/libtest` and if you want to
222 make a new libcurl test that is where you add your code.
223
224### unit tests
225
226 Unit tests are placed in `tests/unit`. There's a tests/unit/README
227 describing the specific set of checks and macros that may be used when
228 writing tests that verify behaviors of specific individual functions.
229
230 The unit tests depend on curl being built with debug enabled.
231