1 _ _ ____ _
2 ___| | | | _ \| |
3 / __| | | | |_) | |
4 | (__| |_| | _ <| |___
5 \___|\___/|_| \_\_____|
6
7The curl Test Suite
8
9 1. Running
10 1.1 Requires to run
11 1.2 Port numbers used by test servers
12 1.3 Test servers
13 1.4 Run
14 1.5 Shell startup scripts
15 1.6 Memory test
16 1.7 Debug
17 1.8 Logs
18 1.9 Test input files
19 1.10 Code coverage
20 1.11 Remote testing
21
22 2. Numbering
23 2.1 Test case numbering
24
25 3. Write tests
26 3.1 test data
27 3.2 curl tests
28 3.3 libcurl tests
29 3.4 unit tests
30
31 4. TODO
32 4.1 More protocols
33 4.2 SOCKS auth
34
35==============================================================================
36
371. Running
38
39 1.1 Requires to run
40
41 perl (and a unix-style shell)
42 python (and a unix-style shell, for SMB and TELNET tests)
43 python-impacket (for SMB tests)
44 diff (when a test fails, a diff is shown)
45 stunnel (for HTTPS and FTPS tests)
46 OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
47 nghttpx (for HTTP/2 tests)
48 nroff (for --manual tests)
49
50 1.1.1 Installation of python-impacket
51
52 The Python-based test servers support both recent Python 2 and 3.
53 You can figure out your default Python interpreter with python -V
54
55 Please install python-impacket in the correct Python environment.
56 You can use pip or your OS' package manager to install 'impacket'.
57
58 On Debian/Ubuntu the package names are:
59 Python 2: 'python-impacket'
60 Python 3: 'python3-impacket'
61
62 On FreeBSD the package names are:
63 Python 2: 'py27-impacket'
64 Python 3: 'py37-impacket'
65
66 On any system where pip is available:
67 Python 2: 'pip2 install impacket'
68 Python 3: 'pip3 install impacket'
69
70 You may also need to manually install the Python package 'six'
71 as that may be a missing requirement for impacket on Python 3.
72
73 1.2 Port numbers used by test servers
74
75 Tests are written to use as few fixed fixed port numbers as possible and all
76 tests should be written to use suitable variables instead of port numbers so
77 that test cases continue to work independent on what port numbers the test
78 servers actually use.
79
80 The remaining fixed-port test servers that are still used, use the port
81 range 8890 - 8904 by default, but can be moved with runtests' -b option.
82
83 See the FILEFORMAT for a listing of existing port number variables.
84
85 1.3 Test servers
86
87 The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
88 servers on the ports listed above to which it makes requests. For SSL tests,
89 it runs stunnel to handle encryption to the regular servers. For SSH, it
90 runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
91 the SOCKS functionality and requires a SSH client and server.
92
93 The base port number (8990), which all the individual port numbers are
94 indexed from, can be set explicitly using runtests.pl' -b option to allow
95 running more than one instance of the test suite simultaneously on one
96 machine, or just move the servers in case you have local services on any of
97 those ports.
98
99 The HTTP server supports listening on a Unix domain socket, the default
100 location is 'http.sock'.
101
102 1.4 Run
103
104 './configure && make && make test'. This builds the test suite support code
105 and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
106 variables of that script in case you have some specific needs, or run the
107 script manually (after the support code has been built).
108
109 The script breaks on the first test that doesn't do OK. Use -a to prevent
110 the script from aborting on the first error. Run the script with -v for more
111 verbose output. Use -d to run the test servers with debug output enabled as
112 well. Specifying -k keeps all the log files generated by the test intact.
113
114 Use -s for shorter output, or pass test numbers to run specific tests only
115 (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
116 ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
117 3 to 9. Any test numbers starting with ! are disabled, as are any test
118 numbers found in the files data/DISABLED or data/DISABLED.local (one per
119 line). The latter is meant for local temporary disables and will be ignored
120 by git.
121
122 When -s is not present, each successful test will display on one line the
123 test number and description and on the next line a set of flags, the test
124 result, current test sequence, total number of tests to be run and an
125 estimated amount of time to complete the test run. The flags consist of
126 these letters describing what is checked in this test:
127
128 s stdout
129 d data
130 u upload
131 p protocol
132 o output
133 e exit code
134 m memory
135 v valgrind
136
137 1.5 Shell startup scripts
138
139 Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
140 influenced by the output of system wide or user specific shell startup
141 scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
142 output text messages or escape sequences on user login. When these shell
143 startup messages or escape sequences are output they might corrupt the
144 expected stream of data which flows to the sftp-server or from the ssh
145 client which can result in bad test behaviour or even prevent the test
146 server from running.
147
148 If the test suite ssh or sftp server fails to start up and logs the message
149 'Received message too long' then you are certainly suffering the unwanted
150 output of a shell startup script. Locate, cleanup or adjust the shell
151 script.
152
153 1.6 Memory test
154
155 The test script will check that all allocated memory is freed properly IF
156 curl has been built with the CURLDEBUG define set. The script will
157 automatically detect if that is the case, and it will use the
158 'memanalyze.pl' script to analyze the memory debugging output.
159
160 Also, if you run tests on a machine where valgrind is found, the script will
161 use valgrind to run the test with (unless you use -n) to further verify
162 correctness.
163
164 runtests.pl's -t option will enable torture testing mode, which runs each
165 test many times and makes each different memory allocation fail on each
166 successive run. This tests the out of memory error handling code to ensure
167 that memory leaks do not occur even in those situations. It can help to
168 compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
169 ensure that the memory log file is properly written even if curl crashes.
170
171 1.7 Debug
172
173 If a test case fails, you can conveniently get the script to invoke the
174 debugger (gdb) for you with the server running and the exact same command
175 line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
176 then just type 'run' in the debugger to perform the command through the
177 debugger.
178
179 1.8 Logs
180
181 All logs are generated in the log/ subdirectory (it is emptied first in the
182 runtests.pl script). Use runtests.pl -k to force it to keep the temporary
183 files after the test run since successful runs will clean it up otherwise.
184
185 1.9 Test input files
186
187 All test cases are put in the data/ subdirectory. Each test is stored in the
188 file named according to the test number.
189
190 See FILEFORMAT for the description of the test case files.
191
192 1.10 Code coverage
193
194 gcc provides a tool that can determine the code coverage figures for
195 the test suite. To use it, configure curl with
196 CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
197 and torture tests to get more full coverage, i.e. do:
198
199 make test
200 make test-torture
201
202 The graphical tool ggcov can be used to browse the source and create
203 coverage reports on *NIX hosts:
204
205 ggcov -r lib src
206
207 The text mode tool gcov may also be used, but it doesn't handle object files
208 in more than one directory very well.
209
210 1.11 Remote testing
211
212 The runtests.pl script provides some hooks to allow curl to be tested on a
213 machine where perl can not be run. The test framework in this case runs on
214 a workstation where perl is available, while curl itself is run on a remote
215 system using ssh or some other remote execution method. See the comments at
216 the beginning of runtests.pl for details.
217
2182. Numbering
219
220 2.1 Test case numbering
221
222 Test cases used to be numbered by category, but the ranges filled
223 up. Subsets of tests can now be selected by passing keywords to the
224 runtests.pl script via the make TFLAGS variable.
225
226 New tests should now be added by finding a free number in
227 tests/data/Makefile.inc.
228
2293. Write tests
230
231 Here's a quick description on writing test cases. We basically have three
232 kinds of tests: the ones that test the curl tool, the ones that build small
233 applications and test libcurl directly and the unit tests that test
234 individual (possibly internal) functions.
235
236 3.1 test data
237
238 Each test has a master file that controls all the test data. What to read,
239 what the protocol exchange should look like, what exit code to expect and
240 what command line arguments to use etc.
241
242 These files are tests/data/test[num] where [num] is described in section 2
243 of this document, and the XML-like file format of them is described in the
244 separate tests/FILEFORMAT document.
245
246 3.2 curl tests
247
248 A test case that runs the curl tool and verifies that it gets the correct
249 data, it sends the correct data, it uses the correct protocol primitives
250 etc.
251
252 3.3 libcurl tests
253
254 The libcurl tests are identical to the curl ones, except that they use a
255 specific and dedicated custom-built program to run instead of "curl". This
256 tool is built from source code placed in tests/libtest and if you want to
257 make a new libcurl test that is where you add your code.
258
259 3.4 unit tests
260
261 Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
262 There's a tests/unit/README describing the specific set of checks and macros
263 that may be used when writing tests that verify behaviors of specific
264 individual functions.
265
266 The unit tests depend on curl being built with debug enabled.
267
2684. TODO
269
270 4.1 More protocols
271
272 Add tests for TELNET, LDAP, DICT...
273
274 4.2 SOCKS auth
275
276 SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
277 test mechanism) doesn't support them
278