1# The curl Test Suite 2 3# Running 4 5## Requires to run 6 7 - perl (and a unix-style shell) 8 - python (and a unix-style shell, for SMB and TELNET tests) 9 - python-impacket (for SMB tests) 10 - diff (when a test fails, a diff is shown) 11 - stunnel (for HTTPS and FTPS tests) 12 - OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests) 13 - nghttpx (for HTTP/2 tests) 14 - nroff (for --manual tests) 15 - An available `en_US.UTF-8` locale 16 17### Installation of python-impacket 18 19 The Python-based test servers support both recent Python 2 and 3. 20 You can figure out your default Python interpreter with python -V 21 22 Please install python-impacket in the correct Python environment. 23 You can use pip or your OS' package manager to install 'impacket'. 24 25 On Debian/Ubuntu the package names are: 26 27 - Python 2: 'python-impacket' 28 - Python 3: 'python3-impacket' 29 30 On FreeBSD the package names are: 31 32 - Python 2: 'py27-impacket' 33 - Python 3: 'py37-impacket' 34 35 On any system where pip is available: 36 37 - Python 2: 'pip2 install impacket' 38 - Python 3: 'pip3 install impacket' 39 40 You may also need to manually install the Python package 'six' 41 as that may be a missing requirement for impacket on Python 3. 42 43### Port numbers used by test servers 44 45 All test servers run on "random" port numbers. All tests should be written 46 to use suitable variables instead of fixed port numbers so that test cases 47 continue to work independent on what port numbers the test servers actually 48 use. 49 50 See [FILEFORMAT](FILEFORMAT.md) for the port number variables. 51 52### Test servers 53 54 The test suite runs stand-alone servers on random ports to which it makes 55 requests. For SSL tests, it runs stunnel to handle encryption to the regular 56 servers. For SSH, it runs a standard OpenSSH server. For SOCKS4/5 tests SSH 57 is used to perform the SOCKS functionality and requires a SSH client and 58 server. 59 60 The listen port numbers for the test servers are picked randomly to allow 61 users to run multiple test cases concurrently and to not collide with other 62 existing services that might listen to ports on the machine. 63 64 The HTTP server supports listening on a Unix domain socket, the default 65 location is 'http.sock'. 66 67### Run 68 69 `./configure && make && make test`. This builds the test suite support code 70 and invokes the 'runtests.pl' perl script to run all the tests. Edit the top 71 variables of that script in case you have some specific needs, or run the 72 script manually (after the support code has been built). 73 74 The script breaks on the first test that doesn't do OK. Use `-a` to prevent 75 the script from aborting on the first error. Run the script with `-v` for 76 more verbose output. Use `-d` to run the test servers with debug output 77 enabled as well. Specifying `-k` keeps all the log files generated by the 78 test intact. 79 80 Use `-s` for shorter output, or pass test numbers to run specific tests only 81 (like `./runtests.pl 3 4` to test 3 and 4 only). It also supports test case 82 ranges with 'to', as in `./runtests.pl 3 to 9` which runs the seven tests 83 from 3 to 9. Any test numbers starting with ! are disabled, as are any test 84 numbers found in the files `data/DISABLED` or `data/DISABLED.local` (one per 85 line). The latter is meant for local temporary disables and will be ignored 86 by git. 87 88 Test cases mentioned in `DISABLED` can still be run if `-f` is provided. 89 90 When `-s` is not present, each successful test will display on one line the 91 test number and description and on the next line a set of flags, the test 92 result, current test sequence, total number of tests to be run and an 93 estimated amount of time to complete the test run. The flags consist of 94 these letters describing what is checked in this test: 95 96 s stdout 97 d data 98 u upload 99 p protocol 100 o output 101 e exit code 102 m memory 103 v valgrind 104 105### Shell startup scripts 106 107 Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly 108 influenced by the output of system wide or user specific shell startup 109 scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which 110 output text messages or escape sequences on user login. When these shell 111 startup messages or escape sequences are output they might corrupt the 112 expected stream of data which flows to the sftp-server or from the ssh 113 client which can result in bad test behavior or even prevent the test 114 server from running. 115 116 If the test suite ssh or sftp server fails to start up and logs the message 117 'Received message too long' then you are certainly suffering the unwanted 118 output of a shell startup script. Locate, cleanup or adjust the shell 119 script. 120 121### Memory test 122 123 The test script will check that all allocated memory is freed properly IF 124 curl has been built with the `CURLDEBUG` define set. The script will 125 automatically detect if that is the case, and it will use the 126 'memanalyze.pl' script to analyze the memory debugging output. 127 128 Also, if you run tests on a machine where valgrind is found, the script will 129 use valgrind to run the test with (unless you use `-n`) to further verify 130 correctness. 131 132 runtests.pl's `-t` option will enable torture testing mode, which runs each 133 test many times and makes each different memory allocation fail on each 134 successive run. This tests the out of memory error handling code to ensure 135 that memory leaks do not occur even in those situations. It can help to 136 compile curl with `CPPFLAGS=-DMEMDEBUG_LOG_SYNC` when using this option, to 137 ensure that the memory log file is properly written even if curl crashes. 138 139### Debug 140 141 If a test case fails, you can conveniently get the script to invoke the 142 debugger (gdb) for you with the server running and the exact same command 143 line parameters that failed. Just invoke `runtests.pl <test number> -g` and 144 then just type 'run' in the debugger to perform the command through the 145 debugger. 146 147### Logs 148 149 All logs are generated in the log/ subdirectory (it is emptied first in the 150 runtests.pl script). They remain in there after a test run. 151 152### Test input files 153 154 All test cases are put in the `data/` subdirectory. Each test is stored in 155 the file named according to the test number. 156 157 See [FILEFORMAT.md](FILEFORMAT.md) for a description of the test case file 158 format. 159 160### Code coverage 161 162 gcc provides a tool that can determine the code coverage figures for the 163 test suite. To use it, configure curl with `CFLAGS='-fprofile-arcs 164 -ftest-coverage -g -O0'`. Make sure you run the normal and torture tests to 165 get more full coverage, i.e. do: 166 167 make test 168 make test-torture 169 170 The graphical tool ggcov can be used to browse the source and create 171 coverage reports on *NIX hosts: 172 173 ggcov -r lib src 174 175 The text mode tool gcov may also be used, but it doesn't handle object files 176 in more than one directory very well. 177 178### Remote testing 179 180 The runtests.pl script provides some hooks to allow curl to be tested on a 181 machine where perl can not be run. The test framework in this case runs on 182 a workstation where perl is available, while curl itself is run on a remote 183 system using ssh or some other remote execution method. See the comments at 184 the beginning of runtests.pl for details. 185 186## Test case numbering 187 188 Test cases used to be numbered by category ranges, but the ranges filled 189 up. Subsets of tests can now be selected by passing keywords to the 190 runtests.pl script via the make `TFLAGS` variable. 191 192 New tests are added by finding a free number in `tests/data/Makefile.inc`. 193 194## Write tests 195 196 Here's a quick description on writing test cases. We basically have three 197 kinds of tests: the ones that test the curl tool, the ones that build small 198 applications and test libcurl directly and the unit tests that test 199 individual (possibly internal) functions. 200 201### test data 202 203 Each test has a master file that controls all the test data. What to read, 204 what the protocol exchange should look like, what exit code to expect and 205 what command line arguments to use etc. 206 207 These files are `tests/data/test[num]` where `[num]` is just a unique 208 identifier described above, and the XML-like file format of them is 209 described in the separate [FILEFORMAT.md](FILEFORMAT.md) document. 210 211### curl tests 212 213 A test case that runs the curl tool and verifies that it gets the correct 214 data, it sends the correct data, it uses the correct protocol primitives 215 etc. 216 217### libcurl tests 218 219 The libcurl tests are identical to the curl ones, except that they use a 220 specific and dedicated custom-built program to run instead of "curl". This 221 tool is built from source code placed in `tests/libtest` and if you want to 222 make a new libcurl test that is where you add your code. 223 224### unit tests 225 226 Unit tests are placed in `tests/unit`. There's a tests/unit/README 227 describing the specific set of checks and macros that may be used when 228 writing tests that verify behaviors of specific individual functions. 229 230 The unit tests depend on curl being built with debug enabled. 231