1# curl test suite file format 2 3The curl test suite's file format is very simple and extensible, closely 4resembling XML. All data for a single test case resides in a single ASCII 5file. Labels mark the beginning and the end of all sections, and each label 6must be written in its own line. Comments are either XML-style (enclosed with 7`<!--` and `-->`) or shell script style (beginning with `#`) and must appear 8on their own lines and not alongside actual test data. Most test data files 9are syntactically valid XML, although a few files are not (lack of support for 10character entities and the preservation of CR/LF characters at the end of 11lines are the biggest differences). 12 13Each test case source exists as a file matching the format 14`tests/data/testNUM`, where NUM is the unique test number, and must begin with 15a 'testcase' tag, which encompasses the remainder of the file. 16 17# Preprocessing 18 19When a test is to be executed, the source file is first preprocessed and 20variables are substituted by the their respective contents and the output 21version of the test file is stored as `log/testNUM`. That version is what will 22be read and used by the test servers. 23 24## Base64 Encoding 25 26In the preprocess stage, a special instruction can be used to have runtests.pl 27base64 encode a certain section and insert in the generated output file. This 28is in particular good for test cases where the test tool is expected to pass 29in base64 encoded content that might use dynamic information that is unique 30for this particular test invocation, like the server port number. 31 32To insert a base64 encoded string into the output, use this syntax: 33 34 %b64[ data to encode ]b64% 35 36The data to encode can then use any of the existing variables mentioned below, 37or even percent-encoded individual bytes. As an example, insert the HTTP 38server's port number (in ASCII) followed by a space and the hexadecimal byte 399a: 40 41 %b64[%HTTPPORT %9a]b64% 42 43## Hexadecimal decoding 44 45In the preprocess stage, a special instruction can be used to have runtests.pl 46generate a sequence of binary bytes. 47 48To insert a sequence of bytes from a hex encoded string, use this syntax: 49 50 %hex[ %XX-encoded data to decode ]hex% 51 52For example, to insert the binary octets 0, 1 and 255 into the test file: 53 54 %hex[ %00%01%FF ]hex% 55 56## Repeat content 57 58In the preprocess stage, a special instruction can be used to have runtests.pl 59generate a repetetive sequence of bytes. 60 61To insert a sequence of repeat bytes, use this syntax to make the `<string>` 62get repeated `<number>` of times. The number has to be 1 or large and the 63string may contain `%HH` hexadecimal codes: 64 65 %repeat[<number> x <string>]% 66 67For example, to insert the word hello a 100 times: 68 69 %repeat[100 x hello]% 70 71## Conditional lines 72 73Lines in the test file can be made to appear conditionally on a specific 74feature (see the "features" section below) being set or not set. If the 75specific feature is present, the following lines will be output, otherwise it 76outputs nothing, until a following else or endif clause. Like this: 77 78 %if brotli 79 Accept-Encoding 80 %endif 81 82It can also check for the inversed condition, so if the feature us *not* set by 83the use of an exclamation mark: 84 85 %if !brotli 86 Accept-Encoding: not-brotli 87 %endif 88 89You can also make an "else" clause to get output for the opposite condition, 90like: 91 92 %if brotli 93 Accept-Encoding: brotli 94 %else 95 Accept-Encoding: nothing 96 %endif 97 98**Note** that there can be no nested conditions. You can only do one 99conditional at a time and you can only check for a single feature in it. 100 101# Variables 102 103When the test is preprocessed, a range of "variables" in the test file will be 104replaced by their content at that time. 105 106Available substitute variables include: 107 108- `%CLIENT6IP` - IPv6 address of the client running curl 109- `%CLIENTIP` - IPv4 address of the client running curl 110- `%CURL` - Path to the curl executable 111- `%FILE_PWD` - Current directory, on windows prefixed with a slash 112- `%FTP6PORT` - IPv6 port number of the FTP server 113- `%FTPPORT` - Port number of the FTP server 114- `%FTPSPORT` - Port number of the FTPS server 115- `%FTPTIME2` - Timeout in seconds that should be just sufficient to receive a 116 response from the test FTP server 117- `%FTPTIME3` - Even longer than %FTPTIME2 118- `%GOPHER6PORT` - IPv6 port number of the Gopher server 119- `%GOPHERPORT` - Port number of the Gopher server 120- `%GOPHERSPORT` - Port number of the Gophers server 121- `%HOST6IP` - IPv6 address of the host running this test 122- `%HOSTIP` - IPv4 address of the host running this test 123- `%HTTP6PORT` - IPv6 port number of the HTTP server 124- `%HTTPPORT` - Port number of the HTTP server 125- `%HTTP2PORT` - Port number of the HTTP/2 server 126- `%HTTPSPORT` - Port number of the HTTPS server 127- `%HTTPSPROXYPORT` - Port number of the HTTPS-proxy 128- `%HTTPTLS6PORT` - IPv6 port number of the HTTP TLS server 129- `%HTTPTLSPORT` - Port number of the HTTP TLS server 130- `%HTTPUNIXPATH` - Path to the Unix socket of the HTTP server 131- `%IMAP6PORT` - IPv6 port number of the IMAP server 132- `%IMAPPORT` - Port number of the IMAP server 133- `%MQTTPORT` - Port number of the MQTT server 134- `%TELNETPORT` - Port number of the telnet server 135- `%NOLISTENPORT` - Port number where no service is listening 136- `%POP36PORT` - IPv6 port number of the POP3 server 137- `%POP3PORT` - Port number of the POP3 server 138- `%POSIX_PWD` - Current directory somewhat mingw friendly 139- `%PROXYPORT` - Port number of the HTTP proxy 140- `%PWD` - Current directory 141- `%RTSP6PORT` - IPv6 port number of the RTSP server 142- `%RTSPPORT` - Port number of the RTSP server 143- `%SMBPORT` - Port number of the SMB server 144- `%SMBSPORT` - Port number of the SMBS server 145- `%SMTP6PORT` - IPv6 port number of the SMTP server 146- `%SMTPPORT` - Port number of the SMTP server 147- `%SOCKSPORT` - Port number of the SOCKS4/5 server 148- `%SRCDIR` - Full path to the source dir 149- `%SSHPORT` - Port number of the SCP/SFTP server 150- `%SSHSRVMD5` - MD5 of SSH server's public key 151- `%SSHSRVSHA256` - SHA256 of SSH server's public key 152- `%SSH_PWD` - Current directory friendly for the SSH server 153- `%TESTNUMBER` - Number of the test case 154- `%TFTP6PORT` - IPv6 port number of the TFTP server 155- `%TFTPPORT` - Port number of the TFTP server 156- `%USER` - Login ID of the user running the test 157- `%VERSION` - the full version number of the tested curl 158 159# `<testcase>` 160 161Each test is always specified entirely within the testcase tag. Each test case 162is split up in four main sections: `info`, `reply`, `client` and `verify`. 163 164- **info** provides information about the test case 165 166- **reply** is used for the server to know what to send as a reply for the 167requests curl sends 168 169- **client** defines how the client should behave 170 171- **verify** defines how to verify that the data stored after a command has 172been run ended up correctly 173 174Each main section has a number of available subsections that can be specified, 175that will be checked/used if specified. 176 177## `<info>` 178 179### `<keywords>` 180A newline-separated list of keywords describing what this test case uses and 181tests. Try to use already used keywords. These keywords will be used for 182statistical/informational purposes and for choosing or skipping classes of 183tests. "Keywords" must begin with an alphabetic character, "-", "[" or "{" 184and may actually consist of multiple words separated by spaces which are 185treated together as a single identifier. 186 187When using curl built with Hyper, the keywords must include HTTP or HTTPS for 188'hyper mode' to kick in and make line ending checks work for tests. 189## `<reply>` 190 191### `<data [nocheck="yes"] [sendzero="yes"] [base64="yes"] [hex="yes"]>` 192 193data to be sent to the client on its request and later verified that it 194arrived safely. Set `nocheck="yes"` to prevent the test script from verifying 195the arrival of this data. 196 197If the data contains `swsclose` anywhere within the start and end tag, and 198this is a HTTP test, then the connection will be closed by the server after 199this response is sent. If not, the connection will be kept persistent. 200 201If the data contains `swsbounce` anywhere within the start and end tag, the 202HTTP server will detect if this is a second request using the same test and 203part number and will then increase the part number with one. This is useful 204for auth tests and similar. 205 206`sendzero=yes` means that the (FTP) server will "send" the data even if the 207size is zero bytes. Used to verify curl's behavior on zero bytes transfers. 208 209`base64=yes` means that the data provided in the test-file is a chunk of data 210encoded with base64. It is the only way a test case can contain binary 211data. (This attribute can in fact be used on any section, but it doesn't make 212much sense for other sections than "data"). 213 214`hex=yes` means that the data is a sequence of hex pairs. It will get decoded 215and used as "raw" data. 216 217For FTP file listings, the `<data>` section will be used *only* if you make 218sure that there has been a CWD done first to a directory named `test-[num]` 219where [num] is the test case number. Otherwise the ftp server can't know from 220which test file to load the list content. 221 222### `<dataNUM>` 223 224Send back this contents instead of the <data> one. The num is set by: 225 226 - The test number in the request line is >10000 and this is the remainder 227 of [test case number]%10000. 228 - The request was HTTP and included digest details, which adds 1000 to NUM 229 - If a HTTP request is NTLM type-1, it adds 1001 to num 230 - If a HTTP request is NTLM type-3, it adds 1002 to num 231 - If a HTTP request is Basic and num is already >=1000, it adds 1 to num 232 - If a HTTP request is Negotiate, num gets incremented by one for each 233 request with Negotiate authorization header on the same test case. 234 235Dynamically changing num in this way allows the test harness to be used to 236test authentication negotiation where several different requests must be sent 237to complete a transfer. The response to each request is found in its own data 238section. Validating the entire negotiation sequence can be done by specifying 239a datacheck section. 240 241### `<connect>` 242The connect section is used instead of the 'data' for all CONNECT 243requests. The remainder of the rules for the data section then apply but with 244a connect prefix. 245 246### `<datacheck [mode="text"] [nonewline="yes"]>` 247if the data is sent but this is what should be checked afterwards. If 248`nonewline=yes` is set, runtests will cut off the trailing newline from the 249data before comparing with the one actually received by the client. 250 251Use the `mode="text"` attribute if the output is in text mode on platforms 252that have a text/binary difference. 253 254### `<datacheckNUM [nonewline="yes"] [mode="text"]>` 255The contents of numbered datacheck sections are appended to the non-numbered 256one. 257 258### `<size>` 259number to return on a ftp SIZE command (set to -1 to make this command fail) 260 261### `<mdtm>` 262what to send back if the client sends a (FTP) MDTM command, set to -1 to 263have it return that the file doesn't exist 264 265### `<postcmd>` 266special purpose server-command to control its behavior *after* the 267reply is sent 268For HTTP/HTTPS, these are supported: 269 270`wait [secs]` - Pause for the given time 271 272### `<servercmd>` 273Special-commands for the server. 274 275The first line of this file will always be set to `Testnum [number]` by the 276test script, to allow servers to read that to know what test the client is 277about to issue. 278 279#### For FTP/SMTP/POP/IMAP 280 281- `REPLY [command] [return value] [response string]` - Changes how the server 282 responds to the [command]. [response string] is evaluated as a perl string, 283 so it can contain embedded \r\n, for example. There's a special [command] 284 named "welcome" (without quotes) which is the string sent immediately on 285 connect as a welcome. 286- `REPLYLF` (like above but sends the response terminated with LF-only and not 287 CRLF) 288- `COUNT [command] [num]` - Do the `REPLY` change for `[command]` only `[num]` 289 times and then go back to the built-in approach 290- `DELAY [command] [secs]` - Delay responding to this command for the given 291 time 292- `RETRWEIRDO` - Enable the "weirdo" RETR case when multiple response lines 293 appear at once when a file is transferred 294- `RETRNOSIZE` - Make sure the RETR response doesn't contain the size of the 295 file 296- `NOSAVE` - Don't actually save what is received 297- `SLOWDOWN` - Send FTP responses with 0.01 sec delay between each byte 298- `PASVBADIP` - makes PASV send back an illegal IP in its 227 response 299- `CAPA [capabilities]` - Enables support for and specifies a list of space 300 separated capabilities to return to the client for the IMAP `CAPABILITY`, 301 POP3 `CAPA` and SMTP `EHLO` commands 302- `AUTH [mechanisms]` - Enables support for SASL authentication and specifies 303 a list of space separated mechanisms for IMAP, POP3 and SMTP 304- `STOR [msg]` respond with this instead of default after `STOR` 305 306#### For HTTP/HTTPS 307 308- `auth_required` if this is set and a POST/PUT is made without auth, the 309 server will NOT wait for the full request body to get sent 310- `idle` - do nothing after receiving the request, just "sit idle" 311- `stream` - continuously send data to the client, never-ending 312- `writedelay: [secs]` delay this amount between reply packets 313- `skip: [num]` - instructs the server to ignore reading this many bytes from 314 a PUT or POST request 315- `rtp: part [num] channel [num] size [num]` - stream a fake RTP packet for 316 the given part on a chosen channel with the given payload size 317- `connection-monitor` - When used, this will log `[DISCONNECT]` to the 318 `server.input` log when the connection is disconnected. 319- `upgrade` - when an HTTP upgrade header is found, the server will upgrade to 320 http2 321- `swsclose` - instruct server to close connection after response 322- `no-expect` - don't read the request body if Expect: is present 323 324#### For TFTP 325`writedelay: [secs]` delay this amount between reply packets (each packet 326 being 512 bytes payload) 327 328## `<client>` 329 330### `<server>` 331What server(s) this test case requires/uses. Available servers: 332 333- `file` 334- `ftp-ipv6` 335- `ftp` 336- `ftps` 337- `gopher` 338- `gophers` 339- `http-ipv6` 340- `http-proxy` 341- `http-unix` 342- `http/2` 343- `http` 344- `https` 345- `httptls+srp-ipv6` 346- `httptls+srp` 347- `imap` 348- `mqtt` 349- `none` 350- `pop3` 351- `rtsp-ipv6` 352- `rtsp` 353- `scp` 354- `sftp` 355- `smtp` 356- `socks4` 357- `socks5` 358 359Give only one per line. This subsection is mandatory. 360 361### `<features>` 362A list of features that MUST be present in the client/library for this test to 363be able to run. If a required feature is not present then the test will be 364SKIPPED. 365 366Alternatively a feature can be prefixed with an exclamation mark to indicate a 367feature is NOT required. If the feature is present then the test will be 368SKIPPED. 369 370Features testable here are: 371 372- `alt-svc` 373- `c-ares` 374- `cookies` 375- `crypto` 376- `debug` 377- `DoH` 378- `getrlimit` 379- `GnuTLS` 380- `GSS-API` 381- `HSTS` 382- `HTTP-auth` 383- `http/2` 384- `hyper` 385- `idn` 386- `ipv6` 387- `Kerberos` 388- `large_file` 389- `ld_preload` 390- `libz` 391- `manual` 392- `Mime` 393- `netrc` 394- `NSS` 395- `NTLM` 396- `OpenSSL` 397- `parsedate` 398- `proxy` 399- `PSL` 400- `Schannel` 401- `sectransp` 402- `shuffle-dns` 403- `socks` 404- `SPNEGO` 405- `SSL` 406- `SSLpinning` 407- `SSPI` 408- `threaded-resolver` 409- `TLS-SRP` 410- `TrackMemory` 411- `typecheck` 412- `Unicode` 413- `unittest` 414- `unix-sockets` 415- `verbose-strings` 416- `wakeup` 417- `win32` 418 419as well as each protocol that curl supports. A protocol only needs to be 420specified if it is different from the server (useful when the server 421is `none`). 422 423### `<killserver>` 424Using the same syntax as in `<server>` but when mentioned here these servers 425are explicitly KILLED when this test case is completed. Only use this if there 426is no other alternatives. Using this of course requires subsequent tests to 427restart servers. 428 429### `<precheck>` 430A command line that if set gets run by the test script before the test. If an 431output is displayed by the command or if the return code is non-zero, the test 432will be skipped and the (single-line) output will be displayed as reason for 433not running the test. 434 435### `<postcheck>` 436A command line that if set gets run by the test script after the test. If 437the command exists with a non-zero status code, the test will be considered 438to have failed. 439 440### `<tool>` 441Name of tool to invoke instead of "curl". This tool must be built and exist 442either in the libtest/ directory (if the tool name starts with 'lib') or in 443the unit/ directory (if the tool name starts with 'unit'). 444 445### `<name>` 446Brief test case description, shown when the test runs. 447 448### `<setenv>` 449 variable1=contents1 450 variable2=contents2 451 452Set the given environment variables to the specified value before the actual 453command is run. They are cleared again after the command has been run. 454 455### `<command [option="no-output/no-include/force-output/binary-trace"] [timeout="secs"][delay="secs"][type="perl/shell"]>` 456Command line to run. 457 458Note that the URL that gets passed to the server actually controls what data 459that is returned. The last slash in the URL must be followed by a number. That 460number (N) will be used by the test-server to load test case N and return the 461data that is defined within the `<reply><data></data></reply>` section. 462 463If there's no test number found above, the HTTP test server will use the 464number following the last dot in the given hostname (made so that a CONNECT 465can still pass on test number) so that "foo.bar.123" gets treated as test case 466123. Alternatively, if an IPv6 address is provided to CONNECT, the last 467hexadecimal group in the address will be used as the test number! For example 468the address "[1234::ff]" would be treated as test case 255. 469 470Set `type="perl"` to write the test case as a perl script. It implies that 471there's no memory debugging and valgrind gets shut off for this test. 472 473Set `type="shell"` to write the test case as a shell script. It implies that 474there's no memory debugging and valgrind gets shut off for this test. 475 476Set `option="no-output"` to prevent the test script to slap on the `--output` 477argument that directs the output to a file. The `--output` is also not added 478if the verify/stdout section is used. 479 480Set `option="force-output"` to make use of `--output` even when the test is 481otherwise written to verify stdout. 482 483Set `option="no-include"` to prevent the test script to slap on the 484`--include` argument. 485 486Set `option="binary-trace"` to use `--trace` instead of `--trace-ascii` for 487tracing. Suitable for binary-oriented protocols such as MQTT. 488 489Set `timeout="secs"` to override default server logs advisor read lock 490timeout. This timeout is used by the test harness, once that the command has 491completed execution, to wait for the test server to write out server side log 492files and remove the lock that advised not to read them. The "secs" parameter 493is the not negative integer number of seconds for the timeout. This `timeout` 494attribute is documented for completeness sake, but is deep test harness stuff 495and only needed for very singular and specific test cases. Avoid using it. 496 497Set `delay="secs"` to introduce a time delay once that the command has 498completed execution and before the `<postcheck>` section runs. The "secs" 499parameter is the not negative integer number of seconds for the delay. This 500'delay' attribute is intended for very specific test cases, and normally not 501needed. 502 503### `<file name="log/filename" [nonewline="yes"]>` 504This creates the named file with this content before the test case is run, 505which is useful if the test case needs a file to act on. 506 507If 'nonewline="yes"` is used, the created file will have the final newline 508stripped off. 509 510### `<stdin [nonewline="yes"]>` 511Pass this given data on stdin to the tool. 512 513If 'nonewline' is set, we will cut off the trailing newline of this given data 514before comparing with the one actually received by the client 515 516## `<verify>` 517### `<errorcode>` 518numerical error code curl is supposed to return. Specify a list of accepted 519error codes by separating multiple numbers with comma. See test 237 for an 520example. 521 522### `<strip>` 523One regex per line that is removed from the protocol dumps before the 524comparison is made. This is very useful to remove dependencies on dynamically 525changing protocol data such as port numbers or user-agent strings. 526 527### `<strippart>` 528One perl op per line that operates on the protocol dump. This is pretty 529advanced. Example: `s/^EPRT .*/EPRT stripped/`. 530 531### `<protocol [nonewline="yes"]>` 532 533the protocol dump curl should transmit, if 'nonewline' is set, we will cut off 534the trailing newline of this given data before comparing with the one actually 535sent by the client The `<strip>` and `<strippart>` rules are applied before 536comparisons are made. 537 538### `<proxy [nonewline="yes"]>` 539 540The protocol dump curl should transmit to a HTTP proxy (when the http-proxy 541server is used), if 'nonewline' is set, we will cut off the trailing newline 542of this given data before comparing with the one actually sent by the client 543The `<strip>` and `<strippart>` rules are applied before comparisons are made. 544 545### `<stderr [mode="text"] [nonewline="yes"]>` 546This verifies that this data was passed to stderr. 547 548Use the mode="text" attribute if the output is in text mode on platforms that 549have a text/binary difference. 550 551If 'nonewline' is set, we will cut off the trailing newline of this given data 552before comparing with the one actually received by the client 553 554### `<stdout [mode="text"] [nonewline="yes"]>` 555This verifies that this data was passed to stdout. 556 557Use the mode="text" attribute if the output is in text mode on platforms that 558have a text/binary difference. 559 560If 'nonewline' is set, we will cut off the trailing newline of this given data 561before comparing with the one actually received by the client 562 563### `<file name="log/filename" [mode="text"]>` 564The file's contents must be identical to this after the test is complete. Use 565the mode="text" attribute if the output is in text mode on platforms that have 566a text/binary difference. 567 568### `<file1>` 5691 to 4 can be appended to 'file' to compare more files. 570 571### `<file2>` 572 573### `<file3>` 574 575### `<file4>` 576 577### `<stripfile>` 578One perl op per line that operates on the output file or stdout before being 579compared with what is stored in the test file. This is pretty 580advanced. Example: "s/^EPRT .*/EPRT stripped/" 581 582### `<stripfile1>` 5831 to 4 can be appended to 'stripfile' to strip the corresponding <fileN> 584content 585 586### `<stripfile2>` 587 588### `<stripfile3>` 589 590### `<stripfile4>` 591 592### `<upload>` 593the contents of the upload data curl should have sent 594 595### `<valgrind>` 596disable - disables the valgrind log check for this test 597