1lit - LLVM Integrated Tester 2============================ 3 4.. program:: lit 5 6SYNOPSIS 7-------- 8 9:program:`lit` [*options*] [*tests*] 10 11DESCRIPTION 12----------- 13 14:program:`lit` is a portable tool for executing LLVM and Clang style test 15suites, summarizing their results, and providing indication of failures. 16:program:`lit` is designed to be a lightweight testing tool with as simple a 17user interface as possible. 18 19:program:`lit` should be run with one or more *tests* to run specified on the 20command line. Tests can be either individual test files or directories to 21search for tests (see :ref:`test-discovery`). 22 23Each specified test will be executed (potentially in parallel) and once all 24tests have been run :program:`lit` will print summary information on the number 25of tests which passed or failed (see :ref:`test-status-results`). The 26:program:`lit` program will execute with a non-zero exit code if any tests 27fail. 28 29By default :program:`lit` will use a succinct progress display and will only 30print summary information for test failures. See :ref:`output-options` for 31options controlling the :program:`lit` progress display and output. 32 33:program:`lit` also includes a number of options for controlling how tests are 34executed (specific features may depend on the particular test format). See 35:ref:`execution-options` for more information. 36 37Finally, :program:`lit` also supports additional options for only running a 38subset of the options specified on the command line, see 39:ref:`selection-options` for more information. 40 41:program:`lit` parses options from the environment variable ``LIT_OPTS`` after 42parsing options from the command line. ``LIT_OPTS`` is primarily useful for 43supplementing or overriding the command-line options supplied to :program:`lit` 44by ``check`` targets defined by a project's build system. 45 46Users interested in the :program:`lit` architecture or designing a 47:program:`lit` testing implementation should see :ref:`lit-infrastructure`. 48 49GENERAL OPTIONS 50--------------- 51 52.. option:: -h, --help 53 54 Show the :program:`lit` help message. 55 56.. option:: -j N, --workers=N 57 58 Run ``N`` tests in parallel. By default, this is automatically chosen to 59 match the number of detected available CPUs. 60 61.. option:: --config-prefix=NAME 62 63 Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for 64 test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`. 65 66.. option:: -D NAME[=VALUE], --param NAME[=VALUE] 67 68 Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty 69 string if not given). The meaning and use of these parameters is test suite 70 dependent. 71 72.. _output-options: 73 74OUTPUT OPTIONS 75-------------- 76 77.. option:: -q, --quiet 78 79 Suppress any output except for test failures. 80 81.. option:: -s, --succinct 82 83 Show less output, for example don't show information on tests that pass. 84 85.. option:: -v, --verbose 86 87 Show more information on test failures, for example the entire test output 88 instead of just the test result. 89 90.. option:: -vv, --echo-all-commands 91 92 Echo all commands to stdout, as they are being executed. 93 This can be valuable for debugging test failures, as the last echoed command 94 will be the one which has failed. 95 :program:`lit` normally inserts a no-op command (``:`` in the case of bash) 96 with argument ``'RUN: at line N'`` before each command pipeline, and this 97 option also causes those no-op commands to be echoed to stdout to help you 98 locate the source line of the failed command. 99 This option implies ``--verbose``. 100 101.. option:: -a, --show-all 102 103 Show more information about all tests, for example the entire test 104 commandline and output. 105 106.. option:: --no-progress-bar 107 108 Do not use curses based progress bar. 109 110.. option:: --show-unsupported 111 112 Show the names of unsupported tests. 113 114.. option:: --show-xfail 115 116 Show the names of tests that were expected to fail. 117 118.. _execution-options: 119 120EXECUTION OPTIONS 121----------------- 122 123.. option:: --path=PATH 124 125 Specify an additional ``PATH`` to use when searching for executables in tests. 126 127.. option:: --vg 128 129 Run individual tests under valgrind (using the memcheck tool). The 130 ``--error-exitcode`` argument for valgrind is used so that valgrind failures 131 will cause the program to exit with a non-zero status. 132 133 When this option is enabled, :program:`lit` will also automatically provide a 134 "``valgrind``" feature that can be used to conditionally disable (or expect 135 failure in) certain tests. 136 137.. option:: --vg-arg=ARG 138 139 When :option:`--vg` is used, specify an additional argument to pass to 140 :program:`valgrind` itself. 141 142.. option:: --vg-leak 143 144 When :option:`--vg` is used, enable memory leak checks. When this option is 145 enabled, :program:`lit` will also automatically provide a "``vg_leak``" 146 feature that can be used to conditionally disable (or expect failure in) 147 certain tests. 148 149.. option:: --time-tests 150 151 Track the wall time individual tests take to execute and includes the results 152 in the summary output. This is useful for determining which tests in a test 153 suite take the most time to execute. Note that this option is most useful 154 with ``-j 1``. 155 156.. _selection-options: 157 158SELECTION OPTIONS 159----------------- 160 161.. option:: --max-failures N 162 163 Stop execution after the given number ``N`` of failures. 164 An integer argument should be passed on the command line 165 prior to execution. 166 167.. option:: --max-tests=N 168 169 Run at most ``N`` tests and then terminate. 170 171.. option:: --max-time=N 172 173 Spend at most ``N`` seconds (approximately) running tests and then terminate. 174 Note that this is not an alias for :option:`--timeout`; the two are 175 different kinds of maximums. 176 177.. option:: --num-shards=M 178 179 Divide the set of selected tests into ``M`` equal-sized subsets or 180 "shards", and run only one of them. Must be used with the 181 ``--run-shard=N`` option, which selects the shard to run. The environment 182 variable ``LIT_NUM_SHARDS`` can also be used in place of this 183 option. These two options provide a coarse mechanism for partitioning large 184 testsuites, for parallel execution on separate machines (say in a large 185 testing farm). 186 187.. option:: --run-shard=N 188 189 Select which shard to run, assuming the ``--num-shards=M`` option was 190 provided. The two options must be used together, and the value of ``N`` 191 must be in the range ``1..M``. The environment variable 192 ``LIT_RUN_SHARD`` can also be used in place of this option. 193 194.. option:: --shuffle 195 196 Run the tests in a random order. 197 198.. option:: --timeout=N 199 200 Spend at most ``N`` seconds (approximately) running each individual test. 201 ``0`` means no time limit, and ``0`` is the default. Note that this is not an 202 alias for :option:`--max-time`; the two are different kinds of maximums. 203 204.. option:: --filter=REGEXP 205 206 Run only those tests whose name matches the regular expression specified in 207 ``REGEXP``. The environment variable ``LIT_FILTER`` can be also used in place 208 of this option, which is especially useful in environments where the call 209 to ``lit`` is issued indirectly. 210 211ADDITIONAL OPTIONS 212------------------ 213 214.. option:: --debug 215 216 Run :program:`lit` in debug mode, for debugging configuration issues and 217 :program:`lit` itself. 218 219.. option:: --show-suites 220 221 List the discovered test suites and exit. 222 223.. option:: --show-tests 224 225 List all of the discovered tests and exit. 226 227EXIT STATUS 228----------- 229 230:program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS 231results. Otherwise, it will exit with the status 0. Other exit codes are used 232for non-test related failures (for example a user error or an internal program 233error). 234 235.. _test-discovery: 236 237TEST DISCOVERY 238-------------- 239 240The inputs passed to :program:`lit` can be either individual tests, or entire 241directories or hierarchies of tests to run. When :program:`lit` starts up, the 242first thing it does is convert the inputs into a complete list of tests to run 243as part of *test discovery*. 244 245In the :program:`lit` model, every test must exist inside some *test suite*. 246:program:`lit` resolves the inputs specified on the command line to test suites 247by searching upwards from the input path until it finds a :file:`lit.cfg` or 248:file:`lit.site.cfg` file. These files serve as both a marker of test suites 249and as configuration files which :program:`lit` loads in order to understand 250how to find and run the tests inside the test suite. 251 252Once :program:`lit` has mapped the inputs into test suites it traverses the 253list of inputs adding tests for individual files and recursively searching for 254tests in directories. 255 256This behavior makes it easy to specify a subset of tests to run, while still 257allowing the test suite configuration to control exactly how tests are 258interpreted. In addition, :program:`lit` always identifies tests by the test 259suite they are in, and their relative path inside the test suite. For 260appropriately configured projects, this allows :program:`lit` to provide 261convenient and flexible support for out-of-tree builds. 262 263.. _test-status-results: 264 265TEST STATUS RESULTS 266------------------- 267 268Each test ultimately produces one of the following eight results: 269 270**PASS** 271 272 The test succeeded. 273 274**FLAKYPASS** 275 276 The test succeeded after being re-run more than once. This only applies to 277 tests containing an ``ALLOW_RETRIES:`` annotation. 278 279**XFAIL** 280 281 The test failed, but that is expected. This is used for test formats which allow 282 specifying that a test does not currently work, but wish to leave it in the test 283 suite. 284 285**XPASS** 286 287 The test succeeded, but it was expected to fail. This is used for tests which 288 were specified as expected to fail, but are now succeeding (generally because 289 the feature they test was broken and has been fixed). 290 291**FAIL** 292 293 The test failed. 294 295**UNRESOLVED** 296 297 The test result could not be determined. For example, this occurs when the test 298 could not be run, the test itself is invalid, or the test was interrupted. 299 300**UNSUPPORTED** 301 302 The test is not supported in this environment. This is used by test formats 303 which can report unsupported tests. 304 305**TIMEOUT** 306 307 The test was run, but it timed out before it was able to complete. This is 308 considered a failure. 309 310Depending on the test format tests may produce additional information about 311their status (generally only for failures). See the :ref:`output-options` 312section for more information. 313 314.. _lit-infrastructure: 315 316LIT INFRASTRUCTURE 317------------------ 318 319This section describes the :program:`lit` testing architecture for users interested in 320creating a new :program:`lit` testing implementation, or extending an existing one. 321 322:program:`lit` proper is primarily an infrastructure for discovering and running 323arbitrary tests, and to expose a single convenient interface to these 324tests. :program:`lit` itself doesn't know how to run tests, rather this logic is 325defined by *test suites*. 326 327TEST SUITES 328~~~~~~~~~~~ 329 330As described in :ref:`test-discovery`, tests are always located inside a *test 331suite*. Test suites serve to define the format of the tests they contain, the 332logic for finding those tests, and any additional information to run the tests. 333 334:program:`lit` identifies test suites as directories containing ``lit.cfg`` or 335``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are 336initially discovered by recursively searching up the directory hierarchy for 337all the input files passed on the command line. You can use 338:option:`--show-suites` to display the discovered test suites at startup. 339 340Once a test suite is discovered, its config file is loaded. Config files 341themselves are Python modules which will be executed. When the config file is 342executed, two important global variables are predefined: 343 344**lit_config** 345 346 The global **lit** configuration object (a *LitConfig* instance), which defines 347 the builtin test formats, global configuration parameters, and other helper 348 routines for implementing test configurations. 349 350**config** 351 352 This is the config object (a *TestingConfig* instance) for the test suite, 353 which the config file is expected to populate. The following variables are also 354 available on the *config* object, some of which must be set by the config and 355 others are optional or predefined: 356 357 **name** *[required]* The name of the test suite, for use in reports and 358 diagnostics. 359 360 **test_format** *[required]* The test format object which will be used to 361 discover and run tests in the test suite. Generally this will be a builtin test 362 format available from the *lit.formats* module. 363 364 **test_source_root** The filesystem path to the test suite root. For out-of-dir 365 builds this is the directory that will be scanned for tests. 366 367 **test_exec_root** For out-of-dir builds, the path to the test suite root inside 368 the object directory. This is where tests will be run and temporary output files 369 placed. 370 371 **environment** A dictionary representing the environment to use when executing 372 tests in the suite. 373 374 **suffixes** For **lit** test formats which scan directories for tests, this 375 variable is a list of suffixes to identify test files. Used by: *ShTest*. 376 377 **substitutions** For **lit** test formats which substitute variables into a test 378 script, the list of substitutions to perform. Used by: *ShTest*. 379 380 **unsupported** Mark an unsupported directory, all tests within it will be 381 reported as unsupported. Used by: *ShTest*. 382 383 **parent** The parent configuration, this is the config object for the directory 384 containing the test suite, or None. 385 386 **root** The root configuration. This is the top-most :program:`lit` configuration in 387 the project. 388 389 **pipefail** Normally a test using a shell pipe fails if any of the commands 390 on the pipe fail. If this is not desired, setting this variable to false 391 makes the test fail only if the last command in the pipe fails. 392 393 **available_features** A set of features that can be used in `XFAIL`, 394 `REQUIRES`, and `UNSUPPORTED` directives. 395 396TEST DISCOVERY 397~~~~~~~~~~~~~~ 398 399Once test suites are located, :program:`lit` recursively traverses the source 400directory (following *test_source_root*) looking for tests. When :program:`lit` 401enters a sub-directory, it first checks to see if a nested test suite is 402defined in that directory. If so, it loads that test suite recursively, 403otherwise it instantiates a local test config for the directory (see 404:ref:`local-configuration-files`). 405 406Tests are identified by the test suite they are contained within, and the 407relative path inside that suite. Note that the relative path may not refer to 408an actual file on disk; some test formats (such as *GoogleTest*) define 409"virtual tests" which have a path that contains both the path to the actual 410test file and a subpath to identify the virtual test. 411 412.. _local-configuration-files: 413 414LOCAL CONFIGURATION FILES 415~~~~~~~~~~~~~~~~~~~~~~~~~ 416 417When :program:`lit` loads a subdirectory in a test suite, it instantiates a 418local test configuration by cloning the configuration for the parent directory 419--- the root of this configuration chain will always be a test suite. Once the 420test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file 421in the subdirectory. If present, this file will be loaded and can be used to 422specialize the configuration for each individual directory. This facility can 423be used to define subdirectories of optional tests, or to change other 424configuration parameters --- for example, to change the test format, or the 425suffixes which identify test files. 426 427SUBSTITUTIONS 428~~~~~~~~~~~~~ 429 430:program:`lit` allows patterns to be substituted inside RUN commands. It also 431provides the following base set of substitutions, which are defined in 432TestRunner.py: 433 434 ======================= ============== 435 Macro Substitution 436 ======================= ============== 437 %s source path (path to the file currently being run) 438 %S source dir (directory of the file currently being run) 439 %p same as %S 440 %{pathsep} path separator 441 %t temporary file name unique to the test 442 %basename_t The last path component of %t but without the ``.tmp`` extension 443 %T parent directory of %t (not unique, deprecated, do not use) 444 %% % 445 %/s %s but ``\`` is replaced by ``/`` 446 %/S %S but ``\`` is replaced by ``/`` 447 %/p %p but ``\`` is replaced by ``/`` 448 %/t %t but ``\`` is replaced by ``/`` 449 %/T %T but ``\`` is replaced by ``/`` 450 %{/s:regex_replacement} %/s but escaped for use in the replacement of a ``s@@@`` command in sed 451 %{/S:regex_replacement} %/S but escaped for use in the replacement of a ``s@@@`` command in sed 452 %{/p:regex_replacement} %/p but escaped for use in the replacement of a ``s@@@`` command in sed 453 %{/t:regex_replacement} %/t but escaped for use in the replacement of a ``s@@@`` command in sed 454 %{/T:regex_replacement} %/T but escaped for use in the replacement of a ``s@@@`` command in sed 455 %:s On Windows, %/s but a ``:`` is removed if its the second character. 456 Otherwise, %s but with a single leading ``/`` removed. 457 %:S On Windows, %/S but a ``:`` is removed if its the second character. 458 Otherwise, %S but with a single leading ``/`` removed. 459 %:p On Windows, %/p but a ``:`` is removed if its the second character. 460 Otherwise, %p but with a single leading ``/`` removed. 461 %:t On Windows, %/t but a ``:`` is removed if its the second character. 462 Otherwise, %t but with a single leading ``/`` removed. 463 %:T On Windows, %/T but a ``:`` is removed if its the second character. 464 Otherwise, %T but with a single leading ``/`` removed. 465 ======================= ============== 466 467Other substitutions are provided that are variations on this base set and 468further substitution patterns can be defined by each test module. See the 469modules :ref:`local-configuration-files`. 470 471By default, substitutions are expanded exactly once, so that if e.g. a 472substitution ``%build`` is defined in top of another substitution ``%cxx``, 473``%build`` will expand to ``%cxx`` textually, not to what ``%cxx`` expands to. 474However, if the ``recursiveExpansionLimit`` property of the ``TestingConfig`` 475is set to a non-negative integer, substitutions will be expanded recursively 476until that limit is reached. It is an error if the limit is reached and 477expanding substitutions again would yield a different result. 478 479More detailed information on substitutions can be found in the 480:doc:`../TestingGuide`. 481 482TEST RUN OUTPUT FORMAT 483~~~~~~~~~~~~~~~~~~~~~~ 484 485The :program:`lit` output for a test run conforms to the following schema, in 486both short and verbose modes (although in short mode no PASS lines will be 487shown). This schema has been chosen to be relatively easy to reliably parse by 488a machine (for example in buildbot log scraping), and for other tools to 489generate. 490 491Each test result is expected to appear on a line that matches: 492 493.. code-block:: none 494 495 <result code>: <test name> (<progress info>) 496 497where ``<result-code>`` is a standard test result such as PASS, FAIL, XFAIL, 498XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and 499REGRESSED are also allowed. 500 501The ``<test name>`` field can consist of an arbitrary string containing no 502newline. 503 504The ``<progress info>`` field can be used to report progress information such 505as (1/300) or can be empty, but even when empty the parentheses are required. 506 507Each test result may include additional (multiline) log information in the 508following format: 509 510.. code-block:: none 511 512 <log delineator> TEST '(<test name>)' <trailing delineator> 513 ... log message ... 514 <log delineator> 515 516where ``<test name>`` should be the name of a preceding reported test, ``<log 517delineator>`` is a string of "*" characters *at least* four characters long 518(the recommended length is 20), and ``<trailing delineator>`` is an arbitrary 519(unparsed) string. 520 521The following is an example of a test run output which consists of four tests A, 522B, C, and D, and a log message for the failing test C: 523 524.. code-block:: none 525 526 PASS: A (1 of 4) 527 PASS: B (2 of 4) 528 FAIL: C (3 of 4) 529 ******************** TEST 'C' FAILED ******************** 530 Test 'C' failed as a result of exit code 1. 531 ******************** 532 PASS: D (4 of 4) 533 534LIT EXAMPLE TESTS 535~~~~~~~~~~~~~~~~~ 536 537The :program:`lit` distribution contains several example implementations of 538test suites in the *ExampleTests* directory. 539 540SEE ALSO 541-------- 542 543valgrind(1) 544