1================================= 2LLVM Testing Infrastructure Guide 3================================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 TestSuiteMakefileGuide 12 13Overview 14======== 15 16This document is the reference manual for the LLVM testing 17infrastructure. It documents the structure of the LLVM testing 18infrastructure, the tools needed to use it, and how to add and run 19tests. 20 21Requirements 22============ 23 24In order to use the LLVM testing infrastructure, you will need all of the 25software required to build LLVM, as well as `Python <http://python.org>`_ 2.5 or 26later. 27 28LLVM testing infrastructure organization 29======================================== 30 31The LLVM testing infrastructure contains two major categories of tests: 32regression tests and whole programs. The regression tests are contained 33inside the LLVM repository itself under ``llvm/test`` and are expected 34to always pass -- they should be run before every commit. 35 36The whole programs tests are referred to as the "LLVM test suite" (or 37"test-suite") and are in the ``test-suite`` module in subversion. For 38historical reasons, these tests are also referred to as the "nightly 39tests" in places, which is less ambiguous than "test-suite" and remains 40in use although we run them much more often than nightly. 41 42Regression tests 43---------------- 44 45The regression tests are small pieces of code that test a specific 46feature of LLVM or trigger a specific bug in LLVM. The language they are 47written in depends on the part of LLVM being tested. These tests are driven by 48the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and 49are located in the ``llvm/test`` directory. 50 51Typically when a bug is found in LLVM, a regression test containing just 52enough code to reproduce the problem should be written and placed 53somewhere underneath this directory. For example, it can be a small 54piece of LLVM IR distilled from an actual application or benchmark. 55 56``test-suite`` 57-------------- 58 59The test suite contains whole programs, which are pieces of code which 60can be compiled and linked into a stand-alone program that can be 61executed. These programs are generally written in high level languages 62such as C or C++. 63 64These programs are compiled using a user specified compiler and set of 65flags, and then executed to capture the program output and timing 66information. The output of these programs is compared to a reference 67output to ensure that the program is being compiled correctly. 68 69In addition to compiling and executing programs, whole program tests 70serve as a way of benchmarking LLVM performance, both in terms of the 71efficiency of the programs generated as well as the speed with which 72LLVM compiles, optimizes, and generates code. 73 74The test-suite is located in the ``test-suite`` Subversion module. 75 76Debugging Information tests 77--------------------------- 78 79The test suite contains tests to check quality of debugging information. 80The test are written in C based languages or in LLVM assembly language. 81 82These tests are compiled and run under a debugger. The debugger output 83is checked to validate of debugging information. See README.txt in the 84test suite for more information . This test suite is located in the 85``debuginfo-tests`` Subversion module. 86 87Quick start 88=========== 89 90The tests are located in two separate Subversion modules. The 91regressions tests are in the main "llvm" module under the directory 92``llvm/test`` (so you get these tests for free with the main LLVM tree). 93Use ``make check-all`` to run the regression tests after building LLVM. 94 95The more comprehensive test suite that includes whole programs in C and C++ 96is in the ``test-suite`` module. See :ref:`test-suite Quickstart 97<test-suite-quickstart>` for more information on running these tests. 98 99Regression tests 100---------------- 101 102To run all of the LLVM regression tests, use the master Makefile in the 103``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM 104Makefile Guide <MakefileGuide>` for more details): 105 106.. code-block:: bash 107 108 % make -C llvm/test 109 110or: 111 112.. code-block:: bash 113 114 % make check 115 116If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you 117can run the LLVM and Clang tests simultaneously using: 118 119.. code-block:: bash 120 121 % make check-all 122 123To run the tests with Valgrind (Memcheck by default), use the ``LIT_ARGS`` make 124variable to pass the required options to lit. For example, you can use: 125 126.. code-block:: bash 127 128 % make check LIT_ARGS="-v --vg --vg-leak" 129 130to enable testing with valgrind and with leak checking enabled. 131 132To run individual tests or subsets of tests, you can use the ``llvm-lit`` 133script which is built as part of LLVM. For example, to run the 134``Integer/BitPacked.ll`` test by itself you can run: 135 136.. code-block:: bash 137 138 % llvm-lit ~/llvm/test/Integer/BitPacked.ll 139 140or to run all of the ARM CodeGen tests: 141 142.. code-block:: bash 143 144 % llvm-lit ~/llvm/test/CodeGen/ARM 145 146For more information on using the :program:`lit` tool, see ``llvm-lit --help`` 147or the :doc:`lit man page <CommandGuide/lit>`. 148 149Debugging Information tests 150--------------------------- 151 152To run debugging information tests simply checkout the tests inside 153clang/test directory. 154 155.. code-block:: bash 156 157 % cd clang/test 158 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests 159 160These tests are already set up to run as part of clang regression tests. 161 162Regression test structure 163========================= 164 165The LLVM regression tests are driven by :program:`lit` and are located in the 166``llvm/test`` directory. 167 168This directory contains a large array of small tests that exercise 169various features of LLVM and to ensure that regressions do not occur. 170The directory is broken into several sub-directories, each focused on a 171particular area of LLVM. 172 173Writing new regression tests 174---------------------------- 175 176The regression test structure is very simple, but does require some 177information to be set. This information is gathered via ``configure`` 178and is written to a file, ``test/lit.site.cfg`` in the build directory. 179The ``llvm/test`` Makefile does this work for you. 180 181In order for the regression tests to work, each directory of tests must 182have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine 183how to run the tests. This file is just Python code and thus is very 184flexible, but we've standardized it for the LLVM regression tests. If 185you're adding a directory of tests, just copy ``lit.local.cfg`` from 186another directory to get running. The standard ``lit.local.cfg`` simply 187specifies which files to look in for tests. Any directory that contains 188only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit 189documentation <CommandGuide/lit>` for more information. 190 191Each test file must contain lines starting with "RUN:" that tell :program:`lit` 192how to run it. If there are no RUN lines, :program:`lit` will issue an error 193while running a test. 194 195RUN lines are specified in the comments of the test program using the 196keyword ``RUN`` followed by a colon, and lastly the command (pipeline) 197to execute. Together, these lines form the "script" that :program:`lit` 198executes to run the test case. The syntax of the RUN lines is similar to a 199shell's syntax for pipelines including I/O redirection and variable 200substitution. However, even though these lines may *look* like a shell 201script, they are not. RUN lines are interpreted by :program:`lit`. 202Consequently, the syntax differs from shell in a few ways. You can specify 203as many RUN lines as needed. 204 205:program:`lit` performs substitution on each RUN line to replace LLVM tool names 206with the full paths to the executable built for each tool (in 207``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does 208not invoke any stray LLVM tools in the user's path during testing. 209 210Each RUN line is executed on its own, distinct from other lines unless 211its last character is ``\``. This continuation character causes the RUN 212line to be concatenated with the next one. In this way you can build up 213long pipelines of commands without making huge line lengths. The lines 214ending in ``\`` are concatenated until a RUN line that doesn't end in 215``\`` is found. This concatenated set of RUN lines then constitutes one 216execution. :program:`lit` will substitute variables and arrange for the pipeline 217to be executed. If any process in the pipeline fails, the entire line (and 218test case) fails too. 219 220Below is an example of legal RUN lines in a ``.ll`` file: 221 222.. code-block:: llvm 223 224 ; RUN: llvm-as < %s | llvm-dis > %t1 225 ; RUN: llvm-dis < %s.bc-13 > %t2 226 ; RUN: diff %t1 %t2 227 228As with a Unix shell, the RUN lines permit pipelines and I/O 229redirection to be used. 230 231There are some quoting rules that you must pay attention to when writing 232your RUN lines. In general nothing needs to be quoted. :program:`lit` won't 233strip off any quote characters so they will get passed to the invoked program. 234To avoid this use curly braces to tell :program:`lit` that it should treat 235everything enclosed as one value. 236 237In general, you should strive to keep your RUN lines as simple as possible, 238using them only to run tools that generate textual output you can then examine. 239The recommended way to examine output to figure out if the test passes is using 240the :doc:`FileCheck tool <CommandGuide/FileCheck>`. *[The usage of grep in RUN 241lines is deprecated - please do not send or commit patches that use it.]* 242 243Fragile tests 244------------- 245 246It is easy to write a fragile test that would fail spuriously if the tool being 247tested outputs a full path to the input file. For example, :program:`opt` by 248default outputs a ``ModuleID``: 249 250.. code-block:: console 251 252 $ cat example.ll 253 define i32 @main() nounwind { 254 ret i32 0 255 } 256 257 $ opt -S /path/to/example.ll 258 ; ModuleID = '/path/to/example.ll' 259 260 define i32 @main() nounwind { 261 ret i32 0 262 } 263 264``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: 265 266.. code-block:: llvm 267 268 ; RUN: opt -S %s | FileCheck 269 270 define i32 @main() nounwind { 271 ; CHECK-NOT: load 272 ret i32 0 273 } 274 275This test will fail if placed into a ``download`` directory. 276 277To make your tests robust, always use ``opt ... < %s`` in the RUN line. 278:program:`opt` does not output a ``ModuleID`` when input comes from stdin. 279 280Platform-Specific Tests 281----------------------- 282 283Whenever adding tests that require the knowledge of a specific platform, 284either related to code generated, specific output or back-end features, 285you must make sure to isolate the features, so that buildbots that 286run on different architectures (and don't even compile all back-ends), 287don't fail. 288 289The first problem is to check for target-specific output, for example sizes 290of structures, paths and architecture names, for example: 291 292* Tests containing Windows paths will fail on Linux and vice-versa. 293* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else. 294* Tests where the debug information calculates the size of types and structures. 295 296Also, if the test rely on any behaviour that is coded in any back-end, it must 297go in its own directory. So, for instance, code generator tests for ARM go 298into ``test/CodeGen/ARM`` and so on. Those directories contain a special 299``lit`` configuration file that ensure all tests in that directory will 300only run if a specific back-end is compiled and available. 301 302For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: 303 304.. code-block:: python 305 306 config.suffixes = ['.ll', '.c', '.cpp', '.test'] 307 targets = set(config.root.targets_to_build.split()) 308 if not 'ARM' in targets: 309 config.unsupported = True 310 311Other platform-specific tests are those that depend on a specific feature 312of a specific sub-architecture, for example only to Intel chips that support ``AVX2``. 313 314For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture 315variants: 316 317.. code-block:: llvm 318 319 ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2 320 ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1 321 ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2 322 323And the checks are different: 324 325.. code-block:: llvm 326 327 ; SSE2: @test1 328 ; SSE2: psubusw LCPI0_0(%rip), %xmm0 329 ; AVX1: @test1 330 ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 331 ; AVX2: @test1 332 ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 333 334So, if you're testing for a behaviour that you know is platform-specific or 335depends on special features of sub-architectures, you must add the specific 336triple, test with the specific FileCheck and put it into the specific 337directory that will filter out all other architectures. 338 339 340Variables and substitutions 341--------------------------- 342 343With a RUN line there are a number of substitutions that are permitted. 344To make a substitution just write the variable's name preceded by a ``$``. 345Additionally, for compatibility reasons with previous versions of the 346test library, certain names can be accessed with an alternate syntax: a 347% prefix. These alternates are deprecated and may go away in a future 348version. 349 350Here are the available variable names. The alternate syntax is listed in 351parentheses. 352 353``$test`` (``%s``) 354 The full path to the test case's source. This is suitable for passing on 355 the command line as the input to an LLVM tool. 356 357``%(line)``, ``%(line+<number>)``, ``%(line-<number>)`` 358 The number of the line where this variable is used, with an optional 359 integer offset. This can be used in tests with multiple RUN lines, 360 which reference test file's line numbers. 361 362``$srcdir`` 363 The source directory from where the ``make check`` was run. 364 365``objdir`` 366 The object directory that corresponds to the ``$srcdir``. 367 368``subdir`` 369 A partial path from the ``test`` directory that contains the 370 sub-directory that contains the test source being executed. 371 372``srcroot`` 373 The root directory of the LLVM src tree. 374 375``objroot`` 376 The root directory of the LLVM object tree. This could be the same as 377 the srcroot. 378 379``path`` 380 The path to the directory that contains the test case source. This is 381 for locating any supporting files that are not generated by the test, 382 but used by the test. 383 384``tmp`` 385 The path to a temporary file name that could be used for this test case. 386 The file name won't conflict with other test cases. You can append to it 387 if you need multiple temporaries. This is useful as the destination of 388 some redirected output. 389 390``target_triplet`` (``%target_triplet``) 391 The target triplet that corresponds to the current host machine (the one 392 running the test cases). This should probably be called "host". 393 394``link`` (``%link``) 395 This full link command used to link LLVM executables. This has all the 396 configured ``-I``, ``-L`` and ``-l`` options. 397 398``shlibext`` (``%shlibext``) 399 The suffix for the host platforms shared library (DLL) files. This 400 includes the period as the first character. 401 402To add more variables, look at ``test/lit.cfg``. 403 404Other Features 405-------------- 406 407To make RUN line writing easier, there are several helper scripts and programs 408in the ``llvm/test/Scripts`` directory. This directory is in the PATH 409when running tests, so you can just call these scripts using their name. 410For example: 411 412``ignore`` 413 This script runs its arguments and then always returns 0. This is useful 414 in cases where the test needs to cause a tool to generate an error (e.g. 415 to check the error output). However, any program in a pipeline that 416 returns a non-zero result will cause the test to fail. This script 417 overcomes that issue and nicely documents that the test case is 418 purposefully ignoring the result code of the tool 419``not`` 420 This script runs its arguments and then inverts the result code from it. 421 Zero result codes become 1. Non-zero result codes become 0. 422 423Sometimes it is necessary to mark a test case as "expected fail" or 424XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` 425on a line near the top of the file. This signals that the test case 426should succeed if the test fails. Such test cases are counted separately 427by the testing tool. To specify an expected fail, use the XFAIL keyword 428in the comments of the test program followed by a colon and one or more 429failure patterns. Each failure pattern can be either ``*`` (to specify 430fail everywhere), or a part of a target triple (indicating the test 431should fail on that platform), or the name of a configurable feature 432(for example, ``loadable_module``). If there is a match, the test is 433expected to fail. If not, the test is expected to succeed. To XFAIL 434everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL`` 435line: 436 437.. code-block:: llvm 438 439 ; XFAIL: darwin,sun 440 441To make the output more useful, :program:`lit` will scan 442the lines of the test case for ones that contain a pattern that matches 443``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number 444that is related to the test case. The number after "PR" specifies the 445LLVM bugzilla number. When a PR number is specified, it will be used in 446the pass/fail reporting. This is useful to quickly get some context when 447a test fails. 448 449Finally, any line that contains "END." will cause the special 450interpretation of lines to terminate. This is generally done right after 451the last RUN: line. This has two side effects: 452 453(a) it prevents special interpretation of lines that are part of the test 454 program, not the instructions to the test case, and 455 456(b) it speeds things up for really big test cases by avoiding 457 interpretation of the remainder of the file. 458 459``test-suite`` Overview 460======================= 461 462The ``test-suite`` module contains a number of programs that can be 463compiled and executed. The ``test-suite`` includes reference outputs for 464all of the programs, so that the output of the executed program can be 465checked for correctness. 466 467``test-suite`` tests are divided into three types of tests: MultiSource, 468SingleSource, and External. 469 470- ``test-suite/SingleSource`` 471 472 The SingleSource directory contains test programs that are only a 473 single source file in size. These are usually small benchmark 474 programs or small programs that calculate a particular value. Several 475 such programs are grouped together in each directory. 476 477- ``test-suite/MultiSource`` 478 479 The MultiSource directory contains subdirectories which contain 480 entire programs with multiple source files. Large benchmarks and 481 whole applications go here. 482 483- ``test-suite/External`` 484 485 The External directory contains Makefiles for building code that is 486 external to (i.e., not distributed with) LLVM. The most prominent 487 members of this directory are the SPEC 95 and SPEC 2000 benchmark 488 suites. The ``External`` directory does not contain these actual 489 tests, but only the Makefiles that know how to properly compile these 490 programs from somewhere else. When using ``LNT``, use the 491 ``--test-externals`` option to include these tests in the results. 492 493.. _test-suite-quickstart: 494 495``test-suite`` Quickstart 496------------------------- 497 498The modern way of running the ``test-suite`` is focused on testing and 499benchmarking complete compilers using the 500`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure. 501 502For more information on using LNT to execute the ``test-suite``, please 503see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_ 504documentation. 505 506``test-suite`` Makefiles 507------------------------ 508 509Historically, the ``test-suite`` was executed using a complicated setup 510of Makefiles. The LNT based approach above is recommended for most 511users, but there are some testing scenarios which are not supported by 512the LNT approach. In addition, LNT currently uses the Makefile setup 513under the covers and so developers who are interested in how LNT works 514under the hood may want to understand the Makefile based setup. 515 516For more information on the ``test-suite`` Makefile setup, please see 517the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`. 518