1============== 2Testing libc++ 3============== 4 5.. contents:: 6 :local: 7 8Getting Started 9=============== 10 11libc++ uses LIT to configure and run its tests. 12 13The primary way to run the libc++ tests is by using ``make check-cxx``. 14 15However since libc++ can be used in any number of possible 16configurations it is important to customize the way LIT builds and runs 17the tests. This guide provides information on how to use LIT directly to 18test libc++. 19 20Please see the `Lit Command Guide`_ for more information about LIT. 21 22.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html 23 24Usage 25----- 26 27After building libc++, you can run parts of the libc++ test suite by simply 28running ``llvm-lit`` on a specified test or directory. If you're unsure 29whether the required libraries have been built, you can use the 30`check-cxx-deps` target. For example: 31 32.. code-block:: bash 33 34 $ cd <monorepo-root> 35 $ make -C <build> check-cxx-deps # If you want to make sure the targets get rebuilt 36 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests 37 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test 38 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic 39 40Sometimes you'll want to change the way LIT is running the tests. Custom options 41can be specified using the `--param=<name>=<val>` flag. The most common option 42you'll want to change is the standard dialect (ie -std=c++XX). By default the 43test suite will select the newest C++ dialect supported by the compiler and use 44that. However if you want to manually specify the option like so: 45 46.. code-block:: bash 47 48 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std 49 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param=std=c++03 # Run the tests in C++03 50 51Occasionally you'll want to add extra compile or link flags when testing. 52You can do this as follows: 53 54.. code-block:: bash 55 56 $ <build>/bin/llvm-lit -sv libcxx/test --param=compile_flags='-Wcustom-warning' 57 $ <build>/bin/llvm-lit -sv libcxx/test --param=link_flags='-L/custom/library/path' 58 59Some other common examples include: 60 61.. code-block:: bash 62 63 # Specify a custom compiler. 64 $ <build>/bin/llvm-lit -sv libcxx/test/std --param=cxx_under_test=/opt/bin/g++ 65 66 # Enable warnings in the test suite 67 $ <build>/bin/llvm-lit -sv libcxx/test --param=enable_warnings=true 68 69 # Use UBSAN when running the tests. 70 $ <build>/bin/llvm-lit -sv libcxx/test --param=use_sanitizer=Undefined 71 72Using a custom site configuration 73--------------------------------- 74 75By default, the libc++ test suite will use a site configuration that matches 76the current CMake configuration. It does so by generating a ``lit.site.cfg`` 77file in the build directory from the ``libcxx/test/lit.site.cfg.in`` template, 78and pointing ``llvm-lit`` (which is a wrapper around ``llvm/utils/lit/lit.py``) 79to that file. So when you're running ``<build>/bin/llvm-lit``, the generated 80``lit.site.cfg`` file is always loaded instead of ``libcxx/test/lit.cfg.py``. 81If you want to use a custom site configuration, simply point the CMake build 82to it using ``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site 83configuration will be used instead. That file can use CMake variables inside 84itself to make configuration easier. 85 86 .. code-block:: bash 87 88 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config> 89 $ make -C <build> check-cxx-deps 90 $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file 91 92 93LIT Options 94=========== 95 96:program:`lit` [*options*...] [*filenames*...] 97 98Command Line Options 99-------------------- 100 101To use these options you pass them on the LIT command line as ``--param NAME`` 102or ``--param NAME=VALUE``. Some options have default values specified during 103CMake's configuration. Passing the option on the command line will override the 104default. 105 106.. program:: lit 107 108.. option:: cxx_under_test=<path/to/compiler> 109 110 Specify the compiler used to build the tests. 111 112.. option:: stdlib=<stdlib name> 113 114 **Values**: libc++, libstdc++, msvc 115 116 Specify the C++ standard library being tested. The default is libc++ if this 117 option is not provided. This option is intended to allow running the libc++ 118 test suite against other standard library implementations. 119 120.. option:: std=<standard version> 121 122 **Values**: c++03, c++11, c++14, c++17, c++2a 123 124 Change the standard version used when building the tests. 125 126.. option:: cxx_headers=<path/to/headers> 127 128 Specify the c++ standard library headers that are tested. By default the 129 headers in the source tree are used. 130 131.. option:: cxx_library_root=<path/to/lib/> 132 133 Specify the directory of the libc++ library to be tested. By default the 134 library folder of the build directory is used. 135 136 137.. option:: cxx_runtime_root=<path/to/lib/> 138 139 Specify the directory of the libc++ library to use at runtime. This directory 140 is not added to the linkers search path. This can be used to compile tests 141 against one version of libc++ and run them using another. The default value 142 for this option is `cxx_library_root`. 143 144.. option:: use_system_cxx_lib=<bool> 145 146 **Default**: False 147 148 Enable or disable testing against the installed version of libc++ library. 149 This impacts whether the ``with_system_cxx_lib`` Lit feature is defined or 150 not. The ``cxx_library_root`` and ``cxx_runtime_root`` parameters should 151 still be used to specify the path of the library to link to and run against, 152 respectively. 153 154.. option:: debug_level=<level> 155 156 **Values**: 0, 1 157 158 Enable the use of debug mode. Level 0 enables assertions and level 1 enables 159 assertions and debugging of iterator misuse. 160 161.. option:: use_sanitizer=<sanitizer name> 162 163 **Values**: Memory, MemoryWithOrigins, Address, Undefined 164 165 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when 166 building libc++ then that sanitizer will be used by default. 167 168.. option:: llvm_unwinder 169 170 Enable the use of LLVM unwinder instead of libgcc. 171 172.. option:: builtins_library 173 174 Path to the builtins library to use instead of libgcc. 175 176 177Writing Tests 178------------- 179 180When writing tests for the libc++ test suite, you should follow a few guidelines. 181This will ensure that your tests can run on a wide variety of hardware and under 182a wide variety of configurations. We have several unusual configurations such as 183building the tests on one host but running them on a different host, which add a 184few requirements to the test suite. Here's some stuff you should know: 185 186- All tests are run in a temporary directory that is unique to that test and 187 cleaned up after the test is done. 188- When a test needs data files as inputs, these data files can be saved in the 189 repository (when reasonable) and referrenced by the test as 190 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or 191 directories will be made available to the test in the temporary directory 192 where it is run. 193- You should never hardcode a path from the build-host in a test, because that 194 path will not necessarily be available on the host where the tests are run. 195- You should try to reduce the runtime dependencies of each test to the minimum. 196 For example, requiring Python to run a test is bad, since Python is not 197 necessarily available on all devices we may want to run the tests on (even 198 though supporting Python is probably trivial for the build-host). 199 200Benchmarks 201========== 202 203Libc++ contains benchmark tests separately from the test of the test suite. 204The benchmarks are written using the `Google Benchmark`_ library, a copy of which 205is stored in the libc++ repository. 206 207For more information about using the Google Benchmark library see the 208`official documentation <https://github.com/google/benchmark>`_. 209 210.. _`Google Benchmark`: https://github.com/google/benchmark 211 212Building Benchmarks 213------------------- 214 215The benchmark tests are not built by default. The benchmarks can be built using 216the ``cxx-benchmarks`` target. 217 218An example build would look like: 219 220.. code-block:: bash 221 222 $ cd build 223 $ cmake [options] <path to libcxx sources> 224 $ make cxx-benchmarks 225 226This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be 227built against the just-built libc++. The compiled tests are output into 228``build/benchmarks``. 229 230The benchmarks can also be built against the platforms native standard library 231using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This 232is useful for comparing the performance of libc++ to other standard libraries. 233The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and 234``<test>.native.out`` otherwise. 235 236Also See: 237 238 * :ref:`Building Libc++ <build instructions>` 239 * :ref:`CMake Options` 240 241Running Benchmarks 242------------------ 243 244The benchmarks must be run manually by the user. Currently there is no way 245to run them as part of the build. 246 247For example: 248 249.. code-block:: bash 250 251 $ cd build/benchmarks 252 $ make cxx-benchmarks 253 $ ./algorithms.libcxx.out # Runs all the benchmarks 254 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks 255 256For more information about running benchmarks see `Google Benchmark`_. 257