1============== 2Testing libc++ 3============== 4 5.. contents:: 6 :local: 7 8Getting Started 9=============== 10 11libc++ uses LIT to configure and run its tests. 12 13The primary way to run the libc++ tests is by using ``make check-cxx``. 14 15However since libc++ can be used in any number of possible 16configurations it is important to customize the way LIT builds and runs 17the tests. This guide provides information on how to use LIT directly to 18test libc++. 19 20Please see the `Lit Command Guide`_ for more information about LIT. 21 22.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html 23 24Usage 25----- 26 27After building libc++, you can run parts of the libc++ test suite by simply 28running ``llvm-lit`` on a specified test or directory. If you're unsure 29whether the required libraries have been built, you can use the 30`cxx-test-depends` target. For example: 31 32.. code-block:: bash 33 34 $ cd <monorepo-root> 35 $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt 36 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests 37 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test 38 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic 39 40In the default configuration, the tests are built against headers that form a 41fake installation root of libc++. This installation root has to be updated when 42changes are made to the headers, so you should re-run the `cxx-test-depends` 43target before running the tests manually with `lit` when you make any sort of 44change, including to the headers. 45 46Sometimes you'll want to change the way LIT is running the tests. Custom options 47can be specified using the `--param=<name>=<val>` flag. The most common option 48you'll want to change is the standard dialect (ie -std=c++XX). By default the 49test suite will select the newest C++ dialect supported by the compiler and use 50that. However if you want to manually specify the option like so: 51 52.. code-block:: bash 53 54 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std 55 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param=std=c++03 # Run the tests in C++03 56 57Occasionally you'll want to add extra compile or link flags when testing. 58You can do this as follows: 59 60.. code-block:: bash 61 62 $ <build>/bin/llvm-lit -sv libcxx/test --param=compile_flags='-Wcustom-warning' 63 $ <build>/bin/llvm-lit -sv libcxx/test --param=link_flags='-L/custom/library/path' 64 65Some other common examples include: 66 67.. code-block:: bash 68 69 # Specify a custom compiler. 70 $ <build>/bin/llvm-lit -sv libcxx/test/std --param=cxx_under_test=/opt/bin/g++ 71 72 # Disable warnings in the test suite 73 $ <build>/bin/llvm-lit -sv libcxx/test --param=enable_warnings=False 74 75 # Use UBSAN when running the tests. 76 $ <build>/bin/llvm-lit -sv libcxx/test --param=use_sanitizer=Undefined 77 78Using a custom site configuration 79--------------------------------- 80 81By default, the libc++ test suite will use a site configuration that matches 82the current CMake configuration. It does so by generating a ``lit.site.cfg`` 83file in the build directory from one of the configuration file templates in 84``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around 85``llvm/utils/lit/lit.py``) to that file. So when you're running 86``<build>/bin/llvm-lit``, the generated ``lit.site.cfg`` file is always loaded 87instead of ``libcxx/test/lit.cfg.py``. If you want to use a custom site 88configuration, simply point the CMake build to it using 89``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration 90will be used instead. That file can use CMake variables inside it to make 91configuration easier. 92 93 .. code-block:: bash 94 95 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config> 96 $ make -C <build> cxx-test-depends 97 $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file 98 99 100LIT Options 101=========== 102 103:program:`lit` [*options*...] [*filenames*...] 104 105Command Line Options 106-------------------- 107 108To use these options you pass them on the LIT command line as ``--param NAME`` 109or ``--param NAME=VALUE``. Some options have default values specified during 110CMake's configuration. Passing the option on the command line will override the 111default. 112 113.. program:: lit 114 115.. option:: cxx_under_test=<path/to/compiler> 116 117 Specify the compiler used to build the tests. 118 119.. option:: stdlib=<stdlib name> 120 121 **Values**: libc++, libstdc++, msvc 122 123 Specify the C++ standard library being tested. The default is libc++ if this 124 option is not provided. This option is intended to allow running the libc++ 125 test suite against other standard library implementations. 126 127.. option:: std=<standard version> 128 129 **Values**: c++03, c++11, c++14, c++17, c++20, c++2b 130 131 Change the standard version used when building the tests. 132 133.. option:: cxx_headers=<path/to/headers> 134 135 Specify the c++ standard library headers that are tested. By default the 136 headers in the source tree are used. 137 138.. option:: cxx_library_root=<path/to/lib/> 139 140 Specify the directory of the libc++ library to be tested. By default the 141 library folder of the build directory is used. 142 143 144.. option:: cxx_runtime_root=<path/to/lib/> 145 146 Specify the directory of the libc++ library to use at runtime. This directory 147 is not added to the linkers search path. This can be used to compile tests 148 against one version of libc++ and run them using another. The default value 149 for this option is `cxx_library_root`. 150 151.. option:: use_system_cxx_lib=<bool> 152 153 **Default**: False 154 155 Enable or disable testing against the installed version of libc++ library. 156 This impacts whether the ``use_system_cxx_lib`` Lit feature is defined or 157 not. The ``cxx_library_root`` and ``cxx_runtime_root`` parameters should 158 still be used to specify the path of the library to link to and run against, 159 respectively. 160 161.. option:: debug_level=<level> 162 163 **Values**: 0, 1 164 165 Enable the use of debug mode. Level 0 enables assertions and level 1 enables 166 assertions and debugging of iterator misuse. 167 168.. option:: use_sanitizer=<sanitizer name> 169 170 **Values**: Memory, MemoryWithOrigins, Address, Undefined 171 172 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when 173 building libc++ then that sanitizer will be used by default. 174 175.. option:: llvm_unwinder 176 177 Enable the use of LLVM unwinder instead of libgcc. 178 179.. option:: builtins_library 180 181 Path to the builtins library to use instead of libgcc. 182 183 184Writing Tests 185------------- 186 187When writing tests for the libc++ test suite, you should follow a few guidelines. 188This will ensure that your tests can run on a wide variety of hardware and under 189a wide variety of configurations. We have several unusual configurations such as 190building the tests on one host but running them on a different host, which add a 191few requirements to the test suite. Here's some stuff you should know: 192 193- All tests are run in a temporary directory that is unique to that test and 194 cleaned up after the test is done. 195- When a test needs data files as inputs, these data files can be saved in the 196 repository (when reasonable) and referenced by the test as 197 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or 198 directories will be made available to the test in the temporary directory 199 where it is run. 200- You should never hardcode a path from the build-host in a test, because that 201 path will not necessarily be available on the host where the tests are run. 202- You should try to reduce the runtime dependencies of each test to the minimum. 203 For example, requiring Python to run a test is bad, since Python is not 204 necessarily available on all devices we may want to run the tests on (even 205 though supporting Python is probably trivial for the build-host). 206 207Benchmarks 208========== 209 210Libc++ contains benchmark tests separately from the test of the test suite. 211The benchmarks are written using the `Google Benchmark`_ library, a copy of which 212is stored in the libc++ repository. 213 214For more information about using the Google Benchmark library see the 215`official documentation <https://github.com/google/benchmark>`_. 216 217.. _`Google Benchmark`: https://github.com/google/benchmark 218 219Building Benchmarks 220------------------- 221 222The benchmark tests are not built by default. The benchmarks can be built using 223the ``cxx-benchmarks`` target. 224 225An example build would look like: 226 227.. code-block:: bash 228 229 $ cd build 230 $ cmake [options] <path to libcxx sources> 231 $ make cxx-benchmarks 232 233This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be 234built against the just-built libc++. The compiled tests are output into 235``build/benchmarks``. 236 237The benchmarks can also be built against the platforms native standard library 238using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This 239is useful for comparing the performance of libc++ to other standard libraries. 240The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and 241``<test>.native.out`` otherwise. 242 243Also See: 244 245 * :ref:`Building Libc++ <build instructions>` 246 * :ref:`CMake Options` 247 248Running Benchmarks 249------------------ 250 251The benchmarks must be run manually by the user. Currently there is no way 252to run them as part of the build. 253 254For example: 255 256.. code-block:: bash 257 258 $ cd build/benchmarks 259 $ make cxx-benchmarks 260 $ ./algorithms.libcxx.out # Runs all the benchmarks 261 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks 262 263For more information about running benchmarks see `Google Benchmark`_. 264