1=======
2ThinLTO
3=======
4
5.. contents::
6   :local:
7
8Introduction
9============
10
11*ThinLTO* compilation is a new type of LTO that is both scalable and
12incremental. *LTO* (Link Time Optimization) achieves better
13runtime performance through whole-program analysis and cross-module
14optimization. However, monolithic LTO implements this by merging all
15input into a single module, which is not scalable
16in time or memory, and also prevents fast incremental compiles.
17
18In ThinLTO mode, as with regular LTO, clang emits LLVM bitcode after the
19compile phase. The ThinLTO bitcode is augmented with a compact summary
20of the module. During the link step, only the summaries are read and
21merged into a combined summary index, which includes an index of function
22locations for later cross-module function importing. Fast and efficient
23whole-program analysis is then performed on the combined summary index.
24
25However, all transformations, including function importing, occur
26later when the modules are optimized in fully parallel backends.
27By default, linkers_ that support ThinLTO are set up to launch
28the ThinLTO backends in threads. So the usage model is not affected
29as the distinction between the fast serial thin link step and the backends
30is transparent to the user.
31
32For more information on the ThinLTO design and current performance,
33see the LLVM blog post `ThinLTO: Scalable and Incremental LTO
34<http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html>`_.
35While tuning is still in progress, results in the blog post show that
36ThinLTO already performs well compared to LTO, in many cases matching
37the performance improvement.
38
39Current Status
40==============
41
42Clang/LLVM
43----------
44.. _compiler:
45
46The 3.9 release of clang includes ThinLTO support. However, ThinLTO
47is under active development, and new features, improvements and bugfixes
48are being added for the next release. For the latest ThinLTO support,
49`build a recent version of clang and LLVM
50<http://llvm.org/docs/CMake.html>`_.
51
52Linkers
53-------
54.. _linkers:
55.. _linker:
56
57ThinLTO is currently supported for the following linkers:
58
59- **gold (via the gold-plugin)**:
60  Similar to monolithic LTO, this requires using
61  a `gold linker configured with plugins enabled
62  <http://llvm.org/docs/GoldPlugin.html>`_.
63- **ld64**:
64  Starting with `Xcode 8 <https://developer.apple.com/xcode/>`_.
65- **lld**:
66  Starting with r284050 for ELF, r298942 for COFF.
67
68Usage
69=====
70
71Basic
72-----
73
74To utilize ThinLTO, simply add the -flto=thin option to compile and link. E.g.
75
76.. code-block:: console
77
78  % clang -flto=thin -O2 file1.c file2.c -c
79  % clang -flto=thin -O2 file1.o file2.o -o a.out
80
81When using lld-link, the -flto option need only be added to the compile step:
82
83.. code-block:: console
84
85  % clang-cl -flto=thin -O2 -c file1.c file2.c
86  % lld-link /out:a.exe file1.obj file2.obj
87
88As mentioned earlier, by default the linkers will launch the ThinLTO backend
89threads in parallel, passing the resulting native object files back to the
90linker for the final native link.  As such, the usage model the same as
91non-LTO.
92
93With gold, if you see an error during the link of the form:
94
95.. code-block:: console
96
97  /usr/bin/ld: error: /path/to/clang/bin/../lib/LLVMgold.so: could not load plugin library: /path/to/clang/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
98
99Then either gold was not configured with plugins enabled, or clang
100was not built with ``-DLLVM_BINUTILS_INCDIR`` set properly. See
101the instructions for the
102`LLVM gold plugin <http://llvm.org/docs/GoldPlugin.html#how-to-build-it>`_.
103
104Controlling Backend Parallelism
105-------------------------------
106.. _parallelism:
107
108By default, the ThinLTO link step will launch up to
109``std::thread::hardware_concurrency`` number of threads in parallel.
110For machines with hyper-threading, this is the total number of
111virtual cores. For some applications and machine configurations this
112may be too aggressive, in which case the amount of parallelism can
113be reduced to ``N`` via:
114
115- gold:
116  ``-Wl,-plugin-opt,jobs=N``
117- ld64:
118  ``-Wl,-mllvm,-threads=N``
119- lld:
120  ``-Wl,--thinlto-jobs=N``
121- lld-link:
122  ``/opt:lldltojobs=N``
123
124Incremental
125-----------
126.. _incremental:
127
128ThinLTO supports fast incremental builds through the use of a cache,
129which currently must be enabled through a linker option.
130
131- gold (as of LLVM 4.0):
132  ``-Wl,-plugin-opt,cache-dir=/path/to/cache``
133- ld64 (support in clang 3.9 and Xcode 8):
134  ``-Wl,-cache_path_lto,/path/to/cache``
135- ELF lld (as of LLVM 5.0):
136  ``-Wl,--thinlto-cache-dir=/path/to/cache``
137- COFF lld-link (as of LLVM 6.0):
138  ``/lldltocache:/path/to/cache``
139
140Cache Pruning
141-------------
142
143To help keep the size of the cache under control, ThinLTO supports cache
144pruning. Cache pruning is supported with gold, ld64 and ELF and COFF lld, but
145currently only gold, ELF and COFF lld allow you to control the policy with a
146policy string. The cache policy must be specified with a linker option.
147
148- gold (as of LLVM 6.0):
149  ``-Wl,-plugin-opt,cache-policy=POLICY``
150- ELF lld (as of LLVM 5.0):
151  ``-Wl,--thinlto-cache-policy,POLICY``
152- COFF lld-link (as of LLVM 6.0):
153  ``/lldltocachepolicy:POLICY``
154
155A policy string is a series of key-value pairs separated by ``:`` characters.
156Possible key-value pairs are:
157
158- ``cache_size=X%``: The maximum size for the cache directory is ``X`` percent
159  of the available space on the disk. Set to 100 to indicate no limit,
160  50 to indicate that the cache size will not be left over half the available
161  disk space. A value over 100 is invalid. A value of 0 disables the percentage
162  size-based pruning. The default is 75%.
163
164- ``cache_size_bytes=X``, ``cache_size_bytes=Xk``, ``cache_size_bytes=Xm``,
165  ``cache_size_bytes=Xg``:
166  Sets the maximum size for the cache directory to ``X`` bytes (or KB, MB,
167  GB respectively). A value over the amount of available space on the disk
168  will be reduced to the amount of available space. A value of 0 disables
169  the byte size-based pruning. The default is no byte size-based pruning.
170
171  Note that ThinLTO will apply both size-based pruning policies simultaneously,
172  and changing one does not affect the other. For example, a policy of
173  ``cache_size_bytes=1g`` on its own will cause both the 1GB and default 75%
174  policies to be applied unless the default ``cache_size`` is overridden.
175
176- ``cache_size_files=X``:
177  Set the maximum number of files in the cache directory. Set to 0 to indicate
178  no limit. The default is 1000000 files.
179
180- ``prune_after=Xs``, ``prune_after=Xm``, ``prune_after=Xh``: Sets the
181  expiration time for cache files to ``X`` seconds (or minutes, hours
182  respectively).  When a file hasn't been accessed for ``prune_after`` seconds,
183  it is removed from the cache. A value of 0 disables the expiration-based
184  pruning. The default is 1 week.
185
186- ``prune_interval=Xs``, ``prune_interval=Xm``, ``prune_interval=Xh``:
187  Sets the pruning interval to ``X`` seconds (or minutes, hours
188  respectively). This is intended to be used to avoid scanning the directory
189  too often. It does not impact the decision of which files to prune. A
190  value of 0 forces the scan to occur. The default is every 20 minutes.
191
192Clang Bootstrap
193---------------
194
195To bootstrap clang/LLVM with ThinLTO, follow these steps:
196
1971. The host compiler_ must be a version of clang that supports ThinLTO.
198#. The host linker_ must support ThinLTO (and in the case of gold, must be
199   `configured with plugins enabled <http://llvm.org/docs/GoldPlugin.html>`_.
200#. Use the following additional `CMake variables
201   <http://llvm.org/docs/CMake.html#options-and-variables>`_
202   when configuring the bootstrap compiler build:
203
204  * ``-DLLVM_ENABLE_LTO=Thin``
205  * ``-DCMAKE_C_COMPILER=/path/to/host/clang``
206  * ``-DCMAKE_CXX_COMPILER=/path/to/host/clang++``
207  * ``-DCMAKE_RANLIB=/path/to/host/llvm-ranlib``
208  * ``-DCMAKE_AR=/path/to/host/llvm-ar``
209
210  Or, on Windows:
211
212  * ``-DLLVM_ENABLE_LTO=Thin``
213  * ``-DCMAKE_C_COMPILER=/path/to/host/clang-cl.exe``
214  * ``-DCMAKE_CXX_COMPILER=/path/to/host/clang-cl.exe``
215  * ``-DCMAKE_LINKER=/path/to/host/lld-link.exe``
216  * ``-DCMAKE_RANLIB=/path/to/host/llvm-ranlib.exe``
217  * ``-DCMAKE_AR=/path/to/host/llvm-ar.exe``
218
219#. To use additional linker arguments for controlling the backend
220   parallelism_ or enabling incremental_ builds of the bootstrap compiler,
221   after configuring the build, modify the resulting CMakeCache.txt file in the
222   build directory. Specify any additional linker options after
223   ``CMAKE_EXE_LINKER_FLAGS:STRING=``. Note the configure may fail if
224   linker plugin options are instead specified directly in the previous step.
225
226More Information
227================
228
229* From LLVM project blog:
230  `ThinLTO: Scalable and Incremental LTO
231  <http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html>`_
232