1====================================
2LLVM bugpoint tool: design and usage
3====================================
4
5.. contents::
6   :local:
7
8Description
9===========
10
11``bugpoint`` narrows down the source of problems in LLVM tools and passes.  It
12can be used to debug three types of failures: optimizer crashes, miscompilations
13by optimizers, or bad native code generation (including problems in the static
14and JIT compilers).  It aims to reduce large test cases to small, useful ones.
15For example, if ``opt`` crashes while optimizing a file, it will identify the
16optimization (or combination of optimizations) that causes the crash, and reduce
17the file down to a small example which triggers the crash.
18
19For detailed case scenarios, such as debugging ``opt``, or one of the LLVM code
20generators, see :doc:`HowToSubmitABug`.
21
22Design Philosophy
23=================
24
25``bugpoint`` is designed to be a useful tool without requiring any hooks into
26the LLVM infrastructure at all.  It works with any and all LLVM passes and code
27generators, and does not need to "know" how they work.  Because of this, it may
28appear to do stupid things or miss obvious simplifications.  ``bugpoint`` is
29also designed to trade off programmer time for computer time in the
30compiler-debugging process; consequently, it may take a long period of
31(unattended) time to reduce a test case, but we feel it is still worth it. Note
32that ``bugpoint`` is generally very quick unless debugging a miscompilation
33where each test of the program (which requires executing it) takes a long time.
34
35Automatic Debugger Selection
36----------------------------
37
38``bugpoint`` reads each ``.bc`` or ``.ll`` file specified on the command line
39and links them together into a single module, called the test program.  If any
40LLVM passes are specified on the command line, it runs these passes on the test
41program.  If any of the passes crash, or if they produce malformed output (which
42causes the verifier to abort), ``bugpoint`` starts the `crash debugger`_.
43
44Otherwise, if the ``-output`` option was not specified, ``bugpoint`` runs the
45test program with the "safe" backend (which is assumed to generate good code) to
46generate a reference output.  Once ``bugpoint`` has a reference output for the
47test program, it tries executing it with the selected code generator.  If the
48selected code generator crashes, ``bugpoint`` starts the `crash debugger`_ on
49the code generator.  Otherwise, if the resulting output differs from the
50reference output, it assumes the difference resulted from a code generator
51failure, and starts the `code generator debugger`_.
52
53Finally, if the output of the selected code generator matches the reference
54output, ``bugpoint`` runs the test program after all of the LLVM passes have
55been applied to it.  If its output differs from the reference output, it assumes
56the difference resulted from a failure in one of the LLVM passes, and enters the
57`miscompilation debugger`_.  Otherwise, there is no problem ``bugpoint`` can
58debug.
59
60.. _crash debugger:
61
62Crash debugger
63--------------
64
65If an optimizer or code generator crashes, ``bugpoint`` will try as hard as it
66can to reduce the list of passes (for optimizer crashes) and the size of the
67test program.  First, ``bugpoint`` figures out which combination of optimizer
68passes triggers the bug. This is useful when debugging a problem exposed by
69``opt``, for example, because it runs over 38 passes.
70
71Next, ``bugpoint`` tries removing functions from the test program, to reduce its
72size.  Usually it is able to reduce a test program to a single function, when
73debugging intraprocedural optimizations.  Once the number of functions has been
74reduced, it attempts to delete various edges in the control flow graph, to
75reduce the size of the function as much as possible.  Finally, ``bugpoint``
76deletes any individual LLVM instructions whose absence does not eliminate the
77failure.  At the end, ``bugpoint`` should tell you what passes crash, give you a
78bitcode file, and give you instructions on how to reproduce the failure with
79``opt`` or ``llc``.
80
81.. _code generator debugger:
82
83Code generator debugger
84-----------------------
85
86The code generator debugger attempts to narrow down the amount of code that is
87being miscompiled by the selected code generator.  To do this, it takes the test
88program and partitions it into two pieces: one piece which it compiles with the
89"safe" backend (into a shared object), and one piece which it runs with either
90the JIT or the static LLC compiler.  It uses several techniques to reduce the
91amount of code pushed through the LLVM code generator, to reduce the potential
92scope of the problem.  After it is finished, it emits two bitcode files (called
93"test" [to be compiled with the code generator] and "safe" [to be compiled with
94the "safe" backend], respectively), and instructions for reproducing the
95problem.  The code generator debugger assumes that the "safe" backend produces
96good code.
97
98.. _miscompilation debugger:
99
100Miscompilation debugger
101-----------------------
102
103The miscompilation debugger works similarly to the code generator debugger.  It
104works by splitting the test program into two pieces, running the optimizations
105specified on one piece, linking the two pieces back together, and then executing
106the result.  It attempts to narrow down the list of passes to the one (or few)
107which are causing the miscompilation, then reduce the portion of the test
108program which is being miscompiled.  The miscompilation debugger assumes that
109the selected code generator is working properly.
110
111Advice for using bugpoint
112=========================
113
114``bugpoint`` can be a remarkably useful tool, but it sometimes works in
115non-obvious ways.  Here are some hints and tips:
116
117* In the code generator and miscompilation debuggers, ``bugpoint`` only works
118  with programs that have deterministic output.  Thus, if the program outputs
119  ``argv[0]``, the date, time, or any other "random" data, ``bugpoint`` may
120  misinterpret differences in these data, when output, as the result of a
121  miscompilation.  Programs should be temporarily modified to disable outputs
122  that are likely to vary from run to run.
123
124* In the `crash debugger`_, ``bugpoint`` does not distinguish different crashes
125  during reduction. Thus, if new crash or miscompilation happens, ``bugpoint``
126  will continue with the new crash instead. If you would like to stick to
127  particular crash, you should write check scripts to validate the error
128  message, see ``-compile-command`` in :doc:`CommandGuide/bugpoint`.
129
130* In the code generator and miscompilation debuggers, debugging will go faster
131  if you manually modify the program or its inputs to reduce the runtime, but
132  still exhibit the problem.
133
134* ``bugpoint`` is extremely useful when working on a new optimization: it helps
135  track down regressions quickly.  To avoid having to relink ``bugpoint`` every
136  time you change your optimization however, have ``bugpoint`` dynamically load
137  your optimization with the ``-load`` option.
138
139* ``bugpoint`` can generate a lot of output and run for a long period of time.
140  It is often useful to capture the output of the program to file.  For example,
141  in the C shell, you can run:
142
143  .. code-block:: console
144
145    $ bugpoint  ... |& tee bugpoint.log
146
147  to get a copy of ``bugpoint``'s output in the file ``bugpoint.log``, as well
148  as on your terminal.
149
150* ``bugpoint`` cannot debug problems with the LLVM linker. If ``bugpoint``
151  crashes before you see its "All input ok" message, you might try ``llvm-link
152  -v`` on the same set of input files. If that also crashes, you may be
153  experiencing a linker bug.
154
155* ``bugpoint`` is useful for proactively finding bugs in LLVM.  Invoking
156  ``bugpoint`` with the ``-find-bugs`` option will cause the list of specified
157  optimizations to be randomized and applied to the program. This process will
158  repeat until a bug is found or the user kills ``bugpoint``.
159
160* ``bugpoint`` can produce IR which contains long names. Run ``opt
161  -metarenamer`` over the IR to rename everything using easy-to-read,
162  metasyntactic names. Alternatively, run ``opt -strip -instnamer`` to rename
163  everything with very short (often purely numeric) names.
164
165What to do when bugpoint isn't enough
166=====================================
167
168Sometimes, ``bugpoint`` is not enough. In particular, InstCombine and
169TargetLowering both have visitor structured code with lots of potential
170transformations.  If the process of using bugpoint has left you with still too
171much code to figure out and the problem seems to be in instcombine, the
172following steps may help.  These same techniques are useful with TargetLowering
173as well.
174
175Turn on ``-debug-only=instcombine`` and see which transformations within
176instcombine are firing by selecting out lines with "``IC``" in them.
177
178At this point, you have a decision to make.  Is the number of transformations
179small enough to step through them using a debugger?  If so, then try that.
180
181If there are too many transformations, then a source modification approach may
182be helpful.  In this approach, you can modify the source code of instcombine to
183disable just those transformations that are being performed on your test input
184and perform a binary search over the set of transformations.  One set of places
185to modify are the "``visit*``" methods of ``InstCombiner`` (*e.g.*
186``visitICmpInst``) by adding a "``return false``" as the first line of the
187method.
188
189If that still doesn't remove enough, then change the caller of
190``InstCombiner::DoOneIteration``, ``InstCombiner::runOnFunction`` to limit the
191number of iterations.
192
193You may also find it useful to use "``-stats``" now to see what parts of
194instcombine are firing.  This can guide where to put additional reporting code.
195
196At this point, if the amount of transformations is still too large, then
197inserting code to limit whether or not to execute the body of the code in the
198visit function can be helpful.  Add a static counter which is incremented on
199every invocation of the function.  Then add code which simply returns false on
200desired ranges.  For example:
201
202.. code-block:: c++
203
204
205  static int calledCount = 0;
206  calledCount++;
207  LLVM_DEBUG(if (calledCount < 212) return false);
208  LLVM_DEBUG(if (calledCount > 217) return false);
209  LLVM_DEBUG(if (calledCount == 213) return false);
210  LLVM_DEBUG(if (calledCount == 214) return false);
211  LLVM_DEBUG(if (calledCount == 215) return false);
212  LLVM_DEBUG(if (calledCount == 216) return false);
213  LLVM_DEBUG(dbgs() << "visitXOR calledCount: " << calledCount << "\n");
214  LLVM_DEBUG(dbgs() << "I: "; I->dump());
215
216could be added to ``visitXOR`` to limit ``visitXor`` to being applied only to
217calls 212 and 217. This is from an actual test case and raises an important
218point---a simple binary search may not be sufficient, as transformations that
219interact may require isolating more than one call.  In TargetLowering, use
220``return SDNode();`` instead of ``return false;``.
221
222Now that the number of transformations is down to a manageable number, try
223examining the output to see if you can figure out which transformations are
224being done.  If that can be figured out, then do the usual debugging.  If which
225code corresponds to the transformation being performed isn't obvious, set a
226breakpoint after the call count based disabling and step through the code.
227Alternatively, you can use "``printf``" style debugging to report waypoints.
228