README.md
1# DExTer (Debugging Experience Tester)
2
3## Introduction
4
5DExTer is a suite of tools used to evaluate the "User Debugging Experience". DExTer drives an external debugger, running on small test programs, and collects information on the behavior at each debugger step to provide quantitative values that indicate the quality of the debugging experience.
6
7## Supported Debuggers
8
9DExTer currently supports the Visual Studio 2015 and Visual Studio 2017 debuggers via the [DTE interface](https://docs.microsoft.com/en-us/dotnet/api/envdte.dte), and LLDB via its [Python interface](https://lldb.llvm.org/python-reference.html). GDB is not currently supported.
10
11The following command evaluates your environment, listing the available and compatible debuggers:
12
13 dexter.py list-debuggers
14
15## Dependencies
16[TODO] Add a requirements.txt or an install.py and document it here.
17
18### Python 3.6
19
20DExTer requires python version 3.6 or greater.
21
22### pywin32 python package
23
24This is required to access the DTE interface for the Visual Studio debuggers.
25
26 <python-executable> -m pip install pywin32
27
28### clang
29
30DExTer is current compatible with 'clang' and 'clang-cl' compiler drivers. The compiler must be available for DExTer, for example the following command should successfully build a runnable executable.
31
32 <compiler-executable> tests/nostdlib/fibonacci/test.cpp
33
34## Running a test case
35
36The following DExTer commands build the test.cpp from the tests/nostdlib/fibonacci directory and quietly runs it on the Visual Studio debugger, reporting the debug experience heuristic. The first command builds with no optimizations (/Od) and scores 1.0000. The second command builds with optimizations (/Ox) and scores 0.2832 which suggests a worse debugging experience.
37
38 dexter.py test --builder clang-cl_vs2015 --debugger vs2017 --cflags="/Od /Zi" --ldflags="/Zi" -- tests/nostdlib/fibonacci
39 fibonacci = (1.0000)
40
41 dexter.py test --builder clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -- tests/nostdlib/fibonacci
42 fibonacci = (0.2832)
43
44## An example test case
45
46The sample test case (tests/nostdlib/fibonacci) looks like this:
47
48 1. #ifdef _MSC_VER
49 2. # define DEX_NOINLINE __declspec(noinline)
50 3. #else
51 4. # define DEX_NOINLINE __attribute__((__noinline__))
52 5. #endif
53 6.
54 7. DEX_NOINLINE
55 8. void Fibonacci(int terms, int& total)
56 9. {
57 0. int first = 0;
58 11. int second = 1;
59 12. for (int i = 0; i < terms; ++i)
60 13. {
61 14. int next = first + second; // DexLabel('start')
62 15. total += first;
63 16. first = second;
64 17. second = next; // DexLabel('end')
65 18. }
66 19. }
67 20.
68 21. int main()
69 22. {
70 23. int total = 0;
71 24. Fibonacci(5, total);
72 25. return total;
73 26. }
74 27.
75 28. /*
76 29. DexExpectWatchValue('i', '0', '1', '2', '3', '4',
77 30. from_line='start', to_line='end')
78 31. DexExpectWatchValue('first', '0', '1', '2', '3', '5',
79 32. from_line='start', to_line='end')
80 33. DexExpectWatchValue('second', '1', '2', '3', '5',
81 34 from_line='start', to_line='end')
82 35. DexExpectWatchValue('total', '0', '1', '2', '4', '7',
83 36. from_line='start', to_line='end')
84 37. DexExpectWatchValue('next', '1', '2', '3', '5', '8',
85 38. from_line='start', to_line='end')
86 39. DexExpectWatchValue('total', '7', on_line=25)
87 40. DexExpectStepKind('FUNC_EXTERNAL', 0)
88 41. */
89
90[DexLabel][1] is used to give a name to a line number.
91
92The [DexExpectWatchValue][2] command states that an expression, e.g. `i`, should
93have particular values, `'0', '1', '2', '3','4'`, sequentially over the program
94lifetime on particular lines. You can refer to a named line or simply the line
95number (See line 39).
96
97At the end of the test is the following line:
98
99 DexExpectStepKind('FUNC_EXTERNAL', 0)
100
101This [DexExpectStepKind][3] command indicates that we do not expect the debugger
102to step into a file outside of the test directory.
103
104[1]: Commands.md#DexLabel
105[2]: Commands.md#DexExpectWatchValue
106[3]: Commands.md#DexExpectStepKind
107
108## Detailed DExTer reports
109
110Running the command below launches the tests/nostdlib/fibonacci test case in DExTer, using clang-cl as the compiler, Visual Studio 2017 as the debugger, and producing a detailed report:
111
112 $ dexter.py test --builder clang-cl_vs2015 --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi" -v -- tests/nostdlib/fibonacci
113
114The detailed report is enabled by `-v` and shows a breakdown of the information from each debugger step. For example:
115
116 fibonacci = (0.2832)
117
118 ## BEGIN ##
119 [1, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 23, 1, "BREAKPOINT", "FUNC", {}]
120 [2, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 24, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
121 [3, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 25, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
122 . [4, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "FUNC", {}]
123 . [5, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
124 . [6, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
125 . [7, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
126 . [8, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
127 . [9, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "0"}]
128 . [10, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
129 . [11, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
130 . [12, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "1", "total": "0", "first": "1"}]
131 . [13, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
132 . [14, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
133 . [15, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "2", "total": "0", "first": "1"}]
134 . [16, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
135 . [17, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
136 . [18, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "3", "total": "0", "first": "2"}]
137 . [19, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
138 . [20, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
139 . [21, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 15, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {"i": "Variable is optimized away and not available.", "second": "5", "total": "0", "first": "3"}]
140 . [22, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 13, 1, "BREAKPOINT", "VERTICAL_BACKWARD", {}]
141 . [23, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 16, 1, "BREAKPOINT", "VERTICAL_FORWARD", {"i": "Variable is optimized away and not available.", "next": "Variable is optimized away and not available.", "second": "Variable is optimized away and not available.", "total": "0", "first": "Variable is optimized away and not available."}]
142 . [24, "Fibonacci", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 20, 1, "BREAKPOINT", "VERTICAL_FORWARD", {}]
143 [25, "main", "c:\\dexter\\tests\\nostdlib\\fibonacci\\test.cpp", 26, 1, "BREAKPOINT", "FUNC", {"total": "7"}]
144 ## END (25 steps) ##
145
146
147 step kind differences [0/1]
148 FUNC_EXTERNAL:
149 0
150
151 test.cpp:15-18 [first] [9/21]
152 expected encountered values:
153 0
154 1
155 2
156 3
157
158 missing values:
159 5 [-6]
160
161 result optimized away:
162 step 5 (Variable is optimized away and not available.) [-3]
163 step 7 (Variable is optimized away and not available.)
164 step 8 (Variable is optimized away and not available.)
165 step 11 (Variable is optimized away and not available.)
166 step 14 (Variable is optimized away and not available.)
167 step 17 (Variable is optimized away and not available.)
168 step 20 (Variable is optimized away and not available.)
169 step 23 (Variable is optimized away and not available.)
170
171 test.cpp:15-18 [i] [15/21]
172 result optimized away:
173 step 5 (Variable is optimized away and not available.) [-3]
174 step 7 (Variable is optimized away and not available.) [-3]
175 step 8 (Variable is optimized away and not available.) [-3]
176 step 9 (Variable is optimized away and not available.) [-3]
177 step 11 (Variable is optimized away and not available.) [-3]
178 step 12 (Variable is optimized away and not available.)
179 step 14 (Variable is optimized away and not available.)
180 step 15 (Variable is optimized away and not available.)
181 step 17 (Variable is optimized away and not available.)
182 step 18 (Variable is optimized away and not available.)
183 step 20 (Variable is optimized away and not available.)
184 step 21 (Variable is optimized away and not available.)
185 step 23 (Variable is optimized away and not available.)
186
187 test.cpp:15-18 [second] [21/21]
188 expected encountered values:
189 1
190 2
191 3
192 5
193
194 result optimized away:
195 step 5 (Variable is optimized away and not available.) [-3]
196 step 7 (Variable is optimized away and not available.) [-3]
197 step 8 (Variable is optimized away and not available.) [-3]
198 step 11 (Variable is optimized away and not available.) [-3]
199 step 14 (Variable is optimized away and not available.) [-3]
200 step 17 (Variable is optimized away and not available.) [-3]
201 step 20 (Variable is optimized away and not available.) [-3]
202 step 23 (Variable is optimized away and not available.)
203
204 test.cpp:15-18 [total] [21/21]
205 expected encountered values:
206 0
207
208 missing values:
209 1 [-6]
210 2 [-6]
211 4 [-6]
212 7 [-3]
213
214 test.cpp:16-18 [next] [15/21]
215 result optimized away:
216 step 5 (Variable is optimized away and not available.) [-3]
217 step 8 (Variable is optimized away and not available.) [-3]
218 step 11 (Variable is optimized away and not available.) [-3]
219 step 14 (Variable is optimized away and not available.) [-3]
220 step 17 (Variable is optimized away and not available.) [-3]
221 step 20 (Variable is optimized away and not available.)
222 step 23 (Variable is optimized away and not available.)
223
224 test.cpp:26 [total] [0/7]
225 expected encountered values:
226 7
227
228The first line
229
230 fibonacci = (0.2832)
231
232shows a score of 0.2832 suggesting that unexpected behavior has been seen. This score is on scale of 0.0000 to 1.000, with 0.000 being the worst score possible and 1.000 being the best score possible. The verbose output shows the reason for any scoring. For example:
233
234 test.cpp:15-18 [first] [9/21]
235 expected encountered values:
236 0
237 1
238 2
239 3
240
241 missing values:
242 5 [-6]
243
244 result optimized away:
245 step 5 (Variable is optimized away and not available.) [-3]
246 step 7 (Variable is optimized away and not available.)
247 step 8 (Variable is optimized away and not available.)
248 step 11 (Variable is optimized away and not available.)
249 step 14 (Variable is optimized away and not available.)
250 step 17 (Variable is optimized away and not available.)
251 step 20 (Variable is optimized away and not available.)
252 step 23 (Variable is optimized away and not available.)
253
254shows that for `first` the expected values 0, 1, 2 and 3 were seen, 5 was not. On some steps the variable was reported as being optimized away.
255
256## Writing new test cases
257
258Each test requires a `test.cfg` file. Currently the contents of this file are not read, but its presence is used to determine the root directory of a test. In the future, configuration variables for the test such as supported language modes may be stored in this file. Use the various [commands](Commands.md) to encode debugging expectations.
259
260## Additional tools
261
262For clang-based compilers, the `clang-opt-bisect` tool can be used to get a breakdown of which LLVM passes may be contributing to debugging experience issues. For example:
263
264 $ dexter.py clang-opt-bisect tests/nostdlib/fibonacci --builder clang-cl --debugger vs2017 --cflags="/Ox /Zi" --ldflags="/Zi"
265
266 pass 1/211 = (1.0000) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
267 pass 2/211 = (0.7611) (-0.2389) [SROA on function (?Fibonacci@@YAXHAEAH@Z)]
268 pass 3/211 = (0.7611) (0.0000) [Early CSE on function (?Fibonacci@@YAXHAEAH@Z)]
269 pass 4/211 = (0.7611) (0.0000) [Simplify the CFG on function (main)]
270 pass 5/211 = (0.7611) (0.0000) [SROA on function (main)]
271 pass 6/211 = (0.7611) (0.0000) [Early CSE on function (main)]
272 pass 7/211 = (0.7611) (0.0000) [Infer set function attributes on module (c:\dexter\tests\fibonacci\test.cpp)]
273 pass 8/211 = (0.7611) (0.0000) [Interprocedural Sparse Conditional Constant Propagation on module (c:\dexter\tests\fibonacci\test.cpp)]
274 pass 9/211 = (0.7611) (0.0000) [Called Value Propagation on module (c:\dexter\tests\fibonacci\test.cpp)]
275 pass 10/211 = (0.7611) (0.0000) [Global Variable Optimizer on module (c:\dexter\tests\fibonacci\test.cpp)]
276 pass 11/211 = (0.7611) (0.0000) [Promote Memory to Register on function (?Fibonacci@@YAXHAEAH@Z)]
277 pass 12/211 = (0.7611) (0.0000) [Promote Memory to Register on function (main)]
278 pass 13/211 = (0.7611) (0.0000) [Dead Argument Elimination on module (c:\dexter\tests\fibonacci\test.cpp)]
279 pass 14/211 = (0.7611) (0.0000) [Combine redundant instructions on function (?Fibonacci@@YAXHAEAH@Z)]
280 pass 15/211 = (0.7611) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]a
281 pass 16/211 = (0.7345) (-0.0265) [Combine redundant instructions on function (main)]
282 pass 17/211 = (0.7345) (0.0000) [Simplify the CFG on function (main)]
283 pass 18/211 = (0.7345) (0.0000) [Remove unused exception handling info on SCC (?Fibonacci@@YAXHAEAH@Z)]
284 pass 19/211 = (0.7345) (0.0000) [Function Integration/Inlining on SCC (?Fibonacci@@YAXHAEAH@Z)]
285 pass 20/211 = (0.7345) (0.0000) [Deduce function attributes on SCC (?Fibonacci@@YAXHAEAH@Z)]
286 pass 21/211 = (0.7345) (0.0000) [SROA on function (?Fibonacci@@YAXHAEAH@Z)]
287 pass 22/211 = (0.7345) (0.0000) [Early CSE w/ MemorySSA on function (?Fibonacci@@YAXHAEAH@Z)]
288 pass 23/211 = (0.7345) (0.0000) [Speculatively execute instructions if target has divergent branches on function (?Fibonacci@@YAXHAEAH@Z)]
289 pass 24/211 = (0.7345) (0.0000) [Jump Threading on function (?Fibonacci@@YAXHAEAH@Z)]
290 pass 25/211 = (0.7345) (0.0000) [Value Propagation on function (?Fibonacci@@YAXHAEAH@Z)]
291 pass 26/211 = (0.7345) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
292 pass 27/211 = (0.7345) (0.0000) [Combine redundant instructions on function (?Fibonacci@@YAXHAEAH@Z)]
293 pass 28/211 = (0.7345) (0.0000) [Tail Call Elimination on function (?Fibonacci@@YAXHAEAH@Z)]
294 pass 29/211 = (0.7345) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
295 pass 30/211 = (0.7345) (0.0000) [Reassociate expressions on function (?Fibonacci@@YAXHAEAH@Z)]
296 pass 31/211 = (0.8673) (0.1327) [Rotate Loops on loop]
297 pass 32/211 = (0.5575) (-0.3097) [Loop Invariant Code Motion on loop]
298 pass 33/211 = (0.5575) (0.0000) [Unswitch loops on loop]
299 pass 34/211 = (0.5575) (0.0000) [Simplify the CFG on function (?Fibonacci@@YAXHAEAH@Z)]
300 pass 35/211 = (0.5575) (0.0000) [Combine redundant instructions on function (?Fibonacci@@YAXHAEAH@Z)]
301 pass 36/211 = (0.5575) (0.0000) [Induction Variable Simplification on loop]
302 pass 37/211 = (0.5575) (0.0000) [Recognize loop idioms on loop]
303 <output-snipped>
304
305