1This directory defines the source code and make rules for a C++
2test coverage tool. This tool is provisionally called "covtool"
3but the name may change. See the file COPYRIGHT for your rights
4to use this code.
5
6This tool differs from similar tools available on some systems
7(tcov, purecov, or gcov) in that instead of instrumenting the
8object modules, it is the source code that gets instrumented --
9albeit without editing any of the source. More about this later.
10
11A test coverage tool helps you determine whether or not all the
12lines of code in a C++ program have been executed during a
13sequence of tests. It does not analyze program behavior in any
14other way. It does not magically tell you what is wrong with
15your program or if your program is in fact working correctly,
16but it does tell you whether or not every executable statement
17in your program has been executed.
18
19It is your reponsibility to ensure that the program's outputs
20are being properly checked for accuracy. This tool only helps
21make sure that you got all the lines of code executed. Well,
22not _all_ lines of code: static initialization and destruction
23are periods of time in which no collection occurs -- so as to
24prevent bugs caused by C++ run time library reentrancy.
25
26Instrumentation doesn't beginning until your program invokes
27"CvT_StartRecording____()". Normally, the instrumentor
28automatically invokes a call to this function before the first
29line in your "main()" function. If you choose not to
30instrument your main function for some reason, you'll have to
31call the function yourself. You'll have to forward declare it
32like this:
33
34 extern void CvT_StartRecording____();
35
36Instrumentation doesn't start until this function returns --
37so lines of source in the module that calls this function won't
38show up as instrumented until the function is called.
39
40An instrumented executables runs very slowly compared to your
41normal executable performance -- so be ware of the additional
42time your tests will take when instrumented.
43
44See the subdirectory, example, for a full scale example of
45instrumenting a program. See commentary below about cov++
46the compile/instrumentation driver script used in the example.
47
48Here is an overview of running the tool:
49
50 1. run make in this directory and verify that you have built
51 the following:
52
53 covtool.exe
54 covmerge.exe
55 covannotate.exe
56 covtoolhelper.o
57
58 then, as root, run 'make install'. You don't actually have
59 to do the make install step. If you want to use the tools
60 from the directory where you compiled it that is fine.
61
62 If you do run the make install step, you can test your
63 installation using the following steps. If you don't do the
64 install, then installation test will fail (so don't run it).
65
66 As yourself, then run 'make install_tests'. You can see an
67 example of an annotated .c file in install_tests/covmerge.ann
68 or read_database.ann in the same directory. You should also
69 add /usr/local/covtool/man to your MANPATH. And add
70 /usr/local/covtool to your command search path.
71
72
73 2. in some C++ program you wish analyze, modify the make
74 rules so that instead of compile your .c files directly
75 into .o files, you do this instead:
76
77 Build instrumented versions of your source files by
78 modifying your make rules to compile your .c files
79 into .o files using the instrumentor. You should
80 probably be using the cov++ wrapper for the
81 instrumentor.
82
83 Here is an example of doing that:
84
85 .SUFFIXES: .c .o
86 .c.o:
87 cov++ -skip /usr/ -Dflags -I. -I.. -c $<
88
89 program: program.o
90 cov++ -o $@ program.o
91
92
93 These examples assume that the either /usr/local/covtool
94 or the compilation directory is in your path.
95
96 Alternately, you could use the low level instrumentor
97 program directly:
98
99 .SUFFIXES: .c .o
100 .c.o:
101 g++ -E -Dflags -I. -I.. $< | \
102 covtool.exe -instrument >`basename $< .o`.c++
103 g++ -c -O2 `basename $< .o`.c++
104 rm `basename $< .o`.c++
105
106 program: program.o
107 g++ -o $@ program.o $COVTOOL_PATH/covtoolhelper.o
108
109 If you are not naming your .c sources '*.c', but rather
110 '*.cxx', or '*.cpp' or something, you will need to tell
111 cov++ the names of the sources and a different name
112 excepted by your compiler for C++ code. See the -EXT
113 option. The actual instrumentor doesn't know about file
114 name extensions. For example: "-EXT .cpp .cpp"
115
116 3. run all your tests using the now instrumented executable
117
118 4. Use the covmerge.exe program to create a merged version
119 of the instrumentation logs created by the runs you have
120 just made:
121
122 covmerge.exe *.covexp >merged.db
123
124 If you are like me, you will have many tests distributed
125 throughout a variety of subdirectories. To collect a merged
126 coverage database, do this:
127
128 covmerge.exe `find * -name '*.covexp' -print` >merged.db
129
130 5. If you want to know the percent coverage totals for
131 complete set of tests, use
132
133 grep 'totals:' merged.db
134
135 You could also examine the individual totals: lines from each
136 of the *.covexp files.
137
138 If you want to know the coverage numbers for a given
139 source file do this:
140
141 grep 'file:.*YOUR_FILE_NAME' merged.db
142
143 It will print out records like this:
144
145 file: /dir/YOUR_FILE_NAME instrumented-lines executed-lines percentage
146
147 where 'percentage' is the executed-lines as a percentage of
148 instrumented lines.
149
150 6. If you would like to see which lines from which files
151 are NOT covered by your tests, use the program,
152 covannotate.exe to annotate your files with coverate
153 information stored in .covexp file or the merged
154 database above.
155
156 Note the script, gen_html, can be used to autmoatically run
157 the annotator for you. You must merge your .covexp files into
158 a merged database to use gen_html. You run gen_html like this:
159
160 gen_html merged_database_filename
161
162 It creates a subdirectory, coverage_html, and creates .html files
163 including annotated versions of your source code (it is likely to
164 be slow on a large project). See the description for the
165 covannotate in the next paragraphs for an explanation of the
166 annoated source format.
167
168 If you wish to manually run the coverage annotator, you invoke it
169 like this:
170
171 covannotate.exe your-c-file-name coverage-database-or-log ...
172
173 All lines in the annotated output file will begin with either
174 ' ', '-', or '+'. These characters have the following
175 significance:
176
177 ' ' -- line was not instrumented
178 '-' -- instrumented but not executed
179 '+' -- executed
180
181 For example, your annotated source file might look like this:
182
183 #include <stdio.h>
184 int main()
185 +{
186 + printf("hello world\n");
187 + exit(0);
188 }
189
190 Note: the .covexp files (and the merged version thereof)
191 contain filenames with the full directory part attached.
192 When using covannotate.exe you do not have to specify the
193 full pathname. That program figures out the pathname for
194 the file you specify and uses that to search the .covexp
195 files.
196
197 7. Add additional tests to make sure that all your lines of code
198 get tested. At least to 85% coverage in the totals: section.
199
200 8. To simplify your life, you should probably modify your
201 makefile to allow you to provide a make directive that will
202 let you build everything with coverage or not. Here's an
203 example:
204
205 INSTRUMENTATION=false
206
207 ifeq ($(INSTRUMENTATION),true)
208 CC=g++
209 else
210 CC=g++ -EXT .cpp .c++ -skip /usr
211 endif
212
213 In this example, when you normally run make, you don't get
214 instrumentation. If you want to turn on instrumentation
215 just do this:
216
217 make clean
218
219 make INSTRUMENTATION=true
220
221 You will of course have to have a clean target of your own
222 construction.
223
224WIERDNESSES
225
226 As stated earlier, covtool does not handle instrumentation during
227 static initialization and destruction. This is is because of
228 ordering issues with the standard library 'standard allocator'.
229 However, I might figure out how to fix this later.
230
231 Also, threads are not supported! Both the threading and the
232 static initialization issues could be eliminated by having the
233 covtoolhelper.c file implemented to communicate with a separate
234 process which does the collection. Feel free to do this and
235 send me the source ;->.
236
237REDUCING VOLUME
238
239 The covtool.exe program allows you to reduce the amount of
240 instrumentation injected by specifying directories which
241 should not be instrumented on its command line. To skip
242 the C and C++ standard headers you would use an invocation
243 like this:
244
245 covtool.exe -instrument -skip /usr/include/ <your_file.i
246
247 Multiple -skip commands can be given, so that you pick and
248 choose the directories to ignore. Actually, the skip option
249 does not define directories, but rather a prefix. You could
250 skip all file names beginning with 'h' like this: "-skip h".
251 I guess, I should make -skip accept filename pattern match
252 rules, but I haven't yet.
253
254 The parameter to the skip function does not have the same
255 characteristics as the text you find in .covexp files,
256 described below. The -skip directives are in the format
257 defined by the compiler when it #includes files. If you
258 can't get your skip pattern to work, grep the preprocessed
259 output from the compiler like this:
260
261 grep '^#'
262
263 the file name patterns it presents define the format of
264 the file names to the -skip directive. So, if you use a
265 -I.. compiler directive, and you want to skip ../somedir
266 then your skip directive should be something like
267
268 -skip ../somedir
269
270 rather than skipping the full pathname of the directory as
271 you would expect from examining the .covexp files.
272
273
274COVEXP/MERGED DATABASE FILE FORMAT
275
276 The runtime data collection routines found in covtoolhelper.c
277 produce data of the following forms:
278
279 file: [filename] [instrumented_lines] [executed_lines]
280
281 el: [number] ...
282
283 il: [number] ...
284
285 totals: [instrumented_lines] [executed_lines] [percent_covered]
286
287 For aesthetic reasons, the data is formatted as attractively as
288 possible, but the official format does not require attractive
289 presentation in the database files. The data in the files should
290 be considered as a stream having 5 token types -- with blanks and
291 newlines as the separator. The token types are:
292
293 numbers
294 file:
295 el:
296 il:
297 totals:
298
299 There should only be one 'totals:' line and it should be at the
300 end. The el: and il: tags may be absent or may be followed by
301 0 or more numbers.
302
303 The filename will contain the full pathname.
304
305
306NAMING .covexp FILES
307
308
309 When running tests with instrumented executables, the .covexp
310 files will normally be named something like this:
311
312 cov-run-3287.covexp
313
314 With each run, you will get a new cov-run file. The number in
315 the file name is the process id of the running program. You can
316 specify the file name prefix for the .covexp files by using the
317 environment variable, COVTOOL_PREFIX. For example:
318
319 export COVTOOL_PREFIX=test1
320 program_that_performs_the_test
321
322 And instead of getting
323
324 cov-run-????.covexp
325
326 You will get
327
328 test1-????.covexp
329
330
331PROGRAM BUGS UNCOVERED BY INSTRUMENTATION
332
333
334While the instrumentor and runtime data collector probably have
335bugs I have not yet caught, the act of instrumentation itself
336sometimes brings to light bugs in your program which have not
337yet given symptoms.
338
3391. The missing return statement
340
341 I have a program that the instrumentor told me was 85 percent
342 covered -- which is great. However the instrumented version
343 crashed when I ran one particular test. After scratching my
344 head and debugging for some time, I discovered that a function
345 had no return statement and it was supposed to return pointer
346 to string. G++ did not give me any errors, and so my test had
347 been passing only by accident. Here is what the code looked
348 like:
349
350 string const *function(int parm)
351 {
352 static string rv;
353
354 if(parm != 99)
355 {
356 rv = "not 99";
357 return &rv;
358 }
359 else
360 {
361 rv = "yep, 99";
362 }
363
364 }
365
366 As you can see, if the parm is 99, then there is no return
367 statement for 'function'. What gets returned? Apparantly,
368 the &rv was getting returned because my test was looking for
369 "yep, 99" and got it.
370
371 Apparantly the act of injecting instrumentation calls, caused
372 the return value not to accidentally be &rv as it was before
373 but some random pointer.
374
375 Putting in a statement of the form 'return &rv' at the end
376 of the function solved the real problem instrumentation had
377 detected.
378
3792. Using a pointer after it was deleted
380
381 In another program, adding instrumentation caused the program
382 output to change without a crash. Instead of printing the
383 number 34, it produced 704502365. A bit of debugging led
384 to the following discovery: a function was returning a data
385 structure containing a pointer to a deleted object -- but
386 the program was printing a member of that object. The act
387 of instrumentation did not cause the number to change, but
388 the instrumentation runtime activity was re-using the
389 deleted heap packet.
390
391Thus, if your tests fail after you add instrumentation, do not
392immediately assume that the problem is the result of a bug in
393the instrumentation itself. On the other hand, if your program
394won't compile after instrumentation, it most certain is a problem
395with covtool. Please report it as described below.
396
397
398REPORTING PROBLEMS
399
400
401Unfortunately, covtool itself is no more immune to program bugs
402than any other. Let me apologize in advance for any you
403encounter.
404
405When an problem occurs, please send email to lowell.boggs@attbi.com
406and I will attempt to get back with you within a couple of days.
407Sorry, but I cannot guarantee any specific turn around time. Also,
408I cannot provide general programming help. I can only attempt to
409resolve problems with covtool -- and am anxious to do so. Please
410do as much to debug the problem yourself as you can -- as there is
411only one of me and (hopefully) there will be many of you.
412
413When problem does occur, please provide the information specifically
414requested below and as much detail as possible as to the scenario
415that led to the problem. Also, please state clearly what you thought
416was supposed to happen and what did happen. I am not much of a mind
417reader.
418
419There are likely to be the following kinds of problems:
420
421 0. threads and static initialization and destruction. covtool
422 does not support threads and it does not support you turning
423 on recording during static initialization or destruction!
424
425 1. covtool.exe might incorrectly instrument your program. This
426 is likely to result in a failure of it to compile with
427 instrumentation. I have compiled all the /usr/include/g++
428 headers and have not had any problems, but you never know.
429 A work around for this might be to use the -skip directives
430 to suppress the instrumentation for the offending file. On
431 the other hand, it might not fix the problem. In either
432 event, please do the following to report the problem:
433
434 A. produce a .i file from your source that won't properly
435 instrument.
436
437 B. email me the compile error you are seeing and the
438 instrumented file. See the email address above.
439
440 If you have an interest in the source code for covtool.exe,
441 you might try diagnosing the problem yourself. If you run
442 covtool.exe without the -instrument option, it will produce
443 a commentary describing the lines of text, the { brace depth,
444 and the kind of C++ structure it thinks it is processing:
445 enum, class, block, function, etc. You can sometimes guess
446 what is wrong by looking at the commentary and verifying that
447 it doesn't match what the source code is doing. For example,
448 the nest depth at file scope should always be 0. If it isn't
449 zero then the parser is lost.
450
451 2. The instrumentation runtime might malfunction causing a crash.
452 This is unlikely because the datastructures used are entirely
453 STL containers with little or no pointer arithmetic. There are
454 no vectors or arrays, and no deletes occur during
455 instrumentation. However, things happen. There may well be
456 bugs I haven't yet caught. If you encounter a crash and you
457 suspect the instrumentation code do the following:
458
459 1. Verify that the program doesn't crash if you don't
460 instrument it. If it crashes without instrumentation,
461 obviously it isn't covtool's fault.
462
463 2. Verify that the program doesn't crash if you instrument
464 but don't actually record. The easiest way to do this
465 is to compile your main function without
466 instrumentation. That is, g++ not cov++ to compile it.
467
468 If the program crashes with instrumentation but without
469 recording, see if you have a missing return statement
470 somewhere as described above in the section PROGRAM BUGS
471 UNCOVERED BY INSTRUMENTATION.
472
473 It is unlikely that the a successfully compiling program
474 will have an instrumentation bug that leads to
475 misbehavior, but things happen. Send me the .i file
476 containing the misbehaving function at the address above
477 and I will see if I can figure out what is going on.
478
479 3. If the program runs correctly when instrumented but not
480 recording and fails when you turn recording on, it is
481 most likely that instrumentation data structures are
482 being overwritten during your program's execution.
483 Or, you might be using a pointer which you have deleted.
484 See PROGRAM BUGS UNCOVERED BY INSTRUMENTATION. The
485 instrumentation runtime code using STL maps, sets, etc,
486 and no writable pointers.
487
488 As a work around, try not instrumenting some files and
489 see how this affects behavior. It might help you isolate
490 which modules have a missing return statement or are
491 using a deleted heap packet.
492
493 If the program is still crashing and you can't figure
494 out why, compile the module 'covtoolhelper.o' for
495 debug and verify that the program crashes in one of
496 its functions. Sadly, you'll probably have to debug
497 this problem yourself. I'll be happy to consult about
498 the implementation of covtoolhelper, but when programs
499 write on the wrong data structures, it becomes almost
500 impossible for an outsider to have much insight.
501
502 4. If covannotate.exe tells you that it does not have any
503 information on a file you wish to annotate, verify that
504 the file name does in fact exist in your *.covexp or *.db
505 file and that you are spelling the name correctly. If
506 the *.covexp files don't have any information about your
507 file, verify that you have instrumented it. If it still
508 can't find the file, try specifying the exact pathname
509 in the file as found in the *.covexp file. If that doesn't
510 work, then send me any one of the *.covexp files containing
511 the collected data and the full pathname of the file you
512 are trying to get an annotation for.
513
514 5. If you have any problem with cov++, other than being happy
515 with the limitations it places on you by giving you error
516 messages, use the -VER option as the first option and send me
517 the output it gives.
518
519
520Good luck!
521