1[/
2 / Copyright (c) 2003 Boost.Test team
3 /
4 / Distributed under the Boost Software License, Version 1.0. (See accompanying
5 / file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
6 /]
7
8
9[section:test_output Controlling outputs]
10The output produced by a test module is one of the major assets the __UTF__ brings to users. In comparison with any
11kind of manual/assert based solution the __UTF__ provide following services:
12
13[/TOFIX into proper list]
14[itemized_list
15[All test errors are reported uniformly[br]
16  The test execution monitor along with standardized output from all included
17  [link boost_test.testing_tools testing tools] provides uniform reporting for all errors
18  including fatal errors, like memory assess violation and uncaught exceptions.]
19[Detailed information on the source of an error[br]
20  The __UTF__ test tool's based assertion provides as much information as possible about cause of error,
21  usually  allowing you to deduce what is wrong without entering the debugger or core analysis.]
22[Separation of the test errors description (test log) from the results report summary (test results report)
23  The information produced during test execution, including all error, warning and info messages from the test
24  tools, executed test units notification constitute the test log. By default all entries in the test log are
25  directed to the standard output. Once testing is completed the __UTF__ may produce a summary test report with
26  different levels of detail. The test report is by default directed to the standard error output.]
27[Flexibility in what is shown in the output[br]
28The __UTF__ provides the ability to configure what is shown in both the test log and the test report. The
29  configuration is supported both at runtime, during test module invocation and at compile time from within a
30  test module.]
31[Flexibility in how output is formatted[br]
32The __UTF__ provides the ability to configure the format of the test module output. At the moment only 2 formats
33  are supported by the __UTF__ itself, the well defined public interface allows you to customize an output for
34  your purposes almost any way you want.]
35]
36
37[tip More details about the class implementing the formatting of the logs can be found in the following reference sections.
38
39* [classref boost::unit_test::unit_test_log_formatter] defines the interface for all loggers. Built-in HRF and XML
40  interfaces derive from this class
41* [classref boost::unit_test::test_results] defines the information carried by the tests to provide reports and logs.
42]
43
44[section Test log output]
45
46The test log is produced during the test execution. All entries in the test log are assigned a particular log
47level. Only the entries with level that exceeds the ['active log level threshold] actually
48appear in the test log output. Log levels are arranged by the 'importance' of the log entries. Here is
49the list of all levels in order of increasing 'importance':
50
51
52[table:id_messages Messages
53    [
54      [Notifications]
55      [Meaning]
56    ]
57    [
58      [Success]
59      [This category includes messages that provide information on successfully passed assertions]
60    ]
61    [
62      [Test tree traversal]
63      [This category includes messages that are produced by the __UTF__ core and indicate which test suites/cases are currently being executed or skipped]
64    ]
65    [
66      [General information]
67      [This category includes general information messages produced in most cases by a test module author using the
68       macro __BOOST_TEST_MESSAGE__]
69    ]
70    [
71      [Warning]
72      [This category includes messages produced by failed warning level assertions]
73    ]
74    [
75      [Non fatal error]
76      [This category includes messages produced by failed check level assertions]
77    ]
78
79    [
80      [Uncaught C++ exceptions]
81      [This category includes messages that are produced by the __UTF__ and provide detailed information on the C++
82       exceptions uncaught by the test case body.
83      ]
84    ]
85    [
86      [Non-fatal system error]
87      [This category includes messages that are produced by the __UTF__ itself and provides information about caught
88       non-fatal system error. For example it includes messages produced in the case of test case timeout or if
89       floating  point values calculation errors are caught.
90      ]
91   ]
92   [
93      [Fatal system error]
94      [This category includes messages produced by failed require level assertions and by the __UTF__ itself in case of
95       abnormal test case termination.]
96   ]
97]
98
99[note
100    The active log level works namely as threshold, not as selector. For the given active log level threshold, all
101    test log entries with ['importance] higher than threshold are enabled and all test log entries with
102    ['importance] below threshold are disabled.
103]
104
105In addition to the levels described above the test log defines two special log levels. The current log level can
106be set to:
107
108* All messages[br]
109  If active log level threshold is set to this value, all test log entries appear in the output. In practice
110  this is equivalent to setting the active log level threshold to ['success information messages]
111* Nothing[br]
112  If the active log level threshold is set to this value, none of test log entries appear in the output. This log level
113  is used to execute a ['silent] test that doesn't produce any test log and only generates a result code indicating
114  whether test failed or passed.
115
116
117By default the active log level threshold is set to "non fatal error messages" and the test log output
118is generated in the human readable format. The active log level threshold and the output format can be configured
119at runtime during a test module invocation and at compile time from within a test module using the test log
120public interfaces. For example, for automated test module output processing it might be more convenient to use
121the XML based format.
122
123In most cases, the __UTF__ can't provide an exact location, where system error occurs or uncaught C++ exception
124is thrown from. To be able to pinpoint it as close as possible the __UTF__ keeps track of checkpoints - the
125location a test module passed through. A test case entrance and exit points, a test tool invocation point the
126__UTF__ tracks automatically. Any other checkpoints should be entered by you manually. The test log provides two
127macros for this purpose:
128
129* __BOOST_TEST_CHECKPOINT__ to specify a ['named] checkpoint and
130* __BOOST_TEST_PASSPOINT__ to specify an ['unnamed] checkpoint.
131
132
133
134[endsect] [/ test log output]
135
136
137
138[/ -------------------------------------------------------------------------------------------------- ]
139[section:log_runtime_config Verbosity of the logs]
140
141The active log level threshold can be configured at runtime using the parameter __param_log_level__.
142The test log output format can be selected using either parameter __param_log_format__ or the
143parameter __param_output_format__.
144
145[endsect] [/section:log_runtime_config]
146
147
148
149
150
151[/ -------------------------------------------------------------------------------------------------- ]
152[section:testing_tool_output_disable Disabling automatic printing for specific types]
153
154Most of the [link boost_test.testing_tools testing tools] print values of their
155arguments to the output stream in some form of log statement. If arguments type does not support
156``
157  operator<<(std::ostream&, ArgumentType const&);
158``
159interface you will get a compilation error. You can either implement above interface or prohibit
160the [link boost_test.testing_tools testing tools] from logging argument values for
161specified type. To do so, use following statement on file level before first test case that
162includes statement failing to compile:
163
164``
165  BOOST_TEST_DONT_PRINT_LOG_VALUE(ArgumentType)
166``
167
168[bt_example example32..BOOST_TEST_DONT_PRINT_LOG_VALUE usage..run-fail]
169
170[endsect] [/section:testing_tool_output_disable]
171
172
173
174
175
176[/ -------------------------------------------------------------------------------------------------- ]
177[section:test_output_macro_message Custom messages]
178The macro __BOOST_TEST_MESSAGE__ is intended to be used for the purpose of injecting an additional message into the
179__UTF__ test log. These messages are not intended to indicate any error or warning conditions, but rather as
180information/status notifications. The macro signature is as follows:
181
182  __BOOST_TEST_MESSAGE__(test_message);
183
184The test_message argument can be as simple as C string literal or any custom expression that you can produce with in a
185manner similar to standard `std::iostream` operation.
186
187[important Messages generated by this tool do not appear in test log output with default value of the active log level
188   threshold. For these messages to appear the active log level threshold has to be set to a value below or equal
189   to "message".
190]
191
192[bt_example example21..__BOOST_TEST_MESSAGE__ usage..run]
193
194[endsect] [/section:test_output_macro_message]
195
196
197[/ -------------------------------------------------------------------------------------------------- ]
198
199[include checkpoints.qbk]
200[include contexts.qbk]
201[include logging_floating_point.qbk]
202[include log_format.qbk]
203[include compilation_options.qbk]
204
205
206[#ref_tests_report][section Test reports]
207[warning Section empty]
208[endsect]
209
210[include progress_display.qbk]
211[include testout_summary.qbk]
212
213[endsect]  [/test_output]
214
215