1Catch works quite nicely without any command line options at all - but for those times when you want greater control the following options are available. 2Click one of the followings links to take you straight to that option - or scroll on to browse the available options. 3 4<a href="#specifying-which-tests-to-run"> ` <test-spec> ...`</a><br /> 5<a href="#usage"> ` -h, -?, --help`</a><br /> 6<a href="#listing-available-tests-tags-or-reporters"> ` -l, --list-tests`</a><br /> 7<a href="#listing-available-tests-tags-or-reporters"> ` -t, --list-tags`</a><br /> 8<a href="#showing-results-for-successful-tests"> ` -s, --success`</a><br /> 9<a href="#breaking-into-the-debugger"> ` -b, --break`</a><br /> 10<a href="#eliding-assertions-expected-to-throw"> ` -e, --nothrow`</a><br /> 11<a href="#invisibles"> ` -i, --invisibles`</a><br /> 12<a href="#sending-output-to-a-file"> ` -o, --out`</a><br /> 13<a href="#choosing-a-reporter-to-use"> ` -r, --reporter`</a><br /> 14<a href="#naming-a-test-run"> ` -n, --name`</a><br /> 15<a href="#aborting-after-a-certain-number-of-failures"> ` -a, --abort`</a><br /> 16<a href="#aborting-after-a-certain-number-of-failures"> ` -x, --abortx`</a><br /> 17<a href="#warnings"> ` -w, --warn`</a><br /> 18<a href="#reporting-timings"> ` -d, --durations`</a><br /> 19<a href="#input-file"> ` -f, --input-file`</a><br /> 20<a href="#run-section"> ` -c, --section`</a><br /> 21<a href="#filenames-as-tags"> ` -#, --filenames-as-tags`</a><br /> 22 23 24</br> 25 26<a href="#list-test-names-only"> ` --list-test-names-only`</a><br /> 27<a href="#listing-available-tests-tags-or-reporters"> ` --list-reporters`</a><br /> 28<a href="#order"> ` --order`</a><br /> 29<a href="#rng-seed"> ` --rng-seed`</a><br /> 30<a href="#libidentify"> ` --libidentify`</a><br /> 31<a href="#wait-for-keypress"> ` --wait-for-keypress`</a><br /> 32 33</br> 34 35 36 37<a id="specifying-which-tests-to-run"></a> 38## Specifying which tests to run 39 40<pre><test-spec> ...</pre> 41 42Test cases, wildcarded test cases, tags and tag expressions are all passed directly as arguments. Tags are distinguished by being enclosed in square brackets. 43 44If no test specs are supplied then all test cases, except "hidden" tests, are run. 45A test is hidden by giving it any tag starting with (or just) a period (```.```) - or, in the deprecated case, tagged ```[hide]``` or given name starting with `'./'`. To specify hidden tests from the command line ```[.]``` or ```[hide]``` can be used *regardless of how they were declared*. 46 47Specs must be enclosed in quotes if they contain spaces. If they do not contain spaces the quotes are optional. 48 49Wildcards consist of the `*` character at the beginning and/or end of test case names and can substitute for any number of any characters (including none). 50 51Test specs are case insensitive. 52 53If a spec is prefixed with `exclude:` or the `~` character then the pattern matches an exclusion. This means that tests matching the pattern are excluded from the set - even if a prior inclusion spec included them. Subsequent inclusion specs will take precendence, however. 54Inclusions and exclusions are evaluated in left-to-right order. 55 56Test case examples: 57 58<pre>thisTestOnly Matches the test case called, 'thisTestOnly' 59"this test only" Matches the test case called, 'this test only' 60these* Matches all cases starting with 'these' 61exclude:notThis Matches all tests except, 'notThis' 62~notThis Matches all tests except, 'notThis' 63~*private* Matches all tests except those that contain 'private' 64a* ~ab* abc Matches all tests that start with 'a', except those that 65 start with 'ab', except 'abc', which is included 66</pre> 67 68Names within square brackets are interpreted as tags. 69A series of tags form an AND expression wheras a comma-separated sequence forms an OR expression. e.g.: 70 71<pre>[one][two],[three]</pre> 72This matches all tests tagged `[one]` and `[two]`, as well as all tests tagged `[three]` 73 74Test names containing special characters, such as `,` or `[` can specify them on the command line using `\`. 75`\` also escapes itself. 76 77<a id="choosing-a-reporter-to-use"></a> 78## Choosing a reporter to use 79 80<pre>-r, --reporter <reporter></pre> 81 82A reporter is an object that formats and structures the output of running tests, and potentially summarises the results. By default a console reporter is used that writes, IDE friendly, textual output. Catch comes bundled with some alternative reporters, but more can be added in client code.<br /> 83The bundled reporters are: 84 85<pre>-r console 86-r compact 87-r xml 88-r junit 89</pre> 90 91The JUnit reporter is an xml format that follows the structure of the JUnit XML Report ANT task, as consumed by a number of third-party tools, including Continuous Integration servers such as Hudson. If not otherwise needed, the standard XML reporter is preferred as this is a streaming reporter, whereas the Junit reporter needs to hold all its results until the end so it can write the overall results into attributes of the root node. 92 93<a id="breaking-into-the-debugger"></a> 94## Breaking into the debugger 95<pre>-b, --break</pre> 96 97In some IDEs (currently XCode and Visual Studio) it is possible for Catch to break into the debugger on a test failure. This can be very helpful during debug sessions - especially when there is more than one path through a particular test. 98 99<a id="showing-results-for-successful-tests"></a> 100## Showing results for successful tests 101<pre>-s, --success</pre> 102 103Usually you only want to see reporting for failed tests. Sometimes it's useful to see *all* the output (especially when you don't trust that that test you just added worked first time!). 104To see successful, as well as failing, test results just pass this option. Note that each reporter may treat this option differently. The Junit reporter, for example, logs all results regardless. 105 106<a id="aborting-after-a-certain-number-of-failures"></a> 107## Aborting after a certain number of failures 108<pre>-a, --abort 109-x, --abortx [<failure threshold>] 110</pre> 111 112If a ```REQUIRE``` assertion fails the test case aborts, but subsequent test cases are still run. 113If a ```CHECK``` assertion fails even the current test case is not aborted. 114 115Sometimes this results in a flood of failure messages and you'd rather just see the first few. Specifying ```-a``` or ```--abort``` on its own will abort the whole test run on the first failed assertion of any kind. Use ```-x``` or ```--abortx``` followed by a number to abort after that number of assertion failures. 116 117<a id="listing-available-tests-tags-or-reporters"></a> 118## Listing available tests, tags or reporters 119<pre>-l, --list-tests 120-t, --list-tags 121--list-reporters 122</pre> 123 124```-l``` or ```--list-tests``` will list all registered tests, along with any tags. 125If one or more test-specs have been supplied too then only the matching tests will be listed. 126 127```-t``` or ```--list-tags``` lists all available tags, along with the number of test cases they match. Again, supplying test specs limits the tags that match. 128 129```--list-reporters``` lists the available reporters. 130 131<a id="sending-output-to-a-file"></a> 132## Sending output to a file 133<pre>-o, --out <filename> 134</pre> 135 136Use this option to send all output to a file. By default output is sent to stdout (note that uses of stdout and stderr *from within test cases* are redirected and included in the report - so even stderr will effectively end up on stdout). 137 138<a id="naming-a-test-run"></a> 139## Naming a test run 140<pre>-n, --name <name for test run></pre> 141 142If a name is supplied it will be used by the reporter to provide an overall name for the test run. This can be useful if you are sending to a file, for example, and need to distinguish different test runs - either from different Catch executables or runs of the same executable with different options. If not supplied the name is defaulted to the name of the executable. 143 144<a id="eliding-assertions-expected-to-throw"></a> 145## Eliding assertions expected to throw 146<pre>-e, --nothrow</pre> 147 148Skips all assertions that test that an exception is thrown, e.g. ```REQUIRE_THROWS```. 149 150These can be a nuisance in certain debugging environments that may break when exceptions are thrown (while this is usually optional for handled exceptions, it can be useful to have enabled if you are trying to track down something unexpected). 151 152Sometimes exceptions are expected outside of one of the assertions that tests for them (perhaps thrown and caught within the code-under-test). The whole test case can be skipped when using ```-e``` by marking it with the ```[!throws]``` tag. 153 154When running with this option any throw checking assertions are skipped so as not to contribute additional noise. Be careful if this affects the behaviour of subsequent tests. 155 156<a id="invisibles"></a> 157## Make whitespace visible 158<pre>-i, --invisibles</pre> 159 160If a string comparison fails due to differences in whitespace - especially leading or trailing whitespace - it can be hard to see what's going on. 161This option transforms tabs and newline characters into ```\t``` and ```\n``` respectively when printing. 162 163<a id="warnings"></a> 164## Warnings 165<pre>-w, --warn <warning name></pre> 166 167Enables reporting of warnings (only one, at time of this writing). If a warning is issued it fails the test. 168 169The ony available warning, presently, is ```NoAssertions```. This warning fails a test case, or (leaf) section if no assertions (```REQUIRE```/ ```CHECK``` etc) are encountered. 170 171<a id="reporting-timings"></a> 172## Reporting timings 173<pre>-d, --durations <yes/no></pre> 174 175When set to ```yes``` Catch will report the duration of each test case, in milliseconds. Note that it does this regardless of whether a test case passes or fails. Note, also, the certain reporters (e.g. Junit) always report test case durations regardless of this option being set or not. 176 177<a id="input-file"></a> 178## Load test names to run from a file 179<pre>-f, --input-file <filename></pre> 180 181Provide the name of a file that contains a list of test case names - one per line. Blank lines are skipped and anything after the comment character, ```#```, is ignored. 182 183A useful way to generate an initial instance of this file is to use the <a href="#list-test-names-only">list-test-names-only</a> option. This can then be manually curated to specify a specific subset of tests - or in a specific order. 184 185<a id="list-test-names-only"></a> 186## Just test names 187<pre>--list-test-names-only</pre> 188 189This option lists all available tests in a non-indented form, one on each line. This makes it ideal for saving to a file and feeding back into the <a href="#input-file">```-f``` or ```--input-file```</a> option. 190 191 192<a id="order"></a> 193## Specify the order test cases are run 194<pre>--order <decl|lex|rand></pre> 195 196Test cases are ordered one of three ways: 197 198 199### decl 200Declaration order. The order the tests were originally declared in. Note that ordering between files is not guaranteed and is implementation dependent. 201 202### lex 203Lexicographically sorted. Tests are sorted, alpha-numerically, by name. 204 205### rand 206Randomly sorted. Test names are sorted using ```std::random_shuffle()```. By default the random number generator is seeded with 0 - and so the order is repeatable. To control the random seed see <a href="#rng-seed">rng-seed</a>. 207 208<a id="rng-seed"></a> 209## Specify a seed for the Random Number Generator 210<pre>--rng-seed <'time'|number></pre> 211 212Sets a seed for the random number generator using ```std::srand()```. 213If a number is provided this is used directly as the seed so the random pattern is repeatable. 214Alternatively if the keyword ```time``` is provided then the result of calling ```std::time(0)``` is used and so the pattern becomes unpredictable. 215 216In either case the actual value for the seed is printed as part of Catch's output so if an issue is discovered that is sensitive to test ordering the ordering can be reproduced - even if it was originally seeded from ```std::time(0)```. 217 218<a id="libidentify"></a> 219## Identify framework and version according to the libIdentify standard 220<pre>--libidentify</pre> 221 222See [The LibIdentify repo for more information and examples](https://github.com/janwilmans/LibIdentify). 223 224<a id="wait-for-keypress"></a> 225## Wait for key before continuing 226<pre>--wait-for-keypress <start|exit|both></pre> 227 228Will cause the executable to print a message and wait until the return/ enter key is pressed before continuing - 229either before running any tests, after running all tests - or both, depending on the argument. 230 231 232<a id="usage"></a> 233## Usage 234<pre>-h, -?, --help</pre> 235 236Prints the command line arguments to stdout 237 238 239<a id="run-section"></a> 240## Specify the section to run 241<pre>-c, --section <section name></pre> 242 243To limit execution to a specific section within a test case, use this option one or more times. 244To narrow to sub-sections use multiple instances, where each subsequent instance specifies a deeper nesting level. 245 246E.g. if you have: 247 248<pre> 249TEST_CASE( "Test" ) { 250 SECTION( "sa" ) { 251 SECTION( "sb" ) { 252 /*...*/ 253 } 254 SECTION( "sc" ) { 255 /*...*/ 256 } 257 } 258 SECTION( "sd" ) { 259 /*...*/ 260 } 261} 262</pre> 263 264Then you can run `sb` with: 265<pre>./MyExe Test -c sa -c sb</pre> 266 267Or run just `sd` with: 268<pre>./MyExe Test -c sd</pre> 269 270To run all of `sa`, including `sb` and `sc` use: 271<pre>./MyExe Test -c sa</pre> 272 273There are some limitations of this feature to be aware of: 274- Code outside of sections being skipped will still be executed - e.g. any set-up code in the TEST_CASE before the 275start of the first section.</br> 276- At time of writing, wildcards are not supported in section names. 277- If you specify a section without narrowing to a test case first then all test cases will be executed 278(but only matching sections within them). 279 280 281<a id="filenames-as-tags"></a> 282## Filenames as tags 283<pre>-#, --filenames-as-tags</pre> 284 285When this option is used then every test is given an additional tag which is formed of the unqualified 286filename it is found in, with any extension stripped, prefixed with the `#` character. 287 288So, for example, tests within the file `~\Dev\MyProject\Ferrets.cpp` would be tagged `[#Ferrets]`. 289 290 291--- 292 293[Home](Readme.md) 294