1OutputCheck 2=========== 3 4[![Build Status](https://travis-ci.org/stp/OutputCheck.png?branch=master)](https://travis-ci.org/stp/OutputCheck) 5 6OutputCheck is a tool for checking the output of console programs 7that is inspired by the FileCheck tool used by LLVM. It has its own 8small language (Check Directives) for describing the expected output of 9a tool that is considerably more convenient and more powerful than GNU ``grep``. 10 11This tool was originally written for [STP](http://github.com/stp/stp) 12but it ended up being a very useful tool for other projects so it 13became its own project! 14 15Obtaining OutputCheck 16===================== 17 18OutputCheck can be obtained in multple ways. 19 20Cloning Git repository 21---------------------- 22 23``` 24$ git clone https://github.com/stp/OutputCheck.git 25``` 26 27the tool can now be run as 28 29``` 30$ cd OutputCheck/ 31$ bin/OutputCheck --help 32``` 33 34Installing PyPi package 35----------------------- 36 37The tool is [available](https://pypi.python.org/pypi/OutputCheck/) in the [Python package index](https://pypi.python.org/pypi). It can be installed using the ``pip`` tool. 38 39``` 40$ pip install OutputCheck 41``` 42 43the tool can now be run as 44 45``` 46$ OutputCheck --help 47``` 48 49Please note that this package may not be up to date. 50 51It is recommended that you use [virtualenv](http://www.virtualenv.org/en/latest/) in conjunction with ``pip`` so that you do not need to install python packages as root. 52 53Check Directives 54================ 55 56Check Directives declare what output is expected from a tool. They are 57written as single line comments in a file (this file is usually used 58to by the tool being tested by OutputCheck). 59 60The advantage of writing directives in this way is that the directives 61can be written next to the code that generates the output that the directive 62is checking for. 63 64All directives use the regular expression syntax used by the ``re`` python 65module. It is also important to note that the any spaces after the ``:`` 66until the first non-whitespace character are not considered part of the 67regular expression. 68 69The following directives are supported 70 71CHECK: ``<regex>`` 72------------------ 73 74This declares that that regular expression ``<regex>`` should match somewhere 75on a single line. This match must occur after previously declared Check directives. 76 77**Succesful example** 78 79``HelloWorld.c`` 80```C 81#include <stdio.h> 82 83int main() 84{ 85 // CHECK: Hello World 86 printf("Hello World\n"); 87 88 // CHECK: Goodbye 89 printf("Goodbye\n"); 90 91 return 0; 92} 93``` 94 95``` 96$ cc HelloWorld.c -o HelloWorld 97$ ./HelloWorld | OutputCheck HelloWorld.c 98``` 99 100This example shows a simple ``C`` program being compiled and its output being checked. There are two ``CHECK:`` declarations which effectively say... 101 1021. At least one of the lines must match the regular expression ``Hello World``. 1032. At least one line after the previous match must match regular expression ``Goodbye``. 104 105It can be seen that the order the ``CHECK:`` directives are declared is important. If the directives were specified the other way round then the OutputCheck tool would report an error as shown below 106 107**Failing example** 108 109``BrokenHelloWorld.c`` 110```C 111#include <stdio.h> 112 113int main() 114{ 115 // CHECK: Goodbye 116 printf("Hello World\n"); 117 118 // CHECK: Hello World 119 printf("Goodbye\n"); 120 121 return 0; 122} 123``` 124 125``` 126$ cc BrokenHelloWorld.c -o BrokenHelloWorld 127$ ./BrokenHelloWorld | OutputCheck BrokenHelloWorld.c 128ERROR: Could not find a match for Check Directive (BrokenHelloWorld.c:8 Pattern: 'Hello World') 129``` 130 131CHECK-L: ``<string>`` 132--------------------- 133 134This is the string literal version of the ``CHECK:`` directive. This is identical to the ``CHECK:`` directive except that ``<string>`` is a literal string rather than a regular expression. This is useful if using python's regular expression syntax is too cumbersome. 135 136For example 137 138``` 139// CHECK: foo\.c\(5\) '\*' is not a valid identifier\. 140// CHECK-L: foo.c(5) '*' is not a valid identifier. 141``` 142 143the above directives are equivalent but the ``CHECK-L:`` is much easier to write. 144 145CHECK-NEXT: ``<regex>`` 146----------------------- 147 148This declares that the next line after the previous match must match the regular expression ``<regex>``. If there was no previous directive then ``CHECK-NEXT:`` matches the first line of the tool's output. 149 150**Succesful example** 151 152``HelloWorld2.c`` 153```C 154#include <stdio.h> 155 156int main() 157{ 158 // CHECK: Hello World 159 // CHECK-NEXT: Goodbye 160 printf("Hello World\nGoodbye"); 161 162 return 0; 163} 164``` 165 166``` 167$ cc HelloWorld2.c -o HelloWorld2 168$ ./HelloWorld2 | OutputCheck HelloWorld2.c 169``` 170 171**Failing example** 172 173``BrokenHelloWorld2.c`` 174```C 175#include <stdio.h> 176 177int main() 178{ 179 // CHECK: Hello World 180 181 printf("Hello World\n"); 182 printf("Testing...\n"); 183 184 // CHECK-NEXT: Goodbye 185 printf("Goodbye\n"); 186 187 return 0; 188} 189``` 190 191``` 192$ cc BrokenHelloWorld2.c -o BrokenHelloWorld2 193$ ./BrokenHelloWorld2 | OutputCheck BrokenHelloWorld2.c 194ERROR: Could not find a match for CheckNext Directive (BrokenHelloWorld2.c:10 Pattern: 'Goodbye') expected at <stdin>:2 195``` 196 197CHECK-NEXT-L: ``<string>`` 198-------------------------- 199 200This is the string literal version of the ``CHECK-NEXT:`` directive. This is identical to the ``CHECK-NEXT:`` directive except that ``<string>`` is a literal string rather than a regular expression. This is useful if using python's regular expression syntax is too cumbersome. 201 202CHECK-NOT: ``<regex>`` 203---------------------- 204 205This declares that between the previous match (if there is none, search starts from the first line of tool's output) and the next match (if there is none the search will search to the end of the tool's output) that no line will match the regular expression ``<regex>``. 206 207**Succesful example** 208 209``HelloWorld3.c`` 210```C 211#include <stdio.h> 212 213int main() 214{ 215 // CHECK: Hello World 216 // CHECK-NOT: Testing 217 // CHECK: Goodbye 218 printf("Hello World\nGoodbye"); 219 220 return 0; 221} 222``` 223 224``` 225$ cc HelloWorld3.c -o HelloWorld3 226$ ./HelloWorld3 | OutputCheck HelloWorld3.c 227``` 228 229**Failing example** 230 231``BrokenHelloWorld3.c`` 232```C 233#include <stdio.h> 234 235int main() 236{ 237 // CHECK: Hello World 238 printf("Hello World\n"); 239 240 // CHECK-NOT: Testing 241 // CHECK: Goodbye 242 printf("Testing...\n"); 243 printf("Goodbye\n"); 244 245 return 0; 246} 247``` 248 249``` 250$ cc BrokenHelloWorld3.c -o BrokenHelloWorld3 251$ ./BrokenHelloWorld3 | OutputCheck BrokenHelloWorld3.c 252ERROR: Found a match for CheckNot Directive (BrokenHelloWorld3.c:8 : Pattern: 'Testing') in <stdin>:2 253``` 254 255CHECK-NOT-L: ``<string>`` 256------------------------- 257 258This is the string literal version of the ``CHECK-NOT:`` directive. This is identical to the ``CHECK-NOT:`` directive except that ``<string>`` is a literal string rather than a regular expression. This is useful if using python's regular expression syntax is too cumbersome. 259 260Substitutions 261============= 262 263For convenience several substitutions are provided for use in ``OutputCheck``. These substitutions can used in ``<regex>`` or ``<string>`` for all the check directives. All directives can be escaped by prepending ``\``. 264 265``${LINE}`` 266----------- 267 268This is substituted by the line number that the check directive is on in the check file. 269 270``` 271// CHECK-L : file.c(${LINE}) 272``` 273 274``${LINE:+N}`` 275-------------- 276 277This is substituted by the line number that the check directive is on in the check file plus an offset (``N``). 278 279 280``` 281// CHECK-L : file.c(${LINE:+5}) 282``` 283 284``${LINE:-N}`` 285-------------- 286 287This is substituted by the line number that the check directive is on in the check file minus an offset (``N``). 288 289 290``` 291// CHECK-L : file.c(${LINE:-5}) 292``` 293 294``${CHECKFILE_NAME}`` 295--------------------- 296 297This is substituted by the name of the check file. 298 299``` 300// CHECK-L : ${CHECKFILE_NAME}(5) 301``` 302 303``${CHECKFILE_ABS_PATH}`` 304--------------------- 305 306This is substituted by the absolute path to check file. 307 308``` 309// CHECK-L : ${CHECKFILE_ABS_PATH}(5) 310``` 311 312Tests 313===== 314 315A small set of tests are present in the ``tests/`` directory. These tests are designed to be driven using ``llvm-lit`` from LLVM >=3.4 . These tests are not cross platform and will only work on systems with the ``sed`` and ``grep`` programs. It should be noted that the OutputCheck tool is implemented purely in python so the tool should work on platforms that support Python. 316