1====================== 2 Cram: It's test time 3====================== 4 5Cram is a functional testing framework for command line applications. 6Cram tests look like snippets of interactive shell sessions. Cram runs 7each command and compares the command output in the test with the 8command's actual output. 9 10Here's a snippet from `Cram's own test suite`_:: 11 12 The $PYTHON environment variable should be set when running this test 13 from Python. 14 15 $ [ -n "$PYTHON" ] || PYTHON="`which python`" 16 $ [ -n "$PYTHONPATH" ] || PYTHONPATH="$TESTDIR/.." && export PYTHONPATH 17 $ if [ -n "$COVERAGE" ]; then 18 > coverage erase 19 > alias cram="`which coverage` run --branch -a $TESTDIR/../scripts/cram" 20 > else 21 > alias cram="$PYTHON $TESTDIR/../scripts/cram" 22 > fi 23 $ command -v md5 > /dev/null || alias md5=md5sum 24 25 Usage: 26 27 $ cram -h 28 [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re) 29 30 [Oo]ptions: (re) 31 -h, --help show this help message and exit 32 -V, --version show version information and exit 33 -q, --quiet don't print diffs 34 -v, --verbose show filenames and test status 35 -i, --interactive interactively merge changed test output 36 -d, --debug write script output directly to the terminal 37 -y, --yes answer yes to all questions 38 -n, --no answer no to all questions 39 -E, --preserve-env don't reset common environment variables 40 -e, --no-err-files don't write .err files on test failures 41 --keep-tmpdir keep temporary directories 42 --shell=PATH shell to use for running tests (default: /bin/sh) 43 --shell-opts=OPTS arguments to invoke shell with 44 --indent=NUM number of spaces to use for indentation (default: 2) 45 --xunit-file=PATH path to write xUnit XML output 46 47The format in a nutshell: 48 49* Cram tests use the ``.t`` file extension. 50 51* Lines beginning with two spaces, a dollar sign, and a space are run 52 in the shell. 53 54* Lines beginning with two spaces, a greater than sign, and a space 55 allow multi-line commands. 56 57* All other lines beginning with two spaces are considered command 58 output. 59 60* Output lines ending with a space and the keyword ``(re)`` are 61 matched as `Perl-compatible regular expressions`_. 62 63* Lines ending with a space and the keyword ``(glob)`` are matched 64 with a glob-like syntax. The only special characters supported are 65 ``*`` and ``?``. Both characters can be escaped using ``\``, and the 66 backslash can be escaped itself. 67 68* Output lines ending with either of the above keywords are always 69 first matched literally with actual command output. 70 71* Lines ending with a space and the keyword ``(no-eol)`` will match 72 actual output that doesn't end in a newline. 73 74* Actual output lines containing unprintable characters are escaped 75 and suffixed with a space and the keyword ``(esc)``. Lines matching 76 unprintable output must also contain the keyword. 77 78* Anything else is a comment. 79 80.. _Cram's own test suite: https://bitbucket.org/brodie/cram/src/0.6/tests/cram.t 81.. _Perl-compatible regular expressions: https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions 82 83 84Download 85-------- 86 87* `cram-0.7.tar.gz`_ (32 KB, requires Python 2.4-2.7 or Python 3.1 or newer) 88 89.. _cram-0.7.tar.gz: https://bitheap.org/cram/cram-0.7.tar.gz 90 91 92Installation 93------------ 94 95Install Cram using make:: 96 97 $ wget https://bitheap.org/cram/cram-0.7.tar.gz 98 $ tar zxvf cram-0.7.tar.gz 99 $ cd cram-0.7 100 $ make install 101 102 103Usage 104----- 105 106Cram will print a dot for each passing test. If a test fails, a 107`unified context diff`_ is printed showing the test's expected output 108and the actual output. Skipped tests (empty tests and tests that exit 109with return code ``80``) are marked with ``s`` instead of a dot. 110 111For example, if we run Cram on `its own example tests`_:: 112 113 .s.! 114 --- examples/fail.t 115 +++ examples/fail.t.err 116 @@ -3,21 +3,22 @@ 117 $ echo 1 118 1 119 $ echo 1 120 - 2 121 + 1 122 $ echo 1 123 1 124 125 Invalid regex: 126 127 $ echo 1 128 - +++ (re) 129 + 1 130 131 Offset regular expression: 132 133 $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n' 134 foo 135 + bar 136 baz 137 138 \d (re) 139 [A-Z] (re) 140 - # 141 + @ 142 s. 143 # Ran 6 tests, 2 skipped, 1 failed. 144 145Unless run with ``-e`` or ``--no-err-files``, Cram will also write the 146test with its actual output to ``examples/fail.t.err``, allowing you to 147use other diff tools. This file is automatically removed the next time 148the test passes. 149 150When you're first writing a test, you might just write the commands 151and run the test to see what happens. If you run Cram with ``-i`` or 152``--interactive``, you'll be prompted to merge the actual output back 153into the test. This makes it easy to quickly prototype new tests. 154 155You can specify a default set of options by creating a ``.cramrc`` 156file. For example:: 157 158 [cram] 159 verbose = True 160 indent = 4 161 162Is the same as invoking Cram with ``--verbose`` and ``--indent=4``. 163 164To change what configuration file Cram loads, you can set the 165``CRAMRC`` environment variable. You can also specify command line 166options in the ``CRAM`` environment variable. 167 168Note that the following environment variables are reset before tests 169are run: 170 171* ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's 172 ``tmp`` directory. 173 174* ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``. 175 176* ``TZ`` is set to ``GMT``. 177 178* ``COLUMNS`` is set to ``80``. (Note: When using ``--shell=zsh``, 179 this cannot be reset. It will reflect the actual terminal's width.) 180 181* ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string. 182 183Cram also provides the following environment variables to tests: 184 185* ``CRAMTMP``, set to the test runner's temporary directory. 186 187* ``TESTDIR``, set to the directory containing the test file. 188 189* ``TESTFILE``, set to the basename of the current test file. 190 191* ``TESTSHELL``, set to the value specified by ``--shell``. 192 193Also note that care should be taken with commands that close the test 194shell's ``stdin``. For example, if you're trying to invoke ``ssh`` in 195a test, try adding the ``-n`` option to prevent it from closing 196``stdin``. Similarly, if you invoke a daemon process that inherits 197``stdout`` and fails to close it, it may cause Cram to hang while 198waiting for the test shell's ``stdout`` to be fully closed. 199 200.. _unified context diff: https://en.wikipedia.org/wiki/Diff#Unified_format 201.. _its own example tests: https://bitbucket.org/brodie/cram/src/default/examples/ 202 203 204Development 205----------- 206 207Download the official development repository using Mercurial_:: 208 209 hg clone https://bitbucket.org/brodie/cram 210 211Or Git_:: 212 213 git clone https://github.com/brodie/cram.git 214 215Test Cram using Cram:: 216 217 pip install -r requirements.txt 218 make test 219 220Visit Bitbucket_ or GitHub_ if you'd like to fork the project, watch 221for new changes, or report issues. 222 223.. _Mercurial: http://mercurial.selenic.com/ 224.. _Git: http://git-scm.com/ 225.. _coverage.py: http://nedbatchelder.com/code/coverage/ 226.. _Bitbucket: https://bitbucket.org/brodie/cram 227.. _GitHub: https://github.com/brodie/cram 228