README.rst
1======================
2 Cram: It's test time
3======================
4
5Cram is a functional testing framework for command line applications.
6Cram tests look like snippets of interactive shell sessions. Cram runs
7each command and compares the command output in the test with the
8command's actual output.
9
10Here's a snippet from `Cram's own test suite`_::
11
12 The $PYTHON environment variable should be set when running this test
13 from Python.
14
15 $ [ -n "$PYTHON" ] || PYTHON="`which python`"
16 $ [ -n "$PYTHONPATH" ] || PYTHONPATH="$TESTDIR/.." && export PYTHONPATH
17 $ if [ -n "$COVERAGE" ]; then
18 > coverage erase
19 > alias cram="`which coverage` run --branch -a $TESTDIR/../scripts/cram"
20 > else
21 > alias cram="$PYTHON $TESTDIR/../scripts/cram"
22 > fi
23 $ command -v md5 > /dev/null || alias md5=md5sum
24
25 Usage:
26
27 $ cram -h
28 [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)
29
30 [Oo]ptions: (re)
31 -h, --help show this help message and exit
32 -V, --version show version information and exit
33 -q, --quiet don't print diffs
34 -v, --verbose show filenames and test status
35 -i, --interactive interactively merge changed test output
36 -d, --debug write script output directly to the terminal
37 -y, --yes answer yes to all questions
38 -n, --no answer no to all questions
39 -E, --preserve-env don't reset common environment variables
40 --keep-tmpdir keep temporary directories
41 --shell=PATH shell to use for running tests (default: /bin/sh)
42 --shell-opts=OPTS arguments to invoke shell with
43 --indent=NUM number of spaces to use for indentation (default: 2)
44 --xunit-file=PATH path to write xUnit XML output
45
46The format in a nutshell:
47
48* Cram tests use the ``.t`` file extension.
49
50* Lines beginning with two spaces, a dollar sign, and a space are run
51 in the shell.
52
53* Lines beginning with two spaces, a greater than sign, and a space
54 allow multi-line commands.
55
56* All other lines beginning with two spaces are considered command
57 output.
58
59* Output lines ending with a space and the keyword ``(re)`` are
60 matched as `Perl-compatible regular expressions`_.
61
62* Lines ending with a space and the keyword ``(glob)`` are matched
63 with a glob-like syntax. The only special characters supported are
64 ``*`` and ``?``. Both characters can be escaped using ``\``, and the
65 backslash can be escaped itself.
66
67* Output lines ending with either of the above keywords are always
68 first matched literally with actual command output.
69
70* Lines ending with a space and the keyword ``(no-eol)`` will match
71 actual output that doesn't end in a newline.
72
73* Actual output lines containing unprintable characters are escaped
74 and suffixed with a space and the keyword ``(esc)``. Lines matching
75 unprintable output must also contain the keyword.
76
77* Anything else is a comment.
78
79.. _Cram's own test suite: https://bitbucket.org/brodie/cram/src/default/tests/cram.t
80.. _Perl-compatible regular expressions: https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions
81
82
83Download
84--------
85
86* `cram-0.7.tar.gz`_ (32 KB, requires Python 2.4-2.7 or Python 3.1 or newer)
87
88.. _cram-0.7.tar.gz: https://bitheap.org/cram/cram-0.7.tar.gz
89
90
91Installation
92------------
93
94Install Cram using make::
95
96 $ wget https://bitheap.org/cram/cram-0.7.tar.gz
97 $ tar zxvf cram-0.7.tar.gz
98 $ cd cram-0.7
99 $ make install
100
101
102Usage
103-----
104
105Cram will print a dot for each passing test. If a test fails, a
106`unified context diff`_ is printed showing the test's expected output
107and the actual output. Skipped tests (empty tests and tests that exit
108with return code ``80``) are marked with ``s`` instead of a dot.
109
110For example, if we run Cram on `its own example tests`_::
111
112 .s.!
113 --- examples/fail.t
114 +++ examples/fail.t.err
115 @@ -3,21 +3,22 @@
116 $ echo 1
117 1
118 $ echo 1
119 - 2
120 + 1
121 $ echo 1
122 1
123
124 Invalid regex:
125
126 $ echo 1
127 - +++ (re)
128 + 1
129
130 Offset regular expression:
131
132 $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n'
133 foo
134 + bar
135 baz
136
137 \d (re)
138 [A-Z] (re)
139 - #
140 + @
141 s.
142 # Ran 6 tests, 2 skipped, 1 failed.
143
144Cram will also write the test with its actual output to
145``examples/fail.t.err``, allowing you to use other diff tools. This
146file is automatically removed the next time the test passes.
147
148When you're first writing a test, you might just write the commands
149and run the test to see what happens. If you run Cram with ``-i`` or
150``--interactive``, you'll be prompted to merge the actual output back
151into the test. This makes it easy to quickly prototype new tests.
152
153You can specify a default set of options by creating a ``.cramrc``
154file. For example::
155
156 [cram]
157 verbose = True
158 indent = 4
159
160Is the same as invoking Cram with ``--verbose`` and ``--indent=4``.
161
162To change what configuration file Cram loads, you can set the
163``CRAMRC`` environment variable. You can also specify command line
164options in the ``CRAM`` environment variable.
165
166Note that the following environment variables are reset before tests
167are run:
168
169* ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's
170 ``tmp`` directory.
171
172* ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``.
173
174* ``TZ`` is set to ``GMT``.
175
176* ``COLUMNS`` is set to ``80``. (Note: When using ``--shell=zsh``,
177 this cannot be reset. It will reflect the actual terminal's width.)
178
179* ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string.
180
181Cram also provides the following environment variables to tests:
182
183* ``CRAMTMP``, set to the test runner's temporary directory.
184
185* ``TESTDIR``, set to the directory containing the test file.
186
187* ``TESTFILE``, set to the basename of the current test file.
188
189* ``TESTSHELL``, set to the value specified by ``--shell``.
190
191Also note that care should be taken with commands that close the test
192shell's ``stdin``. For example, if you're trying to invoke ``ssh`` in
193a test, try adding the ``-n`` option to prevent it from closing
194``stdin``. Similarly, if you invoke a daemon process that inherits
195``stdout`` and fails to close it, it may cause Cram to hang while
196waiting for the test shell's ``stdout`` to be fully closed.
197
198.. _unified context diff: https://en.wikipedia.org/wiki/Diff#Unified_format
199.. _its own example tests: https://bitbucket.org/brodie/cram/src/default/examples/
200
201
202Development
203-----------
204
205Download the official development repository using Mercurial_::
206
207 hg clone https://bitbucket.org/brodie/cram
208
209Or Git_::
210
211 git clone https://github.com/brodie/cram.git
212
213Test Cram using Cram::
214
215 pip install -r requirements.txt
216 make test
217
218Visit Bitbucket_ or GitHub_ if you'd like to fork the project, watch
219for new changes, or report issues.
220
221.. _Mercurial: http://mercurial.selenic.com/
222.. _Git: http://git-scm.com/
223.. _coverage.py: http://nedbatchelder.com/code/coverage/
224.. _Bitbucket: https://bitbucket.org/brodie/cram
225.. _GitHub: https://github.com/brodie/cram
226