README
1# vim: ft=rst
2
3This directory contains test scripts that are useful for running a
4bunch of tests all at once.
5
6There are two parts to this:
7
8 * The test runner (selftest/selftest.pl)
9 * The test formatter
10
11selftest.pl simply outputs subunit, which can then be formatted or analyzed
12by tools that understand the subunit protocol. One of these tools is
13format-subunit, which is used by default as part of "make test".
14
15Available testsuites
16====================
17The available testsuites are obtained from a script, usually
18source{3,4}/selftest/tests.py. This script should for each testsuite output
19the name of the test, the command to run and the environment that should be
20provided. Use the included "plantest" function to generate the required output.
21
22Testsuite behaviour
23===================
24
25Exit code
26------------
27The testsuites should exit with a non-zero exit code if at least one
28test failed. Skipped tests should not influence the exit code.
29
30Output format
31-------------
32Testsuites can simply use the exit code to indicate whether all of their
33tests have succeeded or one or more have failed. It is also possible to
34provide more granular information using the Subunit protocol.
35
36This protocol works by writing simple messages to standard output. Any
37messages that can not be interpreted by this protocol are considered comments
38for the last announced test.
39
40For a full description of the subunit protocol, see the README file in the subunit
41repository at http://github.com/testing-cabal/subunit.
42
43The following commands are Samba extensions to Subunit:
44
45start-testsuite
46~~~~~~~~~~~~~~~
47start-testsuite: name
48
49The testsuite name is used as prefix for all containing tests.
50
51skip-testsuite
52~~~~~~~~~~~~~~
53skip-testsuite: name
54
55Mark the testsuite with the specified name as skipped.
56
57testsuite-success
58~~~~~~~~~~~~~~~~~
59testsuite-success: name
60
61Indicate that the testsuite has succeeded successfully.
62
63testsuite-fail
64~~~~~~~~~~~~~~
65testsuite-fail: name
66
67Indicate that a testsuite has failed.
68
69Environments
70============
71Tests often need to run against a server with particular things set up,
72a "environment". This environment is provided by the test "target": Samba 3,
73Samba 4 or Windows.
74
75The environments are currently available include
76
77 - none: No server set up, no variables set.
78 - dc,s3dc: Domain controller set up. The following environment variables will
79 be set:
80
81 * USERNAME: Administrator user name
82 * PASSWORD: Administrator password
83 * DOMAIN: Domain name
84 * REALM: Realm name
85 * SERVER: DC host name
86 * SERVER_IP: DC IPv4 address
87 * SERVER_IPV6: DC IPv6 address
88 * NETBIOSNAME: DC NetBIOS name
89 * NETIOSALIAS: DC NetBIOS alias
90
91 - member,s4member,s3member: Domain controller and member server that is joined to it set up. The
92 following environment variables will be set:
93
94 * USERNAME: Domain administrator user name
95 * PASSWORD: Domain administrator password
96 * DOMAIN: Domain name
97 * REALM: Realm name
98 * SERVER: Name of the member server
99
100See Samba.pm, Samba3.pm and Samba4.pm for the full list.
101
102Running tests
103=============
104
105To run all the tests use::
106
107 make test
108
109To run a quicker subset run::
110
111 make quicktest
112
113To run a specific test, use this syntax::
114
115 make test TESTS=testname
116
117for example::
118
119 make test TESTS=samba4.BASE-DELETE
120
121