1How to add recipes
2==================
3
4For any test that you want to perform, you write a script located in
5test/recipes/, named {nn}-test_{name}.t, where {nn} is a two digit number and
6{name} is a unique name of your choice.
7
8Please note that if a test involves a new testing executable, you will need to
9do some additions in test/Makefile.  More on this later.
10
11
12Naming conventions
13=================
14
15A test executable is named test/{name}test.c
16
17A test recipe is named test/recipes/{nn}-test_{name}.t, where {nn} is a two
18digit number and {name} is a unique name of your choice.
19
20The number {nn} is (somewhat loosely) grouped as follows:
21
2200-04  sanity, internal and essential API tests
2305-09  individual symmetric cipher algorithms
2410-14  math (bignum)
2515-19  individual asymmetric cipher algorithms
2620-24  openssl commands (some otherwise not tested)
2725-29  certificate forms, generation and verification
2830-35  engine and evp
2960-79  APIs
30   70  PACKET layer
3180-89  "larger" protocols (CA, CMS, OCSP, SSL, TSA)
3290-98  misc
3399     most time consuming tests [such as test_fuzz]
34
35
36A recipe that just runs a test executable
37=========================================
38
39A script that just runs a program looks like this:
40
41    #! /usr/bin/perl
42
43    use OpenSSL::Test::Simple;
44
45    simple_test("test_{name}", "{name}test", "{name}");
46
47{name} is the unique name you have chosen for your test.
48
49The second argument to `simple_test' is the test executable, and `simple_test'
50expects it to be located in test/
51
52For documentation on OpenSSL::Test::Simple, do
53`perldoc util/perl/OpenSSL/Test/Simple.pm'.
54
55
56A recipe that runs a more complex test
57======================================
58
59For more complex tests, you will need to read up on Test::More and
60OpenSSL::Test.  Test::More is normally preinstalled, do `man Test::More' for
61documentation.  For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm'.
62
63A script to start from could be this:
64
65    #! /usr/bin/perl
66
67    use strict;
68    use warnings;
69    use OpenSSL::Test;
70
71    setup("test_{name}");
72
73    plan tests => 2;                # The number of tests being performed
74
75    ok(test1, "test1");
76    ok(test2, "test1");
77
78    sub test1
79    {
80        # test feature 1
81    }
82
83    sub test2
84    {
85        # test feature 2
86    }
87
88
89Changes to test/build.info
90==========================
91
92Whenever a new test involves a new test executable you need to do the
93following (at all times, replace {NAME} and {name} with the name of your
94test):
95
96* add {name} to the list of programs under PROGRAMS_NO_INST
97
98* create a three line description of how to build the test, you will have
99to modify the include paths and source files if you don't want to use the
100basic test framework:
101
102    SOURCE[{name}]={name}.c
103    INCLUDE[{name}]=.. ../include
104    DEPEND[{name}]=../libcrypto libtestutil.a
105
106Generic form of C test executables
107==================================
108
109    #include "testutil.h"
110
111    static int my_test(void)
112    {
113        int testresult = 0;                 /* Assume the test will fail    */
114        int observed;
115
116        observed = function();              /* Call the code under test     */
117        if (!TEST_int_eq(observed, 2))      /* Check the result is correct  */
118            goto end;                       /* Exit on failure - optional   */
119
120        testresult = 1;                     /* Mark the test case a success */
121    end:
122        cleanup();                          /* Any cleanup you require      */
123        return testresult;
124    }
125
126    int setup_tests(void)
127    {
128        ADD_TEST(my_test);                  /* Add each test separately     */
129        return 1;                           /* Indicate success             */
130    }
131
132You should use the TEST_xxx macros provided by testutil.h to test all failure
133conditions.  These macros produce an error message in a standard format if the
134condition is not met (and nothing if the condition is met).  Additional
135information can be presented with the TEST_info macro that takes a printf
136format string and arguments.  TEST_error is useful for complicated conditions,
137it also takes a printf format string and argument.  In all cases the TEST_xxx
138macros are guaranteed to evaluate their arguments exactly once.  This means
139that expressions with side effects are allowed as parameters.  Thus,
140
141    if (!TEST_ptr(ptr = OPENSSL_malloc(..)))
142
143works fine and can be used in place of:
144
145    ptr = OPENSSL_malloc(..);
146    if (!TEST_ptr(ptr))
147
148The former produces a more meaningful message on failure than the latter.
149
150