xref: /openssl/test/README-dev.md (revision badf3c16)
1Guidelines for test developers
2==============================
3
4How to add recipes
5------------------
6
7For any test that you want to perform, you write a script located in
8`test/recipes/`, named `{nn}-test_{name}.t`,
9where `{nn}` is a two digit number and
10`{name}` is a unique name of your choice.
11
12Please note that if a test involves a new testing executable, you will need to
13do some additions in test/build.info. Please refer to the section
14["Changes to test/build.info"](README.md#changes-to-testbuildinfo) below.
15
16Naming conventions
17------------------
18
19A test executable is named `test/{name}test.c`
20
21A test recipe is named `test/recipes/{nn}-test_{name}.t`, where `{nn}` is a two
22digit number and `{name}` is a unique name of your choice.
23
24The number `{nn}` is (somewhat loosely) grouped as follows:
25
26    00-04  sanity, internal and essential API tests
27    05-09  individual symmetric cipher algorithms
28    10-14  math (bignum)
29    15-19  individual asymmetric cipher algorithms
30    20-24  openssl commands (some otherwise not tested)
31    25-29  certificate forms, generation and verification
32    30-35  engine and evp
33    60-79  APIs:
34       60  X509 subsystem
35       61  BIO subsystem
36       65  CMP subsystem
37       70  PACKET layer
38    80-89  "larger" protocols (CA, CMS, OCSP, SSL, TSA)
39    90-98  misc
40    99     most time consuming tests [such as test_fuzz]
41
42A recipe that just runs a test executable
43-----------------------------------------
44
45A script that just runs a program looks like this:
46
47    #! /usr/bin/env perl
48
49    use OpenSSL::Test::Simple;
50
51    simple_test("test_{name}", "{name}test", "{name}");
52
53`{name}` is the unique name you have chosen for your test.
54
55The second argument to `simple_test` is the test executable, and `simple_test`
56expects it to be located in `test/`
57
58For documentation on `OpenSSL::Test::Simple`,
59do `perldoc util/perl/OpenSSL/Test/Simple.pm`.
60
61A recipe that runs a more complex test
62--------------------------------------
63
64For more complex tests, you will need to read up on Test::More and
65OpenSSL::Test.  Test::More is normally preinstalled, do `man Test::More` for
66documentation.  For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm`.
67
68A script to start from could be this:
69
70    #! /usr/bin/env perl
71
72    use strict;
73    use warnings;
74    use OpenSSL::Test;
75
76    setup("test_{name}");
77
78    plan tests => 2;                # The number of tests being performed
79
80    ok(test1, "test1");
81    ok(test2, "test1");
82
83    sub test1
84    {
85        # test feature 1
86    }
87
88    sub test2
89    {
90        # test feature 2
91    }
92
93Changes to test/build.info
94--------------------------
95
96Whenever a new test involves a new test executable you need to do the
97following (at all times, replace {NAME} and {name} with the name of your
98test):
99
100 * add `{name}` to the list of programs under `PROGRAMS_NO_INST`
101
102 * create a three line description of how to build the test, you will have
103   to modify the include paths and source files if you don't want to use the
104   basic test framework:
105
106       SOURCE[{name}]={name}.c
107       INCLUDE[{name}]=.. ../include ../apps/include
108       DEPEND[{name}]=../libcrypto libtestutil.a
109
110Generic form of C test executables
111----------------------------------
112
113    #include "testutil.h"
114
115    static int my_test(void)
116    {
117        int testresult = 0;                 /* Assume the test will fail    */
118        int observed;
119
120        observed = function();              /* Call the code under test     */
121        if (!TEST_int_eq(observed, 2))      /* Check the result is correct  */
122            goto end;                       /* Exit on failure - optional   */
123
124        testresult = 1;                     /* Mark the test case a success */
125    end:
126        cleanup();                          /* Any cleanup you require      */
127        return testresult;
128    }
129
130    int setup_tests(void)
131    {
132        ADD_TEST(my_test);                  /* Add each test separately     */
133        return 1;                           /* Indicates success.  Return 0 */
134                                            /* to produce an error with a   */
135                                            /* usage message and -1 for     */
136                                            /* failure to set up with no    */
137                                            /* usage message.               */
138    }
139
140You should use the `TEST_xxx` macros provided by `testutil.h` to test all failure
141conditions.  These macros produce an error message in a standard format if the
142condition is not met (and nothing if the condition is met).  Additional
143information can be presented with the `TEST_info` macro that takes a `printf`
144format string and arguments.  `TEST_error` is useful for complicated conditions,
145it also takes a `printf` format string and argument.  In all cases the `TEST_xxx`
146macros are guaranteed to evaluate their arguments exactly once.  This means
147that expressions with side effects are allowed as parameters.  Thus,
148
149    if (!TEST_ptr(ptr = OPENSSL_malloc(..)))
150
151works fine and can be used in place of:
152
153    ptr = OPENSSL_malloc(..);
154    if (!TEST_ptr(ptr))
155
156The former produces a more meaningful message on failure than the latter.
157
158Note that the test infrastructure automatically sets up all required environment
159variables (such as `OPENSSL_MODULES`, `OPENSSL_CONF`, etc.) for the tests.
160Individual tests may choose to override the default settings as required.
161