xref: /openssl/test/README.md (revision cf424d1d)
1Using OpenSSL Tests
2===================
3
4After a successful build, and before installing, the libraries should be tested.
5Run:
6
7    $ make test                                      # Unix
8    $ mms test                                       ! OpenVMS
9    $ nmake test                                     # Windows
10
11**Warning:** you MUST run the tests from an unprivileged account
12(or disable your privileges temporarily if your platform allows it).
13
14If some tests fail, take a look at the section Test Failures below.
15
16Test Failures
17-------------
18
19If some tests fail, look at the output.  There may be reasons for the failure
20that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue).
21You may want increased verbosity, that can be accomplished like this:
22
23Full verbosity, showing full output of all successful and failed test cases
24(`make` macro `VERBOSE` or `V`):
25
26    $ make V=1 test                                  # Unix
27    $ mms /macro=(V=1) test                          ! OpenVMS
28    $ nmake V=1 test                                 # Windows
29
30Verbosity on failed (sub-)tests only
31(`VERBOSE_FAILURE` or `VF` or `REPORT_FAILURES`):
32
33    $ make test VF=1
34
35Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests
36(`VERBOSE_FAILURE_PROGRESS` or `VFP` or `REPORT_FAILURES_PROGRESS`):
37
38    $ make test VFP=1
39
40If you want to run just one or a few specific tests, you can use
41the make variable TESTS to specify them, like this:
42
43    $ make TESTS='test_rsa test_dsa' test            # Unix
44    $ mms/macro="TESTS=test_rsa test_dsa" test       ! OpenVMS
45    $ nmake TESTS="test_rsa test_dsa" test           # Windows
46
47And of course, you can combine (Unix examples shown):
48
49    $ make test TESTS='test_rsa test_dsa' VF=1
50    $ make test TESTS="test_cmp_*" VFP=1
51
52You can find the list of available tests like this:
53
54    $ make list-tests                                # Unix
55    $ mms list-tests                                 ! OpenVMS
56    $ nmake list-tests                               # Windows
57
58Have a look at the manual for the perl module Test::Harness to
59see what other HARNESS_* variables there are.
60
61To report a bug please open an issue on GitHub, at
62<https://github.com/openssl/openssl/issues>.
63
64For more details on how the `make` variables `TESTS` can be used,
65see section Running Selected Tests below.
66
67Running Selected Tests
68----------------------
69
70The `make` variable `TESTS` supports a versatile set of space separated tokens
71with which you can specify a set of tests to be performed.  With a "current
72set of tests" in mind, initially being empty, here are the possible tokens:
73
74     alltests      The current set of tests becomes the whole set of available
75                   tests (as listed when you do 'make list-tests' or similar).
76
77     xxx           Adds the test 'xxx' to the current set of tests.
78
79    -xxx           Removes 'xxx' from the current set of tests.  If this is the
80                   first token in the list, the current set of tests is first
81                   assigned the whole set of available tests, effectively making
82                   this token equivalent to TESTS="alltests -xxx".
83
84     nn            Adds the test group 'nn' (which is a number) to the current
85                   set of tests.
86
87    -nn            Removes the test group 'nn' from the current set of tests.
88                   If this is the first token in the list, the current set of
89                   tests is first assigned the whole set of available tests,
90                   effectively making this token equivalent to
91                   TESTS="alltests -xxx".
92
93Also, all tokens except for "alltests" may have wildcards, such as *.
94(on Unix and Windows, BSD style wildcards are supported, while on VMS,
95it's VMS style wildcards)
96
97### Examples
98
99Run all tests except for the fuzz tests:
100
101    $ make TESTS='-test_fuzz*' test
102
103or, if you want to be explicit:
104
105    $ make TESTS='alltests -test_fuzz*' test
106
107Run all tests that have a name starting with "test_ssl" but not those
108starting with "test_ssl_":
109
110    $ make TESTS='test_ssl* -test_ssl_*' test
111
112Run only test group 10:
113
114    $ make TESTS='10' test
115
116Run all tests except the slow group (group 99):
117
118    $ make TESTS='-99' test
119
120Run all tests in test groups 80 to 99 except for tests in group 90:
121
122    $ make TESTS='[89]? -90' test
123
124To run specific fuzz tests you can use for instance:
125
126    $ make test TESTS='test_fuzz_cmp test_fuzz_cms'
127
128To stochastically verify that the algorithm that produces uniformly distributed
129random numbers is operating correctly (with a false positive rate of 0.01%):
130
131    $ ./util/wrap.sh test/bntest -stochastic
132
133Running Tests in Parallel
134-------------------------
135
136By default the test harness will execute the selected tests sequentially.
137Depending on the platform characteristics, running more than one test job in
138parallel may speed up test execution.
139This can be requested by setting the `HARNESS_JOBS` environment variable to a
140positive integer value. This specifies the maximum number of test jobs to run in
141parallel.
142
143Depending on the Perl version different strategies could be adopted to select
144which test recipes can be run in parallel.  In recent versions of Perl, unless
145specified otherwise, any task can be run in parallel. Consult the documentation
146for `TAP::Harness` to know more.
147
148To run up to four tests in parallel at any given time:
149
150    $ make HARNESS_JOBS=4 test
151
152Random numbers in tests
153-----------------------
154
155Some tests use random numbers as part of the test. In some cases a test failure
156may occur for some random numbers, but not for others. The seed used for the
157rand number generator can be set via the `OPENSSL_TEST_RAND_SEED` environment
158variable. It can also be set via the `OPENSSL_TEST_RAND_ORDER` environment
159variable which additionally randomises the order tests are run in (see below).
160
161When a test fails the test harness will display the seed used during the test
162(displaying either the `OPENSSL_TEST_RAND_SEED` or `OPENSSL_TEST_RAND_ORDER`
163environment variable value that must be used to recreate the results), e.g.
164
165    $ make OPENSSL_TEST_RAND_SEED=42 test
166
167Randomisation of Test Ordering
168------------------------------
169
170By default, the test harness will execute tests in the order they were added.
171By setting the `OPENSSL_TEST_RAND_ORDER` environment variable to zero, the
172test ordering will be randomised. This additionally seeds the random number
173generator used within the tests as described in the section above. If a randomly
174ordered test fails, the seed value used will be reported.  Setting the
175`OPENSSL_TEST_RAND_ORDER` environment variable to this value will rerun the
176tests in the same order and will also seed the test random number generator.
177This assures repeatability of randomly ordered test runs. This repeatability is
178independent of the operating system, processor or platform used.
179
180To randomise the test ordering:
181
182    $ make OPENSSL_TEST_RAND_ORDER=0 test
183
184To run the tests using the order defined by the random seed `42`:
185
186    $ make OPENSSL_TEST_RAND_ORDER=42 test
187