Centralise Environment Variables for the tests

The test_includes test was failing if OPENSSL_CONF_INCLUDE happened to
be set in the user's environment. To ensure that no tests accidental

Centralise Environment Variables for the tests

The test_includes test was failing if OPENSSL_CONF_INCLUDE happened to
be set in the user's environment. To ensure that no tests accidentally
use this or other enviroment variables from the user's environment we
automatically set them centrally for all tests.

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/11691)

show more ...

Rework test/run_tests.pl to support selective verbosity and TAP copy

This includes a complete rework of how we use TAP::Harness, by adding
a TAP::Parser subclass that allows additional c

Rework test/run_tests.pl to support selective verbosity and TAP copy

This includes a complete rework of how we use TAP::Harness, by adding
a TAP::Parser subclass that allows additional callbacks to be passed
to perform what we need. The TAP::Parser callbacks we add are:

ALL to print all the TAP output to a file (conditionally)
to collect all the TAP output to an array (conditionally)
EOF to print all the collected TAP output (if there is any)
if any subtest failed

To get TAP output to file, the environment variable HARNESS_TAP_COPY
must be defined, with a file name as value. That file will be
overwritten unconditionally.

To get TAP output displayed on failure, the make variable VERBOSE_FAILURE
or VF must be defined with a non-emoty value.

Additionally, the output of test recipe names has been changed to only
display its basename.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9862)

show more ...

Following the license change, modify the boilerplates in test/

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7767)

Update copyright year

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6145)

Use the config file from the source not the host for the tests

Fixes: #6046

Reviewed-by: Rich Salz <rsalz@openssl.org>

GH: #6125

test/run_tests.pl: don't use Module::Load::Conditional.

Ironically enough not all installations get Module::Load::Conditional
installed by default... [It's a bit half-hearted, because su

test/run_tests.pl: don't use Module::Load::Conditional.

Ironically enough not all installations get Module::Load::Conditional
installed by default... [It's a bit half-hearted, because such
installations are likely to lack more stuffi that is needed, but
nevertheless, it proved to be helpful.]

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4323)

show more ...

File::Glob option ':bsd_glob' doesn't work everywhere, replace w/ a wrapper

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4069)

Consolidate the locations where we have our internal perl modules

Instead of having perl modules under test/testlib, util and util/perl,
consolidate them all to be inside util/perl.

Consolidate the locations where we have our internal perl modules

Instead of having perl modules under test/testlib, util and util/perl,
consolidate them all to be inside util/perl.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4069)

show more ...

Perl: Use File::Glob::bsd_glob rather than File::Glob::glob

File::Glob::glob is deprecated, it's use generates this kind of
message:

File::Glob::glob() will disappear in per

Perl: Use File::Glob::bsd_glob rather than File::Glob::glob

File::Glob::glob is deprecated, it's use generates this kind of
message:

File::Glob::glob() will disappear in perl 5.30. Use File::Glob::bsd_glob() instead. at ../master/Configure line 277.

So instead, use a construction that makes the caller glob() use
File::Glob::bsd_glob().

Note that we're still excluding VMS, as it's directory specs use '['
and ']', which have a different meaning with bsd_glob and would need
some extra quoting. This might change, but later.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4040)

show more ...

test/run_tests.pl: Make sure to exit with a code that's understood universally

TAP::Parser::Aggregator::has_errors may return any number, not just 0
and 1. With Perl on VMS, any number

test/run_tests.pl: Make sure to exit with a code that's understood universally

TAP::Parser::Aggregator::has_errors may return any number, not just 0
and 1. With Perl on VMS, any number from 2 and on is interpreted as a
VMS status, the 3 lower bits are the encoded severity (1 = SUCCESS,
for example), so depending on what has_errors returns, a test failure
might be interpreted as a success. Therefore, it's better to make
sure the exit code is 0 or 1, nothing else (they are special on VMS,
and mean SUCCESS or FAILURE, to match Unix conventions).

Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3880)

show more ...

Fix use of "can_load()" in run_tests.pl.

CLA: Trivial

Fixes #3563.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>

Fix use of "can_load()" in run_tests.pl.

CLA: Trivial

Fixes #3563.

Reviewed-by: Andy Polyakov <appro@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3564)

show more ...

test/run_tests.pl: don't mask test failures.

Switch to TAP::Harness inadvertently masked test failures.
Test::Harness::runtests was terminating with non-zero exit code in case
of fai

test/run_tests.pl: don't mask test failures.

Switch to TAP::Harness inadvertently masked test failures.
Test::Harness::runtests was terminating with non-zero exit code in case
of failure[s], while TAP::Harness apparently holds caller responsible
for doing so.

Reviewed-by: Rich Salz <rsalz@openssl.org>

show more ...

Prefer TAP::Harness over Test::Harness

TAP:Harness came along in perl 5.10.1, and since we claim to support
perl 5.10.0 in configuration and testing, we can only load it
conditionall

Prefer TAP::Harness over Test::Harness

TAP:Harness came along in perl 5.10.1, and since we claim to support
perl 5.10.0 in configuration and testing, we can only load it
conditionally.

The main reason to use TAP::Harness rather than Test::Harness is its
capability to merge stdout and stderr output from the test recipes,
which Test::Harness can't. The merge gives much more comprehensible
output when testing verbosely.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3424)

show more ...

Make it possible to select or deselect test groups by number

Examples of possible expressions (adapt to your platform):

make test TESTS=-99
make test TESTS=10
ma

Make it possible to select or deselect test groups by number

Examples of possible expressions (adapt to your platform):

make test TESTS=-99
make test TESTS=10
make test TESTS=-9?
make test TESTS=-[89]0
make test TESTS=[89]0

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2887)

show more ...

Add a more versatile test chooser

So far, the test runner (test/run_tests.pl) could get a list of tests
to run, and if non were given, it assumes all available tests should
be perfor

Add a more versatile test chooser

So far, the test runner (test/run_tests.pl) could get a list of tests
to run, and if non were given, it assumes all available tests should
be performed.

However, that makes skipping just one or two tests a bit of a pain.
This change makes the possibilities more versatile, run_checker.pl
takes these arguments and will process them in the given order,
starting with an empty set of tests to perform:

alltests The current set becomes the whole set of
available tests.
test_xxx Adds 'test_xxx' to the current set.
-test_xxx Removes 'test_xxx' from the current set. If
nothing has been added to the set before this
argument, the current set is first initialised
to the whole set of available tests, then
'test_xxx' is removed from the current set.
list Display all available tests, then stop.

If no arguments are given, 'alltests' is assumed.

Reviewed-by: Emilia Käsper <emilia@openssl.org>

show more ...

perl: use the 'if' module to conditionally load File::Glob

Trying to use normal perl conditions to conditionally 'use' a perl
module didn't quite work. Using the 'if' module to do so do

perl: use the 'if' module to conditionally load File::Glob

Trying to use normal perl conditions to conditionally 'use' a perl
module didn't quite work. Using the 'if' module to do so does work.

Reviewed-by: Andy Polyakov <appro@openssl.org>

show more ...

Configure,test/recipes: "pin" glob to File::Glob::glob.

As it turns out default glob's behaviour for quoted argument varies
from version to version, making it impossible to Configure or

Configure,test/recipes: "pin" glob to File::Glob::glob.

As it turns out default glob's behaviour for quoted argument varies
from version to version, making it impossible to Configure or run
tests in some cases. The reason for quoting globs was to accommodate
source path with spaces in its name, which was treated by default glob
as multiple paths. File::Glob::glob on the other hand doesn't consider
spaces as delimiters and therefore works with unquoted patterns.

[Unfortunaltely File::Glob::glob, being too csh-ly, doesn't work
on VMS, hence the "pinning" is conditional.]

Reviewed-by: Richard Levitte <levitte@openssl.org>

show more ...

perl glob: make sure to put quotes around the pattern, in case of spaces

RT#4486

Reviewed-by: Matt Caswell <matt@openssl.org>

Replace cipherlist test

The old cipherlist test in ssltest.c only tests the internal order of
the cipher table, which is pretty useless.

Replace this test with a test that catch

Replace cipherlist test

The old cipherlist test in ssltest.c only tests the internal order of
the cipher table, which is pretty useless.

Replace this test with a test that catches inadvertent changes to the
default cipherlist.

Fix run_tests.pl to correctly filter tests that have "list" in their name.

(Also includes a small drive-by fix in .gitignore.)

Reviewed-by: Rich Salz <rsalz@openssl.org>

show more ...

Recognise VERBOSE and V as well as HARNESS_VERBOSE

PR#4462

Reviewed-by: Rich Salz <rsalz@openssl.org>

Copyright consolidation: perl files

Add copyright to most .pl files
This does NOT cover any .pl file that has other copyright in it.
Most of those are Andy's but some are public doma

Copyright consolidation: perl files

Add copyright to most .pl files
This does NOT cover any .pl file that has other copyright in it.
Most of those are Andy's but some are public domain.
Fix typo's in some existing files.

Reviewed-by: Richard Levitte <levitte@openssl.org>

show more ...

unified build scheme: adjust test framework for out of source build tree

To be able to run tests when we've built in a directory other than
the source tree, the testing framework needs a

unified build scheme: adjust test framework for out of source build tree

To be able to run tests when we've built in a directory other than
the source tree, the testing framework needs a few adjustments.

test/testlib/OpenSSL/Test.pm needs to know where it can find
shlib_wrap.sh, and a number of other tests need to be told a different
place to find engines than what they may be able to figure out on
their own. Relying to $TOP is not enough, $SRCTOP and $BLDTOP can be
used as an alternative.

As part of this change, top_file and top_dir are removed and
srctop_file, bldtop_file, srctop_dir and bldtop_dir take their place.

Reviewed-by: Ben Laurie <ben@openssl.org>

show more ...

few typo fixes

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Add a method to list available tests

Reviewed-by: Emilia Käsper <emilia@openssl.org>

Adapt the libssl test harness testing scripts to new testing framework

This involves adding $TOP/util as perl library in test/run_tests.pl.

Reviewed-by: Rich Salz <rsalz@openssl.org>