Use GH action commands to group/collapse filtered output

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/open

Use GH action commands to group/collapse filtered output

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/22678)

show more ...

augment test/run_tests.pl to filter indirect leaks

When verbosity isn't set to 1 or higher, suppress indirect leaks (i.e.
only print direct leaks) to make output more human-readable. Se

augment test/run_tests.pl to filter indirect leaks

When verbosity isn't set to 1 or higher, suppress indirect leaks (i.e.
only print direct leaks) to make output more human-readable. Setting
V=1 on make test produces all leaks (direct and indirect)

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/22678)

show more ...

Update copyright year

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Release: yes

Fix test failure when testing with Test::Harness

Fixes an issue where, when the test suite was being run with the older
Test::Harness package, the test suite would not complete correctly

Fix test failure when testing with Test::Harness

Fixes an issue where, when the test suite was being run with the older
Test::Harness package, the test suite would not complete correctly due
to evaluation of $harness->runtests()->has_errors, which is only
available for the newer TAP::Parser::Aggregator code path.

Fixes #17818.

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17823)

show more ...

Perturb the malloc output during tests

Set the environment variable MALLOC_PERTURB_ during tests to perturb the
output from OPENSSL_malloc() calls (see the mallopt man page for details

Perturb the malloc output during tests

Set the environment variable MALLOC_PERTURB_ during tests to perturb the
output from OPENSSL_malloc() calls (see the mallopt man page for details
about this environment variable). This could be a low cost way of spotting
uninit reads in "make test" runs in some situations.

In tests I have found it to be a little unreliable (sometimes it seemed to
not perturb the output for inexplicable reasons) - but since this is easy
to implement I think it is worthwhile.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17768)

show more ...

Fix exit code for VMS in util/wrap.pl and test/run_tests.pl

The exit code for VMS is a bit tricky, and while perl translates the
VMS status code from a typical C program to posix terms,

Fix exit code for VMS in util/wrap.pl and test/run_tests.pl

The exit code for VMS is a bit tricky, and while perl translates the
VMS status code from a typical C program to posix terms, it doesn't
automatically translate its exit code into the typical C program VMS
status code. Perl scripts are recommended to do so explicitly.

Therefore, we make util/wrap.pl and test/run_tests.pl simulate the
typical C program VMS status code for all non-zero exit codes, except
we give them all the error severity (according to the VMS C library
reference manual, exit codes 2 and above are treated as success...).

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15787)

show more ...

TEST: Change 'catdir' to 'catfile' when dealing with files, in run_tests.pl

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https

TEST: Change 'catdir' to 'catfile' when dealing with files, in run_tests.pl

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15767)

show more ...

Make the 00-prep_*.t recipe truly mandatory

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/open

Make the 00-prep_*.t recipe truly mandatory

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15550)

show more ...

TEST: Add test specific fipsmodule.cnf, and use it

We add the concept of preparation recipes, which are performed
unconditionally. They are all expected to match the pattern
test/re

TEST: Add test specific fipsmodule.cnf, and use it

We add the concept of preparation recipes, which are performed
unconditionally. They are all expected to match the pattern
test/recipes/00-prep_*.t.

We add one such preparation recipe, test/recipes/00-prep_fipsmodule_cnf.t,
which helps us generate a test specific fipsmodule.cnf, to be used by
all other tests.

Fixes #15166

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15436)

show more ...

Update copyright year

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14235)

test: add an option to output timing information from tests.

Fixes #14141

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull

test: add an option to output timing information from tests.

Fixes #14141

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14149)

show more ...

test: turn off parallel tests in verbose mode.

The existing code prints a warning saying that verbose mode is ignored with
parallel jobs. This seems backward, more useful is disabling p

test: turn off parallel tests in verbose mode.

The existing code prints a warning saying that verbose mode is ignored with
parallel jobs. This seems backward, more useful is disabling parallel jobs
when verbose is enabled.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14137)

show more ...

run_tests.pl: Improve diagnostics on the use of HARNESS_JOBS

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13551)

Make TAP::Harness and TAP::Parser optional.

In OpenSSL 1.1.1 the script run_tests.pl has an effectiver
workaround to fall back to Test::Harness, if TAP::Harness
is not available. Tha

Make TAP::Harness and TAP::Parser optional.

In OpenSSL 1.1.1 the script run_tests.pl has an effectiver
workaround to fall back to Test::Harness, if TAP::Harness
is not available. That code has substantially changed,
but it seems it should still fall back but doesn't.

Observed on SuSE Linux Enterprise Server 11 (SLES11).

Error messages:

Can't locate TAP/Parser.pm in @inc (@inc contains: /path/to/bld/openssl300/test/../util/perl /path/to/local/perl/lib/perl5 /usr/lib/perl5/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/5.10.0 /usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5/vendor_perl .) at /path/to/local/perl/lib/perl5/parent.pm line 20.
BEGIN failed--compilation aborted at /path/to/bld/openssl300/test/run_tests.pl line 131.

and

Can't locate TAP/Harness.pm in @inc (@inc contains: /path/to/bld/openssl300/test/../util/perl /path/to/local/perl/lib/perl5 /usr/lib/perl5/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/5.10.0 /usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5/vendor_perl .) at /path/to/local/perl/lib/perl5/parent.pm line 20.
BEGIN failed--compilation aborted at /path/to/bld/openssl300/test/run_tests.pl line 215.

Concerning the fix: the docs for parent.pm show, that without
the "-norequire" it puts the require statement in a BEGIN block
which probably runs before the eval, to the loading is no
longer encapsulated by the eval. Without the additional require
line, the loading doesn't happen at all, so the availability
testing fails. Combining the "-norequire" and an explicit
"require" worked for me.

Tested on the original problem platform SLES 11, but also on
SLES 12 and 15, RHEL 6, 7 and 8 plus Solaris 10 Sparc.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/12500)

show more ...

Avoid uninitialised variable warning for jobs

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail

Avoid uninitialised variable warning for jobs

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/12762)

show more ...

run_tests.pl: Add warning that HARNESS_JOBS > 1 overrides HARNESS_VERBOSE

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/12682)

test/run_tests.pl: In parallel runs, start those tests first that run longest

Also untabify the Perl source file.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from

test/run_tests.pl: In parallel runs, start those tests first that run longest

Also untabify the Perl source file.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/12359)

show more ...

test/run_tests.pl: Add alias REPORT_FAILURES{,_PROGRESS} for VF and VFP

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/12279)

test/run_tests.pl: Add visual separator after failed test case for VFP and VFP modes

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/12

test/run_tests.pl: Add visual separator after failed test case for VFP and VFP modes

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/12279)

show more ...

test/run_tests.pl: Enhance the semantics of HARNESS_VERBOSE_FAILURES (VF)

Make the improved semantics of VFO replace the previous VF and remove VFO
Add warnings about overriding use of H

test/run_tests.pl: Enhance the semantics of HARNESS_VERBOSE_FAILURES (VF)

Make the improved semantics of VFO replace the previous VF and remove VFO
Add warnings about overriding use of HARNESS_VERBOSE* variables

Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/12279)

show more ...

Run tests in parallel

The environment variable `HARNESS_JOBS` can be used to control how many
jobs to run in parallel. The default is still to run jobs sequentially.

This commi

Run tests in parallel

The environment variable `HARNESS_JOBS` can be used to control how many
jobs to run in parallel. The default is still to run jobs sequentially.

This commit does not define custom `rules`, and different versions of
`TAP::Harness` come with different strategies regarding the default
`rules` that define which test recipes can be run in parallel.
In recent versions of Perl, unless specified otherwise any task can be
run in parallel.

Reviewed-by: David von Oheimb <david.von.oheimb@siemens.com>
(Merged from https://github.com/openssl/openssl/pull/12326)

show more ...

test/run_tests.pl: Improve indentation parsing workaround for VFO and VFP mode

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/12175)

test/run_tests.pl: Improve newline output for VFO and VFP mode

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/12175)

run_tests.pl: Add options for focussing output on failed (sub-)tests

VERBOSE_FAILURES_ONLY (VFO): verbose output only of failed (sub-)tests
VERBOSE_FAILURES_PROGRESS (VFP): in addition s

run_tests.pl: Add options for focussing output on failed (sub-)tests

VERBOSE_FAILURES_ONLY (VFO): verbose output only of failed (sub-)tests
VERBOSE_FAILURES_PROGRESS (VFP): in addition summary for passed tests
This adds a workaroud for TAP::Parser not coping well with indentation.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/12088)

show more ...

Update copyright year

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11839)