runtests: support creating more than one runner process

The controller currently only creates and uses one, but more are now
possible.

Ref: #10818

runtests: spawn a new process for the test runner

When the -j option is given, a new process is spawned in which the test
programs are run and from which test servers are started. Only o

runtests: spawn a new process for the test runner

When the -j option is given, a new process is spawned in which the test
programs are run and from which test servers are started. Only one
process can be started at once, but this is sufficient to test that the
infrastructure can isolate those functions in a new task. There should
be no visible difference between the two modes at the moment.

Ref: #10818
Closes #11064

show more ...

runtests: turn singletest() into a state machine

This allows it to run in a non-blocking manner.

Ref: #10818

runtests: change runner interface to be asynchronous

Program arguments are marshalled and then written to the end of a pipe
which is later read from and the arguments unmarshalled before

runtests: change runner interface to be asynchronous

Program arguments are marshalled and then written to the end of a pipe
which is later read from and the arguments unmarshalled before the
desired function is called normally. The function return values are
then marshalled and written into another pipe when is later read from
and unmarshalled before being returned to the caller.

The implementation is currently blocking but can be made non-blocking
without any changes to the API. This allows calling multiple runners
without blocking in the future.

Ref: #10818

show more ...

runtests: add a runner initialization function

This sets up the runner environment to start running tests.

Ref: #10818

runtests: remove directory from server filename variables

There will soon be multiple log directories so the paths will no longer
be static in runtests.pl. Also, get rid of $SERVER2IN wh

runtests: remove directory from server filename variables

There will soon be multiple log directories so the paths will no longer
be static in runtests.pl. Also, get rid of $SERVER2IN which was not
used.

Ref: #10818

show more ...

runtests: use a function to obtain $LOGDIR for a test

This will no longer be static soon.

Ref: #10818

runtests: support buffering log messages in runner & servers

Log messages generated with logmsg can now be buffered and returned from
the runner as a return value. This will be needed w

runtests: support buffering log messages in runner & servers

Log messages generated with logmsg can now be buffered and returned from
the runner as a return value. This will be needed with parallel testing
to allow all messages for one test to be displayed together instead of
interspersed with messages of multiple tests. Buffering can be disabled
by setting a logging callback function with setlogfunc, which is
currently being done to preserve existing logging behaviour for now.

Some additional output is generated in verbose and debugprotocol modes,
which don't always use logmsg. These modes also impact some servers
which generate extra messages. No attempt is made to buffer everything
if these modes are enabled.

Ref: #10818
Closes #11016

show more ...

runtests: create runner functions for clearlocks and stopservers

runtests.pl now uses runner for all server actions beyond the initial
variable configuration.

Ref: #10818

runtests: tightened servers package exports

The defaults are intended for runtests.pl, whereas runner.pm needs to
explicitly specify them.

runtests: display logs on server failure in singletest()

This is closer to the place where logs are displayed on test failure.
Also, only display these logs if -p is given, which is the

runtests: display logs on server failure in singletest()

This is closer to the place where logs are displayed on test failure.
Also, only display these logs if -p is given, which is the same flag
that controls display of test failure logs. Some server log files
need to be deleted later so that they stay around long enough to be
displayed on failure.

Ref: #10818

show more ...

runtests: turn a print into a logmsg

Also enable another couple of useful messages in verbose mode.

Ref: #10818

runtests: refactor the main test loop into two

The test loop now has an initial loop that first runs through all
possible tests to build a set of those to attempt on this run based on

runtests: refactor the main test loop into two

The test loop now has an initial loop that first runs through all
possible tests to build a set of those to attempt on this run based on
features and keywords and only then goes through that new list to run
them. This actually makes it three loops through all tests cases, as
there is an existing loop that gathers possible test numbers from the
test files on disk.

This has two minor effects on the output: all the tests that will be
skipped are displayed at the start (instead of being interspersed with
other tests) and the -l option no longer shows a count of tests at the
end or a (misleading) statement that tests have run successfully. The
skipped tests are also omitted from the test results sent to AppVeyor
and Azure in CI builds.

Another effect is a reduction in the amount of work considered part of
the "Test definition reading and preparation time" reported with -r
making those figures slightly lower than before.

Ref: #10818

show more ...

runtests: track only the current test timings in runner.pm

This avoids passing these data through through global variables, which
soon won't be possible.

Ref: #10818

runtests: simplify value returned regarding use of valgrind

As a side effect this will now also show in verbose mode that valgrind
is being skipped on tests that explicitly disable it, s

runtests: simplify value returned regarding use of valgrind

As a side effect this will now also show in verbose mode that valgrind
is being skipped on tests that explicitly disable it, such as 600.

Ref: #10818

show more ...

runtests: fix problems on <killserver> failure

The verify time must be set in this case, like all cases. An error
message needs to be displayed as well.

runtests: fix perl warning when <tool> is wrong

runtests: refactor into more packages

testutil.pm now contains a few miscellaneous functions that are used in
several places but have no better place to live. subvariables moves to

runtests: refactor into more packages

testutil.pm now contains a few miscellaneous functions that are used in
several places but have no better place to live. subvariables moves to
servers.pm since most variables that it substitutes relate to servers,
so this is the most appropriate place. Rename a few functions for better
naming consistency.

Ref: #10818
Closes #10995

show more ...

runtests: call timestampskippedevents() in singletest

..rather than by the runner

runtests: assume a newer Valgrind by default

The tests for an older Valgrind version should probably just be deleted,
given that they're testing for an 18-year-old version.

runtests: refactor test runner code into runner.pm

This is code that is directly responsible for running a single test.
This will eventually run in a separate process as part of the para

runtests: refactor test runner code into runner.pm

This is code that is directly responsible for running a single test.
This will eventually run in a separate process as part of the parallel
testing project.

Ref: #10818

show more ...