Make unrepeatable tests retriable

This was an error in thinking on my side. The reason these tests are
unrepeatable is because --repeat executes the same request in the same process,

Make unrepeatable tests retriable

This was an error in thinking on my side. The reason these tests are
unrepeatable is because --repeat executes the same request in the same process,
and does not run the CLEAN section in between runs. This is not the case when
retrying tests.

We could potentially make CLEAN tests repeatable by including the CLEAN section
in the tested script. This does however not work for all tests (e.g. tests that
set open_basedir).

Closes GH-12072

show more ...

Fix passing null to parameter of type string (#12014)

This fixes builds without cgi or phpdbg:
./configure --disable-cgi --disable-phpdbg
make
./sapi/cli/php run-tests.php

Fix passing null to parameter of type string (#12014)

This fixes builds without cgi or phpdbg:
./configure --disable-cgi --disable-phpdbg
make
./sapi/cli/php run-tests.php

Otherwise, deprecation warnings (since PHP-8.1) are emitted:
Deprecated: escapeshellarg(): Passing null to parameter #1 ($arg) of
type string is deprecated in run-tests.php...

show more ...

Skip xleak tests on asan (#11610)

Fixes GH-11077

Implement flaky test section

This re-uses the already used for automatic retesting. That's certainly better
than XFAIL.

Closes GH-11325

[skip ci] Add more patterns to run-tests.php retry list

CURL: 404: Page Not Found
IMAP: Can't create a temporary mailbox: [ALREADYEXISTS] Mailbox already exists
Sockets: socket_bind(

[skip ci] Add more patterns to run-tests.php retry list

CURL: 404: Page Not Found
IMAP: Can't create a temporary mailbox: [ALREADYEXISTS] Mailbox already exists
Sockets: socket_bind(): Unable to bind address [98]: Address already in use

show more ...

Add retry mechanism in run-tests.php

We have lots of spurious failures in CI, many of them with the "all" CONFLICT.
We're limiting the retrying to specific error messages. In the future

Add retry mechanism in run-tests.php

We have lots of spurious failures in CI, many of them with the "all" CONFLICT.
We're limiting the retrying to specific error messages. In the future we may
also provide a FLAKY section to retry specific tests.

Closes GH-10892

show more ...

Add zend_alloc XLEAK support

In the future we may want to use a different exit code to warn for tests that
didn't leak.

Closes GH-10999

Add ASAN XLEAK support

Only disable LSAN instead of skipping the test. This way we can still detect
memory issues which is arguably more important anyway.

Closes GH-10996

[skip ci] Avoid infinite loop in differ

TBH I don't understand why this can happen here but not in sebastian/diff. I'll
have to take a closer look.

Fix one more differ direct comparison (through in_array)

Fix direct comparison in run-tests.php differ

[skip ci] Fix var_dump statement in run-tests.php

Implement better diff for run-tests.php

Borrow sebastianbergmann/diff with
MemoryEfficientLongestCommonSubsequenceCalculator

Fixes GH-10806
Closes GH-10875

Another attempt to fix MSAN nightly on master

use_tls=0 on MSAN

Attempt to fix MSAN failure in CI

Closes GH-10851

[skip ci] Fix diff cmd env name in run-tests.php

[skip ci] Remove accidental run-tests.php line

Fix warning in run-tests when PHP compiled without generating phpdbg support. (#10745)

Fix GH-10489: run-tests.php does not escape path when building cmd (#10560)

Multiple tests had to be changed to escape the arguments in shell
commands. Some tests are skipped because th

Fix GH-10489: run-tests.php does not escape path when building cmd (#10560)

Multiple tests had to be changed to escape the arguments in shell
commands. Some tests are skipped because they behave differently with
spaces in the path versus without. One notable example of this is the
hashbang test which does not work because spaces in hashbangs paths are
not supported in Linux.

Co-authored-by: Michael Voříšek <mvorisek@mvorisek.cz>

show more ...

Fix run-tests.php hanging when a worker process dies without notice (#9931)

run-tests.php with `-jN` can hang if a parallel worker dies without notice. This
can happen due to fatal error

Fix run-tests.php hanging when a worker process dies without notice (#9931)

run-tests.php with `-jN` can hang if a parallel worker dies without notice. This
can happen due to fatal errors in the worker, or if the worker is killed.

- run-tests.php (main process)
\_ run-tests.php (worker #0) // main process hangs if this one crashes
\_ test-001.php (test-001.phpt)

show more ...

Migrate i386 to GitHub actions

Closes GH-9856

Make run-tests.php check for tcp fwrite edge cases (#9242)

When the recipient is busy or the payload is large, fwrite can block
or return a value smaller than the length of the stream.

Make run-tests.php check for tcp fwrite edge cases (#9242)

When the recipient is busy or the payload is large, fwrite can block
or return a value smaller than the length of the stream.

workers in run-tests.php communicates over tcp sockets with the manager.

https://cirrus-ci.com/task/5315675320221696?logs=tests#L130
showed notices for fwrite/unserialize

This is a similar approach to the approach used in
https://github.com/phan/phan/blob/v5/src/Phan/LanguageServer/ProtocolStreamWriter.php
for the tcp language server writing.

show more ...

Do not require opcache.preload_user in cli SAPIs

Fix run-tests.php for explicitly given test cases

The recent improvement to list skipped extensions explicitly[1] missed
to properly initialize `$ignored_by_ext` for the case where an ex

Fix run-tests.php for explicitly given test cases

The recent improvement to list skipped extensions explicitly[1] missed
to properly initialize `$ignored_by_ext` for the case where an explicit
set of test cases are given; this was not a problem previously, since
the undefined *global* variable was coerced to int. We fix this by
initializing the variable earlier.

[1] <https://github.com/php/php-src/commit/baef47ef3eafc37a4e69ef4ca530206517425768>

Closes GH-9617.

show more ...

List skipped extensions explicitly

We also fix the summary alignment for 10k+ tests.

Closes GH-8363.