GHA/linux: merge torture jobs into the main workflow

They complete in 4 and 7 minutes, and do not hold back the main Linux
workflow.

Also:
- bump default parallelism for `te

GHA/linux: merge torture jobs into the main workflow

They complete in 4 and 7 minutes, and do not hold back the main Linux
workflow.

Also:
- bump default parallelism for `test-torture` target to `-j20`
(was: `-j2`).
- drop redundant package install from `rustls` jobs.

Closes #15360

show more ...

CI: improvements in test reliability and performance

- CI default: raise parallelism to 20
- CI valgind: set parallelism to 6
- CI non-native: adapt parallelism for OS builds
- C

CI: improvements in test reliability and performance

- CI default: raise parallelism to 20
- CI valgind: set parallelism to 6
- CI non-native: adapt parallelism for OS builds
- CI Windows: no longer ignore FTP, TFTP, MQTT and SMTP
- CI Windows: restrict test timeout to 10 minutes
- CI Windows: do not run tests for msh3 build
- tests, various: restrict curl invocation to ipv4 to avoid talking to
any ipv6 test server running in parallel
- tests: requiring http/2 server no longer needs to mention http server
- test 190: use a fixed timeout of 10 seconds instead of %FTPTIME2, as
that value rises under parallel load beyond what the server is waiting
and then produces different results
- test 1540: add debug logging
- testrunner: add verify checks for http/2 and http/3 server

Closes #15040

show more ...

build: add pytest targets

It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)

Also:

- drop pytest logic l

build: add pytest targets

It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)

Also:

- drop pytest logic launching `make` and exiting in case of failure.
Maybe there is a better way and keep this functionality somehow, bind
it to a command-line option? make it fail softly?

- GHA/linux: invoke pytest via the build, not directly.

- autotools: add missing dummy runtests targets when cross-compiling.

Closes #15034

show more ...

tests: speed up builds with single-binary test bundles

Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools:

tests: speed up builds with single-binary test bundles

Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools: `--enable-test-bundles`
- cmake: `-DCURL_TEST_BUNDLES=ON`

(They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON`
options, for further speed-up.)

Makes libtests and unit tests build _fast_, needing little disk space
even in static mode. Similar to CMake unity mode, but with a custom
script, also supporting autotools builds.

The price is having to deal with symbols/macros colliding between
`lib*.c` and `unit*.c` sources. Maybe with naming conventions or other
solutions this can be improved gradually and reduce the need for manual
intervention by `mk-bundle.mk`. I've included a script that does the bulk
of detecting name collisions.

Also:
- CI: enable test bundles.
- CI: build tests in more jobs.
- lib2305: fix FILE handle leak.
- unit1661: fix memleak found by torture test by releasing the `bufref`
structure in `unit_stop()` that was allocated in `unit_setup()`.
```
test 1661...[bufref unit tests]
Leak detected: memory still allocated: 13 bytes
allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70
1661: torture FAILED: function number 1 in test.
```
Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41

Similar test suite builds with autotools default and cmake+bundle+unity:
- GHA/Linux: 33s vs 7s
https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
- GHA/macOS 34s vs 2s
https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
- GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
- GHA/Cygwin: 9m52 vs 32s
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 3m52 vs 14s
https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 5m45 vs 30s
https://github.com/curl/curl/actions/runs/10705668808/job/29681628787

Autotools test suite builds compared between master -> `--enable-test-bundles`:
- GHA/Linux: 33s -> 9s (run tests: 22m23 -> 20m44)
https://github.com/curl/curl/actions/runs/10710030193/job/29695932185
https://github.com/curl/curl/actions/runs/10967831456/job/30458220344
- GHA/macOS: 25s -> 4s (run tests: 2m58 -> 2m24)
https://github.com/curl/curl/actions/runs/10710030195/job/29695938444
https://github.com/curl/curl/actions/runs/10967831452/job/30458225762
- GHA/non-native (FreeBSD): 4m8 -> 3m12 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10710030198/job/29695928401
https://github.com/curl/curl/actions/runs/10967831458/job/30458212692
- GHA/Cygwin: 9m25 -> 1m9 (run tests: 9m19 -> 3m28)
https://github.com/curl/curl/actions/runs/10710030212/job/29695928213
https://github.com/curl/curl/actions/runs/10967831453/job/30458213268
- GHA/MSYS2: 3m54 -> 32s (run tests: 6m3 -> 3m59)
https://github.com/curl/curl/actions/runs/10710030190/job/29704850591
https://github.com/curl/curl/actions/runs/10967831449/job/30459280005
- GHA/mingw-w64: 5m42 -> 1m5 (run tests: 7m41 -> 5m36)
https://github.com/curl/curl/actions/runs/10710030190/job/29704852058
https://github.com/curl/curl/actions/runs/10967831449/job/30459280862
- Azure MSYS2 mingw64 openssl: 38m55 -> 11m58
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
- Azure Ubuntu default: 2m15 -> 55s (all build)
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7

Cmake test suite builds compared between master -> `-DCURL_TEST_BUNDLES=ON` + unity:
- GHA/Linux: 29s -> 7s (run tests: 4m50 -> 4m57, 20m43 -> 20m45)
https://github.com/curl/curl/actions/runs/10710030193/job/29695941814
https://github.com/curl/curl/actions/runs/10705668823/job/29681622201
- GHA/Linux old: 44s -> 13s (bundle+no unity) (run tests: 5m5 -> 5m6)
https://github.com/curl/curl/actions/runs/10718264094/job/29719794727
https://github.com/curl/curl/actions/runs/10718653175/job/29721009613
- GHA/macOS: 32s -> 2s (run tests: 2m43 -> 2m40)
https://github.com/curl/curl/actions/runs/10710030195/job/29695931956
https://github.com/curl/curl/actions/runs/10705668813/job/29681638937
- GHA/non-native (*BSD): inconclusive (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10710030198
https://github.com/curl/curl/actions/runs/10705668811
- GHA/Cygwin: 3m9 -> 32s
https://github.com/curl/curl/actions/runs/10710030212/job/29695929075
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 2m24 -> 14s
https://github.com/curl/curl/actions/runs/10710030190/job/29704850996
https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 3m56 -> 30s (run tests: 4m2 -> 3m52)
https://github.com/curl/curl/actions/runs/10710030190/job/29704852219
https://github.com/curl/curl/actions/runs/10705668808/job/29681631393
- GHA/mingw-w64-old: 7m19 -> 1m44 (run tests: 3m30 -> 2m53)
https://github.com/curl/curl/actions/runs/10710030190/job/29704849763
https://github.com/curl/curl/actions/runs/10705668808/job/29681622329
- GHA/MSVC: 3m22 -> 13s (run tests: 9m43 -> 4m22)
https://github.com/curl/curl/actions/runs/10710030190/job/29704850411
https://github.com/curl/curl/actions/runs/10705668808/job/29681623313
- AppVeyor CI MSVC 2008: 4m3 -> 45s (full build)
- AppVeyor CI MSVC 2010: 2m56 -> 1m8 (full build)
- AppVeyor CI MSVC 2022: 10m19 -> 2m23 (full build)
https://ci.appveyor.com/project/curlorg/curl/builds/50538455
https://ci.appveyor.com/project/curlorg/curl/builds/50536558
- AppVeyor CI total build time: 10m30 (master) -> 6m48 (unity) -> 4m5 (bundle) -> 3m24 (bundle+unity) -> 5m7 (bundle+unity+all jobs building tests)

Closes #14772

show more ...

build: use `configurehelp.pm.in` with autotools and cmake

Before this patch, each build tool generated `tests/configurehelp.pm`
manually.

Ref: https://github.com/curl/curl/pull/

build: use `configurehelp.pm.in` with autotools and cmake

Before this patch, each build tool generated `tests/configurehelp.pm`
manually.

Ref: https://github.com/curl/curl/pull/14802#issuecomment-2332734326
Closes #14819

show more ...

tests: testrunner reliability improvements

- perform torture tests with '-j2' for shorter runtime
- when waiting on test results overly long, log the tests
waited for and eventuall

tests: testrunner reliability improvements

- perform torture tests with '-j2' for shorter runtime
- when waiting on test results overly long, log the tests
waited for and eventually log the test log directories
for easier analysis what is wrong in CI jobs.
- sockfilt.c: treat the windows errno 109 (ERROR_BROKEN_PIPE)
as a socket closed by the client and do not exit.
- when verifying https server, do not in addition check
the http server behind it also
- when tearing down the stunnel of a non-responsive https
server, tear down the http server with it

Closes #14960

show more ...

tests: constrain http pytest to tests/http directory

Running the http pytest had to be done from tests directory or above,
because the repeat argument fixture was defined in tests/confte

tests: constrain http pytest to tests/http directory

Running the http pytest had to be done from tests directory or above,
because the repeat argument fixture was defined in tests/conftest.py.
However, the repeat argument is not needed because its functionality
can be provided by pytest-repeat as documented in the test's
README.md. So, removed the pytest_addoption function for the repeat
argument and the pytest_report_header function is moved to
tests/http/conftest.py.

TODO: Remove repeat argument from all tests. As a stopgap, a
one-element list is defined for it for now.

Closes #14611

show more ...

curl: --help [option] displays documentation for given cmdline option

Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It

curl: --help [option] displays documentation for given cmdline option

Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".

Then, it looks for the start to display at "\n -[option]" and stops
again at "\n -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.

Test 1707 to 1710 verify

Closes #13997

show more ...

CI: enable parallel testing in CI builds

The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on

CI: enable parallel testing in CI builds

The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs. Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.

Here's a summary of the number of test processes by CI service:

Appveyor - 2 (Windows MSVC), 1 (others)
Azure - 2
Circle CI - 14
Cirrus - 28 (macOS), 14 (Linux), 7 (FreeBSD), 5 (macOS torture), 2 (Windows)
GitHub Actions - 3 (macOS), 2 (Linux)

Some of these are a bit conservative to keep timing-induced flakiness down.

The net result is that the first test results should arrive only
3 minutes after a commit submission.

Changes merged via separate commits:
- 2a7c8b27fdd266894fe24d6d0f5d853ea843dff4 #14171
- 72341068a2d8f1b94a26add16830c725cc4054e6
- efce544418971b064d9fc8183a6caa582c54559b #14244
- c6cf411bacf97f230db160e543d0cc3bbe5b9aba

Ref: #10818
Closes #11510

show more ...

tests: provide docs a as curldown, not nroff

As runtests.md and testcurl.md. Very few people actually need these as
manpages anyway.

With this, we have no more nroff formatted d

tests: provide docs a as curldown, not nroff

As runtests.md and testcurl.md. Very few people actually need these as
manpages anyway.

With this, we have no more nroff formatted documents in git.

Closes #14324

show more ...

runtests: fold timing details with GHA, sync `-r` tflags

- move timing details into a foldable group when run in GitHub Actions.
Spec:
https://docs.github.com/en/actions/using-wo

runtests: fold timing details with GHA, sync `-r` tflags

- move timing details into a foldable group when run in GitHub Actions.
Spec:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#grouping-log-lines

- enable `-r` (run time stats) option in autotools' `test-ci` target,
syncing it with cmake.

Closes #14284

show more ...

curldown: make 'added-in:' a mandatory header field

- generate AVAILABILITY manpage sections automatically - for consistent
wording

- allows us to double-check against other d

curldown: make 'added-in:' a mandatory header field

- generate AVAILABILITY manpage sections automatically - for consistent
wording

- allows us to double-check against other documumentation (symbols-in-versions
etc)

- enables proper automation/scripting based on this data

- lots of them were wrong or missing in the manpages

- several of them repeated (sometimes mismatching) backend support info

Add test 1488 to verify "added-in" version numbers against
symbols-in-versions.

Closes #14217

show more ...

test1486: verify that write-out.md and tool_writeout.c are in sync

- also verify alphabetialal order in the source
- add two missing variables to write-out.md

Closes #13920

build: untangle `CURLDEBUG` and `DEBUGBUILD` macros

`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be gua

build: untangle `CURLDEBUG` and `DEBUGBUILD` macros

`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be guarded with
`DEBUGBUILD`.

Replace these uses with `DEBUGBUILD`.

This leaves `CURLDEBUG` uses solely for its intended purpose: to enable
the memory tracking debug feature.

Also:
- autotools: rely on `DEBUGBUILD` to enable `checksrc`.
Instead of `CURLDEBUG`, which worked in most cases because debug
builds enable `CURLDEBUG` by default, but it's not accurate.
- include `lib/easyif.h` instead of keeping a copy of a declaration.
- add CI test jobs for the build issues discovered.

Ref: https://github.com/curl/curl/pull/13694#issuecomment-2120311894
Closes #13718

show more ...

dist: add ECH files to tarball

Also sort `EXTRA_DIST` list in `tests/Makefile.am` and make it diffable.

Follow-up to a362962b7289ec02b412890c9515657cf0ed50ac #11922
Closes #13381

dist: add files missing from release tarball

Closes #13346

docs: introduce "curldown" for libcurl man page format

curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of

docs: introduce "curldown" for libcurl man page format

curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)

tools:

- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown

This setup generates .3 versions of all the curldown versions at build time.

CI:

Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...

Closes #12730

show more ...

tests: rename tests scripts to the test number

It is hard to name the scripts sensibly. Lots of them are similarly
named and the name did not tell which test that used them.

The

tests: rename tests scripts to the test number

It is hard to name the scripts sensibly. Lots of them are similarly
named and the name did not tell which test that used them.

The new approach is rather to name them based on the test number that
runs them. Also helps us see which scripts are for individual tests
rather than for general test infra.

- badsymbols.pl -> test1167.pl
- check-deprecated.pl -> test1222.pl
- check-translatable-options.pl -> test1544.pl
- disable-scan.pl -> test1165.pl
- error-codes.pl -> test1175.pl
- errorcodes.pl -> test1477.pl
- extern-scan.pl -> test1135.pl
- manpage-scan.pl -> test1139.pl
- manpage-syntax.pl -> test1173.pl
- markdown-uppercase.pl -> test1275.pl
- mem-include-scan.pl -> test1132.pl
- nroff-scan.pl -> test1140.pl
- option-check.pl -> test1276.pl
- options-scan.pl -> test971.pl
- symbol-scan.pl -> test1119.pl
- version-scan.pl -> test1177.pl

Closes #12487

show more ...

dist: add tests/errorcodes.pl to the tarball

Used by test 1477

Reported-by: Xi Ruoyao
Follow-up to 0ca3a4ec9a7
Fixes #12462
Closes #12463

CI: ignore the "flaky" and "timing-dependent" test results

CI builds will now run these tests, but will ignore the results if they
fail. The relevant tests are ones that are sensitive to

CI: ignore the "flaky" and "timing-dependent" test results

CI builds will now run these tests, but will ignore the results if they
fail. The relevant tests are ones that are sensitive to timing or
have edge conditions that make them more likely to fail on CI servers,
which are often heavily overloaded and slow.

This change only adds two additional tests to be ignored, since the
others already had the flaky keyword.

Closes #11865

show more ...

build: fix portability of mancheck and checksrc targets

At least FreeBSD preserves cwd across makefile lines, so rules
consisting of more than one "cd X; do_something" must be explicitly

build: fix portability of mancheck and checksrc targets

At least FreeBSD preserves cwd across makefile lines, so rules
consisting of more than one "cd X; do_something" must be explicitly run
in a subshell to avoid this. This problem caused the Cirrus FreeBSD
build to fail when parallel make jobs were enabled.

show more ...

tests/Makefile: add check-translatable-options.pl to tarball

Used in test 1544

Follow-up to ae806395abc8c

tests/http: more tests with specific clients

- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_0

tests/http: more tests with specific clients

- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_02 for server PUSH_PROMISEs using clients/h2-serverpush
- added test_02_21 for lib based downloads and pausing/unpausing transfers

curl url parser:
- added internal method `curl_url_set_authority()` for setting the
authority part of a url (used for PUSH_PROMISE)

http2:
- made logging of PUSH_PROMISE handling nicer

Placing python test requirements in requirements.txt files
- separate files to base test suite and http tests since use
and module lists differ
- using the files in the gh workflows

websocket test cases, fixes for we and bufq
- bufq: account for spare chunks in space calculation
- bufq: reset chunks that are skipped empty
- ws: correctly encode frames with 126 bytes payload
- ws: update frame meta information on first call of collect
callback that fills user buffer
- test client ws-data: some test/reporting improvements

Closes #11006

show more ...

devtest: add a new script for testing the test harness

This is currently useful for starting a test server on its own without
an associated test, which can be used for interactive curl t

devtest: add a new script for testing the test harness

This is currently useful for starting a test server on its own without
an associated test, which can be used for interactive curl testing or
for validating parts of the test harness itself. More commands can be
added to perform additional functions in the future.

Ref: #10818
Closes #11008

show more ...

runtests: refactor into more packages

testutil.pm now contains a few miscellaneous functions that are used in
several places but have no better place to live. subvariables moves to

runtests: refactor into more packages

testutil.pm now contains a few miscellaneous functions that are used in
several places but have no better place to live. subvariables moves to
servers.pm since most variables that it substitutes relate to servers,
so this is the most appropriate place. Rename a few functions for better
naming consistency.

Ref: #10818
Closes #10995

show more ...