tests: speed up builds with single-binary test bundles

Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools:

tests: speed up builds with single-binary test bundles

Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools: `--enable-test-bundles`
- cmake: `-DCURL_TEST_BUNDLES=ON`

(They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON`
options, for further speed-up.)

Makes libtests and unit tests build _fast_, needing little disk space
even in static mode. Similar to CMake unity mode, but with a custom
script, also supporting autotools builds.

The price is having to deal with symbols/macros colliding between
`lib*.c` and `unit*.c` sources. Maybe with naming conventions or other
solutions this can be improved gradually and reduce the need for manual
intervention by `mk-bundle.mk`. I've included a script that does the bulk
of detecting name collisions.

Also:
- CI: enable test bundles.
- CI: build tests in more jobs.
- lib2305: fix FILE handle leak.
- unit1661: fix memleak found by torture test by releasing the `bufref`
structure in `unit_stop()` that was allocated in `unit_setup()`.
```
test 1661...[bufref unit tests]
Leak detected: memory still allocated: 13 bytes
allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70
1661: torture FAILED: function number 1 in test.
```
Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41

Similar test suite builds with autotools default and cmake+bundle+unity:
- GHA/Linux: 33s vs 7s
https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
- GHA/macOS 34s vs 2s
https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
- GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
- GHA/Cygwin: 9m52 vs 32s
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 3m52 vs 14s
https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 5m45 vs 30s
https://github.com/curl/curl/actions/runs/10705668808/job/29681628787

Autotools test suite builds compared between master -> `--enable-test-bundles`:
- GHA/Linux: 33s -> 9s (run tests: 22m23 -> 20m44)
https://github.com/curl/curl/actions/runs/10710030193/job/29695932185
https://github.com/curl/curl/actions/runs/10967831456/job/30458220344
- GHA/macOS: 25s -> 4s (run tests: 2m58 -> 2m24)
https://github.com/curl/curl/actions/runs/10710030195/job/29695938444
https://github.com/curl/curl/actions/runs/10967831452/job/30458225762
- GHA/non-native (FreeBSD): 4m8 -> 3m12 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10710030198/job/29695928401
https://github.com/curl/curl/actions/runs/10967831458/job/30458212692
- GHA/Cygwin: 9m25 -> 1m9 (run tests: 9m19 -> 3m28)
https://github.com/curl/curl/actions/runs/10710030212/job/29695928213
https://github.com/curl/curl/actions/runs/10967831453/job/30458213268
- GHA/MSYS2: 3m54 -> 32s (run tests: 6m3 -> 3m59)
https://github.com/curl/curl/actions/runs/10710030190/job/29704850591
https://github.com/curl/curl/actions/runs/10967831449/job/30459280005
- GHA/mingw-w64: 5m42 -> 1m5 (run tests: 7m41 -> 5m36)
https://github.com/curl/curl/actions/runs/10710030190/job/29704852058
https://github.com/curl/curl/actions/runs/10967831449/job/30459280862
- Azure MSYS2 mingw64 openssl: 38m55 -> 11m58
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
- Azure Ubuntu default: 2m15 -> 55s (all build)
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7

Cmake test suite builds compared between master -> `-DCURL_TEST_BUNDLES=ON` + unity:
- GHA/Linux: 29s -> 7s (run tests: 4m50 -> 4m57, 20m43 -> 20m45)
https://github.com/curl/curl/actions/runs/10710030193/job/29695941814
https://github.com/curl/curl/actions/runs/10705668823/job/29681622201
- GHA/Linux old: 44s -> 13s (bundle+no unity) (run tests: 5m5 -> 5m6)
https://github.com/curl/curl/actions/runs/10718264094/job/29719794727
https://github.com/curl/curl/actions/runs/10718653175/job/29721009613
- GHA/macOS: 32s -> 2s (run tests: 2m43 -> 2m40)
https://github.com/curl/curl/actions/runs/10710030195/job/29695931956
https://github.com/curl/curl/actions/runs/10705668813/job/29681638937
- GHA/non-native (*BSD): inconclusive (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10710030198
https://github.com/curl/curl/actions/runs/10705668811
- GHA/Cygwin: 3m9 -> 32s
https://github.com/curl/curl/actions/runs/10710030212/job/29695929075
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 2m24 -> 14s
https://github.com/curl/curl/actions/runs/10710030190/job/29704850996
https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 3m56 -> 30s (run tests: 4m2 -> 3m52)
https://github.com/curl/curl/actions/runs/10710030190/job/29704852219
https://github.com/curl/curl/actions/runs/10705668808/job/29681631393
- GHA/mingw-w64-old: 7m19 -> 1m44 (run tests: 3m30 -> 2m53)
https://github.com/curl/curl/actions/runs/10710030190/job/29704849763
https://github.com/curl/curl/actions/runs/10705668808/job/29681622329
- GHA/MSVC: 3m22 -> 13s (run tests: 9m43 -> 4m22)
https://github.com/curl/curl/actions/runs/10710030190/job/29704850411
https://github.com/curl/curl/actions/runs/10705668808/job/29681623313
- AppVeyor CI MSVC 2008: 4m3 -> 45s (full build)
- AppVeyor CI MSVC 2010: 2m56 -> 1m8 (full build)
- AppVeyor CI MSVC 2022: 10m19 -> 2m23 (full build)
https://ci.appveyor.com/project/curlorg/curl/builds/50538455
https://ci.appveyor.com/project/curlorg/curl/builds/50536558
- AppVeyor CI total build time: 10m30 (master) -> 6m48 (unity) -> 4m5 (bundle) -> 3m24 (bundle+unity) -> 5m7 (bundle+unity+all jobs building tests)

Closes #14772

show more ...

autotools: add support for 'unity' builds, enable in CI

Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `t

autotools: add support for 'unity' builds, enable in CI

Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).

Enable with: `--enable-unity` (disabled by default)

Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.

This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).

The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.

Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)

Autotools test suite builds compared between master -> `--enable-unity`:
- GHA/Linux: 32s -> 12s
https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
https://github.com/curl/curl/actions/runs/10742978889/job/29796766297
- GHA/macOS: 37s -> 10s
https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
https://github.com/curl/curl/actions/runs/10742978699/job/29796768875
- GHA/FreeBSD: 15m25 -> 10m58 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
https://github.com/curl/curl/actions/runs/10742978937/job/29796766115
- GHA/Cygwin: 3m32 -> 1m21
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
https://github.com/curl/curl/actions/runs/10742978645/job/29796756933
- GHA/MSYS2: 2m42 -> 50s
https://github.com/curl/curl/actions/runs/10705668808/job/29681621166
https://github.com/curl/curl/actions/runs/10742978662/job/29799739289
- GHA/mingw-w64: 5m32 -> 1m23
https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
https://github.com/curl/curl/actions/runs/10742978662/job/29799741568

Closes #14815

show more ...

GHA: keep default pkgconf, do not replace with pkg-config on Linux

Ubuntu has the `pkgconf` package installed by default that implements
the `pkg-config` command. Switch CI `apt` command

GHA: keep default pkgconf, do not replace with pkg-config on Linux

Ubuntu has the `pkgconf` package installed by default that implements
the `pkg-config` command. Switch CI `apt` commands over to `pkgconf`
to avoid replacing it in every job run.

Avoids:
```
The following packages will be REMOVED:
pkgconf r-base-dev
The following NEW packages will be installed:
[...] pkg-config [...]
```
https://github.com/curl/curl/actions/runs/10949915766/job/30404126342?pr=14972#step:2:20

Closes #14974

show more ...

CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks

CI:

- GHA/windows: enable OpenSSH server, SysInternals `handle`, `impacket`.
Skip `impacket` on MSYS2 due

CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks

CI:

- GHA/windows: enable OpenSSH server, SysInternals `handle`, `impacket`.
Skip `impacket` on MSYS2 due to install failure.
Skip OpenSSH server for old/standalone mingw-w64 (building curl
without SSH there.)
- GHA/windows: make test tool installs a separate step.
- GHA/cygwin: enable OpenSSH server.
Skip `impacket`: it's compiling for 7 minutes then breaks.
Skip `stunnel` due to sluggish test run performance.
(This update is unrelated to Azure jobs.)
- GHA/linux: migrate Linux jobs from Azure CI.
- GHA/linux: migrate scanbuild job from Azure CI.
- GHA/linux: enable libssh2 in a job. Also enable valgrind.
- CI/windows: enable SSPI in two jobs.
- CI/windows: disable zlib in one more job.
- CI/windows: improve `if` condition checking GnuTLS.
- CI/windows: ignore SFTP/SCP tests as necessary.
- universally ignore SCP tests, they fail everywhere.
- ignore test 612.
- ignore test 613 616 618 with MSYS2 mingw-w64.
- ignore test 614 with libssh.
- ignore all SFTP with MSYS2 native.
- ignore all SFTP with vcpkg with `libssh2[core,zlib]`.
- ignore a couple of SFTP tests with MSYS2 mingw-w64.
(This matches settings on Azure CI.)
- GHA/windows: ignore failing 1451 'Basic SMB request' test for
old mingw-w64 7.3.0 (but not for 9.5.0!):
```
2024-09-11 21:45:59,738 ERROR smbComNegotiate: b'NT LM 0.12\x00' is not in list
[...]
curl: (7) Could not connect to server
[...]
FAIL 1451: 'Basic SMB request' SMB
```
Ref: https://github.com/curl/curl/actions/runs/10816280746/job/30007130770#step:13:3546
- CI/Azure: delete, now moved to GHA.

sshserver:

- sshserver: fix permissions for SSH host key on Windows, allowing sshd
to launch.
- sshserver: fix initializing config paths with Cygwin/MSYS2 sshd.
- sshserver: fix initializing config paths in parallel builds.
- sshserver: delete redundant `DenyUsers`. This also opens the way
to allow multiple usernames.
- sshserver: fix `AllowUsers` for GHA/windows, by allowing the
domainless username again.
Follow-up to 3ee7c676ec8ef32e3a9cb4d0fb2929a8335d58db #5721
- sshserver: fix `Deprecated option` warnings in `sshd_config`:
```
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 7: Deprecated option AuthorizedKeysFile2
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 25: Deprecated option KeyRegenerationInterval
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 35: Deprecated option RhostsRSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 36: Deprecated option RSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 37: Deprecated option ServerKeyBits
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 41: Deprecated option UseLogin
```
- sshserver: fix `Deprecated option` warnings in `ssh_config` with
Cygwin/MSYS2.
- sshserver: fix dumping config files due to the filenames missing their
full paths.
- sshserver: add workaround to make `logmsg` messages visible.
Before this patch they only went to a file and never shown.

runtests:

- runtests: log details when these Windows commands are called:
`handle`, `taskkill`, `tasklist`.
- runtests: add documentation links to Windows tools:
`handle`, `taskkill`, `tasklist`, `icacls`.
- runtests: add `-t` (kill whole tree) option to `taskkill` in
`servers.pm`, syncing it with the other `taskkill` call.
Follow-up to bc72a78a11764558639131d51fa8e7c81ee1e113 #14488
- runtests: show warning if Sysinternals `handle` tool is missing.
- runtests: drop Windows XP Home compatibility `tskill` call.
The call was made on all Windows versions. It's possibly overkill to
do this, because XP Home is probably rarely used for running curl
tests these days. In case it's needed, it'd be better to put it under
an explicit option.
- runtests: show Perl version and path.

Fix/silence fallouts:

- unit2603: fix building with disabled HTTP support.
- unit2604: silence `-Woverlength-strings` warnings in C89 mode.
- test437, test1614: fix to pass with no-IPv6 builds.

Closes #14859

show more ...

GHA: update CI dependencies

- update github/codeql-action digest to 2c779ab
- update awslabs/aws-lc to v1.34.2
- update ngtcp2/nghttp3 to v1.5.0
- update ngtcp2/ngtcp2 to v1.7.0

GHA: update CI dependencies

- update github/codeql-action digest to 2c779ab
- update awslabs/aws-lc to v1.34.2
- update ngtcp2/nghttp3 to v1.5.0
- update ngtcp2/ngtcp2 to v1.7.0

Closes #14638
Closes #14639
Closes #14675
Closes #14676

show more ...

GHA: add yamlcheck

Uses `yamllint`.

Also:
- fix warning for existing YAML files:
```
[truthy] truthy value should be one of [false, true]
```

Closes #

GHA: add yamlcheck

Uses `yamllint`.

Also:
- fix warning for existing YAML files:
```
[truthy] truthy value should be one of [false, true]
```

Closes #14650

show more ...

tests: constrain http pytest to tests/http directory

Running the http pytest had to be done from tests directory or above,
because the repeat argument fixture was defined in tests/confte

tests: constrain http pytest to tests/http directory

Running the http pytest had to be done from tests directory or above,
because the repeat argument fixture was defined in tests/conftest.py.
However, the repeat argument is not needed because its functionality
can be provided by pytest-repeat as documented in the test's
README.md. So, removed the pytest_addoption function for the repeat
argument and the pytest_report_header function is moved to
tests/http/conftest.py.

TODO: Remove repeat argument from all tests. As a stopgap, a
one-element list is defined for it for now.

Closes #14611

show more ...

GHA: update dependency gnutls/gnutls to v3.8.7

Closes #14554

GHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI

- run tests via `make test-ci` instead of `make check` with autotools.
- add `x86_64` job for FreeBSD, with test

GHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI

- run tests via `make test-ci` instead of `make check` with autotools.
- add `x86_64` job for FreeBSD, with tests.
It matches the existing Cirrus CI job, with these differences:
- finishes 3x faster (thanks to parallel tests enabled).
- librtmp is not enabled because it's slated for removal by FreeBSD.
(already past the removal deadline, thought the package still
installs.)
- DICT and TELNET servers fail to start. Couldn't figure out why.
It means skipping test 1450 and 1452.
- it runs more tests, e.g. websockets and ip6-localhost.
- no `pkg update -f`.
- it misses the `CRYPTOGRAPHY_DONT_BUILD_RUST=1`, `pkg delete curl`,
`chmod 777`, `sudo -u nobody` and `sysctl net.inet.tcp.blackhole`
tricks. The latter is the default in these runners, the others did
not affect results.
- set `-j0` for tests in the NetBSD job. Flaky otherwise.

Closes #14244

show more ...

GHA: update wolfSSL and mod_h2

- wolfSSL/wolfssl to v5.7.2
- icing/mod_h2 to v2.0.29

Closes #14131
Closes #14148

GHA: configure OpenSSL's libdir as 'lib' only

Also mention in HTTP3.md

OpenSSL has a bug that messes the config `--libdir=path` to become the
wrong path in its pkgconfig files.

GHA: configure OpenSSL's libdir as 'lib' only

Also mention in HTTP3.md

OpenSSL has a bug that messes the config `--libdir=path` to become the
wrong path in its pkgconfig files. If we just pass `--libdir=lib` it
should avoid this.

Ref: #14099
See also: https://github.com/openssl/openssl/issues/23569

Closes #14102

show more ...

GHA: update dependency gnutls/gnutls to v3.8.6

Closes #14094

quic: update to quiche 0.22.0

quiche 0.22.0 will set SONAME in libquiche.so (libquiche.so.0) for
linux/BSDs. Install a symlink with SONAME.

Closes #14030
Closes #14046

GHA: add --enable-werror to the quiche job

Closes #14041

GHA: update pinned actions

- github/codeql-action digest to 23acc5c
- actions/checkout digest to 692973e
- rojopolis/spellcheck-github-actions digest to d354a4d

Closes #1393

GHA: update pinned actions

- github/codeql-action digest to 23acc5c
- actions/checkout digest to 692973e
- rojopolis/spellcheck-github-actions digest to d354a4d

Closes #13935
Closes #13945
Closes #13946

show more ...

ci: update dependency ngtcp2/ngtcp2 to v1.6.0

Closes #13939

ci: update ngtcp2/nghttp3 to v1.4.0

Closes #13938

GHA: bump all build jobs to nproc+1

- bump rest of the workflows (windows, macos, distrocheck).

- non-native virtualized envs have 2 CPUs, bump down accordingly.
(for `vmactio

GHA: bump all build jobs to nproc+1

- bump rest of the workflows (windows, macos, distrocheck).

- non-native virtualized envs have 2 CPUs, bump down accordingly.
(for `vmactions/omnios-vm` it's just a guess.)

- bump all to nproc + 1.

Follow-up to e838b341a08b44d4a8486fb0d3f15d12fc794c62 #12927
Closes #13807

show more ...

ci: pin actions/github-script action to 60a0d83

Closes #13846

GHA: unify http3 workflows into one

This commit unifies the following http3 workflows into http3-linux.yml:

- ngtcp2-linux.yml
- osslq-linux.yml
- quiche-linux.yml

GHA: unify http3 workflows into one

This commit unifies the following http3 workflows into http3-linux.yml:

- ngtcp2-linux.yml
- osslq-linux.yml
- quiche-linux.yml

The idea is better use of the build cache. Previously, they
independently create caches with the same key. Some of the caches
include source code and intermediate object files, which makes cache
quite large. In this commit, only built artifacts are cached, which
drastically reduces the cache size. OpenSSL v3, mod_h2 and quiche caches
still include all stuff, but they are left for the later improvement.
Because the contents of the cache have been changed, the cache keys are
also changed to include the word "http3".

Closes #13841

show more ...