select: use poll() if existing, avoid poll() with no sockets

poll() on macOS 10.12 was deemed broken in 2016 when we discovered that
it misbehaves when provided with no sockets to wait f

select: use poll() if existing, avoid poll() with no sockets

poll() on macOS 10.12 was deemed broken in 2016 when we discovered that
it misbehaves when provided with no sockets to wait for. The
HAVE_POLL_FINE is used to mark a poll() implementation that behaves
correctly: it *should* still wait the timeout time.

curl has therefore opted to use select() on Apple operating systems ever
since. To avoid the risk that this or other breakage cause problems.

However, using select() internally is also bad because it suffers from
problems when using file descriptors beyond 1024.

This change makes poll() used if it is present, but if there is no
sockets to wait for it avoids using poll() and instead falls back to
select() - but without any sockets to wait for there is no 1024 problem.

This removes all previous special-handling involving HAVE_POLL_FINE.

ref: https://daniel.haxx.se/blog/2016/10/11/poll-on-mac-10-12-is-broken/

Closes #15096

show more ...

tests: speed up builds with single-binary test bundles

Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools:

tests: speed up builds with single-binary test bundles

Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools: `--enable-test-bundles`
- cmake: `-DCURL_TEST_BUNDLES=ON`

(They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON`
options, for further speed-up.)

Makes libtests and unit tests build _fast_, needing little disk space
even in static mode. Similar to CMake unity mode, but with a custom
script, also supporting autotools builds.

The price is having to deal with symbols/macros colliding between
`lib*.c` and `unit*.c` sources. Maybe with naming conventions or other
solutions this can be improved gradually and reduce the need for manual
intervention by `mk-bundle.mk`. I've included a script that does the bulk
of detecting name collisions.

Also:
- CI: enable test bundles.
- CI: build tests in more jobs.
- lib2305: fix FILE handle leak.
- unit1661: fix memleak found by torture test by releasing the `bufref`
structure in `unit_stop()` that was allocated in `unit_setup()`.
```
test 1661...[bufref unit tests]
Leak detected: memory still allocated: 13 bytes
allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70
1661: torture FAILED: function number 1 in test.
```
Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41

Similar test suite builds with autotools default and cmake+bundle+unity:
- GHA/Linux: 33s vs 7s
https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
- GHA/macOS 34s vs 2s
https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
- GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
- GHA/Cygwin: 9m52 vs 32s
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 3m52 vs 14s
https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 5m45 vs 30s
https://github.com/curl/curl/actions/runs/10705668808/job/29681628787

Autotools test suite builds compared between master -> `--enable-test-bundles`:
- GHA/Linux: 33s -> 9s (run tests: 22m23 -> 20m44)
https://github.com/curl/curl/actions/runs/10710030193/job/29695932185
https://github.com/curl/curl/actions/runs/10967831456/job/30458220344
- GHA/macOS: 25s -> 4s (run tests: 2m58 -> 2m24)
https://github.com/curl/curl/actions/runs/10710030195/job/29695938444
https://github.com/curl/curl/actions/runs/10967831452/job/30458225762
- GHA/non-native (FreeBSD): 4m8 -> 3m12 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10710030198/job/29695928401
https://github.com/curl/curl/actions/runs/10967831458/job/30458212692
- GHA/Cygwin: 9m25 -> 1m9 (run tests: 9m19 -> 3m28)
https://github.com/curl/curl/actions/runs/10710030212/job/29695928213
https://github.com/curl/curl/actions/runs/10967831453/job/30458213268
- GHA/MSYS2: 3m54 -> 32s (run tests: 6m3 -> 3m59)
https://github.com/curl/curl/actions/runs/10710030190/job/29704850591
https://github.com/curl/curl/actions/runs/10967831449/job/30459280005
- GHA/mingw-w64: 5m42 -> 1m5 (run tests: 7m41 -> 5m36)
https://github.com/curl/curl/actions/runs/10710030190/job/29704852058
https://github.com/curl/curl/actions/runs/10967831449/job/30459280862
- Azure MSYS2 mingw64 openssl: 38m55 -> 11m58
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
- Azure Ubuntu default: 2m15 -> 55s (all build)
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7

Cmake test suite builds compared between master -> `-DCURL_TEST_BUNDLES=ON` + unity:
- GHA/Linux: 29s -> 7s (run tests: 4m50 -> 4m57, 20m43 -> 20m45)
https://github.com/curl/curl/actions/runs/10710030193/job/29695941814
https://github.com/curl/curl/actions/runs/10705668823/job/29681622201
- GHA/Linux old: 44s -> 13s (bundle+no unity) (run tests: 5m5 -> 5m6)
https://github.com/curl/curl/actions/runs/10718264094/job/29719794727
https://github.com/curl/curl/actions/runs/10718653175/job/29721009613
- GHA/macOS: 32s -> 2s (run tests: 2m43 -> 2m40)
https://github.com/curl/curl/actions/runs/10710030195/job/29695931956
https://github.com/curl/curl/actions/runs/10705668813/job/29681638937
- GHA/non-native (*BSD): inconclusive (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10710030198
https://github.com/curl/curl/actions/runs/10705668811
- GHA/Cygwin: 3m9 -> 32s
https://github.com/curl/curl/actions/runs/10710030212/job/29695929075
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 2m24 -> 14s
https://github.com/curl/curl/actions/runs/10710030190/job/29704850996
https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 3m56 -> 30s (run tests: 4m2 -> 3m52)
https://github.com/curl/curl/actions/runs/10710030190/job/29704852219
https://github.com/curl/curl/actions/runs/10705668808/job/29681631393
- GHA/mingw-w64-old: 7m19 -> 1m44 (run tests: 3m30 -> 2m53)
https://github.com/curl/curl/actions/runs/10710030190/job/29704849763
https://github.com/curl/curl/actions/runs/10705668808/job/29681622329
- GHA/MSVC: 3m22 -> 13s (run tests: 9m43 -> 4m22)
https://github.com/curl/curl/actions/runs/10710030190/job/29704850411
https://github.com/curl/curl/actions/runs/10705668808/job/29681623313
- AppVeyor CI MSVC 2008: 4m3 -> 45s (full build)
- AppVeyor CI MSVC 2010: 2m56 -> 1m8 (full build)
- AppVeyor CI MSVC 2022: 10m19 -> 2m23 (full build)
https://ci.appveyor.com/project/curlorg/curl/builds/50538455
https://ci.appveyor.com/project/curlorg/curl/builds/50536558
- AppVeyor CI total build time: 10m30 (master) -> 6m48 (unity) -> 4m5 (bundle) -> 3m24 (bundle+unity) -> 5m7 (bundle+unity+all jobs building tests)

Closes #14772

show more ...

test537: cap the rlimit max this test runs

Modern debian reports a soft limit of 134217724 which makes this test
run for minutes to allocate all file descriptors. Impose a cap of 256k,

test537: cap the rlimit max this test runs

Modern debian reports a soft limit of 134217724 which makes this test
run for minutes to allocate all file descriptors. Impose a cap of 256k,
so we do not run this tests on systems with larger rlimits.

Closes #14857

show more ...

tests: make the unit test result type `CURLcode`

Before this patch, the result code was a mixture of `int` and
`CURLcode`.

Also adjust casts and fix a couple of minor issues fou

tests: make the unit test result type `CURLcode`

Before this patch, the result code was a mixture of `int` and
`CURLcode`.

Also adjust casts and fix a couple of minor issues found along the way.

Cherry-picked from #13489
Closes #13600

show more ...

misc: Fix typos in docs and lib

This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.

Author: RainRat on Git

misc: Fix typos in docs and lib

This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.

Author: RainRat on Github
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Dan Fandrich <dan@coneharvesters.com>
Closes: #13019

show more ...

build: delete/replace 3 more clang warning pragmas

- tool_msgs: delete redundant `-Wformat-nonliteral` suppression pragma.

- whitespace formatting in `mprintf.h`, lib518, lib537.

build: delete/replace 3 more clang warning pragmas

- tool_msgs: delete redundant `-Wformat-nonliteral` suppression pragma.

- whitespace formatting in `mprintf.h`, lib518, lib537.

- lib518: fix wrong variable in `sizeof()`.

- lib518: bump variables to `rlim_t`.
Follow-up to e2b394106d543c4615a60795b7fdce04bd4e5090 #1469

- lib518: sync error message with lib537
Follow-up to 365322b8bcf9efb6a361473d227b70f2032212ce

- lib518, lib537: replace `-Wformat-nonliteral` suppression pragmas
by reworking test code.

Follow-up to 5b286c250829e06a135a6ba998e80beb7f43a734 #12812
Follow-up to aee4ebe59161d0a5281743f96e7738ad97fe1cd4 #12803
Follow-up to 09230127589eccc7e01c1a7217787ef8e64f3328 #12540
Follow-up to 3829759bd042c03225ae862062560f568ba1a231 #12489

Reviewed-by: Daniel Stenberg
Closes #12814

show more ...

build: enable missing OpenSSF-recommended warnings, with fixes

https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1]

build: enable missing OpenSSF-recommended warnings, with fixes

https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].

Enable new recommended warnings (except `-Wsign-conversion`):

- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
printf arguments with it. This is a copy of existing
`CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
with redefinting the `printf` symbol:
https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
Follow-up to d5c0351055d5709da8f3e16c91348092fdb481aa #2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
checks. Previously it was always disabled due to the internal `printf`
macro.

Fix them:

- fix bug where an `set_ipv6_v6only()` call was missed in builds with
`--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.

[1] https://github.com/ossf/wg-best-practices-os-developers/blob/56c0fde3895bfc55c8a973ef49a2572c507b2ae1/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.md

Closes #12489

show more ...

windows: use built-in `_WIN32` macro to detect Windows

Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. T

windows: use built-in `_WIN32` macro to detect Windows

Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.

Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.

There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.

grepping for `WIN32` remains useful to discover Windows-specific code.

Also:

- extend `checksrc` to ensure we're not using `WIN32` anymore.

- apply minor formatting here and there.

- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.

Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg

Closes #12376

show more ...

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- save

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205

show more ...

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869

show more ...

TPF: drop support

There has been no TPF related changes done since September 2010 (commit
7e1a45e224e57) and since this is a platform that is relatively different
than many others (=

TPF: drop support

There has been no TPF related changes done since September 2010 (commit
7e1a45e224e57) and since this is a platform that is relatively different
than many others (== needs attention), I draw the conclusion that this
build is broken since a long time.

Closes #8378

show more ...

checksrc: detect more kinds of NULL comparisons we avoid

Co-authored-by: Jay Satiro
Closes #8180

tidy-up: make conditional checks more consistent

... remove '== NULL' and '!= 0'

Closes #6912

copyright: fix year ranges

Follow-up from 4d2f8006777

curl.se: new home

Closes #6172

snprintf: renamed and we now only use msnprintf()

The function does not return the same value as snprintf() normally does,
so readers may be mislead into thinking the code works differen

snprintf: renamed and we now only use msnprintf()

The function does not return the same value as snprintf() normally does,
so readers may be mislead into thinking the code works differently than
it actually does. A different function name makes this easier to detect.

Reported-by: Tomas Hoger
Assisted-by: Daniel Gustafsson
Fixes #3296
Closes #3297

show more ...

build: remove HAVE_LIMITS_H check

.. because limits.h presence isn't optional, it's required by C89.

Ref: http://port70.net/~nsz/c/c89/c89-draft.html#2.2.4.2

Closes https:/

build: remove HAVE_LIMITS_H check

.. because limits.h presence isn't optional, it's required by C89.

Ref: http://port70.net/~nsz/c/c89/c89-draft.html#2.2.4.2

Closes https://github.com/curl/curl/pull/2215

show more ...

tests: Fix up issues with errno in test files

Closes https://github.com/curl/curl/pull/1671

test537: use correct variable type

Avoids narrowing conversion warnings because rlim_t is usually
unsigned long.

Closes https://github.com/curl/curl/pull/1469

checksrc: warn for assignments within if() expressions

... they're already frowned upon in our source code style guide, this
now enforces the rule harder.

code: style updates

tests/libtest: follow our code style guidelines better

... checksrc of all test code is pending.

curl/mprintf.h: remove support for _MPRINTF_REPLACE

The define is not in our name space and is therefore not protected by
our API promises.

It was only really used by libcurl in

curl/mprintf.h: remove support for _MPRINTF_REPLACE

The define is not in our name space and is therefore not protected by
our API promises.

It was only really used by libcurl internals but was mostly erased from
there already in 8aabbf5 (March 2015). This is supposedly the final
death blow to that define from everywhere.

As a side-effect, making sure _MPRINTF_REPLACE is gone and not used, I
made the lib tests in tests/libtest/ use curl_printf.h for its redefine
magic and then subsequently the use of sprintf() got banned in the tests
as well (as it is in libcurl internals) and I then replaced them all
with snprintf().

In the unlikely event that any users is actually using this define and
gets sad by this change, it is very easily copied to the user's own
code.

show more ...

URLs: change all http:// URLs to https://

always-multi: always use non-blocking internals

Remove internal separated behavior of the easy vs multi intercace.
curl_easy_perform() is now using the multi interface itself.

S

always-multi: always use non-blocking internals

Remove internal separated behavior of the easy vs multi intercace.
curl_easy_perform() is now using the multi interface itself.

Several minor multi interface quirks and bugs have been fixed in the
process.

Much help with debugging this has been provided by: Yang Tse

show more ...