ngtcp2: build with 0.17.0 and nghttp3 0.13.0

- ngtcp2_crypto_openssl was renamed to ngtcp2_crypto_quictls.

Closes #11428

configure: the --without forms of the options are also gone

--without-darwin-ssl and --without-metalink

Closes #11378

configure: add check for ldap_init_fd

... as otherwise the configure script will say it is OpenLDAP in the
summary, but not set the USE_OPENLDAP define, therefor not using the
intend

configure: add check for ldap_init_fd

... as otherwise the configure script will say it is OpenLDAP in the
summary, but not set the USE_OPENLDAP define, therefor not using the
intended OpenLDAP code paths.

Regression since 4d7385446 (7.85.0)
Fixes #11372
Closes #11374
Reported-by: vlkl-sap on github

show more ...

timeval: use CLOCK_MONOTONIC_RAW if available

Reported-by: Harry Sintonen
Ref: #11288
Closes #11291

build: drop unused/redundant `HAVE_WINLDAP_H`

Sources did not use it. Autotools used it when checking for the
`winldap` library, which is redundant.

With CMake, detection was br

build: drop unused/redundant `HAVE_WINLDAP_H`

Sources did not use it. Autotools used it when checking for the
`winldap` library, which is redundant.

With CMake, detection was broken:
```
Run Build Command(s):/usr/local/Cellar/cmake/3.26.3/bin/cmake -E env VERBOSE=1 /usr/bin/make -f Makefile cmTC_2d8fe/fast && /Library/Developer/CommandLineTools/usr/bin/make -f CMakeFiles/cmTC_2d8fe.dir/build.make CMakeFiles/cmTC_2d8fe.dir/build
Building C object CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj
/usr/local/opt/llvm/bin/clang --target=x86_64-w64-mingw32 --sysroot=/usr/local/opt/mingw-w64/toolchain-x86_64 -D_WINSOCKAPI_="" -I/my/quictls/x64-ucrt/usr/include -I/my/zlib/x64-ucrt/usr/include -I/my/brotli/x64-ucrt/usr/include -Wno-unused-command-line-argument -D_UCRT -DCURL_HIDDEN_SYMBOLS -DHAVE_SSL_SET0_WBIO -DHAS_ALPN -DNGHTTP2_STATICLIB -DNGHTTP3_STATICLIB -DNGTCP2_STATICLIB -DUSE_MANUAL=1 -fuse-ld=lld -Wl,-s -static-libgcc -lucrt -Wextra -Wall -pedantic -Wbad-function-cast -Wconversion -Winline -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wno-long-long -Wno-multichar -Wpointer-arith -Wshadow -Wsign-compare -Wundef -Wunused -Wwrite-strings -Wcast-align -Wdeclaration-after-statement -Wempty-body -Wendif-labels -Wfloat-equal -Wignored-qualifiers -Wno-format-nonliteral -Wno-sign-conversion -Wno-system-headers -Wstrict-prototypes -Wtype-limits -Wvla -Wshift-sign-overflow -Wshorten-64-to-32 -Wdouble-promotion -Wenum-conversion -Wunused-const-variable -Wcomma -Wmissing-variable-declarations -Wassign-enum -Wextra-semi-stmt -MD -MT CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj -MF CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj.d -o CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj -c /my/curl/bld-cmake-llvm-x64-shared/CMakeFiles/CMakeScratch/TryCompile-3JP6dR/HAVE_WINLDAP_H.c
In file included from /my/curl/bld-cmake-llvm-x64-shared/CMakeFiles/CMakeScratch/TryCompile-3JP6dR/HAVE_WINLDAP_H.c:2:
In file included from /usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/winldap.h:17:
In file included from /usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/schnlsp.h:9:
In file included from /usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/schannel.h:10:
/usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/wincrypt.h:5041:254: error: unknown type name 'PSYSTEMTIME'
WINIMPM PCCERT_CONTEXT WINAPI CertCreateSelfSignCertificate (HCRYPTPROV_OR_NCRYPT_KEY_HANDLE hCryptProvOrNCryptKey, PCERT_NAME_BLOB pSubjectIssuerBlob, DWORD dwFlags, PCRYPT_KEY_PROV_INFO pKeyProvInfo, PCRYPT_ALGORITHM_IDENTIFIER pSignatureAlgorithm, PSYSTEMTIME pStartTime, PSYSTEMTIME pEndTime, PCERT_EXTENSIONS pExtensions);
^
/usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/wincrypt.h:5041:278: error: unknown type name 'PSYSTEMTIME'
WINIMPM PCCERT_CONTEXT WINAPI CertCreateSelfSignCertificate (HCRYPTPROV_OR_NCRYPT_KEY_HANDLE hCryptProvOrNCryptKey, PCERT_NAME_BLOB pSubjectIssuerBlob, DWORD dwFlags, PCRYPT_KEY_PROV_INFO pKeyProvInfo, PCRYPT_ALGORITHM_IDENTIFIER pSignatureAlgorithm, PSYSTEMTIME pStartTime, PSYSTEMTIME pEndTime, PCERT_EXTENSIONS pExtensions);
^
2 errors generated.
make[1]: *** [CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj] Error 1
make: *** [cmTC_2d8fe/fast] Error 2
exitCode: 2
```

Cherry-picked from #11095 88e4a21ff70ccef391cf99c8165281ff81374503
Reviewed-by: Daniel Stenberg
Closes #11245

show more ...

configure: fix run-compiler for old /bin/sh

If you try to assign and export on the same line on some older /bin/sh
implementations, it complains:

```
$ export "NAME=value"

configure: fix run-compiler for old /bin/sh

If you try to assign and export on the same line on some older /bin/sh
implementations, it complains:

```
$ export "NAME=value"
NAME=value: is not an identifier
```

This commit rewrites run-compiler's assignments and exports to work with
old /bin/sh, splitting assignment and export into two separate
statements, and only quote the value. So now we have:

```
NAME="value"
export NAME
```

While we're here, make the same change to the two supporting
assign+export lines preceeding the script to be consistent with how
exports work throughout the rest of configure.ac.

Closes #11228

show more ...

configure: without pkg-config and no custom path, use -lnghttp2

Reported-by: correctmost on github
Fixes #11186
Closes #11210

configure: fix build with arbitrary CC and LD_LIBRARY_PATH

Since ./configure and processes that inherit its environment variables
are the only callers of the run-compiler script, we can

configure: fix build with arbitrary CC and LD_LIBRARY_PATH

Since ./configure and processes that inherit its environment variables
are the only callers of the run-compiler script, we can just save the
current value of the LD_LIBRARY_PATH and CC variables to another pair of
environment variables, and make run-compiler a static script that
simply restores CC and LD_LIBRARY_PATH to the saved value, and before
running the compiler.

This avoids having to inject the values of the variables in the script,
possibly causing problems if they contains spaces, quotes, and other
special characters.

Also add exports in the script just in case LD_LIBRARY_PATH and CC are
not already in the environment.

follow-up from 471dab2

Closes #11182

show more ...

configure: quote the assignments for run-compiler

Building for multilib failed, as the compiler command contains an
extra argument. That needs quoting.

Regression from b78ca50cb

configure: quote the assignments for run-compiler

Building for multilib failed, as the compiler command contains an
extra argument. That needs quoting.

Regression from b78ca50cb3dda361f9c1

Fixes #11179
Closes #11180

show more ...

configure: fix --help alignment

AC_ARG_ENABLE seems to only trim off whitespace from the start and end
of its help-string argument, while prepending two spaces of indentation
to all

configure: fix --help alignment

AC_ARG_ENABLE seems to only trim off whitespace from the start and end
of its help-string argument, while prepending two spaces of indentation
to all lines.

This means that the two spaces of indentation between the --enable-rtsp
and the --disable-rtsp line were not removed causing ./configure --help
to print:

Optional Features:
[...]
--enable-rtsp Enable RTSP support
--disable-rtsp Disable RTSP support

I removed the indentation to fix the issue, now it prints:

Optional Features:
[...]
--enable-rtsp Enable RTSP support
--disable-rtsp Disable RTSP support

The --enable-hsts and --disable-hsts lines had the same problems, and
have been fixed too.

Closes #11142

show more ...

configure: generate a script to run the compiler

in the CURL_RUN_IFELSE macro, with LD_LIBRARY_PATH set to the value of
the configure invoke, and not the value that might be used later,

configure: generate a script to run the compiler

in the CURL_RUN_IFELSE macro, with LD_LIBRARY_PATH set to the value of
the configure invoke, and not the value that might be used later,
intended for the execution of the output the compiler ouputs.

For example when the compiler uses the same library (like libz) that
configure checks for.

Reported-by: Jonas Bülow
Fixes #11114
Closes #11120

show more ...

configure: fix detection of apxs (for httpd)

The condition check was turned the wrong way around!

Closes #11051

tests/http: more tests with specific clients

- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_0

tests/http: more tests with specific clients

- Makefile support for building test specific clients in tests/http/clients
- auto-make of clients when invoking pytest
- added test_09_02 for server PUSH_PROMISEs using clients/h2-serverpush
- added test_02_21 for lib based downloads and pausing/unpausing transfers

curl url parser:
- added internal method `curl_url_set_authority()` for setting the
authority part of a url (used for PUSH_PROMISE)

http2:
- made logging of PUSH_PROMISE handling nicer

Placing python test requirements in requirements.txt files
- separate files to base test suite and http tests since use
and module lists differ
- using the files in the gh workflows

websocket test cases, fixes for we and bufq
- bufq: account for spare chunks in space calculation
- bufq: reset chunks that are skipped empty
- ws: correctly encode frames with 126 bytes payload
- ws: update frame meta information on first call of collect
callback that fills user buffer
- test client ws-data: some test/reporting improvements

Closes #11006

show more ...

configure: don't set HAVE_WRITABLE_ARGV on Windows

Ref: #10888
Closes #10896

tests/http: improved httpd detection

- better error messages when not found/complete
- handling of `--without-test-httpd`

Reported-by: kwind on github
Fixes #10879
Close

tests/http: improved httpd detection

- better error messages when not found/complete
- handling of `--without-test-httpd`

Reported-by: kwind on github
Fixes #10879
Closes #10883

show more ...

configure: make quiche require quiche_conn_send_ack_eliciting

curl now requires quiche version >= 1.17.1 to be used and this function
was added in this version and makes a convenient che

configure: make quiche require quiche_conn_send_ack_eliciting

curl now requires quiche version >= 1.17.1 to be used and this function
was added in this version and makes a convenient check.

This requirement is because this is the lowest quiche version that
supports peer-initiated key updates correctly.

Closes #10886

show more ...

lib: use correct printf flags for sockets and timediffs

Introduces CURL_FORMAT_SOCKET_T for outputting socket numbers.

Fixes #10737
Reported-by: Gisle Vanem
Closes #10855

ngtcp2: adjust config and code checks for ngtcp2 without nghttp3

- make configure show on HTTP3 feature that both ngtcp2 and nghttp3
are in play
- define ENABLE_QUIC only when USE_

ngtcp2: adjust config and code checks for ngtcp2 without nghttp3

- make configure show on HTTP3 feature that both ngtcp2 and nghttp3
are in play
- define ENABLE_QUIC only when USE_NGTCP2 and USE_NGHTTP3 are defined
- add USE_NGHTTP3 in the ngtcp2 implementation

Fixes #10793
Closes #10821

show more ...

tests/http: add pytest to GHA and improve tests

- added to: ngtcp2-quictls, ngtcp2-gnutls and the linux varians
quiche, bearssl, libressl, mbedtls, openssl3, rustls
- added disable

tests/http: add pytest to GHA and improve tests

- added to: ngtcp2-quictls, ngtcp2-gnutls and the linux varians
quiche, bearssl, libressl, mbedtls, openssl3, rustls
- added disabled in ngtcp2-wolfssl due to weird SSL_connect() errors
not reproducable locally

Improvements on pytest:

- handling of systems with nghttpx in $PATH
- configure will seach $PATH got nghttpx used in pytest
- pytest fixes for managing nghttpx without h3 support
- ngtcp2-wolfssl: use a fully enabled wolfssl build

- lower parallel count for http/1.1 tests, since we do not
want to test excessive connections.
- check built curl for HTTPS-proxy support in proxy tests
- bearssl does not like one of our critical cert extensions, making
it non-critical now
- bearssl is too slow for test_12, skipping
- making sure we do h3 tests only when curl and server support is there

Closes #10699

show more ...

DEPRECATE: the original legacy mingw version 1

Remove completely in September 2023

Closes #10667

rand: use arc4random as fallback when available

Normally curl uses cryptographically strong random provided by the
selected SSL backend. If compiled without SSL support, a naive built-in

rand: use arc4random as fallback when available

Normally curl uses cryptographically strong random provided by the
selected SSL backend. If compiled without SSL support, a naive built-in
function was used instead.

Generally this was okay, but it will result in some downsides for non-
SSL builds, such as predictable temporary file names.

This change ensures that arc4random will be used instead, if available.

Closes #10672

show more ...

build: drop the use of XC_AMEND_DISTCLEAN

Because automake used to delete depdirs at once (.deps) and there was an issue
with portability, curl's XC_AMEND_DISTCLEAN greps the Makefiles i

build: drop the use of XC_AMEND_DISTCLEAN

Because automake used to delete depdirs at once (.deps) and there was an issue
with portability, curl's XC_AMEND_DISTCLEAN greps the Makefiles in an attempt
to build a list of all depfiles and delete them individually instead.

Since commit 08849db866b44510f6b8fd49e313c91a43a3dfd3, automake switched from
deleting directories to individual files. curl's custom logic now finds a lot
more results with the grep (the filtering of these results isn't great), which
causes a massive bloating of the Makefile in the order of O(n^2).

Also remove now-unused XC_AMEND_DISTCLEAN macro group

References: https://github.com/curl/curl/issues/9843
References: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59288

Reported-by: Ilmari Lauhakangas
Fixes #9843
Closes #10661

show more ...

tests: rename tests/tests-httpd to tests/http

- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more b

tests: rename tests/tests-httpd to tests/http

- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more beefy environment

Closes #10654

show more ...

misc: remove support for curl_off_t < 8 bytes

Closes #10597

connections: introduce http/3 happy eyeballs

New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURL

connections: introduce http/3 happy eyeballs

New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.

Closes #10349

show more ...