TLS: TLSv1.3 earlydata support for curl

Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.

If a known TLS

TLS: TLSv1.3 earlydata support for curl

Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.

If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.

- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
bytes sent and accepted/rejected by the server.

Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
that ALPN protocol for negoptiation with the server. When the
sessions ALPN does not match the connections ALPN, earlydata
will not be enabled.
- Check that the server selected the correct ALPN protocol for
an earlydata connect. If the server does not confirm or reports
something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
if not connect-only.

Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature

Assisted-by: ad-chaos on github
Closes #15211

show more ...

lib: use bool/TRUE/FALSE properly

booleans should use the type 'bool' and set the value to TRUE/FALSE

non-booleans should not be 'bool' and should not set the value to
TRUE/FALS

lib: use bool/TRUE/FALSE properly

booleans should use the type 'bool' and set the value to TRUE/FALSE

non-booleans should not be 'bool' and should not set the value to
TRUE/FALSE

Closes #15123

show more ...

lib/src: white space edits to comply better with code style

... as checksrc now finds and complains about these.

Closes #14921

lib: use FMT_ as prefix instead of CURL_FORMAT_

For printf format defines used internally. Makes the code slighly
easier to read.

Closes #14764

getinfo: add CURLINFO_POSTTRANSFER_TIME_T

Returns the time, in microseconds, from the start until the last byte is
sent by libcurl (i.e. the request is sent off).

Closes #14189

progress: ratelimit/progress tweaks

- multi.c: when ratelimiting a transfer stops (MSTATE_RATELIMITING ->
MSTATE_PERFORMING), run the MSTATE_PERFORMING state right away

- urld

progress: ratelimit/progress tweaks

- multi.c: when ratelimiting a transfer stops (MSTATE_RATELIMITING ->
MSTATE_PERFORMING), run the MSTATE_PERFORMING state right away

- urldata.h: factor out upload and download progress counters into a
struct, use that for passing these to progress update functions

- progress.c/getinfo.c: change names of moved progress counters

- progress.c: use new structs and a helper struct to factor repeated
calculation into static helpers

Closes #14335

show more ...

multi: do a final progress update on connect failure

To fix timing metric etc

Co-authored-by: Justin Maggard
Fixes #14204
Closes #14239

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period

Closes #14073

show more ...

getinfo: CURLINFO_QUEUE_TIME_T

Returns the time, in microseconds, during which this transfer was held
in a waiting queue before it started "for real". A transfer might be put
in a qu

getinfo: CURLINFO_QUEUE_TIME_T

Returns the time, in microseconds, during which this transfer was held
in a waiting queue before it started "for real". A transfer might be put
in a queue if after getting started, it cannot create a new connection
etc due to set conditions and limits imposed by the application.

Ref: #12293
Closes #12368

show more ...

multi: during ratelimit multi_getsock should return no sockets

... as there is nothing to wait for then, it just waits. Otherwise, this
causes much more CPU work and updates than necessa

multi: during ratelimit multi_getsock should return no sockets

... as there is nothing to wait for then, it just waits. Otherwise, this
causes much more CPU work and updates than necessary during ratelimit
periods.

Ref: https://curl.se/mail/lib-2023-11/0056.html
Closes #12430

show more ...

lib: client writer, part 2, accounting + logging

This PR has these changes:

Renaming of unencode_* to cwriter, e.g. client writers
- documentation of sendf.h functions
- mov

lib: client writer, part 2, accounting + logging

This PR has these changes:

Renaming of unencode_* to cwriter, e.g. client writers
- documentation of sendf.h functions
- move max decode stack checks back to content_encoding.c
- define writer phase which was used as order before
- introduce phases for monitoring inbetween decode phases
- offering default implementations for init/write/close

Add type paramter to client writer's do_write()
- always pass all writes through the writer stack
- writers who only care about BODY data will pass other writes unchanged

add RAW and PROTOCOL client writers
- RAW used for Curl_debug() logging of CURLINFO_DATA_IN
- PROTOCOL used for updates to data->req.bytecount, max_filesize checks and
Curl_pgrsSetDownloadCounter()
- remove all updates of data->req.bytecount and calls to
Curl_pgrsSetDownloadCounter() and Curl_debug() from other code
- adjust test457 expected output to no longer see the excess write

Closes #12184

show more ...

lib: let the max filesize option stop too big transfers too

Previously it would only stop them from getting started if the size is
known to be too big then.

Update the libcurl a

lib: let the max filesize option stop too big transfers too

Previously it would only stop them from getting started if the size is
known to be too big then.

Update the libcurl and curl docs accordingly.

Fixes #11810
Reported-by: Elliot Killick
Assisted-by: Jay Satiro
Closes #11820

show more ...

misc: remove support for curl_off_t < 8 bytes

Closes #10597

connections: introduce http/3 happy eyeballs

New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURL

connections: introduce http/3 happy eyeballs

New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.

HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.

HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue

HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.

GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).

- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.

Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.

HTTP/3 documentation on how https eyeballing works.

TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).

Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy

pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.

Closes #10349

show more ...

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- save

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205

show more ...

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869

show more ...

progress: make trspeed avoid floats

and compiler warnings for data conversions.

Reported-by: Michał Antoniak
Fixes #7645
Closes #7653

progress: fix a compile warning on some systems

lib/progress.c:380:40: warning: conversion to 'long double' from
'curl_off_t {aka long long int}' may alter its value [-Wconversion]

progress: fix a compile warning on some systems

lib/progress.c:380:40: warning: conversion to 'long double' from
'curl_off_t {aka long long int}' may alter its value [-Wconversion]

Closes #7549

show more ...

progress: reset limit_size variables at transfer start

Otherwise the old value would linger from a previous use and would mess
up the network speed cap logic.

Reported-by: Ymir1

progress: reset limit_size variables at transfer start

Otherwise the old value would linger from a previous use and would mess
up the network speed cap logic.

Reported-by: Ymir1711 on github

Fixes #7042
Closes #7043

show more ...

progress/trspeed: use a local convenient pointer to beautify code

The function becomes easier to read and understand with less repetition.

trspeed: use long double for transfer speed calculation

progress: move transfer speed calc into function

This silences two scan-build-11 warnings: "The result of the '/'
expression is undefined"

Bug: https://curl.se/mail/lib-2021-05/

progress: move transfer speed calc into function

This silences two scan-build-11 warnings: "The result of the '/'
expression is undefined"

Bug: https://curl.se/mail/lib-2021-05/0022.html
Closes #7035

show more ...

progress: when possible, calculate transfer speeds with microseconds

... this improves precision, especially for transfers in the few or even
sub millisecond range.

Reported-by:

progress: when possible, calculate transfer speeds with microseconds

... this improves precision, especially for transfers in the few or even
sub millisecond range.

Reported-by: J. Bromley
Fixes #7017
Closes #7020

show more ...

send_speed: simplify the checks for if a speed limit is set

... as we know the value cannot be set to negative: enforced by
setopt()

config: remove CURL_SIZEOF_CURL_OFF_T use only SIZEOF_CURL_OFF_T

Make the code consistently use a single name for the size of the
"curl_off_t" type.

Closes #6702