TLS: TLSv1.3 earlydata support for curl

Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.

If a known TLS

TLS: TLSv1.3 earlydata support for curl

Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.

If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.

- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
bytes sent and accepted/rejected by the server.

Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
that ALPN protocol for negoptiation with the server. When the
sessions ALPN does not match the connections ALPN, earlydata
will not be enabled.
- Check that the server selected the correct ALPN protocol for
an earlydata connect. If the server does not confirm or reports
something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
if not connect-only.

Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature

Assisted-by: ad-chaos on github
Closes #15211

show more ...

curl: add --skip-existing

With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.

Test 994, 995 and

curl: add --skip-existing

With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.

Test 994, 995 and 996 verify.

Ref: #11012
Closes #13993

show more ...

tool_getparam: make --show-headers the same as --include

Simply a name alias that better explains what the option does.

Closes #13987

curl: support embedding a CA bundle

Add the ability to embed a CA bundle into the curl binary. It is used
when no other runtime or build-time option set one.

This helps curl-for

curl: support embedding a CA bundle

Add the ability to embed a CA bundle into the curl binary. It is used
when no other runtime or build-time option set one.

This helps curl-for-win macOS and Linux builds to run standalone, and
also helps Windows builds to avoid picking up the CA bundle from an
arbitrary (possibly world-writable) location (though this behaviour is
not currently disablable).

Usage:
- cmake: `-DCURL_CA_EMBED=/path/to/curl-ca-bundle.crt`
- autotools: `--with-ca-embed=/path/to/curl-ca-bundle.crt`
- Makefile.mk: `CURL_CA_EMBED=/path/to/curl-ca-bundle.crt`

Also add new command-line option `--dump-ca-embed` to dump the embedded
CA bundle to standard output.

Closes #14059

show more ...

tcpkeepalive: add CURLOPT_TCP_KEEPCNT and --keepalive-cnt

Closes #13885

curl: support VLAN Priority: --vlan-priority

Add --vlan-priority option to the command line tool for setting VLAN
priority.

Closes #13907

curl: (on linux) add MPTCP support

Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.

Multipath TCP has

curl: (on linux) add MPTCP support

Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.

Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.

To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.

This attached patch adds an --mptcp option which allows the creation of
an MPTCP socket instead of TCP on Linux. If Multipath TCP is not
supported on the system, an error will be reported. It is important to
note that if the end server doesn't support MPTCP, the connection will
continue after a seamless fallback to TCP.

Link: https://www.rfc-editor.org/rfc/rfc8684.html [1]
Link: https://www.tessares.net/apples-mptcp-story-so-far/ [2]
Link: https://www.mptcp.dev [3]
Co-developed-by: Dorian Craps (@CrapsDorian) <doriancraps@gmail.com>
Co-developed-by: Olivier Bonaventure (@obonaventure) <Olivier.Bonaventure@uclouvain.be>
Co-developed-by: Matthieu Baerts (@matttbe) <matttbe@kernel.org>
Signed-off-by: Dorian Craps <dorian.craps@student.vinci.be>

Closes #13278

show more ...

curl: support IP Type of Service / Traffic Class: --ip-tos

Add --ip-tos option to the command line tool for setting TOS for IPv4 or
Traffic Class for IPv6.

Closes #13606

TLS: add support for ECH (Encrypted Client Hello)

An EXPERIMENTAL feature used with CURLOPT_ECH and --ech.

Closes #11922

curl: add support for the IPFS protocols:

- ipfs://<cid>
- ipns://<cid>

This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>

Fo

curl: add support for the IPFS protocols:

- ipfs://<cid>
- ipns://<cid>

This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>

For more information consult the readme at:
https://curl.se/docs/ipfs.html

Closes #8805

show more ...

trace: make tracing available in non-debug builds

Add --trace-config to curl

Add curl_global_trace() to libcurl

Closes #11421

tool: add "variable" support

Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single

tool: add "variable" support

Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single dash (-)).

Variable content is expanded in option parameters using "{{name}}"
(without the quotes) if the option name is prefixed with
"--expand-". This gets the contents of the variable "name" inserted, or
a blank if the name does not exist as a variable. Insert "{{" verbatim
in the string by prefixing it with a backslash, like "\\{{".

Import an environment variable with --variable %name. It makes curl exit
with an error if the environment variable is not set. It can also rather
get a default value if the variable does not exist, using =content or
@file like shown above.

Example: get the USER environment variable into the URL:

--variable %USER
--expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make
the variable contents more convenient to use. It can trim leading and
trailing white space with "trim", output the contents as a JSON quoted
string with "json", URL encode it with "url" and base 64 encode it with
"b64". To apply functions to a variable expansion, add them colon
separated to the right side of the variable. They are then performed in
a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable
called "fix". Make sure that the content is trimmed and percent-encoded
sent as POST data:

--variable %HOME=/home/default
--expand-variable fix@{{HOME}}/.secret
--expand-data "{{fix:trim:url}}"
https://example.com/

Documented. Many new test cases.

Co-brainstormed-by: Emanuele Torre
Assisted-by: Jat Satiro
Closes #11346

show more ...

tool: add curl command line option `--trace-ids`

- added and documented --trace-ids to prepend (after the timestamp)
the transfer and connection identifiers to each verbose log line

tool: add curl command line option `--trace-ids`

- added and documented --trace-ids to prepend (after the timestamp)
the transfer and connection identifiers to each verbose log line
- format is [n-m] with `n` being the transfer id and `m` being the
connection id. In case there is not valid connection id, print 'x'.
- Log calls with a handle that has no transfer id yet, are written
without any ids.

Closes #11185

show more ...

haproxy: add --haproxy-clientip flag to spoof client IPs

CURLOPT_HAPROXY_CLIENT_IP in the library

Closes #10779

curl: add --ca-native and --proxy-ca-native

These are two boolean options to ask curl to use the native OS's CA
store when verifying TLS servers. For peers and for proxies
respective

curl: add --ca-native and --proxy-ca-native

These are two boolean options to ask curl to use the native OS's CA
store when verifying TLS servers. For peers and for proxies
respectively.

They currently only have an effect for curl on Windows when built to use
OpenSSL for TLS.

Closes #11049

show more ...

curl: add --proxy-http2

For trying HTTP/2 with an HTTPS proxy.

Closes #10926

src: add --http3-only

Warning: --http3 and --http3-only are subject to change again (or be
removed) before HTTP/3 support goes non-experimental.

Closes #10264

curl: add --url-query

This option adds a piece of data, usually a name + value pair, to the
end of the URL query part. The syntax is identical to that used for
--data-urlencode with

curl: add --url-query

This option adds a piece of data, usually a name + value pair, to the
end of the URL query part. The syntax is identical to that used for
--data-urlencode with one extension:

If the argument starts with a '+' (plus), the rest of the string is
provided as-is unencoded.

This allows users to "build" query parts with options and URL encoding
even when not doing GET requests, which the already provided option -G
(--get) is limited to.

This idea was born in a Twitter thread.

Closes #9691

show more ...

curl: add --rate to set max request rate per time unit

--rate "12/m" - for 12 per minute or
--rate "5/h" - for 5 per hour

Removed from TODO

Closes #8671

curl: add --no-clobber

Does not overwrite output files if they already exist

Closes #7708
Co-authored-by: Daniel Stenberg

curl: add --remove-on-error

If a transfer returns an error, using this option makes curl remove the
leftover downloded (partial) local file before exiting.

Added test 376 to ver

curl: add --remove-on-error

If a transfer returns an error, using this option makes curl remove the
leftover downloded (partial) local file before exiting.

Added test 376 to verify

Closes #8503

show more ...

tool_getparam: initial --json support

Adds these test cases:

383 - simple single command line option
384 - reading it from stdin
385 - getting two --json options on comma

tool_getparam: initial --json support

Adds these test cases:

383 - simple single command line option
384 - reading it from stdin
385 - getting two --json options on command line
386 - --next works after --json

Closes #8314

show more ...

mime: use percent-escaping for multipart form field and file names

Until now, form field and file names where escaped using the
backslash-escaping algorithm defined for multipart mails.

mime: use percent-escaping for multipart form field and file names

Until now, form field and file names where escaped using the
backslash-escaping algorithm defined for multipart mails. This commit
replaces this with the percent-escaping method for URLs.

As this may introduce incompatibilities with server-side applications, a
new libcurl option CURLOPT_MIME_OPTIONS with bitmask
CURLMIMEOPT_FORMESCAPE is introduced to revert to legacy use of
backslash-escaping. This is controlled by new cli tool option
--form-escape.

New tests and documentation are provided for this feature.

Reported by: Ryan Sleevi
Fixes #7789
Closes #7805

show more ...

libssh2: add SHA256 fingerprint support

Added support for SHA256 fingerprint in command line curl and in
libcurl.

Closes #7646

schannel: Disable auto credentials; add an option to enable it

- Disable auto credentials by default. This is a breaking change
for clients that are using it, wittingly or not.

schannel: Disable auto credentials; add an option to enable it

- Disable auto credentials by default. This is a breaking change
for clients that are using it, wittingly or not.

- New libcurl ssl option value CURLSSLOPT_AUTO_CLIENT_CERT tells libcurl
to automatically locate and use a client certificate for
authentication, when requested by the server.

- New curl tool options --ssl-auto-client-cert and
--proxy-ssl-auto-client-cert map to CURLSSLOPT_AUTO_CLIENT_CERT.

This option is only supported for Schannel (the native Windows SSL
library). Prior to this change Schannel would, with no notification to
the client, attempt to locate a client certificate and send it to the
server, when requested by the server. Since the server can request any
certificate that supports client authentication in the OS certificate
store it could be a privacy violation and unexpected.

Fixes https://github.com/curl/curl/issues/2262
Reported-by: Jeroen Ooms
Assisted-by: Wes Hinsley
Assisted-by: Rich FitzJohn

Ref: https://curl.se/mail/lib-2021-02/0066.html
Reported-by: Morten Minde Neergaard

Closes https://github.com/curl/curl/pull/6673

show more ...