curl: improved IPFS and IPNS URL support

Previously just ipfs://<cid> and ipns://<cid> was supported, which is
too strict for some usecases.

This patch allows paths and query ar

curl: improved IPFS and IPNS URL support

Previously just ipfs://<cid> and ipns://<cid> was supported, which is
too strict for some usecases.

This patch allows paths and query arguments to be used too.
Making this work according to normal http semantics:

ipfs://<cid>/foo/bar?key=val
ipns://<cid>/foo/bar?key=val

The gateway url support is changed.
It now only supports gateways in the form of:

http://<gateway>/foo/bar
http://<gateway>

Query arguments here are explicitly not allowed and trigger an intended
malformed url error.

There also was a crash when IPFS_PATH was set with a non trailing
forward slash. This has been fixed.

Lastly, a load of test cases have been added to verify the above.

Reported-by: Steven Allen
Fixes #12148
Closes #12152

show more ...

tool: fix --capath when proxy support is disabled

After 95e8515ca0, --capath always sets CURLOPT_PROXY_CAPATH, which fails
with CURLE_UNKNOWN_OPTION when proxy support is disabled.

tool: fix --capath when proxy support is disabled

After 95e8515ca0, --capath always sets CURLOPT_PROXY_CAPATH, which fails
with CURLE_UNKNOWN_OPTION when proxy support is disabled.

Closes #12089

show more ...

tool_operate: fix links in ipfs errors

URL fragment links generated from headers in
https://curl.se/docs/ipfs.html are lowercase.

Closes #12133

tool: use our own stderr variable

Earlier this year we changed our own stderr variable to use the standard
name `stderr` (to avoid bugs where someone is using `stderr` instead of
the

tool: use our own stderr variable

Earlier this year we changed our own stderr variable to use the standard
name `stderr` (to avoid bugs where someone is using `stderr` instead of
the curl-tool specific variable). This solution needed to override the
standard `stderr` symbol via the preprocessor. This in turn didn't play
well with unity builds and caused curl tool to crash or stay silent due
to an uninitialized stderr. This was a hard to find issue, fixed by
manually breaking out one file from the unity sources.

To avoid two these two tricks, this patch implements a different
solution: Restore using our own local variable for our stderr output and
leave `stderr` as-is. To avoid using `stderr` by mistake, add a
`checksrc` rule (based on logic we already used in lib for `strerror`)
that detects any `stderr` use in `src` and points to using our own
variable instead: `tool_stderr`.

Follow-up to 06133d3e9b8aeb9e9ca0b3370c246bdfbfc8619e
Follow-up to 2f17a9b654121dd1ecf4fc043c6d08a9da3522db

Closes #11958

show more ...

tool_operate: free 'gateway' correctly

Pointed out by Coverity. The fix in 93885cf3a8d4e was incomplete.

Also removed repeated wording in IPFS related error messages.

Close

tool_operate: free 'gateway' correctly

Pointed out by Coverity. The fix in 93885cf3a8d4e was incomplete.

Also removed repeated wording in IPFS related error messages.

Closes #11969

show more ...

tool_operate: avoid strlen() -1 on zero length content from file

Follow-up to 65b563a96a226649ba12cb1e

Closes #11959

tool_operate: fix memory mixups

Switch to plain getenv() from curl_getenv() to avoid the allocation and
having to keep track of which free() or curl_free() that need to be
used.

tool_operate: fix memory mixups

Switch to plain getenv() from curl_getenv() to avoid the allocation and
having to keep track of which free() or curl_free() that need to be
used.

Coverity found issues and a memory leak.

Follow-up to 65b563a96a226649ba12cb1e

Closes #11959

show more ...

curl: add support for the IPFS protocols:

- ipfs://<cid>
- ipns://<cid>

This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>

Fo

curl: add support for the IPFS protocols:

- ipfs://<cid>
- ipns://<cid>

This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>

For more information consult the readme at:
https://curl.se/docs/ipfs.html

Closes #8805

show more ...

tool_operate: make aws-sigv4 not require TLS to be used

Maybe not used too often, but we want it for testing and it should work.

tool_operate: fix memory leak when SSL_CERT_DIR is used

Detected by Coverity

Follow-up to 29bce9857a12b6cfa726a5

Closes #11577

tool_cb_wrt: fix invalid unicode for windows console

- Suppress an incomplete UTF-8 sequence at the end of the buffer.

- Attempt to reconstruct incomplete UTF-8 sequence from prior

tool_cb_wrt: fix invalid unicode for windows console

- Suppress an incomplete UTF-8 sequence at the end of the buffer.

- Attempt to reconstruct incomplete UTF-8 sequence from prior call(s)
in current call.

Prior to this change, in Windows console UTF-8 sequences split between
two or more calls to the write callback would cause invalid "replacement
characters" U+FFFD to be printed instead of the actual Unicode
character. This is because in Windows only UTF-16 encoded characters are
printed to the console, therefore we convert the UTF-8 contents to
UTF-16, which cannot be done with partial UTF-8 sequences.

Reported-by: Maksim Arhipov

Fixes https://github.com/curl/curl/issues/9841
Closes https://github.com/curl/curl/pull/10890

show more ...

tool: add "variable" support

Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single

tool: add "variable" support

Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single dash (-)).

Variable content is expanded in option parameters using "{{name}}"
(without the quotes) if the option name is prefixed with
"--expand-". This gets the contents of the variable "name" inserted, or
a blank if the name does not exist as a variable. Insert "{{" verbatim
in the string by prefixing it with a backslash, like "\\{{".

Import an environment variable with --variable %name. It makes curl exit
with an error if the environment variable is not set. It can also rather
get a default value if the variable does not exist, using =content or
@file like shown above.

Example: get the USER environment variable into the URL:

--variable %USER
--expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make
the variable contents more convenient to use. It can trim leading and
trailing white space with "trim", output the contents as a JSON quoted
string with "json", URL encode it with "url" and base 64 encode it with
"b64". To apply functions to a variable expansion, add them colon
separated to the right side of the variable. They are then performed in
a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable
called "fix". Make sure that the content is trimmed and percent-encoded
sent as POST data:

--variable %HOME=/home/default
--expand-variable fix@{{HOME}}/.secret
--expand-data "{{fix:trim:url}}"
https://example.com/

Documented. Many new test cases.

Co-brainstormed-by: Emanuele Torre
Assisted-by: Jat Satiro
Closes #11346

show more ...

tool_operate: allow SSL_CERT_FILE and SSL_CERT_DIR

... used at once.

Reported-by: Gabriel Corona
Fixes #11325
Closes #11531

tool_operate: allow cookie lines up to 8200 bytes

Since this option might set multiple cookies in the same line, it does
not make total sense to cap this at 4096 bytes, which is the limi

tool_operate: allow cookie lines up to 8200 bytes

Since this option might set multiple cookies in the same line, it does
not make total sense to cap this at 4096 bytes, which is the limit for a
single cookie name or value.

Closes #11303

show more ...

haproxy: add --haproxy-clientip flag to spoof client IPs

CURLOPT_HAPROXY_CLIENT_IP in the library

Closes #10779

curl: add --ca-native and --proxy-ca-native

These are two boolean options to ask curl to use the native OS's CA
store when verifying TLS servers. For peers and for proxies
respective

curl: add --ca-native and --proxy-ca-native

These are two boolean options to ask curl to use the native OS's CA
store when verifying TLS servers. For peers and for proxies
respectively.

They currently only have an effect for curl on Windows when built to use
OpenSSL for TLS.

Closes #11049

show more ...

CURLOPT_MAIL_RCPT_ALLOWFAILS: replace CURLOPT_MAIL_RCPT_ALLLOWFAILS

Deprecate the name using three Ls and prefer the name with two.

Replaces #10047
Closes #11218

test447: test PUTting a file that grows

... and have curl trim the end when it reaches the expected total amount
of bytes instead of over-sending.

Reported-by: JustAnotherArchiv

test447: test PUTting a file that grows

... and have curl trim the end when it reaches the expected total amount
of bytes instead of over-sending.

Reported-by: JustAnotherArchivist on github
Closes #11223

show more ...

curl: count uploaded data to stop at the originally given size

Closes #11223
Fixes #11222
Reported-by: JustAnotherArchivist on github

tool: remove exclamation marks from error/warning messages

tool: use errorf() for error output

Convert a number of fprintf() calls.

tool: remove newlines from all helpf/notef/warnf/errorf calls

Make voutf() always add one.

Closes #11226

tool_urlglob: use curl_off_t instead of longs

To handle more globs better (especially on Windows)

Closes #11224

checksrc: disallow spaces before labels

Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
th

checksrc: disallow spaces before labels

Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
the line is the favoured style overall with 329 instances.

Out of the 86 labels not at the start of the line:
* 75 were indented with the same indentation level of the following line
* 8 were indented with exactly one space
* 2 were indented with one fewer indentation level then the following
line
* 1 was indented with the indentation level of the following line minus
three space (probably unintentional)

Co-Authored-By: Viktor Szakats

Closes #11134

show more ...

tool_operate: refuse (--data or --form) and --continue-at combo

libcurl assumes that a --continue-at resumption is done to continue an
upload using the read callback and neither --data n

tool_operate: refuse (--data or --form) and --continue-at combo

libcurl assumes that a --continue-at resumption is done to continue an
upload using the read callback and neither --data nor --form use
that and thus won't do what the user wants. Whatever the user wants
with this strange combination.

Add test 426 to verify.

Reported-by: Smackd0wn on github
Fixes #11081
Closes #11083

show more ...