http: check for "Host:" case insensitively

When checking if the user wants to replace the header, the check should
be case insensitive.

Adding test 461 to verify

Found-

http: check for "Host:" case insensitively

When checking if the user wants to replace the header, the check should
be case insensitive.

Adding test 461 to verify

Found-by: Dan Fandrich
Ref: #12782
Closes #12784

show more ...

http: remove comment reference to a removed solution

Follow-up to 58974d25d

Closes #12785

websockets: refactor decode chain

- use client writer stack for decoding frames
- move websocket protocol handler to ws.c

Closes #12713

lib: replace readwrite with write_resp

This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
m

lib: replace readwrite with write_resp

This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.

The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.

Curl's protocol handler `readwrite()` method been changed:

```diff
- CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
- const char *buf, size_t blen,
- size_t *pconsumed, bool *readmore);
+ CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+ bool is_eos, bool *done);
```

The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:

* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
finished

This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.

The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.

HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.

The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:

```
static CURLcode readwrite_data(data, ...)
{
do {
Curl_xfer_recv_resp(data, buf)
...
Curl_xfer_write_resp(data, buf)
...
} while(interested);
...
}
```

All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.

All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.

Closes #12480

show more ...

lib: add debug log outputs for CURLE_BAD_FUNCTION_ARGUMENT

Closes #12658

curl.h: add CURLE_TOO_LARGE

A new error code to be used when an internal field grows too large, like
when a dynbuf reaches its maximum. Previously it would return
CURLE_OUT_OF_MEMORY

curl.h: add CURLE_TOO_LARGE

A new error code to be used when an internal field grows too large, like
when a dynbuf reaches its maximum. Previously it would return
CURLE_OUT_OF_MEMORY for this, which is highly misleading.

Ref: #12268
Closes #12269

show more ...

build: enable missing OpenSSF-recommended warnings, with fixes

https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1]

build: enable missing OpenSSF-recommended warnings, with fixes

https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].

Enable new recommended warnings (except `-Wsign-conversion`):

- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
printf arguments with it. This is a copy of existing
`CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
with redefinting the `printf` symbol:
https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
Follow-up to d5c0351055d5709da8f3e16c91348092fdb481aa #2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
checks. Previously it was always disabled due to the internal `printf`
macro.

Fix them:

- fix bug where an `set_ipv6_v6only()` call was missed in builds with
`--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.

[1] https://github.com/ossf/wg-best-practices-os-developers/blob/56c0fde3895bfc55c8a973ef49a2572c507b2ae1/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.md

Closes #12489

show more ...

http: fix off-by-one error in request method length check

It should allow one more byte.

Closes #12534

lib: rename Curl_strndup to Curl_memdup0 to avoid misunderstanding

Since the copy does not stop at a null byte, let's not call it anything
that makes you think it works like the common s

lib: rename Curl_strndup to Curl_memdup0 to avoid misunderstanding

Since the copy does not stop at a null byte, let's not call it anything
that makes you think it works like the common strndup() function.

Based on feedback from Jay Satiro, Stefan Eissing and Patrick Monnerat

Closes #12490

show more ...

lib: strndup/memdup instead of malloc, memcpy and null-terminate

- bufref: use strndup
- cookie: use strndup
- formdata: use strndup
- ftp: use strndup
- gtls: use aprin

lib: strndup/memdup instead of malloc, memcpy and null-terminate

- bufref: use strndup
- cookie: use strndup
- formdata: use strndup
- ftp: use strndup
- gtls: use aprintf instead of malloc + strcpy * 2
- http: use strndup
- mbedtls: use strndup
- md4: use memdup
- ntlm: use memdup
- ntlm_sspi: use strndup
- pingpong: use memdup
- rtsp: use strndup instead of malloc, memcpy and null-terminate
- sectransp: use strndup
- socks_gssapi.c: use memdup
- vtls: use dynbuf instead of malloc, snprintf and memcpy
- vtls: use strdup instead of malloc + memcpy
- wolfssh: use strndup

Closes #12453

show more ...

Curl_http_body: cleanup properly when Curl_getformdata errors

Reported-by: yushicheng7788 on github
Based-on-work-by: yushicheng7788 on github
Fixes #12410
Closes #12421

transfer: cleanup done+excess handling

- add `SingleRequest->download_done` as indicator that
all download bytes have been received
- remove `stop_reading` bool from readwrite func

transfer: cleanup done+excess handling

- add `SingleRequest->download_done` as indicator that
all download bytes have been received
- remove `stop_reading` bool from readwrite functions
- move excess body handling into client download writer

Closes #12371

show more ...

lib: fix comment typos

Five separate ones, found by codespell

Closes #12390

transfer: readwrite improvements

- changed header/chunk/handler->readwrite prototypes to accept `buf`,
`blen` and a `pconsumed` pointer. They now get the buffer to work on
and re

transfer: readwrite improvements

- changed header/chunk/handler->readwrite prototypes to accept `buf`,
`blen` and a `pconsumed` pointer. They now get the buffer to work on
and report back how many bytes they consumed
- eliminated `k->str` in SingleRequest
- improved excess data handling to properly calculate with any body data
left in the headerb buffer
- eliminated `k->badheader` enum to only be a bool

Closes #12283

show more ...

http: fix `-Wunused-parameter` with no auth and no proxy

```
lib/http.c:734:26: warning: unused parameter 'proxy' [-Wunused-parameter]
bool proxy)

http: fix `-Wunused-parameter` with no auth and no proxy

```
lib/http.c:734:26: warning: unused parameter 'proxy' [-Wunused-parameter]
bool proxy)
^
```

Reviewed-by: Marcel Raad
Closes #12338

show more ...

http: fix `-Wunused-variable` compiler warning

Fix compiler warnings in builds with disabled auths, NTLM and SPNEGO.

E.g. with `CURL_DISABLE_BASIC_AUTH` + `CURL_DISABLE_BEARER_AUTH`

http: fix `-Wunused-variable` compiler warning

Fix compiler warnings in builds with disabled auths, NTLM and SPNEGO.

E.g. with `CURL_DISABLE_BASIC_AUTH` + `CURL_DISABLE_BEARER_AUTH` +
`CURL_DISABLE_DIGEST_AUTH` + `CURL_DISABLE_NEGOTIATE_AUTH` +
`CURL_DISABLE_NTLM` on non-Windows.

```
./curl/lib/http.c:737:12: warning: unused variable 'result' [-Wunused-variable]
CURLcode result = CURLE_OK;
^
./curl/lib/http.c:995:18: warning: variable 'availp' set but not used [-Wunused-but-set-variable]
unsigned long *availp;
^
./curl/lib/http.c:996:16: warning: variable 'authp' set but not used [-Wunused-but-set-variable]
struct auth *authp;
^
```

Regression from e92edfbef64448ef461117769881f3ed776dec4e #11490

Fixes #12228
Closes #12335

show more ...

misc: fix -Walloc-size warnings

GCC 14 introduces a new -Walloc-size included in -Wextra which gives:

```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operat

misc: fix -Walloc-size warnings

GCC 14 introduces a new -Walloc-size included in -Wextra which gives:

```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operate.c:213:5: warning: allocation of insufficient size ‘1’ for type ‘struct per_transfer’ with size ‘480’ [-Walloc-size]
213 | p = calloc(sizeof(struct per_transfer), 1);
| ^
src/var.c: In function ‘addvariable’:
src/var.c:361:5: warning: allocation of insufficient size ‘1’ for type ‘struct var’ with size ‘32’ [-Walloc-size]
361 | p = calloc(sizeof(struct var), 1);
| ^
```

The calloc prototype is:
```
void *calloc(size_t nmemb, size_t size);
```

So, just swap the number of members and size arguments to match the
prototype, as we're initialising 1 struct of size `sizeof(struct
...)`. GCC then sees we're not doing anything wrong.

Closes #12292

show more ...

vtls: use ALPN "http/1.1" for HTTP/1.x, including HTTP/1.0

Some servers don't support the ALPN protocol "http/1.0" (e.g. IIS 10),
avoid it and use "http/1.1" instead.

This rever

vtls: use ALPN "http/1.1" for HTTP/1.x, including HTTP/1.0

Some servers don't support the ALPN protocol "http/1.0" (e.g. IIS 10),
avoid it and use "http/1.1" instead.

This reverts commit df856cb5c9 (#10183).

Fixes #12259
Closes #12285

show more ...

strdup: do Curl_strndup without strncpy

To avoid (false positive) gcc-13 compiler warnings.

Follow-up to 4855debd8a2c1cb

Assisted-by: Jay Satiro
Reported-by: Viktor Sza

strdup: do Curl_strndup without strncpy

To avoid (false positive) gcc-13 compiler warnings.

Follow-up to 4855debd8a2c1cb

Assisted-by: Jay Satiro
Reported-by: Viktor Szakats
Fixes #12258

show more ...

HTTP: fix empty-body warning

This change fixes a compiler warning with gcc-12.2.0 when
`-DCURL_DISABLE_BEARER_AUTH=ON` is used.

/home/tox/src/curl/lib/http.c: In function 'C

HTTP: fix empty-body warning

This change fixes a compiler warning with gcc-12.2.0 when
`-DCURL_DISABLE_BEARER_AUTH=ON` is used.

/home/tox/src/curl/lib/http.c: In function 'Curl_http_input_auth':
/home/tox/src/curl/lib/http.c:1147:12: warning: suggest braces around empty body in an 'else' statement [-Wempty-body]
1147 | ;
| ^

Closes #12262

show more ...

http: consider resume with CURLOPT_FAILONERRROR and 416 to be fine

Finding a 'Content-Range:' in the response changed the handling.

Add test case 1475 to verify -C - with 416 and Co

http: consider resume with CURLOPT_FAILONERRROR and 416 to be fine

Finding a 'Content-Range:' in the response changed the handling.

Add test case 1475 to verify -C - with 416 and Content-Range: header,
which is almost exactly like test 194 which instead uses a fixed -C
offset. Adjusted test 194 to also be considered fine.

Fixes #10521
Reported-by: Smackd0wn
Fixes #12174
Reported-by: Anubhav Rai
Closes #12176

show more ...

build: fix compiler warning with auths disabled

```
./curl/lib/http.c:979:12: warning: unused function 'is_valid_auth_separator' [-Wunused-function]
static int is_valid_auth_separato

build: fix compiler warning with auths disabled

```
./curl/lib/http.c:979:12: warning: unused function 'is_valid_auth_separator' [-Wunused-function]
static int is_valid_auth_separator(char ch)
^
5 warnings generated.
```

Follow-up to e92edfbef64448ef461117769881f3ed776dec4e #11490

Closes #12227

show more ...

RTSP: improved RTP parser

- fix HTTP header parsing to report incomplete
lines it buffers as consumed!
- re-implement the RTP parser for interleave RTP
messages for robustnes

RTSP: improved RTP parser

- fix HTTP header parsing to report incomplete
lines it buffers as consumed!
- re-implement the RTP parser for interleave RTP
messages for robustness. It is now keeping its
state at the connection
- RTSP protocol handler "readwrite" implementation
now tracks if the response is before/in/after
header parsing or "in" a bod by calling
"Curl_http_readwrite_headers()" itself. This
allows it to know when non-RTP bytes are "junk"
or HEADER or BODY.
- tested with #12035 and various small receive
sizes where current master fails

Closes #12052

show more ...

http: avoid Expect: 100-continue if Upgrade: is used

Reported-by: Daniel Jelinski
Fixes #12022
Closes #12062

lib: move handling of `data->req.writer_stack` into Curl_client_write()

- move definitions from content_encoding.h to sendf.h
- move create/cleanup/add code into sendf.c
- installed

lib: move handling of `data->req.writer_stack` into Curl_client_write()

- move definitions from content_encoding.h to sendf.h
- move create/cleanup/add code into sendf.c
- installed content_encoding writers will always be called
on Curl_client_write(CLIENTWRITE_BODY)
- Curl_client_cleanup() frees writers and tempbuffers from
paused transfers, irregardless of protocol

Closes #11908

show more ...