WebSockets: make support official (non-experimental)

Inverts the configure/cmake options to instead provide options that
disable WebSockets and have them (ws + wss) enabled by default.

WebSockets: make support official (non-experimental)

Inverts the configure/cmake options to instead provide options that
disable WebSockets and have them (ws + wss) enabled by default.

Closes #14936

show more ...

lib/src: white space edits to comply better with code style

... as checksrc now finds and complains about these.

Closes #14921

lib: use FMT_ as prefix instead of CURL_FORMAT_

For printf format defines used internally. Makes the code slighly
easier to read.

Closes #14764

cpool: rename "connection cache/conncache" to "Connection Pools/cpool"

This is a better match for what they do and the general "cpool"
var/function prefix works well.

The pool n

cpool: rename "connection cache/conncache" to "Connection Pools/cpool"

This is a better match for what they do and the general "cpool"
var/function prefix works well.

The pool now handles very long hostnames correctly.

The following changes have been made:

* 'struct connectdata', e.g. connections, keep new members
named `destination` and ' destination_len' that fully specifies
interface+port+hostname of where the connection is going to.
This is used in the pool for "bundling" of connections with
the same destination. There is no limit on the length any more.
* Locking: all locks are done inside conncache.c when calling
into the pool and released on return. This eliminates hazards
of the callers keeping track.
* 'struct connectbundle' is now internal to the pool. It is no
longer referenced by a connection.
* 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters
no longer need to set it. Instead, the multi checks on leaving
MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now
multiplexed and new, e.g. not conn->bits.reuse. In that case
the processing of pending handles is triggered.
* The pool's init is provided with a callback to invoke on all
connections being discarded. This allows the cleanups in
`Curl_disconnect` to run, wherever it is decided to retire
a connection.
* Several pool operations can now be fully done with one call.
Pruning dead connections, upkeep and checks on pool limits
can now directly discard connections and need no longer return
those to the caller for doing that (as we have now the callback
described above).
* Finding a connection for reuse is now done via `Curl_cpool_find()`
and the caller provides callbacks to evaluate the connection
candidates.
* The 'Curl_cpool_check_limits()' now directly uses the max values
that may be set in the transfer's multi. No need to pass them
around. Curl_multi_max_host_connections() and
Curl_multi_max_total_connections() are gone.
* Add method 'Curl_node_llist()' to get the llist a node is in.
Used in cpool to verify connection are indeed in the list (or
not in any list) as they need to.

I left the conncache.[ch] as is for now and also did not touch the
documentation. If we update that outside the feature window, we can
do this in a separate PR.

Multi-thread safety is not achieved by this PR, but since more details
on how pools operate are now "internal" it is a better starting
point to go for this in the future.

Closes #14662

show more ...

transfer: Curl_sendrecv() and event related improvements

- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to

transfer: Curl_sendrecv() and event related improvements

- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to the
client. Add a `nowp` parameter since the single caller already
has this.
- Curl_sendrecv() now runs all possible operations whenever it is
called and either it had been polling sockets or the 'select_bits'
are set.
POLL_IN/POLL_OUT are not always directly related to send/recv
operations. Filters like HTTP/2, QUIC or TLS may monitor reverse
directions. If a transfer does not want to send (KEEP_SEND), it
will not do so, as before. Same for receives.
- Curl_update_timer() now checks the absolute timestamp of an expiry
and the last/new timeout to determine if the application needs
to stop/start/restart its timer. This fixes edge cases where
updates did not happen as they should have.
- improved --test-event curl_easy_perform() simulation to handle
situations where no sockets are registered but a timeout is
in place.
- fixed bug in events_socket() that complained about removing
a socket that was unknown, when indeed it had removed the socket
just before, only it was the last in the list
- fixed conncache's internal handle to carry the multi instance
(where the cache has one) so that operations on the closure handle
trigger event callbacks correctly.
- fixed conncache to not POLL_REMOVE a socket twice when a conneciton
was closed.

Closes #14561

show more ...

build: silence C4232 MSVC warnings in vcpkg ngtcp2 builds

Silence bogus MSVC warning C4232. Use the method already used
for similar cases earlier.

Also fixup existing suppressio

build: silence C4232 MSVC warnings in vcpkg ngtcp2 builds

Silence bogus MSVC warning C4232. Use the method already used
for similar cases earlier.

Also fixup existing suppressions to use pragma push/pop.

```
lib\vquic\curl_ngtcp2.c(709,40): error C2220: the following warning is treated as an error
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'client_initial': address of dllimport 'ngtcp2_crypto_client_initial_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'recv_crypto_data': address of dllimport 'ngtcp2_crypto_recv_crypto_data_cb' is not static, identity not guaran
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'encrypt': address of dllimport 'ngtcp2_crypto_encrypt_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'decrypt': address of dllimport 'ngtcp2_crypto_decrypt_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'hp_mask': address of dllimport 'ngtcp2_crypto_hp_mask_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'recv_retry': address of dllimport 'ngtcp2_crypto_recv_retry_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'update_key': address of dllimport 'ngtcp2_crypto_update_key_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'delete_crypto_aead_ctx': address of dllimport 'ngtcp2_crypto_delete_crypto_aead_ctx_cb' is not static, identit
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'delete_crypto_cipher_ctx': address of dllimport 'ngtcp2_crypto_delete_crypto_cipher_ctx_cb' is not static, ide
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'get_path_challenge_data': address of dllimport 'ngtcp2_crypto_get_path_challenge_data_cb' is not static, ident
```
Ref: https://github.com/curl/curl/actions/runs/10343459009/job/28627621355#step:10:30

Cherry-picked from #14495
Co-authored-by: Tal Regev
Ref: #14383
Closes #14510

show more ...

llist: remove direct struct accesses, use only functions

- Turned them all into functions to also do asserts etc.

- The llist related structs got all their fields renamed in order t

llist: remove direct struct accesses, use only functions

- Turned them all into functions to also do asserts etc.

- The llist related structs got all their fields renamed in order to make
sure no existing code remains using direct access.

- Each list node struct now points back to the list it "lives in", so
Curl_node_remove() no longer needs the list pointer.

- Rename the node struct and some of the access functions.

- Added lots of ASSERTs to verify API being used correctly

- Fix some cases of API misuse

Add docs/LLIST.md documenting the internal linked list API.

Closes #14485

show more ...

multi: make the "general" list of easy handles a Curl_llist

Instead of having an especially "unique" linked list handler for the
main list of easy handles within the multi handle, this n

multi: make the "general" list of easy handles a Curl_llist

Instead of having an especially "unique" linked list handler for the
main list of easy handles within the multi handle, this now uses a
regular Curl_llist for this as well.

With this change, it is also clearer that every easy handle added to a
multi handle belongs to one and only one out of three different lists:

process - the general one for normal transfer processing

pending - queued up waiting to get a connection (MSTATE_PENDING)

msgsent - transfer completed (MSTATE_MSGSENT)

An easy handle must therefore be removed from the current list before it
gets added to another.

Closes #14474

show more ...

lib: fix AIX build issues

- memdebug: replace keyword `malloc` with `__malloc__` to
not interfere with envs where `malloc` is redefined. Also apply
the fix to `alloc_size`.

lib: fix AIX build issues

- memdebug: replace keyword `malloc` with `__malloc__` to
not interfere with envs where `malloc` is redefined. Also apply
the fix to `alloc_size`.
Fixes:
```
lib/memdebug.h:107:13: warning: unknown attribute 'vec_malloc' ignored [-Wunknown-attributes]
CURL_EXTERN ALLOC_FUNC FILE *curl_dbg_fdopen(int filedes, const char *mode,
^~~~~~~~~~
lib/memdebug.h:37:37: note: expanded from macro 'ALLOC_FUNC'
# define ALLOC_FUNC __attribute__((malloc))
^~~~~~
/usr/include/stdlib.h:753:16: note: expanded from macro 'malloc'
#define malloc vec_malloc
^~~~~~~~~~
```

- memdebug: always undef before defining.
Also do this for the rest of functions redefined in the same block.
Avoids warning on AIX:
```
lib/memdebug.h:117:9: warning: 'malloc' macro redefined [-Wmacro-redefined]
#define malloc(size) curl_dbg_malloc(size, __LINE__, __FILE__)
^
/usr/include/stdlib.h:753:9: note: previous definition is here
#define malloc vec_malloc
^
```

- easy: fix `-Wformat` warning on AIX by adding a cast.
```
lib/easy.c:608:47: warning: format specifies type 'int' but the argument has type 'long' [-Wformat]
"%" CURL_FORMAT_SOCKET_T ")", fds[i].fd);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
```

- if2ip: silence compiler warning inside AIX system header.

```
/lib/if2ip.c:219:19: warning: signed shift result (0x80000000) sets the sign bit of the shift expression's type ('int') and becomes negative [-Wshift-sign-overflow]
if(ioctl(dummy, SIOCGIFADDR, &req) < 0) {
^~~~~~~~~~~
/usr/include/sys/ioctl.h:401:26: note: expanded from macro 'SIOCGIFADDR'
#define SIOCGIFADDR (int)_IOWR('i',33, struct oifreq) /* get ifnet address */
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/sys/ioctl.h:174:23: note: expanded from macro '_IOWR'
#define _IOWR(x,y,t) (IOC_INOUT|((sizeof(t)&IOCPARM_MASK)<<16)|(x<<8)|y)
^~~~~~~~~
/usr/include/sys/ioctl.h:168:20: note: expanded from macro 'IOC_INOUT'
#define IOC_INOUT (IOC_IN|IOC_OUT)
^~~~~~
/usr/include/sys/ioctl.h:167:28: note: expanded from macro 'IOC_IN'
#define IOC_IN (0x40000000<<1) /* copy in parameters */
~~~~~~~~~~^ ~
```

Ref: https://curl.se/dev/log.cgi?id=20240808180420-3809007
Assisted-by: Dan Fandrich
Closes #14464

show more ...

easy: fix curl_easy_upkeep for shared connection caches

- Determine which connection cache is or will be used by the easy handle
and perform connection upkeep on that cache.

T

easy: fix curl_easy_upkeep for shared connection caches

- Determine which connection cache is or will be used by the easy handle
and perform connection upkeep on that cache.

This change allows curl_easy_upkeep to be effective on easy handles that
are using a shared connection cache, either from a user created shared
connection cache or a user created multi which has its own shared
connection cache.

Prior to this change curl_easy_upkeep would upkeep the connection cache
for the easy handle only if that cache was from the multi owned by the
easy handle (ie curl_easy_perform was previously called and there's a
connection cache exclusive to the easy handle in
data->multi_easy->conn_cache).

Ref: https://curl.se/mail/lib-2024-01/0016.html

Closes https://github.com/curl/curl/pull/12677

show more ...

lib: add eos flag to send methods

Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.

This will

lib: add eos flag to send methods

Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.

Closes #14220

show more ...

connect: fix connection shutdown for event based processing

connections being shutdown would register sockets for events, but then
never remove these sockets again. Nor would the shutdow

connect: fix connection shutdown for event based processing

connections being shutdown would register sockets for events, but then
never remove these sockets again. Nor would the shutdown effectively
been performed.

- If a socket event involves a transfer, check if that is the
connection cache internal handle and run its multi_perform()
instead (the internal handle is used for all shutdowns).
- When a timer triggers for a transfer, check also if it is
about the connection cache internal handle.
- During processing shutdowns in the connection cache, assess
the shutdown timeouts. Register a Curl_expire() of the lowest
value for the cache's internal handle.

Reported-by: Gordon Parke
Fixes #14280
Closes #14296

show more ...

Revert "lib: send eos flag"

This reverts commit be93299f10ef0b2bf7fe5c82140120073831867a.

lib: send eos flag

Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.

This will help protocol

lib: send eos flag

Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.

Closes #14220

show more ...

multi: fix pollset during RESOLVING phase

- add a DEBUGASSERT for when a transfer's pollset should not be empty.
- move write unpausing from transfer loop into curl_easy_pause. This

multi: fix pollset during RESOLVING phase

- add a DEBUGASSERT for when a transfer's pollset should not be empty.
- move write unpausing from transfer loop into curl_easy_pause. This
make sure that the url_updatesocket() finds the correct state when
updating socket events.
- fix HTTP/2 proxy during connect phase to set sockets correctly
- fix test2600 to simulate a socket set
- move write unpausing from transfer loop into curl_easy_pause. This
make sure that the url_updatesocket() finds the correct state when
updating socket events.
- waiting for the resolver to deliver might not involve any sockets to
wait for. Do not generate a warning.

Fixes #14047
Closes #14074

show more ...

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period

Closes #14073

show more ...

lib: tidy up types and casts

Cherry-picked from #13489
Closes #13862

build: untangle `CURLDEBUG` and `DEBUGBUILD` macros

`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be gua

build: untangle `CURLDEBUG` and `DEBUGBUILD` macros

`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be guarded with
`DEBUGBUILD`.

Replace these uses with `DEBUGBUILD`.

This leaves `CURLDEBUG` uses solely for its intended purpose: to enable
the memory tracking debug feature.

Also:
- autotools: rely on `DEBUGBUILD` to enable `checksrc`.
Instead of `CURLDEBUG`, which worked in most cases because debug
builds enable `CURLDEBUG` by default, but it's not accurate.
- include `lib/easyif.h` instead of keeping a copy of a declaration.
- add CI test jobs for the build issues discovered.

Ref: https://github.com/curl/curl/pull/13694#issuecomment-2120311894
Closes #13718

show more ...

lib: clear the easy handle's saved errno before transfer

- Clear data->state.os_errno before transfer.

- Explain the change in behavior in the CURLINFO_OS_ERRNO doc.

- Add

lib: clear the easy handle's saved errno before transfer

- Clear data->state.os_errno before transfer.

- Explain the change in behavior in the CURLINFO_OS_ERRNO doc.

- Add to the CURLINFO_OS_ERRNO doc the list of libcurl network-related
errors that may cause the errno to be saved.

data->state.os_errno is saved before libcurl returns a network-related
failure such as connection failure. It is accessible to the user via
CURLINFO_OS_ERRNO so they can get more information about the failure.

Prior to this change it wasn't cleared before transfer, so if a user
retrieved the saved errno it could be from a previous transfer. That is
because an errno is not always saved for network-related errors.

Closes https://github.com/curl/curl/pull/13574

show more ...

cw-out: improved error handling

- remember error encountered in invoking write callback and always fail
afterwards without further invokes

- check behaviour in test_02_17 with

cw-out: improved error handling

- remember error encountered in invoking write callback and always fail
afterwards without further invokes

- check behaviour in test_02_17 with h2-pausing client

Reported-by: Pavel Kropachev
Fixes #13337
Closes #13340

show more ...

lib: use multi instead of multi_easy for the active multi

- Use data->multi and not data->multi_easy to refer to the active multi.

The easy handle's active multi is always data->mul

lib: use multi instead of multi_easy for the active multi

- Use data->multi and not data->multi_easy to refer to the active multi.

The easy handle's active multi is always data->multi.

This is a follow up to 757dfdf which changed curl so that an easy handle
used with the easy interface and then multi interface cannot have two
different multi handles associated with it at the same time
(data->multi_easy from the easy interface and data->multi from the multi
interface).

Closes https://github.com/curl/curl/pull/12665

show more ...

lib: move 'done' parameter to SingleRequests

A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connecti

lib: move 'done' parameter to SingleRequests

A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connections.
The "readwrite()" calls and functions connected to those carried a `bool
*done` parameter to indicate that the current `SingleRequest` is over.
This may happen before `upload_done` or `download_done` bits of
`SingleRequest` are set.

The problem with that is now `write_resp()` protocol handlers are
invoked in places where the `bool *done` cannot be passed up to the
caller. Instead of being a bool in the call chain, it needs to become a
member of `SingleRequest`, reflecting its state.

This removes the `bool *done` parameter and adds the `done` bit to
`SingleRequest` instead. It adds `Curl_req_soft_reset()` for using a
`SingleRequest` in a follow up, clearing `done` and other
flags/counters.

Closes #13096

show more ...

hyper: implement unpausing via client reader

Just a tidy up to contain 'ifdef' pollution of common
code parts with implementation specifics.

- remove the ifdef hyper unpausing i

hyper: implement unpausing via client reader

Just a tidy up to contain 'ifdef' pollution of common
code parts with implementation specifics.

- remove the ifdef hyper unpausing in easy.c
- add hyper client reader for CURL_CR_PROTOCOL phase
that implements the unpause method for calling
the hyper waker if it is set

Closes #13075

show more ...

mime: add client reader

Add `mime` client reader. Encapsulates reading from mime parts, getting
their length, rewinding and unpausing.

- remove special mime handling from sendf.

mime: add client reader

Add `mime` client reader. Encapsulates reading from mime parts, getting
their length, rewinding and unpausing.

- remove special mime handling from sendf.c and easy.c
- add general "unpause" method to client readers
- use new reader in http/imap/smtp
- make some mime functions static that are now only used internally

In addition:
- remove flag 'forbidchunk' as no longer needed

Closes #13039

show more ...

lib: Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
clarify when and at what level they operate
- send/recv of transfer related

lib: Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
clarify when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer
setup of `conn->sockfd` and `conn->writesockfd` on which
connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index
as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
naming consistency
- clarify that the special CURLE_AGAIN hangling to return
`CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
when it arrived in more than a single chunk

The method for sending not just raw bytes, but bytes that are either
"headers" or "body". The send abstraction stack, to to bottom, now is:

* `Curl_req_send()`: has parameter to indicate amount of header bytes,
buffers all data.
* `Curl_xfer_send()`: knows on which socket index to send, returns
amount of bytes sent.
* `Curl_conn_send()`: called with socket index, returns amount of bytes
sent.

In addition there is `Curl_req_flush()` for writing out all buffered
bytes.

`Curl_req_send()` is active for requests without body,
`Curl_buffer_send()` still being used for others. This is because the
special quirks need to be addressed in future parts:

* `expect-100` handling
* `Curl_fillreadbuffer()` needs to add directly to the new
`data->req.sendbuf`
* special body handlings, like `chunked` encodings and line end
conversions will be moved into something like a Client Reader.

In functions of the pattern `CURLcode xxx_send(..., ssize_t *written)`,
replace the `ssize_t` with a `size_t`. It makes no sense to allow for negative
values as the returned `CURLcode` already specifies error conditions. This
allows easier handling of lengths without casting.

Closes #12964

show more ...