cpool: rename "connection cache/conncache" to "Connection Pools/cpool"

This is a better match for what they do and the general "cpool"
var/function prefix works well.

The pool n

cpool: rename "connection cache/conncache" to "Connection Pools/cpool"

This is a better match for what they do and the general "cpool"
var/function prefix works well.

The pool now handles very long hostnames correctly.

The following changes have been made:

* 'struct connectdata', e.g. connections, keep new members
named `destination` and ' destination_len' that fully specifies
interface+port+hostname of where the connection is going to.
This is used in the pool for "bundling" of connections with
the same destination. There is no limit on the length any more.
* Locking: all locks are done inside conncache.c when calling
into the pool and released on return. This eliminates hazards
of the callers keeping track.
* 'struct connectbundle' is now internal to the pool. It is no
longer referenced by a connection.
* 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters
no longer need to set it. Instead, the multi checks on leaving
MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now
multiplexed and new, e.g. not conn->bits.reuse. In that case
the processing of pending handles is triggered.
* The pool's init is provided with a callback to invoke on all
connections being discarded. This allows the cleanups in
`Curl_disconnect` to run, wherever it is decided to retire
a connection.
* Several pool operations can now be fully done with one call.
Pruning dead connections, upkeep and checks on pool limits
can now directly discard connections and need no longer return
those to the caller for doing that (as we have now the callback
described above).
* Finding a connection for reuse is now done via `Curl_cpool_find()`
and the caller provides callbacks to evaluate the connection
candidates.
* The 'Curl_cpool_check_limits()' now directly uses the max values
that may be set in the transfer's multi. No need to pass them
around. Curl_multi_max_host_connections() and
Curl_multi_max_total_connections() are gone.
* Add method 'Curl_node_llist()' to get the llist a node is in.
Used in cpool to verify connection are indeed in the list (or
not in any list) as they need to.

I left the conncache.[ch] as is for now and also did not touch the
documentation. If we update that outside the feature window, we can
do this in a separate PR.

Multi-thread safety is not achieved by this PR, but since more details
on how pools operate are now "internal" it is a better starting
point to go for this in the future.

Closes #14662

show more ...

transfer: Curl_sendrecv() and event related improvements

- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to

transfer: Curl_sendrecv() and event related improvements

- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to the
client. Add a `nowp` parameter since the single caller already
has this.
- Curl_sendrecv() now runs all possible operations whenever it is
called and either it had been polling sockets or the 'select_bits'
are set.
POLL_IN/POLL_OUT are not always directly related to send/recv
operations. Filters like HTTP/2, QUIC or TLS may monitor reverse
directions. If a transfer does not want to send (KEEP_SEND), it
will not do so, as before. Same for receives.
- Curl_update_timer() now checks the absolute timestamp of an expiry
and the last/new timeout to determine if the application needs
to stop/start/restart its timer. This fixes edge cases where
updates did not happen as they should have.
- improved --test-event curl_easy_perform() simulation to handle
situations where no sockets are registered but a timeout is
in place.
- fixed bug in events_socket() that complained about removing
a socket that was unknown, when indeed it had removed the socket
just before, only it was the last in the list
- fixed conncache's internal handle to carry the multi instance
(where the cache has one) so that operations on the closure handle
trigger event callbacks correctly.
- fixed conncache to not POLL_REMOVE a socket twice when a conneciton
was closed.

Closes #14561

show more ...

urldata: introduce `data->mid`, a unique identifier inside a multi

`data->id` is unique in *most* situations, but not in all. If a libcurl
application uses more than one connection cache

urldata: introduce `data->mid`, a unique identifier inside a multi

`data->id` is unique in *most* situations, but not in all. If a libcurl
application uses more than one connection cache, they will overlap. This
is a rare situations, but libcurl apps do crazy things. However, for
informative things, like tracing, `data->id` is superior, since it
assigns new ids in curl's serial curl_easy_perform() use.

Introduce `data->mid` which is a unique identifer inside one multi
instance, assigned on multi_add_handle() and cleared on
multi_remove_handle().

Use the `mid` in DoH operations and also in h2/h3 stream hashes.

Reported-by: 罗朝辉
Fixes #14414
Closes #14499

show more ...

conncache: connection shutdown, multi_socket handling

- implement the socket hash user/reader/writer processing also
for connections that are being shut down by the connection cache.

conncache: connection shutdown, multi_socket handling

- implement the socket hash user/reader/writer processing also
for connections that are being shut down by the connection cache.
- split out handling of current vs. last pollset socket event handling
into a function available in other code parts
- add `shutdown_poll` pollset to `connectdata` struct so that changes
in the pollset can be recorded during shutdown. (The internal handle
cannot keep it since it might be used for many connections)

Reported-by: calvin2021y on github
Fixes #14252
Closes #14257

show more ...

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period

Closes #14073

show more ...

lib: bump hash sizes to `size_t`

Follow-up to cc907e80a2498c0599253271a6f657f614b52a4e #13502
Cherry-picked from #13489
Closes #13601

lib: use multi instead of multi_easy for the active multi

- Use data->multi and not data->multi_easy to refer to the active multi.

The easy handle's active multi is always data->mul

lib: use multi instead of multi_easy for the active multi

- Use data->multi and not data->multi_easy to refer to the active multi.

The easy handle's active multi is always data->multi.

This is a follow up to 757dfdf which changed curl so that an easy handle
used with the easy interface and then multi interface cannot have two
different multi handles associated with it at the same time
(data->multi_easy from the easy interface and data->multi from the multi
interface).

Closes https://github.com/curl/curl/pull/12665

show more ...

lib: further send/upload handling polish

- Move all the "upload_done" handling to request.c

- add possibility to abort sending of a request
- add `Curl_req_done_sending()` f

lib: further send/upload handling polish

- Move all the "upload_done" handling to request.c

- add possibility to abort sending of a request
- add `Curl_req_done_sending()` for checks
- transfer.c: readwrite_upload() now clean

- removing data->state.ulbuf and data->req.upload_fromhere

- as well as data->req.upload_present
- set data->req.upload_done on having read all from
the client and completely flushed the send buffer

- tftp, remove setting of data->req.upload_fromhere

- serves no purpose as `upload_present` is not set
and the data itself is directly `sendto()` anyway

- smtp, make upload EOB conversion a client reader
- xfer_ulbuf addition

- add xfer_ulbuf for borrowing, similar to xfer_buf
- use in file upload
- use in c-hyper body sending

- h1-proxy, remove init of data->state.uilbuf that is never used
- smb, add own send_buf instead of using data->state.ulbuf

Closes #13010

show more ...

multi: add xfer_buf to multi handle

- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`

multi: add xfer_buf to multi handle

- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`
- used in transfer.c readwrite_data()

Closes #12805

show more ...

multiif.h: remove Curl_multi_dump declaration

Follow-up to d850eea2 which removed the Curl_multi_dump definition.

Closes https://github.com/curl/curl/pull/11946

multi: more efficient pollfd count for poll

- do not use separate pollfds for sockets that have POLLIN+POLLOUT

Closes #11792

http2: avoid too early connection re-use/multiplexing

HTTP/1 connections that are upgraded to HTTP/2 should not be picked up
for reuse and multiplexing by other handles until the 101 swi

http2: avoid too early connection re-use/multiplexing

HTTP/1 connections that are upgraded to HTTP/2 should not be picked up
for reuse and multiplexing by other handles until the 101 switching
process is completed.

Lots-of-debgging-by: Stefan Eissing
Reported-by: Richard W.M. Jones
Bug: https://curl.se/mail/lib-2023-07/0045.html
Closes #11557

show more ...

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- save

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205

show more ...

multi: use larger dns hash table for multi interface

Have curl_multi_init() use a much larger DNS hash table than used for
the easy interface to scale and perform better when used with _

multi: use larger dns hash table for multi interface

Have curl_multi_init() use a much larger DNS hash table than used for
the easy interface to scale and perform better when used with _many_
host names.

curl_share_init() sets an in-between size.

Inspired-by: Ivan Tsybulin
See #9340
Closes #9376

show more ...

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869

show more ...

conn: fix typo 'connnection' -> 'connection' in two function names

Closes #8759

multi: handle errors returned from socket/timer callbacks

The callbacks were partially documented to support this. Now the
behavior is documented and returning error from either of these

multi: handle errors returned from socket/timer callbacks

The callbacks were partially documented to support this. Now the
behavior is documented and returning error from either of these
callbacks will effectively kill all currently ongoing transfers.

Added test 530 to verify

Reported-by: Marcelo Juchem
Fixes #8083
Closes #8089

show more ...

lib: pass in 'struct Curl_easy *' to most functions

... in most cases instead of 'struct connectdata *' but in some cases in
addition to.

- We mostly operate on transfers and no

lib: pass in 'struct Curl_easy *' to most functions

... in most cases instead of 'struct connectdata *' but in some cases in
addition to.

- We mostly operate on transfers and not connections.

- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).

- This work clarifies and separates the transfers from the connections
better.

- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.

Closes #6425

show more ...

curl.se: new home

Closes #6172

url: make sure pushed streams get an allocated download buffer

Follow-up to c4e6968127e876b0

When a new transfer is created, as a resuly of an acknowledged push,
that transfer n

url: make sure pushed streams get an allocated download buffer

Follow-up to c4e6968127e876b0

When a new transfer is created, as a resuly of an acknowledged push,
that transfer needs a download buffer allocated.

Closes #5590

show more ...

conncache: various concept cleanups

More connection cache accesses are protected by locks.

CONNCACHE_* is a beter prefix for the connection cache lock macros.

Curl_attach_c

conncache: various concept cleanups

More connection cache accesses are protected by locks.

CONNCACHE_* is a beter prefix for the connection cache lock macros.

Curl_attach_connnection: now called as soon as there's a connection
struct available and before the connection is added to the connection
cache.

Curl_disconnect: now assumes that the connection is already removed from
the connection cache.

Ref: #4915
Closes #5009

show more ...

ConnectionExists: respect the max_concurrent_streams limits

A regression made the code use 'multiplexed' as a boolean instead of the
counter it is intended to be. This made curl try to "

ConnectionExists: respect the max_concurrent_streams limits

A regression made the code use 'multiplexed' as a boolean instead of the
counter it is intended to be. This made curl try to "over-populate"
connections with new streams.

This regression came with 41fcdf71a1, shipped in curl 7.65.0.

Also, respect the CURLMOPT_MAX_CONCURRENT_STREAMS value in the same
check.

Reported-by: Kunal Ekawde
Fixes #4779
Closes #4784

show more ...

CURLMOPT_MAX_CONCURRENT_STREAMS: new setopt

Closes #4410

curl_multi_poll: a sister to curl_multi_wait() that waits more

Repeatedly we see problems where using curl_multi_wait() is difficult or
just awkward because if it has no file descriptor

curl_multi_poll: a sister to curl_multi_wait() that waits more

Repeatedly we see problems where using curl_multi_wait() is difficult or
just awkward because if it has no file descriptor to wait for
internally, it returns immediately and leaves it to the caller to wait
for a small amount of time in order to avoid occasional busy-looping.

This is often missed or misunderstood, leading to underperforming
applications.

This change introduces curl_multi_poll() as a replacement drop-in
function that accepts the exact same set of arguments. This function
works identically to curl_multi_wait() - EXCEPT - for the case when
there's nothing to wait for internally, as then this function will by
itself wait for a "suitable" short time before it returns. This
effectiely avoids all risks of busy-looping and should also make it less
likely that apps "over-wait".

This also changes the curl tool to use this funtion internally when
doing parallel transfers and changes curl_easy_perform() to use it
internally.

Closes #4163

show more ...

timediff: make it 64 bit (if possible) even with 32 bit time_t

... to make it hold microseconds too.

Fixes #4165
Closes #4168