#
1be704e1 |
| 23-Aug-2024 |
Stefan Eissing |
cpool: rename "connection cache/conncache" to "Connection Pools/cpool" This is a better match for what they do and the general "cpool" var/function prefix works well. The pool n
cpool: rename "connection cache/conncache" to "Connection Pools/cpool" This is a better match for what they do and the general "cpool" var/function prefix works well. The pool now handles very long hostnames correctly. The following changes have been made: * 'struct connectdata', e.g. connections, keep new members named `destination` and ' destination_len' that fully specifies interface+port+hostname of where the connection is going to. This is used in the pool for "bundling" of connections with the same destination. There is no limit on the length any more. * Locking: all locks are done inside conncache.c when calling into the pool and released on return. This eliminates hazards of the callers keeping track. * 'struct connectbundle' is now internal to the pool. It is no longer referenced by a connection. * 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters no longer need to set it. Instead, the multi checks on leaving MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now multiplexed and new, e.g. not conn->bits.reuse. In that case the processing of pending handles is triggered. * The pool's init is provided with a callback to invoke on all connections being discarded. This allows the cleanups in `Curl_disconnect` to run, wherever it is decided to retire a connection. * Several pool operations can now be fully done with one call. Pruning dead connections, upkeep and checks on pool limits can now directly discard connections and need no longer return those to the caller for doing that (as we have now the callback described above). * Finding a connection for reuse is now done via `Curl_cpool_find()` and the caller provides callbacks to evaluate the connection candidates. * The 'Curl_cpool_check_limits()' now directly uses the max values that may be set in the transfer's multi. No need to pass them around. Curl_multi_max_host_connections() and Curl_multi_max_total_connections() are gone. * Add method 'Curl_node_llist()' to get the llist a node is in. Used in cpool to verify connection are indeed in the list (or not in any list) as they need to. I left the conncache.[ch] as is for now and also did not touch the documentation. If we update that outside the feature window, we can do this in a separate PR. Multi-thread safety is not achieved by this PR, but since more details on how pools operate are now "internal" it is a better starting point to go for this in the future. Closes #14662
show more ...
|
#
a58b50fc |
| 15-Aug-2024 |
Stefan Eissing |
transfer: Curl_sendrecv() and event related improvements - Renames Curl_readwrite() to Curl_sendrecv() to reflect that it is mainly about talking to the server, not reads or writes to
transfer: Curl_sendrecv() and event related improvements - Renames Curl_readwrite() to Curl_sendrecv() to reflect that it is mainly about talking to the server, not reads or writes to the client. Add a `nowp` parameter since the single caller already has this. - Curl_sendrecv() now runs all possible operations whenever it is called and either it had been polling sockets or the 'select_bits' are set. POLL_IN/POLL_OUT are not always directly related to send/recv operations. Filters like HTTP/2, QUIC or TLS may monitor reverse directions. If a transfer does not want to send (KEEP_SEND), it will not do so, as before. Same for receives. - Curl_update_timer() now checks the absolute timestamp of an expiry and the last/new timeout to determine if the application needs to stop/start/restart its timer. This fixes edge cases where updates did not happen as they should have. - improved --test-event curl_easy_perform() simulation to handle situations where no sockets are registered but a timeout is in place. - fixed bug in events_socket() that complained about removing a socket that was unknown, when indeed it had removed the socket just before, only it was the last in the list - fixed conncache's internal handle to carry the multi instance (where the cache has one) so that operations on the closure handle trigger event callbacks correctly. - fixed conncache to not POLL_REMOVE a socket twice when a conneciton was closed. Closes #14561
show more ...
|
#
22d292b3 |
| 12-Aug-2024 |
Stefan Eissing |
urldata: introduce `data->mid`, a unique identifier inside a multi `data->id` is unique in *most* situations, but not in all. If a libcurl application uses more than one connection cache
urldata: introduce `data->mid`, a unique identifier inside a multi `data->id` is unique in *most* situations, but not in all. If a libcurl application uses more than one connection cache, they will overlap. This is a rare situations, but libcurl apps do crazy things. However, for informative things, like tracing, `data->id` is superior, since it assigns new ids in curl's serial curl_easy_perform() use. Introduce `data->mid` which is a unique identifer inside one multi instance, assigned on multi_add_handle() and cleared on multi_remove_handle(). Use the `mid` in DoH operations and also in h2/h3 stream hashes. Reported-by: 罗朝辉 Fixes #14414 Closes #14499
show more ...
|
#
ba235ab2 |
| 10-Aug-2024 |
Daniel Stenberg |
llist: remove direct struct accesses, use only functions - Turned them all into functions to also do asserts etc. - The llist related structs got all their fields renamed in order t
llist: remove direct struct accesses, use only functions - Turned them all into functions to also do asserts etc. - The llist related structs got all their fields renamed in order to make sure no existing code remains using direct access. - Each list node struct now points back to the list it "lives in", so Curl_node_remove() no longer needs the list pointer. - Rename the node struct and some of the access functions. - Added lots of ASSERTs to verify API being used correctly - Fix some cases of API misuse Add docs/LLIST.md documenting the internal linked list API. Closes #14485
show more ...
|
#
2c15ee4b |
| 09-Aug-2024 |
Daniel Stenberg |
multi: make the "general" list of easy handles a Curl_llist Instead of having an especially "unique" linked list handler for the main list of easy handles within the multi handle, this n
multi: make the "general" list of easy handles a Curl_llist Instead of having an especially "unique" linked list handler for the main list of easy handles within the multi handle, this now uses a regular Curl_llist for this as well. With this change, it is also clearer that every easy handle added to a multi handle belongs to one and only one out of three different lists: process - the general one for normal transfer processing pending - queued up waiting to get a connection (MSTATE_PENDING) msgsent - transfer completed (MSTATE_MSGSENT) An easy handle must therefore be removed from the current list before it gets added to another. Closes #14474
show more ...
|
#
f81f351b |
| 02-Aug-2024 |
Viktor Szakats |
tidy-up: OS names Use these words and casing more consistently across text, comments and one curl tool output: AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
tidy-up: OS names Use these words and casing more consistently across text, comments and one curl tool output: AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux, macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode, WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock. Mostly OS names and a few more. Also a couple of other minor text fixups. Closes #14360
show more ...
|
#
c074ba64 |
| 01-Jul-2024 |
Daniel Stenberg |
code: language cleanup in comments Based on the standards and guidelines we use for our documentation. - expand contractions (they're => they are etc) - host name = > hostname
code: language cleanup in comments Based on the standards and guidelines we use for our documentation. - expand contractions (they're => they are etc) - host name = > hostname - file name => filename - user name = username - man page => manpage - run-time => runtime - set-up => setup - back-end => backend - a HTTP => an HTTP - Two spaces after a period => one space after period Closes #14073
show more ...
|
#
c9b95c0b |
| 19-Jun-2024 |
Stefan Eissing |
lib: graceful connection shutdown When libcurl discards a connection there are two phases this may go through: "shutdown" and "closing". If a connection is aborted, the shutdown phas
lib: graceful connection shutdown When libcurl discards a connection there are two phases this may go through: "shutdown" and "closing". If a connection is aborted, the shutdown phase is skipped and it is closed right away. The connection filters attached to the connection implement the phases in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a `shutdown` flags next to `connected` to keep track of the shutdown operation. Filters are shut down from top to bottom. If a filter is not connected, its shutdown is skipped. Notable filters that *do* something during shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends its close notify and expects to receive a close notify from the server. As sends and receives may EAGAIN on the network, a shutdown is often not successful right away and needs to poll the connection's socket(s). To facilitate this, such connections are placed on a new shutdown list inside the connection cache. Since managing this list requires the cooperation of a multi handle, only the connection cache belonging to a multi handle is used. If a connection was in another cache when being discarded, it is removed there and added to the multi's cache. If no multi handle is available at that time, the connection is shutdown and closed in a one-time, best-effort attempt. When a multi handle is destroyed, all connection still on the shutdown list are discarded with a final shutdown attempt and close. In curl debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be set to make this graceful with a timeout in milliseconds given by the variable. The shutdown list is limited to the max number of connections configured for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the limit is reached, the oldest connection on the shutdown list is discarded. - In multi_wait() and multi_waitfds(), collect all connection caches involved (each transfer might carry its own) into a temporary list. Let each connection cache on the list contribute sockets and POLLIN/OUT events it's connections are waiting for. - in multi_perform() collect the connection caches the same way and let them peform their maintenance. This will make another non-blocking attempt to shutdown all connections on its shutdown list. - for event based multis (multi->socket_cb set), add the sockets and their poll events via the callback. When `multi_socket()` is invoked for a socket not known by an active transfer, forward this to the multi's cache for processing. On closing a connection, remove its socket(s) via the callback. TLS connection filters MUST NOT send close nofity messages in their `do_close()` implementation. The reason is that a TLS close notify signals a success. When a connection is aborted and skips its shutdown phase, the server needs to see a missing close notify to detect something has gone wrong. A graceful shutdown of FTP's data connection is performed implicitly before regarding the upload/download as complete and continuing on the control connection. For FTP without TLS, there is just the socket close happening. But with TLS, the sent/received close notify signals that the transfer is complete and healthy. Servers like `vsftpd` verify that and reject uploads without a TLS close notify. - added test_19_* for shutdown related tests - test_19_01 and test_19_02 test for TCP RST packets which happen without a graceful shutdown and should no longer appear otherwise. - add test_19_03 for handling shutdowns by the server - add test_19_04 for handling shutdowns by curl - add test_19_05 for event based shutdowny by server - add test_30_06/07 and test_31_06/07 for shutdown checks on FTP up- and downloads. Closes #13976
show more ...
|
#
23fe1a52 |
| 01-Jun-2024 |
Andy Pan |
socketpair: add `eventfd` and use `SOCK_NONBLOCK` for `socketpair()` Currently, we use `pipe` for `wakeup_create`, which requires ***two*** file descriptors. Furthermore, given its compl
socketpair: add `eventfd` and use `SOCK_NONBLOCK` for `socketpair()` Currently, we use `pipe` for `wakeup_create`, which requires ***two*** file descriptors. Furthermore, given its complexity inside, `pipe` is a bit heavyweight for just a simple event wait/notify mechanism. `eventfd` would be a more suitable solution for this kind of scenario, kernel also advocates for developers to use `eventfd` instead of `pipe` in some simple use cases: Applications can use an eventfd file descriptor instead of a pipe (see pipe(2) in all cases where a pipe is used simply to signal events. The kernel overhead of an eventfd file descriptor is much lower than that of a pipe, and only one file descriptor is required (versus the two required for a pipe). This change adds the new backend of `eventfd` for `wakeup_create` and uses it where available, eliminating the overhead of `pipe`. Also, it optimizes the `wakeup_create` to eliminate the system calls that make file descriptors non-blocking by moving the logic of setting non-blocking flags on file descriptors to `socketpair.c` and using `SOCK_NONBLOCK` for `socketpair(2)`, `EFD_NONBLOCK` for `eventfd(2)`. Ref: https://man7.org/linux/man-pages/man7/pipe.7.html https://man7.org/linux/man-pages/man2/eventfd.2.html https://man7.org/linux/man-pages/man2/socketpair.2.html https://www.gnu.org/software/gnulib/manual/html_node/eventfd.html Closes #13874
show more ...
|
#
e101a7a8 |
| 11-Apr-2024 |
Stefan Eissing |
multi: add multi->proto_hash, a key-value store for protocol data - add `Curl_hash_add2()` that passes a destructor function for the element added. Call element destructor instead of h
multi: add multi->proto_hash, a key-value store for protocol data - add `Curl_hash_add2()` that passes a destructor function for the element added. Call element destructor instead of hash destructor if present. - multi: add `proto_hash` for protocol related information, remove `struct multi_ssl_backend_data`. - openssl: use multi->proto_hash to keep x509 shared store - schannel: use multi->proto_hash to keep x509 shared store - vtls: remove Curl_free_multi_ssl_backend_data() and its equivalents in the TLS backends Closes #13345
show more ...
|
#
fd0d2ed7 |
| 13-May-2024 |
Andrew |
wakeup_create: use FD_CLOEXEC/SOCK_CLOEXEC for `pipe()`/`socketpair()` Fixes #13618 Closes #13625
|
#
be659030 |
| 05-Apr-2024 |
Daniel Stenberg |
multi: introduce SETUP state for better timeouts Since we can go to the CONNECT state from PENDING, potentially multiple times for a single transfer, this change introdues a SETUP state
multi: introduce SETUP state for better timeouts Since we can go to the CONNECT state from PENDING, potentially multiple times for a single transfer, this change introdues a SETUP state that happens before CONNECT when doing a new transfer. Now, doing a redirect on a handle goes back to SETUP (not CONNECT like before) and we initilize the connect timeout etc in SETUP. Previously, we would do it in CONNECT but that would make it unreliable in cases where a transfer goes in and out between CONNECT and PENDING multiple times. SETUP is transient, so the handle never actually stays in that state. Additionally: take care of timeouts of PENDING transfers in curl_multi_perform() Ref: #13227 Closes #13371
show more ...
|
#
c296abd4 |
| 10-Apr-2024 |
Stefan Eissing |
llist: add Curl_llist_append() - use for better readability in all places where the "insert_next" actually performs an append to the list - add some tests in unit1300 Clos
llist: add Curl_llist_append() - use for better readability in all places where the "insert_next" actually performs an append to the list - add some tests in unit1300 Closes #13336
show more ...
|
#
e3905de8 |
| 28-Feb-2024 |
Stefan Eissing |
lib: further send/upload handling polish - Move all the "upload_done" handling to request.c - add possibility to abort sending of a request - add `Curl_req_done_sending()` f
lib: further send/upload handling polish - Move all the "upload_done" handling to request.c - add possibility to abort sending of a request - add `Curl_req_done_sending()` for checks - transfer.c: readwrite_upload() now clean - removing data->state.ulbuf and data->req.upload_fromhere - as well as data->req.upload_present - set data->req.upload_done on having read all from the client and completely flushed the send buffer - tftp, remove setting of data->req.upload_fromhere - serves no purpose as `upload_present` is not set and the data itself is directly `sendto()` anyway - smtp, make upload EOB conversion a client reader - xfer_ulbuf addition - add xfer_ulbuf for borrowing, similar to xfer_buf - use in file upload - use in c-hyper body sending - h1-proxy, remove init of data->state.uilbuf that is never used - smb, add own send_buf instead of using data->state.ulbuf Closes #13010
show more ...
|
#
476adfea |
| 26-Jan-2024 |
Stefan Eissing |
multi: add xfer_buf to multi handle - can be borrowed by transfer during recv-write operation - needs to be released before borrowing again - adjustis size to `data->set.buffer_size`
multi: add xfer_buf to multi handle - can be borrowed by transfer during recv-write operation - needs to be released before borrowing again - adjustis size to `data->set.buffer_size` - used in transfer.c readwrite_data() Closes #12805
show more ...
|
#
2d06eebf |
| 21-Nov-2023 |
Daniel Stenberg |
urldata: make maxconnects a 32 bit value "2^32 idle connections ought to be enough for anybody" Closes #12375
|
#
843b3baa |
| 26-Mar-2023 |
Daniel Stenberg |
multi: remove PENDING + MSGSENT handles from the main linked list As they are not driving transfers or any socket activity, the main loop does not need to iterate over these handles. A p
multi: remove PENDING + MSGSENT handles from the main linked list As they are not driving transfers or any socket activity, the main loop does not need to iterate over these handles. A performance improvement. They are instead only held in their own separate lists. 'data->multi' is kept a pointer to the multi handle as long as the easy handle is actually part of it even when the handle is moved to the pending/msgsent lists. It needs to know which multi handle it belongs to, if for example curl_easy_cleanup() is called before the handle is removed from the multi handle. Alll 'data->multi' pointers of handles still part of the multi handle gets cleared by curl_multi_cleanup() which "orphans" all previously attached easy handles. This is take 2. The first version was reverted for the 8.0.1 release. Assisted-by: Stefan Eissing Closes #10801
show more ...
|
#
c2df780a |
| 20-Mar-2023 |
Daniel Stenberg |
Revert "multi: remove PENDING + MSGSENT handles from the main linked list" This reverts commit f6d6f3ce01e377932f1ce7c24ee34d45a36950b8. The commits caused issues in the 8.0.0 relea
Revert "multi: remove PENDING + MSGSENT handles from the main linked list" This reverts commit f6d6f3ce01e377932f1ce7c24ee34d45a36950b8. The commits caused issues in the 8.0.0 release. Needs a retake. Reported-by: Kamil Dudka Closes #10795
show more ...
|
#
f6d6f3ce |
| 14-Mar-2023 |
Daniel Stenberg |
multi: remove PENDING + MSGSENT handles from the main linked list As they are not driving transfers or any socket activity, the main loop does not need to iterate over these handles. A p
multi: remove PENDING + MSGSENT handles from the main linked list As they are not driving transfers or any socket activity, the main loop does not need to iterate over these handles. A performance improvement. They are instead only held in their own separate lists. Assisted-by: Stefan Eissing Ref: #10743 Closes #10762
show more ...
|
#
7d3b167f |
| 11-Jan-2023 |
Daniel Stenberg |
curl: output warning at --verbose output for debug-enabled version + a libcurl warning in the debug output Assisted-by: Jay Satiro Ref: https://curl.se/mail/lib-2023-01/003
curl: output warning at --verbose output for debug-enabled version + a libcurl warning in the debug output Assisted-by: Jay Satiro Ref: https://curl.se/mail/lib-2023-01/0039.html Closes #10278
show more ...
|
#
2bc1d775 |
| 02-Jan-2023 |
Daniel Stenberg |
copyright: update all copyright lines and remove year ranges - they are mostly pointless in all major jurisdictions - many big corporations and projects already don't use them - save
copyright: update all copyright lines and remove year ranges - they are mostly pointless in all major jurisdictions - many big corporations and projects already don't use them - saves us from pointless churn - git keeps history for us - the year range is kept in COPYING checksrc is updated to allow non-year using copyright statements Closes #10205
show more ...
|
#
1c007960 |
| 29-Dec-2022 |
Daniel Stenberg |
multihandle: turn bool struct fields into bits Closes #10179
|
#
3c16697e |
| 28-Sep-2022 |
Michael Drake |
openssl: reduce CA certificate bundle reparsing by caching Closes #9620
|
#
46f3fe0e |
| 21-Sep-2022 |
Daniel Stenberg |
hostip: lazily wait to figure out if IPv6 works until needed The check may take many milliseconds, so now it is performed once the value is first needed. Also, this change makes sure tha
hostip: lazily wait to figure out if IPv6 works until needed The check may take many milliseconds, so now it is performed once the value is first needed. Also, this change makes sure that the value is not used if the resolve is set to be IPv4-only. Closes #9553
show more ...
|
#
ad9bc597 |
| 17-May-2022 |
max.mehl |
copyright: make repository REUSE compliant Add licensing and copyright information for all files in this repository. This either happens in the file itself as a comment header or in the
copyright: make repository REUSE compliant Add licensing and copyright information for all files in this repository. This either happens in the file itself as a comment header or in the file `.reuse/dep5`. This commit also adds a Github workflow to check pull requests and adapts copyright.pl to the changes. Closes #8869
show more ...
|