#
1be704e1 |
| 23-Aug-2024 |
Stefan Eissing |
cpool: rename "connection cache/conncache" to "Connection Pools/cpool" This is a better match for what they do and the general "cpool" var/function prefix works well. The pool n
cpool: rename "connection cache/conncache" to "Connection Pools/cpool" This is a better match for what they do and the general "cpool" var/function prefix works well. The pool now handles very long hostnames correctly. The following changes have been made: * 'struct connectdata', e.g. connections, keep new members named `destination` and ' destination_len' that fully specifies interface+port+hostname of where the connection is going to. This is used in the pool for "bundling" of connections with the same destination. There is no limit on the length any more. * Locking: all locks are done inside conncache.c when calling into the pool and released on return. This eliminates hazards of the callers keeping track. * 'struct connectbundle' is now internal to the pool. It is no longer referenced by a connection. * 'bundle->multiuse' no longer exists. HTTP/2 and 3 and TLS filters no longer need to set it. Instead, the multi checks on leaving MSTATE_CONNECT or MSTATE_CONNECTING if the connection is now multiplexed and new, e.g. not conn->bits.reuse. In that case the processing of pending handles is triggered. * The pool's init is provided with a callback to invoke on all connections being discarded. This allows the cleanups in `Curl_disconnect` to run, wherever it is decided to retire a connection. * Several pool operations can now be fully done with one call. Pruning dead connections, upkeep and checks on pool limits can now directly discard connections and need no longer return those to the caller for doing that (as we have now the callback described above). * Finding a connection for reuse is now done via `Curl_cpool_find()` and the caller provides callbacks to evaluate the connection candidates. * The 'Curl_cpool_check_limits()' now directly uses the max values that may be set in the transfer's multi. No need to pass them around. Curl_multi_max_host_connections() and Curl_multi_max_total_connections() are gone. * Add method 'Curl_node_llist()' to get the llist a node is in. Used in cpool to verify connection are indeed in the list (or not in any list) as they need to. I left the conncache.[ch] as is for now and also did not touch the documentation. If we update that outside the feature window, we can do this in a separate PR. Multi-thread safety is not achieved by this PR, but since more details on how pools operate are now "internal" it is a better starting point to go for this in the future. Closes #14662
show more ...
|
#
c9b95c0b |
| 19-Jun-2024 |
Stefan Eissing |
lib: graceful connection shutdown When libcurl discards a connection there are two phases this may go through: "shutdown" and "closing". If a connection is aborted, the shutdown phas
lib: graceful connection shutdown When libcurl discards a connection there are two phases this may go through: "shutdown" and "closing". If a connection is aborted, the shutdown phase is skipped and it is closed right away. The connection filters attached to the connection implement the phases in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a `shutdown` flags next to `connected` to keep track of the shutdown operation. Filters are shut down from top to bottom. If a filter is not connected, its shutdown is skipped. Notable filters that *do* something during shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends its close notify and expects to receive a close notify from the server. As sends and receives may EAGAIN on the network, a shutdown is often not successful right away and needs to poll the connection's socket(s). To facilitate this, such connections are placed on a new shutdown list inside the connection cache. Since managing this list requires the cooperation of a multi handle, only the connection cache belonging to a multi handle is used. If a connection was in another cache when being discarded, it is removed there and added to the multi's cache. If no multi handle is available at that time, the connection is shutdown and closed in a one-time, best-effort attempt. When a multi handle is destroyed, all connection still on the shutdown list are discarded with a final shutdown attempt and close. In curl debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be set to make this graceful with a timeout in milliseconds given by the variable. The shutdown list is limited to the max number of connections configured for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the limit is reached, the oldest connection on the shutdown list is discarded. - In multi_wait() and multi_waitfds(), collect all connection caches involved (each transfer might carry its own) into a temporary list. Let each connection cache on the list contribute sockets and POLLIN/OUT events it's connections are waiting for. - in multi_perform() collect the connection caches the same way and let them peform their maintenance. This will make another non-blocking attempt to shutdown all connections on its shutdown list. - for event based multis (multi->socket_cb set), add the sockets and their poll events via the callback. When `multi_socket()` is invoked for a socket not known by an active transfer, forward this to the multi's cache for processing. On closing a connection, remove its socket(s) via the callback. TLS connection filters MUST NOT send close nofity messages in their `do_close()` implementation. The reason is that a TLS close notify signals a success. When a connection is aborted and skips its shutdown phase, the server needs to see a missing close notify to detect something has gone wrong. A graceful shutdown of FTP's data connection is performed implicitly before regarding the upload/download as complete and continuing on the control connection. For FTP without TLS, there is just the socket close happening. But with TLS, the sent/received close notify signals that the transfer is complete and healthy. Servers like `vsftpd` verify that and reject uploads without a TLS close notify. - added test_19_* for shutdown related tests - test_19_01 and test_19_02 test for TCP RST packets which happen without a graceful shutdown and should no longer appear otherwise. - add test_19_03 for handling shutdowns by the server - add test_19_04 for handling shutdowns by curl - add test_19_05 for event based shutdowny by server - add test_30_06/07 and test_31_06/07 for shutdown checks on FTP up- and downloads. Closes #13976
show more ...
|
#
fcef00db |
| 08-Mar-2024 |
Stefan Eissing |
lib: keep conn IP information together new struct ip_quadruple for holding local/remote addr+port - used in data->info and conn and cf-socket.c - copy back and forth complete st
lib: keep conn IP information together new struct ip_quadruple for holding local/remote addr+port - used in data->info and conn and cf-socket.c - copy back and forth complete struct - add 'secondary' to conn - use secondary in reporting success for ftp 2nd connection Reported-by: DasKutti on github Fixes #13084 Closes #13090
show more ...
|
#
59298221 |
| 14-Feb-2024 |
Stefan Eissing |
lib: send rework Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify when and at what level they operate - sen
lib: send rework Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify when and at what level they operate - send/recv of transfer related data is now done via `Curl_xfer_send()/Curl_xfer_recv()` which no longer has socket/socketindex as parameter. It decides on the transfer setup of `conn->sockfd` and `conn->writesockfd` on which connection filter chain to operate. - send/recv on a specific connection filter chain is done via `Curl_conn_send()/Curl_conn_recv()` which get the socket index as parameter. - rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming consistency - clarify that the special CURLE_AGAIN handling to return `CURLE_OK` with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is returned by all other send() variants. SingleRequest reshuffling - move functions into request.[ch] - differentiate between reset and free - add Curl_req_done() to perform last actions - add a send `bufq` to SingleRequest for future use in keeping upload data Closes #12963
show more ...
|
#
39547ae6 |
| 27-Oct-2023 |
Stefan Eissing |
url: protocol handler lookup tidy-up - rename lookup to what it does - use ARRAYSIZE instead of NULL check for end - offer alternate lookup for 0-terminated strings Closes #
url: protocol handler lookup tidy-up - rename lookup to what it does - use ARRAYSIZE instead of NULL check for end - offer alternate lookup for 0-terminated strings Closes #12216
show more ...
|
#
8e85764b |
| 22-May-2023 |
Daniel Stenberg |
lib: remove unused functions, make single-use static Closes #11174
|
#
2bc1d775 |
| 02-Jan-2023 |
Daniel Stenberg |
copyright: update all copyright lines and remove year ranges - they are mostly pointless in all major jurisdictions - many big corporations and projects already don't use them - save
copyright: update all copyright lines and remove year ranges - they are mostly pointless in all major jurisdictions - many big corporations and projects already don't use them - saves us from pointless churn - git keeps history for us - the year range is kept in COPYING checksrc is updated to allow non-year using copyright statements Closes #10205
show more ...
|
#
71b7e016 |
| 30-Dec-2022 |
Stefan Eissing |
lib: connect/h2/h3 refactor Refactoring of connection setup and happy eyeballing. Move nghttp2. ngtcp2, quiche and msh3 into connection filters. - eyeballing cfilter that uses
lib: connect/h2/h3 refactor Refactoring of connection setup and happy eyeballing. Move nghttp2. ngtcp2, quiche and msh3 into connection filters. - eyeballing cfilter that uses sub-filters for performing parallel connects - socket cfilter for all transport types, including QUIC - QUIC implementations in cfilter, can now participate in eyeballing - connection setup is more dynamic in order to adapt to what filter did really connect. Relevant to see if a SSL filter needs to be added or if SSL has already been provided - HTTP/3 test cases similar to HTTP/2 - multiuse of parallel transfers for HTTP/3, tested for ngtcp2 and quiche - Fix for data attach/detach in VTLS filters that could lead to crashes during parallel transfers. - Eliminating setup() methods in cfilters, no longer needed. - Improving Curl_conn_is_alive() to replace Curl_connalive() and integrated ssl alive checks into cfilter. - Adding CF_CNTRL_CONN_INFO_UPDATE to tell filters to update connection into and persist it at the easy handle. - Several more cfilter related cleanups and moves: - stream_weigth and dependency info is now wrapped in struct Curl_data_priority - Curl_data_priority members depend is available in HTTP2|HTTP3 - Curl_data_priority members depend on NGHTTP2 support - handling init/reset/cleanup of priority part of url.c - data->state.priority same struct, but shallow copy for compares only - PROTOPT_STREAM has been removed - Curl_conn_is_mulitplex() now available to check on capability - Adding query method to connection filters. - ngtcp2+quiche: implementing query for max concurrent transfers. - Adding is_alive and keep_alive cfilter methods. Adding DATA_SETUP event. - setting keepalive timestamp on connect - DATA_SETUP is called after the connection has been completely setup (but may not connected yet) to allow filters to initialize data members they use. - there is no socket to be had with msh3, it is unclear how select shall work - manual test via "curl --http3 https://curl.se" fail with "empty reply from server". - Various socket/conn related cleanups: - Curl_socket is now Curl_socket_open and in cf-socket.c - Curl_closesocket is now Curl_socket_close and in cf-socket.c - Curl_ssl_use has been replaced with Cur_conn_is_ssl - Curl_conn_tcp_accepted_set has been split into Curl_conn_tcp_listen_set and Curl_conn_tcp_accepted_set with a clearer purpose Closes #10141
show more ...
|
#
cc0aaf6e |
| 14-Dec-2022 |
Daniel Stenberg |
idn: rename the files to idn.[ch] and hold all IDN functions Closes #10094
|
#
dafdb20a |
| 11-Nov-2022 |
Stefan Eissing |
lib: connection filters (cfilter) addition to curl: - general construct/destroy in connectdata - default implementations of callback functions - connect: cfilters for connect and
lib: connection filters (cfilter) addition to curl: - general construct/destroy in connectdata - default implementations of callback functions - connect: cfilters for connect and accept - socks: cfilter for socks proxying - http_proxy: cfilter for http proxy tunneling - vtls: cfilters for primary and proxy ssl - change in general handling of data/conn - Curl_cfilter_setup() sets up filter chain based on data settings, if none are installed by the protocol handler setup - Curl_cfilter_connect() boot straps filters into `connected` status, used by handlers and multi to reach further stages - Curl_cfilter_is_connected() to check if a conn is connected, e.g. all filters have done their work - Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE indicators for multi select to work - Curl_cfilter_data_pending() asks filters if the have incoming data pending for recv - Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks installed in conn->recv/conn->send for io handling - Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters and addition/removal of a `data` from their connection - adding vtl functions to prevent use of Curl_ssl globals directly in other parts of the code. Reviewed-by: Daniel Stenberg Closes #9855
show more ...
|
#
9d513290 |
| 15-Sep-2022 |
Patrick Monnerat |
setopt: use the handler table for protocol name to number conversions This also returns error CURLE_UNSUPPORTED_PROTOCOL rather than CURLE_BAD_FUNCTION_ARGUMENT when a listed protocol na
setopt: use the handler table for protocol name to number conversions This also returns error CURLE_UNSUPPORTED_PROTOCOL rather than CURLE_BAD_FUNCTION_ARGUMENT when a listed protocol name is not found. A new schemelen parameter is added to Curl_builtin_scheme() to support this extended use. Note that disabled protocols are not recognized anymore. Tests adapted accordingly. Closes #9472
show more ...
|
#
ad9bc597 |
| 17-May-2022 |
max.mehl |
copyright: make repository REUSE compliant Add licensing and copyright information for all files in this repository. This either happens in the file itself as a comment header or in the
copyright: make repository REUSE compliant Add licensing and copyright information for all files in this repository. This either happens in the file itself as a comment header or in the file `.reuse/dep5`. This commit also adds a Github workflow to check pull requests and adapts copyright.pl to the changes. Closes #8869
show more ...
|
#
e74a6b7b |
| 19-Jan-2022 |
Daniel Stenberg |
url: make Curl_disconnect return void 1. The function would only ever return CURLE_OK anyway 2. Only one caller actually used the return code 3. Most callers did (void)Curl_discon
url: make Curl_disconnect return void 1. The function would only ever return CURLE_OK anyway 2. Only one caller actually used the return code 3. Most callers did (void)Curl_disconnect() Closes #8303
show more ...
|
Revision tags: curl-7_76_1, curl-7_76_0, curl-7_75_0 |
|
#
560fc170 |
| 24-Jan-2021 |
Daniel Stenberg |
url: reduce conn->data references ... there are a few left but let's keep them to last Closes #6512
|
#
d0688dcb |
| 19-Jan-2021 |
Daniel Stenberg |
socks: use the download buffer instead The SOCKS code now uses the generic download buffer for temporary storage during the connection procedure, instead of having its own private 60
socks: use the download buffer instead The SOCKS code now uses the generic download buffer for temporary storage during the connection procedure, instead of having its own private 600 byte buffer that adds to the connectdata struct size. This works fine because this point the buffer is allocated but is not use for download yet since the connection hasn't completed. This reduces the connection struct size by 22% on a 64bit arch! The SOCKS buffer needs to be at least 600 bytes, and the download buffer is guaranteed to never be smaller than 1000 bytes. Closes #6491
show more ...
|
#
215db086 |
| 08-Jan-2021 |
Daniel Stenberg |
lib: pass in 'struct Curl_easy *' to most functions ... in most cases instead of 'struct connectdata *' but in some cases in addition to. - We mostly operate on transfers and no
lib: pass in 'struct Curl_easy *' to most functions ... in most cases instead of 'struct connectdata *' but in some cases in addition to. - We mostly operate on transfers and not connections. - We need the transfer handle to log, store data and more. Everything in libcurl is driven by a transfer (the CURL * in the public API). - This work clarifies and separates the transfers from the connections better. - We should avoid "conn->data". Since individual connections can be used by many transfers when multiplexing, making sure that conn->data points to the current and correct transfer at all times is difficult and has been notoriously error-prone over the years. The goal is to ultimately remove the conn->data pointer for this reason. Closes #6425
show more ...
|
Revision tags: curl-7_74_0 |
|
#
4d2f8006 |
| 04-Nov-2020 |
Daniel Stenberg |
curl.se: new home Closes #6172
|
Revision tags: curl-7_73_0, tiny-curl-7_72_0, curl-7_72_0, curl-7_71_1, curl-7_71_0 |
|
#
f3d501dc |
| 27-May-2020 |
Daniel Stenberg |
build: disable more code/data when built without proxy support Added build to travis to verify Closes #5466
|
#
66b07757 |
| 13-May-2020 |
Daniel Stenberg |
checksrc: enhance the ASTERISKSPACE and update code accordingly Fine: "struct hello *world" Not fine: "struct hello* world" (and variations) Closes #5386
|
Revision tags: curl-7_70_0, curl-7_69_1, curl-7_69_0 |
|
#
9352a407 |
| 07-Feb-2020 |
Steve Holme |
url: Make the IDN conversion functions available to others
|
Revision tags: curl-7_68_0, curl-7_67_0 |
|
#
dcd7e37c |
| 28-Oct-2019 |
Daniel Stenberg |
url: make Curl_close() NULLify the pointer too This is the common pattern used in the code and by a unified approach we avoid mistakes. Closes #4534
|
Revision tags: curl-7_66_0 |
|
#
5050edb1 |
| 02-Sep-2019 |
Daniel Stenberg |
cleanup: move functions out of url.c and make them static Closes #4289
|
#
a55faf33 |
| 30-Jul-2019 |
Daniel Stenberg |
cleanup: remove the 'numsocks' argument used in many places It was used (intended) to pass in the size of the 'socks' array that is also passed to these functions, but was rarely actuall
cleanup: remove the 'numsocks' argument used in many places It was used (intended) to pass in the size of the 'socks' array that is also passed to these functions, but was rarely actually checked/used and the array is defined to a fixed size of MAX_SOCKSPEREASYHANDLE entries that should be used instead. Closes #4169
show more ...
|
Revision tags: curl-7_65_3, curl-7_65_2 |
|
#
11d7fe11 |
| 10-Jul-2019 |
Daniel Gustafsson |
headers: Remove no longer exported functions There were a leftover few prototypes of Curl_ functions that we used to export but no longer do, this removes those prototypes and cleans up
headers: Remove no longer exported functions There were a leftover few prototypes of Curl_ functions that we used to export but no longer do, this removes those prototypes and cleans up any comments still referring to them. Curl_write32_le(), Curl_strcpy_url(), Curl_strlen_url(), Curl_up_free() Curl_concat_url(), Curl_detach_connnection(), Curl_http_setup_conn() were made static in 05b100aee247bb9bec8e9a1b0166496aa4248d1c. Curl_http_perhapsrewind() made static in 574aecee208f79d391f10d57520b3. For the remainder, I didn't trawl the Git logs hard enough to capture their exact time of deletion, but they were all gone: Curl_splayprint(), Curl_http2_send_request(), Curl_global_host_cache_dtor(), Curl_scan_cache_used(), Curl_hostcache_destroy(), Curl_second_connect(), Curl_http_auth_stage() and Curl_close_connections(). Closes #4096 Reviewed-by: Daniel Stenberg <daniel@haxx.se>
show more ...
|
Revision tags: curl-7_65_1, curl-7_65_0 |
|
#
2f44e94e |
| 05-Apr-2019 |
Daniel Stenberg |
pipelining: removed As previously planned and documented in DEPRECATE.md, all pipelining code is removed. Closes #3651
|