vtls: TLS session storage overhaul

- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object,

vtls: TLS session storage overhaul

- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object, freeing it also on failures
- change tls backend use
- test_17, add tests for SSL session resumption

Closes #13386

show more ...

TLS: add support for ECH (Encrypted Client Hello)

An EXPERIMENTAL feature used with CURLOPT_ECH and --ech.

Closes #11922

multi: introduce SETUP state for better timeouts

Since we can go to the CONNECT state from PENDING, potentially multiple
times for a single transfer, this change introdues a SETUP state

multi: introduce SETUP state for better timeouts

Since we can go to the CONNECT state from PENDING, potentially multiple
times for a single transfer, this change introdues a SETUP state that
happens before CONNECT when doing a new transfer.

Now, doing a redirect on a handle goes back to SETUP (not CONNECT like
before) and we initilize the connect timeout etc in SETUP. Previously,
we would do it in CONNECT but that would make it unreliable in cases
where a transfer goes in and out between CONNECT and PENDING multiple
times.

SETUP is transient, so the handle never actually stays in that state.

Additionally: take care of timeouts of PENDING transfers in
curl_multi_perform()

Ref: #13227
Closes #13371

show more ...

build: prefer `USE_IPV6` macro internally (was: `ENABLE_IPV6`)

Before this patch, two macros were used to guard IPv6 features in curl
sources: `ENABLE_IPV6` and `USE_IPV6`. This patch ma

build: prefer `USE_IPV6` macro internally (was: `ENABLE_IPV6`)

Before this patch, two macros were used to guard IPv6 features in curl
sources: `ENABLE_IPV6` and `USE_IPV6`. This patch makes the source use
the latter for consistency with other similar switches.

`-DENABLE_IPV6` remains accepted for compatibility as a synonym for
`-DUSE_IPV6`, when passed to the compiler.

`ENABLE_IPV6` also remains the name of the CMake and `Makefile.vc`
options to control this feature.

Closes #13349

show more ...

file: add support for getting basic directory listings

Not supported on Windows (yet)

Closes #13137

lib: add Curl_xfer_write_resp_hd

Add method in protocol handlers to allow writing of a single,
0-terminated header line. Avoids parsing and copying these lines.

Closes #13165

tls: use shared init code for TCP+QUIC

Closes #13172

NTLM_WB: drop support

The feature has not worked for months and has been marked as DEPRECATED
for six+ months.

Closes #13249

urldata: remove fields not used depending on used features

Reduced size of dynamically_allocated_data structure.

Reduced number of stored values in enum dupstring and enum dupblob.

urldata: remove fields not used depending on used features

Reduced size of dynamically_allocated_data structure.

Reduced number of stored values in enum dupstring and enum dupblob. This
affects the reduced array placed in the UserDefined structure.

Closes #13188

show more ...

http: expect 100 rework

Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general

http: expect 100 rework

Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general events like a timer.

HTTP installs a `CURL_CR_PROTOCOL` reader when announcing `Expect:
100-continue`. That reader works as follows:

- on first invocation, records time, starts the `EXPIRE_100_TIMEOUT`
timer, disables `KEEP_SEND`, enables `KEEP_SEND_TIMER` and returns 0,
eos=FALSE like a paused upload.

- on subsequent invocation it checks if the timer has expired. If so, it
enables `KEEP_SEND` and switches to passing through reads to the
underlying readers.

Transfer handling's `readwrite()` will be invoked when a timer expires
(like `EXPIRE_100_TIMEOUT`) or when data from the server arrives. Seeing
`KEEP_SEND_TIMER`, it will try to upload more data, which triggers
reading from the client readers again. Which then may lead to a new
pausing or cause the upload to start.

Flags and timestamps connected to this have been moved from
`SingleRequest` into the reader's context.

Closes #13110

show more ...

lib: move 'done' parameter to SingleRequests

A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connecti

lib: move 'done' parameter to SingleRequests

A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connections.
The "readwrite()" calls and functions connected to those carried a `bool
*done` parameter to indicate that the current `SingleRequest` is over.
This may happen before `upload_done` or `download_done` bits of
`SingleRequest` are set.

The problem with that is now `write_resp()` protocol handlers are
invoked in places where the `bool *done` cannot be passed up to the
caller. Instead of being a bool in the call chain, it needs to become a
member of `SingleRequest`, reflecting its state.

This removes the `bool *done` parameter and adds the `done` bit to
`SingleRequest` instead. It adds `Curl_req_soft_reset()` for using a
`SingleRequest` in a follow up, clearing `done` and other
flags/counters.

Closes #13096

show more ...

lib: keep conn IP information together

new struct ip_quadruple for holding local/remote addr+port

- used in data->info and conn and cf-socket.c
- copy back and forth complete st

lib: keep conn IP information together

new struct ip_quadruple for holding local/remote addr+port

- used in data->info and conn and cf-socket.c
- copy back and forth complete struct
- add 'secondary' to conn
- use secondary in reporting success for ftp 2nd connection

Reported-by: DasKutti on github
Fixes #13084
Closes #13090

show more ...

lib: client reader polish

- seek_func/seek_client, use transfer values only
- remove copies held in `struct connectdata`, use only
ever `data->set.seek_func`
- reso

lib: client reader polish

- seek_func/seek_client, use transfer values only
- remove copies held in `struct connectdata`, use only
ever `data->set.seek_func`
- resolves possible issues in multiuse connections
- new mime post reader eliminates need to ever overwriting this

- websockets, remove empty Curl_ws_done() function

Closes #13079

show more ...

lib: enhance client reader resume + rewind

- update client reader documentation
- client reader, add rewind capabilities
- tell creader to rewind on next start
- Curl_cli

lib: enhance client reader resume + rewind

- update client reader documentation
- client reader, add rewind capabilities
- tell creader to rewind on next start
- Curl_client_reset() will keep reader for future rewind if requested
- add Curl_client_cleanup() for freeing all resources independent of
rewinds
- add Curl_client_start() to trigger rewinds
- move rewind code from multi.c to sendf.c and make part of
"cr-in"'s implementation
- http, move the "resume_from" handling into the client readers
- the setup of a HTTP request is reshuffled to follow:
* determine method, target, auth negotiation
* install the client reader(s) for the request, including crlf
conversions and "chunked" encoding
* apply ranges to client reader
* concat request headers, upgrades, cookies, etc.
* complete request by determining Content-Length of installed
readers in combination with method
* send
- add methods for client readers to
* return the overall length they will generate (or -1 when unknown)
* return the amount of data on the CLIENT level, so that
expect-100 can decide if it want to apply itself
* set a "resume_from" offset or fail if unsupported
- struct HTTP has become largely empty now
- rename `Client_reader_*` to `Curl_creader_*`

Closes #13026

show more ...

lib: further send/upload handling polish

- Move all the "upload_done" handling to request.c

- add possibility to abort sending of a request
- add `Curl_req_done_sending()` f

lib: further send/upload handling polish

- Move all the "upload_done" handling to request.c

- add possibility to abort sending of a request
- add `Curl_req_done_sending()` for checks
- transfer.c: readwrite_upload() now clean

- removing data->state.ulbuf and data->req.upload_fromhere

- as well as data->req.upload_present
- set data->req.upload_done on having read all from
the client and completely flushed the send buffer

- tftp, remove setting of data->req.upload_fromhere

- serves no purpose as `upload_present` is not set
and the data itself is directly `sendto()` anyway

- smtp, make upload EOB conversion a client reader
- xfer_ulbuf addition

- add xfer_ulbuf for borrowing, similar to xfer_buf
- use in file upload
- use in c-hyper body sending

- h1-proxy, remove init of data->state.uilbuf that is never used
- smb, add own send_buf instead of using data->state.ulbuf

Closes #13010

show more ...

lib: send rework

Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
when and at what level they operate

- sen

lib: send rework

Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
when and at what level they operate

- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer setup of
`conn->sockfd` and `conn->writesockfd` on which connection filter
chain to operate.

- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index as
parameter.

- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
consistency

- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
returned by all other send() variants.

SingleRequest reshuffling

- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data

Closes #12963

show more ...

lib: move client writer into own source

Refactoring of the client writer that passes the data to the
client/application's callback functions.

- split out into own source cw-out.

lib: move client writer into own source

Refactoring of the client writer that passes the data to the
client/application's callback functions.

- split out into own source cw-out.[ch] from sendf.c

- move tempwrite and tempcount from data->state into the context of the
client writer

- redesign the 3 tempwrite dynbufs as a linked list of dynbufs. On
paused transfers, this allows to "record" interleaved HEADER/BODY
chunks to be "played back" in the same order on unpausing.

- keep the overall size limit of all buffered data to DYN_PAUSE_BUFFER.
On exceeding that, return CURLE_TOO_LARGE instead of
CURLE_OUT_OF_MEMORY as before.

- add method to be called when a transfer is DONE to allow writing of
any data still buffered

- when paused, record HEADER writes exactly as they come for later
playback. HEADERs are documented to be written one-by-one.

Closes #12898

show more ...

urldata: move authneg bit from conn to Curl_easy

- from `conn->bits.authneg` to `data->req.authneg`
- this is a property of the request about to be made
and not a property of the c

urldata: move authneg bit from conn to Curl_easy

- from `conn->bits.authneg` to `data->req.authneg`
- this is a property of the request about to be made
and not a property of the connection
- in multiuse connections, transfer could step on each others
toes here potentially.

Closes #12949

show more ...

lib: remove curl_mimepart object when CURL_DISABLE_MIME

Remove curl_mimepart object from UserDefined structure when
CURL_DISABLE_MIME flag is active. Reduce size of UserDefined structure

lib: remove curl_mimepart object when CURL_DISABLE_MIME

Remove curl_mimepart object from UserDefined structure when
CURL_DISABLE_MIME flag is active. Reduce size of UserDefined structure.

Also remove unreachable code: when CURL_DISABLE_MIME is set, httpreq can
never have HTTPREQ_POST_MIME value and the same goes for the
CURL_DISABLE_FORM_API flag and the HTTPREQ_POST_FORM value

Closes #12948

show more ...

CURLINFO_USED_PROXY: return bool whether the proxy was used

Adds test536 to verify

Closes #12719

DoH: add trace configuration

- refs #12397 where it is dicussed how to en-/disable verbose output
of DoH operations
- introducing `struct curl_trc_feat` to track a curl feature for

DoH: add trace configuration

- refs #12397 where it is dicussed how to en-/disable verbose output
of DoH operations
- introducing `struct curl_trc_feat` to track a curl feature for
tracing
- adding `data->state.feat` optionally pointing to the feature a
transfer belongs to
- adding trace functions and verbosity checks on features
- using trace feature in DoH code
- documenting `doh` as feature for `--trace-config`

Closes #12411

show more ...

vtls: fix tls proxy peer verification

- When verifying a proxy certificate for an ip address, use the correct
ip family.

Prior to this change the "connection" ip family was us

vtls: fix tls proxy peer verification

- When verifying a proxy certificate for an ip address, use the correct
ip family.

Prior to this change the "connection" ip family was used, which was not
necessarily the same.

Reported-by: HsiehYuho@users.noreply.github.com

Fixes https://github.com/curl/curl/issues/12831
Closes https://github.com/curl/curl/pull/12931

show more ...

transfer: improve Windows SO_SNDBUF update limit

- Change the 1 second SO_SNDBUF update limit from per transfer to per
connection.

Prior to this change many transfers over the

transfer: improve Windows SO_SNDBUF update limit

- Change the 1 second SO_SNDBUF update limit from per transfer to per
connection.

Prior to this change many transfers over the same connection could cause
many SO_SNDBUF updates made to that connection per second, which was
unnecessary.

Closes https://github.com/curl/curl/pull/12911

show more ...

multi: add xfer_buf to multi handle

- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`

multi: add xfer_buf to multi handle

- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`
- used in transfer.c readwrite_data()

Closes #12805

show more ...

ftp: do lineend conversions in client writer

- remove the ftp special handling from sendf.c
- let ftp_do() add a client writer that does
the linened conversions
- change the li

ftp: do lineend conversions in client writer

- remove the ftp special handling from sendf.c
- let ftp_do() add a client writer that does
the linened conversions
- change the lineend conversion to no longer
modify the passed buffer, but write smaller
chunks to the next cwriter instead. The
inefficiency of this will be mitigated once
we add output buffering for all client writes.

Closes #12878

show more ...