lib: bump hash sizes to `size_t`

Follow-up to cc907e80a2498c0599253271a6f657f614b52a4e #13502
Cherry-picked from #13489
Closes #13601

lib: clear the easy handle's saved errno before transfer

- Clear data->state.os_errno before transfer.

- Explain the change in behavior in the CURLINFO_OS_ERRNO doc.

- Add

lib: clear the easy handle's saved errno before transfer

- Clear data->state.os_errno before transfer.

- Explain the change in behavior in the CURLINFO_OS_ERRNO doc.

- Add to the CURLINFO_OS_ERRNO doc the list of libcurl network-related
errors that may cause the errno to be saved.

data->state.os_errno is saved before libcurl returns a network-related
failure such as connection failure. It is accessible to the user via
CURLINFO_OS_ERRNO so they can get more information about the failure.

Prior to this change it wasn't cleared before transfer, so if a user
retrieved the saved errno it could be from a previous transfer. That is
because an errno is not always saved for network-related errors.

Closes https://github.com/curl/curl/pull/13574

show more ...

multi: remove useless assignment

Spotted by CodeSonar

Closes #13500

multi: multi_wait improvements

- only call `multi_getsock()` once for all transfers
- realloc pollset array on demand
- fold repeated sockets

Closes #13150

multi: avoid memory-leak risk

'newurl' is allocated in some conditions and used in a few scenarios,
but there were theoretical combinations in which it would not get freed.
Move the

multi: avoid memory-leak risk

'newurl' is allocated in some conditions and used in a few scenarios,
but there were theoretical combinations in which it would not get freed.
Move the free to happen unconditionally. Never triggered by tests, but
spotted by Coverity.

Closes #13471

show more ...

multi: remove the unused Curl_preconnect function

The implementation has been removed, no point in keeping it around.

Follow-up to 476adfeac019ed

Closes #13422

cw-out: improved error handling

- remember error encountered in invoking write callback and always fail
afterwards without further invokes

- check behaviour in test_02_17 with

cw-out: improved error handling

- remember error encountered in invoking write callback and always fail
afterwards without further invokes

- check behaviour in test_02_17 with h2-pausing client

Reported-by: Pavel Kropachev
Fixes #13337
Closes #13340

show more ...

multi: introduce SETUP state for better timeouts

Since we can go to the CONNECT state from PENDING, potentially multiple
times for a single transfer, this change introdues a SETUP state

multi: introduce SETUP state for better timeouts

Since we can go to the CONNECT state from PENDING, potentially multiple
times for a single transfer, this change introdues a SETUP state that
happens before CONNECT when doing a new transfer.

Now, doing a redirect on a handle goes back to SETUP (not CONNECT like
before) and we initilize the connect timeout etc in SETUP. Previously,
we would do it in CONNECT but that would make it unreliable in cases
where a transfer goes in and out between CONNECT and PENDING multiple
times.

SETUP is transient, so the handle never actually stays in that state.

Additionally: take care of timeouts of PENDING transfers in
curl_multi_perform()

Ref: #13227
Closes #13371

show more ...

multi: timeout handles even without connection

When there is a "change" in a multi handle and pending handles are moved
back to the main list to be retested if they can proceed further (

multi: timeout handles even without connection

When there is a "change" in a multi handle and pending handles are moved
back to the main list to be retested if they can proceed further (for
example a previous transfer completed or a connection has a confirmed
multiplexed state), the timeout check in multi_runsingle() would not
trigger because it required an established connection.

This could make a pending tranfer go back to pending state even though
it had been "in progress" for a longer time than permitted. By removing
the requirement for an associated connection, the timeout check will be
done proper even for transfers that has not yet been assigned one.

Ref #13227
Reported-by: Rahul Krishna M
Closes #13276

show more ...

misc: fix typos

Closes #13344

llist: add Curl_llist_append()

- use for better readability in all places where the "insert_next"
actually performs an append to the list
- add some tests in unit1300

Clos

llist: add Curl_llist_append()

- use for better readability in all places where the "insert_next"
actually performs an append to the list
- add some tests in unit1300

Closes #13336

show more ...

lib: add curl_multi_waitfds

New function call, similar to curl_multi_fdset()

Closes #13135

lib: use multi instead of multi_easy for the active multi

- Use data->multi and not data->multi_easy to refer to the active multi.

The easy handle's active multi is always data->mul

lib: use multi instead of multi_easy for the active multi

- Use data->multi and not data->multi_easy to refer to the active multi.

The easy handle's active multi is always data->multi.

This is a follow up to 757dfdf which changed curl so that an easy handle
used with the easy interface and then multi interface cannot have two
different multi handles associated with it at the same time
(data->multi_easy from the easy interface and data->multi from the multi
interface).

Closes https://github.com/curl/curl/pull/12665

show more ...

lib: move 'done' parameter to SingleRequests

A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connecti

lib: move 'done' parameter to SingleRequests

A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connections.
The "readwrite()" calls and functions connected to those carried a `bool
*done` parameter to indicate that the current `SingleRequest` is over.
This may happen before `upload_done` or `download_done` bits of
`SingleRequest` are set.

The problem with that is now `write_resp()` protocol handlers are
invoked in places where the `bool *done` cannot be passed up to the
caller. Instead of being a bool in the call chain, it needs to become a
member of `SingleRequest`, reflecting its state.

This removes the `bool *done` parameter and adds the `done` bit to
`SingleRequest` instead. It adds `Curl_req_soft_reset()` for using a
`SingleRequest` in a follow up, clearing `done` and other
flags/counters.

Closes #13096

show more ...

lib: keep conn IP information together

new struct ip_quadruple for holding local/remote addr+port

- used in data->info and conn and cf-socket.c
- copy back and forth complete st

lib: keep conn IP information together

new struct ip_quadruple for holding local/remote addr+port

- used in data->info and conn and cf-socket.c
- copy back and forth complete struct
- add 'secondary' to conn
- use secondary in reporting success for ftp 2nd connection

Reported-by: DasKutti on github
Fixes #13084
Closes #13090

show more ...

lib: enhance client reader resume + rewind

- update client reader documentation
- client reader, add rewind capabilities
- tell creader to rewind on next start
- Curl_cli

lib: enhance client reader resume + rewind

- update client reader documentation
- client reader, add rewind capabilities
- tell creader to rewind on next start
- Curl_client_reset() will keep reader for future rewind if requested
- add Curl_client_cleanup() for freeing all resources independent of
rewinds
- add Curl_client_start() to trigger rewinds
- move rewind code from multi.c to sendf.c and make part of
"cr-in"'s implementation
- http, move the "resume_from" handling into the client readers
- the setup of a HTTP request is reshuffled to follow:
* determine method, target, auth negotiation
* install the client reader(s) for the request, including crlf
conversions and "chunked" encoding
* apply ranges to client reader
* concat request headers, upgrades, cookies, etc.
* complete request by determining Content-Length of installed
readers in combination with method
* send
- add methods for client readers to
* return the overall length they will generate (or -1 when unknown)
* return the amount of data on the CLIENT level, so that
expect-100 can decide if it want to apply itself
* set a "resume_from" offset or fail if unsupported
- struct HTTP has become largely empty now
- rename `Client_reader_*` to `Curl_creader_*`

Closes #13026

show more ...

lib: further send/upload handling polish

- Move all the "upload_done" handling to request.c

- add possibility to abort sending of a request
- add `Curl_req_done_sending()` f

lib: further send/upload handling polish

- Move all the "upload_done" handling to request.c

- add possibility to abort sending of a request
- add `Curl_req_done_sending()` for checks
- transfer.c: readwrite_upload() now clean

- removing data->state.ulbuf and data->req.upload_fromhere

- as well as data->req.upload_present
- set data->req.upload_done on having read all from
the client and completely flushed the send buffer

- tftp, remove setting of data->req.upload_fromhere

- serves no purpose as `upload_present` is not set
and the data itself is directly `sendto()` anyway

- smtp, make upload EOB conversion a client reader
- xfer_ulbuf addition

- add xfer_ulbuf for borrowing, similar to xfer_buf
- use in file upload
- use in c-hyper body sending

- h1-proxy, remove init of data->state.uilbuf that is never used
- smb, add own send_buf instead of using data->state.ulbuf

Closes #13010

show more ...

multi: make add_handle free any multi_easy

If the easy handle that is being added to a multi handle has previously
been used for curl_easy_perform(), there is a private multi handle here

multi: make add_handle free any multi_easy

If the easy handle that is being added to a multi handle has previously
been used for curl_easy_perform(), there is a private multi handle here
that we can kill off. While it flushes some caches etc for the easy
handle would it be used for an easy interface transfer again after being
used in the multi stack, this cleanup simplifies behavior and uses less
memory.

Closes #12992

show more ...

lib: send rework

Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
when and at what level they operate

- sen

lib: send rework

Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
when and at what level they operate

- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer setup of
`conn->sockfd` and `conn->writesockfd` on which connection filter
chain to operate.

- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index as
parameter.

- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
consistency

- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
returned by all other send() variants.

SingleRequest reshuffling

- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data

Closes #12963

show more ...

lib: move client writer into own source

Refactoring of the client writer that passes the data to the
client/application's callback functions.

- split out into own source cw-out.

lib: move client writer into own source

Refactoring of the client writer that passes the data to the
client/application's callback functions.

- split out into own source cw-out.[ch] from sendf.c

- move tempwrite and tempcount from data->state into the context of the
client writer

- redesign the 3 tempwrite dynbufs as a linked list of dynbufs. On
paused transfers, this allows to "record" interleaved HEADER/BODY
chunks to be "played back" in the same order on unpausing.

- keep the overall size limit of all buffered data to DYN_PAUSE_BUFFER.
On exceeding that, return CURLE_TOO_LARGE instead of
CURLE_OUT_OF_MEMORY as before.

- add method to be called when a transfer is DONE to allow writing of
any data still buffered

- when paused, record HEADER writes exactly as they come for later
playback. HEADERs are documented to be written one-by-one.

Closes #12898

show more ...

lib: remove curl_mimepart object when CURL_DISABLE_MIME

Remove curl_mimepart object from UserDefined structure when
CURL_DISABLE_MIME flag is active. Reduce size of UserDefined structure

lib: remove curl_mimepart object when CURL_DISABLE_MIME

Remove curl_mimepart object from UserDefined structure when
CURL_DISABLE_MIME flag is active. Reduce size of UserDefined structure.

Also remove unreachable code: when CURL_DISABLE_MIME is set, httpreq can
never have HTTPREQ_POST_MIME value and the same goes for the
CURL_DISABLE_FORM_API flag and the HTTPREQ_POST_FORM value

Closes #12948

show more ...

multi: fix multi_sock handling of select_bits

- OR the event bitmask to data->state.select_bits instead of overwriting
them. They are cleared again on use.

Reported-by: 5533as

multi: fix multi_sock handling of select_bits

- OR the event bitmask to data->state.select_bits instead of overwriting
them. They are cleared again on use.

Reported-by: 5533asdg on github
Fixes #12971
Closes #12972

show more ...

multi: add xfer_buf to multi handle

- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`

multi: add xfer_buf to multi handle

- can be borrowed by transfer during recv-write operation
- needs to be released before borrowing again
- adjustis size to `data->set.buffer_size`
- used in transfer.c readwrite_data()

Closes #12805

show more ...

lib: replace readwrite with write_resp

This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
m

lib: replace readwrite with write_resp

This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.

The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.

Curl's protocol handler `readwrite()` method been changed:

```diff
- CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
- const char *buf, size_t blen,
- size_t *pconsumed, bool *readmore);
+ CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+ bool is_eos, bool *done);
```

The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:

* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
finished

This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.

The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.

HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.

The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:

```
static CURLcode readwrite_data(data, ...)
{
do {
Curl_xfer_recv_resp(data, buf)
...
Curl_xfer_write_resp(data, buf)
...
} while(interested);
...
}
```

All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.

All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.

Closes #12480

show more ...

multi: pollset adjust, init with FIRSTSOCKET during connect

- `conn->sockfd` is set by `Curl_setup_transfer()`, but that
is called *after* the connection has been established
- use

multi: pollset adjust, init with FIRSTSOCKET during connect

- `conn->sockfd` is set by `Curl_setup_transfer()`, but that
is called *after* the connection has been established
- use `conn->sock[FIRSTSOCKET]` instead

Follow-up to a0f94800d507de
Closes #12664

show more ...