multi: getsock improvements for QUIC connecting

curl_multi_poll: a sister to curl_multi_wait() that waits more

Repeatedly we see problems where using curl_multi_wait() is difficult or
just awkward because if it has no file descriptor

curl_multi_poll: a sister to curl_multi_wait() that waits more

Repeatedly we see problems where using curl_multi_wait() is difficult or
just awkward because if it has no file descriptor to wait for
internally, it returns immediately and leaves it to the caller to wait
for a small amount of time in order to avoid occasional busy-looping.

This is often missed or misunderstood, leading to underperforming
applications.

This change introduces curl_multi_poll() as a replacement drop-in
function that accepts the exact same set of arguments. This function
works identically to curl_multi_wait() - EXCEPT - for the case when
there's nothing to wait for internally, as then this function will by
itself wait for a "suitable" short time before it returns. This
effectiely avoids all risks of busy-looping and should also make it less
likely that apps "over-wait".

This also changes the curl tool to use this funtion internally when
doing parallel transfers and changes curl_easy_perform() to use it
internally.

Closes #4163

show more ...

timediff: make it 64 bit (if possible) even with 32 bit time_t

... to make it hold microseconds too.

Fixes #4165
Closes #4168

cleanup: remove the 'numsocks' argument used in many places

It was used (intended) to pass in the size of the 'socks' array that is
also passed to these functions, but was rarely actuall

cleanup: remove the 'numsocks' argument used in many places

It was used (intended) to pass in the size of the 'socks' array that is
also passed to these functions, but was rarely actually checked/used and
the array is defined to a fixed size of MAX_SOCKSPEREASYHANDLE entries
that should be used instead.

Closes #4169

show more ...

asyn-thread: create a socketpair to wait on

Closes #4157

multi: call detach_connection before Curl_disconnect

Curl_disconnect bails out if conn->easyq is not empty, detach_connection
needs to be called first to remove the current easy from the

multi: call detach_connection before Curl_disconnect

Curl_disconnect bails out if conn->easyq is not empty, detach_connection
needs to be called first to remove the current easy from the queue.

Fixes #4144
Closes #4151

show more ...

multi: enable multiplexing by default (again)

It was originally made default in d7c4213bd0c (7.62.0) but mistakenly
reverted in commit 2f44e94efb3d (7.65.0). Now enabled again.

multi: enable multiplexing by default (again)

It was originally made default in d7c4213bd0c (7.62.0) but mistakenly
reverted in commit 2f44e94efb3d (7.65.0). Now enabled again.

Closes #4051

show more ...

multi: fix the transfer hash function

Follow-up from 8b987cc7eb

Reported-by: Tom van der Woerdt
Fixes #4018
Closes #4024

multi: fix the transfer hashes in the socket hash entries

- The transfer hashes weren't using the correct keys so removing entries
failed.

- Simplified the iteration logic ove

multi: fix the transfer hashes in the socket hash entries

- The transfer hashes weren't using the correct keys so removing entries
failed.

- Simplified the iteration logic over transfers sharing the same socket and
they now simply are set to expire and thus get handled in the "regular"
timer loop instead.

Reported-by: Tom van der Woerdt
Fixes #4012
Closes #4014

show more ...

multi: make sure 'data' can present in several sockhash entries

Since more than one socket can be used by each transfer at a given time,
each sockhash entry how has its own hash table wi

multi: make sure 'data' can present in several sockhash entries

Since more than one socket can be used by each transfer at a given time,
each sockhash entry how has its own hash table with transfers using that
socket.

In addition, the sockhash entry can now be marked 'blocked = TRUE'"
which then makes the delete function just set 'removed = TRUE' instead
of removing it "for real", as a way to not rip out the carpet under the
feet of a parent function that iterates over the transfers of that same
sockhash entry.

Reported-by: Tom van der Woerdt
Fixes #3961
Fixes #3986
Fixes #3995
Fixes #4004
Closes #3997

show more ...

unpause: trigger a timeout for event-based transfers

... so that timeouts or other state machine actions get going again
after a changing pause state. For example, if the last delivery w

unpause: trigger a timeout for event-based transfers

... so that timeouts or other state machine actions get going again
after a changing pause state. For example, if the last delivery was
paused there's no pending socket activity.

Reported-by: sstruchtrup on github
Fixes #3994
Closes #4001

show more ...

singlesocket: use separate variable for inner loop

An inner loop within the singlesocket() function wrongly re-used the
variable for the outer loop which then could cause an infinite

singlesocket: use separate variable for inner loop

An inner loop within the singlesocket() function wrongly re-used the
variable for the outer loop which then could cause an infinite
loop. Change to using a separate variable!

Reported-by: Eric Wu
Fixes #3970
Closes #3973

show more ...

multi: track users of a socket better

They need to be removed from the socket hash linked list with more care.

When sh_delentry() is called to remove a sockethash entry, remove all

multi: track users of a socket better

They need to be removed from the socket hash linked list with more care.

When sh_delentry() is called to remove a sockethash entry, remove all
individual transfers from the list first. To enable this, each Curl_easy struct
now stores a pointer to the sockethash entry to know how to remove itself.

Reported-by: Tom van der Woerdt and Kunal Ekawde

Fixes #3952
Fixes #3904
Closes #3953

show more ...

cleanup: remove FIXME and TODO comments

They serve very little purpose and mostly just add noise. Most of them
have been around for a very long time. I read them all before removing

cleanup: remove FIXME and TODO comments

They serve very little purpose and mostly just add noise. Most of them
have been around for a very long time. I read them all before removing
or rephrasing them.

Ref: #3876
Closes #3883

show more ...

http_negotiate: Move the Negotiate state out of the negotiatedata structure

Given that this member variable is not used by the SASL based protocols
there is no need to have it here.

http_negotiate: Move the Negotiate state out of the negotiatedata structure

Given that this member variable is not used by the SASL based protocols
there is no need to have it here.

Closes #3882

show more ...

http_ntlm: Move the NTLM state out of the ntlmdata structure

Given that this member variable is not used by the SASL based protocols
there is no need to have it here.

wildcard: disable from build when FTP isn't present

build: fix "clarify calculation precedence" warnings

Codacy/CppCheck warns about this. Consistently use parentheses as we
already do in some places to silence the warning.

Close

build: fix "clarify calculation precedence" warnings

Codacy/CppCheck warns about this. Consistently use parentheses as we
already do in some places to silence the warning.

Closes https://github.com/curl/curl/pull/3866

show more ...

Revert "multi: support verbose conncache closure handle"

This reverts commit b0972bc.

- No longer show verbose output for the conncache closure handle.

The offending commit

Revert "multi: support verbose conncache closure handle"

This reverts commit b0972bc.

- No longer show verbose output for the conncache closure handle.

The offending commit was added so that the conncache closure handle
would inherit verbose mode from the user's easy handle. (Note there is
no way for the user to set options for the closure handle which is why
that was necessary.) Other debug settings such as the debug function
were not also inherited since we determined that could lead to crashes
if the user's per-handle private data was used on an unexpected handle.

The reporter here says he has a debug function to capture the verbose
output, and does not expect or want any output to stderr; however
because the conncache closure handle does not inherit the debug function
the verbose output for that handle does go to stderr.

There are other plausible scenarios as well such as the user redirects
stderr on their handle, which is also not inherited since it could lead
to crashes when used on an unexpected handle.

Short of allowing the user to set options for the conncache closure
handle I don't think there's much we can safely do except no longer
inherit the verbose setting.

Bug: https://curl.haxx.se/mail/lib-2019-05/0021.html
Reported-by: Kristoffer Gleditsch

Ref: https://github.com/curl/curl/pull/3598
Ref: https://github.com/curl/curl/pull/3618

Closes https://github.com/curl/curl/pull/3856

show more ...

multi: provide Curl_multiuse_state to update information

As soon as a TLS backend gets ALPN conformation about the specific HTTP
version it can now set the multiplex situation for the "b

multi: provide Curl_multiuse_state to update information

As soon as a TLS backend gets ALPN conformation about the specific HTTP
version it can now set the multiplex situation for the "bundle" and
trigger moving potentially queued up transfers to the CONNECT state.

show more ...

process_pending_handles: mark queued transfers as previously pending

With transfers being queued up, we only move one at a a time back to the
CONNECT state but now we mark moved transfer

process_pending_handles: mark queued transfers as previously pending

With transfers being queued up, we only move one at a a time back to the
CONNECT state but now we mark moved transfers so that when a moved
transfer is confirmed "successful" (it connected) it will trigger the
move of another pending transfer. Previously, it would otherwise wait
until the transfer was done before doing this. This makes queued up
pending transfers get processed (much) faster.

show more ...

multi: fix the statenames (follow-up fix from 2f44e94efb3df8e)

The list of names must be in sync with the defined states in the header
file!

CURLOPT_DNS_USE_GLOBAL_CACHE: remove

Remove the code too. The functionality has been disabled in code since
7.62.0. Setting this option will from now on simply be ignored and have
no

CURLOPT_DNS_USE_GLOBAL_CACHE: remove

Remove the code too. The functionality has been disabled in code since
7.62.0. Setting this option will from now on simply be ignored and have
no function.

Closes #3654

show more ...

multi_runsingle(): fix use-after-free

Fixes #3745
Closes #3746

The following snippet
```

int main()
{
CURL* hCurlHandle = curl_easy_init();
curl

multi_runsingle(): fix use-after-free

Fixes #3745
Closes #3746

The following snippet
```

int main()
{
CURL* hCurlHandle = curl_easy_init();
curl_easy_setopt(hCurlHandle, CURLOPT_URL, "http://example.com");
curl_easy_setopt(hCurlHandle, CURLOPT_PROXY, "1");
curl_easy_perform(hCurlHandle);
curl_easy_cleanup(hCurlHandle);
return 0;
}
```
triggers the following Valgrind warning

```
==4125== Invalid read of size 8
==4125== at 0x4E7D1EE: Curl_llist_remove (llist.c:97)
==4125== by 0x4E7EF5C: detach_connnection (multi.c:798)
==4125== by 0x4E80545: multi_runsingle (multi.c:1451)
==4125== by 0x4E8197C: curl_multi_perform (multi.c:2072)
==4125== by 0x4E766A0: easy_transfer (easy.c:625)
==4125== by 0x4E76915: easy_perform (easy.c:719)
==4125== by 0x4E7697C: curl_easy_perform (easy.c:738)
==4125== by 0x4008BE: main (in /home/even/curl/test)
==4125== Address 0x9b3d1d0 is 1,120 bytes inside a block of size 1,600 free'd
==4125== at 0x4C2ECF0: free (vg_replace_malloc.c:530)
==4125== by 0x4E62C36: conn_free (url.c:756)
==4125== by 0x4E62D34: Curl_disconnect (url.c:818)
==4125== by 0x4E48DF9: Curl_once_resolved (hostip.c:1097)
==4125== by 0x4E8052D: multi_runsingle (multi.c:1446)
==4125== by 0x4E8197C: curl_multi_perform (multi.c:2072)
==4125== by 0x4E766A0: easy_transfer (easy.c:625)
==4125== by 0x4E76915: easy_perform (easy.c:719)
==4125== by 0x4E7697C: curl_easy_perform (easy.c:738)
==4125== by 0x4008BE: main (in /home/even/curl/test)
==4125== Block was alloc'd at
==4125== at 0x4C2F988: calloc (vg_replace_malloc.c:711)
==4125== by 0x4E6438E: allocate_conn (url.c:1654)
==4125== by 0x4E685B4: create_conn (url.c:3496)
==4125== by 0x4E6968F: Curl_connect (url.c:4023)
==4125== by 0x4E802E7: multi_runsingle (multi.c:1368)
==4125== by 0x4E8197C: curl_multi_perform (multi.c:2072)
==4125== by 0x4E766A0: easy_transfer (easy.c:625)
==4125== by 0x4E76915: easy_perform (easy.c:719)
==4125== by 0x4E7697C: curl_easy_perform (easy.c:738)
==4125== by 0x4008BE: main (in /home/even/curl/test)
```

This has been bisected to commit 2f44e94

Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=14109
Credit to OSS Fuzz

show more ...

pipelining: removed

As previously planned and documented in DEPRECATE.md, all pipelining
code is removed.

Closes #3651