lib: introduce struct easy_poll_set for poll information

Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of

lib: introduce struct easy_poll_set for poll information

Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.

Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.

This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.

Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.

After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.

Example:

* transfer wants to send, adds POLLOUT
* http/2 filter has a flow control block, removes POLLOUT and adds
POLLIN (it is waiting on a WINDOW_UPDATE from the server)
* TLS filter is connected and changes nothing
* h2-proxy filter also has a flow control block on its tunnel stream,
removes POLLOUT and adds POLLIN also.
* socket filter is connected and changes nothing
* The resulting pollset is then mixed together with all other transfers
and their pollsets, just as before.

Use of `SEND_HOLD` is no longer necessary in the filters.

All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.

The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.

No noticeable difference in performance was detected in local scorecard
runs.

Closes #11833

show more ...

spelling: fix codespell 2.2.6 typos

Closes #12019

lib: disambiguate Curl_client_write flag semantics

- use CLIENTWRITE_BODY *only* when data is actually body data
- add CLIENTWRITE_INFO for meta data that is *not* a HEADER
- debug a

lib: disambiguate Curl_client_write flag semantics

- use CLIENTWRITE_BODY *only* when data is actually body data
- add CLIENTWRITE_INFO for meta data that is *not* a HEADER
- debug assertions that BODY/INFO/HEADER is not used mixed
- move `data->set.include_header` check into Curl_client_write
so protocol handlers no longer have to care
- add special in FTP for `data->set.include_header` for historic,
backward compatible reasons
- move unpausing of client writes from easy.c to sendf.c, so that
code is in one place and can forward flags correctly

Closes #11885

show more ...

ftp: fix temp write of ipv6 address

- During the check to differentiate between a port and IPv6 address
without brackets, write the binary IPv6 address to an in6_addr.

Prior t

ftp: fix temp write of ipv6 address

- During the check to differentiate between a port and IPv6 address
without brackets, write the binary IPv6 address to an in6_addr.

Prior to this change the binary IPv6 address was erroneously written to
a sockaddr_in6 'sa6' when it should have been written to its in6_addr
member 'sin6_addr'. There's no fallout because no members of 'sa6' are
accessed before it is later overwritten.

Closes https://github.com/curl/curl/pull/11747

show more ...

spelling: use 'reuse' not 're-use' in code and elsewhere

Unify the spelling as both versions were previously used intermittently

Closes #11717

cmake: cache more config and delete unused ones

- cache more Windows config results for faster initialization.

- delete unused config macros `HAVE_SYS_UTSNAME_H`, `HAVE_SSL_H`.

cmake: cache more config and delete unused ones

- cache more Windows config results for faster initialization.

- delete unused config macros `HAVE_SYS_UTSNAME_H`, `HAVE_SSL_H`.

- delete dead references to `sys/utsname.h`.

Closes #11551

show more ...

cmake: add support for "unity" builds

Aka "jumbo" or "amalgamation" builds. It means to compile all sources
per target as a single C source. This is experimental.

You can enable

cmake: add support for "unity" builds

Aka "jumbo" or "amalgamation" builds. It means to compile all sources
per target as a single C source. This is experimental.

You can enable it by passing `-DCMAKE_UNITY_BUILD=ON` to cmake.
It requires CMake 3.16 or newer.

It makes builds (much) faster, allows for better optimizations and tends
to promote less ambiguous code.

Also add a new AppVeyor CI job and convert an existing one to use
"unity" mode (one MSVC, one MinGW), and enable it for one macOS CI job.

Fix related issues:
- add missing include guard to `easy_lock.h`.
- rename static variables and functions (and a macro) with names reused
across sources, or shadowed by local variables.
- add an `#undef` after use.
- add a missing `#undef` before use.
- move internal definitions from `ftp.h` to `ftp.c`.
- `curl_memory.h` fixes to make it work when included repeatedly.
- stop building/linking curlx bits twice for a static-mode curl tool.
These caused doubly defined symbols in unity builds.
- silence missing extern declarations compiler warning for ` _CRT_glob`.
- fix extern declarations for `tool_freq` and `tool_isVistaOrGreater`.
- fix colliding static symbols in debug mode: `debugtime()` and
`statename`.
- rename `ssl_backend_data` structure to unique names for each
TLS-backend, along with the `ssl_connect_data` struct member
referencing them. This required adding casts for each access.
- add workaround for missing `[P]UNICODE_STRING` types in certain Windows
builds when compiling `lib/ldap.c`. To support "unity" builds, we had
to enable `SCHANNEL_USE_BLACKLISTS` for Schannel (a Windows
`schannel.h` option) _globally_. This caused an indirect inclusion of
Windows `schannel.h` from `ldap.c` via `winldap.h` to have it enabled
as well. This requires `[P]UNICODE_STRING` types, which is apperantly
not defined automatically (as seen with both MSVS and mingw-w64).
This patch includes `<subauth.h>` to fix it.
Ref: https://github.com/curl/curl/runs/13987772013
Ref: https://dev.azure.com/daniel0244/curl/_build/results?buildId=15827&view=logs&jobId=2c9f582d-e278-56b6-4354-f38a4d851906&j=2c9f582d-e278-56b6-4354-f38a4d851906&t=90509b00-34fa-5a81-35d7-5ed9569d331c
- tweak unity builds to compile `lib/memdebug.c` separately in memory
trace builds to avoid PP confusion.
- force-disable unity for test programs.
- do not compile and link libcurl sources to libtests _twice_ when libcurl
is built in static mode.

KNOWN ISSUES:
- running tests with unity builds may fail in cases.
- some build configurations/env may not compile in unity mode. E.g.:
https://ci.appveyor.com/project/curlorg/curl/builds/47230972/job/51wfesgnfuauwl8q#L250

Ref: https://github.com/libssh2/libssh2/issues/1034
Ref: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html
Ref: https://en.wikipedia.org/wiki/Unity_build

Closes #11095

show more ...

checksrc: disallow spaces before labels

Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
th

checksrc: disallow spaces before labels

Out of 415 labels throughout the code base, 86 of those labels were
not at the start of the line. Which means labels always at the start of
the line is the favoured style overall with 329 instances.

Out of the 86 labels not at the start of the line:
* 75 were indented with the same indentation level of the following line
* 8 were indented with exactly one space
* 2 were indented with one fewer indentation level then the following
line
* 1 was indented with the indentation level of the following line minus
three space (probably unintentional)

Co-Authored-By: Viktor Szakats

Closes #11134

show more ...

checksrc: find bad indentation in conditions without open brace

If the previous line starts with if/while/for AND ends with a closed
parenthesis and there's an equal number of open and c

checksrc: find bad indentation in conditions without open brace

If the previous line starts with if/while/for AND ends with a closed
parenthesis and there's an equal number of open and closed parentheses
on that line, verify that this line is indented $indent more steps, if
not a cpp line.

Also adjust the fall-out from this fix.

Closes #11054

show more ...

checksrc: fix SPACEBEFOREPAREN for conditions starting with "*"

The open paren check wants to warn for spaces before open parenthesis
for if/while/for but also for any function call. In

checksrc: fix SPACEBEFOREPAREN for conditions starting with "*"

The open paren check wants to warn for spaces before open parenthesis
for if/while/for but also for any function call. In order to avoid
catching function pointer declarations, the logic allows a space if the
first character after the open parenthesis is an asterisk.

I also spotted what we did not include "switch" in the check but we should.

This check is a little lame, but we reduce this problem by not allowing
that space for if/while/for/switch.

Reported-by: Emanuele Torre
Closes #11044

show more ...

lib: unify the upload/method handling

By making sure we set state.upload based on the set.method value and not
independently as set.upload, we reduce confusion and mixup risks, both

lib: unify the upload/method handling

By making sure we set state.upload based on the set.method value and not
independently as set.upload, we reduce confusion and mixup risks, both
internally and externally.

Closes #11017

show more ...

ftp: fix 'portsock' variable was assigned the same value

Pointed out by PVS

Ref: #10929
Closes #10955

ftp: remove dead code

This condition can never be true here since it is handled already 28
lines above.

Pointed out by PVS.

Ref: #10929
Closes #10957

multi: remove PENDING + MSGSENT handles from the main linked list

As they are not driving transfers or any socket activity, the main loop
does not need to iterate over these handles. A p

multi: remove PENDING + MSGSENT handles from the main linked list

As they are not driving transfers or any socket activity, the main loop
does not need to iterate over these handles. A performance improvement.

They are instead only held in their own separate lists.

'data->multi' is kept a pointer to the multi handle as long as the easy
handle is actually part of it even when the handle is moved to the
pending/msgsent lists. It needs to know which multi handle it belongs
to, if for example curl_easy_cleanup() is called before the handle is
removed from the multi handle.

Alll 'data->multi' pointers of handles still part of the multi handle
gets cleared by curl_multi_cleanup() which "orphans" all previously
attached easy handles.

This is take 2. The first version was reverted for the 8.0.1 release.

Assisted-by: Stefan Eissing
Closes #10801

show more ...

ftp: add more conditions for connection reuse

Reported-by: Harry Sintonen
Closes #10730

ftp: make the 'ftpauth' a more normal 'char *'-array

Closes #10703

ftp: active mode with SSL, add the damn filter

- since 7.87.0 we lost adding the SSL filter for an active
FTP connection that uses SSL. This leads to hangers and timeouts
as repo

ftp: active mode with SSL, add the damn filter

- since 7.87.0 we lost adding the SSL filter for an active
FTP connection that uses SSL. This leads to hangers and timeouts
as reported in #10666.

Reported-by: SandakovMM on github
Fixes #10666
Closes #10669

show more ...

ftp: allocate the wildcard struct on demand

The feature is rarely used so this frees up data for the vast majority
of easy handles that don't use it.

Rename "protdata" to "ftpwc

ftp: allocate the wildcard struct on demand

The feature is rarely used so this frees up data for the vast majority
of easy handles that don't use it.

Rename "protdata" to "ftpwc" since it is always an FTP wildcard struct
pointer. Made the state struct field an unsigned char to save space.

Closes #10639

show more ...

setopt: move the CURLOPT_CHUNK_DATA pointer to the set struct

To make duphandle work etc

Closes #10635

ftp: replace sscanf for MDTM 213 response parsing

Closes #10590

ftp: replace sscanf for PASV parsing

Closes #10590

ftp: make the EPSV response parser not use sscanf

Closes #10590

lib: fix typos

Closes https://github.com/curl/curl/pull/10307

misc: reduce struct and struct field sizes

- by using BIT() instead of bool
- imap: shrink struct
- ftp: make state 'unsigned char'
- ftp: sort ftp_conn struct entries on size

misc: reduce struct and struct field sizes

- by using BIT() instead of bool
- imap: shrink struct
- ftp: make state 'unsigned char'
- ftp: sort ftp_conn struct entries on size
- urldata: use smaller fields for SSL version info storage
- pop3: reduce the pop3_conn struct size
- smtp: reduce the size of the smtp structs

Closes #10186

show more ...

cf-socket: keep sockaddr local in the socket filters

- copy `struct Curl_addrinfo` on filter setup into context
- remove `struct Curl_addrinfoi *` with `struct Curl_sockaddr_ex *` in

cf-socket: keep sockaddr local in the socket filters

- copy `struct Curl_addrinfo` on filter setup into context
- remove `struct Curl_addrinfoi *` with `struct Curl_sockaddr_ex *` in
connectdata that is set and NULLed by the socket filter
- this means we have no reference to the resolver info in connectdata or
its filters
- trigger the CF_CTRL_CONN_INFO_UPDATE event when the complete filter
chain reaches connected status
- update easy handle connection information on CF_CTRL_DATA_SETUP event.

Closes #10213

show more ...