NTLM_WB: delete remains in tests, docs updates

Also update more docs to reflect the removal of NTLM_WB.

Follow-up to 50def7c881ba560ab6e0235990e8f07fa69f4bc8 #13249
Closes #14832

tracing: allow CURL_DEBUG override

On debug builds, allow environment variable CURL_DEBUG to override any
setting done via '-v' or '--no-verbose'.

Closes #14436

tidy-up: OS names

Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,

tidy-up: OS names

Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.

Mostly OS names and a few more.

Also a couple of other minor text fixups.

Closes #14360

show more ...

curldown: make 'added-in:' a mandatory header field

- generate AVAILABILITY manpage sections automatically - for consistent
wording

- allows us to double-check against other d

curldown: make 'added-in:' a mandatory header field

- generate AVAILABILITY manpage sections automatically - for consistent
wording

- allows us to double-check against other documumentation (symbols-in-versions
etc)

- enables proper automation/scripting based on this data

- lots of them were wrong or missing in the manpages

- several of them repeated (sometimes mismatching) backend support info

Add test 1488 to verify "added-in" version numbers against
symbols-in-versions.

Closes #14217

show more ...

CI: add whitespace checker

Fix issues detected.

Also:

- One of the `.vc` files used LF EOLs, while the other didn't.
Make that one also use LF EOLs, as this is appare

CI: add whitespace checker

Fix issues detected.

Also:

- One of the `.vc` files used LF EOLs, while the other didn't.
Make that one also use LF EOLs, as this is apparently supported by
`nmake`.

- Drop `.dsw` and `.btn` types from `.gitattributes`.
The repository doesn't use them.

- Sync section order with the rest of files in
`tests/certs/EdelCurlRoot-ca.prm`.

- Indent/align `.prm` and `.pem` files.

- Delete dummy `[something]` section from `.prm` and `.pem` files.

Mental note:
MSVC `.sln` files seem to accept spaces for indentation and also support
LF line-endings. I cannot test this and I don't know what's more
convenient when updating them, so left them as-is, with specific
exclusions.

Closes #14031

show more ...

lib: graceful connection shutdown

When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phas

lib: graceful connection shutdown

When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.

Closes #13976

show more ...

docs/libcurl: generate PROTOCOLS from meta-data

Remove the PROTOCOLS section from the source files completely and
instead generate them based on the header data in the curldown files.

docs/libcurl: generate PROTOCOLS from meta-data

Remove the PROTOCOLS section from the source files completely and
instead generate them based on the header data in the curldown files.

It also generates TLS backend information for options marked for TLS as
protocol.

Closes #13175

show more ...

docs: make each libcurl man specify protocol(s)

The mandatory header now has a mandatory list of protocols for which the
manpage is relevant.

Most man pages already has a "PROTO

docs: make each libcurl man specify protocol(s)

The mandatory header now has a mandatory list of protocols for which the
manpage is relevant.

Most man pages already has a "PROTOCOLS" section, but this introduces a
stricter way to specify the relevant protocols.

cd2nroff verifies that at least one protocol is mentioned (which can be
`*`).

This information is not used just yet, but A) the PROTOCOLS section can
now instead get generated and get a unified wording across all manpages
and B) this allows us to more reliably filter/search for protocol
specific manpages/options.

Closes #13166

show more ...

curldown: Fix email address in Copyright

The curldown conversion accidentally replaced daniel@haxx.se with
just daniel.se. This reverts back to the proper email address in
the curld

curldown: Fix email address in Copyright

The curldown conversion accidentally replaced daniel@haxx.se with
just daniel.se. This reverts back to the proper email address in
the curldown docs as well as in a few other stray places where it
was incorrect (while unrelated to curldown).

Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Closes: #12997

show more ...

websocket: fix curl_ws_recv()

- when data arrived in several chunks, the collection into
the passed buffer always started at offset 0, overwriting
the data already there.

websocket: fix curl_ws_recv()

- when data arrived in several chunks, the collection into
the passed buffer always started at offset 0, overwriting
the data already there.

adding test_20_07 to verify fix

- debug environment var CURL_WS_CHUNK_SIZE can be used to
influence the buffer chunk size used for en-/decoding.

Closes #12945

show more ...

GHA: add a job scanning for "bad words" in markdown

This means words, phrases or things we have decided not to use - words that
are spelled right according to the dictionary but we want

GHA: add a job scanning for "bad words" in markdown

This means words, phrases or things we have decided not to use - words that
are spelled right according to the dictionary but we want to avoid. In the
name of consistency and better documentation.

Closes #12764

show more ...

docs: introduce "curldown" for libcurl man page format

curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of

docs: introduce "curldown" for libcurl man page format

curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)

tools:

- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown

This setup generates .3 versions of all the curldown versions at build time.

CI:

Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...

Closes #12730

show more ...