http2: auto reset stream on server eos

When a server signals EOS from its side and the curl upload is
unfinished and the server has not given a positive HTTP status response,
auto RS

http2: auto reset stream on server eos

When a server signals EOS from its side and the curl upload is
unfinished and the server has not given a positive HTTP status response,
auto RST the stream to signal that the upload is incomplete and that the
whole transfer can be stopped.

Fixes the case where the server responds with 413 on an upload but does
not RST the stream from its side, as httpd and others do.

Reported-by: jkamp-aws on github
Fixes #15316
Closes #15325

show more ...

lib/src: white space edits to comply better with code style

... as checksrc now finds and complains about these.

Closes #14921

lib, src, tests: added space around ternary expressions

Closes #14912

misc: general C style cleanups

... in code that previously was not checksrc'ed

Closes #14625

pytests: scorecard upload tests

- add upload tests to scorecard, invoke with
> python3 tests/http/scorecard.py -u h1|h2|h3
- add a reverse proxy setup from Caddy to httpd for

pytests: scorecard upload tests

- add upload tests to scorecard, invoke with
> python3 tests/http/scorecard.py -u h1|h2|h3
- add a reverse proxy setup from Caddy to httpd for
upload tests since Caddy does not have other PUT/POST handling
- add caddy tests in test_08 for POST/PUT
- increase read buffer in mod_curltest for larger reads

Closes #14208

show more ...

lib: graceful connection shutdown

When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phas

lib: graceful connection shutdown

When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.

Closes #13976

show more ...

transfer: conn close on paused upload

- add 2 variations on test_07_42 which PAUSEs uploads
and response connections terminating either right away
or after the 100-continue respo

transfer: conn close on paused upload

- add 2 variations on test_07_42 which PAUSEs uploads
and response connections terminating either right away
or after the 100-continue response
- when detecting the connection being closed in transfer.c
readwrite_data(), clear ALL send bits in data->req.keepon.
It no longer makes send to wait for a KEEP_SEND_PAUSE or HOLD.
- in the protocol client writer add the check for incomplete
response bodies. When an EOS is seen and the length is known,
check that and fail if bytes are missing.

Reported-by: Sergey Bronnikov
Fixes #13740
Closes #13750

show more ...

pytest: add DELETE tests, check server version

- add tests for DELETE working
- check apache version in keepalive test
- fix some comments

Closes #13679

tests: add SNI and peer name checks

- connect to DNS names with trailing dot
- connect to DNS names with double trailing dot
- rustls, always give `peer->hostname` and let it
f

tests: add SNI and peer name checks

- connect to DNS names with trailing dot
- connect to DNS names with double trailing dot
- rustls, always give `peer->hostname` and let it
figure out SNI itself
- add SNI tests for ip address and localhost
- document in code and TODO that QUIC with ngtcp2+wolfssl
does not do proper peer verification of the certificate
- mbedtls, skip tests with ip address verification as not
supported by the library

Closes #13486

show more ...

vtls: TLS session storage overhaul

- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object,

vtls: TLS session storage overhaul

- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object, freeing it also on failures
- change tls backend use
- test_17, add tests for SSL session resumption

Closes #13386

show more ...

request: paused upload on completed download, assess connection

A transfer with a completed download that is still uploading needs to
check the connection state when it is PAUSEd, since

request: paused upload on completed download, assess connection

A transfer with a completed download that is still uploading needs to
check the connection state when it is PAUSEd, since connection
close/errors would otherwise go unnoticed.

Reported-by: Sergey Bronnikov
Fixes #13260
Closes #13271

show more ...

http: expect 100 rework

Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general

http: expect 100 rework

Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general events like a timer.

HTTP installs a `CURL_CR_PROTOCOL` reader when announcing `Expect:
100-continue`. That reader works as follows:

- on first invocation, records time, starts the `EXPIRE_100_TIMEOUT`
timer, disables `KEEP_SEND`, enables `KEEP_SEND_TIMER` and returns 0,
eos=FALSE like a paused upload.

- on subsequent invocation it checks if the timer has expired. If so, it
enables `KEEP_SEND` and switches to passing through reads to the
underlying readers.

Transfer handling's `readwrite()` will be invoked when a timer expires
(like `EXPIRE_100_TIMEOUT`) or when data from the server arrives. Seeing
`KEEP_SEND_TIMER`, it will try to upload more data, which triggers
reading from the client readers again. Which then may lead to a new
pausing or cause the upload to start.

Flags and timestamps connected to this have been moved from
`SingleRequest` into the reader's context.

Closes #13110

show more ...

vtls: revert "receive max buffer" + add test case

- add test_05_04 for requests using http/1.0, http/1.1 and h2 against an
Apache resource that does an unclean TLS shutdown.
- reve

vtls: revert "receive max buffer" + add test case

- add test_05_04 for requests using http/1.0, http/1.1 and h2 against an
Apache resource that does an unclean TLS shutdown.
- revert special workarund in openssl.c for suppressing shutdown errors
on multiplexed connections
- vlts.c restore to its state before 9a90c9dd64d2f03601833a70786d485851bd1b53

Fixes #12885
Fixes #12844

Closes #12848

show more ...

transfer: fix upload rate limiting, add test cases

- add test cases for rate limiting uploads for all
http versions
- fix transfer loop handling of limits. Signal a re-receive

transfer: fix upload rate limiting, add test cases

- add test cases for rate limiting uploads for all
http versions
- fix transfer loop handling of limits. Signal a re-receive
attempt only on exhausting maxloops without an EAGAIN
- fix `data->state.selectbits` forcing re-receive to also
set re-sending when transfer is doing this.

Reported-by: Karthikdasari0423 on github
Fixes #12559
Closes #12586

show more ...

http2: error stream resets with code CURLE_HTTP2_STREAM

- refs #11357, where it was reported that HTTP/1.1 downgrades
no longer works
- fixed with suggested change
- added test

http2: error stream resets with code CURLE_HTTP2_STREAM

- refs #11357, where it was reported that HTTP/1.1 downgrades
no longer works
- fixed with suggested change
- added test_05_03 and a new handler in the curltest module
to reproduce that downgrades work

Fixes #11357
Closes #11362
Reported-by: Jay Satiro

show more ...

tests: rename tests/tests-httpd to tests/http

- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more b

tests: rename tests/tests-httpd to tests/http

- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more beefy environment

Closes #10654

show more ...