tidy-up: whitespace [ci skip]

request: paused upload on completed download, assess connection

A transfer with a completed download that is still uploading needs to
check the connection state when it is PAUSEd, since

request: paused upload on completed download, assess connection

A transfer with a completed download that is still uploading needs to
check the connection state when it is PAUSEd, since connection
close/errors would otherwise go unnoticed.

Reported-by: Sergey Bronnikov
Fixes #13260
Closes #13271

show more ...

http: expect 100 rework

Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general

http: expect 100 rework

Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general events like a timer.

HTTP installs a `CURL_CR_PROTOCOL` reader when announcing `Expect:
100-continue`. That reader works as follows:

- on first invocation, records time, starts the `EXPIRE_100_TIMEOUT`
timer, disables `KEEP_SEND`, enables `KEEP_SEND_TIMER` and returns 0,
eos=FALSE like a paused upload.

- on subsequent invocation it checks if the timer has expired. If so, it
enables `KEEP_SEND` and switches to passing through reads to the
underlying readers.

Transfer handling's `readwrite()` will be invoked when a timer expires
(like `EXPIRE_100_TIMEOUT`) or when data from the server arrives. Seeing
`KEEP_SEND_TIMER`, it will try to upload more data, which triggers
reading from the client readers again. Which then may lead to a new
pausing or cause the upload to start.

Flags and timestamps connected to this have been moved from
`SingleRequest` into the reader's context.

Closes #13110

show more ...

pytest: Scorecard tracking CPU and RSS

Closes #12765

transfer: fix upload rate limiting, add test cases

- add test cases for rate limiting uploads for all
http versions
- fix transfer loop handling of limits. Signal a re-receive

transfer: fix upload rate limiting, add test cases

- add test cases for rate limiting uploads for all
http versions
- fix transfer loop handling of limits. Signal a re-receive
attempt only on exhausting maxloops without an EAGAIN
- fix `data->state.selectbits` forcing re-receive to also
set re-sending when transfer is doing this.

Reported-by: Karthikdasari0423 on github
Fixes #12559
Closes #12586

show more ...

http2: improved on_stream_close/data_done handling

- there seems to be a code path that cleans up easy handles without
triggering DONE or DETACH events to the connection filters. This

http2: improved on_stream_close/data_done handling

- there seems to be a code path that cleans up easy handles without
triggering DONE or DETACH events to the connection filters. This
would explain wh nghttp2 still holds stream user data
- add GOOD check to easy handle used in on_close_callback to
prevent crashes, ASSERTs in debug builds.
- NULL the stream user data early before submitting RST
- add checks in on_stream_close() to identify UNGOOD easy handles

Reported-by: Hans-Christian Egtvedt
Fixes #10936
Closes #12562

show more ...

pytest: use lower count in repeat tests

- lower large iteration counts in some tests somewhat for
the same coverage with less duration

Closes #12248

http: fix sending of large requests

- refs #11342 where errors with git https interactions
were observed
- problem was caused by 1st sends of size larger than 64KB
which resu

http: fix sending of large requests

- refs #11342 where errors with git https interactions
were observed
- problem was caused by 1st sends of size larger than 64KB
which resulted in later retries of 64KB only
- limit sending of 1st block to 64KB
- adjust h2/h3 filters to cope with parsing the HTTP/1.1
formatted request in chunks

- introducing Curl_nwrite() as companion to Curl_write()
for the many cases where the sockindex is already known

Fixes #11342 (again)
Closes #11803

show more ...

http2: polish things around POST

- added test cases for various code paths
- fixed handling of blocked write when stream had
been closed inbetween attempts
- re-enabled DEBUGAS

http2: polish things around POST

- added test cases for various code paths
- fixed handling of blocked write when stream had
been closed inbetween attempts
- re-enabled DEBUGASSERT on send with smaller data size

- in debug builds, environment variables can be set to simulate a slow
network when sending data. cf-socket.c and vquic.c support
* CURL_DBG_SOCK_WBLOCK: percentage of send() calls that should be
answered with a EAGAIN. TCP/UNIX sockets.
This is chosen randomly.
* CURL_DBG_SOCK_WPARTIAL: percentage of data that shall be written
to the network. TCP/UNIX sockets.
Example: 80 means a send with 1000 bytes would only send 800
This is applied to every send.
* CURL_DBG_QUIC_WBLOCK: percentage of send() calls that should be
answered with EAGAIN. QUIC only.
This is chosen randomly.

Closes #11756

show more ...

tidy-up: mostly whitespace nits

- delete completed TODO from `./CMakeLists.txt`.
- convert a C++ comment to C89 in `./CMake/CurlTests.c`.
- delete duplicate EOLs from EOF.
- add

tidy-up: mostly whitespace nits

- delete completed TODO from `./CMakeLists.txt`.
- convert a C++ comment to C89 in `./CMake/CurlTests.c`.
- delete duplicate EOLs from EOF.
- add missing EOL at EOF.
- delete whitespace at EOL (except from expected test results).
- convert tabs to spaces.
- convert CRLF EOLs to LF in GHA yaml.
- text casing fixes in `./CMakeLists.txt`.
- fix a codespell typo in `packages/OS400/initscript.sh`.

Closes #11772

show more ...

test_07_upload.py: fix test_07_34 curl args

- Pass correct filename to --data-binary.

Prior to this change --data-binary was passed an incorrect filename due
to a missing separa

test_07_upload.py: fix test_07_34 curl args

- Pass correct filename to --data-binary.

Prior to this change --data-binary was passed an incorrect filename due
to a missing separator in the arguments list. Since aacbeae7 curl will
error on incorrect filenames for POST.

Fixes https://github.com/curl/curl/issues/11761
Closes https://github.com/curl/curl/pull/11763

show more ...

http2: upgrade tests and add fix for non-existing stream

- check in h2 filter recv that stream actually exists
and return error if not
- add test for parallel, extreme h2 upgrades

http2: upgrade tests and add fix for non-existing stream

- check in h2 filter recv that stream actually exists
and return error if not
- add test for parallel, extreme h2 upgrades that fail if
connections get reused before fully switched
- add h2 upgrade upload test just for completeness

Closes #11563

show more ...

http2: fix EOF handling on uploads with auth negotiation

- doing a POST with `--digest` does an override on the initial request
with `Content-Length: 0`, but the http2 filter was unawa

http2: fix EOF handling on uploads with auth negotiation

- doing a POST with `--digest` does an override on the initial request
with `Content-Length: 0`, but the http2 filter was unaware of that
and expected the originally request body. It did therefore not
send a final DATA frame with EOF flag to the server.
- The fix overrides any initial notion of post size when the `done_send`
event is triggered by the transfer loop, leading to the EOF that
is necessary.
- refs #11194. The fault did not happen in testing, as Apache httpd
never tries to read the request body of the initial request,
sends the 401 reply and closes the stream. The server used in the
reported issue however tried to read the EOF and timed out on the
request.

Reported-by: Aleksander Mazur
Fixes #11194
Cloes #11200

show more ...

http/2: unstick uploads

- refs #11157 and #11175 where uploads get stuck or lead to RST streams
- fixes our h2 send behaviour to continue sending in the nghttp2 session
as long as

http/2: unstick uploads

- refs #11157 and #11175 where uploads get stuck or lead to RST streams
- fixes our h2 send behaviour to continue sending in the nghttp2 session
as long as it wants to. This will empty our send buffer as long as
the remote stream/connection window allows.
- in case the window is exhausted, the data remaining in the send buffer
will wait for a WINDOW_UPDATE from the server. Which is a socket event
that engages our transfer loop again
- the problem in the issue was that we did not exhaust the window, but
left data in the sendbuffer and no further socket events did happen.
The server was just waiting for us to send more.
- relatedly, there was an issue fixed that closing a stream with KEEP_HOLD
set kept the transfer from shutting down - as it should have - leading
to a timeout.

Closes #11176

show more ...

http2: upload improvements

Make send buffer smaller to have progress and "upload done" reporting
closer to reality. Fix handling of send "drain" condition to no longer
trigger once t

http2: upload improvements

Make send buffer smaller to have progress and "upload done" reporting
closer to reality. Fix handling of send "drain" condition to no longer
trigger once the transfer loop reports it is done sending. Also do not
trigger the send "drain" on RST streams.

Background:
- a upload stall was reported in #11157 that timed out
- test_07_33a reproduces a problem with such a stall if the
server 404s the request and RSTs the stream.
- test_07_33b verifies a successful PUT, using the parameters
from #11157 and checks success

Ref: #11157
Closes #11165

show more ...

cf-socket: completely remove the disabled USE_RECV_BEFORE_SEND_WORKAROUND

Closes #11118

proxy: http2 proxy tunnel implementation

- currently only on debug build and when env variable
CURL_PROXY_TUNNEL_H2 is present.
- will ALPN negotiate with the proxy server and swit

proxy: http2 proxy tunnel implementation

- currently only on debug build and when env variable
CURL_PROXY_TUNNEL_H2 is present.
- will ALPN negotiate with the proxy server and switch
tunnel filter based on the protocol negotiated.
- http/1.1 tunnel code moved into cf-h1-proxy.[ch]
- http/2 tunnel code implemented in cf-h2-proxy.[ch]
- tunnel start and ALPN set remains in http_proxy.c
- moving all haproxy related code into cf-haproxy.[ch]

VTLS changes
- SSL filters rely solely on the "alpn" specification they
are created with and no longer check conn->bits.tls_enable_alpn.
- checks on which ALPN specification to use (or none at all) are
done in vtls.c when creating the filter.

Testing
- added a nghttpx forward proxy to the pytest setup that
speaks HTTP/2 and forwards all requests to the Apache httpd
forward proxy server.
- extending test coverage in test_10 cases
- adding proxy tests for direct/tunnel h1/h2 use of basic auth.
- adding test for http/1.1 and h2 proxy tunneling to pytest

Closes #10780

show more ...

http3: improvements across backends

- ngtcp2: using bufq for recv stream data
- internal stream_ctx instead of `struct HTTP` members
for quiche, ngtcp2 and msh3
- no more QUIC

http3: improvements across backends

- ngtcp2: using bufq for recv stream data
- internal stream_ctx instead of `struct HTTP` members
for quiche, ngtcp2 and msh3
- no more QUIC related members in `struct HTTP`
- experimental use of recvmmsg(), disabled by default
- testing on my old debian box shows no throughput improvements.
- leaving it in, but disabled, for future revisit
- vquic: common UDP receive code for ngtcp2 and quiche
- vquic: common UDP send code for ngtcp2 and quiche
- added pytest skips for known msh3 failures
- fix unit2601 to survive torture testing
- quiche: using latest `master` from quiche and enabling large download
tests, now that key change is supported
- fixing test_07_21 where retry handling of starting a stream
was faulty
- msh3: use bufq for recv buffering headers and data
- msh3: replace fprintf debug logging with LOG_CF where possible
- msh3: force QUIC expire timers on recv/send to have more than
1 request per second served

Closes #10772

show more ...

http2: flow control and buffer improvements

- use bufq for send/receive of network data
- usd bufq for send/receive of stream data
- use HTTP/2 flow control with no-auto updates to c

http2: flow control and buffer improvements

- use bufq for send/receive of network data
- usd bufq for send/receive of stream data
- use HTTP/2 flow control with no-auto updates to control the
amount of data we are buffering for a stream
HTTP/2 stream window set to 128K after local tests, defined
code constant for now
- elminiating PAUSEing nghttp2 processing when receiving data
since a stream can now take in all DATA nghttp2 forwards

Improved scorecard and adjuste http2 stream window sizes
- scorecard improved output formatting and options default
- scorecard now also benchmarks small requests / second

Closes #10771

show more ...

pytest: improvements for suitable curl and error output

- will check built curl for http and https support and
skip all tests if not there
- will dump stdout/stderr/trace output on

pytest: improvements for suitable curl and error output

- will check built curl for http and https support and
skip all tests if not there
- will dump stdout/stderr/trace output on errored responses

Closes #10829

show more ...

tests/http: remove year ranges from copyrights

Closes #10763

tests/http: add pytest to GHA and improve tests

- added to: ngtcp2-quictls, ngtcp2-gnutls and the linux varians
quiche, bearssl, libressl, mbedtls, openssl3, rustls
- added disable

tests/http: add pytest to GHA and improve tests

- added to: ngtcp2-quictls, ngtcp2-gnutls and the linux varians
quiche, bearssl, libressl, mbedtls, openssl3, rustls
- added disabled in ngtcp2-wolfssl due to weird SSL_connect() errors
not reproducable locally

Improvements on pytest:

- handling of systems with nghttpx in $PATH
- configure will seach $PATH got nghttpx used in pytest
- pytest fixes for managing nghttpx without h3 support
- ngtcp2-wolfssl: use a fully enabled wolfssl build

- lower parallel count for http/1.1 tests, since we do not
want to test excessive connections.
- check built curl for HTTPS-proxy support in proxy tests
- bearssl does not like one of our critical cert extensions, making
it non-critical now
- bearssl is too slow for test_12, skipping
- making sure we do h3 tests only when curl and server support is there

Closes #10699

show more ...

tests: rename tests/tests-httpd to tests/http

- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more b

tests: rename tests/tests-httpd to tests/http

- httpd is only one server we test with
- the suite coveres the HTTP protocol in general where
the default test cases need a more beefy environment

Closes #10654

show more ...