build: fix clang-cl builds, add CI job

- appveyor: add build-only job for clang-cl.

- cmake: `-pedantic-errors` enables `-Werror,-Wlanguage-extension-token`
automatically, whi

build: fix clang-cl builds, add CI job

- appveyor: add build-only job for clang-cl.

- cmake: `-pedantic-errors` enables `-Werror,-Wlanguage-extension-token`
automatically, which makes `__int64` detection fail.
Explictly disable this compiler warning for clang-cl to make the
feature detection work and to accept `__int64` in the source code.

- cmake: disable `-Wlanguage-extension-token` warning for clang-cl
to fix these when encountering `__int64`:
```
lib/formdata.c(797,29): error : extension used [-Werror,-Wlanguage-extension-token]
lib/warnless.c(117,33): error : extension used [-Werror,-Wlanguage-extension-token]
lib/warnless.c(60,28): message : expanded from macro 'CURL_MASK_SCOFFT'
lib/warnless.c(59,38): message : expanded from macro 'CURL_MASK_UCOFFT'
include\curl/system.h(352,40): message : expanded from macro 'CURL_TYPEOF_CURL_OFF_T'
```

- make `__GNUC__` warning suppressions apply to `__clang__` too.
Necessary for clang-cl, which defines the latter, but not the former.
(Regular clang defines both.)

- examples: fix clang-cl compiler warning in `http2-upload.c`.
```
docs\examples\http2-upload.c(56,5): error : no previous prototype for function 'my_gettimeofday' [-Werror,-Wmissing-prototypes]
docs\examples\http2-upload.c(56,1): message : declare 'static' if the function is not intended to be used outside of this translation unit
```

- unit2604: add missing `#pragma GCC diagnostic pop`.
Follow-up to e53523fef07894991c69d907a7c7794c7ada4ff4 #14859

- unit1652: limit compiler warning suppression to GCC.
They do not affect clang builds.
Follow-up to 71cf0d1fca9e1f53524e1545ef0c08d174458d80 #14772

Closes #15449

show more ...

lib/src: white space edits to comply better with code style

... as checksrc now finds and complains about these.

Closes #14921

content_encoding: avoid getting all encodings unless necessary

The error_do_write() function may very well return witout needing the
listing of all encoding types so postpone that call u

content_encoding: avoid getting all encodings unless necessary

The error_do_write() function may very well return witout needing the
listing of all encoding types so postpone that call until it is needed.

Closes #14831

show more ...

tidy-up: misc spelling (bit, ASCII)

Closes #14559

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname

code: language cleanup in comments

Based on the standards and guidelines we use for our documentation.

- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period

Closes #14073

show more ...

lib: tidy up types and casts

Cherry-picked from #13489
Closes #13862

test: add test1546, chunked not last transfer encoding

with more than one transfer-encoding, 'chunked' must be the last added
to the writer stack (and therefore the first to decode). RFC

test: add test1546, chunked not last transfer encoding

with more than one transfer-encoding, 'chunked' must be the last added
to the writer stack (and therefore the first to decode). RFC 9112, ch.
6.1.

Closes #13736

show more ...

content_encoding: reject transfer-encoding after chunked

reject a response that applies a transfer-encoding after a 'chunked'
encoding. RFC 9112 ch. 6.1 required chunked to be the final

content_encoding: reject transfer-encoding after chunked

reject a response that applies a transfer-encoding after a 'chunked'
encoding. RFC 9112 ch. 6.1 required chunked to be the final encoding.

Closes #13733

show more ...

content_encoding: ignore duplicate chunked encoding

- ignore duplicate "chunked" transfer-encodings from
a server to accomodate for broken implementations
- add test1482 and test14

content_encoding: ignore duplicate chunked encoding

- ignore duplicate "chunked" transfer-encodings from
a server to accomodate for broken implementations
- add test1482 and test1483

Reported-by: Mel Zuser
Fixes #13451
Closes #13461

show more ...

content_encoding: brotli and others, pass through 0-length writes

- curl's transfer handling may write 0-length chunks at the end of the
download with an EOS flag. (HTTP/2 does this co

content_encoding: brotli and others, pass through 0-length writes

- curl's transfer handling may write 0-length chunks at the end of the
download with an EOS flag. (HTTP/2 does this commonly)

- content encoders need to pass-through such a write and not count this
as error in case they are finished decoding

Fixes #13209
Fixes #13212
Closes #13219

show more ...

lib: replace readwrite with write_resp

This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
m

lib: replace readwrite with write_resp

This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.

The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.

Curl's protocol handler `readwrite()` method been changed:

```diff
- CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
- const char *buf, size_t blen,
- size_t *pconsumed, bool *readmore);
+ CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+ bool is_eos, bool *done);
```

The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:

* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
finished

This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.

The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.

HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.

The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:

```
static CURLcode readwrite_data(data, ...)
{
do {
Curl_xfer_recv_resp(data, buf)
...
Curl_xfer_write_resp(data, buf)
...
} while(interested);
...
}
```

All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.

All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.

Closes #12480

show more ...

content_encoding: change return code to typedef'ed enum

... to work around a clang ubsan warning.

Fixes #12618
Closes #12622

content_encoding: make Curl_all_content_encodings allocless

- Fixes a memory leak pointed out by Coverity
- Also found by OSS-Fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=

content_encoding: make Curl_all_content_encodings allocless

- Fixes a memory leak pointed out by Coverity
- Also found by OSS-Fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63947
- Avoids unncessary allocations

Follow-up ad051e1cbec68b2456a22661b

Closes #12289

show more ...

lib: client writer, part 2, accounting + logging

This PR has these changes:

Renaming of unencode_* to cwriter, e.g. client writers
- documentation of sendf.h functions
- mov

lib: client writer, part 2, accounting + logging

This PR has these changes:

Renaming of unencode_* to cwriter, e.g. client writers
- documentation of sendf.h functions
- move max decode stack checks back to content_encoding.c
- define writer phase which was used as order before
- introduce phases for monitoring inbetween decode phases
- offering default implementations for init/write/close

Add type paramter to client writer's do_write()
- always pass all writes through the writer stack
- writers who only care about BODY data will pass other writes unchanged

add RAW and PROTOCOL client writers
- RAW used for Curl_debug() logging of CURLINFO_DATA_IN
- PROTOCOL used for updates to data->req.bytecount, max_filesize checks and
Curl_pgrsSetDownloadCounter()
- remove all updates of data->req.bytecount and calls to
Curl_pgrsSetDownloadCounter() and Curl_debug() from other code
- adjust test457 expected output to no longer see the excess write

Closes #12184

show more ...

lib: move handling of `data->req.writer_stack` into Curl_client_write()

- move definitions from content_encoding.h to sendf.h
- move create/cleanup/add code into sendf.c
- installed

lib: move handling of `data->req.writer_stack` into Curl_client_write()

- move definitions from content_encoding.h to sendf.h
- move create/cleanup/add code into sendf.c
- installed content_encoding writers will always be called
on Curl_client_write(CLIENTWRITE_BODY)
- Curl_client_cleanup() frees writers and tempbuffers from
paused transfers, irregardless of protocol

Closes #11908

show more ...

content_encoding: only do tranfer-encoding compression if asked to

To reduce surprises. Update test 387 and 418 accordingly.

Closes #10899

lib: silence clang/gcc -Wvla warnings in brotli headers

brotli v1.0.0 throughout current latest v1.0.9 and latest master [1]
trigger this warning.

It happened with CMake and GNU

lib: silence clang/gcc -Wvla warnings in brotli headers

brotli v1.0.0 throughout current latest v1.0.9 and latest master [1]
trigger this warning.

It happened with CMake and GNU Make. autotools builds avoid it with
the `convert -I options to -isystem` macro.

llvm/clang:
```
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:204:34: warning: variable length array used [-Wvla]
const uint8_t encoded_buffer[BROTLI_ARRAY_PARAM(encoded_size)],
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./brotli/x64-ucrt/usr/include/brotli/port.h:253:34: note: expanded from macro 'BROTLI_ARRAY_PARAM'
^~~~~~
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:206:48: warning: variable length array used [-Wvla]
uint8_t decoded_buffer[BROTLI_ARRAY_PARAM(*decoded_size)]);
~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
./brotli/x64-ucrt/usr/include/brotli/port.h:253:35: note: expanded from macro 'BROTLI_ARRAY_PARAM'
~^~~~~
```

gcc:
```
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:204:5: warning: ISO C90 forbids variable length array 'encoded_buffer' [-Wvla]
204 | const uint8_t encoded_buffer[BROTLI_ARRAY_PARAM(encoded_size)],
| ^~~~~
./brotli/x64-ucrt/usr/include/brotli/decode.h:206:5: warning: ISO C90 forbids variable length array 'decoded_buffer' [-Wvla]
206 | uint8_t decoded_buffer[BROTLI_ARRAY_PARAM(*decoded_size)]);
| ^~~~~~~
```

[1] https://github.com/google/brotli/commit/ed1995b6bda19244070ab5d331111f16f67c8054

Reviewed-by: Daniel Stenberg
Reviewed-by: Marcel Raad
Closes #10738

show more ...

content_encoding: do not reset stage counter for each header

Test 418 verifies

Closes #10492

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- save

copyright: update all copyright lines and remove year ranges

- they are mostly pointless in all major jurisdictions
- many big corporations and projects already don't use them
- saves us from pointless churn
- git keeps history for us
- the year range is kept in COPYING

checksrc is updated to allow non-year using copyright statements

Closes #10205

show more ...

http: decode transfer encoding first

The unencoding stack is added to as Transfer-Encoding and
Content-Encoding fields are encountered with no distinction between the
two, meaning th

http: decode transfer encoding first

The unencoding stack is added to as Transfer-Encoding and
Content-Encoding fields are encountered with no distinction between the
two, meaning the stack will be incorrect if, e.g., the message has both
fields and a non-chunked Transfer-Encoding comes first. This commit
fixes this by ordering the stack with transfer encodings first.

Reviewed-by: Patrick Monnerat
Closes #10187

show more ...

tidy-up: delete parallel/unused feature flags

Detecting headers and lib separately makes sense when headers come in
variations or with extra ones, but this wasn't the case here. These we

tidy-up: delete parallel/unused feature flags

Detecting headers and lib separately makes sense when headers come in
variations or with extra ones, but this wasn't the case here. These were
duplicate/parallel macros that we had to keep in sync with each other
for a working build. This patch leaves a single macro for each of these
dependencies:

- Rely on `HAVE_LIBZ`, delete parallel `HAVE_ZLIB_H`.

Also delete CMake logic making sure these two were in sync, along with
a toggle to turn off that logic, called `CURL_SPECIAL_LIBZ`.

Also delete stray `HAVE_ZLIB` defines.

There is also a `USE_ZLIB` variant in `lib/config-dos.h`. This patch
retains it for compatibility and deprecates it.

- Rely on `USE_LIBSSH2`, delete parallel `HAVE_LIBSSH2_H`.

Also delete `LIBSSH2_WIN32`, `LIBSSH2_LIBRARY` from
`winbuild/MakefileBuild.vc`, these have a role when building libssh2
itself. And `CURL_USE_LIBSSH`, which had no use at all.

Also delete stray `HAVE_LIBSSH2` defines.

- Rely on `USE_LIBSSH`, delete parallel `HAVE_LIBSSH_LIBSSH_H`.

Also delete `LIBSSH_WIN32`, `LIBSSH_LIBRARY` and `HAVE_LIBSSH` from
`winbuild/MakefileBuild.vc`, these were the result of copy-pasting the
libssh2 line, and were not having any use.

- Delete unused `HAVE_LIBPSL_H` and `HAVE_LIBPSL`.

Reviewed-by: Daniel Stenberg

Closes #9652

show more ...

content_encoding: use writer struct subclasses for different encodings

The variable-sized encoding-specific storage of a struct contenc_writer
currently relies on void * alignment that m

content_encoding: use writer struct subclasses for different encodings

The variable-sized encoding-specific storage of a struct contenc_writer
currently relies on void * alignment that may be insufficient with
regards to the specific storage fields, although having not caused any
problems yet.

In addition, gcc 11.3 issues a warning on access to fields of partially
allocated structures that can occur when the specific storage size is 0:

content_encoding.c: In function ‘Curl_build_unencoding_stack’:
content_encoding.c:980:21: warning: array subscript ‘struct contenc_writer[0]’ is partly outside array bounds of ‘unsigned char[16]’ [-Warray-bounds]
980 | writer->handler = handler;
| ~~~~~~~~~~~~~~~~^~~~~~~~~
In file included from content_encoding.c:49:
memdebug.h:115:29: note: referencing an object of size 16 allocated by ‘curl_dbg_calloc’
115 | #define calloc(nbelem,size) curl_dbg_calloc(nbelem, size, __LINE__, __FILE__)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
content_encoding.c:977:60: note: in expansion of macro ‘calloc’
977 | struct contenc_writer *writer = (struct contenc_writer *)calloc(1, sz);

To solve both these problems, the current commit replaces the
contenc_writer/params structure pairs by "subclasses" of struct
contenc_writer. These are structures that contain a contenc_writer at
offset 0. Proper field alignment is therefore handled by the compiler and
full structure allocation is performed, silencing the warnings.

Closes #9455

show more ...

misc: ISSPACE() => ISBLANK()

Instances of ISSPACE() use that should rather use ISBLANK(). I think
somewhat carelessly used because it sounds as if it checks for space or
whitespace,

misc: ISSPACE() => ISBLANK()

Instances of ISSPACE() use that should rather use ISBLANK(). I think
somewhat carelessly used because it sounds as if it checks for space or
whitespace, but also includes %0a to %0d.

For parsing purposes, we should only accept what we must and not be
overly liberal. It leads to surprises and surprises lead to bad things.

Closes #9432

show more ...

content_encoding: return error on too many compression steps

The max allowed steps is arbitrarily set to 5.

Bug: https://curl.se/docs/CVE-2022-32206.html
CVE-2022-32206
Repo

content_encoding: return error on too many compression steps

The max allowed steps is arbitrarily set to 5.

Bug: https://curl.se/docs/CVE-2022-32206.html
CVE-2022-32206
Reported-by: Harry Sintonen
Closes #9049

show more ...

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the

copyright: make repository REUSE compliant

Add licensing and copyright information for all files in this repository. This
either happens in the file itself as a comment header or in the file
`.reuse/dep5`.

This commit also adds a Github workflow to check pull requests and adapts
copyright.pl to the changes.

Closes #8869

show more ...