#
c294f9cb |
| 07-May-2024 |
Daniel Stenberg |
lib: make protocol handlers store scheme name lowercase - saves a lowercase operation when the "[scheme]_proxy" name is generated - appears less "shouting" - update test 970, 9
lib: make protocol handlers store scheme name lowercase - saves a lowercase operation when the "[scheme]_proxy" name is generated - appears less "shouting" - update test 970, 972, 1438 and 1536 Closes #13553
show more ...
|
#
8dd81bd5 |
| 21-Mar-2024 |
Stefan Eissing |
lib: add Curl_xfer_write_resp_hd Add method in protocol handlers to allow writing of a single, 0-terminated header line. Avoids parsing and copying these lines. Closes #13165
|
#
e3905de8 |
| 28-Feb-2024 |
Stefan Eissing |
lib: further send/upload handling polish - Move all the "upload_done" handling to request.c - add possibility to abort sending of a request - add `Curl_req_done_sending()` f
lib: further send/upload handling polish - Move all the "upload_done" handling to request.c - add possibility to abort sending of a request - add `Curl_req_done_sending()` for checks - transfer.c: readwrite_upload() now clean - removing data->state.ulbuf and data->req.upload_fromhere - as well as data->req.upload_present - set data->req.upload_done on having read all from the client and completely flushed the send buffer - tftp, remove setting of data->req.upload_fromhere - serves no purpose as `upload_present` is not set and the data itself is directly `sendto()` anyway - smtp, make upload EOB conversion a client reader - xfer_ulbuf addition - add xfer_ulbuf for borrowing, similar to xfer_buf - use in file upload - use in c-hyper body sending - h1-proxy, remove init of data->state.uilbuf that is never used - smb, add own send_buf instead of using data->state.ulbuf Closes #13010
show more ...
|
#
9369c30c |
| 15-Feb-2024 |
Stefan Eissing |
lib: Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to clarify when and at what level they operate - send/recv of transfer related
lib: Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to clarify when and at what level they operate - send/recv of transfer related data is now done via `Curl_xfer_send()/Curl_xfer_recv()` which no longer has socket/socketindex as parameter. It decides on the transfer setup of `conn->sockfd` and `conn->writesockfd` on which connection filter chain to operate. - send/recv on a specific connection filter chain is done via `Curl_conn_send()/Curl_conn_recv()` which get the socket index as parameter. - rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming consistency - clarify that the special CURLE_AGAIN hangling to return `CURLE_OK` with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is returned by all other send() variants. - fix a bug in websocket `curl_ws_recv()` that mixed up data when it arrived in more than a single chunk (to be made into a sperate PR, also) Added as documented [in CLIENT-READER.md](https://github.com/curl/curl/blob/5b1f31dfbab8aef467c419c68aa06dc738cb75d4/docs/CLIENT-READERS.md). - old `Curl_buffer_send()` completely replaced by new `Curl_req_send()` - old `Curl_fillreadbuffer()` replaced with `Curl_client_read()` - HTTP chunked uploads are now formatted in a client reader added when needed. - FTP line-end conversions are done in a client reader added when needed. - when sending requests headers, remaining buffer space is filled with body data for sending in "one go". This is independent of the request body size. Resolves #12938 as now small and large requests have the same code path. Changes done to test cases: - test513: now fails before sending request headers as this initial "client read" triggers the setup fault. Behaves now the same as in hyper build - test547, test555, test1620: fix the length check in the lib code to only fail for reads *smaller* than expected. This was a bug in the test code that never triggered in the old implementation. Closes #12969
show more ...
|
#
37551535 |
| 15-Feb-2024 |
Stefan Eissing |
lib: Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to clarify when and at what level they operate - send/recv of transfer related
lib: Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to clarify when and at what level they operate - send/recv of transfer related data is now done via `Curl_xfer_send()/Curl_xfer_recv()` which no longer has socket/socketindex as parameter. It decides on the transfer setup of `conn->sockfd` and `conn->writesockfd` on which connection filter chain to operate. - send/recv on a specific connection filter chain is done via `Curl_conn_send()/Curl_conn_recv()` which get the socket index as parameter. - rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming consistency - clarify that the special CURLE_AGAIN hangling to return `CURLE_OK` with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is returned by all other send() variants. - fix a bug in websocket `curl_ws_recv()` that mixed up data when it arrived in more than a single chunk The method for sending not just raw bytes, but bytes that are either "headers" or "body". The send abstraction stack, to to bottom, now is: * `Curl_req_send()`: has parameter to indicate amount of header bytes, buffers all data. * `Curl_xfer_send()`: knows on which socket index to send, returns amount of bytes sent. * `Curl_conn_send()`: called with socket index, returns amount of bytes sent. In addition there is `Curl_req_flush()` for writing out all buffered bytes. `Curl_req_send()` is active for requests without body, `Curl_buffer_send()` still being used for others. This is because the special quirks need to be addressed in future parts: * `expect-100` handling * `Curl_fillreadbuffer()` needs to add directly to the new `data->req.sendbuf` * special body handlings, like `chunked` encodings and line end conversions will be moved into something like a Client Reader. In functions of the pattern `CURLcode xxx_send(..., ssize_t *written)`, replace the `ssize_t` with a `size_t`. It makes no sense to allow for negative values as the returned `CURLcode` already specifies error conditions. This allows easier handling of lengths without casting. Closes #12964
show more ...
|
#
59298221 |
| 14-Feb-2024 |
Stefan Eissing |
lib: send rework Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify when and at what level they operate - sen
lib: send rework Curl_read/Curl_write clarifications - replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify when and at what level they operate - send/recv of transfer related data is now done via `Curl_xfer_send()/Curl_xfer_recv()` which no longer has socket/socketindex as parameter. It decides on the transfer setup of `conn->sockfd` and `conn->writesockfd` on which connection filter chain to operate. - send/recv on a specific connection filter chain is done via `Curl_conn_send()/Curl_conn_recv()` which get the socket index as parameter. - rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming consistency - clarify that the special CURLE_AGAIN handling to return `CURLE_OK` with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is returned by all other send() variants. SingleRequest reshuffling - move functions into request.[ch] - differentiate between reset and free - add Curl_req_done() to perform last actions - add a send `bufq` to SingleRequest for future use in keeping upload data Closes #12963
show more ...
|
#
d7b6ce64 |
| 01-Dec-2023 |
Stefan Eissing |
lib: replace readwrite with write_resp This clarifies the handling of server responses by folding the code for the complicated protocols into their protocol handlers. This concerns m
lib: replace readwrite with write_resp This clarifies the handling of server responses by folding the code for the complicated protocols into their protocol handlers. This concerns mainly HTTP and its bastard sibling RTSP. The terms "read" and "write" are often used without clear context if they refer to the connect or the client/application side of a transfer. This PR uses "read/write" for operations on the client side and "send/receive" for the connection, e.g. server side. If this is considered useful, we can revisit renaming of further methods in another PR. Curl's protocol handler `readwrite()` method been changed: ```diff - CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn, - const char *buf, size_t blen, - size_t *pconsumed, bool *readmore); + CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen, + bool is_eos, bool *done); ``` The name was changed to clarify that this writes reponse data to the client side. The parameter changes are: * `conn` removed as it always operates on `data->conn` * `pconsumed` removed as the method needs to handle all data on success * `readmore` removed as no longer necessary * `is_eos` as indicator that this is the last call for the transfer response (end-of-stream). * `done` TRUE on return iff the transfer response is to be treated as finished This change affects many files only because of updated comments in handlers that provide no implementation. The real change is that the HTTP protocol handlers now provide an implementation. The HTTP protocol handlers `write_resp()` implementation will get passed **all** raw data of a server response for the transfer. The HTTP/1.x formatted status and headers, as well as the undecoded response body. `Curl_http_write_resp_hds()` is used internally to parse the response headers and pass them on. This method is public as the RTSP protocol handler also uses it. HTTP/1.1 "chunked" transport encoding is now part of the general *content encoding* writer stack, just like other encodings. A new flag `CLIENTWRITE_EOS` was added for the last client write. This allows writers to verify that they are in a valid end state. The chunked decoder will check if it indeed has seen the last chunk. The general response handling in `transfer.c:466` happens in function `readwrite_data()`. This mainly operates now like: ``` static CURLcode readwrite_data(data, ...) { do { Curl_xfer_recv_resp(data, buf) ... Curl_xfer_write_resp(data, buf) ... } while(interested); ... } ``` All the response data handling is implemented in `Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()` implementation if available, or does the default behaviour. All raw response data needs to pass through this function. Which also means that anyone in possession of such data may call `Curl_xfer_write_resp()`. Closes #12480
show more ...
|
#
e9a7d4a1 |
| 21-Nov-2023 |
Viktor Szakats |
windows: use built-in `_WIN32` macro to detect Windows Windows compilers define `_WIN32` automatically. Windows SDK headers or build env defines `WIN32`, or we have to take care of it. T
windows: use built-in `_WIN32` macro to detect Windows Windows compilers define `_WIN32` automatically. Windows SDK headers or build env defines `WIN32`, or we have to take care of it. The agreement seems to be that `_WIN32` is the preferred practice here. Make the source code rely on that to detect we're building for Windows. Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for Windows detection, next to the official `_WIN32`. After this patch it only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`. There is a slight chance these break compatibility with Windows compilers that fail to define `_WIN32`. I'm not aware of any obsolete or modern compiler affected, but in case there is one, one possible solution is to define this macro manually. grepping for `WIN32` remains useful to discover Windows-specific code. Also: - extend `checksrc` to ensure we're not using `WIN32` anymore. - apply minor formatting here and there. - delete unnecessary checks for `!MSDOS` when `_WIN32` is present. Co-authored-by: Jay Satiro Reviewed-by: Daniel Stenberg Closes #12376
show more ...
|
#
ad051e1c |
| 23-Oct-2023 |
Stefan Eissing |
lib: client writer, part 2, accounting + logging This PR has these changes: Renaming of unencode_* to cwriter, e.g. client writers - documentation of sendf.h functions - mov
lib: client writer, part 2, accounting + logging This PR has these changes: Renaming of unencode_* to cwriter, e.g. client writers - documentation of sendf.h functions - move max decode stack checks back to content_encoding.c - define writer phase which was used as order before - introduce phases for monitoring inbetween decode phases - offering default implementations for init/write/close Add type paramter to client writer's do_write() - always pass all writes through the writer stack - writers who only care about BODY data will pass other writes unchanged add RAW and PROTOCOL client writers - RAW used for Curl_debug() logging of CURLINFO_DATA_IN - PROTOCOL used for updates to data->req.bytecount, max_filesize checks and Curl_pgrsSetDownloadCounter() - remove all updates of data->req.bytecount and calls to Curl_pgrsSetDownloadCounter() and Curl_debug() from other code - adjust test457 expected output to no longer see the excess write Closes #12184
show more ...
|
#
914e49b9 |
| 23-Sep-2023 |
Daniel Stenberg |
lib: let the max filesize option stop too big transfers too Previously it would only stop them from getting started if the size is known to be too big then. Update the libcurl a
lib: let the max filesize option stop too big transfers too Previously it would only stop them from getting started if the size is known to be too big then. Update the libcurl and curl docs accordingly. Fixes #11810 Reported-by: Elliot Killick Assisted-by: Jay Satiro Closes #11820
show more ...
|
#
2485547d |
| 05-Sep-2023 |
Stefan Eissing |
http: fix sending of large requests - refs #11342 where errors with git https interactions were observed - problem was caused by 1st sends of size larger than 64KB which resu
http: fix sending of large requests - refs #11342 where errors with git https interactions were observed - problem was caused by 1st sends of size larger than 64KB which resulted in later retries of 64KB only - limit sending of 1st block to 64KB - adjust h2/h3 filters to cope with parsing the HTTP/1.1 formatted request in chunks - introducing Curl_nwrite() as companion to Curl_write() for the many cases where the sockindex is already known Fixes #11342 (again) Closes #11803
show more ...
|
#
3f8fc257 |
| 09-May-2023 |
Viktor Szakats |
cmake: add support for "unity" builds Aka "jumbo" or "amalgamation" builds. It means to compile all sources per target as a single C source. This is experimental. You can enable
cmake: add support for "unity" builds Aka "jumbo" or "amalgamation" builds. It means to compile all sources per target as a single C source. This is experimental. You can enable it by passing `-DCMAKE_UNITY_BUILD=ON` to cmake. It requires CMake 3.16 or newer. It makes builds (much) faster, allows for better optimizations and tends to promote less ambiguous code. Also add a new AppVeyor CI job and convert an existing one to use "unity" mode (one MSVC, one MinGW), and enable it for one macOS CI job. Fix related issues: - add missing include guard to `easy_lock.h`. - rename static variables and functions (and a macro) with names reused across sources, or shadowed by local variables. - add an `#undef` after use. - add a missing `#undef` before use. - move internal definitions from `ftp.h` to `ftp.c`. - `curl_memory.h` fixes to make it work when included repeatedly. - stop building/linking curlx bits twice for a static-mode curl tool. These caused doubly defined symbols in unity builds. - silence missing extern declarations compiler warning for ` _CRT_glob`. - fix extern declarations for `tool_freq` and `tool_isVistaOrGreater`. - fix colliding static symbols in debug mode: `debugtime()` and `statename`. - rename `ssl_backend_data` structure to unique names for each TLS-backend, along with the `ssl_connect_data` struct member referencing them. This required adding casts for each access. - add workaround for missing `[P]UNICODE_STRING` types in certain Windows builds when compiling `lib/ldap.c`. To support "unity" builds, we had to enable `SCHANNEL_USE_BLACKLISTS` for Schannel (a Windows `schannel.h` option) _globally_. This caused an indirect inclusion of Windows `schannel.h` from `ldap.c` via `winldap.h` to have it enabled as well. This requires `[P]UNICODE_STRING` types, which is apperantly not defined automatically (as seen with both MSVS and mingw-w64). This patch includes `<subauth.h>` to fix it. Ref: https://github.com/curl/curl/runs/13987772013 Ref: https://dev.azure.com/daniel0244/curl/_build/results?buildId=15827&view=logs&jobId=2c9f582d-e278-56b6-4354-f38a4d851906&j=2c9f582d-e278-56b6-4354-f38a4d851906&t=90509b00-34fa-5a81-35d7-5ed9569d331c - tweak unity builds to compile `lib/memdebug.c` separately in memory trace builds to avoid PP confusion. - force-disable unity for test programs. - do not compile and link libcurl sources to libtests _twice_ when libcurl is built in static mode. KNOWN ISSUES: - running tests with unity builds may fail in cases. - some build configurations/env may not compile in unity mode. E.g.: https://ci.appveyor.com/project/curlorg/curl/builds/47230972/job/51wfesgnfuauwl8q#L250 Ref: https://github.com/libssh2/libssh2/issues/1034 Ref: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html Ref: https://en.wikipedia.org/wiki/Unity_build Closes #11095
show more ...
|
#
7815647d |
| 25-Apr-2023 |
Daniel Stenberg |
lib: unify the upload/method handling By making sure we set state.upload based on the set.method value and not independently as set.upload, we reduce confusion and mixup risks, both
lib: unify the upload/method handling By making sure we set state.upload based on the set.method value and not independently as set.upload, we reduce confusion and mixup risks, both internally and externally. Closes #11017
show more ...
|
#
d5004550 |
| 13-Apr-2023 |
Daniel Stenberg |
smb: remove double assign The same value is assigned the same value already a few lines above. Pointed out by PVS. Ref: #10929 Closes #10945
|
#
83568266 |
| 23-Feb-2023 |
Daniel Stenberg |
misc: remove support for curl_off_t < 8 bytes Closes #10597
|
#
efbf0211 |
| 12-Feb-2023 |
Daniel Stenberg |
smb: return error on upload without size The protocol needs to know the size ahead of time, this is now a known restriction and not a bug. Also output a clearer error if the URL
smb: return error on upload without size The protocol needs to know the size ahead of time, this is now a known restriction and not a bug. Also output a clearer error if the URL path does not contain proper share. Ref: #7896 Closes #10484
show more ...
|
#
2bc1d775 |
| 02-Jan-2023 |
Daniel Stenberg |
copyright: update all copyright lines and remove year ranges - they are mostly pointless in all major jurisdictions - many big corporations and projects already don't use them - save
copyright: update all copyright lines and remove year ranges - they are mostly pointless in all major jurisdictions - many big corporations and projects already don't use them - saves us from pointless churn - git keeps history for us - the year range is kept in COPYING checksrc is updated to allow non-year using copyright statements Closes #10205
show more ...
|
#
4f20188a |
| 19-Dec-2022 |
Daniel Stenberg |
smb/telnet: do not free the protocol struct in *_done() It is managed by the generic layer. Reported-by: Trail of Bits Closes #10112
|
#
af22c2a5 |
| 22-Nov-2022 |
Stefan Eissing |
vtls: localization of state data in filters - almost all backend calls pass the Curl_cfilter intance instead of connectdata+sockindex - ssl_connect_data is remove from struct co
vtls: localization of state data in filters - almost all backend calls pass the Curl_cfilter intance instead of connectdata+sockindex - ssl_connect_data is remove from struct connectdata and made internal to vtls - ssl_connect_data is allocated in the added filter, kept at cf->ctx - added function to let a ssl filter access its ssl_primary_config and ssl_config_data this selects the propert subfields in conn and data, for filters added as plain or proxy - adjusted all backends to use the changed api - adjusted all backends to access config data via the exposed functions, no longer using conn or data directly cfilter renames for clear purpose: - methods `Curl_conn_*(data, conn, sockindex)` work on the complete filter chain at `sockindex` and connection `conn`. - methods `Curl_cf_*(cf, ...)` work on a specific Curl_cfilter instance. - methods `Curl_conn_cf()` work on/with filter instances at a connection. - rebased and resolved some naming conflicts - hostname validation (und session lookup) on SECONDARY use the same name as on FIRST (again). new debug macros and removing connectdata from function signatures where not needed. adapting schannel for new Curl_read_plain paramter. Closes #9919
show more ...
|
#
dafdb20a |
| 11-Nov-2022 |
Stefan Eissing |
lib: connection filters (cfilter) addition to curl: - general construct/destroy in connectdata - default implementations of callback functions - connect: cfilters for connect and
lib: connection filters (cfilter) addition to curl: - general construct/destroy in connectdata - default implementations of callback functions - connect: cfilters for connect and accept - socks: cfilter for socks proxying - http_proxy: cfilter for http proxy tunneling - vtls: cfilters for primary and proxy ssl - change in general handling of data/conn - Curl_cfilter_setup() sets up filter chain based on data settings, if none are installed by the protocol handler setup - Curl_cfilter_connect() boot straps filters into `connected` status, used by handlers and multi to reach further stages - Curl_cfilter_is_connected() to check if a conn is connected, e.g. all filters have done their work - Curl_cfilter_get_select_socks() gets the sockets and READ/WRITE indicators for multi select to work - Curl_cfilter_data_pending() asks filters if the have incoming data pending for recv - Curl_cfilter_recv()/Curl_cfilter_send are the general callbacks installed in conn->recv/conn->send for io handling - Curl_cfilter_attach_data()/Curl_cfilter_detach_data() inform filters and addition/removal of a `data` from their connection - adding vtl functions to prevent use of Curl_ssl globals directly in other parts of the code. Reviewed-by: Daniel Stenberg Closes #9855
show more ...
|
#
b563a92c |
| 01-Nov-2022 |
Viktor Szakats |
tidy-up: process.h detection and use This patch aims to cleanup the use of `process.h` header and the macro `HAVE_PROCESS_H` associated with it. - `process.h` is always availabl
tidy-up: process.h detection and use This patch aims to cleanup the use of `process.h` header and the macro `HAVE_PROCESS_H` associated with it. - `process.h` is always available on Windows. In curl, it is required only for `_beginthreadex()` in `lib/curl_threads.c`. - `process.h` is also available in MS-DOS. In curl, its only use was in `lib/smb.c` for `getpid()`. But `getpid()` is in fact declared by `unistd.h`, which is always enabled via `lib/config-dos.h`. So the header is not necessary. - `HAVE_PROCESS_H` was detected by CMake, forced to 1 on Windows and left to real detection for other platforms. It was also set to always-on in `lib/config-win32.h` and `lib/config-dos.h`. In autotools builds, there was no detection and the macro was never set. Based on these observations, in this patch we: - Rework Windows `getpid` logic in `lib/smb.c` to always use the equivalent direct Win32 API function `GetCurrentProcessId()`, as we already did for Windows UWP apps. This makes `process.h` unnecessary here on Windows. - Stop #including `process.h` into files where it was not necessary. This is everywhere, except `lib/curl_threads.c`. > Strangely enough, `lib/curl_threads.c` compiled fine with autotools > because `process.h` is also indirecty included via `unistd.h`. This > might have been broken in autotools MSVC builds, where the latter > header is missing. - Delete all remaining `HAVE_PROCESS_H` feature guards, for they were unnecessary. - Delete `HAVE_PROCESS_H` detection from CMake and predefined values from `lib/config-*.h` headers. Reviewed-by: Jay Satiro Closes #9703
show more ...
|
#
28edcbe1 |
| 11-Oct-2022 |
Viktor Szakats |
smb: replace CURL_WIN32 with WIN32 PR #9255 aimed to fix a Cygwin/MSYS issue (#8220). It used the `CURL_WIN32` macro, but that one is not defined here, while compiling curl itself. T
smb: replace CURL_WIN32 with WIN32 PR #9255 aimed to fix a Cygwin/MSYS issue (#8220). It used the `CURL_WIN32` macro, but that one is not defined here, while compiling curl itself. This patch changes this to `WIN32`, assuming this was the original intent. Regression from 1c52e8a3795ccdf8ec9c308f4f8f19cf10ea1f1a Reviewed-by: Marcel Raad Closes #9701
show more ...
|
#
1c52e8a3 |
| 05-Aug-2022 |
Orgad Shaneh |
fix Cygwin/MSYS compilation _getpid is Windows API. On Cygwin variants it should remain getpid. Fixes #8220 Closes #9255
|
#
ad9bc597 |
| 17-May-2022 |
max.mehl |
copyright: make repository REUSE compliant Add licensing and copyright information for all files in this repository. This either happens in the file itself as a comment header or in the
copyright: make repository REUSE compliant Add licensing and copyright information for all files in this repository. This either happens in the file itself as a comment header or in the file `.reuse/dep5`. This commit also adds a Github workflow to check pull requests and adapts copyright.pl to the changes. Closes #8869
show more ...
|
#
7d600ad1 |
| 14-Feb-2022 |
Daniel Stenberg |
urldata: remove conn->bits.user_passwd The authentication status should be told by the transfer and not the connection. Reported-by: John H. Ayad Fixes #8449 Closes #8451
|