QUIC API Overview ================= This document sets out the objectives of the QUIC API design process, describes the new and changed APIs, and the design constraints motivating those API designs and the relevant design decisions. - [QUIC API Overview](#quic-api-overview) * [Overview and Implementation Status](#overview-and-implementation-status) * [Objectives](#objectives) * [SSL Objects](#ssl-objects) + [Structure of Documentation](#structure-of-documentation) + [Existing APIs](#existing-apis) - [`SSL_set_connect_state`](#-ssl-set-connect-state-) - [`SSL_set_accept_state`](#-ssl-set-accept-state-) - [`SSL_is_server`](#-ssl-is-server-) - [`SSL_connect`](#-ssl-connect-) - [`SSL_accept`](#-ssl-accept-) - [`SSL_do_handshake`](#-ssl-do-handshake-) - [`SSL_read`, `SSL_read_ex`, `SSL_peek`, `SSL_peek_ex`](#-ssl-read----ssl-read-ex----ssl-peek----ssl-peek-ex-) - [`SSL_write`, `SSL_write_ex`](#-ssl-write----ssl-write-ex-) - [`SSL_pending`](#-ssl-pending-) - [`SSL_has_pending`](#-ssl-has-pending-) - [`SSL_shutdown`](#-ssl-shutdown-) - [`SSL_clear`](#-ssl-clear-) - [`SSL_free`](#-ssl-free-) - [`SSL_set0_rbio`, `SSL_set0_wbio`, `SSL_set_bio`](#-ssl-set0-rbio----ssl-set0-wbio----ssl-set-bio-) - [`SSL_set_[rw]fd`](#-ssl-set--rw-fd-) - [`SSL_get_[rw]fd`](#-ssl-get--rw-fd-) - [`SSL_CTRL_MODE`, `SSL_CTRL_CLEAR_MODE`](#-ssl-ctrl-mode----ssl-ctrl-clear-mode-) - [SSL Modes](#ssl-modes) + [New APIs for Single-Stream Operation](#new-apis-for-single-stream-operation) - [`SSL_handle_events`](#-ssl-handle-events-) - [`SSL_get_event_timeout`](#-ssl-get-event-timeout-) - [`SSL_set_blocking_mode`, `SSL_get_blocking_mode`](#-ssl-set-blocking-mode----ssl-get-blocking-mode-) - [`SSL_get_rpoll_descriptor`, `SSL_get_wpoll_descriptor`](#-ssl-get-rpoll-descriptor----ssl-get-wpoll-descriptor-) - [`SSL_net_read_desired`, `SSL_net_write_desired`](#-ssl-want-net-read----ssl-want-net-write-) - [`SSL_want`, `SSL_want_read`, `SSL_want_write`](#-ssl-want----ssl-want-read----ssl-want-write-) - [`SSL_set1_initial_peer_addr`](#-ssl-set-initial-peer-addr-) - [`SSL_shutdown_ex`](#-ssl-shutdown-ex-) - [`SSL_stream_conclude`](#-ssl-stream-conclude-) - [`SSL_stream_reset`](#-ssl-stream-reset-) - [`SSL_get_stream_state`](#-ssl-get-stream-state-) - [`SSL_get_stream_read_error_code`, `SSL_get_stream_write_error_code`](#-ssl-get-stream-read-error-code----ssl-get-stream-write-error-code-) - [`SSL_get_conn_close_info`](#-ssl-get-conn-close-info-) + [New APIs for Multi-Stream Operation](#new-apis-for-multi-stream-operation) - [Notes on Multi-Threaded Operation](#notes-on-multi-threaded-operation) - [Notes on Blocking](#notes-on-blocking) - [Notes on Application-Level Polling](#notes-on-application-level-polling) - [`SSL_get0_connection`](#-ssl-get0-connection-) - [`SSL_is_connection`](#-ssl-is-connection-) - [`SSL_get_stream_type`](#-ssl-get-stream-type-) - [`SSL_get_stream_id`](#-ssl-get-stream-id-) - [`SSL_is_stream_local`](#-ssl-is-stream-local-) - [`SSL_new_stream`](#-ssl-new-stream-) - [`SSL_accept_stream`](#-ssl-accept-stream-) - [`SSL_get_accept_stream_queue_len`](#-ssl-get-accept-stream-queue-len-) - [`SSL_set_incoming_stream_policy`](#-ssl-set-incoming-stream-policy-) - [`SSL_set_default_stream_mode`](#-ssl-set-default-stream-mode-) + [Future APIs](#future-apis) * [BIO Objects](#bio-objects) + [Existing APIs](#existing-apis-1) - [`BIO_s_connect`, `BIO_new_ssl_connect`, `BIO_set_conn_hostname`](#-bio-s-connect----bio-new-ssl-connect----bio-set-conn-hostname-) - [`BIO_new_bio_pair`](#-bio-new-bio-pair-) - [Interactions with `BIO_f_buffer`](#interactions-with--bio-f-buffer-) - [MTU Signalling](#mtu-signalling) + [New APIs](#new-apis) - [`BIO_sendmmsg` and `BIO_recvmmsg`](#-bio-sendmmsg--and--bio-recvmmsg-) - [Truncation Mode](#truncation-mode) - [Capability Negotiation](#capability-negotiation) - [Local Address Support](#local-address-support) - [`BIO_s_dgram_pair`](#-bio-s-dgram-pair-) - [`BIO_POLL_DESCRIPTOR`](#-bio-poll-descriptor-) - [`BIO_s_dgram_mem`](#-bio-s-dgram-mem-) - [`BIO_err_is_non_fatal`](#-bio-err-is-non-fatal-) * [Q & A](#q---a) Overview and Implementation Status ---------------------------------- A listing of all SSL object APIs and their implications for QUIC, including current implementation status, can be found in [quic-api-ssl-funcs.md](./quic-api-ssl-funcs.md). Non-SSL object APIs which are new or changed, or otherwise discussed in this document are listed below, along with their implementation status. SSL object APIs are not listed here; see [quic-api-ssl-funcs.md](./quic-api-ssl-funcs.md) for details on SSL object APIs. | Semantics | API | Status | |-----------|---------------------------------|--------| | Changed | `BIO_s_connect` | Done | | Unchanged | `BIO_set_conn_hostname` | Done | | N/A | `BIO_new_bio_pair` | N/A (see `BIO_new_bio_dgram_pair`) | | New | `BIO_s_dgram_pair` | Done | | Unchanged | `BIO_dgram_get_mtu` | Done | | Unchanged | `BIO_dgram_set_mtu` | Done | | New | `BIO_sendmmsg` | Done | | New | `BIO_recvmmsg` | Done | | New | `BIO_dgram_set_no_trunc` | Done | | New | `BIO_dgram_get_no_trunc` | Done | | New | `BIO_dgram_set_caps` | Done | | New | `BIO_dgram_get_caps` | Done | | New | `BIO_dgram_get_effective_caps` | Done | | New | `BIO_dgram_get_local_addr_cap` | Done | | New | `BIO_dgram_set_local_addr_enable` | Done | | New | `BIO_dgram_get_local_addr_enable` | Done | | New | `BIO_get_rpoll_descriptor` | Done | | New | `BIO_get_wpoll_descriptor` | Done | | New | `BIO_err_is_non_fatal` | Done | Objectives ---------- The objectives of the QUIC API design are: - to provide an API suitable for use with QUIC, now and in the future; - to reuse the existing libssl APIs to the extent feasible; - to enable existing applications to adapt to using QUIC with only minimal API changes. SSL Objects ----------- ### Structure of Documentation Each API listed below has an information table with the following fields: - **Semantics**: This can be one of: - **Unchanged**: The semantics of this existing libssl API call are unchanged. - **Changed**: The semantics are changed for QUIC. - **New**: The API is new for QUIC. - `SSL_get_error`: Can this API, when used with QUIC, change the state returned by `SSL_get_error`? This can be any combination of: - **Never**: Does not interact with `SSL_get_error`. - **Error**: Non-`WANT_READ`/`WANT_WRITE` errors can be raised. - **Want**: `WANT_READ`/`WANT_WRITE` can be raised. - **Can Tick?**: Whether this function is allowed to perform event processing for the QUIC state machine and potentially perform network I/O. - **CSHL:** Connection/Stream/Handshake Layer classification. This can be one of: - **HL:** This is a handshake layer related call. It should be supported on a QUIC connection SSL object, forwarding to the handshake layer SSL object. QUIC stream SSL objects do not allow these calls to be forwarded. - **HL-Forbidden:** This is a handshake layer related call, but it is inapplicable to QUIC, so it is not supported. - **C:** Not handshake-layer related. QUIC connection SSL object usage only. Fails on a QUIC stream SSL object. - **CS:** Not handshake-layer related. Can be used on any QUIC SSL object. - **S**: Requires a QUIC stream SSL object or a QUIC connection SSL object with a default stream attached. ### Existing APIs #### `SSL_set_connect_state` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Never | No | HL | #### `SSL_set_accept_state` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Never | No | HL | **Note:** Attempting to proceed in this state will not function for now because we do not implement server support at this time. However, the semantics of this function as such are unchanged. #### `SSL_is_server` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Never | No | HL | #### `SSL_connect` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Error/Want | Yes | HL | Simple composition of `SSL_set_connect_state` and `SSL_do_handshake`. #### `SSL_accept` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Error/Want | Yes | HL | Simple composition of `SSL_set_accept_state` and `SSL_do_handshake`. #### `SSL_do_handshake` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Error/Want | Yes | HL | **Note:** Idempotent if handshake already completed. **Blocking Considerations:** Blocks until handshake completed if in blocking mode. #### `SSL_read`, `SSL_read_ex`, `SSL_peek`, `SSL_peek_ex` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Error/Want | Yes | CS | **Blocking Considerations:** Blocks until at least one byte is available or an error occurs if in blocking mode (including the peek functions). If the read part of the stream has been finished by the peer, calls to `SSL_read` will fail with `SSL_ERROR_ZERO_RETURN`. If a stream has terminated in a non-normal fashion (for example because the stream has been reset, or the connection has terminated), calls to `SSL_read` will fail with `SSL_ERROR_SSL`. `SSL_get_stream_read_state` can be used to clarify the stream state when an error occurs. #### `SSL_write`, `SSL_write_ex` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Error/Want | Yes | CS | We have to implement all of the following modes: - `SSL_MODE_ENABLE_PARTIAL_WRITE` on or off - `SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER` on or off - Blocking mode on or off **Blocking Considerations:** Blocks until libssl has accepted responsibility for (i.e., copied) all data provided, or an error occurs, if in blocking mode. In other words, it blocks until it can buffer the data. This does not necessarily mean that the data has actually been sent. `SSL_get_stream_write_state` can be used to clarify the stream state when an error occurs. #### `SSL_pending` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Never | No | CS | #### `SSL_has_pending` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Never | No | CS | **TBD.** Options: - Semantics unchanged or approximated (essentially, `SSL_pending() || any RXE queued || any URXE queued`). - Change semantics to only determine the return value based on if there is data in the stream receive buffer. #### `SSL_shutdown` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Error | Yes | CS | See `SSL_shutdown_ex` below for discussion of how this will work for QUIC. Calling `SSL_shutdown` is always exactly identical in function to calling `SSL_shutdown_ex` with `flags` set to 0 and `args` set to `NULL`. #### `SSL_clear` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | TBD | TBD | No | C | There are potential implementation hazards: >SSL_clear() resets the SSL object to allow for another connection. The reset >operation however keeps several settings of the last sessions (some of these >settings were made automatically during the last handshake). It only makes sense >for a new connection with the exact same peer that shares these settings, and >may fail if that peer changes its settings between connections. **TBD:** How should `SSL_clear` be implemented? Either: - Modernised implementation which resets everything, handshake layer re-instantiated (safer); - Preserve `SSL_clear` semantics at the handshake layer, reset all QUIC state (`QUIC_CHANNEL` torn down, CSM reset). **TBD:** Semantics of this on stream objects. #### `SSL_free` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Changed | Never | No | CS | **QUIC stream SSL objects.** When used on a QUIC stream SSL object, parts of the stream state may continue to exist internally, managed inside the QUIC connection SSL object, until they can be correctly torn down, or until the QUIC connection SSL object is freed. If a QUIC stream SSL object is freed for a stream which has not reached a terminal state for all of its parts (both send and receive, as applicable), the stream is automatically reset (non-normal termination) with an application error code of 0. To explicitly reset a stream with a different application error code, call `SSL_stream_reset` before calling this function. If the peer continues to send data on the stream before it processes the notification of the stream's termination, that incoming data will be discarded. However, the peer will be reliably notified of the non-normal termination of the stream assuming that the connection remains healthy. When freeing a QUIC stream SSL object which was terminated in a non-normal fashion, or which was terminated automatically due to a call to this function, any data which was appended to the stream via `SSL_write` may or may not have already been transmitted, and even if already transmitted, may or may not be retransmitted in the event of loss. When freeing a QUIC stream SSL object which was terminated normally (for example via `SSL_stream_conclude`), data appended to the stream via `SSL_write` will still be transmitted or retransmitted as necessary, assuming that the QUIC connection SSL object is not freed and that the connection remains healthy. **QUIC connection SSL objects.** `SSL_free` is largely unchanged for QUIC connection SSL objects on the client side. When freeing a QUIC connection SSL object being used in client mode, there is immediate termination of any QUIC network I/O processing as the resources needed to handle the connection are immediately freed. This means that, if a QUIC connection SSL object which has not been shutdown properly is freed using this function: - Any data which was pending transmission or retransmission will not be transmitted, including in streams which were terminated normally; - The connection closure process will not function correctly or in an RFC-compliant manner. Connection closure will not be signalled to the peer and the connection will simply disappear from the perspective of the peer. The connection will appear to remain active until the connection's idle timeout (if negotiated) takes effect. For further discussion of this issue, see `SSL_shutdown_ex` and the Q&A. #### `SSL_set0_rbio`, `SSL_set0_wbio`, `SSL_set_bio` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Changed | Never | No | C | Sets network-side BIO. The changes to the semantics of these calls are as follows: - The BIO MUST be a BIO with datagram semantics (this is a change relative to TLS, though not to DTLS). - If the BIO is non-pollable (see below), application-level blocking mode will be forced off. #### `SSL_set_[rw]fd` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Changed | Never | No | C | Sets network-side socket FD. Existing behaviour: Instantiates a `BIO_s_socket`, sets an FD on it, and sets it as the BIO. New proposed behaviour: - Instantiate a `BIO_s_dgram` instead for a QUIC connection SSL object. - Fails (no-op) for a QUIC stream SSL object. #### `SSL_get_[rw]fd` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Never | No | C | Should not require any changes. #### `SSL_CTRL_MODE`, `SSL_CTRL_CLEAR_MODE` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | Unchanged | Never | No | CS | #### SSL Modes - `SSL_MODE_ENABLE_PARTIAL_WRITE`: Implemented. If this mode is set during a non-partial-write `SSL_write` operation spanning multiple `SSL_write` calls, this mode does not take effect until the non-partial write operation is completed. - `SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER`: Implemented. - `SSL_MODE_AUTO_RETRY`: TBD. - `SSL_MODE_RELEASE_BUFFERS`: Ignored. This is an optimization and if it has any sensible semantic correspondence to QUIC, this can be considered later. - `SSL_MODE_SEND_FALLBACK_SCSV`: TBD: Either ignore or fail if the client attempts to set this prior to handshake. The latter is probably safer. Ignored if set after handshake (existing behaviour). - `SSL_MODE_ASYNC`: TBD. ### New APIs for Single-Stream Operation #### `SSL_handle_events` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | Yes | CS | Advances the QUIC state machine to the extent feasible, potentially performing network I/O. Also compatible with DTLSv1 and supersedes `DTLSv1_handle_timeout` for all use cases. #### `SSL_get_event_timeout` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | CS | Gets the time until the QUIC state machine next wants to receive a timeout event, if any. This is similar to the existing `DTLSv1_get_timeout` function, but it is not specific to DTLSv1. It is also usable for DTLSv1 and can become a protocol-agnostic API for this purpose, superseding `DTLSv1_get_timeout` for all use cases. The design is similar to that of `DTLSv1_get_timeout` and uses a `struct timeval`. However, this function can also output an infinite timeout using the `is_infinite` argument, whereas whereas `DTLSv1_get_timeout` represents an infinite timeout using a 0 return value, which does not allow a failure condition to be distinguished. #### `SSL_set_blocking_mode`, `SSL_get_blocking_mode` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | CS | Turns blocking mode on or off. This is necessary because up until now libssl has operated in blocking or non-blocking mode automatically as an emergent consequence of whether the underlying network socket is blocking. For QUIC, this is no longer viable, thus blocking semantics at the application level must be explicitly configured. Use on stream objects: It may be feasible to implement this such that different QUIC stream SSL objects can have different settings for this option. Not supported for non-QUIC SSL objects. #### `SSL_get_rpoll_descriptor`, `SSL_get_wpoll_descriptor` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | CS | These functions output poll descriptors which can be used to determine when the QUIC state machine next needs to have events handled. `SSL_get_rpoll_descriptor` is relevant if `SSL_net_read_desired` returns 1, and `SSL_get_wpoll_descriptor` is relevant if `SSL_net_write_desired` returns 1. The implementation of these functions is a simple forward to `BIO_get_rpoll_descriptor` and `BIO_get_wpoll_descriptor` on the underlying network BIOs. TODO: Support these for non-QUIC SSL objects #### `SSL_net_read_desired`, `SSL_net_write_desired` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | CS | These calls return 1 if the QUIC state machine is interested in receiving further data from the network, or writing to the network, respectively. The return values of these calls should be used to determine which wakeup events should cause an application to call `SSL_handle_events`. These functions do not mutate any state, and their return values may change after a call to any SSL function other than `SSL_net_read_desired`, `SSL_net_write_desired`, `SSL_get_rpoll_descriptor`, `SSL_get_wpoll_descriptor` and `SSL_get_event_timeout`. TODO: Support these for non-QUIC SSL objects, turning this into a unified replacement for `SSL_want` #### `SSL_want`, `SSL_want_read`, `SSL_want_write` The existing API `SSL_want`, and the macros defined in terms of it, are traditionally used to determine if the SSL state machine has exited in non-blocking mode due to a desire to read from or write to the underlying network BIO. However, this API is unsuitable for use with QUIC because the return value of `SSL_want` can only express one I/O direction at a time (read or write), not both. This call will not be implemented for QUIC (e.g. always returns `SSL_NOTHING`) and `SSL_net_read_desired` and `SSL_net_write_desired` will be used instead. #### `SSL_set1_initial_peer_addr` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | CS | `SSL_set1_initial_peer_addr` sets the initial L4 UDP peer address for an outgoing QUIC connection. The initial peer address may be autodetected if no peer address has already been set explicitly and the QUIC connection SSL object is provided with a `BIO_s_dgram` with a peer set. `SSL_set1_initial_peer_addr` cannot be called after a connection is established. #### `SSL_shutdown_ex` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Error | Yes | C | ```c typedef struct ssl_shutdown_ex_args_st { /* These arguments pertain only to QUIC connections. */ uint64_t quic_error_code; /* [0, 2**62-1] */ const char *quic_reason; } SSL_SHUTDOWN_EX_ARGS; #define SSL_SHUTDOWN_FLAG_RAPID (1U << 0) #define SSL_SHUTDOWN_FLAG_IMMEDIATE (1U << 1) int SSL_shutdown_ex(SSL *ssl, uint64_t flags, const SSL_SHUTDOWN_EX_ARGS *args, size_t args_len); ``` `SSL_shutdown_ex` is an extended version of `SSL_shutdown`. `args` specifies arguments which control how the SSL object is shut down. `args` are read only on the first call to `SSL_shutdown_ex` for a given SSL object and subsequent calls to `SSL_shutdown_ex` ignore the `args` argument. `args_len` should be set to `sizeof(*args)`. This function is idempotent; once the shutdown process for an SSL object is complete, further calls are a no-op and return 1. Calling `SSL_shutdown_ex` on a QUIC connection SSL object causes the immediate close of the QUIC connection. “Immediate close” is as defined by RFC 9000. If no QUIC connection attempt was ever initiated using the given SSL object, the QUIC connection transitions immediately to the Terminated state. Otherwise, the connection closure process is initiated if it has not already begun. Any application stream data on a non-terminated or normally terminated stream which has yet to be transmitted is flushed to the network before the termination process begins. This ensures that where an application which calls `SSL_write` and performs a connection closure in a way which is considered normal to the application protocol being used, all of the data written is delivered to the peer. This behaviour may be skipped by setting the `SSL_SHUTDOWN_FLAG_IMMEDIATE` flag, in which case any data appended to streams via `SSL_write` (or any end-of-stream conditions) may not be transmitted to the peer. This flag may be useful where a non-normal application condition has occurred and the delivery of data written to streams via `SSL_write` is no longer relevant. Application stream data on streams which were terminated non-normally (for example via `SSL_stream_reset`) is not transmitted by this function. A QUIC connection can be shut down using this function in two different ways: - **RFC compliant mode.** In this mode, which provides the most robust operation, the shutdown process may take a period of time up to three times the current estimated RTT to the peer. It is possible for the closure process to complete much faster in some circumstances but this cannot be relied upon. In blocking mode, the function will return once the closure process is complete. In non-blocking mode, `SSL_shutdown_ex` should be called until it returns 1, indicating the closure process is complete and the connection is now terminated. - **Rapid mode.** In this mode, a `CONNECTION_CLOSE` frame is sent in a best-effort manner and the connection is terminated immediately. If the `CONNECTION_CLOSE` frame sent is lost, the peer will not know that the connection has terminated until the negotiated idle timeout (if any) expires. This will generally return 0 on success, indicating that the connection has not yet reached the Terminating state (unless it has already done so, in which case it will return 1). In blocking mode, this blocks until at least one `CONNECTION_CLOSE` frame is sent but does not otherwise block. In non-blocking mode, this should be called until it returns a non-negative value. A negative value indicates failure or an I/O would-block condition. It is permissible for an application to implement a hybrid approach, for example by initiating a rapid or non-blocking shutdown and continuing to call `SSL_handle_events` for a duration it chooses. If `SSL_SHUTDOWN_FLAG_RAPID` is specified in `flags`, a rapid shutdown is performed, otherwise an RFC-compliant shutdown is performed. The principal effect of this flag is to partially disable blocking behaviour in blocking mode, and the QUIC implementation will still attempt to implement the Terminating state semantics if the application happens to call `SSL_handle_events`, until it reaches the Terminated state or is freed. An application can change its mind about performing a rapid shutdown by making a subsequent call to `SSL_shutdown_ex` without the flag set. Calling `SSL_shutdown_ex` on a QUIC stream SSL object is not valid; such a call will fail and has no effect. The rationale for this is that an application may well want to pass around SSL objects for individual QUIC streams to existing parts of its own code which expect something which behaves like a typical SSL object (i.e., a single bytestream); those components may well already call `SSL_shutdown` and it is not desired for such calls to affect the whole connection. The `args->quic_error_code` and `args->reason` fields allow the application error code and reason string for the closure of a QUIC connection to be specified. If `args` or `args->reason` is `NULL`, a zero-length string is used for the reason. If `args` is `NULL`, an error code of 0 is used. `args->quic_error_code` must be in the range `[0, 2**62-1]`, else this function fails. These fields are ignored for SSL objects which do not represent QUIC connections. #### `SSL_stream_conclude` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Error | Yes | S | ```c int SSL_stream_conclude(SSL *ssl, uint64_t flags); ``` `SSL_stream_conclude` signals the normal end-of-stream condition to the send part of a QUIC stream. If called on a QUIC connection SSL object with a default stream, it signals the end of that stream to the peer. If called on a QUIC stream SSL object, it signals the end of that stream to the peer. This function may only be called for bidirectional streams and for outgoing unidirectional streams. It is a no-op if it has already been called for a given stream, or if either the stream or connection have entered an error state. Any data already queued for transmission via a call to `SSL_write()` will still be written in a reliable manner before the end-of-stream is signalled, assuming the connection remains healthy. This function can be thought of as appending a logical end-of-stream marker after any data which has previously been written to the stream via calls to `SSL_write`. Further attempts to call `SSL_write` after calling this function will fail. When calling this on a bidirectional stream, the receive part of the stream remains unaffected, and the peer may continue to send data via it until the peer also signals the end of the stream. Thus, `SSL_read()` can still be used. This function is used to conclude the send part of a stream in a normal manner. To perform non-normal termination of both the sending and receiving parts of a stream, see `SSL_stream_reset`. `flags` is reserved and should be set to 0. #### `SSL_stream_reset` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Error | Yes | S | ```c typedef struct ssl_stream_reset_args_st { uint64_t quic_error_code; /* [0, 2**62-1] */ } SSL_STREAM_RESET_ARGS; int SSL_stream_reset(SSL *ssl, const SSL_STREAM_RESET_ARGS *args, size_t args_len); ``` Conducts a non-normal termination of a bidirectional or outgoing unidirectional stream. For QUIC, this corresponds to a stream reset using a `RESET_STREAM` frame. It may be called on either a QUIC stream SSL object or a QUIC connection SSL object with a default stream; the given stream is reset. The QUIC connection is not affected. For bidirectional streams, this terminates both sending and receiving parts of the stream. It may not be called on an incoming unidirectional stream. If `args` is `NULL`, an application error code of 0 is used. Otherwise, the application error code to use is specified in `args->quic_error_code`, which must be in the range `[0, 2**62-1]`. `args_len` must be set to `sizeof(*args)` if `args` is non-NULL. Only the first call to this function has any effect; subsequent calls are no-ops. This is considered a success case. #### `SSL_get_stream_state` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | S | ```c /* * e.g. Non-QUIC SSL object, or QUIC connection SSL object without a default * stream. */ #define SSL_STREAM_STATE_NONE 0 /* * The read or write part of the stream is still available and has not been * terminated in a normal or non-normal manner. */ #define SSL_STREAM_STATE_OK 1 /* * The stream is a unidirectional stream and this direction cannot be used; for * example, a remotely initiated unidirectional stream where * SSL_get_stream_write_state is called, or a locally initiated unidirectional * stream where SSL_get_stream_read_state is called. */ #define SSL_STREAM_STATE_WRONG_DIR 2 /* * The read or write part of the stream has been finished in a normal manner. * * For SSL_get_stream_read_state, this means that there is no more data to read, * and that any future SSL_read calls will return any residual data waiting to * be read followed by an SSL_ERROR_ZERO_RETURN condition. * * For SSL_get_stream_write_state, this means that the local application has * already indicated the end of the stream by calling SSL_stream_conclude, * and that future calls to SSL_write will fail. */ #define SSL_STREAM_STATE_FINISHED 3 /* * The stream was reset by the local party. * * For SSL_get_stream_read_state, this means that the stream was aborted using a * locally transmitted STOP_SENDING frame. Attempts to read from the stream via * SSL_read will fail, though SSL_read may allow any residual data waiting to * be read to be read first. * * For SSL_get_stream_write_state, this means that the stream was aborted * using a locally transmitted RESET_STREAM frame. Attempts to write to * the stream will fail. */ #define SSL_STREAM_STATE_RESET_LOCAL 4 /* * The stream was reset by the remote party. * * For SSL_get_stream_read_state, this means the peer sent a STREAM_RESET * frame for the stream. * * For SSL_get_stream_write_state, this means the peer sent a STOP_SENDING * frame for the stream. */ #define SSL_STREAM_STATE_RESET_REMOTE 5 /* * The underlying connection supporting the stream has closed or otherwise * failed. * * For SSL_get_stream_read_state, this means that attempts to read from the * stream via SSL_read will fail, though SSL_read may allow any residual * data waiting to be read to be read first. * * For SSL_get_stream_write_state, this means that attempts to write to the * stream will fail. */ #define SSL_STREAM_STATE_CONN_CLOSED 6 int SSL_get_stream_read_state(SSL *ssl); int SSL_get_stream_write_state(SSL *ssl); ``` This API allows the current state of a stream to be queried. This allows an application to determine whether a stream is still usable and why a stream has reached an error state. #### `SSL_get_stream_read_error_code`, `SSL_get_stream_write_error_code` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | S | ```c int SSL_get_stream_read_error_code(SSL *ssl, uint64_t *app_error_code); int SSL_get_stream_write_error_code(SSL *ssl, uint64_t *app_error_code); ``` `SSL_get_stream_read_error_code` gets the error code for the read part of the stream. `SSL_get_stream_write_error_code` gets the error code for the write part of the stream. If a stream has been terminated normally, returns 0. If a stream has been terminated non-normally, returns 1 and writes the applicable application error code to `*app_error_code`. If a stream is still healthy, or was healthy at the time the connection was closed, or the respective part of the stream does not exist (e.g. for a unidirectional stream), returns -1. #### `SSL_get_conn_close_info` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | C | ```c #define SSL_CONN_CLOSE_FLAG_LOCAL #define SSL_CONN_CLOSE_FLAG_TRANSPORT typedef struct ssl_conn_close_info_st { uint64_t error_code; char *reason; size_t reason_len; uint32_t flags; } SSL_CONN_CLOSE_INFO; int SSL_get_conn_close_info(SSL *ssl, SSL_CONN_CLOSE_INFO *info, size_t info_len); ``` If a connection is still healthy, returns 0. Otherwise, fills `*info` with information about the error causing connection termination and returns 1. `info_len` must be set to `sizeof(*info)`. Returns -1 if called on a non-QUIC SSL object or if the connection status cannot be determined. `info->reason` is set to point to a buffer containing a reason string. The buffer is valid for the lifetime of the SSL object. The reason string will always be zero terminated, but since it is received from a potentially untrusted peer, may also contain zero bytes. `info->reason_len` is the true length of the reason string in bytes. `info->flags` has `SSL_CONN_CLOSE_FLAG_LOCAL` set if the connection closure was locally initiated. `info->flags` has `SSL_CONN_CLOSE_FLAG_TRANSPORT` if the connection closure was initiated by QUIC, and 0 if it was initiated by the application. The namespace of `info->error_code` is determined by this parameter. ### New APIs for Multi-Stream Operation The above new APIs are built on constructively to facilitate multi-stream operation. The concept of a QUIC stream SSL object is introduced. A QUIC SSL object is either a QUIC connection SSL object or a QUIC stream SSL object. A QUIC stream SSL object belongs to a QUIC connection SSL object. A QUIC connection SSL object may or may not have an associated default stream. There may only be at most one default stream for a QUIC connection SSL object. Reading or writing application data to a QUIC connection SSL object with a default stream is equivalent to reading or writing to that stream. It is an error to attempt to read or write application data, or perform other stream-specific operations, on a QUIC connection SSL object without a default stream associated. #### Notes on Multi-Threaded Operation Initially these APIs will not be thread safe over the same connection, but in the longer term we intend to support multiple threads using different QUIC stream SSL objects on different threads over the same connection without the application having to do any locking. This is referred to as multi-stream multi-thread (MSMT) operation. Only APIs explicitly denoted below will eventually be MSMT-safe. #### Notes on Blocking The blocking mode can be configured on each SSL object individually. When a QUIC stream SSL object is created it inherits its blocking state from the currently configured blocking state of the QUIC connection SSL object at the time the stream is created. This can be changed independently. For example, a QUIC connection SSL object can be in blocking mode to allow for blocking `SSL_accept_stream` calls, yet have some or all QUIC stream SSL objects be in non-blocking mode concurrently. #### Notes on Application-Level Polling An API may be added in the future to allow applications to poll multiple QUIC connection SSL objects efficiently for new stream and stream readability events. This is not yet urgent but will be more relevant for concurrent server applications. #### `SSL_get0_connection` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | CS | ```c /* * Get the SSL object representing the connection associated with this object. * * If the SSL object represents a non-QUIC method or a QUIC connection, this * returns the same object passed. * * If the SSL object represents a QUIC stream returns the QUIC connection * object. */ SSL *SSL_get0_connection(SSL *ssl); ``` #### `SSL_is_connection` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | CS | ```c /* * Returns 1 if the object represents a connection. This always returns 1 for * non-QUIC methods, but returns 0 for SSL objects for QUIC streams which are * not also the QUIC connection object. * * This is exactly equivalent to (SSL_get0_connection(ssl) == ssl). */ int SSL_is_connection(SSL *ssl); ``` #### `SSL_get_stream_type` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | S | ```c /* * If the object represents a stream, returns an SSL_STREAM_TYPE value * designating whether the stream can be used for transmission, reception, * or both. * * This always returns SSL_STREAM_TYPE_BIDI for non-QUIC methods. * * It returns SSL_STREAM_TYPE_NONE for a QUIC connection object if it * does not have a default stream. */ #define SSL_STREAM_TYPE_NONE 0 #define SSL_STREAM_TYPE_READ 1 #define SSL_STREAM_TYPE_WRITE 2 #define SSL_STREAM_TYPE_BIDI (SSL_STREAM_TYPE_READ | SSL_STREAM_TYPE_WRITE) __owur int SSL_get_stream_type(SSL *ssl); ``` #### `SSL_get_stream_id` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | S | ```c /* * QUIC: Returns the unique stream ID for the stream, an integer in range [0, 2**62-1], * or UINT64_MAX if the stream ID is not available. If called on a QUIC * connection, returns the unique stream ID for the default stream if there is * one, and otherwise returns UINT64_MAX. * * TLS, DTLS: Returns UINT64_MAX. */ __owur uint64_t SSL_get_stream_id(SSL *ssl); ``` #### `SSL_is_stream_local` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | S | ```c /* * QUIC: Returns 1 if the stream was locally initiated, or 0 otherwise. * * TLS, DTLS: Returns -1. */ __owur int SSL_is_stream_local(SSL *ssl); ``` #### `SSL_new_stream` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | C | ```c /* * Create a new SSL object representing a single additional stream. * * There is no need to call SSL_connect on the resulting object, and * any such call is a no-op. * * For QUIC: * Creates a new stream. Must be called only on a QUIC connection SSL object. * Can be used on client or server. If the SSL_STREAM_FLAG_UNI flag is set, * the created stream is unidirectional, otherwise it is bidirectional. * * To be MSMT-safe. * * For TLS and DTLS SSL objects: * Always fails. */ #define SSL_STREAM_FLAG_UNI 1 SSL *SSL_new_stream(SSL *ssl, uint64_t flags); ``` #### `SSL_accept_stream` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | Yes | C | ```c /* * Create a new SSL object representing an additional stream which was created * by the peer. * * There is no need to call SSL_accept on the resulting object, and * any such call is a no-op. * * For QUIC: * Must be called only on a QUIC connection SSL object. Fails if called on a * stream object. Checks if a new stream has been created by the peer. If it * has, creates a new SSL object to represent it and returns it. Otherwise, * returns NULL. If multiple streams are available to be accepted, the oldest * stream (that is, the stream with the lowest stream ID) is accepted. * * For all other methods: * Returns NULL. * * The flags argument is unused and should be set to zero. * * To be MSMT-safe (i.e., can be called from multiple threads). * * If the QUIC connection SSL object is configured in blocking mode, this * function will block unless the SSL_ACCEPT_STREAM_NO_BLOCK flag is passed. * * This function returns NULL if the effective incoming stream reject policy is * `REJECT`. */ #define SSL_ACCEPT_STREAM_NO_BLOCK 1 SSL *SSL_accept_stream(SSL *ssl, uint64_t flags); ``` #### `SSL_get_accept_stream_queue_len` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | C | ```c /* * Determine the number of streams waiting to be returned on a subsequent call * to SSL_accept_stream. If this returns a non-zero value, the next call to * SSL_accept_stream (on any thread) is guaranteed to work. Returns 0 for * non-QUIC objects, or for QUIC stream objects. * * To be MSMT-safe. */ size_t SSL_get_accept_stream_queue_len(SSL *ssl); ``` #### `SSL_set_incoming_stream_policy` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | C | ```c /* * Sets the policy for incoming streams. If `policy` is `AUTO` (the default): * * - if the default stream mode is * `SSL_DEFAULT_STREAM_MODE_AUTO_BIDI` or * `SSL_DEFAULT_STREAM_MODE_AUTO_UNI`, this is equivalent to `REJECT`; * * - otherwise, this is equivalent to `ACCEPT`. * * If configured to `ACCEPT`, incoming streams are placed on the accept queue * for application consumption. `aec` is ignored in this case. * * If configured to `REJECT`, incoming streams automatically have both their * receiving and sending parts handled via non-normal termination. `aec` is an * application error code used for the `STOP_SENDING` and `RESET_STREAM` frames * used for the purposes of this termination. The default AEC value used if this * function is never called is 0. */ #define SSL_INCOMING_STREAM_POLICY_AUTO 0 #define SSL_INCOMING_STREAM_POLICY_ACCEPT 1 #define SSL_INCOMING_STREAM_POLICY_REJECT 2 int SSL_set_incoming_stream_policy(SSL *ssl, int policy, uint64_t aec); ``` #### `SSL_set_default_stream_mode` | Semantics | `SSL_get_error` | Can Tick? | CSHL | | --------- | ------------- | --------- | ------------- | | New | Never | No | C | ```c /* * Used to control single stream operation. Calling this function determines the * nature of the default stream which will automatically be created on the QUIC * connection SSL object. * * The default mode is `SSL_DEFAULT_STREAM_MODE_AUTO_BIDI`. * * The modes are as follows: * * - `SSL_DEFAULT_STREAM_MODE_NONE`: No default stream will ever be created. * The application is assumed to understand multi-stream operation. * Remotely-initiated streams are placed in the accept queue for application * consumption. `SSL_read` and `SSL_write` calls must be made on a QUIC * stream SSL object, not the QUIC connection SSL object, as no default * stream will be associated with it. * * - `SSL_DEFAULT_STREAM_MODE_AUTO_BIDI`: "First stream wins" mode of * operation for single-stream usage. If `SSL_write` is called before the * peer opens a remotely-initiated stream, a locally-initiated bidirectional * stream is created and bound as the default stream. If the peer opens a * remotely-initiated stream before the local application calls `SSL_write` * (with `len > 0`) for the first time, that stream is bound as the default * stream, which may be bidirectional or unidirectional; if it is * unidirectional, calls to `SSL_write` will fail. Attempts to create * additional streams by the peer are automatically rejected unless * the application opts in (API TBD). * * - `SSL_DEFAULT_STREAM_MODE_AUTO_UNI`: "First stream wins" mode of * operation for single-stream usage, with a unidirectional stream. This * functions identically to `SSL_DEFAULT_STREAM_MODE_AUTO_BIDI`, but if the * local application calls `SSL_write` prior to the peer creating a * remotely-initiated stream, a unidirectional TX-only stream is created and * bound as the default stream. Thereafter, calls to `SSL_read` will fail. * If the peer creates a remotely-initiated stream prior to the first call * to `SSL_write` (with `len > 0`), that stream will be bound as the default * stream; note that a bidirectional stream may be bound in this case. * Attempts to create additional streams by the peer are automatically * rejected unless the application opts in (API TBD). * * This function must be called before a default stream object is created, for * example before initiating a connection. If the function is too late to have * an effect, this function fails and returns 0. */ #define SSL_DEFAULT_STREAM_MODE_NONE 0 #define SSL_DEFAULT_STREAM_MODE_AUTO_BIDI 1 #define SSL_DEFAULT_STREAM_MODE_AUTO_UNI 2 __owur int SSL_set_default_stream_mode(SSL *ssl, uint32_t mode); ``` ### Future APIs A custom poller interface may be provided in the future. For more information, see the QUIC I/O Architecture design document. BIO Objects ----------- ### Existing APIs #### `BIO_s_connect`, `BIO_new_ssl_connect`, `BIO_set_conn_hostname` We are aiming to support use of the existing `BIO_new_ssl_connect` API with only minimal changes. This will require internal changes to `BIO_s_connect`, which should automatically detect when it is being used with a QUIC `SSL_CTX` and act accordingly. #### `BIO_new_bio_pair` Unsuitable for use with QUIC on the network side; instead, applications can make use of the new `BIO_s_dgram_pair` which provides equivalent functionality with datagram semantics. #### Interactions with `BIO_f_buffer` Existing applications sometimes combine a network socket BIO with a `BIO_f_buffer`. This is problematic because the datagram semantics of writes are not preserved, therefore the BIO provided to libssl is, as provided, unusable for the purposes of implementing QUIC. Moreover, output buffering is not a relevant or desirable performance optimisation for the transmission of UDP datagrams and will actually undermine QUIC performance by causing incorrect calculation of ACK delays and consequently inaccurate RTT calculation. Options: - Require applications to be changed to not use QUIC with a `BIO_f_buffer`. - Detect when a `BIO_f_buffer` is part of a BIO stack and bypass it (yucky and surprising). #### MTU Signalling **See also:** [BIO_s_dgram_pair(3)](https://www.openssl.org/docs/manmaster/man3/BIO_s_dgram_pair.html) `BIO_dgram_get_mtu` (`BIO_CTRL_DGRAM_GET_MTU`) and `BIO_dgram_set_mtu` (`BIO_CTRL_DGRAM_SET_MTU`) already exist for `BIO_s_dgram` and are implemented on a `BIO_s_dgram_pair` to allow the MTU to be determined and configured. One side of a pair can configure the MTU to allow the other side to detect it. `BIO_s_dgram` also has pre-existing support for getting the correct MTU value from the OS using `BIO_CTRL_DGRAM_QUERY_MTU`. ### New APIs #### `BIO_sendmmsg` and `BIO_recvmmsg` **See also:** [BIO_sendmmsg(3)](https://www.openssl.org/docs/manmaster/man3/BIO_sendmmsg.html) The BIO interface features a new high-performance API for the execution of multiple read or write operations in a single system call, on supported OSes. On other OSes, a compatible fallback implementation is used. Unlike all other BIO APIs, this API is intended for concurrent threaded use and as such operates in a stateless fashion with regards to a BIO. This means, for example, that retry indications are made using explicit API inputs and outputs rather than setting an internal flag on the BIO. This new BIO API includes: - Local address support (getting the destination address of an incoming packet; setting the source address of an outgoing packet), where support for this is available; - Peer address support (setting the destination address of an outgoing packet; getting the source address of an incoming packet), where support for this is available. The following functionality was intentionally left out of this design because not all OSes can provide support: - Iovecs (which have also been determined not to be necessary for a performant QUIC implementation); - Features such as `MSG_DONTWAIT`, etc. This BIO API is intended to be extensible. For more information on this API, see BIO_sendmmsg(3) and BIO_recvmmsg(3). Custom BIO implementers may set their own implementation of these APIs via corresponding `BIO_meth` getter/setter functions. #### Truncation Mode **See also:** [BIO_s_dgram_pair(3)](https://www.openssl.org/docs/manmaster/man3/BIO_s_dgram_pair.html) The controls `BIO_dgram_set_no_trunc` (`BIO_CTRL_DGRAM_SET_NO_TRUNC`) and `BIO_dgram_get_no_trunc` (`BIO_CTRL_DGRAM_GET_NO_TRUNC`) are introduced. This is a boolean value which may be implemented by BIOs with datagram semantics. When enabled, attempting to receive a datagram such that the datagram would ordinarily be truncated (as per the design of the Berkeley sockets API) instead results in a failure. This is intended for implementation by `BIO_s_dgram_pair`. For compatibility, the default behaviour is off. #### Capability Negotiation **See also:** [BIO_s_dgram_pair(3)](https://www.openssl.org/docs/manmaster/man3/BIO_s_dgram_pair.html) Where a `BIO_s_dgram_pair` is used, there is the potential for such a memory BIO to be used by existing application code which is being adapted for use with QUIC. A problem arises whereby one end of a `BIO_s_dgram_pair` (for example, the side being used by OpenSSL's QUIC implementation) may assume that the other end supports certain capabilities (for example, specifying a peer address), when in actual fact the opposite end of the `BIO_s_dgram_pair` does not. A capability signalling mechanism is introduced which allows one end of a `BIO_s_dgram_pair` to indicate to the user of the opposite BIO the following capabilities and related information: - Whether source addresses the peer specifies will be processed. - Whether destination addresses the peer specifies will be processed. - Whether source addresses will be provided to the opposite BIO when it receives datagrams. - Whether destination addresses will be provided to the opposite BIO when it receives datagrams. The usage is as follows: - One side of a BIO pair calls `BIO_dgram_set_caps` with zero or more of the following flags to advertise its capabilities: - `BIO_DGRAM_CAP_HANDLES_SRC_ADDR` - `BIO_DGRAM_CAP_HANDLES_DST_ADDR` - `BIO_DGRAM_CAP_PROVIDES_SRC_ADDR` - `BIO_DGRAM_CAP_PROVIDES_DST_ADDR` - The other side of the BIO pair calls `BIO_dgram_get_effective_caps` to learn the effective capabilities of the BIO. These are the capabilities set by the opposite BIO. - The above process can also be repeated in the opposite direction. #### Local Address Support **See also:** [BIO_s_dgram_pair(3)](https://www.openssl.org/docs/manmaster/man3/BIO_s_dgram_pair.html) Support for local addressing (the reception of destination addresses for incoming packets, and the specification of source addresses for outgoing packets) varies by OS. Thus, it may not be available in all circumstances. A feature negotiation mechanism is introduced to facilitate this. `BIO_dgram_get_local_addr_cap` (`BIO_CTRL_DGRAM_GET_LOCAL_ADDR_CAP`) determines if a BIO is potentially capable of supporting local addressing on the current platform. If it determines that support is available, local addressing support must then be explicitly enabled via `BIO_dgram_set_local_addr_enable` (`BIO_CTRL_DGRAM_SET_LOCAL_ADDR_ENABLE`). If local addressing support has not been enabled, attempts to use local addressing (for example via `BIO_sendmmsg` or `BIO_recvmmsg` with a `BIO_MSG` with a non-NULL `local` field) fails. An explicit enablement call is required because setting up local addressing support requires system calls on most operating systems prior to sending or receiving packets and we do not wish to do this automatically inside the `BIO_sendmmsg`/`BIO_recvmmsg` fastpaths, particularly since the process of enabling support could fail due to lack of OS support, etc. `BIO_dgram_get_local_addr_enable` (`BIO_CTRL_DGRAM_GET_LOCAL_ADDR_ENABLE`) is also available. It is important to note that `BIO_dgram_get_local_addr_cap` is entirely distinct from the application capability negotiation mechanism discussed above. Whereas the capability negotiation mechanism discussed above allows *applications* to signal what they are capable of handling in their usage of a given BIO, `BIO_dgram_local_addr_cap` allows a *BIO implementation* to indicate to the users of that BIO whether it is able to support local addressing (where enabled). #### `BIO_s_dgram_pair` **See also:** [BIO_s_dgram_pair(3)](https://www.openssl.org/docs/manmaster/man3/BIO_s_dgram_pair.html) A new BIO implementation, `BIO_s_dgram_pair`, is provided. This is similar to the existing BIO pair but provides datagram semantics. It provides full support for the new APIs `BIO_sendmmsg`, `BIO_recvmmsg`, the capability negotiation mechanism described above, local address support and the MTU signalling mechanism described above. It can be instantiated using the new API `BIO_new_dgram_pair`. #### `BIO_POLL_DESCRIPTOR` The concept of *poll descriptors* are introduced. A poll descriptor is a tagged union structure which represents an abstraction over some unspecified kind of OS descriptor which can be used for synchronization and waiting. The most commonly used kind of poll descriptor is one which describes a network socket (i.e., on POSIX-like platforms, a file descriptor), however other kinds of poll descriptor may be defined. A BIO may be queried for whether it has a poll descriptor for read or write operations respectively: - Where `BIO_get_rpoll_descriptor` (`BIO_CTRL_GET_RPOLL_DESCRIPTOR`) is called, the BIO should output a poll descriptor which describes a resource which can be used to determine when the BIO will next become readable via a call to `BIO_read` or, if supported by the BIO, `BIO_recvmmsg`. - Where `BIO_get_wpoll_descriptor` (`BIO_CTRL_GET_WPOLL_DESCRIPTOR`) is called, the BIO should output a poll descriptor which describes a resource which can be used to determine when the BIO will next become writeable via a call to `BIO_write` or, if supported by the BIO, `BIO_sendmmsg`. A BIO may not necessarily be able to provide a poll descriptor. For example, memory-based BIOs such as `BIO_s_dgram_pair` do not correspond to any OS synchronisation resource, and thus the `BIO_get_rpoll_descriptor` and `BIO_get_wpoll_descriptor` calls are not supported for such BIOs. A BIO which supports these functions is known as pollable, and a BIO which does not is known as non-pollable. `BIO_s_dgram` supports these functions. The implementation of these functions for a `BIO_f_ssl` forwards to `SSL_get_rpoll_descriptor` and `SSL_get_wpoll_descriptor` respectively. The #### `BIO_s_dgram_mem` This is a basic memory buffer BIO with datagram semantics. Unlike `BIO_s_dgram_pair`, it is unidirectional and does not support peer addressing or local addressing. #### `BIO_err_is_non_fatal` A new predicate function `BIO_err_is_non_fatal` is defined which determines if an error code represents a non-fatal or transient error. For details, see [BIO_sendmmsg(3)](https://www.openssl.org/docs/manmaster/man3/BIO_sendmmsg.html). Q & A ----- To assist in understanding, when a “TBD” listed above is removed, or when a relevant question is raised, the resolution to the question will be placed here. **Q. Should `SSL_do_handshake` wait until the handshake is completed, or until it is confirmed?** **Note:** [The terms *handshake complete* and *handshake confirmed* are defined in RFC 9001 and have specific meanings.](https://www.rfc-editor.org/rfc/rfc9001.html#name-handshake-complete) A. `SSL_do_handshake` should wait until the handshake is completed, because handshake completion represents the completion of the cryptographic authentication of the connection. When a connection's handshake is completed, TLS 1.3 Finished messages have been exchanged by both parties, even if the handshake has not yet been *confirmed*. Moreover, RFC 9001 s. 4.1.2 states: >Additionally, a client MAY consider the handshake to be confirmed when it >receives an acknowledgment for a 1-RTT packet. This logically implies that it is OK for a client to start transmitting 1-RTT packets prior to handshake confirmation, otherwise there would be no in-flight 1-RTT packets for the client to receive ACKs for. **Q. Does `ENABLE_PARTIAL_WRITE` interact with blocking mode?** A. No; this mode is only relevant to non-blocking mode. In blocking mode, `SSL_write` always waits until all data is written unless an error occurs. The semantics of `SSL_write` are preserved unchanged. **Q. Does `SSL_write` block until data is written to the network, or simply until it is buffered?** A. `SSL_write` blocks until it has accepted responsibility for the data passed to it, just like `write(2)` or `send(2)`. In other words, it blocks until it can buffer the data. This does not necessarily mean that the data has actually been sent. **Q. How should connection closure work?** A. **RFC requirements.** After we begin terminating the connection by sending a `CONNECTION_CLOSE` frame, QUIC requires that we continue to process network I/O for a certain period of time so that any further traffic from the peer results in generation of a further `CONNECTION_CLOSE` frame. This is necessary to handle the possibility that the `CONNECTION_CLOSE` frame which was initially sent may be lost. **API issues.** This creates a complication because it implies that the connection closure process may take a fair amount of time, whereas existing API users will generally expect to be able to call `SSL_shutdown` and then immediately free the SSL object. However, if the caller immediately frees the SSL object, this precludes our implementing the applicable logic, at least on the client side. Moreover, existing API users are likely to tear down underlying network BIOs immediately after calling `SSL_free` anyway. In other words, any implementation based on secretly keeping QUIC state around after a call to `SSL_free` does not seem particularly workable on the client side. **Server side considerations.** There is more of a prospect here on the server side, since multiple connections will share the same socket, which will presumably be associated with some kind of enduring listener object. Thus when server support is implemented in the future connection teardown could be handled internally by maintaining the state of connections undergoing termination inside the listener object. However, similar caveats to those discussed here arise when the listener object itself is to be town down. (It is also possible we could optionally allow use of the server-style API to make multiple outgoing client connections with a non-zero-length client-side CID on the same underlying network BIO.) There are only really two ways to handle this: - **RFC conformant mode.** `SSL_shutdown` only indicates that shutdown is complete once the entire connection closure process is complete. This process consists of the Closing and Draining states. In some cases the Closing state may last only briefly, namely if the peer chooses to respond to our `CONNECTION_CLOSE` frame with a `CONNECTION_CLOSE` frame of its own. This allows immediate progression to the Draining state. However, a peer is *not* required to respond with such a frame. Thus in the worst case, this state can be as long as `3*PTO`; for example a peer with a high estimated RTT of 300ms would have us wait for 900ms. In the Draining state we simply ignore all incoming traffic and do not generate outgoing traffic. The purpose of this state is to simply tie up the socket and ensure any data still in flight is discarded. However, RFC 9000 states: Disposing of connection state prior to exiting the closing or draining state could result in an endpoint generating a Stateless Reset unnecessarily when it receives a late-arriving packet. Endpoints that have some alternative means to ensure that late-arriving packets do not induce a response, such as those that are able to close the UDP socket, MAY end these states earlier to allow for faster resource recovery. Servers that retain an open socket for accepting new connections SHOULD NOT end the closing or draining state early Because our client mode implementation uses one socket per connection, it appears to be reasonable based on the above text to omit the implementation of the draining state (the same may not be the case for the server role when implemented in the future). Thus, in general, `SSL_shutdown` can be expected to take about one round trip's time to complete when dealing with a peer whose QUIC implementation happens to respond to a `CONNECTION_CLOSE` frame with a `CONNECTION_CLOSE` frame of its own, and about three round trips otherwise. - **Rapid shutdown mode.** `SSL_shutdown` sends a `CONNECTION_CLOSE` frame once and completes immediately. The Closing and Draining states are not used, and if the `CONNECTION_CLOSE` frame was lost, the peer will have to wait for idle timeout to determine that the connection is gone (there is also the possibility that, if the socket is closed by the application after teardown, a peer will make something of ICMP Port Unreachable messages, but this is unlikely to be reliable and since this message is not authenticated, QUIC implementations probably shouldn't pay much attention to it anyway.) There is little problem with `SSL_shutdown` taking as long as it needs to for some long-running applications, but for others it poses a real issue. For example, a command-line tool which makes one connection, performs one application-specific transaction, and then tears down the connection. In this case an RFC-conformant connection termination would essentially require the process to hang around for a substantial amount of time after the work of the process is done. For this reason, it is concluded that both of these shutdown modes need to be offered. Where connection closure is initiated remotely rather than locally, only the draining state is relevant. Since we conclude above that we do not need to implement the draining state on the client side, this means that connection closure can be completed immediately in the case of a remote closure. **Q. Should we just map `SSL_handle_events` to `DTLS_CTRL_HANDLE_TIMEOUT` internally?** A. No, since the infinite time representation is different between the two calls. **Q. How should `STOP_SENDING` be supported?** We trigger `STOP_SENDING` automatically if an application frees the associated QUIC stream SSL object. **Q. Can data be received on a locally initiated bidirectional stream before any data is sent on that stream?** This is an interesting question without a clear answer to be found in the QUIC RFCs. A close reading of RFC 9000 suggests that the answer is, in principle, yes; however the RFC also grants explicit permission to make design choices in implementations which would preclude this: >An implementation might choose to defer allocating a stream ID to a stream until >it sends the first STREAM frame and enters this state, which can allow for >better stream prioritization. If an ID has not been allocated to a stream, obviously incoming data cannot be addressed to it. However, supposing that an implementation does not do this, RFC 9000 seems basically clear that it is valid for an application to create a stream locally, then receive data on it before sending anything: >The sending part of a stream that the endpoint initiates (types 0 and 2 for >clients, 1 and 3 for servers) is opened by the application. The "Ready" >state represents a newly created stream that is able to accept data from the >application. > >[...] > >For a bidirectional stream, the receiving part enters the "Recv" state when >the sending part initiated by the endpoint (type 0 for a client, type 1 for >a server) enters the "Ready" state. A peer is not generally notified of the creation of a stream which has not sent any data yet, since the creation of a stream is signalled only implicitly via the transmission of data in `STREAM` frames. However, a zero-length STREAM frame could presumably be used to effect such a notification. RFC 9000 contains no specific discussion of this possibility but does not preclude it. As such, in order to receive data on a locally-initiated bidirectional stream before sending any data on that stream, it would be necessary to either - Use a QUIC implementation which signals a bidirectional stream which has not yet sent any data via a zero-length stream frame, or - Use an application protocol which can inform the peer of the stream ID of the created stream in some application protocol-specific way. This is somewhat less plausible because it would require an API between the application and its QUIC library to inform the QUIC library that the peer has in fact created a stream with a given ID and to take its word for it. This is unlikely to be commonly available, especially as application errors in usage of such an API would lead to internal inconsistencies in QUIC connection state. Of course this discussion is somewhat esoteric as it is unclear why an application would want to create a locally-initiated stream and then have the peer transmit on it first, rather than simply use a remotely-initiated stream. Thus this discussion of this edge case is more of a curiosity, however for completeness it needs to be thought about in the API design. **Q. How should single-stream operation support locally and remotely-initiated streams?** Note that the ID of a stream depends on whether it is bidirectional and whether it is initiated by the client or server. Therefore, in single stream operation, it is necessary to know whether single-stream QUIC is being used with client-initiated or server-initiated stream initiation, and whether a bidirectional or unidirectional stream is being used; otherwise, we do not know which stream ID to bind to. The object of single stream operation is to support simple uses cases for simple applications. There seems no need to support esoteric usage of streams such as receiving first on a locally initiated stream here, thus we avoid supporting this to simplify the API. As such, an application which calls `SSL_write` on a QUIC connection SSL object before it calls `SSL_read` by definition is using a locally-initiated stream, and an application which does the opposite is using a remotely-initiated stream. We can use the ordering of initial calls to `SSL_read` and `SSL_write` to infer the desired stream type. Supporting locally-initiated streams (`SSL_write` called first) is simple; we automatically create the stream and queue data for transmission. Supporting remotely-initiated streams (`SSL_read` called first) is a little stranger. We could create the stream with the correct ID when cued to by the initial call to `SSL_read` implying use of a remotely-initiated stream. However, this would mean we are creating state tracking a remotely-initiated stream before the peer has signalled it. This would work in the happy case where the client is connected to a compatible server but may result in strange inconsistencies of QUIC internal state if a client is accidentally connected to an incompatible peer. Since the peer ought to be the authority on the streams it creates, this seems like an undesirable approach. Ergo, creation of a default remotely-initiated stream needs to be deferred until the *peer* signals such a stream. This leads naturally to a "first stream wins" model of implementation: - When a QUIC connection SSL object is created, default stream mode is enabled, meaning that a default stream will be bound to the QUIC connection SSL object at the earliest available opportunity. However, no default stream is bound yet. - One of the following events happened — whichever happens first wins: - The local application calls `SSL_write()` (`len > 0`). A locally-initiated stream with ordinal 0 is created. The stream is bidirectional by default but this can be changed. This stream is bound as the default stream. - The peer creates a stream. This stream is bound as the default stream. If the local application calls `SSL_read()` before either of the above occur, `SSL_read()` fails as though no data is available until one of the above events occurs. Once one of the above events occurs, any additional stream created by the peer is automatically terminated using both `STOP_SENDING` and `STREAM_RESET` frames (to terminate both the receiving and sending parts respectively) and there is no API-visible effect to the local application (unless the application explicitly opts into supporting additional streams).