threads: follow formatting rules

Adjust long lines and correct padding in preprocessor lines to
match the formatting rules

Signed-off-by: Georgi Valkov <gvalkov@gmail.com>

threads: follow formatting rules

Adjust long lines and correct padding in preprocessor lines to
match the formatting rules

Signed-off-by: Georgi Valkov <gvalkov@gmail.com>

Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24941)

show more ...

disable rwlocks on nonstop klt model

It appears nonstops new threading model defines some level of rwlock
pthread api, but its not working properly. Disable rwlocks for
_KLT_MODEL_

disable rwlocks on nonstop klt model

It appears nonstops new threading model defines some level of rwlock
pthread api, but its not working properly. Disable rwlocks for
_KLT_MODEL_ for now

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
(Merged from https://github.com/openssl/openssl/pull/24969)

show more ...

Fix CRYPTO_atomic_store

If the implementation of this function falls to using a pthread lock to
update a value, it should be a write lock, not a read lock

Reviewed-by: Tomas Mra

Fix CRYPTO_atomic_store

If the implementation of this function falls to using a pthread lock to
update a value, it should be a write lock, not a read lock

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
(Merged from https://github.com/openssl/openssl/pull/24969)

show more ...

threads_pthread, threads_win: improve code consistency

Improve code consistency between threads_pthread.c and threads_win.c
threads_pthread.c has good comments, let's copy them to thread

threads_pthread, threads_win: improve code consistency

Improve code consistency between threads_pthread.c and threads_win.c
threads_pthread.c has good comments, let's copy them to threads_win.c
In many places uint64_t or LONG int was used, and assignments were
performed between variables with different sizes.
Unify the code to use uint32_t. In 32 bit architectures it is easier
to perform 32 bit atomic operations. The size is large enough to hold
the list of operations.
Fix result of atomic_or_uint_nv improperly casted to int *
instead of int.

Note:
In general size_t should be preferred for size and index, due to its
descriptive name, however it is more convenient to use uint32_t for
consistency between platforms and atomic calls.

READER_COUNT and ID_VAL return results that fit 32 bit. Cast them to
uint32_t to save a few CPU cycles, since they are used in 32 bit
operations anyway.

TODO:
In struct rcu_lock_st, qp_group can be moved before id_ctr
for better alignment, which would save 8 bytes.

allocate_new_qp_group has a parameter count of type int.
Signed values should be avoided as size or index.
It is better to use unsigned, e.g uint32_t, even though
internally this is assigned to a uint32_t variable.

READER_SIZE is 16 in threads_pthread.c, and 32 in threads_win.c
Using a common size for consistency should be prefered.

Signed-off-by: Georgi Valkov <gvalkov@gmail.com>

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24803)

show more ...

threads_win: fix build error with VS2010 x86

InterlockedAnd64 and InterlockedAdd64 are not available on VS2010 x86.
We already have implemented replacements for other functions, such as

threads_win: fix build error with VS2010 x86

InterlockedAnd64 and InterlockedAdd64 are not available on VS2010 x86.
We already have implemented replacements for other functions, such as
InterlockedOr64. Apply the same approach to fix the errors.
A CRYPTO_RWLOCK rw_lock is added to rcu_lock_st.

Replace InterlockedOr64 and InterlockedOr with CRYPTO_atomic_load and
CRYPTO_atomic_load_int, using the existing design pattern.

Add documentation and tests for the new atomic functions
CRYPTO_atomic_add64, CRYPTO_atomic_and

Fixes:
libcrypto.lib(libcrypto-lib-threads_win.obj) : error LNK2019: unresolved external symbol _InterlockedAdd64 referenced in function _get_hold_current_qp
libcrypto.lib(libcrypto-lib-threads_win.obj) : error LNK2019: unresolved external symbol _InterlockedOr referenced in function _get_hold_current_qp
libcrypto.lib(libcrypto-lib-threads_win.obj) : error LNK2019: unresolved external symbol _InterlockedAnd64 referenced in function _update_qp
libcrypto.lib(libcrypto-lib-threads_win.obj) : error LNK2019: unresolved external symbol _InterlockedOr64 referenced in function _ossl_synchronize_rcu

Signed-off-by: Georgi Valkov <gvalkov@gmail.com>

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24405)

show more ...

Some minor nit corrections in the thread code for rcu

Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.or

Some minor nit corrections in the thread code for rcu

Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24630)

(cherry picked from commit d38d2642287ef9a22f20e662a19c217c227043a6)

show more ...

threads_pthread.c: change inline to ossl_inline

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/op

threads_pthread.c: change inline to ossl_inline

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24502)

show more ...

Add CRYPTO_atomic_store api

Generally we can get away with just using CRYPTO_atomic_load to do
stores by reversing the source and target variables, but doing so
creates a problem for

Add CRYPTO_atomic_store api

Generally we can get away with just using CRYPTO_atomic_load to do
stores by reversing the source and target variables, but doing so
creates a problem for the thread sanitizer as CRYPTO_atomic_load hard
codes an __ATOMIC_ACQUIRE constraint, which confuses tsan into thinking
that loads and stores aren't properly ordered, leading to RAW/WAR
hazzards getting reported. Instead create a CRYPTO_atomic_store api
that is identical to the load variant, save for the fact that the value
is a unit64_t rather than a pointer that gets stored using an
__ATOMIC_RELEASE constraint, satisfying tsan.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23671)

show more ...

Fix list appending in win ossl_rcu_call

The ossl_rcu_call function for windows creates a linked list loop. fix
it to work like the pthread version properly

Reviewed-by: Tomas M

Fix list appending in win ossl_rcu_call

The ossl_rcu_call function for windows creates a linked list loop. fix
it to work like the pthread version properly

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23671)

show more ...

Make thread sanitizer cope with rcu locks

This is unfortunate, but seems necessecary

tsan in gcc/clang tracks data races by recording memory references made
while various locks

Make thread sanitizer cope with rcu locks

This is unfortunate, but seems necessecary

tsan in gcc/clang tracks data races by recording memory references made
while various locks are held. If it finds that a given address is
read/written while under lock (or under no locks without the use of
atomics), it issues a warning

this creates a specific problem for rcu, because on the write side of a
critical section, we write data under the protection of a lock, but by
definition the read side has no lock, and so rcu warns us about it,
which is really a false positive, because we know that, even if a
pointer changes its value, the data it points to will be valid.

The best way to fix it, short of implementing tsan hooks for rcu locks
in any thread sanitizer in the field, is to 'fake it'. If thread
sanitization is activated, then in ossl_rcu_write_[lock|unlock] we add
annotations to make the sanitizer think that, after the write lock is
taken, that we immediately unlock it, and lock it right before we unlock
it again. In this way tsan thinks there are no locks held while
referencing protected data on the read or write side.

we still need to use atomics to ensure that tsan recognizes that we are
doing atomic accesses safely, but thats ok, and we still get warnings if
we don't do that properly

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23671)

show more ...

crypto/threads_pthread.c: Fix typos found by codespell

CLA: trivial

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: T

crypto/threads_pthread.c: Fix typos found by codespell

CLA: trivial

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24206)

show more ...

Make rcu_thread_key context-aware

Currently, rcu has a global bit of data, the CRYPTO_THREAD_LOCAL object
to store per thread data. This works in some cases, but fails in FIPS,
becu

Make rcu_thread_key context-aware

Currently, rcu has a global bit of data, the CRYPTO_THREAD_LOCAL object
to store per thread data. This works in some cases, but fails in FIPS,
becuase it contains its own copy of the global key.

So
1) Make the rcu_thr_key a per-context variable, and force
ossl_rcu_lock_new to be context aware

2) Store a pointer to the context in the lock object

3) Use the context to get the global thread key on read/write lock

4) Use ossl_thread_start_init to properly register a cleanup on thread
exit

5) Fix up missed calls to OSSL_thread_stop() in our tests

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24162)

show more ...

crypto/threads_pthread.c: refactor all atomics fallbacks for type safety

The atomics fallbacks were using 'void *' as a generic transport for all
possible scalar and pointer types, with

crypto/threads_pthread.c: refactor all atomics fallbacks for type safety

The atomics fallbacks were using 'void *' as a generic transport for all
possible scalar and pointer types, with the hypothesis that a pointer is
as large as the largest possible scalar type that we would use.

Then enters the use of uint64_t, which is larger than a pointer on any
32-bit system (or any system that has 32-bit pointer configurations).

We could of course choose a larger type as a generic transport. However,
that only pushes the problem forward in time... and it's still a hack.
It's therefore safer to reimplement the fallbacks per type that atomics
are used for, and deal with missing per type fallbacks when the need
arrises in the future.

For test build purposes, the macro USE_ATOMIC_FALLBACKS is introduced.
If OpenSSL is configured with '-DUSE_ATOMIC_FALLBACKS', the fallbacks
will be used, unconditionally.

Fixes #24096

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24123)

show more ...

crypto/threads_pthread.c: Cleanup misaligned preprocessor directives

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://g

crypto/threads_pthread.c: Cleanup misaligned preprocessor directives

Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24123)

show more ...

Ensure proper memory barriers around ossl_rcu_deref/ossl_rcu_assign_ptr

Since the addition of macos14 M1 runners in our CI jobs we've been
seeing periodic random failures in the test_thr

Ensure proper memory barriers around ossl_rcu_deref/ossl_rcu_assign_ptr

Since the addition of macos14 M1 runners in our CI jobs we've been
seeing periodic random failures in the test_threads CI job.
Specifically we've seen instances in which the shared pointer in the
test (which points to a monotonically incrementing uint64_t went
backwards.

From taking a look at the disassembled code in the failing case, we see
that __atomic_load_n when emitted in clang 15 looks like this
0000000100120488 <_ossl_rcu_uptr_deref>:
100120488: f8bfc000 ldapr x0, [x0]
10012048c: d65f03c0 ret

Notably, when compiling with gcc on the same system we get this output
instead:
0000000100120488 <_ossl_rcu_uptr_deref>:
100120488: f8bfc000 ldar x0, [x0]
10012048c: d65f03c0 ret

Checking the arm docs for the difference between ldar and ldapr:
https://developer.arm.com/documentation/ddi0602/2023-09/Base-Instructions/LDAPR--Load-Acquire-RCpc-Register-
https://developer.arm.com/documentation/dui0802/b/A64-Data-Transfer-Instructions/LDAR

It seems that the ldar instruction provides a global cpu fence, not
completing until all writes in a given cpus writeback queue have
completed

Conversely, the ldapr instruction attmpts to achieve performance
improvements by honoring the Local Ordering register available in the
system coprocessor, only flushing writes in the same address region as
other cpus on the system.

I believe that on M1 virtualized cpus the ldapr is not properly ordering
writes, leading to an out of order read, despite the needed fencing.
I've opened an issue with apple on this here:
https://developer.apple.com/forums/thread/749530

I believe that it is not safe to issue an ldapr instruction unless the
programmer knows that the Local order registers are properly configured
for use on the system.

So to fix it I'm proposing with this patch that we, in the event that:
1) __APPLE__ is defined
AND
2) __clang__ is defined
AND
3) __aarch64__ is defined

during the build, that we override the ATOMIC_LOAD_N macro in the rcu
code such that it uses a custom function with inline assembly to emit
the ldar instruction rather than the ldapr instruction. The above
conditions should get us to where this is only used on more recent MAC
cpus, and only in the case where the affected clang compiler emits the
offending instruction.

I've run this patch 10 times in our CI and failed to reproduce the
issue, whereas previously I could trigger it within 5 runs routinely.

Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23974)

show more ...

Copyright year updates

Reviewed-by: Neil Horman <nhorman@openssl.org>
Release: yes
(cherry picked from commit 0ce7d1f355c1240653e320a3f6f8109c1f05f8c0)

Reviewed-by: Hugo Lan

Copyright year updates

Reviewed-by: Neil Horman <nhorman@openssl.org>
Release: yes
(cherry picked from commit 0ce7d1f355c1240653e320a3f6f8109c1f05f8c0)

Reviewed-by: Hugo Landau <hlandau@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24034)

show more ...

Fix the grammar as suggsted in the review

Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl

Fix the grammar as suggsted in the review

Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23545)

show more ...

Fix new typos found by codespell

Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/235

Fix new typos found by codespell

Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23545)

show more ...

RCU lock implementation

Introduce an RCU lock implementation as an alternative locking mechanism
to openssl. The api is documented in the ossl_rcu.pod
file

Read side implem

RCU lock implementation

Introduce an RCU lock implementation as an alternative locking mechanism
to openssl. The api is documented in the ossl_rcu.pod
file

Read side implementaiton is comparable to that of RWLOCKS:
ossl_rcu_read_lock(lock);
<
critical section in which data can be accessed via
ossl_derefrence
>
ossl_rcu_read_unlock(lock);

Write side implementation is:
ossl_rcu_write_lock(lock);
<
critical section in which data can be updated via
ossl_assign_pointer
and stale data can optionally be scheduled for removal
via ossl_rcu_call
>
ossl_rcu_write_unlock(lock);
...
ossl_synchronize_rcu(lock);

ossl_rcu_call fixup

Reviewed-by: Hugo Landau <hlandau@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/22729)

show more ...

Copyright year updates


Reviewed-by: Richard Levitte <levitte@openssl.org>
Release: yes

issue-21718: remove setting of PTHREAD_MUTEX_NORMAL

issue: https://github.com/openssl/openssl/issues/21718

build break reported:
crypto/threads_pthread.c:76:5: warning: implicit

issue-21718: remove setting of PTHREAD_MUTEX_NORMAL

issue: https://github.com/openssl/openssl/issues/21718

build break reported:
crypto/threads_pthread.c:76:5: warning: implicit declaration of function 'pthread_mutexattr_settype'; did you mean 'pthread_mutexattr_destroy'? [-Wimplicit-function-declaration]
76 | pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_NORMAL);
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| pthread_mutexattr_destroy
crypto/threads_pthread.c:76:38: error: 'PTHREAD_MUTEX_NORMAL' undeclared (first use in this function); did you mean 'PTHREAD_MUTEX_TIMED_NP'?
76 | pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_NORMAL);
| ^~~~~~~~~~~~~~~~~~~~
| PTHREAD_MUTEX_TIMED_NP

This occurs because PTHREAD_MUTEX_NORMAL is only defined in glibc if
__USE_UNIX98 or __USE_XOPEN2K8 is defined, which is derived from setting
__USE_POSIX_C_SOURCE or __XOPEN_SOURCE is selected in the glibc feature
set for a build. Since openssl selects no specific feature set from
glibc, the build break occurs

We could select a feature set of course, but that seems like a
significant discussion to have prior to doing so. Instead, the simpler
solution is to just not set the mutex type at all, given that
pthread_mutexattr_init sets the default mutex type, which should be akin
to normal anyway (i.e. no mutex error checking or allowed-recursive
behavior)

Reviewed-by: Hugo Landau <hlandau@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21726)

show more ...

QUIC: Fix bugs where threading is disabled

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pu

QUIC: Fix bugs where threading is disabled

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20856)

show more ...

CRYPTO_THREAD_lock_new(): Avoid infinite recursion on allocation error

Fixes #19334

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>

CRYPTO_THREAD_lock_new(): Avoid infinite recursion on allocation error

Fixes #19334

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19335)

show more ...

Update copyright year

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Release: yes

Never use `__atomic_*` on macOS 10.7 and 10.8

macOS 10.7 and 10.8 had a bit wired clang which is detected as
`__GNUC__` which has `__ATOMIC_ACQ_REL` but it excepts one option at
`__a

Never use `__atomic_*` on macOS 10.7 and 10.8

macOS 10.7 and 10.8 had a bit wired clang which is detected as
`__GNUC__` which has `__ATOMIC_ACQ_REL` but it excepts one option at
`__atomic_is_lock_free` instead of 2.

This prevents OpenSSL to be compiled on such systems.

Fixes: #18055

Signed-off-by: Kirill A. Korinsky <kirill@korins.ky>

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18056)

show more ...