Segfault and data race between OPENSSL_sk_{num, insert, value} in X509 store/lookup #24480
Description
I have a core dump and separately ThreadSanitizer output, that points to the safe-stack methods (OPENSSL_sk_num, OPENSSL_sk_insert, OPENSSL_sk_value) in OpenSSL 3.0.8 and was wondering if these methods are safely callable from multiple threads, while making a TLS handshake and reading the CA X509 certificates.
There is a small repro as well. It uses CPython 3.12 and OpenSSL 3.0.8, calling S3 endpoints over HTTPS from multiple threads. Total 100 requests. This leads to tsan errors below and it segfaults around 1 in 100000 (approximate) runs. S3 may be irrelevant as the errors happen while doing TLS handshakes / reading and verifying CA certs from different threads.
I am happy to move the issue to the CPython repository, if there is nothing to be done in OpenSSL codebase.
Core dump
CPython 3.12 + OpenSSL 3.0.8
(gdb) info th
Id Target Id Frame
* 1 LWP 22 X509_LOOKUP_by_subject_ex (ctx=0x0, type=type@entry=X509_LU_X509, name=name@entry=0x7fdad0017370,
ret=ret@entry=0x7fdadee44550, libctx=0x0, propq=0x0) at crypto/x509/x509_lu.c:96
2 LWP 20 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fdae4950800, op=137, expected=0,
futex_word=0x7fdae69a11a4 <_PyRuntime+77508>) at futex-internal.c:57
3 LWP 25 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fdadd5417d0, op=137, expected=0,
futex_word=0x7fdae69a11a0 <_PyRuntime+77504>) at futex-internal.c:57
4 LWP 21 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fdadf645730, op=137, expected=0,
futex_word=0x7fdae69a11a0 <_PyRuntime+77504>) at futex-internal.c:57
5 LWP 24 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7fdaddd42560, op=137, expected=0,
futex_word=0x7fdae69a11a4 <_PyRuntime+77508>) at futex-internal.c:57
6 LWP 16 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0,
futex_word=0x28a6a30) at futex-internal.c:57
7 LWP 23 futex_wait (private=0, expected=5, futex_word=0x7fdae69a1188 <_PyRuntime+77480>)
at ../sysdeps/nptl/futex-internal.h:146
(gdb) bt
#0 X509_LOOKUP_by_subject_ex (ctx=0x0, type=type@entry=X509_LU_X509, name=name@entry=0x7fdad0017370, ret=ret@entry=0x7fdadee44550,
libctx=0x0, propq=0x0) at crypto/x509/x509_lu.c:96
#1 0x00007fdae562f67e in X509_STORE_CTX_get_by_subject (vs=vs@entry=0x7fdad005e500, type=type@entry=X509_LU_X509,
name=name@entry=0x7fdad0017370, ret=ret@entry=0x7fdad00ee420) at crypto/x509/x509_lu.c:333
#2 0x00007fdae5633982 in X509_STORE_CTX_get1_issuer (issuer=0x7fdadee44660, ctx=0x7fdad005e500, x=0x7fdad001cae0)
at crypto/x509/x509_lu.c:736
#3 0x00007fdae563c4a8 in get1_trusted_issuer (cert=0x7fdad001cae0, ctx=0x7fdad005e500, issuer=0x7fdadee44660)
at crypto/x509/x509_vfy.c:2982
#4 build_chain (ctx=0x7fdad005e500) at crypto/x509/x509_vfy.c:3103
#5 0x00007fdae56343d6 in verify_chain (ctx=0x7fdad005e500) at crypto/x509/x509_vfy.c:217
#6 0x00007fdae5635532 in X509_verify_cert (ctx=ctx@entry=0x7fdad005e500) at crypto/x509/x509_vfy.c:296
#7 0x00007fdae4f8c0d8 in ssl_verify_cert_chain (s=0x7fdad002d800, sk=<optimized out>) at ssl/ssl_cert.c:436
#8 0x00007fdae4fc568f in ssl_verify_cert_chain (sk=0x7fdad007cfd0, s=0x7fdad002d800) at ssl/ssl_cert.c:380
#9 tls_post_process_server_certificate (wst=<optimized out>, s=0x7fdad002d800) at ssl/statem/statem_clnt.c:1870
#10 ossl_statem_client_post_process_message (s=s@entry=0x7fdad002d800, wst=<optimized out>) at ssl/statem/statem_clnt.c:1085
#11 0x00007fdae4fbf5c8 in read_state_machine (s=0x7fdad002d800) at ssl/statem/statem.c:675
#12 state_machine (s=0x7fdad002d800, server=<optimized out>) at ssl/statem/statem.c:442
#13 0x00007fdae501ad9d in _ssl__SSLSocket_do_handshake_impl (self=0x7fdadcaba0e0) at ./Modules/_ssl.c:962
#14 _ssl__SSLSocket_do_handshake (self=0x7fdadcaba0e0, _unused_ignored=<optimized out>) at ./Modules/clinic/_ssl.c.h:25
#15 0x00007fdae656bd23 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fdae401b3b8, throwflag=<optimized out>)
at Python/bytecodes.c:3150
#16 0x00007fdae65bb228 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdae401a2b0, tstate=0x2844760) at ./Include/internal/pycore_ceval.h:89
#17 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7fdadee44ae0, locals=0x0, func=0x7fdae4a05300,
tstate=0x2844760) at Python/ceval.c:1683
#18 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7fdadee44ae0, func=0x7fdae4a05300)
at Objects/call.c:419
#19 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=0x7fdadee44ae0, callable=0x7fdae4a05300,
tstate=<optimized out>) at ./Include/internal/pycore_call.h:92
#20 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
at Objects/classobject.c:91
#21 0x00007fdae6568e5f in PyCFunction_Call (kwargs=0x7fdadf923280, args=0x7fdadf8fe840, callable=0x7fdadf90e300) at Objects/call.c:387
#22 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fdae401a228, throwflag=<optimized out>) at Python/bytecodes.c:3254
#23 0x00007fdae65bb8e5 in _PyEval_EvalFrame (throwflag=0, frame=0x7fdae401a020, tstate=0x2844760) at ./Include/internal/pycore_ceval.h:89
#24 _PyEval_Vector (kwnames=<optimized out>, argcount=<optimized out>, args=0x7fdadee44da8, locals=0x0, func=0x7fdae5d22340,
tstate=0x2844760) at Python/ceval.c:1683
#25 _PyFunction_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, stack=0x7fdadee44da8, func=0x7fdae5d22340)
at Objects/call.c:419
#26 _PyObject_VectorcallTstate (tstate=<optimized out>, callable=0x7fdae5d22340, args=0x7fdadee44da8, nargsf=<optimized out>,
kwnames=<optimized out>) at ./Include/internal/pycore_call.h:92
#27 0x00007fdae65bb2b8 in method_vectorcall (method=0x7fdadf90e780, args=0x7fdae69a0ce0 <_PyRuntime+76288>, nargsf=0, kwnames=0x0)
at Objects/classobject.c:69
#28 0x00007fdae669e125 in thread_run (boot_raw=0x27f0790) at ./Modules/_threadmodule.c:1114
#29 0x00007fdae665d334 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:237
#30 0x00007fdae61157f2 in start_thread (arg=<optimized out>) at pthread_create.c:443
#31 0x00007fdae60b5314 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100Looking into the disassembly of X509_LOOKUP_by_subject_ex, it is clear that ctx arg is derefecenced while being 0, causing the segfault, which corresponds to L96 ctx->skip in
Lines 92 to 106 in 31157bc
Null ctx is passed from
Lines 331 to 333 in 31157bc
sk_X509_LOOKUP_num and then sk_X509_LOOKUP_value.
It could be that sk_X509_LOOKUP_num returns a value greater than zero, and in the for loop, sk_X509_LOOKUP_value returns zero, causing segfault inside X509_LOOKUP_by_subject_ex, since this code path is exercised from multiple threads. ThreadSanitizer warnings below corroborate this.
Note that CPython enters libcrypto via _ssl__SSLSocket_do_handshake_impl at https://github.com/python/cpython/blob/f6650f9ad73359051f3e558c2431a109bc016664/Modules/_ssl.c#L962
ThreadSanitizer output
CPython 3.12 + OpenSSL 3.0.8 with tsan:
There are two types of data races discovered by tsan, each of them are detected multiple times while making 100 TLS requests in the repro.
Data Race - Type 1
WARNING: ThreadSanitizer: data race (pid=124388)
Read of size 4 at 0x7b080002c080 by thread T24:
#0 OPENSSL_sk_num crypto/stack/stack.c:440 (libcrypto.so.3+0x0000003b0021)
#1 X509_STORE_add_lookup crypto/x509/x509_lu.c:271 (libcrypto.so.3+0x000000401e1b)
#2 X509_STORE_load_file_ex crypto/x509/x509_d2.c:51 (libcrypto.so.3+0x0000003ffc82)
#3 SSL_CTX_load_verify_file ssl/ssl_lib.c:4425 (libssl.so.3+0x000000054f35)
#4 SSL_CTX_load_verify_locations ssl/ssl_lib.c:4445 (libssl.so.3+0x000000055076)
#5 <null> (_ssl.cpython-312-x86_64-linux-gnu.so+0x0000000175fb)
Previous write of size 4 at 0x7b080002c080 by thread T1:
#0 OPENSSL_sk_insert crypto/stack/stack.c:281 (libcrypto.so.3+0x0000003af59d)
#1 OPENSSL_sk_push crypto/stack/stack.c:388 (libcrypto.so.3+0x0000003afcbe)
#2 X509_STORE_add_lookup crypto/x509/x509_lu.c:285 (libcrypto.so.3+0x000000401ecd)
#3 X509_STORE_load_file_ex crypto/x509/x509_d2.c:51 (libcrypto.so.3+0x0000003ffc82)
#4 SSL_CTX_load_verify_file ssl/ssl_lib.c:4425 (libssl.so.3+0x000000054f35)
#5 SSL_CTX_load_verify_locations ssl/ssl_lib.c:4445 (libssl.so.3+0x000000055076)
#6 <null> (_ssl.cpython-312-x86_64-linux-gnu.so+0x0000000175fb)
Location is heap block of size 32 at 0x7b080002c080 allocated by main thread:
#0 malloc <null> (libtsan.so.0.0.0+0x000000027ff1)
#1 CRYPTO_malloc crypto/mem.c:190 (libcrypto.so.3+0x0000002eb38f)
#2 CRYPTO_zalloc crypto/mem.c:197 (libcrypto.so.3+0x0000002eb3d7)
#3 OPENSSL_sk_new_reserve crypto/stack/stack.c:228 (libcrypto.so.3+0x0000003af23c)
#4 OPENSSL_sk_new_null crypto/stack/stack.c:131 (libcrypto.so.3+0x0000003aee12)
#5 X509_STORE_new crypto/x509/x509_lu.c:193 (libcrypto.so.3+0x0000004017e0)
#6 SSL_CTX_new_ex ssl/ssl_lib.c:3262 (libssl.so.3+0x000000050c8e)
#7 SSL_CTX_new ssl/ssl_lib.c:3414 (libssl.so.3+0x00000005135d)
#8 <null> (_ssl.cpython-312-x86_64-linux-gnu.so+0x000000017ad3)
Thread T24 (tid=124423, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0.0.0+0x000000028f93)
#1 <null> (libpython3.12.so.1.0+0x000000303e2f)
Thread T1 (tid=124400, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0.0.0+0x000000028f93)
#1 <null> (libpython3.12.so.1.0+0x000000303e2f)
SUMMARY: ThreadSanitizer: data race crypto/stack/stack.c:440 in OPENSSL_sk_num
Data Race - Type 2
WARNING: ThreadSanitizer: data race (pid=124388)
Read of size 8 at 0x7b080002c088 by thread T9:
#0 OPENSSL_sk_value crypto/stack/stack.c:447 (libcrypto.so.3+0x0000003b0093)
#1 X509_STORE_add_lookup crypto/x509/x509_lu.c:272 (libcrypto.so.3+0x000000401dd8)
#2 X509_STORE_load_file_ex crypto/x509/x509_d2.c:51 (libcrypto.so.3+0x0000003ffc82)
#3 SSL_CTX_load_verify_file ssl/ssl_lib.c:4425 (libssl.so.3+0x000000054f35)
#4 SSL_CTX_load_verify_locations ssl/ssl_lib.c:4445 (libssl.so.3+0x000000055076)
#5 <null> (_ssl.cpython-312-x86_64-linux-gnu.so+0x0000000175fb)
Previous write of size 8 at 0x7b080002c088 by thread T1:
#0 sk_reserve crypto/stack/stack.c:195 (libcrypto.so.3+0x0000003aeff2)
#1 OPENSSL_sk_insert crypto/stack/stack.c:271 (libcrypto.so.3+0x0000003af459)
#2 OPENSSL_sk_push crypto/stack/stack.c:388 (libcrypto.so.3+0x0000003afcbe)
#3 X509_STORE_add_lookup crypto/x509/x509_lu.c:285 (libcrypto.so.3+0x000000401ecd)
#4 X509_STORE_load_file_ex crypto/x509/x509_d2.c:51 (libcrypto.so.3+0x0000003ffc82)
#5 SSL_CTX_load_verify_file ssl/ssl_lib.c:4425 (libssl.so.3+0x000000054f35)
#6 SSL_CTX_load_verify_locations ssl/ssl_lib.c:4445 (libssl.so.3+0x000000055076)
#7 <null> (_ssl.cpython-312-x86_64-linux-gnu.so+0x0000000175fb)
Location is heap block of size 32 at 0x7b080002c080 allocated by main thread:
#0 malloc <null> (libtsan.so.0.0.0+0x000000027ff1)
#1 CRYPTO_malloc crypto/mem.c:190 (libcrypto.so.3+0x0000002eb38f)
#2 CRYPTO_zalloc crypto/mem.c:197 (libcrypto.so.3+0x0000002eb3d7)
#3 OPENSSL_sk_new_reserve crypto/stack/stack.c:228 (libcrypto.so.3+0x0000003af23c)
#4 OPENSSL_sk_new_null crypto/stack/stack.c:131 (libcrypto.so.3+0x0000003aee12)
#5 X509_STORE_new crypto/x509/x509_lu.c:193 (libcrypto.so.3+0x0000004017e0)
#6 SSL_CTX_new_ex ssl/ssl_lib.c:3262 (libssl.so.3+0x000000050c8e)
#7 SSL_CTX_new ssl/ssl_lib.c:3414 (libssl.so.3+0x00000005135d)
#8 <null> (_ssl.cpython-312-x86_64-linux-gnu.so+0x000000017ad3)
Thread T9 (tid=124408, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0.0.0+0x000000028f93)
#1 <null> (libpython3.12.so.1.0+0x000000303e2f)
Thread T1 (tid=124400, running) created by main thread at:
#0 pthread_create <null> (libtsan.so.0.0.0+0x000000028f93)
#1 <null> (libpython3.12.so.1.0+0x000000303e2f)
SUMMARY: ThreadSanitizer: data race crypto/stack/stack.c:447 in OPENSSL_sk_value
Repro
CPython 3.12 + OpenSSL 3.0.8 + boto3
import concurrent.futures
import boto3
REGION_NAME = 'eu-central-1'
# S3 bucket doesn't have to exist.
BUCKET_NAME = 'xxx'
class S3Uploader:
def __init__(self):
self.jitter_factor = 0.1
self.s3_client = self._get_s3_client()
def _get_s3_client(self):
return boto3.client("s3", region_name=REGION_NAME)
def upload_to_s3(self, bucket_name: str, object_key: str, data: str):
self.s3_client.put_object(
Bucket=bucket_name, Key=object_key, Body=data)
class BatchProcessor:
def __init__(self, s3_client: S3Uploader) -> None:
self.s3_client = s3_client
self.executor = concurrent.futures.ThreadPoolExecutor() # max_workers=30
def process_single(self):
print("Uploading to S3...")
self.s3_client.upload_to_s3(BUCKET_NAME, "xxx", "xxx")
def process_records(self, count) -> int:
futures = []
for _ in range(0, count):
futures.append(self.executor.submit(self.process_single))
concurrent.futures.wait(futures)
batch_processor = BatchProcessor(S3Uploader())
def main():
fake_message_count = 100
batch_processor.process_records(fake_message_count)
main()