Low Memory SegFault in dtls #13649
Description
This is from latest OpenSSL 1.1.1 stable branch.
Testing with forced low memory results in a segmentation fault for failing paths of dtls1_new().
While trying to locate the proper code path for this segfault I instrumented the code to simulate
each failing path in dtls1_new() and if a malloc fails the resultant cleanup triggers a segfault.
Depending on the failing location in dtls1_new() the segfault is one of two, this one:
Program received signal SIGSEGV, Segmentation fault.
DTLS_RECORD_LAYER_clear (rl=rl@entry=0x6cc510) at ssl/record/rec_layer_d1.c:68
68 while ((item = pqueue_pop(d->unprocessed_rcds.q)) != NULL) {
(gdb) bt
#0 DTLS_RECORD_LAYER_clear (rl=rl@entry=0x6cc510)
at ssl/record/rec_layer_d1.c:68
#1 0x00007ffff7b6a259 in DTLS_RECORD_LAYER_free (rl=rl@entry=0x6cc510)
at ssl/record/rec_layer_d1.c:49
#2 0x00007ffff7b67770 in dtls1_free (s=0x6cbcd0) at ssl/d1_lib.c:141
#3 0x00007ffff7b85097 in SSL_free (s=0x6cbcd0) at ssl/ssl_lib.c:1215
#4 0x00007ffff7b863b0 in SSL_new (ctx=ctx@entry=0x6c93f0)
at ssl/ssl_lib.c:850
#5 0x0000000000442f7e in s_client_main (argc=<optimized out>,
argv=<optimized out>) at apps/s_client.c:1980
#6 0x00000000004326dd in do_cmd (prog=prog@entry=0x6bbbe0,
argc=argc@entry=5, argv=argv@entry=0x7fffffffde70) at apps/openssl.c:570
#7 0x000000000041e738 in main (argc=5, argv=0x7fffffffde70)
at apps/openssl.c:189
(gdb)
or this one:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7b676e8 in dtls1_clear_received_buffer (s=s@entry=0x6cbcd0)
at ssl/d1_lib.c:119
119 while ((item = pqueue_pop(s->d1->buffered_messages)) != NULL) {
(gdb) bt
#0 0x00007ffff7b676e8 in dtls1_clear_received_buffer (s=s@entry=0x6cbcd0)
at ssl/d1_lib.c:119
#1 0x00007ffff7b67780 in dtls1_clear_queues (s=0x6cbcd0) at ssl/d1_lib.c:110
#2 dtls1_free (s=0x6cbcd0) at ssl/d1_lib.c:145
#3 0x00007ffff7b85107 in SSL_free (s=0x6cbcd0) at ssl/ssl_lib.c:1215
#4 0x00007ffff7b86420 in SSL_new (ctx=ctx@entry=0x6c93f0)
at ssl/ssl_lib.c:850
#5 0x0000000000442f7e in s_client_main (argc=<optimized out>,
argv=<optimized out>) at apps/s_client.c:1980
#6 0x00000000004326dd in do_cmd (prog=prog@entry=0x6bbbe0,
argc=argc@entry=5, argv=argv@entry=0x7fffffffde70) at apps/openssl.c:570
#7 0x000000000041e738 in main (argc=5, argv=0x7fffffffde70)
at apps/openssl.c:189
(gdb)