CPython test suite is currently broken when run with OpenSSL 1.1.1. After some debugging and assistance from @alex and @reaperhulk I found the culprit. The code flow boils down to:

1. establish TLS connection
2. SSL_write(client, "data", 4) optional
3. SSL_read(server, out, 4) optional
4. SSL_shutdown(client)
5. SSL_shutdown(client)

In TLS 1.2, the second SSL_shutdown() call blocks and waits for a TLS shutdown alert from the server side. This is the expected behavior. For two-way shutdown, the calls must be interleaved.

In TLS 1.3 the second shutdown does not block. There is still pending data on the socket. The pending data turns out to be the session ticket from the server. This behavior breaks CPython's test suite and probably other programs, too. It looks like OpenSSL only consumes the session ticket when the client performs a SSL_read() operation. In case the client only writes or does not perform any application data IO, the ticket is stuck on the wire and breaks SSL_shutdown.

Reproducer

Thanks to Alex and Paul, I was able to come up with a simple reproducer in pure C, see https://github.com/tiran/testssl

TLS 1.3 broken case

$ ./testssl
Server version: TLSv1.3
Client version: TLSv1.3
Writing client...
Reading server...
Calling client shutdown once!
Calling client shutdown twice!
result: -1
SSL error: 5
errno: 0
ERR: 0
OpenSSL error

TLS 1.3 working case

$ ./testssl
Server version: TLSv1.3
Client version: TLSv1.3
Writing client...
Reading server...
Writing server...
Reading client...
Calling client shutdown once!
Calling client shutdown twice!
^Cmake: *** [Makefile:10: all] Interrupt

TLS 1.2

$ ./testssl
Server version: TLSv1.2
Client version: TLSv1.2
Writing client...
Reading server...
Calling client shutdown once!
Calling client shutdown twice!
^Cmake: *** [Makefile:10: all] Interrupt