Description

After our project was updated to .NET 5 we found serious regression when trying to send certificate authorized HTTP request using HttpClient to our partner's API and were forced to roll back to .NET Core 3.1 where it works normally.

The problem is a little tricky to reproduce. It is reproducible only when a client is running on a Linux environment (we are using an official docker image based on Debian Buster). The problem is not reproducible on Windows 10.

What it looked like:
After an updated version of our service was released we noticed HTTP requests with certificate authorization started to fail (status code 401) after several successful attempts and then never succeeded again until the service was redeployed. Then we started to debug the application on Windows, but it worked just fine.

To reproduce the problem on Linux we created a simple console application. The code creates HttpClient with certification authorization handler in a loop and tries to make a request to the partner's API. The only first request was successful, the others failed with 401 error.
If we use the same HttpClient for every request then all works fine.
If we create two identical http handlers and corresponding HttpClients, then we try to send requests one by one, then requests made by the first client will succeed while requests made by the second one will fail (the order does not matter). If we will clean SSL sessions cache using reflection before making requests by the other client, it will also work fine. You can see that version of the code in the repository below.
When using .NET Core 3.1 all the requests are made successfully.

In order to investigate the problem by ourselves, we tried to debug .NET internal libraries' code, but we have not found any obvious errors. The only clue was SSL cache. Clearing cache between requests solves the problem, but we have no idea what is wrong with it.

We obviously can't share the production certificate with the private key and provide access to the partner's API, so I've created an alternative server and certificate. The server is written in Go for simplicity and the certificate is issued by Let's Encrypt. We weren't able to reproduce the problem using a self-signed certificate issued by self-signed CA, so we suspect that the problem is somewhere in certificate chain handling.

You can reproduce the problem using the client, server, and certificate provided in the repository below following the instructions:
https://github.com/SukharevAndrey/CertificateAuthBugReproduce

Configuration

• Which version of .NET is the code running on?
  .NET 5.0.2
• What OS and version, and what distro if applicable?
  Debian Buster
• What is the architecture (x64, x86, ARM, ARM64)?
  x64

Regression?

Yes, the problem is not observable using .NET Core 3.1.

Other information

If you use provided client and server, you will see that the first two requests are successful, then the following two will throw an exception, and the last one will be successful again:

System.AggregateException: One or more errors occurred. (An error occurred while sending the request.)
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The decryption operation failed, see inner exception.
 ---> Interop+OpenSsl+SslException: Decrypt failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.Decrypt(SafeSslHandle context, Byte[] outBuffer, Int32 offset, Int32 count, SslErrorCode& errorCode)
   at System.Net.Security.SslStreamPal.EncryptDecryptHelper(SafeDeleteContext securityContext, ReadOnlyMemory`1 input, Int32 offset, Int32 size, Boolean encrypt, Byte[]& output, Int32& resultSize)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)

At the same time go server will print this error to standard output:

http: TLS handshake error from 127.0.0.1:56082: tls: failed to verify client's certificate: x509: certificate signed by unknown authority