libcurl HttpClient incompatibility with HTTP2 protocol #25315
Description
Recently I started running into issues like this one:
OnUnobservedTaskException() System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (An error occurred while sending the request.) ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: Error in the HTTP2 framing layer
at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Discord.Net.Rest.DefaultRestClient.SendInternalAsync(HttpRequestMessage request, CancellationToken cancelToken, Boolean headerOnly)
at Discord.Net.Rest.DefaultRestClient.SendAsync(String method, String endpoint, CancellationToken cancelToken, Boolean headerOnly, String reason)
at Discord.Net.Queue.RestRequest.SendAsync()
at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
at Discord.API.DiscordRestApiClient.SendAsync[TResponse](String method, String endpoint, String bucketId, ClientBucketType clientBucket, RequestOptions options)
at Discord.API.DiscordSocketApiClient.GetGatewayAsync(RequestOptions options)
at Discord.API.DiscordSocketApiClient.ConnectInternalAsync()
at Discord.API.DiscordSocketApiClient.ConnectInternalAsync()
at Discord.API.DiscordSocketApiClient.ConnectAsync()
at Discord.WebSocket.DiscordSocketClient.OnConnectingAsync()
at Discord.WebSocket.DiscordSocketClient.OnConnectingAsync()
at Discord.ConnectionManager.ConnectAsync(CancellationTokenSource reconnectCancelToken)
at Discord.ConnectionManager.<>c__DisplayClass28_0.<<StartAsync>b__0>d.MoveNext()
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: Error in the HTTP2 framing layer
at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Discord.Net.Rest.DefaultRestClient.SendInternalAsync(HttpRequestMessage request, CancellationToken cancelToken, Boolean headerOnly)
at Discord.Net.Rest.DefaultRestClient.SendAsync(String method, String endpoint, CancellationToken cancelToken, Boolean headerOnly, String reason)
at Discord.Net.Queue.RestRequest.SendAsync()
at Discord.Net.Queue.RequestBucket.SendAsync(RestRequest request)
at Discord.Net.Queue.RequestQueue.SendAsync(RestRequest request)
at Discord.API.DiscordRestApiClient.SendInternalAsync(String method, String endpoint, RestRequest request)
at Discord.API.DiscordRestApiClient.SendAsync[TResponse](String method, String endpoint, String bucketId, ClientBucketType clientBucket, RequestOptions options)
at Discord.API.DiscordSocketApiClient.GetGatewayAsync(RequestOptions options)
at Discord.API.DiscordSocketApiClient.ConnectInternalAsync()
at Discord.API.DiscordSocketApiClient.ConnectInternalAsync()
at Discord.API.DiscordSocketApiClient.ConnectAsync()
at Discord.WebSocket.DiscordSocketClient.OnConnectingAsync()
at Discord.WebSocket.DiscordSocketClient.OnConnectingAsync()
at Discord.ConnectionManager.ConnectAsync(CancellationTokenSource reconnectCancelToken)
at Discord.ConnectionManager.<>c__DisplayClass28_0.<<StartAsync>b__0>d.MoveNext()<---I'm almost 100% sure that this issue is the result of some incompatibility with recent libcurl version. I reproduced it with libcurl3 in version 7.58.0 available in Debian testing repo. Temporarily I downgraded myself back to 7.52.1 available in Debian stable and I can't reproduce the issue anymore, at least for now.
The exception itself isn't exactly in my code but in third-party library I'm using, although there is nothing unusual there.
private async Task<RestResponse> SendInternalAsync(HttpRequestMessage request, CancellationToken cancelToken, bool headerOnly)
{
cancelToken = CancellationTokenSource.CreateLinkedTokenSource(_cancelToken, cancelToken).Token;
HttpResponseMessage response = await _client.SendAsync(request, cancelToken).ConfigureAwait(false);
var headers = response.Headers.ToDictionary(x => x.Key, x => x.Value.FirstOrDefault(), StringComparer.OrdinalIgnoreCase);
var stream = !headerOnly ? await response.Content.ReadAsStreamAsync().ConfigureAwait(false) : null;
return new RestResponse(response.StatusCode, headers, stream);
}I'm using one of the recent master builds and I didn't see anything related fixed lately.
.NET Command Line Tools (2.1.300-preview2-008293)
Product Information:
Version: 2.1.300-preview2-008293
Commit SHA-1 hash: 68922e2a51
Runtime Environment:
OS Name: debian
OS Version:
OS Platform: Linux
RID: debian-x64
Base Path: /opt/dotnet/sdk/2.1.300-preview2-008293/
Host (useful for support):
Version: 2.1.0-preview2-26227-01
Commit: 86d1f92013
.NET Core SDKs installed:
2.1.300-preview2-008293 [/opt/dotnet/sdk]
The.NET Core runtimes installed:
Microsoft.AspNetCore.App 2.1.0-preview2-30171 [/opt/dotnet/shared]
Microsoft.NETCore.App 2.1.0-preview2-26227-01 [/opt/dotnet/shared]
Microsoft.AspNetCore.All 2.1.0-preview2-30171 [/opt/dotnet/shared]
I'm now making sure that old libcurl version fixes the issue, since I could've identified the root cause wrong, but for now I can't run into this issue anymore with older libcurl version. I'll make sure to let you know in case it's something else, but if my guess was right then this should be fixed in corefx repo, and stacktrace suggests that even if my guess is wrong then there is some incompatibility in net core internals that should be verified.
Also I have no clue under exactly what condition the exception is happening. It's definitely related to HTTP2 connection, but it seems that not all requests going through HTTP2 throw this exception (?)
Thank you in advance for looking into this.