HTTP2: REFUSED_STREAM not placed in graceful drain state? #13405
Description
Issue found during HTTP2 stress testing. One HttpClient is sending multiple requests in parallel, then canceling them via RST_STREAM sent to the server. For an unknown reason (#13406) the server and client disagree on the active stream count, and the sever eventually sends RST_STREAM (REFUSED_STREAM) and GOAWAY (STREAM_CLOSED).
Stream closed with REFUSED_STREAM should go into a graceful drain state and additional content sent to it from the client is ignored.
Look back in the trace to frame 281508. A bunch of server frames were bundled together, including a RST_STREAM (REFUSED_STREAM) and GOAWAY (STREAM_CLOSED).
It looks like stream 206249 was a POST immediately followed by data. That stream gets refused for some reason we’ll need to investigate, there should be something in the server logs.
Because the stream was refused, it was never added to the stream collection and placed into a graceful drain state. When the data frame is parsed it’s treated as a protocol error (frame sent to a closed stream) and the server initiates a GOAWAY.
Wireshark log: https://www.dropbox.com/s/5ndspth4t0qjyj6/clientreset-error.zip?dl=0
Exception on the client:
Unhandled exception. System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.IO.IOException: The response ended prematurely, with at least 9 additional bytes expected.
at System.Net.Http.Http2Connection.ReadAtLeastAsync(Stream stream, Memory`1 buffer, Int32 minReadBytes)
at System.Net.Http.Http2Connection.EnsureIncomingBytesAsync(Int32 minReadBytes)
at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Grpc.Net.Client.Internal.GrpcCall`2.SendAsync(HttpClient client, HttpRequestMessage message)
at Grpc.Net.Client.Internal.GrpcCall`2.GetResponseHeadersAsync()
Repo is currently not public. Contact @JamesNK or @Tratcher for the repo source code and instruction.