-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Description
HTTP stress tests are failing on the CI with various Failure Types on both Windows and Linux, although the latter seems to happen less frequently.
I keep trying, but so far I was unable to reproduce the failures locally. Based on CI logs, there are several different Failure Types:
1. Windows + HTTP 1.1 connection timeout in HttpConnectionPool.ConnectToTcpHostAsync:
SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type1-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1045441&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=ac0b0b0f-051f-52f7-8fb3-a7e384b0dde9&l=1244
This is the most common failure, happens on almost every build!
2. Windows / Linux + HTTP 2.0 IOException in Http2Connection.ReadFrameAsync:
IOException, The response ended prematurely while waiting for the next frame from the server
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type2-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=702
Note: looks similar to #1586.
3. Windows + HTTP 2.0 SocketException (10054) in Http2Connection.ProcessIncomingFramesAsync:
SocketException (10054): An existing connection was forcibly closed by the remote host. thrown from SslStream.ReadAsync
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type3-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1390
4. Windows/Linux + HTTP 2.0 timeout in Http2Connection.SendStreamDataAsync:
TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type4-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1540
5. Windows/Linux + HTTP 2.0 timeout in Http2Stream.WaitFor100ContinueAsync:
TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type5-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=757