Skip to content

Various HTTP Stress test failures on the CI #49999

@antonfirsov

Description

@antonfirsov

HTTP stress tests are failing on the CI with various Failure Types on both Windows and Linux, although the latter seems to happen less frequently.

I keep trying, but so far I was unable to reproduce the failures locally. Based on CI logs, there are several different Failure Types:

1. Windows + HTTP 1.1 connection timeout in HttpConnectionPool.ConnectToTcpHostAsync:

SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type1-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1045441&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=ac0b0b0f-051f-52f7-8fb3-a7e384b0dde9&l=1244

This is the most common failure, happens on almost every build!

2. Windows / Linux + HTTP 2.0 IOException in Http2Connection.ReadFrameAsync:

IOException, The response ended prematurely while waiting for the next frame from the server
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type2-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=702

Note: looks similar to #1586.

3. Windows + HTTP 2.0 SocketException (10054) in Http2Connection.ProcessIncomingFramesAsync:

SocketException (10054): An existing connection was forcibly closed by the remote host. thrown from SslStream.ReadAsync
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type3-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1390

4. Windows/Linux + HTTP 2.0 timeout in Http2Connection.SendStreamDataAsync:

TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type4-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1027045&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=abae1f68-3c73-5bff-491f-f2b908580ce6&l=1540

5. Windows/Linux + HTTP 2.0 timeout in Http2Stream.WaitFor100ContinueAsync:

TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing
Stack trace:
https://gist.github.com/antonfirsov/dc7af2d9cb4213c835a41f59f09a0775#file-type5-txt
https://dev.azure.com/dnceng/public/_build/results?buildId=1043595&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=757

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions