Skip to content

http: don't close TCP connection when response body is empty and request is retried#5037

Merged
mattklein123 merged 8 commits intoenvoyproxy:masterfrom
julia-stripe:avoid-closing-503s
Nov 15, 2018
Merged

http: don't close TCP connection when response body is empty and request is retried#5037
mattklein123 merged 8 commits intoenvoyproxy:masterfrom
julia-stripe:avoid-closing-503s

Conversation

@julia-stripe
Copy link
Copy Markdown
Contributor

@julia-stripe julia-stripe commented Nov 14, 2018

Description:

This fixes the issue described in #5023: right now we set end_stream=false for many HTTP/1 responses even if the HTTP body is empty (and the stream has ended). Retried requests with end_stream=false close the connection, so this results in the TCP connection being closed in many cases where it could be kept open, which results in unnecessary connection churn.

With this change, when an upstream returns a HTTP/1 response with an empty body and the request is retried, the TCP connection will no longer be closed.

Risk Level:

Low

Testing:

Did manual testing to see that the connection is no longer closed when an upstream returns a 503, and added a unit test to ensure that end_stream is set correctly.

Release note

When a HTTP/1 request is retried due to a 5xx/499 response, we now keep the TCP connection open instead of closing it

Fixes #5023

@mattklein123 mattklein123 self-assigned this Nov 14, 2018
@mattklein123
Copy link
Copy Markdown
Member

Thanks, at a high level this LGTM. Can you check CI?

@julia-stripe
Copy link
Copy Markdown
Contributor Author

The CI issue was that the code was marking the stream as ended for 101 Switching Protocols responses, which is incorrect. I changed it to only mark the stream as ended if the content length is 0 and the status is a 4xx/5xx.

@julia-stripe julia-stripe force-pushed the avoid-closing-503s branch 2 times, most recently from 0441906 to 05c0d9d Compare November 14, 2018 21:53
Copy link
Copy Markdown
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this LGTM. @alyssawilk can you take a quick look?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a test that covers 1xx, but can you add a test for maybe 200 with content_length zero also?

alyssawilk
alyssawilk previously approved these changes Nov 15, 2018
Copy link
Copy Markdown
Contributor

@alyssawilk alyssawilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM, thanks for the fix!

I'd say it's worth a release note though even though it's not technically required. It may not change user-facing behavior but it will have a non-trivial (positive!) effect for folks running H1 upstreams and if anyone is doing whacky non-spec compliant responses it'd be good to give them a heads up.

@mattklein123
Copy link
Copy Markdown
Member

SGTM re: release note. @julia-stripe do you mind adding one?

@julia-stripe
Copy link
Copy Markdown
Contributor Author

added a note!

Copy link
Copy Markdown
Contributor

@alyssawilk alyssawilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did that update get pushed upstream? I'm not seeing a change yet :-)

Signed-off-by: Julia Evans <julia@stripe.com>
Signed-off-by: Julia Evans <julia@stripe.com>
Signed-off-by: Julia Evans <julia@stripe.com>
Signed-off-by: Julia Evans <julia@stripe.com>
Signed-off-by: Julia Evans <julia@stripe.com>
Signed-off-by: Julia Evans <julia@stripe.com>
alyssawilk
alyssawilk previously approved these changes Nov 15, 2018
Signed-off-by: Julia Evans <julia@stripe.com>
@julia-stripe
Copy link
Copy Markdown
Contributor Author

rebased because it said there was a merge conflict

alyssawilk
alyssawilk previously approved these changes Nov 15, 2018
Copy link
Copy Markdown
Contributor

@alyssawilk alyssawilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha - you caught me mid "but you'll have to pull from master" comment :-)

* http: augmented the `sendLocalReply` filter API to accept an optional `GrpcStatus`
value to override the default HTTP to gRPC status mapping.
* load balancer: added a `configuration <envoy_api_msg_Cluster.LeastRequestLbConfig>` option to specify the number of choices made in P2C.
* http: no longer close the TCP connection when a HTTP/1 request is retried due
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: sorry, can you alpha order this

Signed-off-by: Julia Evans <julia@stripe.com>
@mattklein123 mattklein123 merged commit 9aa509b into envoyproxy:master Nov 15, 2018
fredlas pushed a commit to fredlas/envoy that referenced this pull request Mar 5, 2019
…est is retried (envoyproxy#5037)

Signed-off-by: Julia Evans <julia@stripe.com>
Signed-off-by: Fred Douglas <fredlas@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TCP connection is closed when HTTP upstream returns a 503 & retries are enabled

3 participants