Title: TCP connection is closed when HTTP upstream returns a 503

Description:

When a HTTP upstream returns a 503 and retries are enabled, Envoy closes the TCP connection to the upstream (with a FIN packet). This is really confusing because it doesn't happen when retries are disabled, which makes me think that it may be a bug in Envoy.

This is similar to #5022 but it's different enough (a FIN, not a RST, and the behavior depends on the HTTP headers sent) that I decided to open a separate issue.

Repro steps:

To start Envoy & the upstream server:

git clone https://gist.github.com/julia-stripe/41fdc36f96b33c1e0e1708376fb7540d
cd 41fdc36f96b33c1e0e1708376fb7540d
ruby ./server.rb # start a server that always returns 503s
envoy -c ./envoy-config.yaml  --service-cluster test --service-node test

To cause the connection to be closed after each request:

curl localhost:8190 -i -H "x-envoy-retry-on: 5xx" -H "x-envoy-max-retries: 1"

To make a similar request that *doesn't result in the connection being closed:

curl localhost:8190 -i

tcpdump

• tcpdump-503s.pcap. Note that every connection is closed almost immediately by Envoy after the response is received.
• tcpdump-503s-no-retry.pcap. Note that in this packet capture almost all of the requests happen over the same TCP connection.

The only difference between these two packet captures is retry HTTP headers -- they're running with exactly the same Envoy configuration & server.

Admin and Stats Output:

In this gist:

https://gist.github.com/julia-stripe/ced04325aa7ce299a2304f7374b236c0

Config:

https://gist.githubusercontent.com/julia-stripe/41fdc36f96b33c1e0e1708376fb7540d/raw/5b574fe26d9f7cd29139f53bcdf711870c4af5e7/envoy-config.yaml