Envoy produces invalid response when TLS http/1.1 cluster returns upgrade: h2 and transfer-encoding: chunked #10566
Description
Envoy produces invalid response when TLS http/1.1 cluster returns upgrade: h2 and transfer-encoding: chunked
When http/1.1 TLS cluster returns following headers in a response:
Connection: upgrade
Upgrade: h2
Transfer-Encoding: chunked
Envoy returns invalid response downstream.
Direct request to the upstream using curl works correctly:
#> curl -k --http1.1 -v https://localhost:8443/upgrade-h2-chunked
...
< HTTP/1.1 200 OK
< Connection: upgrade
< Upgrade: h2
< Date: Sun, 29 Mar 2020 17:18:13 GMT
< Transfer-Encoding: chunked
<
Service response /upgrade-h2-chunked
Through Envoy we receive something like this:
#> curl -H "host: service" -v http://localhost:6002/upgrade-h2-chunked
< HTTP/1.1 200 OK
< upgrade: h2
< date: Sun, 29 Mar 2020 17:18:31 GMT
< x-envoy-upstream-service-time: 0
< server: envoy
< transfer-encoding: chunked
<
25
Service response /upgrade-h2-chunked
0
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
Responses with transfer-encoding: chunked only or upgrade: h2 only work correctly through Envoy.
It seems like Envoy doesn't decode chunked encoding correctly when upgrade: h2 is present. The strange numbers added to an original response are simply not decoded chunked encoding metadata.
If I see correctly, Envoy doesn't support Upgrade to http2:
Currently, Envoy only supports prior knowledge for upstream connections
Even if Envoy doesn't support it, I think an Upgrade header in a response with status other than 426 should be treated as an advertisment and can be safely ignored:
A server MAY send an Upgrade header field in any other response to
advertise that it implements support for upgrading to the listed
protocols, in order of descending preference, when appropriate for a
future request. (rfc7230)
Somebody already reported similar problem on Istio github: istio/istio#17790
In real life case we encountered this problem when upstream returned upgrade: h2,h2c, so let's not forget about that case too.
How to reproduce
I prepared docker-compose environment where this error can be reproduced. More info in README: https://github.com/MarcinFalkowski/envoy-broken-response-reproduce
The problem can be reproduced on latest master version (commit a29664c)
Logs
In Envoy logs we can see that Envoy enters upgrade mode and then wait for upstream for additional data (despite fact, that all data are already sent) until timeout occurs
envoy logs
envoy_1 | [2020-03-29 16:51:21.858][13][trace][http] [source/common/http/http1/codec_impl.cc:636] [C1] message begin
envoy_1 | [2020-03-29 16:51:21.858][13][trace][http] [source/common/http/http1/codec_impl.cc:425] [C1] completed header: key=Connection value=upgrade
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:425] [C1] completed header: key=Upgrade value=h2
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:425] [C1] completed header: key=Date value=Sun, 29 Mar 2020 16:51:21 GMT
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:559] [C1] onHeadersCompleteBase
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:425] [C1] completed header: key=Transfer-Encoding value=chunked
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:589] [C1] codec entering upgrade mode.
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:924] [C1] Client: onHeadersComplete size=4
envoy_1 | [2020-03-29 16:51:21.859][13][debug][router] [source/common/router/router.cc:1142] [C0][S9047571861382008090] upstream headers complete: end_stream=false
envoy_1 | [2020-03-29 16:51:21.859][13][debug][http] [source/common/http/conn_manager_impl.cc:1702] [C0][S9047571861382008090] encoding headers via codec (end_stream=false):
envoy_1 | ':status', '200'
envoy_1 | 'upgrade', 'h2'
envoy_1 | 'date', 'Sun, 29 Mar 2020 16:51:21 GMT'
envoy_1 | 'x-envoy-upstream-service-time', '6'
envoy_1 | 'server', 'envoy'
envoy_1 |
envoy_1 | [2020-03-29 16:51:21.859][13][trace][connection] [source/common/network/connection_impl.cc:429] [C0] writing 146 bytes, end_stream false
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:615] [C1] message complete
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:621] [C1] Pausing parser due to upgrade.
envoy_1 | [2020-03-29 16:51:21.859][13][trace][http] [source/common/http/http1/codec_impl.cc:483] [C1] parsed 118 bytes
...
envoy_1 | [2020-03-29 16:51:36.851][13][debug][router] [source/common/router/router.cc:812] [C0][S9047571861382008090] upstream timeout
envoy_1 | [2020-03-29 16:51:36.851][13][debug][router] [source/common/router/upstream_request.cc:264] [C0][S9047571861382008090] resetting pool request
Config
cluster
clusters:
- name: service
type: STRICT_DNS
connect_timeout: 1s
load_assignment:
cluster_name: service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: service
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls