Envoy not removing HTTP/1.1 TE and Connection header for HTTP/2 backend. #8623
Description
Title: HTTP/1.1 TE and connection headers are not being removed when making calls to HTTP/2 backends.
Description:
If you make a HTTP/1.1 request than includes a TE header with a valid HTTP/1.1 value like TE: gzip or TE: deflate,gzip;q=0.3 along with Connection: TE, close, it will be passed along to the HTTP/2 backend where it causes a 503 since it's invalid for HTTP/2.
Shouldn't Envoy be removing the TE and Connection header when making the backend HTTP/2 request? The RFC mentions
The only exception to this is the TE header field, which MAY be
present in an HTTP/2 request; when it is, it MUST NOT contain any
value other than "trailers".
This means that an intermediary transforming an HTTP/1.x message to
HTTP/2 will need to remove any header fields nominated by the
Connection header field, along with the Connection header field
itself. Such intermediaries SHOULD also remove other connection-
specific header fields, such as Keep-Alive, Proxy-Connection,
Transfer-Encoding, and Upgrade, even if they are not nominated by the
Connection header field.
Repro steps:
You can see this with a frontend and service envoy. A HTTP/1.1 request goes to the frontend and it creates a HTTP/2 connection to the backend with the TE: gzip header.
HTTP/1.1 envoy front *:8080 -> HTTP/2 envoy service *:9080 (static response)
test-envoy-frontend.yaml
$ cat test-envoy-frontend.yaml
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 8080
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: backend_service
http_filters:
- name: envoy.router
typed_config: {}
clusters:
- name: backend_service
connect_timeout: 0.25s
type: static
lb_policy: round_robin
http2_protocol_options: {}
load_assignment:
cluster_name: backend_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 9080
test-envoy-backend.yaml
$ cat test-envoy-backend.yaml
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 9080
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
direct_response:
status: 200
body:
inline_string: "ok\n"
http_filters:
- name: envoy.router
typed_config: {}
Run the envoys (I happened to use this build from Oct 9th. It also happens in the v1.11.2 release):
$ ~/envoy.3582d02f6600057e676629c2afefe5265e669b10 --base-id 1 -c test-envoy-backend.yaml -l trace
$ ~/envoy.3582d02f6600057e676629c2afefe5265e669b10 --base-id 2 -c test-envoy-frontend.yaml -l trace
Actual without TE header:
This works as expected.
$ time curl -vvv 127.0.0.1:8080/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-length: 3
< content-type: text/plain
< location: http://127.0.0.1:8080/
< date: Tue, 15 Oct 2019 20:25:42 GMT
< server: envoy
< x-envoy-upstream-service-time: 42
<
ok
* Connection #0 to host 127.0.0.1 left intact
real 0m0.090s
user 0m0.005s
sys 0m0.012s
Actual with TE trailers:
This works because TE: trailers is the only valid value in HTTP/2 per section 8.1.2.2.
$ time curl -vvv -H "TE: trailers" -H "Connection: TE, close" 127.0.0.1:8080/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.54.0
> Accept: */*
> TE: trailers
> Connection: TE, close
>
< HTTP/1.1 200 OK
< content-length: 3
< content-type: text/plain
< location: http://127.0.0.1:8080/
< date: Tue, 15 Oct 2019 20:26:31 GMT
< server: envoy
< x-envoy-upstream-service-time: 21
<
ok
* Connection #0 to host 127.0.0.1 left intact
real 0m0.054s
user 0m0.006s
sys 0m0.009s
Actual with TE gzip
This does not work because Envoy is passing the TE: gzip header to a HTTP/2 backend and that's not valid.
$ time curl -vvv -H "TE: gzip" -H "Connection: TE, close" 127.0.0.1:8080/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.54.0
> Accept: */*
> TE: gzip
> Connection: TE, close
>
< HTTP/1.1 503 Service Unavailable
< content-length: 95
< content-type: text/plain
< date: Tue, 15 Oct 2019 20:27:11 GMT
< server: envoy
<
* Connection #0 to host 127.0.0.1 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination
real 0m1.028s
user 0m0.005s
sys 0m0.008s
When running in trace on the backend Envoy, it is complaining about gzip:
[2019-10-15 13:27:10.557][2740140][trace][http2] [source/common/http/http2/nghttp2.cc:20] nghttp2: inflatehd: header emission: te: gzip
[2019-10-15 13:27:10.557][2740140][trace][http2] [source/common/http/http2/nghttp2.cc:20] nghttp2: recv: proclen=6
[2019-10-15 13:27:10.557][2740140][trace][http2] [source/common/http/http2/nghttp2.cc:20] nghttp2: recv: HTTP error: type=1, id=5, header te: gzip
[2019-10-15 13:27:10.557][2740140][debug][http2] [source/common/http/http2/codec_impl.cc:643] [C2] invalid frame: Invalid HTTP header field was received on stream 5
[2019-10-15 13:27:10.558][2740140][debug][http] [source/common/http/conn_manager_impl.cc:264] [C2] dispatch error: The user callback function failed
[2019-10-15 13:27:10.558][2740140][debug][http] [source/common/http/conn_manager_impl.cc:1734] [C2][S3055301157300998685] stream reset
[2019-10-15 13:27:10.558][2740140][trace][main] [source/common/event/dispatcher_impl.cc:158] item added to deferred deletion list (size=1)
Workaround 1:
A simple workaround is to have the frontend Envoy strip out the TE header with the request_headers_to_remove: ["TE"] option so it isn't passed along to the HTTP/2 connection.
That seems acceptable since RFC 7230 says:
If the TE field-value is empty or if no TE field is present, the only
acceptable transfer coding is chunked. A message with no transfer
coding is always acceptable.
Workaround 2
I could also disable the HTTP/2 backend connections via commenting out http2_protocol_options: {}.