What is the issue?

Some of our production services are dropping requests with:

inbound:server{port=<port>}: hyper::proto::h2: Connection header illegal in HTTP/2: connection

and on the outbound / source pods we see 504s along with:

linkerd_app_outbound::http::proxy_connection_close: Closing application connection for remote proxy error=connect timed out after 100ms

As I understand it, this means the proxy is receiving a request with the a connection header set, which is invalid in H2. The only real reference I can see to this is here: #1407 which suggests the proxy is supposed to drop headers that are invalid rather than failing the request.

As it stands, we are dropping production requests between two services, I know the source of the traffic thanks to Linkerd logging out the client IP, but it doesn't log the failed request so I don't know what is happening between the services to try and fix it. Debug logs and tap do not show it either.

Ideally proxy would not fail these requests, and just strip out the illegal headers. If it has to fail them, more information to help identify the requests would be great, such as logging the request headers / body.

How can it be reproduced?

Will confirm this. I suspect sending any http/1 request with a connection header to a meshed pod will trigger it.

Logs, error output, etc

INFO ThreadId(02) inbound:server{port=8080}:rescue{client.addr=10.133.71.3:59588}: linkerd_app_core::errors::respond: Request failed error=error trying to connect: connect timed out after 100ms error.sources=[connect timed out after 100ms]

WARN ThreadId(02) inbound:server{port=8080}: hyper::proto::h2: Connection header illegal in HTTP/2: connection

output of linkerd check -o short

Status check results are √

Environment

Kubernetes - v1.22.15-gke.100
Linkerd - 2.12.2

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None