Skip to content

Envoy ignores H/2 max concurrent streams advertised by peers #13933

@kryzthov-stripe

Description

@kryzthov-stripe

Title: Envoy ignores H/2 max concurrent streams advertised by peers

Description:
Envoy does not seem to correctly calculate the proper max concurrent streams to use.
Envoy seems to always use the max concurrent streams from its own configuration, and disregards the max concurrent streams advertised by a backend through the H/2 settings frames.
This can lead to situations where Envoy does not create new H/2 connections, and submits requests through an existing nghttp2 session that is currently at or above stream capacity.
nghttp2 handles this "gracefully" by queuing the requests internally until stream capacity opens again.
Envoy will count such requests as active (ie. will count against upstream_rq_active), despite the requests being queued in the nghttp2 session and not being actually sent to the upstream.

Repro steps:
This can be reproduced by standing up an H/2 backend advertising low max concurrent streams settings:
https://gist.github.com/kryzthov-stripe/8d8b5f5f1bd7cb99a88baba08f1c5058
And fronting this backend with Envoy with a high max concurrent streams settings (the default here is 2 billions):
https://gist.github.com/kryzthov-stripe/95e2309344113b7a282ffc7dbfc80d86
To make the problem more obvious, restrict Envoy's concurrent to 1:

envoy --config-path envoy.yaml --concurrency 1

Then sending concurrent requests with something like hey (go get github.com/rakyll/hey):

hey -c 200 -t 0 http://localhost:10000

In the example above, Envoy will only establish a single H/2 connection to the backend (single worker thread), and direct all concurrent requests to the connection. The backend restricts the number of concurrent streams to 5, which nghttp2 session will respect by queuing the excess streams internally. The backend logs confirm that only 5 requests are being processed at a time.

I would expect Envoy to recognize that the backend can only process 5 streams concurrently, and reconfigure the client accordingly. This should lead Envoy to create new H/2 connections as needed and permitted by circuit breakers. In the example above, I would expect Envoy to create ~40 connections.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions