-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Description
Title: Envoy ignores H/2 max concurrent streams advertised by peers
Description:
Envoy does not seem to correctly calculate the proper max concurrent streams to use.
Envoy seems to always use the max concurrent streams from its own configuration, and disregards the max concurrent streams advertised by a backend through the H/2 settings frames.
This can lead to situations where Envoy does not create new H/2 connections, and submits requests through an existing nghttp2 session that is currently at or above stream capacity.
nghttp2 handles this "gracefully" by queuing the requests internally until stream capacity opens again.
Envoy will count such requests as active (ie. will count against upstream_rq_active), despite the requests being queued in the nghttp2 session and not being actually sent to the upstream.
Repro steps:
This can be reproduced by standing up an H/2 backend advertising low max concurrent streams settings:
https://gist.github.com/kryzthov-stripe/8d8b5f5f1bd7cb99a88baba08f1c5058
And fronting this backend with Envoy with a high max concurrent streams settings (the default here is 2 billions):
https://gist.github.com/kryzthov-stripe/95e2309344113b7a282ffc7dbfc80d86
To make the problem more obvious, restrict Envoy's concurrent to 1:
envoy --config-path envoy.yaml --concurrency 1
Then sending concurrent requests with something like hey (go get github.com/rakyll/hey):
hey -c 200 -t 0 http://localhost:10000
In the example above, Envoy will only establish a single H/2 connection to the backend (single worker thread), and direct all concurrent requests to the connection. The backend restricts the number of concurrent streams to 5, which nghttp2 session will respect by queuing the excess streams internally. The backend logs confirm that only 5 requests are being processed at a time.
I would expect Envoy to recognize that the backend can only process 5 streams concurrently, and reconfigure the client accordingly. This should lead Envoy to create new H/2 connections as needed and permitted by circuit breakers. In the example above, I would expect Envoy to create ~40 connections.