It turns out that gRPC async client won't open more than SETTINGS_MAX_CONCURRENT_STREAMS concurrent streams to xDS gRPC server (good!), but EDS monitors are long-lived streams, waiting forever, and it doesn't look that gRPC async client opens more than a single HTTP/2 connection to backend (bad!), which means that total number of working EDS endpoints is limited by xDS's settings, and only the first SETTINGS_MAX_CONCURRENT_STREAMS EDS will be able to establish HTTP/2 stream and receive responses.
The solution is for gRPC async client to open another HTTP/2 connection once it reaches xDS's SETTINGS_MAX_CONCURRENT_STREAMS.
Temporary workaround is to increase SETTINGS_MAX_CONCURRENT_STREAMS on xDS server, but that breaks once there are middle proxies involved.
See istio/istio#4593 for background.
cc @htuch @mattklein123 @costinm @ldemailly @lizan
It turns out that gRPC async client won't open more than
SETTINGS_MAX_CONCURRENT_STREAMSconcurrent streams to xDS gRPC server (good!), but EDS monitors are long-lived streams, waiting forever, and it doesn't look that gRPC async client opens more than a single HTTP/2 connection to backend (bad!), which means that total number of working EDS endpoints is limited by xDS's settings, and only the firstSETTINGS_MAX_CONCURRENT_STREAMSEDS will be able to establish HTTP/2 stream and receive responses.The solution is for gRPC async client to open another HTTP/2 connection once it reaches xDS's
SETTINGS_MAX_CONCURRENT_STREAMS.Temporary workaround is to increase
SETTINGS_MAX_CONCURRENT_STREAMSon xDS server, but that breaks once there are middle proxies involved.See istio/istio#4593 for background.
cc @htuch @mattklein123 @costinm @ldemailly @lizan