grpc-go post 1.65.0 introduces separate xDS client connections to istiod for each named grpc service connected to that were previously handled by a single xDS client. In our usage scenario, we follow the instructions in https://istio.io/latest/blog/2021/proxyless-grpc/#in-your-kubernetes-deployment having the grpc-agent enabled and provide settings in a bootstrap.json file.
Once we upgrade to the grpc-go client that uses multiple clients, all gRPC requests start failing due to the istio-agent side car dropping all xDS connections after the first (e.g. clients that use gRPC service a, b and c, only the first connection gets created successfully (see log below).
We expect the istio-agent to proxy the additional xDS client connections instead of being dropped.
The end result is that we're blocked on upgrading the grpc-go client beyond 1.65.0 where the multiple xDS client connections are introduced.
{"level":"info","time":"2024-10-11T17:57:26.160505Z","scope":"xdsproxy","msg":"Initializing with upstream address \"istiod-1-22-4.istio-system.svc:15012\" and cluster \"Kubernetes\""}
{"level":"info","time":"2024-10-11T17:57:26.427983Z","scope":"cache","msg":"generated new workload certificate","resourceName":"default","latency":267051261,"ttl":86399572022083}
{"level":"info","time":"2024-10-11T17:57:26.428066Z","scope":"cache","msg":"Root cert has changed, start rotating root cert"}
{"level":"info","time":"2024-10-11T17:57:26.428100Z","scope":"cache","msg":"returned workload certificate from cache","ttl":86399571901173}
{"level":"info","time":"2024-10-11T17:57:26.428366Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":86399571634504}
{"level":"info","time":"2024-10-11T17:57:26.428802Z","scope":"cache","msg":"returned workload trust anchor from cache","ttl":86399571198514}
{"level":"info","time":"2024-10-11T17:57:33.536595Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod-1-22-4.istio-system.svc:15012","id":1}
{"level":"warn","time":"2024-10-11T17:57:33.808855Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
{"level":"warn","time":"2024-10-11T17:57:33.809498Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
{"level":"info","time":"2024-10-11T17:57:33.815859Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod-1-22-4.istio-system.svc:15012","id":2}
{"level":"info","time":"2024-10-11T17:57:33.816460Z","scope":"xdsproxy","msg":"connected to upstream XDS server: istiod-1-22-4.istio-system.svc:15012","id":3}
{"level":"warn","time":"2024-10-11T17:57:34.817348Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
{"level":"warn","time":"2024-10-11T17:57:34.818000Z","scope":"xdsproxy","msg":"registered overlapping stream; closing previous"}
$ istioctl version
client version: 1.22.2
istiod version: 1.22.4
data plane version: 1.22.4 (247 proxies)
$ kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.8-gke.1096000
Is this the right place to submit this?
Bug Description
grpc-go post 1.65.0 introduces separate xDS client connections to istiod for each named grpc service connected to that were previously handled by a single xDS client. In our usage scenario, we follow the instructions in https://istio.io/latest/blog/2021/proxyless-grpc/#in-your-kubernetes-deployment having the grpc-agent enabled and provide settings in a bootstrap.json file.
Once we upgrade to the grpc-go client that uses multiple clients, all gRPC requests start failing due to the istio-agent side car dropping all xDS connections after the first (e.g. clients that use gRPC service a, b and c, only the first connection gets created successfully (see log below).
We expect the istio-agent to proxy the additional xDS client connections instead of being dropped.
The end result is that we're blocked on upgrading the grpc-go client beyond 1.65.0 where the multiple xDS client connections are introduced.
Version
Additional Information
Google GKE
Original grpc-go ticket when upgrade broke gRPC calls : grpc/grpc-go#7691