Envoy's port is not listening in HostNetwork #2813
Description
Description:
Envoy's port is not listening if gw pod is set to use HostNetwork, and also there are some error log in envoy pod.
Repro steps:
Use EnvoyProxy CR to declare gw pod using HostNetwork, like this:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg2
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: eg2
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: eg2
namespace: envoy-gateway-system
spec:
shutdown:
drainTimeout: "1s"
minDrainDuration: "1s"
provider:
type: Kubernetes
kubernetes:
envoyService:
type: ClusterIP
envoyDeployment:
replicas: 1
container:
resources: {}
pod:
hostNetwork: true
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: test
namespace: envoy-gateway-system
spec:
gatewayClassName: eg2
listeners:
- name: test
protocol: HTTPS
port: 8899
tls:
certificateRefs:
- kind: Secret
group: ""
name: test-crt-secret
allowedRoutes:
namespaces:
from: AllAnd the gw pod yaml:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/path: /stats/prometheus
prometheus.io/port: "19001"
prometheus.io/scrape: "true"
creationTimestamp: "2024-03-07T03:12:19Z"
generateName: envoy-envoy-gateway-system-test-c5654c8e-5788f75495-
labels:
app.kubernetes.io/component: proxy
app.kubernetes.io/managed-by: envoy-gateway
app.kubernetes.io/name: envoy
gateway.envoyproxy.io/owning-gateway-name: test
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway-system
pod-template-hash: 5788f75495
name: envoy-envoy-gateway-system-test-c5654c8e-5788f75495-qvjpm
namespace: envoy-gateway-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: envoy-envoy-gateway-system-test-c5654c8e-5788f75495
uid: 962cac76-47c1-4a30-9320-427225e593b5
resourceVersion: "26347853154"
uid: c636dda3-ff23-4026-893d-f0094f8a479e
spec:
automountServiceAccountToken: false
containers:
- args:
- --service-cluster envoy-gateway-system/test
- --service-node $(ENVOY_POD_NAME)
- |
--config-yaml admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 19000
layered_runtime:
layers:
- name: global_config
static_layer:
envoy.restart_features.use_eds_cache_for_ads: true
re2.max_program_size.error_level: 4294967295
re2.max_program_size.warn_level: 1000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
listeners:
- name: envoy-gateway-proxy-ready-0.0.0.0-19001
address:
socket_address:
address: 0.0.0.0
port_value: 19001
protocol: TCP
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: eg-ready-http
route_config:
name: local_route
virtual_hosts:
- name: prometheus_stats
domains:
- "*"
routes:
- match:
prefix: /stats/prometheus
route:
cluster: prometheus_stats
http_filters:
- name: envoy.filters.http.health_check
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
pass_through_mode: false
headers:
- name: ":path"
string_match:
exact: /ready
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: prometheus_stats
connect_timeout: 0.250s
type: STATIC
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: prometheus_stats
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 19000
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- load_balancing_weight: 1
lb_endpoints:
- load_balancing_weight: 1
endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
explicit_http_config:
http2_protocol_options:
connection_keepalive:
interval: 30s
timeout: 5s
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
- --log-level warn
- --cpuset-threads
command:
- envoy
env:
- name: ENVOY_GATEWAY_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: ENVOY_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: envoyproxy/envoy:distroless-v1.29-latest
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
httpGet:
path: /shutdown/ready
port: 19002
scheme: HTTP
name: envoy
ports:
- containerPort: 8899
hostPort: 8899
name: test-9f86d081
protocol: TCP
- containerPort: 19001
hostPort: 19001
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /ready
port: 19001
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /certs
name: certs
readOnly: true
- mountPath: /sds
name: sds
- args:
- envoy
- shutdown-manager
- --ready-timeout=11s
command:
- envoy-gateway
env:
- name: ENVOY_GATEWAY_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: ENVOY_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: envoyproxy/gateway-dev:7d48480
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- envoy-gateway
- envoy
- shutdown
- --drain-timeout=1s
- --min-drain-duration=1s
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 19002
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: shutdown-manager
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 19002
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 10m
memory: 32Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostNetwork: true
nodeName: 10.10.11.17
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: envoy-envoy-gateway-system-test-c5654c8e
serviceAccountName: envoy-envoy-gateway-system-test-c5654c8e
terminationGracePeriodSeconds: 301
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: certs
secret:
defaultMode: 420
secretName: envoy
- configMap:
defaultMode: 420
items:
- key: xds-trusted-ca.json
path: xds-trusted-ca.json
- key: xds-certificate.json
path: xds-certificate.json
name: envoy-envoy-gateway-system-test-c5654c8e
optional: false
name: sds
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-03-07T03:12:19Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-03-07T03:13:00Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-03-07T03:13:00Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-03-07T03:12:19Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://ed7660459ee578a8ae23754f1e040b344e51acd2e076c80ba173193451e0811e
image: docker.io/envoyproxy/envoy:distroless-v1.29-latest
imageID: docker.io/envoyproxy/envoy@sha256:76dcbc0509d51001da1c94937b4498bd30ae6eb074d98863336c522311b19fa4
lastState: {}
name: envoy
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-03-07T03:12:20Z"
- containerID: containerd://d5f018ef18676d84fe526fe1a60498a579cd47beded39e3de1645eb55fbd4396
image: docker.io/envoyproxy/gateway-dev:7d48480
imageID: docker.io/envoyproxy/gateway-dev@sha256:4b217e42eceab975166ab9676f7ad524fa271abfa1a97fd30b01c87b77d8cbdb
lastState: {}
name: shutdown-manager
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-03-07T03:12:20Z"
hostIP: 10.10.11.17
phase: Running
podIP: 10.10.11.17
podIPs:
- ip: 10.10.11.17
qosClass: Burstable
startTime: "2024-03-07T03:12:19Z"The envoy should listen 8899 as it is declared in Gateway CR (envoy listens the same port if Gateway's port > 1024), but envoy did not.
Environment:
envoygateway version: v1.0.0-rc.1
Tested both on single node k3s and the cloud hosted Tencent Kubenretes Engine, got the same result.
Logs:
Logs in envoy container:
[2024-03-07 03:12:20.518][1][warning][main] [source/server/server.cc:910] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.res
ource_monitors.downstream_connections` resource monitor.
[2024-03-07 03:12:35.701][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.co
nfig.cluster.v3.Cluster
[2024-03-07 03:12:50.700][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.co
nfig.listener.v3.Listener
[2024-03-07 03:12:53.005][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 32s ago
: 14, no healthy upstream
[2024-03-07 03:12:58.650][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 37s ago
: 14, no healthy upstream
[2024-03-07 03:13:11.834][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 51s ago
: 14, no healthy upstream
[2024-03-07 03:13:21.367][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 60s ago
: 14, no healthy upstream
[2024-03-07 03:13:35.482][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 74s ago
: 14, no healthy upstream
[2024-03-07 03:13:50.977][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 90s ago
: 14, no healthy upstream
[2024-03-07 03:14:06.127][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 105s ag
o: 14, no healthy upstream
[2024-03-07 03:14:16.377][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 115s ag
o: 14, no healthy upstream
[2024-03-07 03:14:27.038][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 126s ag
o: 14, no healthy upstream
[2024-03-07 03:14:38.318][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 137s ag
o: 14, no healthy upstream
[2024-03-07 03:14:39.265][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 138s ag
o: 14, no healthy upstream
[2024-03-07 03:14:58.874][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 158s ag
o: 14, no healthy upstream
[2024-03-07 03:15:09.898][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 169s ag
o: 14, no healthy upstream
[2024-03-07 03:15:33.649][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 192s ag
o: 14, no healthy upstream
[2024-03-07 03:15:48.446][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 207s ag
o: 14, no healthy upstream
[2024-03-07 03:16:00.839][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 220s ag
o: 14, no healthy upstream
[2024-03-07 03:16:07.694][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 226s ag
o: 14, no healthy upstream
[2024-03-07 03:16:30.617][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 249s ag
o: 14, no healthy upstream
[2024-03-07 03:16:55.820][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 275s ag
o: 14, no healthy upstream
[2024-03-07 03:17:04.787][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 284s ag
o: 14, no healthy upstream
[2024-03-07 03:17:07.830][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 287s ag
o: 14, no healthy upstream
[2024-03-07 03:17:31.054][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 310s ag
o: 14, no healthy upstream
[2024-03-07 03:17:31.257][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 310s ag
o: 14, no healthy upstream
[2024-03-07 03:17:59.267][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 338s ag
o: 14, no healthy upstream
[2024-03-07 03:18:28.889][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 368s ag
o: 14, no healthy upstream
[2024-03-07 03:18:41.524][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 380s ag
o: 14, no healthy upstream
[2024-03-07 03:18:42.768][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 382s ag
o: 14, no healthy upstream
[2024-03-07 03:19:05.548][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 404s ag
o: 14, no healthy upstream
[2024-03-07 03:19:26.006][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 425s ago: 14, no healthy upstream
[2024-03-07 03:19:34.582][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 433s ago: 14, no healthy upstream
[2024-03-07 03:20:02.878][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 462s ago: 14, no healthy upstream
[2024-03-07 03:20:17.984][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 477s ago: 14, no healthy upstream
[2024-03-07 03:20:23.831][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 483s ago: 14, no healthy upstream
[2024-03-07 03:20:27.603][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 486s ago: 14, no healthy upstream
[2024-03-07 03:20:31.735][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 491s ago: 14, no healthy upstream
[2024-03-07 03:20:48.332][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 507s ago: 14, no healthy upstream
[2024-03-07 03:20:57.674][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 516s ago: 14, no healthy upstream
[2024-03-07 03:21:12.390][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 531s ago: 14, no healthy upstream
[2024-03-07 03:21:42.229][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 561s ago: 14, no healthy upstream
[2024-03-07 03:21:58.696][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 577s ago: 14, no healthy upstream
[2024-03-07 03:22:24.603][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 603s ago: 14, no healthy upstream
[2024-03-07 03:22:48.130][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 627s ago: 14, no healthy upstream
[2024-03-07 03:23:17.094][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 656s ago: 14, no healthy upstream
[2024-03-07 03:23:39.588][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 678s ago: 14, no healthy upstream
[2024-03-07 03:23:52.943][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 692s ago: 14, no healthy upstream
[2024-03-07 03:23:56.546][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 695s ago: 14, no healthy upstream
[2024-03-07 03:24:03.108][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 702s ago: 14, no healthy upstream
[2024-03-07 03:24:12.983][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 712s ago: 14, no healthy upstream
[2024-03-07 03:24:29.579][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 728s ago: 14, no healthy upstream
[2024-03-07 03:24:34.658][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 733s ago: 14, no healthy upstream
[2024-03-07 03:24:37.604][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 736s ago: 14, no healthy upstream
[2024-03-07 03:24:54.716][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 754s ago: 14, no healthy upstream
[2024-03-07 03:25:18.185][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 777s ago: 14, no healthy upstream
[2024-03-07 03:25:48.098][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 807s ago: 14, no healthy upstream
[2024-03-07 03:25:50.520][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 809s ago: 14, no healthy upstream
[2024-03-07 03:26:01.690][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 820s ago: 14, no healthy upstream
[2024-03-07 03:26:30.518][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 849s ago: 14, no healthy upstream
[2024-03-07 03:26:52.494][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 871s ago: 14, no healthy upstream
[2024-03-07 03:27:00.670][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 879s ago: 14, no healthy upstream
[2024-03-07 03:27:26.575][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 905s ago: 14, no healthy upstream
[2024-03-07 03:27:33.402][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 912s ago: 14, no healthy upstream
[2024-03-07 03:27:44.040][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 923s ago: 14, no healthy upstream
[2024-03-07 03:27:59.109][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 938s ago: 14, no healthy upstream
[2024-03-07 03:28:02.022][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to xds_cluster closed since 941s ago: 14, no healthy upstream