HTTPRoute w/ two parent gateways - Both get configured, only one is shown in status, and which switches #4264
Description
Description:
I have 3 different gateways up and running, only 2 of which are used in this issue.
Only one gateway is shown in status, while both are configured. This results in things like external-dns only setting up DNS records for one gateway address.
Repro steps:
Overview:
─❯ kubectl get envoyproxy -A
NAMESPACE NAME AGE
envoy-gateway cluster 2d22h
envoy-gateway external 2d22h
envoy-gateway internal 2d22h
─❯ kubectl get gatewayclass -A
NAME CONTROLLER ACCEPTED AGE
cluster gateway.envoyproxy.io/gatewayclass-controller True 2d22h
external gateway.envoyproxy.io/gatewayclass-controller True 2d22h
internal gateway.envoyproxy.io/gatewayclass-controller True 13d
─❯ kubectl get gateway -A
NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE
envoy-gateway cluster cluster 10.182.165.187 True 2d22h
envoy-gateway external external 10.182.83.109 True 2d22h
envoy-gateway internal internal 10.180.0.60 True 13d
─❯ kubectl get httproute -A
NAMESPACE NAME HOSTNAMES AGE
dev-ops webapi ["webapi.dev-ops.c.example.org","webapi.dev-ops.i.example.org"] 2d22h # This is attached to internal and cluster
echo-server echo-server ["echo-server-gw.dev.example.org"] 14d # This is attached to externalapiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: internal
namespace: envoy-gateway
spec:
gatewayClassName: internal
listeners:
- allowedRoutes:
namespaces:
from: All
hostname: '*.i.example.org'
name: http
port: 80
protocol: HTTP
status:
addresses:
- type: IPAddress
value: 10.180.0.60
conditions:
- lastTransitionTime: "2024-09-14T11:07:57Z"
message: The Gateway has been scheduled by Envoy Gateway
observedGeneration: 8
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:09:29Z"
message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
observedGeneration: 8
reason: Programmed
status: "True"
type: Programmed
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Sending translated listener configuration to the data plane
observedGeneration: 8
reason: Programmed
status: "True"
type: Programmed
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener has been successfully translated
observedGeneration: 8
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener references have been resolved
observedGeneration: 8
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: http
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRouteapiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: cluster
namespace: envoy-gateway
spec:
gatewayClassName: cluster
listeners:
- allowedRoutes:
namespaces:
from: All
hostname: '*.c.example.org'
name: http
port: 80
protocol: HTTP
status:
addresses:
- type: IPAddress
value: 10.182.165.187
conditions:
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: The Gateway has been scheduled by Envoy Gateway
observedGeneration: 3
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:08:37Z"
message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
observedGeneration: 3
reason: Programmed
status: "True"
type: Programmed
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Sending translated listener configuration to the data plane
observedGeneration: 3
reason: Programmed
status: "True"
type: Programmed
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener has been successfully translated
observedGeneration: 3
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-14T11:07:56Z"
message: Listener references have been resolved
observedGeneration: 3
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: http
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRouteSame HTTPRoute switches its status between gateways
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: webapi
namespace: dev-ops
spec:
hostnames:
- webapi.dev-ops.c.example.org
- webapi.dev-ops.i.example.org
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: cluster
namespace: envoy-gateway
- group: gateway.networking.k8s.io
kind: Gateway
name: internal
namespace: envoy-gateway
rules:
- backendRefs:
- group: ""
kind: Service
name: webapi-auth
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /auth
- backendRefs:
- group: ""
kind: Service
name: webapi-browse
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /browse
- backendRefs:
- group: ""
kind: Service
name: webapi-dev
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /dev
- backendRefs:
- group: ""
kind: Service
name: webapi-discover
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /discover
- backendRefs:
- group: ""
kind: Service
name: webapi-images
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /images
- backendRefs:
- group: ""
kind: Service
name: webapi-marketing
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /marketing
- backendRefs:
- group: ""
kind: Service
name: webapi-member
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /member
- backendRefs:
- group: ""
kind: Service
name: webapi-myexample
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /myexample
- backendRefs:
- group: ""
kind: Service
name: webapi-selling
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /selling
- backendRefs:
- group: ""
kind: Service
name: webapi-shopping
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /shopping
status:
parents:
- conditions:
- lastTransitionTime: "2024-09-17T08:00:53Z"
message: Route is accepted
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-17T08:00:53Z"
message: Resolved all the Object references for the Route
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: internal
namespace: envoy-gatewayapiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: webapi
namespace: dev-ops
spec:
hostnames:
- webapi.dev-ops.c.example.org
- webapi.dev-ops.i.example.org
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: cluster
namespace: envoy-gateway
- group: gateway.networking.k8s.io
kind: Gateway
name: internal
namespace: envoy-gateway
rules:
- backendRefs:
- group: ""
kind: Service
name: webapi-auth
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /auth
- backendRefs:
- group: ""
kind: Service
name: webapi-browse
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /browse
- backendRefs:
- group: ""
kind: Service
name: webapi-dev
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /dev
- backendRefs:
- group: ""
kind: Service
name: webapi-discover
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /discover
- backendRefs:
- group: ""
kind: Service
name: webapi-images
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /images
- backendRefs:
- group: ""
kind: Service
name: webapi-marketing
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /marketing
- backendRefs:
- group: ""
kind: Service
name: webapi-member
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /member
- backendRefs:
- group: ""
kind: Service
name: webapi-myexample
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /myexample
- backendRefs:
- group: ""
kind: Service
name: webapi-selling
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /selling
- backendRefs:
- group: ""
kind: Service
name: webapi-shopping
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
replacePrefixMatch: ""
type: ReplacePrefixMatch
matches:
- path:
type: PathPrefix
value: /shopping
status:
parents:
- conditions:
- lastTransitionTime: "2024-09-17T08:31:21Z"
message: Route is accepted
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-09-17T08:31:21Z"
message: Resolved all the Object references for the Route
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: cluster
namespace: envoy-gateway(Part) Config dump to show both gateways are actually configured:
internal:
{
"@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
"dynamicRouteConfigs": [
{
"lastUpdated": "2024-09-14T11:09:04.604Z",
"routeConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"ignorePortInHostMatching": true,
"name": "envoy-gateway/internal/http",
"virtualHosts": [
{
"domains": [
"webapi.dev-ops.i.example.org"
],
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "Gateway",
"name": "internal",
"namespace": "envoy-gateway",
"sectionName": "http"
}
]
}
}
},
"name": "envoy-gateway/internal/http/webapi_dev-ops_i_example_org",
"routes": [
{
"match": {
"pathSeparatedPrefix": "/marketing"
},
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "HTTPRoute",
"name": "webapi",
"namespace": "dev-ops"
}
]
}
}
},
"name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_i_example_org",
"route": {
"cluster": "httproute/dev-ops/webapi/rule/5",
"regexRewrite": {
"pattern": {
"regex": "^/marketing\\/*"
},
"substitution": "/"
},
"upgradeConfigs": [
{
"upgradeType": "websocket"
}
]
}cluster
{
"@type": "type.googleapis.com/envoy.admin.v3.RoutesConfigDump",
"dynamicRouteConfigs": [
{
"lastUpdated": "2024-09-14T11:08:30.023Z",
"routeConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"ignorePortInHostMatching": true,
"name": "envoy-gateway/cluster/http",
"virtualHosts": [
{
"domains": [
"webapi.dev-ops.c.example.org"
],
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "Gateway",
"name": "cluster",
"namespace": "envoy-gateway",
"sectionName": "http"
}
]
}
}
},
"name": "envoy-gateway/cluster/http/webapi_dev-ops_c_example_org",
"routes": [
{
"match": {
"pathSeparatedPrefix": "/marketing"
},
"metadata": {
"filterMetadata": {
"envoy-gateway": {
"resources": [
{
"kind": "HTTPRoute",
"name": "webapi",
"namespace": "dev-ops"
}
]
}
}
},
"name": "httproute/dev-ops/webapi/rule/5/match/0/webapi_dev-ops_c_example.org",
"route": {
"cluster": "httproute/dev-ops/webapi/rule/5",
"regexRewrite": {
"pattern": {
"regex": "^/marketing\\/*"
},
"substitution": "/"
},
"upgradeConfigs": [
{
"upgradeType": "websocket"
}
]
}Environment:
GKE 1.29
Envoy Gateway v1.1.0
Upgraded to envoy gateway v1.1.1 after writing this issue, and it remains an issue