HTTPRouteMatchingAcrossRoutes sometimes fails #483
Description
Description:
HTTPRouteMatchingAcrossRoutes sometimes fails
--- FAIL: TestGatewayAPIConformance (31.84s)
--- FAIL: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes (1.05s)
--- PASS: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/3_request_to_example.com/example_with_headers_should_go_to_infra-backend-v1 (0.02s)
--- PASS: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/0_request_to_example.com/_should_go_to_infra-backend-v1 (0.02s)
--- PASS: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/1_request_to_example.com/example_should_go_to_infra-backend-v1 (0.02s)
--- PASS: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/5_request_to_example.net/v2_should_go_to_infra-backend-v1 (0.02s)
--- PASS: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/2_request_to_example.net/example_should_go_to_infra-backend-v1 (0.02s)
--- FAIL: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/4_request_to_example.com/v2_should_go_to_infra-backend-v2 (30.00s)
--- FAIL: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/6_request_to_example.com/v2/example_should_go_to_infra-backend-v2 (30.00s)
--- FAIL: TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/7_request_to_example.com/_with_headers_should_go_to_infra-backend-v2 (30.00s)
Repro steps:
seen in #480
Logs:
Here's the HTTPRoutes spec https://github.com/kubernetes-sigs/gateway-api/blob/main/conformance/tests/httproute-matching-across-routes.yaml
Here's the Request/Response
ending Request:
< GET /v2 HTTP/1.1
< Host: example.com
< User-Agent: Go-http-client/1.1
< Accept-Encoding: gzip
<
<
Received Response:
< HTTP/1.1 200 OK
< Content-Length: 473
< Content-Type: application/json
< Date: Mon, 03 Oct 2022 18:56:08 GMT
< Server: envoy
< X-Content-Type-Options: nosniff
< X-Envoy-Upstream-Service-Time: 0
<
< {
< "path": "/v2/example",
< "host": "example.com",
< "method": "GET",
< "proto": "HTTP/1.1",
< "headers": {
< "Accept-Encoding": [
< "gzip"
< ],
< "User-Agent": [
< "Go-http-client/1.1"
< ],
< "X-Envoy-Expected-Rq-Timeout-Ms": [
< "15000"
< ],
< "X-Forwarded-Proto": [
< "http"
< ],
< "X-Request-Id": [
< "1ee97601-099c-4889-9fa1-7e833983b2f6"
< ]
< },
< "namespace": "gateway-conformance-infra",
< "ingress": "",
< "service": "",
< "pod": "infra-backend-v1-7bdb567c9f-94nft"
< }
=== CONT TestGatewayAPIConformance/HTTPRouteMatchingAcrossRoutes/6_request_to_example.com/v2/example_should_go_to_infra-backend-v2
http.go:159: Response expectation failed, not ready yet: expected pod name to start with infra-backend-v2, got infra-backend-v1-7bdb567c9f-94nft
From the IR, we can see that that the first match is a prefix match to / for example.com which goes to infra-backend-v1 and there is also a /v2 prefix match for example.com that goes to infra-backend-v2
gateway-conformance-infra-same-namespace:
HTTP:
- Address: 0.0.0.0
Hostnames:
- '*'
Name: gateway-conformance-infra-same-namespace-http
Port: 10080
Routes:
- AddRequestHeaders: null
BackendWeights:
Invalid: 0
Valid: 0
Destinations:
- Host: 10.99.175.224
Port: 8080
Weight: 1
DirectResponse: null
HeaderMatches:
- Exact: example.com
Name: :authority
Prefix: null
SafeRegex: null
Name: gateway-conformance-infra-matching-part1-rule-0-match-0-example.com
PathMatch:
Exact: null
Name: ""
Prefix: /
SafeRegex: null
QueryParamMatches: null
Redirect: null
RemoveRequestHeaders: null
- AddRequestHeaders: null
BackendWeights:
Invalid: 0
Valid: 0
Destinations:
- Host: 10.99.175.224
Port: 8080
Weight: 1
DirectResponse: null
HeaderMatches:
- Exact: example.com
Name: :authority
Prefix: null
SafeRegex: null
- Exact: one
Name: version
Prefix: null
SafeRegex: null
Name: gateway-conformance-infra-matching-part1-rule-0-match-1-example.com
PathMatch:
Exact: null
Name: ""
Prefix: /
SafeRegex: null
QueryParamMatches: null
Redirect: null
RemoveRequestHeaders: null
- AddRequestHeaders: null
BackendWeights:
Invalid: 0
Valid: 0
Destinations:
- Host: 10.99.175.224
Port: 8080
Weight: 1
DirectResponse: null
HeaderMatches:
- Exact: example.net
Name: :authority
Prefix: null
SafeRegex: null
Name: gateway-conformance-infra-matching-part1-rule-0-match-0-example.net
PathMatch:
Exact: null
Name: ""
Prefix: /
SafeRegex: null
QueryParamMatches: null
Redirect: null
RemoveRequestHeaders: null
- AddRequestHeaders: null
BackendWeights:
Invalid: 0
Valid: 0
Destinations:
- Host: 10.99.175.224
Port: 8080
Weight: 1
DirectResponse: null
HeaderMatches:
- Exact: example.net
Name: :authority
Prefix: null
SafeRegex: null
- Exact: one
Name: version
Prefix: null
SafeRegex: null
Name: gateway-conformance-infra-matching-part1-rule-0-match-1-example.net
PathMatch:
Exact: null
Name: ""
Prefix: /
SafeRegex: null
QueryParamMatches: null
Redirect: null
RemoveRequestHeaders: null
- AddRequestHeaders: null
BackendWeights:
Invalid: 0
Valid: 0
Destinations:
- Host: 10.104.210.124
Port: 8080
Weight: 1
DirectResponse: null
HeaderMatches:
- Exact: example.com
Name: :authority
Prefix: null
SafeRegex: null
Name: gateway-conformance-infra-matching-part2-rule-0-match-0-example.com
PathMatch:
Exact: null
Name: ""
Prefix: /v2
SafeRegex: null
QueryParamMatches: null
Redirect: null
RemoveRequestHeaders: null
- AddRequestHeaders: null
BackendWeights:
Invalid: 0
Valid: 0
Destinations:
- Host: 10.104.210.124
Port: 8080
Weight: 1
DirectResponse: null
HeaderMatches:
- Exact: example.com
Name: :authority
Prefix: null
SafeRegex: null
- Exact: two
Name: version
Prefix: null
SafeRegex: null
Name: gateway-conformance-infra-matching-part2-rule-0-match-1-example.com
PathMatch:
Exact: null
Name: ""
Prefix: /
SafeRegex: null
QueryParamMatches: null
Redirect: null
RemoveRequestHeaders: null
TLS: null
This gets translated to this Envoy route config and based on the Envoy route matching rules https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/route_matching the first match gets picked
[name:"route_gateway-conformance-infra-same-namespace-http" virtual_hosts:{name:"route_gateway-conformance-infra-same-namespace-http" domains:"*" routes:{match:{prefix:"/" headers:{name:":authority" string_match:{exact:"example.com"}}} route:{cluster:"cluster_gateway-conformance-infra-matching-part1-rule-0-match-0-example.com"}} routes:{match:{prefix:"/" headers:{name:":authority" string_match:{exact:"example.com"}} headers:{name:"version" string_match:{exact:"one"}}} route:{cluster:"cluster_gateway-conformance-infra-matching-part1-rule-0-match-1-example.com"}} routes:{match:{prefix:"/" headers:{name:":authority" string_match:{exact:"example.net"}}} route:{cluster:"cluster_gateway-conformance-infra-matching-part1-rule-0-match-0-example.net"}} routes:{match:{prefix:"/" headers:{name:":authority" string_match:{exact:"example.net"}} headers:{name:"version" string_match:{exact:"one"}}} route:{cluster:"cluster_gateway-conformance-infra-matching-part1-rule-0-match-1-example.net"}} routes:{match:{prefix:"/v2" headers:{name:":authority" string_match:{exact:"example.com"}}} route:{cluster:"cluster_gateway-conformance-infra-matching-part2-rule-0-match-0-example.com"}} routes:{match:{prefix:"/" headers:{name:":authority" string_match:{exact:"example.com"}} headers:{name:"version" string_match:{exact:"two"}}} route:{cluster:"cluster_gateway-conformance-infra-matching-part2-rule-0-match-1-example.com"}}}]
api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRouteMatch suggests the below logic when there is a tie
If ties still exist across multiple Routes, matching precedence MUST be determined in order of the following criteria, continuing on ties:
The oldest Route based on creation timestamp.
The Route appearing first in alphabetical order by “{namespace}/{name}”.
If ties still exist within an HTTPRoute, matching precedence MUST be granted to the FIRST matching rule (in list order) with a match meeting the above criteria.
When no rules matching a request have been successfully attached to the parent a request is coming from, a HTTP 404 status code MUST be returned.
Here are the HTTPRoutes
kubectl get httproute -A -o yaml
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
creationTimestamp: "2022-10-03T18:55:38Z"
generation: 1
name: matching-part1
namespace: gateway-conformance-infra
resourceVersion: "7403"
uid: db23e763-61ab-40f4-b157-a6b592b7e72c
spec:
hostnames:
- example.com
- example.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: same-namespace
rules:
- backendRefs:
- group: ""
kind: Service
name: infra-backend-v1
port: 8080
weight: 1
matches:
- path:
type: PathPrefix
value: /
- headers:
- name: version
type: Exact
value: one
path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2022-10-03T18:55:38Z"
message: Route is accepted
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: same-namespace
- apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
creationTimestamp: "2022-10-03T18:55:38Z"
generation: 1
name: matching-part2
namespace: gateway-conformance-infra
resourceVersion: "7408"
uid: fe79d86b-552f-451f-8db0-b805b75d9d82
spec:
hostnames:
- example.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: same-namespace
rules:
- backendRefs:
- group: ""
kind: Service
name: infra-backend-v2
port: 8080
weight: 1
matches:
- path:
type: PathPrefix
value: /v2
- headers:
- name: version
type: Exact
value: two
path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2022-10-03T18:55:38Z"
message: Route is accepted
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: same-namespace
kind: List
metadata:
resourceVersion: ""
based on the tie break rules, the logic in EG is correct and the request should be sent to infra-backend-v1 but conformance tests are expecting the request to be sent to infra-backend-v2.