fix: multiple reference grants in same namespace by ardikabs · Pull Request #4008 · envoyproxy/gateway

ardikabs · 2024-08-06T08:49:46Z

What type of PR is this?

The current behavior of Envoy Gateway when working with multiple ReferenceGrants in the same namespace is problematic. When searching for a ReferenceGrant for corresponding resources, such as the HTTPRoute, it only checks the ReferenceGrant.spec.from semantics. However, during translation, the translator validates both the ReferenceGrant.spec.from and ReferenceGrant.spec.to semantics thoroughly.
As a result, the findReferenceGrant method only captures the first ReferenceGrant with the correct From, meaning other ReferenceGrants are not honored. This leads to the corresponding resource that uses two different resource references encountering an error message such as Backend ref to service <namespace>/<service_name> not permitted by any ReferenceGrant during translation.

Code for findReferenceGrant:

gateway/internal/provider/kubernetes/controller.go

Lines 771 to 782 in 3497600

    
           for _, refGrant := range refGrants { 
        
           	if refGrant.Namespace == to.namespace { 
        
           		for _, src := range refGrant.Spec.From { 
        
           			if src.Kind == gwapiv1a2.Kind(from.kind) && string(src.Namespace) == from.namespace { 
        
           				return &refGrant, nil 
        
           			} 
        
           		} 
        
           	} 
        
           } 
        
           // No ReferenceGrant found. 
        
           return nil, nil

Code for translation:

gateway/internal/gatewayapi/validate.go

Lines 832 to 857 in 3497600

    
           var fromAllowed bool 
        
           for _, refGrantFrom := range referenceGrant.Spec.From { 
        
           	if string(refGrantFrom.Namespace) == from.namespace && string(refGrantFrom.Group) == from.group && string(refGrantFrom.Kind) == from.kind { 
        
           		fromAllowed = true 
        
           		break 
        
           	} 
        
           } 
        
           if !fromAllowed { 
        
           	continue 
        
           } 
        
           // Check if the ReferenceGrant has a matching "to". 
        
           var toAllowed bool 
        
           for _, refGrantTo := range referenceGrant.Spec.To { 
        
           	if string(refGrantTo.Group) == to.group && string(refGrantTo.Kind) == to.kind && (refGrantTo.Name == nil || *refGrantTo.Name == "" || string(*refGrantTo.Name) == to.name) { 
        
           		toAllowed = true 
        
           		break 
        
           	} 
        
           } 
        
           if !toAllowed { 
        
           	continue 
        
           } 
        
           // If we got here, both the "from" and the "to" were allowed by this 
        
           // reference grant. 
        
           return true

Proposed Solution

This PR modifies the findReferenceGrant method to also check the To semantics.

Fixes #2149

codecov · 2024-08-06T08:55:35Z

Codecov Report

Attention: Patch coverage is 0% with 17 lines in your changes missing coverage. Please review.

Project coverage is 67.62%. Comparing base (975b1e8) to head (dd4fde8).

Files	Patch %	Lines
internal/provider/kubernetes/controller.go	0.00%	17 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4008      +/-   ##
==========================================
+ Coverage   67.61%   67.62%   +0.01%     
==========================================
  Files         185      185              
  Lines       22578    22591      +13     
==========================================
+ Hits        15265    15277      +12     
- Misses       6217     6221       +4     
+ Partials     1096     1093       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ardikabs · 2024-08-06T09:04:46Z

since there is no existing unit test for the corresponding line, could anyone advise me on whether I should add a unit test elsewhere?

internal/provider/kubernetes/controller.go

shawnh2 · 2024-08-06T09:27:14Z

since there is no existing unit test for the corresponding line, could anyone advise me on whether I should add a unit test elsewhere?

An e2e test would be nice!

ardikabs · 2024-08-06T10:04:48Z

An e2e test would be nice!

@shawnh2 do you mean the integration, like here?

shawnh2 · 2024-08-06T10:08:57Z

An e2e test would be nice!

@shawnh2 do you mean the integration, like here?

no, it's under test/e2e.

ardikabs · 2024-08-07T15:21:55Z

hi @shawnh2,
I've added an e2e test, but it seems that some of the other tests, like TestE2E/Wasm_HTTP_Code_Source, are a bit flaky.

shawnh2 · 2024-08-08T01:04:02Z

they sure are flaky, and we are trying to repair them. 😿

shawnh2

LGTM! Thanks for fixing this

shawnh2 · 2024-08-08T01:06:10Z

/retest

Signed-off-by: Ardika Bagus <me@ardikabs.com>

arkodg

LGTM !
thanks for adding an e2e

* fix: multiple reference grants in same namespace Signed-off-by: Ardika Bagus <me@ardikabs.com> * test: add e2e test Signed-off-by: Ardika Bagus <me@ardikabs.com> * chore: wrong service port Signed-off-by: Ardika Bagus <me@ardikabs.com> --------- Signed-off-by: Ardika Bagus <me@ardikabs.com> (cherry picked from commit b82f4b2) Signed-off-by: Arko Dasgupta <arko@tetrate.io>

* bugfix: fix upstream get unwanted /. (#3990) * bugfix: fix upstream get unwanted /. Signed-off-by: qicz <qiczzhu@gmail.com> * ut for bugfix Signed-off-by: qicz <qiczzhu@gmail.com> --------- Signed-off-by: qicz <qiczzhu@gmail.com> Co-authored-by: Xunzhuo <bitliu@tencent.com> (cherry picked from commit b77f6a4) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * feat: gateway http listener isolation (#4000) Signed-off-by: Kobi Levi <kobilevi503@gmail.com> (cherry picked from commit 97830e9) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * fix: multiple reference grants in same namespace (#4008) * fix: multiple reference grants in same namespace Signed-off-by: Ardika Bagus <me@ardikabs.com> * test: add e2e test Signed-off-by: Ardika Bagus <me@ardikabs.com> * chore: wrong service port Signed-off-by: Ardika Bagus <me@ardikabs.com> --------- Signed-off-by: Ardika Bagus <me@ardikabs.com> (cherry picked from commit b82f4b2) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * reduce readinessProbe failureThreshold and periodSeconds (#4021) * Reduces time for the endpoint to be removed from the endpointSlice from `30s` (3 * 10) to `5s` (1 * 5) * Since kube-proxy and CNIs rely on this info and so do external LBs like GKE https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer Signed-off-by: Arko Dasgupta <arko@tetrate.io> (cherry picked from commit 67575b8) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * fix: add header values as described in the documentation (#4031) Add header values after splitting the provided value string on ',', like described in the documentation. Signed-off-by: Lior Okman <lior.okman@sap.com> (cherry picked from commit eac30d6) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * fix ratelimit statsd not working (#4073) fix ratelimit statd not working Signed-off-by: zirain <zirain2009@gmail.com> (cherry picked from commit 6ab6482) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * fix: active http healthcheck documents a default for expected status, but doesn't use it (#4090) If no expected status was explicitly set, use the default value as described in the documentation. Signed-off-by: Lior Okman <lior.okman@sap.com> (cherry picked from commit 0926b38) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * Fix IsNotFound check for secret and configmap (#4126) fix IsNotFound check for secret and configmap Signed-off-by: TasdidurRahman <tasdid@appscode.com> (cherry picked from commit c20315f) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * fix: assign sugar logger name. (#4144) Signed-off-by: qicz <qiczzhu@gmail.com> Co-authored-by: zirain <zirain2009@gmail.com> (cherry picked from commit b50f5fa) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * use sets and return stable result (#4074) Signed-off-by: zirain <zirain2009@gmail.com> (cherry picked from commit 6066f5a) Signed-off-by: Arko Dasgupta <arko@tetrate.io> * delete internal/gatewayapi/clustersettings.go NA for v1.1 Signed-off-by: Arko Dasgupta <arko@tetrate.io> * bump to go1.22.7 (#4175) * bump to go1.22.6 Signed-off-by: zirain <zirain2009@gmail.com> * bump to 1.22.7 Signed-off-by: zirain <zirain2009@gmail.com> --------- Signed-off-by: zirain <zirain2009@gmail.com> (cherry picked from commit 69bf882) Signed-off-by: Arko Dasgupta <arko@tetrate.io> --------- Signed-off-by: qicz <qiczzhu@gmail.com> Signed-off-by: Arko Dasgupta <arko@tetrate.io> Signed-off-by: Kobi Levi <kobilevi503@gmail.com> Signed-off-by: Ardika Bagus <me@ardikabs.com> Signed-off-by: Lior Okman <lior.okman@sap.com> Signed-off-by: zirain <zirain2009@gmail.com> Signed-off-by: TasdidurRahman <tasdid@appscode.com> Co-authored-by: qi <qiczzhu@gmail.com> Co-authored-by: Xunzhuo <bitliu@tencent.com> Co-authored-by: Kobi Levi <56400138+levikobi@users.noreply.github.com> Co-authored-by: Ardika <me@ardikabs.com> Co-authored-by: Lior Okman <lior.okman@sap.com> Co-authored-by: zirain <zirain2009@gmail.com> Co-authored-by: Tasdidur Rahman <52253951+TasdidurRahman@users.noreply.github.com>

ardikabs requested a review from a team as a code owner August 6, 2024 08:49

shawnh2 reviewed Aug 6, 2024

View reviewed changes

internal/provider/kubernetes/controller.go Outdated Show resolved Hide resolved

internal/provider/kubernetes/controller.go Outdated Show resolved Hide resolved

internal/provider/kubernetes/controller.go Outdated Show resolved Hide resolved

ardikabs force-pushed the fix-multiple-referencegrants-in-same-namespace branch 2 times, most recently from 096abfd to 9ebc349 Compare August 6, 2024 10:02

ardikabs force-pushed the fix-multiple-referencegrants-in-same-namespace branch 4 times, most recently from f327bb8 to bacea2c Compare August 7, 2024 14:46

shawnh2 approved these changes Aug 8, 2024

View reviewed changes

ardikabs added 3 commits August 8, 2024 21:41

fix: multiple reference grants in same namespace

d993599

Signed-off-by: Ardika Bagus <me@ardikabs.com>

test: add e2e test

36fac31

Signed-off-by: Ardika Bagus <me@ardikabs.com>

chore: wrong service port

dd4fde8

Signed-off-by: Ardika Bagus <me@ardikabs.com>

ardikabs force-pushed the fix-multiple-referencegrants-in-same-namespace branch from bacea2c to dd4fde8 Compare August 8, 2024 14:41

arkodg approved these changes Aug 8, 2024

View reviewed changes

arkodg merged commit b82f4b2 into envoyproxy:main Aug 8, 2024

arkodg added cherrypick/release-v1.0 cherrypick/release-v1.1 cherrypick to release/v1.1 cherrypick/release-v1.0.3 cherrypick/release-v1.1.1 and removed cherrypick/release-v1.0 cherrypick/release-v1.1 cherrypick to release/v1.1 labels Aug 8, 2024

ardikabs deleted the fix-multiple-referencegrants-in-same-namespace branch August 9, 2024 02:20

arkodg mentioned this pull request Sep 7, 2024

[release/v1.1] Cherry-pick commits for v1.1.1 #4173

Merged

arkodg mentioned this pull request Sep 7, 2024

[release/v1.0] Cherrypick commits for v1.0.3 #4174

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: multiple reference grants in same namespace#4008

fix: multiple reference grants in same namespace#4008
arkodg merged 3 commits intoenvoyproxy:mainfrom
ardikabs:fix-multiple-referencegrants-in-same-namespace

ardikabs commented Aug 6, 2024 •

edited

Loading

Uh oh!

codecov bot commented Aug 6, 2024 •

edited

Loading

Uh oh!

ardikabs commented Aug 6, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shawnh2 commented Aug 6, 2024

Uh oh!

ardikabs commented Aug 6, 2024

Uh oh!

shawnh2 commented Aug 6, 2024

Uh oh!

ardikabs commented Aug 7, 2024

Uh oh!

shawnh2 commented Aug 8, 2024

Uh oh!

shawnh2 left a comment

Uh oh!

shawnh2 commented Aug 8, 2024

Uh oh!

arkodg left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	for _, refGrant := range refGrants {
	if refGrant.Namespace == to.namespace {
	for _, src := range refGrant.Spec.From {
	if src.Kind == gwapiv1a2.Kind(from.kind) && string(src.Namespace) == from.namespace {
	return &refGrant, nil
	}
	}
	}
	}

	// No ReferenceGrant found.
	return nil, nil

	var fromAllowed bool
	for _, refGrantFrom := range referenceGrant.Spec.From {
	if string(refGrantFrom.Namespace) == from.namespace && string(refGrantFrom.Group) == from.group && string(refGrantFrom.Kind) == from.kind {
	fromAllowed = true
	break
	}
	}
	if !fromAllowed {
	continue
	}

	// Check if the ReferenceGrant has a matching "to".
	var toAllowed bool
	for _, refGrantTo := range referenceGrant.Spec.To {
	if string(refGrantTo.Group) == to.group && string(refGrantTo.Kind) == to.kind && (refGrantTo.Name == nil \|\| refGrantTo.Name == "" \|\| string(refGrantTo.Name) == to.name) {
	toAllowed = true
	break
	}
	}
	if !toAllowed {
	continue
	}

	// If we got here, both the "from" and the "to" were allowed by this
	// reference grant.
	return true

Conversation

ardikabs commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ardikabs commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shawnh2 commented Aug 6, 2024

Uh oh!

ardikabs commented Aug 6, 2024

Uh oh!

shawnh2 commented Aug 6, 2024

Uh oh!

ardikabs commented Aug 7, 2024

Uh oh!

shawnh2 commented Aug 8, 2024

Uh oh!

shawnh2 left a comment

Choose a reason for hiding this comment

Uh oh!

shawnh2 commented Aug 8, 2024

Uh oh!

arkodg left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ardikabs commented Aug 6, 2024 •

edited

Loading

codecov bot commented Aug 6, 2024 •

edited

Loading

ardikabs commented Aug 6, 2024 •

edited

Loading