feat: support infra deployment in the gateway namespace by cnvergence · Pull Request #5137 · envoyproxy/gateway

cnvergence · 2025-01-23T17:29:28Z

What type of PR is this?

feat: support infra deployment in the gateway namespace

What this PR does / why we need it:

Enable deploying Infra envoy proxies in the namespace of related Gateway resources.

Support GatewayNamespaceMode deploy mode in the gateway config map.
Adjust infra resources to reflect non-default namespace.
Set up the auth between xds-server and infra envoy proxy based on sTLS and JWT Auth, set up proper TLS credentials and interceptor in gRPC server.
Update the related helm chart to set up necessary permissions for envoy gateway ServiceAccount.

JWT token are validated using k8s TokenReview API .

Which issue(s) this PR fixes:

Fixes #2629

Release Notes: Yes

codecov · 2025-02-07T22:43:05Z

Codecov Report

Attention: Patch coverage is 50.00000% with 11 lines in your changes missing coverage. Please review.

Project coverage is 65.15%. Comparing base (68a2713) to head (bbddde9).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
api/v1alpha1/envoygateway_helpers.go	0.00%	5 Missing ⚠️
internal/gatewayapi/runner/runner.go	33.33%	2 Missing ⚠️
internal/gatewayapi/securitypolicy.go	33.33%	2 Missing ⚠️
internal/cmd/certgen.go	50.00%	1 Missing ⚠️
internal/extension/registry/extension_manager.go	0.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (50.00%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5137      +/-   ##
==========================================
- Coverage   65.41%   65.15%   -0.26%     
==========================================
  Files         222      224       +2     
  Lines       35643    35857     +214     
==========================================
+ Hits        23315    23364      +49     
- Misses      10882    11048     +166     
+ Partials     1446     1445       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

zhaohuabing · 2025-04-08T08:44:04Z

Hi @cnvergence Just checking in - any updates on this PR? Would love to see it land in 1.4.0.

cnvergence · 2025-04-08T16:21:31Z

hey @zhaohuabing, getting back to this after Kubecon, I need to fix connection between control-plane and proxies, if everything will work again, I think it should be ready for 1.4.0 :)

zhaohuabing · 2025-04-11T14:34:34Z

hey @zhaohuabing, getting back to this after Kubecon, I need to fix connection between control-plane and proxies, if everything will work again, I think it should be ready for 1.4.0 :)

@cnvergence a quick way to solve this issue is to copy the envoy secret into each namespace that contains a Gateway resource. While not ideal, this would unblock some key use cases -- such as using EG as an Ambient Waypoint proxy, and allow us to get this PR into v1.4.0.

This is not the most optimal solution, since all Gateway infras share the same client cert. As the next step, we can introduce logic in Envoy Gateway to automatically generate a unique cert secret per Gateway. This can be handled in a follow-up PR.

Would love your input on this, @envoyproxy/gateway-maintainers

arkodg · 2025-04-11T14:47:21Z

hey @zhaohuabing, getting back to this after Kubecon, I need to fix connection between control-plane and proxies, if everything will work again, I think it should be ready for 1.4.0 :)

@cnvergence a quick way to solve this issue is to copy the envoy secret into each namespace that contains a Gateway resource. While not ideal, this would unblock some key use cases -- such as using EG as an Ambient Waypoint proxy, and allow us to get this PR into v1.4.0.

This is not the most optimal solution, since all Gateway infras share the same client cert. As the next step, we can introduce logic in Envoy Gateway to automatically generate a unique cert secret per Gateway. This can be handled in a follow-up PR.

Would love your input on this, @envoyproxy/gateway-maintainers

-1 to this, app teams have will have access to this, and can take advantage

cnvergence · 2025-04-14T12:20:17Z

@zhaohuabing
Hey, thanks for the hint, we have discussed a few solutions:

generating certs per every Gateway
sharing EG cert per every infra proxy
moving to sTLS with JWT validation

While we can introduce more options as the followup to this, we wanted to do it initially with the option 3.
This way we need only to share the CA cert and set up JWT validation with serviceaccount token for envoy proxies.

zhaohuabing · 2025-04-15T14:44:45Z

To move this forward, I agree that we can start without client cert and use the JWT token for client-side validation for the namespaced mode, but please keep mTLS for the current mode.

We eventually will need mTLS for stronger security posture and compliance reasons. We may end up introducing a shim to exchange the JWT token for a client cert, and then establish the mTLS XDS connection for Envoy. This solution is similar to the approach taken by Istio's pilot agent.

If @arkodg also feels this is the right approach, we can go ahead.

cnvergence · 2025-04-15T19:34:58Z

Yes, mTLS by default was always planned, the JWT method would only be used for the gatewayNamespacedMode.

cnvergence · 2025-04-25T15:45:51Z

/retest

arkodg · 2025-04-25T23:13:49Z

internal/xds/server/kubejwt/tokenreview.go

are we validating that these token belongs to an envoy created by EG ?

Just by the service account, I was thinking about it, we will need to have an exact name of the envoy pod

Can we leverage the IR keys in the snapshot cache which map to the fleet deployment name and namespace?

Tracked in this issue: #5863

I have added the basic check for deployment in snapshot cache, we will need to improve it and make it work with merged gateways mode

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

cnvergence · 2025-04-30T07:57:50Z

Thanks for the review, I will tackle open issues as the followups

zhaohuabing

LGTM. Thanks for the patience!

charts/gateway-helm/templates/infra-manager-rbac.yaml

cnvergence force-pushed the feat-support-infra-different-ns branch 2 times, most recently from 7bf3489 to 91fadc0 Compare February 7, 2025 22:35

cnvergence force-pushed the feat-support-infra-different-ns branch 3 times, most recently from 291e372 to 3d3932b Compare February 12, 2025 15:14

cnvergence force-pushed the feat-support-infra-different-ns branch from 3d3932b to 9cd1e1b Compare February 20, 2025 14:02

cnvergence force-pushed the feat-support-infra-different-ns branch from 9cd1e1b to 9ee9d48 Compare March 11, 2025 10:57

cnvergence force-pushed the feat-support-infra-different-ns branch 3 times, most recently from ab8e446 to 7e27132 Compare March 24, 2025 15:27

cnvergence force-pushed the feat-support-infra-different-ns branch from b48eec8 to b73753e Compare April 15, 2025 19:32

cnvergence force-pushed the feat-support-infra-different-ns branch 4 times, most recently from 31850ce to d46e101 Compare April 22, 2025 17:25

cnvergence changed the title ~~wip feat: support infra deployment in the gateway namespace~~ feat: support infra deployment in the gateway namespace Apr 24, 2025

cnvergence force-pushed the feat-support-infra-different-ns branch from 49fc667 to f6e88f8 Compare April 25, 2025 11:49

cnvergence marked this pull request as ready for review April 25, 2025 11:49

cnvergence requested a review from a team as a code owner April 25, 2025 11:49

arkodg reviewed Apr 25, 2025

View reviewed changes

zirain force-pushed the feat-support-infra-different-ns branch from 079ecae to 38ec262 Compare April 26, 2025 12:47

cnvergence added 15 commits April 30, 2025 09:24

update grpc xds server and share eg ca cert in the infra configmap

f8d0bcd

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

fix tests

c198bfe

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

lint code

20e56ca

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

update stls config for xds server in namespace mode

3f71f3a

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

update codgen

0c64a64

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

update stls+jwt config

d520d8a

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

move to token review API for jwt validation

a5d8e86

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

remove old func

e03dd69

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

go lint

f0ecfc8

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

Update testdata

63a3e51

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

check for serviceaccount group in the tokenReview groups slice

826de23

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

fix tests

77ebebb

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

update testdata

1ef7776

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

Add initial token validation, add tests and review changes

5bc031f

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

lint code

91c297b

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

cnvergence force-pushed the feat-support-infra-different-ns branch from 79afa42 to 91c297b Compare April 30, 2025 07:28

add zone envvar

bbddde9

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

This comment was marked as outdated.

Sign in to view

zhaohuabing approved these changes Apr 30, 2025

View reviewed changes

zirain approved these changes Apr 30, 2025

View reviewed changes

zirain merged commit 0a658b8 into envoyproxy:main Apr 30, 2025
23 of 25 checks passed

cnvergence deleted the feat-support-infra-different-ns branch April 30, 2025 08:28

This was referenced Apr 30, 2025

feat: validate JWT token and use projected token #5871

Merged

feat: add controller namespace field to infrastructure render #5937

Merged

suin mentioned this pull request May 8, 2025

bug: GatewayNamespace mode fails to handle Gateways across multiple namespaces #5952

Closed

zirain reviewed May 8, 2025

View reviewed changes

charts/gateway-helm/templates/infra-manager-rbac.yaml Show resolved Hide resolved

zirain mentioned this pull request May 8, 2025

Better controls of ClusterRole of gateway namespace mode #5962

Closed

kkk777-7 mentioned this pull request May 8, 2025

Set ownerReference on gateway Deployment #3051

Closed

HannaManista mentioned this pull request May 14, 2025

api: support infra deployment in the gateway namespace #4982

Merged

Conversation

cnvergence commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

zhaohuabing commented Apr 8, 2025

Uh oh!

cnvergence commented Apr 8, 2025

Uh oh!

zhaohuabing commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arkodg commented Apr 11, 2025

Uh oh!

cnvergence commented Apr 14, 2025

Uh oh!

zhaohuabing commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cnvergence commented Apr 15, 2025

Uh oh!

cnvergence commented Apr 25, 2025

Uh oh!

arkodg Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

cnvergence Apr 27, 2025

Choose a reason for hiding this comment

Uh oh!

arkodg Apr 27, 2025

Choose a reason for hiding this comment

Uh oh!

zhaohuabing Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

cnvergence Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

cnvergence commented Apr 30, 2025

Uh oh!

zhaohuabing left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cnvergence commented Jan 23, 2025 •

edited

Loading

codecov bot commented Feb 7, 2025 •

edited

Loading

zhaohuabing commented Apr 11, 2025 •

edited

Loading

zhaohuabing commented Apr 15, 2025 •

edited

Loading