feat: add file resource provider for EG standalone mode by shawnh2 · Pull Request #3159 · envoyproxy/gateway

shawnh2 · 2024-04-10T04:06:53Z

What type of PR is this?

What this PR does / why we need it:

This PR implement two component for file resource provider: Notifier and ResourceStore.

Notifier monitors paths(files or dirs) on host via fsnotify, and send Write/Remove event to ResourceStore
ResourceStore load and store resources according to the received event

Which issue(s) this PR fixes:

related #1393

There're some TODOs in this PR, tracked by #3213 #3207

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

… provider Signed-off-by: shawnh2 <shawnhxh@outlook.com>

…rovider

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

…rovider

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

zhaohuabing

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

arkodg · 2024-04-24T00:24:43Z

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

Custom is needed to allow for different resource and infra provider

zhaohuabing · 2024-04-24T00:30:53Z

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

Custom is needed to allow for different resource and infra provider

Add it later when a use case comes out? The File/Host Provider is not a custom provider, it's a known one.

arkodg · 2024-04-24T00:47:36Z

As discussed in today's meeting, I prefer to remove the EnvoyGatewayCustomProvider and add a flat EnvoyGatewayFileProvider(or a better name).

Custom is needed to allow for different resource and infra provider

Add it later when a use case comes out? The File/Host Provider is not a custom provider, it's a known one.

use case exists today - file resource provider and host infra provider to implement standalone case, running EG in a docker container or bare metal directly w/o any API server

Xunzhuo · 2024-05-14T08:26:05Z

/retest

Xunzhuo

Do we plan on releasing this in v1.1.0 ?

cc @arkodg

arkodg · 2024-05-15T00:39:47Z

Do we plan on releasing this in v1.1.0 ?

cc @arkodg

yeah I'm a +1 for this, prioritizing other PRs right now, will get to this one soon

shawnh2 · 2024-06-04T03:42:47Z

will split changes of api/v1alpha1/validation/envoygateway_validate.go into a separate PR, since it's only a refactor and very easily to conflict.

github-actions · 2024-07-04T04:02:11Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. Please feel free to give a status update now, ping for review, when it's ready. Thank you for your contributions!

sky92zwq · 2024-07-30T11:25:26Z

Hi, does it set -c or --config-path to switch to file-mode? how to implement the config file?

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

codecov · 2024-08-13T14:53:44Z

Codecov Report

Attention: Patch coverage is 7.87270% with 550 lines in your changes missing coverage. Please review.

Project coverage is 66.54%. Comparing base (c109131) to head (3c5d0f1).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/provider/file/resources.go	0.00%	234 Missing ⚠️
internal/provider/file/notifier.go	0.00%	158 Missing ⚠️
internal/provider/file/file.go	0.00%	66 Missing ⚠️
internal/provider/runner/runner.go	0.00%	32 Missing ⚠️
internal/provider/file/store.go	0.00%	26 Missing ⚠️
internal/cmd/server.go	0.00%	15 Missing ⚠️
internal/infrastructure/manager.go	0.00%	9 Missing ⚠️
internal/infrastructure/runner/runner.go	0.00%	6 Missing ⚠️
internal/provider/file/path.go	88.88%	1 Missing and 1 partial ⚠️
internal/provider/kubernetes/kubernetes.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3159      +/-   ##
==========================================
- Coverage   67.99%   66.54%   -1.45%     
==========================================
  Files         190      195       +5     
  Lines       23130    23699     +569     
==========================================
+ Hits        15727    15771      +44     
- Misses       6284     6809     +525     
  Partials     1119     1119

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

api/v1alpha1/validation/envoygateway_validate.go

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

api/v1alpha1/envoygateway_types.go

sky92zwq · 2024-08-27T09:25:57Z

internal/provider/file/resources.go

+// - This issue is tracked by https://github.com/envoyproxy/gateway/issues/3207
+//
+// convertKubernetesYAMLToResources converts a Kubernetes YAML string into GatewayAPI Resources.
+func convertKubernetesYAMLToResources(str string) (*gatewayapi.Resources, error) {


should deal with ReferenceGrant type resource in this func?

Yes, it should. This PR only leverage the basic functionality for file-provider, as described in the comments here, this function will be revisited along with the one in egctl x translate.

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

shawnh2 · 2024-09-08T09:10:52Z

Hi, does it set -c or --config-path to switch to file-mode? how to implement the config file?

You can start file-provider mode like envoy-gateway server -c config.yaml, with config.yaml:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
gateway:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
provider:
  type: Custom
  custom:
    resource:
      type: File
      file:
        paths: ["test/"]
logging:
  level:
    default: info

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

…rovider

shawnh2 · 2024-09-08T12:42:50Z

/retest

arkodg

thanks !

Xunzhuo

LGTM. Thanks

shawnh2 added 8 commits April 1, 2024 21:27

add validations for envoy-gateway file resource type

93e4dd3

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

improve eg validation and add resource provider interface for various…

3956911

… provider Signed-off-by: shawnh2 <shawnhxh@outlook.com>

Merge branch 'main' of github.com:envoyproxy/gateway into host-file-p…

9acf325

…rovider

extract common gatewayapi layer translate logic in egctl translate

7de91b3

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

add notifier support

3419af6

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

fix lint and move read yaml bytes function back to translate

e709eae

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

add resources store support

6f3b400

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

Merge branch 'main' of github.com:envoyproxy/gateway into host-file-p…

739c8dc

…rovider

shawnh2 marked this pull request as ready for review April 18, 2024 07:09

shawnh2 requested a review from a team as a code owner April 18, 2024 07:09

fix lint

5726ee5

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

zhaohuabing reviewed Apr 24, 2024

View reviewed changes

Xunzhuo reviewed May 14, 2024

View reviewed changes

github-actions bot added the stale label Jul 4, 2024

Xunzhuo removed the stale label Jul 25, 2024

arkodg added this to the v1.2.0-rc1 milestone Aug 1, 2024

shawnh2 mentioned this pull request Aug 4, 2024

refactor: clean envoygateway validate methods #3997

Merged

shawnh2 added 2 commits August 13, 2024 22:01

resolve conflicts

9460667

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

fix ci

2e1db1b

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

arkodg reviewed Aug 15, 2024

View reviewed changes

api/v1alpha1/validation/envoygateway_validate.go Outdated Show resolved Hide resolved

arkodg reviewed Aug 15, 2024

View reviewed changes

api/v1alpha1/validation/envoygateway_validate.go Outdated Show resolved Hide resolved

shawnh2 added 5 commits August 24, 2024 20:13

update custom provider comments and validate method test

21353e2

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

restore extension manager and add health probe server for file provider

59e9c59

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

update envoy gateway helper functions

be44b88

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

add some unit tests

ec357fa

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

properly handle the remove event for the file provider

4052618

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

shawnh2 dismissed stale reviews from zhaohuabing and arkodg via 4052618 August 25, 2024 08:21

shawnh2 removed the hold do not merge label Aug 25, 2024

shawnh2 requested review from a team, arkodg, cnvergence and zhaohuabing August 25, 2024 08:24

fix lint

381efd6

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

arkodg reviewed Aug 26, 2024

View reviewed changes

api/v1alpha1/envoygateway_types.go Outdated Show resolved Hide resolved

sky92zwq reviewed Aug 27, 2024

View reviewed changes

no default to k8s for infra provider

a30b1ea

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

shawnh2 requested a review from arkodg September 8, 2024 09:09

shawnh2 added 2 commits September 8, 2024 18:21

fix runner

a635d03

Signed-off-by: shawnh2 <shawnhxh@outlook.com>

Merge branch 'main' of github.com:envoyproxy/gateway into host-file-p…

61638ee

…rovider

arkodg approved these changes Sep 9, 2024

View reviewed changes

arkodg requested review from a team September 9, 2024 19:03

Merge branch 'main' into host-file-provider

3c5d0f1

Xunzhuo approved these changes Sep 11, 2024

View reviewed changes

Xunzhuo merged commit f5693f6 into envoyproxy:main Sep 11, 2024

shawnh2 deleted the host-file-provider branch September 11, 2024 02:34

Conversation

shawnh2 commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhaohuabing left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arkodg commented Apr 24, 2024

Uh oh!

zhaohuabing commented Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arkodg commented Apr 24, 2024

Uh oh!

Xunzhuo commented May 14, 2024

Uh oh!

Xunzhuo left a comment

Choose a reason for hiding this comment

Uh oh!

arkodg commented May 15, 2024

Uh oh!

shawnh2 commented Jun 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 4, 2024

Uh oh!

sky92zwq commented Jul 30, 2024

Uh oh!

codecov bot commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sky92zwq Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

shawnh2 Sep 8, 2024

Choose a reason for hiding this comment

Uh oh!

shawnh2 commented Sep 8, 2024

Uh oh!

shawnh2 commented Sep 8, 2024

Uh oh!

arkodg left a comment

Choose a reason for hiding this comment

Uh oh!

Xunzhuo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

shawnh2 commented Apr 10, 2024 •

edited

Loading

zhaohuabing left a comment •

edited

Loading

zhaohuabing commented Apr 24, 2024 •

edited

Loading

shawnh2 commented Jun 4, 2024 •

edited

Loading

codecov bot commented Aug 13, 2024 •

edited

Loading