feat: Add EnvoyProxyTemplate for global Envoy Proxy defaults#7698
Conversation
7da24bf to
02beef5
Compare
fd2a520 to
0ae8e1d
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7698 +/- ##
==========================================
+ Coverage 73.52% 73.78% +0.26%
==========================================
Files 237 237
Lines 35653 35767 +114
==========================================
+ Hits 26214 26392 +178
+ Misses 7576 7518 -58
+ Partials 1863 1857 -6 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
0ae8e1d to
804ff4b
Compare
23b261c to
ff220ab
Compare
e5283b6 to
c198efe
Compare
api/v1alpha1/shared_types.go
Outdated
| @@ -180,8 +180,10 @@ type KubernetesPodSpec struct { | |||
| // Volumes that can be mounted by containers belonging to the pod. | |||
| // More info: https://kubernetes.io/docs/concepts/storage/volumes | |||
| // | |||
| // +patchMergeKey=name | |||
| // +patchStrategy=merge | |||
There was a problem hiding this comment.
thanks, can you also add these for ImagePullSecrets , TopologySpreadConstraints, InitContainers and all other slices in EnvoyProxy
| "github.com/envoyproxy/gateway/internal/utils" | ||
| ) | ||
|
|
||
| // MergeEnvoyProxyConfigs merges EnvoyProxy configurations using a 3-level hierarchy: |
There was a problem hiding this comment.
this changes the spec
3. completely replaces 2. today, there is no merge
so 1. would either merge with 3. if it exists or merge with 2. if 3. is absent
There was a problem hiding this comment.
Yes you are right. The original version only did the merge if the template was present. See below.
api/v1alpha1/envoygateway_types.go
Outdated
| // (e.g., env vars, volumes) are combined from both sources. | ||
| // | ||
| // +optional | ||
| EnvoyProxyTemplate *EnvoyProxySpec `json:"envoyProxyTemplate,omitempty"` |
There was a problem hiding this comment.
we may want create a new struct type and embed EnvoyProxySpec which allows us to add a mergeType or something similar to highlight how this will get merged
There was a problem hiding this comment.
To preserve backward compatibility, added a new EnvoyProxyTemplateSpec struct with a mergeType option - where Replace is the default.
There was a problem hiding this comment.
@arkodg Would you be able to give this another once over? It would be great to get this in.
There was a problem hiding this comment.
sorry for the delay, reviewing rn
dd0a4dc to
67f0b55
Compare
✅ Deploy Preview for cerulean-figolla-1f9435 canceled.
|
api/v1alpha1/envoygateway_types.go
Outdated
| // with or replace template definitions based on the mergeType. | ||
| // | ||
| // +optional | ||
| EnvoyProxyTemplate *EnvoyProxyTemplateSpec `json:"envoyProxyTemplate,omitempty"` |
There was a problem hiding this comment.
should this be EnvoyProxyTemplate or EnvoyProxy
@envoyproxy/gateway-maintainers
(lets time box this to a week to avoid more delay for this PR)
api/v1alpha1/envoygateway_types.go
Outdated
| // | ||
| // +optional | ||
| // +kubebuilder:default=Replace | ||
| MergeType *EnvoyProxyTemplateMergeType `json:"mergeType,omitempty"` |
There was a problem hiding this comment.
for the BackendTrafficPolicy case, the child policy specifies the MergeType and defaults to Replace, right now in this case the parent has this setting, should we move this to the EnvoyProxy resource ?
@envoyproxy/gateway-maintainers
There was a problem hiding this comment.
We chatted about this in the community meeting. I'm a +1 on moving MergeType into EnvoyProxy and using that in EnvoyGateway too so we can define the merge behavior in all layers.
There was a problem hiding this comment.
yeah piggybacking based on what Isaac said, we can split this work into 2 parts/PRs
- Introduce
EnvoyProxy *EnvoyProxySpecin EnvoyGateway and use the same replace semantics we have for deciding what EnvoyProxy wins - Later introduce
MergeTypeintoEnvoyProxySpecand deal with multi level merge
sorry for the churn @mgs255, hope this adds more clarity
There was a problem hiding this comment.
@arkodg Yes, sounds reasonable but does that work in practice?
The way I've installed this in our environments, we have all of the configuration defined centrally in the helm values file. There is large envoyProxyTemplate e.g:
# Lots above....
rateLimitHpa:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
envoyProxyTemplate:
mergeType: StrategicMerge
spec:
mergeGateways: true
logging:
level:
default: info
config: info
upstream: info
jwt: info
oauth2: info
http: info
filter: info
router: info
connection: info
provider:
kubernetes:
envoyHpa:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
# Lots more below....
...We also a minimal EnvoyProxy object in each target cluster using the following template in a different helm chart:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: {{ .Values.proxyName }}
namespace: {{ .Release.Namespace }}
labels: {{- include "pulse-envoy-aws-gateway.labels" . | nindent 4 }}
# Ideally we would not set any properties here and just let the template override everything - unfortunately
# a complete empty spec (even {} or null) appears invalid
spec:
logging:
level:
default: {{ .Values.logging.level.proxy }}
If we only supported the replace option initially then the most specific EnvoyProxy would completely replace the values from the template, right?
There was a problem hiding this comment.
yeah just suggesting the 2 step approach to increase the velocity of code review and getting something in, but agree when merge lands, thats when your use cases and more powerful use cases will be unlocked
There was a problem hiding this comment.
hey @mgs255 would be great if we can get part 1 into v1.7.0-rc.1 (deadline is 01/28)
There was a problem hiding this comment.
Ok - re-done based on those comments. I will work on some further changes to also add the mergeType and start testing this in our environments.
|
hey have some open questions here @mgs255, will raise these in the community meeting next week and get them resolved |
f709f2e to
6adea4a
Compare
Add a new envoyProxy field to the EnvoyGateway spec to support cluster-wide shared defaults. This defines a three-level hierarchy for EnvoyProxy configuration (highest to lowest priority): 1. Gateway-level EnvoyProxy (via Gateway.spec.infrastructure.parametersRef) 2. GatewayClass-level EnvoyProxy (via GatewayClass.spec.parametersRef) 3. EnvoyProxy defaults on EnvoyGateway Currently uses replace semantics where the most specific configuration wins completely. The default spec is almost completely redundant in the sense that it gets completely replaced by any more specific configuration - but this establishes the foundation for a future PR that will add a MergeType to enable merging configurations across levels. Towards envoyproxy#4764 Signed-off-by: Michael Sommerville <msommerville@gmail.com>
6adea4a to
5d64148
Compare
Signed-off-by: Michael Sommerville <msommerville@gmail.com>
zirain
left a comment
There was a problem hiding this comment.
LGTM, it would be good to add some e2e for this with following PR.
e.g. add a dummy/valid EnvoyProxy in test/config/envoy-gateaway-config/default.yaml and the one referenced by GatewayClass will override it, e2e should all pass.
|
thanks, LGTM, +1 to zirain's comments around add some tests |
Signed-off-by: Michael Sommerville <msommerville@gmail.com>
e246274 to
fa0a115
Compare
|
One thing could be imrpove in the next is adding another test case by adding a new GC and it shouldn’t be provision correctly because of the invalid image. |
|
no sure what happen, but this cause e2e always failed on dual+default. |
|
That is really odd. It must be related to the addition in contexts.go. I'll open a new branch removing that section. I'm having a real problem trying to run the e2e tests locally on Mac though so it is a pain to debug this. @arkodg @zirain I've pushed a branch which should completely disable that defaulting behaviour for now. |
…oxy#7698) * Introduce EnvoyProxy default configuration Add a new envoyProxy field to the EnvoyGateway spec to support cluster-wide shared defaults. This defines a three-level hierarchy for EnvoyProxy configuration (highest to lowest priority): 1. Gateway-level EnvoyProxy (via Gateway.spec.infrastructure.parametersRef) 2. GatewayClass-level EnvoyProxy (via GatewayClass.spec.parametersRef) 3. EnvoyProxy defaults on EnvoyGateway Currently uses replace semantics where the most specific configuration wins completely. The default spec is almost completely redundant in the sense that it gets completely replaced by any more specific configuration - but this establishes the foundation for a future PR that will add a MergeType to enable merging configurations across levels. Towards envoyproxy#4764 Signed-off-by: Michael Sommerville <msommerville@gmail.com> Signed-off-by: Sadmi Bouhafs <sadmibouhafs@gmail.com>
|
Just wanted to say thanks for adding this! :) It simplified all my deployments a lot 🫡 |
Introduce EnvoyProxy default configuration
Add a new envoyProxy field to the EnvoyGateway spec to support cluster-wide shared defaults. This defines a three-level hierarchy for EnvoyProxy configuration (highest to lowest priority):
Currently uses replace semantics where the most specific configuration wins completely. The default spec is almost completely redundant in the sense that it gets completely replaced by any more specific configuration - but this establishes the foundation for a future PR that will add a MergeType to enable merging configurations across levels.
Towards #4764
Signed-off-by: Michael Sommerville msommerville@gmail.com