Skip to content

send resource statuses using watchable#395

Merged
arkodg merged 14 commits intoenvoyproxy:mainfrom
arkodg:status-infra
Sep 22, 2022
Merged

send resource statuses using watchable#395
arkodg merged 14 commits intoenvoyproxy:mainfrom
arkodg:status-infra

Conversation

@arkodg
Copy link
Copy Markdown
Contributor

@arkodg arkodg commented Sep 16, 2022

  • Allow multiple runners (such as gatewayapi) to update the status of resources such as gateway and httproute by publishing the updated status of the resource on a watchable map.
  • The provider runner subscribes to these status updates and updates the resource status in the API Server

Signed-off-by: Arko Dasgupta arko@tetrate.io

@arkodg arkodg requested a review from a team as a code owner September 16, 2022 22:00
@arkodg arkodg added the provider/kubernetes Issues related to the Kubernetes provider label Sep 16, 2022
danehans
danehans reviewed Sep 20, 2022
View reviewed changes
danehans
danehans suggested changes Sep 20, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@danehans danehans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arkodg in my testing, I am seeing a lot of:

2022-09-20T20:32:53.057Z	ERROR	status/status.go:81	unable to update status	{"runner": "provider", "name": "eg", "namespace": "default", "error": "Operation cannot be fulfilled on gateways.gateway.networking.k8s.io \"eg\": the object has been modified; please apply your changes to the latest version and try again"}

Are you creating a copy of the Gateway before calling client.Update()?

I'm also seeing a continued reconciliation loop between the GatewayStatus provider and subscriber due to status updates:

2022-09-20T20:36:06.097Z	INFO	kubernetes/httproute.go:256	received a status notification	{"runner": "provider"}
2022-09-20T20:36:06.097Z	INFO	kubernetes/gateway.go:335	received a status notification	{"runner": "provider"}
2022-09-20T20:36:06.197Z	INFO	kubernetes/gateway.go:144	reconciling gateway	{"runner": "provider", "namespace": "default", "name": "eg"}
2022-09-20T20:36:06.197Z	INFO	default.eg	kubernetes/gateway.go:225	reconciled gateway	{"runner": "provider"}
2022-09-20T20:36:06.197Z	INFO	kubernetes/gateway.go:335	received a status notification	{"runner": "provider"}
2022-09-20T20:36:06.197Z	INFO	runner/runner.go:69	received a notification	{"runner": "gateway-api"}

@arkodg
Copy link
Copy Markdown
Contributor Author

arkodg commented Sep 20, 2022

@arkodg in my testing, I am seeing a lot of:

2022-09-20T20:32:53.057Z	ERROR	status/status.go:81	unable to update status	{"runner": "provider", "name": "eg", "namespace": "default", "error": "Operation cannot be fulfilled on gateways.gateway.networking.k8s.io \"eg\": the object has been modified; please apply your changes to the latest version and try again"}

Are you creating a copy of the Gateway before calling client.Update()?

I'm also seeing a continued reconciliation loop between the GatewayStatus provider and subscriber due to status updates:

2022-09-20T20:36:06.097Z	INFO	kubernetes/httproute.go:256	received a status notification	{"runner": "provider"}
2022-09-20T20:36:06.097Z	INFO	kubernetes/gateway.go:335	received a status notification	{"runner": "provider"}
2022-09-20T20:36:06.197Z	INFO	kubernetes/gateway.go:144	reconciling gateway	{"runner": "provider", "namespace": "default", "name": "eg"}
2022-09-20T20:36:06.197Z	INFO	default.eg	kubernetes/gateway.go:225	reconciled gateway	{"runner": "provider"}
2022-09-20T20:36:06.197Z	INFO	kubernetes/gateway.go:335	received a status notification	{"runner": "provider"}
2022-09-20T20:36:06.197Z	INFO	runner/runner.go:69	received a notification	{"runner": "gateway-api"}

Ah not passing a DeepCopy, will edit to do so

Arko Dasgupta added 13 commits September 20, 2022 16:40
send resource statuses using watchable
afad62b 
* Allow multiple runners (such as gatewayapi)
to update the status of resources such as gateway and httproute
by publishing the updated status of the resource on a watchable map.
* The provider runner subscribes to these status updates and updates
the resource status in the API Server

Signed-off-by: Arko Dasgupta <arko@tetrate.io>
update LastTransitionTime condition field
58b99d5 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
fix bugs in the gatewayapi status logic
ca2bfae 
* Use reflect.DeepEqual to compare parentRef values

* Fix the Listener ready condition logic which was causing
the entire translator to panic

* Use snapshot.Updates instead of snaphsot.State

* fix build in kube test

Signed-off-by: Arko Dasgupta <arko@tetrate.io>
fix null deployment check
d56cf65 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
close status messages
821ff35 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
fix testData to include controllerName
0e2210f 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
rename to the coverage file EG creates
da88030 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
fix tests
79e65d8 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
pass copy to update method
a5dcfee 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
fix tests based on listener ready status
b709684 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
deep copy when passing to status updater
680e2cf 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
add help string for release-manifests
14db0f0 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
add status cmp support for HTTPRoute
db5a378 
Signed-off-by: Arko Dasgupta <arko@tetrate.io>
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Sep 21, 2022

Codecov Report

Merging #395 (db5a378) into main (70a0d3c) will decrease coverage by 0.23%.
The diff coverage is 56.25%.

@@            Coverage Diff             @@
##             main     #395      +/-   ##
==========================================
- Coverage   61.20%   60.96%   -0.24%     
==========================================
  Files          36       35       -1     
  Lines        3714     3786      +72     
==========================================
+ Hits         2273     2308      +35     
- Misses       1317     1351      +34     
- Partials      124      127       +3
Impacted Files Coverage Δ
internal/cmd/server.go 8.03% <0.00%> (-0.15%) ⬇️
internal/gatewayapi/runner/runner.go 58.51% <0.00%> (-6.20%) ⬇️
internal/message/types.go 100.00% <ø> (ø)
internal/provider/kubernetes/kubernetes.go 57.50% <0.00%> (ø)
internal/status/gateway.go 0.00% <0.00%> (ø)
internal/status/status.go 0.00% <0.00%> (ø)
internal/provider/kubernetes/httproute.go 59.32% <50.00%> (-2.68%) ⬇️
internal/provider/kubernetes/gateway.go 66.81% <84.37%> (+1.88%) ⬆️
internal/gatewayapi/contexts.go 76.92% <85.71%> (+0.14%) ⬆️
internal/gatewayapi/translator.go 86.12% <100.00%> (+0.01%) ⬆️
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@arkodg
Copy link
Copy Markdown
Contributor Author

arkodg commented Sep 21, 2022

@danehans I think there is an infinite loop being created

  • provider reconciles Gateways
  • Sends it over to the gateway-api runner
  • which recomputes status and sends it over to provider to update status
  • updating status triggers another reconcile and the loop continues

is there a way to skip reconciles when the object status changes ?

fix infinite reconcile loop
054055b 
* reset attached routes before translation to eliminate double count

* dont update the status condition if the condition params havent
changed

Signed-off-by: Arko Dasgupta <arko@tetrate.io>
@arkodg arkodg requested a review from danehans September 21, 2022 15:22
@arkodg
Copy link
Copy Markdown
Contributor Author

arkodg commented Sep 21, 2022

@danehans ptal, fixed the infinite loop reconciliation issue in the latest commit

@arkodg arkodg added the kind/enhancement New feature or request label Sep 21, 2022
@arkodg arkodg added area/message-service Issues related to Gateway's message service used for communication among components. area/translator Issues related to Gateway's translation service, e.g. translating Gateway APIs into the IR. labels Sep 21, 2022
danehans
danehans suggested changes Sep 22, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@danehans danehans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arkodg with this PR, EG no longer provisions the proxy infra after creating a gatewayclass and gateway.

@arkodg
Copy link
Copy Markdown
Contributor Author

arkodg commented Sep 22, 2022 •
edited
Loading

@arkodg with this PR, EG no longer provisions the proxy infra after creating a gatewayclass and gateway.

not seeing this when Ive tested it. Here are the logs

$ kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.2.0-rc1/examples/kubernetes/gatewayclass.yaml
gatewayclass.gateway.networking.k8s.io/eg created
$ kubectl apply -f https://raw.githubusercontent.com//envoyproxy/gateway/v0.2.0-rc1/examples/kubernetes/gateway.yaml
gateway.gateway.networking.k8s.io/eg created
$ kubectl get gateway/eg -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"name":"eg","namespace":"default"},"spec":{"gatewayClassName":"eg","listeners":[{"name":"http","port":8080,"protocol":"HTTP"}]}}
  creationTimestamp: "2022-09-22T16:51:20Z"
  generation: 1
  name: eg
  namespace: default
  resourceVersion: "791"
  uid: 8edc57ca-2221-4056-91e7-84365ce531f6
spec:
  gatewayClassName: eg
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: http
    port: 8080
    protocol: HTTP
status:
  addresses:
  - type: Hostname
    value: localhost
  conditions:
  - lastTransitionTime: "2022-09-22T16:51:20Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-09-22T16:51:21Z"
    message: Address assigned to the Gateway, 1/1 envoy Deployment replicas available
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
$ kubectl get po -A
NAMESPACE              NAME                                           READY   STATUS      RESTARTS   AGE
envoy-gateway-system   envoy-85b9c4684-2rkn5                          1/1     Running     0          35s
envoy-gateway-system   envoy-gateway-78cfc9dd4c-mtw8j                 2/2     Running     0          78s
gateway-system         gateway-api-admission-patch-wb2gp              0/1     Completed   1          2m5s
gateway-system         gateway-api-admission-rzt5d                    0/1     Completed   0          2m5s
gateway-system         gateway-api-admission-server-f849f547f-9rdvc   1/1     Running     0          2m5s
kube-system            coredns-6d4b75cb6d-cgqzr                       1/1     Running     0          2m10s
kube-system            coredns-6d4b75cb6d-p9kfp                       1/1     Running     0          2m10s
kube-system            etcd-docker-desktop                            1/1     Running     74         2m16s
kube-system            kube-apiserver-docker-desktop                  1/1     Running     70         2m10s
kube-system            kube-controller-manager-docker-desktop         1/1     Running     70         2m7s
kube-system            kube-proxy-kqcks                               1/1     Running     0          2m11s
kube-system            kube-scheduler-docker-desktop                  1/1     Running     155        2m15s
kube-system            storage-provisioner                            1/1     Running     0          101s
kube-system            vpnkit-controller                              1/1     Running     0          101s

$ kubectl get httproute -A
No resources found

$ kubectl get deployment/envoy-gateway -n envoy-gateway-system -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"control-plane":"envoy-gateway"},"name":"envoy-gateway","namespace":"envoy-gateway-system"},"spec":{"replicas":1,"selector":{"matchLabels":{"control-plane":"envoy-gateway"}},"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/default-container":"envoy-gateway"},"labels":{"control-plane":"envoy-gateway"}},"spec":{"containers":[{"args":["server","--config-path=/config/envoy-gateway.yaml"],"env":[{"name":"ENVOY_GATEWAY_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"image":"envoyproxy/gateway-dev:latest","imagePullPolicy":"Always","name":"envoy-gateway","resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"10m","memory":"64Mi"}},"securityContext":{"allowPrivilegeEscalation":false},"volumeMounts":[{"mountPath":"/config","name":"envoy-gateway-config","readOnly":true}]},{"args":["--secure-listen-address=0.0.0.0:8443","--upstream=http://127.0.0.1:8080/","--logtostderr=true","--v=0"],"image":"gcr.io/kubebuilder/kube-rbac-proxy:v0.11.0","name":"kube-rbac-proxy","ports":[{"containerPort":8443,"name":"https","protocol":"TCP"}],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"5m","memory":"64Mi"}}}],"securityContext":{"runAsNonRoot":true},"serviceAccountName":"envoy-gateway","terminationGracePeriodSeconds":10,"volumes":[{"configMap":{"defaultMode":420,"name":"envoy-gateway-config"},"name":"envoy-gateway-config"}]}}}}
  creationTimestamp: "2022-09-22T16:49:51Z"
  generation: 2
  labels:
    control-plane: envoy-gateway
  name: envoy-gateway
  namespace: envoy-gateway-system
  resourceVersion: "698"
  uid: 227a0bfb-0885-464a-bee0-15a787a711b1
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      control-plane: envoy-gateway
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: envoy-gateway
      creationTimestamp: null
      labels:
        control-plane: envoy-gateway
    spec:
      containers:
      - args:
        - server
        - --config-path=/config/envoy-gateway.yaml
        env:
        - name: ENVOY_GATEWAY_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: docker.io/arkodg/gateway-dev:status-infra-pr
        imagePullPolicy: Always
        name: envoy-gateway
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config
          name: envoy-gateway-config
          readOnly: true
      - args:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
        - --v=0
        image: gcr.io/kubebuilder/kube-rbac-proxy:v0.11.0
        imagePullPolicy: IfNotPresent
        name: kube-rbac-proxy
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 5m
            memory: 64Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
      serviceAccount: envoy-gateway
      serviceAccountName: envoy-gateway
      terminationGracePeriodSeconds: 10
      volumes:
      - configMap:
          defaultMode: 420
          name: envoy-gateway-config
        name: envoy-gateway-config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-09-22T16:50:15Z"
    lastUpdateTime: "2022-09-22T16:50:15Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-09-22T16:49:51Z"
    lastUpdateTime: "2022-09-22T16:50:44Z"
    message: ReplicaSet "envoy-gateway-78cfc9dd4c" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

@arkodg arkodg requested a review from danehans September 22, 2022 16:56
@danehans danehans mentioned this pull request Sep 22, 2022
Closed
@arkodg arkodg merged commit 3c6dc8d into envoyproxy:main Sep 22, 2022
@arkodg arkodg deleted the status-infra branch September 22, 2022 18:27
@danehans danehans mentioned this pull request Sep 22, 2022
Closed
@arkodg arkodg mentioned this pull request Sep 23, 2022
Closed
@danehans danehans mentioned this pull request Sep 23, 2022
Closed
@chauhanshubham chauhanshubham mentioned this pull request Sep 24, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@skriss skriss Awaiting requested review from skriss

@LukeShu LukeShu Awaiting requested review from LukeShu

1 more reviewer

@danehans danehans danehans approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/message-service Issues related to Gateway's message service used for communication among components. area/translator Issues related to Gateway's translation service, e.g. translating Gateway APIs into the IR. kind/enhancement New feature or request provider/kubernetes Issues related to the Kubernetes provider

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@arkodg @codecov-commenter @danehans