Skip to content

Detect goroutine leak on GCP Workload Identity reconciliation #1901

@tapih

Description

@tapih

Describe the solution you'd like

I use SecretStore with GCP Workload Identity and found that the memory (& CPU) usage of the controller increased steadily.

Memory
image

The metrics go_goroutine grew at the same time, so the controller seems to leak goroutine on GCP Workload Identity reconciliation.

$ kubectl port-forward -n external-secrets $(kubectl get po -n external-secrets -l app.kubernetes.io/name=external-secrets -o name) 8080:8080 &
$ curl -sSLf localhost:8080/metrics | grep "^" | grep goroutines
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 5706

$ cat << 'EOF' | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  data:
  - remoteRef:
      conversionStrategy: Default
      decodingStrategy: None
      key: example
      version: latest
    secretKey: VALUE
  refreshInterval: "1h"
  secretStoreRef:
    kind: ClusterSecretStore
    name: example
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: example
EOF
externalsecret.external-secrets.io/example created

$ curl -sSLf localhost:8080/metrics | grep "^" | grep goroutines
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 5721

What is the added value?

Bug fix

Observations (Constraints, Context, etc):

GKE: 1.24.7-gke.900
ESO: v0.7.1
Go: 1.19.4

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions