Skip to content

AWS Session cache appears to be incompatible with secrets caching #1587

@bobbywatson3

Description

@bobbywatson3

With the following values set on v0.6.0-rc1, a fatal error concurrent map writes crashes the pod:

replicaCount: 3
leaderElect: true
extraArgs:
  enable-secrets-caching: true
  client-qps: 100
  client-burst: 200
  experimental-enable-aws-session-cache: true
resources:
  requests:
    cpu: 200m
    memory: 768M
  limits:
    memory: 768M
concurrent: 1000
prometheus:
  enabled: true

Error seen:

{"level":"info","ts":1664479272.460663,"logger":"provider.aws","msg":"reusing aws session","SecretStore":"secret-store","namespace":"beamy-bike","kind":"SecretStore","resourceversion":"2135774023"}
{"level":"error","ts":1664479272.4606748,"logger":"controllers.ExternalSecret","msg":"could not get existing secret: %w","ExternalSecret":"secrets","SecretStore":"secret-store","error":"Timeout: failed waiting for *v1.Secret Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234"}
external-secrets fatal error: concurrent map writes

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/awsIndicates an issue or PR related to AWS.kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions