Skip to content

Namespace finalizer not getting cleaned up causing hanging namespace deleting #5426

@BadLiveware

Description

@BadLiveware

Describe the bug
The namespace finalizer externalsecrets.external-secrets.io/ces-<xyz> is not getting cleaned up on namespace deletion, causing it to hang indefinitely

To Reproduce
Steps to reproduce the behavior:

    - apiVersion: v1
      kind: Secret
      metadata:
        name: &license-name nservicebus-license
        namespace: '{{ .Release.Namespace }}'
      stringData:
        license.xml: xyz

    - apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: &auth-sa nsb-external-secrets
        namespace: '{{ .Release.Namespace }}'
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: read-source-secret
        namespace: '{{ .Release.Namespace }}'
      rules:
      - apiGroups: [""]
        resourceNames:
        - *license-name
        resources:
        - secrets
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - authorization.k8s.io
        resources:
        - selfsubjectrulesreviews
        verbs:
        - create
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: read-source-secret
        namespace: '{{ .Release.Namespace }}'
      subjects:
      - kind: ServiceAccount
        name: *auth-sa
        namespace: '{{ .Release.Namespace }}'
      roleRef:
        kind: Role
        name: read-source-secret
        apiGroup: rbac.authorization.k8s.io

    - apiVersion: external-secrets.io/v1
      kind: ClusterSecretStore
      metadata:
        name: &store nsb
      spec:
        provider:
          kubernetes:
            remoteNamespace: '{{ .Release.Namespace }}'
            server:
              url: "https://kubernetes.default"
              caProvider:
                type: ConfigMap
                name: kube-root-ca.crt
                key: ca.crt
                namespace: '{{ .Release.Namespace }}'
            auth:
              serviceAccount:
                name: *auth-sa
                namespace: '{{ .Release.Namespace }}'
    - apiVersion: external-secrets.io/v1
      kind: ClusterExternalSecret
      metadata:
        name: "nservicebus-license-replication"
      spec:
        # The name to be used on the ExternalSecrets
        externalSecretName: "nservicebus-license-replicated"
        namespaceSelectors:
        - matchLabels: {}
        refreshTime: "1h" #Cluster External Secret refresh Time
        externalSecretSpec:
          secretStoreRef:
            name: *store
            kind: ClusterSecretStore
          refreshInterval: "10m" #External Secret Refresh Time
          target:
            name: nservicebus-license
            creationPolicy: 'Owner'
          dataFrom:
            - extract: 
                key: *license-name

This is all deployed with argocd version v3.1.7+511ebd7

ESO chart version: 0.20.2

❯ kubectl version
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.32.6-gke.1060000

Expected behavior
The namespace finalizer to be cleaned up so that the namespace can be deleted

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

> kubectl get-all -n tradera-web-beta-451
No resources found.
> kubectl get ns tradera-web-beta-451 -oyaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: tradera-web-beta-451:/Namespace:tradera-web-beta-451/tradera-web-beta-451
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"tradera-web-beta-451:/Namespace:tradera-web-beta-451/tradera-web-beta-451"},"name":"tradera-web-beta-451"}}
  creationTimestamp: "2025-10-06T08:05:18Z"
  deletionTimestamp: "2025-10-06T08:12:50Z"
  finalizers:
  - externalsecrets.external-secrets.io/ces-nservicebus-license-replication
  labels:
    kubernetes.io/metadata.name: tradera-web-beta-451
  name: tradera-web-beta-451
  resourceVersion: "1759738412488175009"
  uid: e302ab21-4214-4c49-a69e-084b6fff0bc9
spec: {}
status:
  conditions:
  - lastTransitionTime: "2025-10-06T08:13:11Z"
    message: All resources successfully discovered
    reason: ResourcesDiscovered
    status: "False"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2025-10-06T08:13:11Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2025-10-06T08:13:11Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2025-10-06T08:13:32Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2025-10-06T08:13:11Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating

Only relevant log:

{
  "jsonPayload": {
    "controllerKind": "ExternalSecret",
    "level": "error",
    "ExternalSecret": {
      "name": "nservicebus-license-replicated",
      "namespace": "tradera-web-beta-451"
    },
    "ts": 1759738384.4430656,
    "msg": "Reconciler error",
    "controller": "externalsecret",
    "controllerGroup": "external-secrets.io",
    "namespace": "tradera-web-beta-451",
    "error": "secrets \"nservicebus-license\" is forbidden: unable to create new content in namespace tradera-web-beta-451 because it is being terminated",
    "name": "nservicebus-license-replicated",
    "reconcileID": "527dcd6f-0eb3-4847-8d21-2905ba36b75d",
    "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "external-secrets",
      "namespace_name": "external-secrets"
    }
  },
  "timestamp": "2025-10-06T08:13:04.443344807Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/app_kubernetes_io/managed-by": "Helm",
    "logging.gke.io/top_level_controller_name": "external-secrets",
    "k8s-pod/app_kubernetes_io/instance": "external-secrets",
    "k8s-pod/app_kubernetes_io/part-of": "external-secrets",
    "k8s-pod/app_kubernetes_io/version": "v0.20.2",
    "k8s-pod/app_kubernetes_io/name": "external-secrets",
    "logging.gke.io/top_level_controller_type": "Deployment",
    "k8s-pod/pod-template-hash": "7c4d7459d",
    "k8s-pod/helm_sh/chart": "external-secrets-0.20.2"
  },
  "logName": "projects/x/logs/stderr",
  "receiveTimestamp": "2025-10-06T08:13:09.357177246Z"
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions