Skip to content

kstatus watcher reports Deployment as InProgress when all Current conditions are met #31824

@mapleeit

Description

@mapleeit

What happened?

When running helm upgrade with --rollback-on-failure (which implies the kstatus watcher wait), the kstatus watcher gets stuck reporting a Deployment as InProgress even though all conditions for Current status are fully satisfied.

The debug log shows:

level=DEBUG msg="waiting for resource" namespace=default name=access-router kind=Deployment expectedStatus=Current actualStatus=InProgress

This repeats indefinitely (for 5-10 minutes) until eventually resolving. During the stuck period, the Deployment status is verified to be completely healthy via kubectl.

Using --wait=legacy resolves the issue entirely — the upgrade completes immediately without any delay.

What did you expect to happen?

The kstatus watcher should detect that the Deployment has reached Current status and proceed without delay, consistent with the behavior of --wait=legacy.

How can we reproduce it (as minimally and precisely as possible)?

Run helm upgrade on a chart containing a Deployment with 2 replicas and a rollingUpdate strategy:

helm upgrade <release> <chart> \
    -n default \
    -f values.yaml \
    --rollback-on-failure \
    --wait-for-jobs \
    --debug \
    --timeout 30m

The affected Deployment spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: access-router
  annotations:
    reloader.stakater.com/auto: "true"  # Reloader is running but does NOT modify the pod template during this upgrade
spec:
  replicas: 2
  strategy:
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 50%
    type: RollingUpdate
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: access-router
        readinessProbe:
          httpGet:
            path: /_readinessProbe
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 15

Evidence collected during the stuck period

While the kstatus watcher was reporting actualStatus=InProgress, the following was observed:

1. Deployment status — all conditions for Current are met:

{
  "generation": 4,
  "observedGeneration": 4,
  "replicas": 2,
  "updatedReplicas": 2,
  "readyReplicas": 2,
  "availableReplicas": 2
}

2. Conditions — both Available and Progressing are healthy:

[
  {
    "type": "Available",
    "status": "True",
    "reason": "MinimumReplicasAvailable",
    "message": "Deployment has minimum availability."
  },
  {
    "type": "Progressing",
    "status": "True",
    "reason": "NewReplicaSetAvailable",
    "message": "ReplicaSet \"access-router-589c46f897\" has successfully progressed."
  }
]

3. ReplicaSets — clean, only one active RS with all replicas ready:

NAME                       DESIRED   CURRENT   READY   AGE
access-router-54b56999d6   0         0         0       174m
access-router-589c46f897   2         2         2       46m
access-router-65f558dbdd   0         0         0       3h11m

4. Pod template annotations — empty (Reloader did NOT trigger a second rollout):

(empty)

5. Ruled out causes:

  • No HPA (kubectl get hpa -n default → no resources)
  • No PDB (kubectl get pdb → no resources)
  • minReadySeconds is not set (defaults to 0)
  • progressDeadlineSeconds is 600 (default)
  • Stakater Reloader is running but did NOT modify the Deployment during the upgrade
  • No other controllers modifying the Deployment

Summary of kstatus Current criteria check

Criteria Value Satisfied?
observedGeneration == generation 4 == 4
updatedReplicas == spec.replicas 2 == 2
readyReplicas == spec.replicas 2 == 2
availableReplicas == spec.replicas 2 == 2
status.replicas <= updatedReplicas 2 <= 2
No ProgressDeadlineExceeded True

All conditions are met, yet the watcher reports InProgress.

Helm version

Details
$ helm version
version.BuildInfo{Version:"v4.1.1", GitCommit:"5caf004", GitTreeState:"clean", GoVersion:"go1.25.5", KubeClientVersion:"v1.34"}

Also reproduced on v4.0.4.

Kubernetes version

Details
$ kubectl version
Client Version: v1.34.3
Kustomize Version: v5.7.1
Server Version: v1.32.2-tke.10

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugCategorizes issue or PR as related to a bug.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions