Summary

Closes #277.

The MinAvailableBreached condition was suppressed whenever scheduledReplicas < MinAvailable. That suppression conflated two situations:

1. Initial startup (correct to suppress) — no pods have been admitted yet, gang termination would be premature.
2. Regression after a healthy state (incorrectly suppressed) — e.g. a node hosting running pods is cordoned or fails and the replacement pod cannot be scheduled. scheduledReplicas falls below MinAvailable, but the breach never fires, so gangterminate.go never deletes the PCS replica's PodCliques, the PodGang is never recreated, and the workload is stuck. This is exactly the gap the new E2E gang-termination tests were exposing.

Idea behind the fix

Split the under-scheduled branch on scheduledReplicas == 0 vs > 0:

if scheduledReplicas < minAvailable {
    if scheduledReplicas == 0 {
        // initial startup OR full regression — suppress (see below)
        return ConditionFalse, InsufficientScheduledPods
    }
    // 0 < scheduled < MinAvailable — must be regression, breach
    return ConditionTrue, ScheduledReplicasRegressed
}

Why 0 < scheduled < MinAvailable is unambiguously a regression. Under gang scheduling the gang scheduler admits MinAvailable pods atomically — a partial-scheduled state is structurally unreachable from a fresh start. The only way to land there is to have been healthy and then lost pods. So we can breach immediately on observing the partial state.

Why we keep scheduled == 0 suppressed. With nothing scheduled, gang termination has no useful work: recreating the PodGang produces the same Pending pods against the same cluster state. Suppressing avoids a churn loop. This matches KAI's behaviour: in general, if reaching MinAvailable legitimately takes longer than TerminationDelay, gang-terminating-and-recreating is the right move, and KAI's gang scheduler arrives at the same conclusion.

Pod scheduling continues throughout. The breach condition is consumed only by gangterminate.go, which still waits the full TerminationDelay (default 4h) before deleting. The pod-creation/replacement path in podclique/components/pod/syncflow.go and the PCLQ-creation path in podcliquescalinggroup/components/podclique/sync.go do not gate on the breach — replacement pods keep being created and re-scheduled while the timer runs. If they succeed, scheduledReplicas recovers, the condition flips back to False, and gang termination is never triggered.

Behaviour on non-gang schedulers

On schedulers without gang admission (e.g. vanilla kube-scheduler), pods are admitted one at a time, so a brief 0 < scheduled < MinAvailable flicker can occur during staged startup. The flicker is debounced by TerminationDelay (default 4h) — only pathological multi-hour rampups would see spurious gang termination, and in that case the existing Spec.Template.TerminationDelay knob is the remedy.

Test plan

• go test ./... passes (operator module)
• go vet ./... clean, go build ./... clean
• Reproducer tests in reconcilestatus_test.go for both PodClique and PodCliqueScalingGroup capture:
  • 0 < scheduled < MinAvailable ⟹ breach (the bug fix)
  • scheduled == 0 after regression ⟹ no breach (sanity-pin against over-correction / churn loop)
  • Fresh PCLQ/PCSG never scheduled ⟹ no breach (initial-startup sanity-pin)
• Updated existing TestComputeMinAvailableBreachedCondition / insufficient_scheduled PCSG test, which had previously asserted the buggy behaviour, to assert the post-fix behaviour and added a paired scheduled == 0 case.
• E2E gang termination tests (referenced in the issue) should now pass.

Open question for reviewers

The scheduled == 0 suppression is a deliberate judgement call: it avoids a churn loop, but it does mean a fully-down PCS replica won't be gang-terminated (the controller keeps trying to schedule replacements forever). If there is a scenario where the PodGang resource itself gets into a bad shape that recreation would fix, we may want to revisit. Flagging for @renormalize / @unmarshall / @sanjaychatterjee.

🤖 Generated with Claude Code