-
Notifications
You must be signed in to change notification settings - Fork 7.5k
Description
What happened?
When upgrading a Helm release where a custom resource changes API version (e.g., from v2beta1 to v2beta2 of the same CRD), Helm 4 incorrectly treats the old and new versions as different resources. This causes it to delete the old version (which is the same underlying object in etcd) after creating the new version, effectively deleting the resource that was just created.
The wait then times out because the resource no longer exists:
timeout waiting for: [CrdUpgradeTestb/default/b status: 'NotFound']
Root Cause
In #12581, isMatchingInfo in pkg/kube/resource.go was changed from comparing Kind + Group (ignoring Version) to comparing the full GroupVersionKind struct (including Version):
// Before (Helm 3.x):
func isMatchingInfo(a, b *resource.Info) bool {
return a.Name == b.Name && a.Namespace == b.Namespace &&
a.Mapping.GroupVersionKind.Kind == b.Mapping.GroupVersionKind.Kind &&
a.Mapping.GroupVersionKind.Group == b.Mapping.GroupVersionKind.Group
}
// After (Helm 4):
func isMatchingInfo(a, b *resource.Info) bool {
return a.Name == b.Name && a.Namespace == b.Namespace &&
a.Mapping.GroupVersionKind == b.Mapping.GroupVersionKind
}This means that when a resource changes API version (e.g., crd-upgrades.example.com/v2beta1 → crd-upgrades.example.com/v2beta2), Difference() considers them as non-matching. Helm's update() method then:
- Creates/updates the new version (
v2beta2) of the resource - Computes
originals.Difference(targets)— the old version (v2beta1) is in the difference set because it doesn't match the new version - Deletes the old version — but since both versions share the same underlying storage in the Kubernetes API server, this deletes the resource that was just created in step 1
- The resource is now gone, and the wait times out with
NotFound
Impact
This affects any chart upgrade where:
- A CRD serves multiple API versions
- Custom resources change their
apiVersionbetween releases (e.g., migrating from a beta to a stable version)
This is a regression from Helm 3, where isMatchingInfo intentionally ignored the Version component.
Reproduction
- Install a chart with a CRD that serves
v2beta1and a custom resource usingapiVersion: .../v2beta1 - Upgrade the chart where the custom resource now uses
apiVersion: .../v2beta2(CRD serves both versions) - The upgrade deletes the resource and then times out waiting for it
See: fluxcd/helm-controller CRDs upgrade e2e tests for a concrete example.
Workaround
None known.
Suggested Fix
isMatchingInfo should compare Group + Kind (without Version), as it did in Helm 3.x. Resources served by the same CRD at different API versions are the same underlying object in etcd and should not be treated as different resources.
Helm Version
v4.1.0
Kubernetes Version
v1.35.0