cilium-operator fails to start when trying to update MultiCluster Services CRDs #44210
Description
Is there an existing issue for this?
- I have searched the existing issues
Version
equal or higher than v1.19.0 and lower than v1.20.0
What happened?
After upgrading to Cilium 1.19.0 via helm the cilium-operator fails to start with
time=2026-02-05T14:50:57.355993168Z level=error msg="Failed to start when elected leader, shutting down" module=operator error="unable to create CRDs: Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"serviceimports.multicluster.x-k8s.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"
If I explicitly set clustermesh.mcsapi.installCRDs to false cilium-operator starts again
How can we reproduce the issue?
Install cilium with helm and have clustermesh + mcs enabled
clustermesh:
mcsapi:
enabled: true
installCRDs: true # default is trueCilium Version
1.19.0
Kernel Version
6.18.6
Kubernetes Version
v1.35.0
Regression
No response
Sysdump
No response
Relevant log output
...
time=2026-02-05T14:50:51.595255277Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumidentities.cilium.io
time=2026-02-05T14:50:52.104406543Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumpodippools.cilium.io
time=2026-02-05T14:50:52.613791079Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumendpoints.cilium.io
time=2026-02-05T14:50:53.127073193Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumnodes.cilium.io
time=2026-02-05T14:50:53.711253539Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumnetworkpolicies.cilium.io
time=2026-02-05T14:50:54.286127327Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumclusterwidenetworkpolicies.cilium.io
time=2026-02-05T14:50:54.79450492Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumcidrgroups.cilium.io
time=2026-02-05T14:50:55.305924094Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumegressgatewaypolicies.cilium.io
time=2026-02-05T14:50:55.816922848Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumloadbalancerippools.cilium.io
time=2026-02-05T14:50:56.327411845Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliuml2announcementpolicies.cilium.io
time=2026-02-05T14:50:56.837617586Z level=info msg="CRD (CustomResourceDefinition) is installed and up-to-date" module=operator.operator-controlplane.leader-lifecycle.create-crds name=ciliumnodeconfigs.cilium.io
time=2026-02-05T14:50:56.844929127Z level=info msg="Updating CRD (CustomResourceDefinition)..." module=operator.operator-controlplane.leader-lifecycle.create-crds name=serviceimports.multicluster.x-k8s.io
time=2026-02-05T14:50:57.355897965Z level=error msg="Unable to update CRD" module=operator.operator-controlplane.leader-lifecycle.create-crds name=serviceimports.multicluster.x-k8s.io error="customresourcedefinitions.apiextensions.k8s.io \"serviceimports.multicluster.x-k8s.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"
time=2026-02-05T14:50:57.35596811Z level=error msg="Start hook failed" module=operator function="apis.createCRDs.func1 (.../k8s/apis/cell.go:64)" error="unable to create CRDs: Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"serviceimports.multicluster.x-k8s.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"
time=2026-02-05T14:50:57.355993168Z level=error msg="Failed to start when elected leader, shutting down" module=operator error="unable to create CRDs: Unable to create custom resource definition: customresourcedefinitions.apiextensions.k8s.io \"serviceimports.multicluster.x-k8s.io\" is forbidden: User \"system:serviceaccount:kube-system:cilium-operator\" cannot update resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"
...
time=2026-02-05T14:50:57.393731154Z level=fatal msg="Leader election lost, shutting down." module=operator operatorID=infrastructure-test-workers-cpx42-7sfh5-f2mbv-p7fgmmgbn5Anything else?
I think the problem is that when the installCRDs option was added (#40729), the appropriate clusterrole permissions for serviceimport/serviceexport weren't added to the helm chart here:
Cilium Users Document
- Are you a user of Cilium? Please add yourself to the Users doc
Code of Conduct
- I agree to follow this project's Code of Conduct