Realtime Compositions: cannot list in CompositionRevision handler #5533
Description
What happened?
Using the flag --enable-realtime-compositions causes crossplane to stop working. The pod won't crash but it's just not reacting to creating/editing claims. Also, there are error logs that aren't present if the flag is omitted.
$ kubectl logs -n crossplane-system -f crossplane-5fddf49df6-grt4c
{"level":"info","ts":"2024-04-01T22:04:35Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaCompositionFunctions"}
{"level":"info","ts":"2024-04-01T22:04:35Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaCompositionFunctionsExtraResources"}
{"level":"info","ts":"2024-04-01T22:04:35Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaCompositionWebhookSchemaValidation"}
{"level":"info","ts":"2024-04-01T22:04:35Z","logger":"crossplane","msg":"Alpha feature enabled","flag":"EnableAlphaRealtimeCompositions"}
{"level":"info","ts":"2024-04-01T22:04:35Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaDeploymentRuntimeConfigs"}
I0401 22:04:37.388729 1 leaderelection.go:250] attempting to acquire leader lease crossplane-system/crossplane-leader-election-core...
I0401 22:04:41.638798 1 leaderelection.go:260] successfully acquired lease crossplane-system/crossplane-leader-election-core
Warning: ControllerConfig.pkg.crossplane.io/v1alpha1 is deprecated. Use DeploymentRuntimeConfig from pkg.crossplane.io/v1beta1 instead.
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
W0401 22:04:42.400546 1 reflector.go:462] k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: watch of somewhere.cloud/v1, Kind=MyStorage ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}
{"level":"info","ts":"2024-04-01T22:04:42Z","logger":"crossplane","msg":"cannot list in CompositionRevision handler","controller":"defined/compositeresourcedefinition.apiextensions.crossplane.io","type":"somewhere.cloud/v1, Kind=MyStorage","error":"Timeout: failed waiting for *unstructured.Unstructured Informer to sync"}The relevant code hints at a programming mistake in the crossplane code:
crossplane/internal/controller/apiextensions/composite/reconciler.go
Lines 752 to 753 in d35ecef
To me it looks a bit suspicious, that the logs mention a timeout but the timestamp of the logs are all the same. Maybe this timeout needs to be configured differently?
How can we reproduce it?
- Install crossplane using helm with:
# values.yaml
args:
- '--enable-realtime-compositions'- Create the following CompositeResourceDefinition:
CompositeResourceDefinition: MyStorage
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: mystorages.somewhere.cloud
namespace: crossplane-system
spec:
group: somewhere.cloud
names:
kind: MyStorage
plural: mystorages
versions:
- name: v1
served: true
referenceable: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
diskName:
type: string
status:
type: object
properties:
bucketUrl:
type: string
websiteUrl:
type: string
userEmail:
type: stringI could imagine that this is an issue that may is more specific. The only information I really provide here is my CompositeResourceDefinition. Please tell me what information you need to find the root cause of this issue. I am willing to share more.
What environment did it happen in?
- Crossplane version: 1.15.1
- repoURL: https://charts.crossplane.io/stable
- chart: crossplane
- Kubernetes version: v1.29.2 (with kind v0.22.0)
- MacOS 14.2.1
- Darwin Kernel Version 23.2.0