enabling realtime composition causes watch error #5957
Description
What happened?
Upgraded to 0.17.0. I didn't see this issue with 0.16.0
Added --enable-realtime-compositions and the logs fill up with
2024-09-11T22:02:04Z DEBUG crossplane Reconciling {"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/irsaroles.aws.ops.cake.ai", "request": {"name":"aws-load-balancer"}}
2024-09-11T22:02:04Z DEBUG crossplane Watch exists for GVK, not starting a new one {"controller": "composite/irsaroles.aws.ops.cake.ai", "watch-type": "ComposedResource", "watched-gvk": "iam.aws.upbound.io/v1beta1, Kind=Policy"}
2024-09-11T22:02:04Z DEBUG crossplane Watch exists for GVK, not starting a new one {"controller": "composite/irsaroles.aws.ops.cake.ai", "watch-type": "ComposedResource", "watched-gvk": "iam.aws.upbound.io/v1beta1, Kind=RolePolicyAttachment"}
2024-09-11T22:02:04Z DEBUG crossplane Watch exists for GVK, not starting a new one {"controller": "composite/irsaroles.aws.ops.cake.ai", "watch-type": "ComposedResource", "watched-gvk": "iam.aws.upbound.io/v1beta1, Kind=Role"}
2024-09-11T22:02:04Z DEBUG crossplane Composed resource is not yet ready {"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/irsaroles.aws.ops.cake.ai", "request": {"name":"aws-load-balancer"}, "uid": "80af72ae-c827-4101-be6f-b67f6bb61f54", "version": "55906", "name": "aws-load-balancer", "id": "irsaAttachment"}
2024-09-11T22:02:04Z DEBUG crossplane Composed resource is not yet ready {"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/irsaroles.aws.ops.cake.ai", "request": {"name":"aws-load-balancer"}, "uid": "80af72ae-c827-4101-be6f-b67f6bb61f54", "version": "55906", "name": "aws-load-balancer", "id": "irsaRole"}
2024-09-11T22:02:04Z DEBUG crossplane Composed resource is not yet ready {"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "controller": "composite/irsaroles.aws.ops.cake.ai", "request": {"name":"aws-load-balancer"}, "uid": "80af72ae-c827-4101-be6f-b67f6bb61f54", "version": "55906", "name": "aws-load-balancer", "id": "irsaPolicy"}
2024-09-11T22:02:04Z DEBUG crossplane Event(v1.ObjectReference{Kind:"IRSARole", Namespace:"", Name:"aws-load-balancer", UID:"80af72ae-c827-4101-be6f-b67f6bb61f54", APIVersion:"aws.ops.cake.ai/v1alpha1", ResourceVersion:"55906", FieldPath:""}): type: 'Normal' reason: 'ComposeResources' Successfully composed resources
2024-09-11T22:02:04Z DEBUG crossplane Event(v1.ObjectReference{Kind:"IRSARole", Namespace:"", Name:"aws-load-balancer", UID:"80af72ae-c827-4101-be6f-b67f6bb61f54", APIVersion:"aws.ops.cake.ai/v1alpha1", ResourceVersion:"55906", FieldPath:""}): type: 'Normal' reason: 'ComposeResources' Composed resource "irsaAttachment" is not yet ready
2024-09-11T22:02:04Z DEBUG crossplane Event(v1.ObjectReference{Kind:"IRSARole", Namespace:"", Name:"aws-load-balancer", UID:"80af72ae-c827-4101-be6f-b67f6bb61f54", APIVersion:"aws.ops.cake.ai/v1alpha1", ResourceVersion:"55906", FieldPath:""}): type: 'Normal' reason: 'ComposeResources' Composed resource "irsaRole" is not yet ready
2024-09-11T22:02:04Z DEBUG crossplane Event(v1.ObjectReference{Kind:"IRSARole", Namespace:"", Name:"aws-load-balancer", UID:"80af72ae-c827-4101-be6f-b67f6bb61f54", APIVersion:"aws.ops.cake.ai/v1alpha1", ResourceVersion:"55906", FieldPath:""}): type: 'Normal' reason: 'ComposeResources' Composed resource "irsaPolicy" is not yet ready
2024-09-11T22:02:10Z ERROR crossplane pkg/mod/k8s.io/client-go@v0.30.0/tools/cache/reflector.go:232: expected type *composed.Unstructured, but watch event object had type *unstructured.Unstructured
k8s.io/client-go/tools/cache.(*Reflector).watch
/go/pkg/mod/k8s.io/client-go@v0.30.0/tools/cache/reflector.go:445
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
/go/pkg/mod/k8s.io/client-go@v0.30.0/tools/cache/reflector.go:365
k8s.io/client-go/tools/cache.(*Reflector).Run.func1
/go/pkg/mod/k8s.io/client-go@v0.30.0/tools/cache/reflector.go:298
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:226
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/backoff.go:227
k8s.io/client-go/tools/cache.(*Reflector).Run
/go/pkg/mod/k8s.io/client-go@v0.30.0/tools/cache/reflector.go:297
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2
/go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:55
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
/go/pkg/mod/k8s.io/apimachinery@v0.30.0/pkg/util/wait/wait.go:72
I am trying to debug why it takes 20 minutes before my IAM resources are touched. I thought adding realtime-compositions would do something but it seems the aws-iam-provider doesn't actually do anything for the first 10m nor does it watch for events which is a separate issue.
How can we reproduce it?
Not sure. I have my own composition which is just creating a AWS role, policy and policy attachment.
What environment did it happen in?
Crossplane version: 0.17.0
EKS 1.30.2
Metadata
Metadata
Assignees
Labels
Type
Projects
Status