Description of your changes

Fixes #5338
Fixes #2645
Closes #5463
Closes #5468
Closes #5422

It's possible this will fix #5400, #5151, #5533, and #5228 given that it makes large changes to how realtime compositions work. I'd rather not assume though - we can see if folks can reproduce if/when this PR is merged.

Crossplane uses a "controller engine" to dynamically start claim and XR controllers when a new XRD is installed, and stop them when the XRD is uninstalled.

Before this PR, each controller gets at least one cache. This is because when I built the controller engine, you couldn't stop a single informer within a cache (a cache is basically a map of informers by GVK). Instead, we stop entire caches.

When realtime composition is enabled, there are even more caches. One per composed resource GVK. A GVK routed cache routes cache lookups to these various delegate caches.

Meanwhile, controller-runtime recently made it possible to stop an informer within a cache. It's also been possible to remove an event handler from an informer for some time (since Kubernetes 1.26):

• 🌱 Proposal for dynamic informer cache kubernetes-sigs/controller-runtime#2285
• ⚠ Support registration and removal of event handler kubernetes-sigs/controller-runtime#2046

This PR uses a single client, backed by a single cache, across all dynamic controllers (specifically the definition, offered, claim, and XR controllers). Each controller tracks what watches ("sources" in controller-runtime) it has active and stops them when it doesn't need them anymore. The controller engine also garbage collects custom resource informers when their CRDs are deleted.

Compared to the current implementation, this:

• Takes fewer global locks when realtime compositions are enabled. Locking is now mostly at the controller scope.
• Works with the breaking changes to source.Source introduced in controller-runtime v0.18. :)

I also think this makes the realtime composition code a little easier to follow by consolodating it into the ControllerEngine (and into one repo), but that's pretty subjective.

Here's a questionable Mermaid diagram of how it works:

graph TD
	subgraph ITC[engine.InformerTrackingCache]
		I1[XFoo Informer]
		I2[XBar Informer]
		I3[ComposedResource Informer]
	end
	
	subgraph CE[engine.ControllerEngine]
		subgraph C1[XFoo Controller]
			R1[XFoo Reconciler] -- Get --> I1
			R1 -- Get --> I3
			S1[XFoo Source] -- Watch --> I1
			S1 -- Enqueue --> R1
			SC1[ComposedResource Source] -- Watch --> I3
			SC1 -- Enqueue --> R1
		end

		subgraph C2[XBar Controller]
			R2[XBar Reconciler] -- Get --> I2
			R2 -- Get --> I2
			S2[XFoo Source] -- Watch --> I2
			S2 -- Enqueue --> R2
			SC2[ComposedResource Source] -- Watch --> I3
			SC2 -- Enqueue --> R2
		end
	end

	subgraph C[cache.Cache]
	  I4[XRD Informer]
	end

	subgraph M[manager.Manager]
		subgraph C3[XRD Controller]
			R3[XRD Reconciler] -- Start --> C1
			R3 -- Start --> C2
			R3 -- Get --> I4
			S3[XRD Source] -- Watch --> I4
			S3 -- Enqueue --> R3
		end
	end

It's conceptually pretty similar to a typical set of controller-runtime controllers run by a controller.Manager, but with the ability to stop watches and informers.

Reviewer Note

This PR depends on crossplane/crossplane-runtime#689.

Unfortunately, this required bumping a bunch of dependencies, mostly to get the latest controller-runtime. Those dependency bumps in turn required a bunch of other little fixes (CRD generation issues, new linter warnings, etc).

I also opted to move the ControllerEngine implementation from c/cr to c/c. I think that will make changes like this easier in future.

While this all makes this PR pretty huge, it's all broken out into in separate commits. The last commit in the PR contains the bulk of the changes - reviewers can focus on that.

I'm opening this as draft since it's a big change. I still need to add more tests for the new code (e.g. the new engine implementation).

I have:

• Read and followed Crossplane's contribution process.
• Run make reviewable to ensure this PR is ready for review.
• Added or updated unit tests.
• Added or updated e2e tests.
• Linked a PR or a docs tracking issue to document this change.
• Added backport release-x.y labels to auto-backport this PR.

Need help with this checklist? See the cheat sheet.