What happened?

While investigating Crossplane performance in #2568 I noticed that the XR and claim controllers were making a surprising amount of calls to the API server.

The above graph is from a test in which I created 1,000 claims. Each claim created an XR composed of 4 managed resources, and exercised most Crossplane features (connection secrets, bidirectional patching, etc).

Note that Crossplane core (crossplane-system/crossplane) is making 4x the API server calls of provider-gcp (crossplane-system/crossplane-providers). I believe this is because:

1. The claim and XR controllers aren't using a read cache. We're passing in the default controller-runtime Kubernetes client from mgr.GetClient(), but these controllers use *unstructured.Unstructured heavily, which is not cached by default. This means we're getting claims, XRs, and composed resources directly from the API server on each reconcile by default.
2. The XR controller is writing two events - SelectedComposition and ComposedResources - on every reconcile. This controller polls (requeues) every minute by default, so we're writing two events per XR per minute even when nothing is really being done (e.g. when the select and composes are no-ops).

How can we reproduce it?

You can find the test config at https://github.com/negz/crossplane-scale. I used a GKE cluster with two e2-medium-4 notes, and Crossplane and the GCP provider both had --max-reconcile-rate=100, with 2 CPU cores and 2GB of memory.

What environment did it happen in?

Crossplane version: