What problem are you facing?

Crossplane installs a lot of CRDs. More specifically, some of the most widely used Crossplane providers install a lot of CRDs. Upbound's Official AWS Provider for example currently installs almost 800 CRDs.

Per the CRD scaling one pager, Kubernetes and the tools in its ecosystem struggle when so many CRDs are installed. The API server uses a lot of memory (around 3MB per CRD). Many tools like kubectl and libraries like client-go are designed under the assumption that only 10s of CRDs will exist, and thus rely on inefficient queries. Crossplane maintainers and others (notably @jonnylangefeld and @apelisse) have made a lot of headway into improving this situation. Unfortunately it has become evident that we can't wait for the ecosystem to catch up to our scale. Despite there being a handful of performance improvements with each new Kubernetes release it may take years for everyone to be running versions of Kubernetes and Kubernetes tooling that handles hundreds or thousands of CRDs well.

In addition to the performance issues that come with loading a lot of CRDs, per #2869, some folks have expressed security concerns about having to deploy APIs and controllers that they don't plan to use. Others have stated they just plain don't like the "bloat" of knowing that there's a bunch of superfluous things installed.

• Add ability to disable certain CRDs on installation. #2869
• Instructions on how to compile a Crossplane Provider with just a subset of the available CRDs docs#336

Per the preceding issues, some folks have started rolling their own Crossplane providers that include only the types they need. These are typically large existing providers (like provider-aws) forked and with the superfluous controllers and types removed.

How could Crossplane help solve your problem?

I propose that we break up large providers by service. That is, instead of provider-aws we would have provider-aws-rds, provider-aws-eks, etc. The goal is to lower the ratio of installed-to-used CRDs, and intuitively doing that by service seems like about the right ratio to me. Thanks to the Crossplane package manager I believe it should be possible (perhaps with a few package manager tweaks) to make for example provider-aws a "meta-package" that pulled in all of the smaller providers for backward compatibility.

I like this approach because:

• It's opt-in, rather than opt-out. You install the things you do need, as opposed to turning off the things you don't need.
• No changes would be required to the many providers that already have a low installed-to-used CRD ratio. Providers like provider-terraform and provider-helm that only have 2-3 CRDs can stay as they are.
• It increases the granularity at which providers can be updated. For example if you had to stick with a certain version of provider-aws-rds you could still update provider-aws-eks to the latest version.
• It will likely increase the scalability of providers, by sharding work across multiple provider processes (i.e. Pods).

Some examples for context. Presuming that each provider mapped to an existing API group (e.g. provider-aws-rds mapped to rds.aws.upbound.io):

• provider-aws-rds would have ~22 CRDs.
• provider-aws-eks would have ~6 CRDs.
• provider-aws-s3 would have ~23 CRDs.
• provider-aws-iam would have ~22 CRDs.