-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Investigate improving resource consumption and performance in clusters with large amount of resources #5220
Description
cert-manager currently watches all Secrets (and Pods?) in a cluster.
There are a number of issues associated with that, mostly due to memory consumption.
A couple alternative ways how this could be solved have been discussed:
-
filter core resources (
Secrets) to be watched by a label. This would require getting to a state where all core resources that cert-manager needs to watch are labelled, this includes resources that already exist in a cluster as well as resources that users have created (i.eSecrets with issuer credentials) -
cache metadata only. This would require a separate call to retrieve the cached resources when they are to be used if the resource spec is needed. Question: would this help in cases where listing a large number of secrets if causing an issue?
This would probably also require rewriting cert-manager controllers as controller runtime controllers. -
allow users to apply a filter via a flag to filter out some secrets that should not be watched, see Add support for restricting the secrets watch list in cainjector #5174. All cases mentioned the large secrets are those of helm releases, but perhaps there are cases out there where users other large secrets in cluster and it's not feasible to specifiy a selector via a flag?
Additionally:
- ensure that for namespace-scoped deployments only secrets in the specific namespaces are watched (needs more investigation)
- is there a similar issue with watching pods (needs more investigation)?
Related issues/PRs:
- High memory usage on cluster with many secrets #4722
- Cert-manager causes API server panic on clusters with more than 20000 secrets. #3748
- Newly elected cert-manager leader replica fails to issue pending CertificateRequest for a CA Issuer when the Kubernetes clusters has a lot of Secrets objects #5216
- cert-manager crash looping in AKS #5195
/kind feature