Describe the bug:
On a cluster with over 30k secrets, the cert-manager controller requires over 8GB of ram to start, takes over a minute to cache all secrets:

I0110 15:36:07.189947       1 trace.go:205] Trace[152527088]: "Reflector ListAndWatch" name:external/io_k8s_client_go/tools/cache/reflector.go:167 (10-Jan-2022 15:34:56.350) (total time: 70839ms):
Trace[152527088]: ---"Objects listed" 70789ms (15:36:07.140)
Trace[152527088]: [1m10.839343767s] [1m10.839343767s] END

Expected behaviour:

I would not have expected cert-manager to use this much memory for only 4 certificates.
Steps to reproduce the bug:

Anything else we need to know?:
This was initially discussed over slack:
https://kubernetes.slack.com/archives/C4NV3DWUC/p1641830882034900

Environment details::

• Kubernetes version: v1.20.11
• Cloud-provider/provisioner: aws
• cert-manager version: 1.6.1
• Install method: static manifests

/kind bug