Skip to content

cert-manager enters infinite re-issuance loop when Issuer returns invalid certificate #8380

@putongyong

Description

@putongyong

Is your feature request related to a problem? Please describe.

Hi cert-manager team, thank you for your great work on this project.

We encountered an issue in a very specific scenario involving our custom Issuer implementation, but we believe there may also be an opportunity to improve cert-manager’s internal behavior in such cases.

Due to a bug in our custom Issuer, in a very rare situation it returned an obsolete certificate, which caused a public key mismatch between the certificate and the CSR.

From our observations on the cert-manager side:

  • cert-manager receives the incorrect certificate from the CertificateRequest
  • it stores the certificate into the Secret
  • it then validates the certificate and detects the key mismatch
  • cert-manager immediately triggers a re-issuance by creating a new CertificateRequest using the same CSR
  • our Issuer (still buggy at that time) returns the same mismatched certificate again

This results in cert-manager entering an infinite loop, continuously creating new CertificateRequests. In our case, this eventually led to millions of CertificateRequest resources being created and ultimately caused the Kubernetes control plane to crash.

We have tested this behavior with cert-manager v1.19.1, and the issue is still present.

Describe the solution you'd like

Although we have fixed the root cause by correcting our custom Issuer, we are wondering if cert-manager itself could be more defensive in this situation.

For example, when cert-manager receives a certificate and attempts to update the Secret, could it:

  • check whether the received certificate is identical to the one already stored in the Secret, and
  • if so, avoid immediately triggering another re-issuance loop, and instead mark the issuance as failed and retry later with a backoff?

This could help prevent runaway loops in case of misbehaving Issuers.

Describe alternatives you've considered

Another possible improvement could be to validate key matching before storing the certificate into the Secret, and fail fast if the certificate does not match the CSR.

Additional context

While browsing existing issues, we noticed some related reports that were labeled as bugs, such as #8234, which seem to touch similar failure modes.

Additionally, we would be happy to join community meetings and contribute to investigating or implementing improvements in this area, as cert-manager is a core component in our infrastructure. Thank you in advance for your time and consideration.

/kind feature

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.kind/featureCategorizes issue or PR as related to a new feature.priority/backlogHigher priority than priority/awaiting-more-evidence.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions