feat: exponential backoff for DigitalOcean API calls by dunglas · Pull Request #8205 · cert-manager/cert-manager

dunglas · 2025-10-24T14:53:23Z

Pull Request Motivation

Currently, if cert-manager hits the rate limit of the DigitalOcean API, it falls into an infinite retry loop and sends a lot of requests to the API, increasing the problem.

For instance, this happens in the scenario described in #3848 (comment).

This patch enables the exponential backoff feature of the DigitalOcean client to mitigate the issue.

Kind

/kind feature

Release Note

Added exponential backoff when encountering 429 or 500 errors when requesting the DigitalOcean API.

cert-manager-prow · 2025-10-24T14:53:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sgtcodfish for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cert-manager-prow · 2025-10-24T14:53:34Z

Hi @dunglas. Thanks for your PR.

I'm waiting for a cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

erikgb · 2025-10-24T15:34:26Z

/ok-to-test

Copilot

Pull Request Overview

This PR adds exponential backoff retry logic to DigitalOcean API calls to prevent infinite retry loops when rate limits are hit. The change configures the DigitalOcean client to automatically retry failed requests (429/500 errors) with exponential backoff, limiting retries to a maximum of 3 attempts.

Key changes:

Enabled exponential backoff for DigitalOcean API client with a retry limit of 3 attempts
Refactored client initialization to use inline option syntax for better readability

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

erikgb · 2025-10-24T15:46:04Z

Thanks for this, @dunglas! The change looks good to me, but I would love to get a second opinion from @inteon. Have you been able to test this end-to-end with DigitalOcean? If this fixes the issue, we should consider back-porting the fix. WDYT @inteon?

/cc @inteon

pkg/issuer/acme/dns/digitalocean/digitalocean.go

dunglas · 2025-10-24T17:00:32Z

I didn't e2e-tested this but I know how to write a test doing this. I'll add it.

dunglas · 2025-10-27T14:08:47Z

E2E test added.

dunglas · 2025-10-27T14:17:36Z

/retest

Copilot

Pull Request Overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-27T18:31:51Z

pkg/issuer/acme/dns/digitalocean/digitalocean.go

-	client, err := godo.New(c, clientOpts...)
+	client, err := godo.New(c,
+		godo.SetUserAgent(userAgent),
+		godo.WithRetryAndBackoffs(godo.RetryConfig{RetryMax: math.MaxInt}),


Setting RetryMax to math.MaxInt creates an effectively infinite retry loop, which contradicts the PR's goal of preventing infinite retries. Consider using a reasonable finite value like 10-20 retries to avoid prolonged blocking operations.

Suggested change

godo.WithRetryAndBackoffs(godo.RetryConfig{RetryMax: math.MaxInt}),

godo.WithRetryAndBackoffs(godo.RetryConfig{RetryMax: 10}),

I tend to agree with Copilot here. What is the benefit of having an infinite retry loop?

The infinite retry loop is already there because cert-manager automatically requeue failed requests.

The main benefit of looping here is to wait before retrying. The native requeue mechanism of cert-manager doesn't.

Without this, the backoff isn't exponential anymore.

Having an infinite loop here would block the reconcile, which in turn would prevent other resources being reconciled. This could mean a network failure or upstream issue would cause cert-manager to hang for all ACME certs, even for other providers.

The main benefit of looping here is to wait before retrying. The native requeue mechanism of cert-manager doesn't.

As I understand it cert-manager follows the standard pattern of having a backoff on reconcile attempts, which should be exponential?

IMO blocking calls in reconcile methods is bad practice, it should be written so failures bubble up and retries are handled by requeuing with a backoff

Indeed, you're right. I was double-checking the code and figured out that it was in the main loop. I updated the code to only try 10 times.

After further digging, the main issue seems to be that the current exponential backoff feature doesn't work for some reason.

It should be 5 seconds to 30 minutes:

cert-manager/pkg/controller/util.go

Line 52 in 1db7f8d

return workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](time.Second*5, time.Minute*30)

But this is definitely not the case. According to the issue we had, failing calls to Present() are re-queued and processed immediately, which increases the issue because the DigitalOcean API is then spammed with thousands of requests, triggering 429 errors for all services (such as external-dns) using it.

I have not dug into this - but one reason the backoff may not be respected is object updates would trigger a reconcile always, so if we update the status with the error every time, it would trigger a reconcile every time

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr>

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr> # Conflicts: # pkg/issuer/acme/dns/digitalocean/digitalocean_test.go

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr>

dunglas · 2025-11-04T07:43:00Z

Should have been fixed by #8221.

cert-manager-prow bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 24, 2025

cert-manager-prow bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Oct 24, 2025

erikgb requested a review from Copilot October 24, 2025 15:34

cert-manager-prow bot added ok-to-test and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 24, 2025

Copilot AI reviewed Oct 24, 2025

View reviewed changes

cert-manager-prow bot requested a review from inteon October 24, 2025 15:46

dunglas commented Oct 24, 2025

View reviewed changes

pkg/issuer/acme/dns/digitalocean/digitalocean.go Outdated Show resolved Hide resolved

dunglas mentioned this pull request Oct 27, 2025

tests: remove empty test #8210

Merged

dunglas force-pushed the feat/do-retry branch from 98af1cb to 4855c23 Compare October 27, 2025 14:05

cert-manager-prow bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. and removed dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. labels Oct 27, 2025

erikgb requested a review from Copilot October 27, 2025 18:31

Copilot AI reviewed Oct 27, 2025

View reviewed changes

cert-manager-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2025

dunglas added 4 commits October 28, 2025 11:31

feat: exponential backoff for DigitalOcean API calls

d6eb0fa

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr>

refactor: pass main context in digitalocean NewDNSProvider

dd296ba

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr>

add test

c080691

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr> # Conflicts: # pkg/issuer/acme/dns/digitalocean/digitalocean_test.go

use math.MaxInt

be073ab

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr>

dunglas force-pushed the feat/do-retry branch from 1bb47b8 to be073ab Compare October 28, 2025 10:32

code review

500fa6c

Signed-off-by: Kévin Dunglas <kevin@dunglas.fr>

dunglas force-pushed the feat/do-retry branch from 6bbd265 to 500fa6c Compare October 28, 2025 11:03

cert-manager-prow bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. and removed dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. labels Oct 28, 2025

wallrj-cyberark mentioned this pull request Oct 31, 2025

Fix unregulated retries with the DigitalOcean, Azure DNS, and AWS Route53 DNS-01 solver #8221

Merged

dunglas closed this Nov 4, 2025

dunglas mentioned this pull request Nov 4, 2025

chore: don't create a useless context for the DigitalOcean HTTP client #8226

Closed

	godo.WithRetryAndBackoffs(godo.RetryConfig{RetryMax: math.MaxInt}),
	godo.WithRetryAndBackoffs(godo.RetryConfig{RetryMax: 10}),

Conversation

dunglas commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Motivation

Kind

Release Note

Uh oh!

cert-manager-prow bot commented Oct 24, 2025

Uh oh!

cert-manager-prow bot commented Oct 24, 2025

Uh oh!

erikgb commented Oct 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

erikgb commented Oct 24, 2025

Uh oh!

Uh oh!

dunglas commented Oct 24, 2025

Uh oh!

dunglas commented Oct 27, 2025

Uh oh!

dunglas commented Oct 27, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

erikgb Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

dunglas Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

ThatsMrTalbot Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

dunglas Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThatsMrTalbot Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

dunglas commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dunglas commented Oct 24, 2025 •

edited

Loading

dunglas Oct 28, 2025 •

edited

Loading