Improve S3 Retry Strategy for backup and restore#84421
Merged
jkartseva merged 11 commits intoClickHouse:masterfrom Aug 9, 2025
Merged
Improve S3 Retry Strategy for backup and restore#84421jkartseva merged 11 commits intoClickHouse:masterfrom
jkartseva merged 11 commits intoClickHouse:masterfrom
Conversation
Contributor
pufit
approved these changes
Jul 26, 2025
Member
pufit
left a comment
There was a problem hiding this comment.
Everything besides the failing test looks good.
CheSema
reviewed
Jul 28, 2025
| assert response == "OK", response | ||
|
|
||
|
|
||
| class Throttler: |
Member
There was a problem hiding this comment.
Cool, you are using broken_s3 mock :)
Member
Author
There was a problem hiding this comment.
Yes, it helped emulate the 'Slow Down' error.
730cd32 to
cc7db5f
Compare
serxa
reviewed
Jul 31, 2025
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR improves the S3 retry strategy for backup and restore operations by introducing jitter, fixing a bug where the first retry had no delay, and adding configurable parameters for the retry backoff strategy.
- Introduces jitter to the S3 retry strategy to avoid thundering herd problems
- Fixes a bug where the first retry attempt had no delay
- Adds configurable parameters for initial delay, max delay, and jitter factor in the S3 retry backoff strategy
Reviewed Changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/IO/S3/PocoHTTPClient.h |
Defines new RetryStrategy struct with configurable parameters |
src/IO/S3/PocoHTTPClient.cpp |
Adds validation for retry strategy parameters |
src/IO/S3/Client.h |
Updates RetryStrategy class to use new configuration |
src/IO/S3/Client.cpp |
Implements improved retry delay calculation with jitter |
src/Core/Settings.cpp |
Adds new settings for retry configuration |
src/Core/SettingsChangesHistory.cpp |
Records new settings in version history |
src/Backups/BackupIO_S3.cpp |
Uses new retry strategy configuration for backups |
tests/integration/test_backup_restore_s3/test.py |
Updates test structure and adds throttling test |
tests/integration/helpers/s3_mocks/broken_s3.py |
Implements S3 throttling mock functionality |
| "enable_s3_requests_logging": "1", | ||
| } | ||
|
|
||
| node = cluster.instances["node"] |
There was a problem hiding this comment.
The variable node is assigned twice in this function (lines 936 and 946). The second assignment is redundant and should be removed.
Suggested change
| node = cluster.instances["node"] |
1 task
Member
Author
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 11, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 12, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 13, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 14, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 15, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 16, 2025
baibaichen
pushed a commit
to Kyligence/gluten
that referenced
this pull request
Aug 18, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://github.com/ClickHouse/clickhouse-private/issues/19914
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Introduce
backup_restore_s3_retry_initial_backoff_ms,backup_restore_s3_retry_max_backoff_ms,backup_restore_s3_retry_jitter_factorto configure the S3 retry backoff strategy used during backup and restore operations.Documentation entry for user-facing changes