Skip to content

Retry streaming get requests (#15)#383

Merged
tustvold merged 4 commits into
apache:mainfrom
tustvold:retry-request-body
May 29, 2025
Merged

Retry streaming get requests (#15)#383
tustvold merged 4 commits into
apache:mainfrom
tustvold:retry-request-body

Conversation

@tustvold

@tustvold tustvold commented May 26, 2025

Copy link
Copy Markdown
Contributor

Draft as needs tests

Which issue does this PR close?

Closes #15

Rationale for this change

With this PR we retry streaming get requests. Technically it is still possible to get response body errors on the other methods, however, I chose to just do get requests as these are the highest impact as they have the largest payloads, and we can always do the others later.

What changes are included in this PR?

Are there any user-facing changes?

Comment thread src/client/retry.rs

retry_error_body: bool,
/// Combines a [`RetryableRequest`] with a [`RetryContext`]
pub(crate) struct RetryableRequestBuilder {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This largely exists to keep the changeset down

Comment thread src/client/get.rs
}
}

fn get_range_meta(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted to split up get_result as it was relatively hard to follow, being a single >100 line function

Comment thread src/client/get.rs
let value = val
.to_str()
.map_err(|source| GetResultError::InvalidContentRange { source })?;
fn retry_stream(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the real meat of the PR - the rest is largely just the plumbing to make this possible

Comment thread src/client/get.rs
},
// Retry all response body errors
(Err(e), Some(etag)) if !ctx.retry_ctx.exhausted() => {
let sleep = ctx.retry_ctx.backoff();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use the same RetryContext for retrying body errors and response errors, I felt this was the most reasonable approach - if I set RetryConfig to have 10 max retries, I'd be suprised if it could make more than 10 requests if there are a mix of errors.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This strategy will make many classes of transient error less severe which is great

However, it is still possible to get timeout errors when actively reading a streaming response body as the retry mechanism kicks in after timeout, even though progress was made

At some point, it may make sense to detect when progress was made and adjust the remaining retries / policy accordingly.

@tustvold tustvold marked this pull request as ready for review May 26, 2025 17:59
@tustvold tustvold requested a review from crepererum May 26, 2025 17:59
Comment thread src/client/http/body.rs
Comment on lines +217 to +223
fn is_end_stream(&self) -> bool {
self.0.is_end_stream()
}

fn size_hint(&self) -> SizeHint {
self.0.size_hint()
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more correct, and without this the HttpServer won't populate the ContentLength header (as it uses the size hint to do this)

Comment thread src/client/get.rs
let (parts, retry_body) = request.into_parts();
let retry_etag = get_etag(&parts.headers).map_err(Self::err)?;

if etag != &retry_etag {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment thread src/client/get.rs
) -> BoxStream<'static, Result<Bytes>> {
futures::stream::try_unfold(
(self, body, etag, range),
|(mut ctx, mut body, etag, mut range)| async move {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that code is rather elegant 👍

@tustvold tustvold merged commit 7a0504b into apache:main May 29, 2025
8 checks passed
@alamb

alamb commented May 29, 2025

Copy link
Copy Markdown
Contributor

I tested this out and it works great. Details at #15 (comment)

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @tustvold and @crepererum -- this is pretty amazing

Comment thread src/client/get.rs
},
// Retry all response body errors
(Err(e), Some(etag)) if !ctx.retry_ctx.exhausted() => {
let sleep = ctx.retry_ctx.backoff();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This strategy will make many classes of transient error less severe which is great

However, it is still possible to get timeout errors when actively reading a streaming response body as the retry mechanism kicks in after timeout, even though progress was made

At some point, it may make sense to detect when progress was made and adjust the remaining retries / policy accordingly.

@alamb

alamb commented May 30, 2025

Copy link
Copy Markdown
Contributor

The CI appears to fail after this PR:

AdamGS pushed a commit to AdamGS/arrow-rs-object-store that referenced this pull request Sep 4, 2025
* Retry streaming get requests (apache#15)

* Add tests

* Fix size hint

* Fix wasm32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry / recover after partially reading a streaming response ( fix timeout errors / error decoding response body )

3 participants