Problem Description

if a request fails mid-stream (after we begin to read data) it is not retried and instead an error is returned. The error message, somewhat confusingly, often says error decoding response body

Some examples:

ExternalError(General("ParquetObjectReader::get_byte_ranges error: Generic MicrosoftAzure error: error decoding response body"))

Generic S3 error: error decoding response body

Workaround

You can often work around this error by increasing the network timeout to something longer than the 30s default

Related Tickets

• there are many related discussions / report error decoding response body after upgrade to object store 0.10 #272
• we see this in DataFusion too Timeouts reading "large" files from object stores over "slow" connections datafusion#15067
• object_store: Retry on connection duration timeouts (retry / recover after partially reading a streaming response) #53
• object_store: retry on response decoding errors arrow-rs#6519
• to_pyarrow_table() on a table in S3 kept getting "Generic S3 error: error decoding response body" delta-io/delta-rs#2595

As @crepererum says on #272 :

So long store short: People agree that this would be a good feature to have, but it requires a proper implementation.

Background

Streaming ✅

Some APIs like ObjectStore::get are "streaming" in the sense that they start returning data as soon as it comes back from the network (as opposed to buffering the response before returning to the caller)

This is great for performance as response processing can happen immediately and limits memory usage for large payloads 🏆

Retries ✅

In order to deal with the intermittent errors that occur processing object store requests, most ObjectStore implementations retry the request if they encounter error (see retry.rs)

Retries + Streaming ❌

However, there is a problem when streaming is mixed with the existing retries. Specifically, if a request fails mid-stream (after some, but not all, of the data has been returned to the client), just retrying the entire request isn't enough because then the client would be potentially be given the same data from the start of the response that it had already been given

Solution

Describe the solution you'd like

Implementing retries for streaming reads would need something more complicated like retrying the request just for the bytes that hadn't been already read

Any solution for this I think needs:

1. Very good tests / clear documentation

Describe alternatives you've considered

@crepererum suggests on #272 :

retrying would need to make a new request with a new range starting after the last received byte and ideally also an ETAG/version check to ensure that the object that is returned by the retry is the the one that was already "in flight". This retry mechanic is obviously chaining/nested, i.e. if the retry fails mid-stream, you wanna have yet another retry that picks up the where the previous one ended.