Have you tested this, I ask as the retry logic occurs before the response body processing

Have you tested this, I ask as the retry logic occurs before the response body processing

I haven't. My assumption was that this error originates from within the reqwest client, so we should be able to catch it here. Even in the streaming case I thought we should still be polling data from the RetryClient, but I will look more closely at the dataflow here to see what I've missed.

Testing this is a bit annoying without mocking the inner client, which isn't exactly a real world scenario. I am not very familiar with the existing testing harness so if you have any recommendations on where to start I would appreciate it.

I thought we should still be polling data from the RetryClient

Unfortunately, more broadly speaking this is not generally possible, as discussed on the ticket. Once response streaming has started, a retry would need to somehow resume from where it left off, the semantics of which will depend on the method in question. I do not know of a good way to handle this.

Testing this is a bit annoying without mocking the inner client, which isn't exactly a real world scenario. I am not very familiar with the existing testing harness so if you have any recommendations on where to start I would appreciate it.

We already have a mock HTTP server harness for running these sorts of tests

Unfortunately, more broadly speaking this is not generally possible, as discussed on the ticket.

Could you link me to where this is discussed? I'm afraid there's been a lot of comments around this spread across various issues, so it is hard to find specific discussions.

Once response streaming has started, a retry would need to somehow resume from where it left off, the semantics of which will depend on the method in question. I do not know of a good way to handle this.

Is "method" in this context referring to the HTTP method?

Perhaps we need a RetryStream to wrap the response stream in? I am not sure how that would hook up to the existing trait methods exactly, but it seems necessary if we want to transparently re-initiate response streaming.

apache/arrow-rs-object-store#53 suggests having a manual way to re-initiate the request, but I'm not sure what that would look like either.

Could you link me to where this is discussed?

apache/arrow-rs-object-store#53

Perhaps we need a RetryStream to wrap the response stream in? I am not sure how that would hook up to the existing trait methods exactly, but it seems necessary if we want to transparently re-initiate response streaming.

I think we could add a method to RetryableRequest to return a Result<Bytes> that can be used by non-idempotent, non-streaming requests, such as ObjectStore::list, and which will retry errors during response streaming by retrying the entire request.

However, ObjectStore::get will require retrying at a higher level, as not only will it need to keep track of the current offset, but compute a new range for the retry, and re-sign the resulting request. Perhaps something in GetClientExt might work 🤔

In my opinion, to move forward we really need an example/test showing the problem so we can evaluate how the proposed solution fixes it. More discussion here: apache/arrow-rs-object-store#53

One thing I've noticed is that I also would see failed decoding bytes, when a timeout is set too low.

Could it be that the request still gets processed even though it was incomplete and should have thrown a timeout error instead?

One thing I've noticed is that I also would see failed decoding bytes, when a timeout is set too low.

Could it be that the request still gets processed even though it was incomplete and should have thrown a timeout error instead?

From what I understand, if a request fails mid-stream (after we begin to read data) just retrying the request isn't enough -- instead it would need something more complicated like retrying the request just for the bytes that hadn't been already read, or something like that

Since this keeps coming up, I filed a separate ticket to track

• Retry / recover after partially reading a streaming response ( fix timeout errors / error decoding response body ) arrow-rs-object-store#15

Thank you for this PR. We are in the process of moving the object_store code to its own repository. Would it be possible for you to create a PR in that repository instead?

• See details on [EPIC] Port object_store content from arrow-rs repository arrow-rs-object-store#2
• Backstory on [DISCUSSION] Proposal move object_store to its own github repo? #6183

(we will handle moving all existing issues to the new repository)

We merged a fix in

• Retry streaming get requests (#15) arrow-rs-object-store#383