Skip to content

Error body missing for 5xx errors after retry exhausted #617

Description

@jackye1995

Describe the bug

When retry is exhausted, the error message is missing and only shows 503 Service Unavailable: with nothing after it. This makes it hard to know if it is a throttling error or service unavailable

To Reproduce

The following test would fail and show an empty body:

    #[tokio::test]
    async fn test_503_error_body_captured() {
        let mock = MockServer::new().await;

        let retry = RetryConfig {
            backoff: Default::default(),
            max_retries: 0,
            retry_timeout: Duration::from_secs(1000),
        };

        let client = HttpClient::new(Client::builder().build().unwrap());

        // Test that 503 SlowDown body is captured for throttling detection
        let slowdown_body = r#"<?xml version="1.0" encoding="UTF-8"?><Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message></Error>"#;
        mock.push(
            Response::builder()
                .status(StatusCode::SERVICE_UNAVAILABLE)
                .body(slowdown_body.to_string())
                .unwrap(),
        );

        let e = client
            .request(Method::GET, mock.url())
            .send_retry(&retry)
            .await
            .unwrap_err();

        assert_eq!(e.status().unwrap(), StatusCode::SERVICE_UNAVAILABLE);
        assert_eq!(e.body(), Some(slowdown_body));
        assert!(e.body().unwrap().contains("SlowDown"));

        mock.shutdown().await
    }

Expected behavior

We should be able to see the error body to better debug the exact S3 underlying error.

Additional context

Example error we see in Lance in multipart upload:

LanceError(IO): Generic S3 error: Error performing PUT https://xxxxxx--use1-az4--x-s3.s3express-use1-az4.us-
east-1.amazonaws.com/xxxxxx.lance?partNumber=2&uploadId=xxxxxx in 2.264798859s, after 10 retries,
 max_retries: 10, retry_timeout: 180s  - Server returned non-2xx status code: 503 Service Unavailable: ,
 /src/lance/rust/lance-file/src/writer.rs:250:9

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions