@hoytak Do you think I should add a config override for this so users can set an env variable to configure this?

Nice work thanks! Would be nice if there was a patch pypi release for this soon!

If this fixes it, then LGTM!

A config var would make sense -- the max timeout -- e.g. large shard timeout? There are situations where 5 min may not be enough.

If this fixes it, then LGTM!

A config var would make sense -- the max timeout -- e.g. large shard timeout? There are situations where 5 min may not be enough.

This PR has the max time be 10m (clamped at 600s) - which I imagine should be enough for 128MB shard - which I believe is the max shard size.

I am building out some tests for this and then will merge.

Fortunately @XciD did some actual testing with this patch and learned that the per-request timeout does not take precedence over the overall read_timeout timing. If overall read_timeout is set then per-request being longer doesn't matter.

Need to rethink this fix. Also, the timeout is silently failing - need to add testing to confirm that when timeout occurs we need to actually fail the upload with a reasonable explanation.

More in internal Slack thread: https://huggingface.slack.com/archives/C087TU2FE3G/p1772992380446739?thread_ts=1772768281.424279&cid=C087TU2FE3G

did you find why the error was silenced ? worth investigating if we are silenting some other errors.
Could have been a retry policy failling over and over in my case, but worth checking

did you find why the error was silenced ? worth investigating if we are silenting some other errors. Could have been a retry policy failling over and over in my case, but worth checking

From the analysis it should have reported an ERROR message in the logs after all retries were exhausted. Each retry should be in the logs at INFO level. What is less clear is what makes its way back to Python - but for the testing you were doing that was in Rust, right?

Here is the writeup for the swallowed errors from Claude:

  Why Timeout Errors Appear Silent

  The error path is: reqwest timeout → RetryWrapper → upload_shard → shard_interface → file_upload_session::finalize → upload_commit → user.

  The error does propagate — it's not silently discarded in code. But here's why it appeared silent:

  Root Cause: Timeouts are classified as Retryable::Transient

  In on_request_failure (line 473):
  if error.is_timeout() || is_connect {
      Some(Retryable::Transient)
  }

  This means:

  1. Each timeout is logged at info level, not error — process_error_response logs transient errors with log_as_info = true (line 115). If the operator or user doesn't have info-level logging enabled, they see nothing during the retry attempts.
  2. The retry loop silently retries 3 times (default retry_max_attempts). With the old 120s read_timeout, that's up to 6 minutes of silent retries before failing. Each retry hits the same timeout.
  3. The final "No more retries; aborting" IS logged at error level (line 320), but only after exhausting all retries.
  4. From the user's perspective: the upload hangs for ~6 minutes (3 × 120s timeouts), then fails with a generic CasClientError that wraps the reqwest timeout. The error message may not clearly indicate "shard upload timed out" — it's wrapped in layers of error types.

  Contributing factors:

  - No progress indication during shard upload — unlike xorb uploads which have progress callbacks, shard upload has no intermediate progress reporting. The user sees the upload "freeze" with no explanation.
  - Default log level may hide the info-level retry messages — if INFORMATION_LOG_LEVEL is DEBUG (which it is by default, line 28-29 of lib.rs), the retry messages are at debug level, invisible to most users.
  - The error message itself may be unclear — a reqwest timeout wraps into CasClientError::from(reqwest_middleware::Error) which may not say "shard upload timed out after 120s" in a user-friendly way.

did you find why the error was silenced ? worth investigating if we are silenting some other errors. Could have been a retry policy failling over and over in my case, but worth checking

From the analysis it should have reported an ERROR message in the logs after all retries were exhausted. Each retry should be in the logs at INFO level. What is less clear is what makes its way back to Python - but for the testing you were doing that was in Rust, right?

Here is the writeup for the swallowed errors from Claude:

  Why Timeout Errors Appear Silent

  The error path is: reqwest timeout → RetryWrapper → upload_shard → shard_interface → file_upload_session::finalize → upload_commit → user.

  The error does propagate — it's not silently discarded in code. But here's why it appeared silent:

  Root Cause: Timeouts are classified as Retryable::Transient

  In on_request_failure (line 473):
  if error.is_timeout() || is_connect {
      Some(Retryable::Transient)
  }

  This means:

  1. Each timeout is logged at info level, not error — process_error_response logs transient errors with log_as_info = true (line 115). If the operator or user doesn't have info-level logging enabled, they see nothing during the retry attempts.
  2. The retry loop silently retries 3 times (default retry_max_attempts). With the old 120s read_timeout, that's up to 6 minutes of silent retries before failing. Each retry hits the same timeout.
  3. The final "No more retries; aborting" IS logged at error level (line 320), but only after exhausting all retries.
  4. From the user's perspective: the upload hangs for ~6 minutes (3 × 120s timeouts), then fails with a generic CasClientError that wraps the reqwest timeout. The error message may not clearly indicate "shard upload timed out" — it's wrapped in layers of error types.

  Contributing factors:

  - No progress indication during shard upload — unlike xorb uploads which have progress callbacks, shard upload has no intermediate progress reporting. The user sees the upload "freeze" with no explanation.
  - Default log level may hide the info-level retry messages — if INFORMATION_LOG_LEVEL is DEBUG (which it is by default, line 28-29 of lib.rs), the retry messages are at debug level, invisible to most users.
  - The error message itself may be unclear — a reqwest timeout wraps into CasClientError::from(reqwest_middleware::Error) which may not say "shard upload timed out after 120s" in a user-friendly way.

It was rust, but in a e2e test were I didnt had the logs. So I was maybe block on a retry loop