Skip to content

400 RequestTimeout - s3cmd is not re-trying #737

@jblachly

Description

@jblachly

s3cmd version 1.6.1
SunOS cloudbackup 5.11 joyent_20160428T170316Z i86pc i386 i86pc Solaris

When doing an initial sync of about 100k files to S3:

s3cmd sync /path/to/photos s3://photo-archive-bucket/

.. plenty of successes for > 12 hours (averaging 600 kB/sec) ...
then:

WARNING: photos/Photos/20 - Mexico - Group1.jpg: Owner groupname not known. Storing GID=100 instead.
upload: 'photos/Photos/20 - Mexico - Group1.jpg' -> 's3://photo-archive-bucket/photos/Photos/20 - Mexico - Group1.jpg'  [15078 of 53083]
313221 of 313221   100% in   33s     9.00 kB/s  done
ERROR: S3 error: 400 (RequestTimeout): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

I found the following (very old) issue in AWS' official repository:
aws/aws-sdk-js#281
It seems like in most cases the S3 error 400 RequestTimeout Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. is related to incorrect content-length header. However, I had not run with --debug and so cannot confirm.

Importantly, and the reason I am filing this issue, is that I also found the following on the s3tools mailing list:
https://sourceforge.net/p/s3tools/mailman/message/25713745/
This indicates that the code had been fixed to retry on 400 status codes.

Indeed, looking at the s3cmd source code:
https://github.com/s3tools/s3cmd/blob/master/S3/S3.py#L1289
It should be retrying (up to 5 times total as the S3 class _max_retries is initialized at 5), but it is not. Why not?

The 9.00 kB/s (my other files all transmitted at > 600 kB/sec) suggest perhaps I had an intermittent internet connectivity issue. EVEN SO, s3cmd should have retried several additional times.

Looking at the code path this could happen if

  1. retries is initialized as 0 -- I don't see this happening, but I could have missed it
  2. if err.code in [ 'BadDigest', 'OperationAborted', 'TokenRefreshRequired', 'RequestTimeout' ]: evaluates to false, which would then skip the if try_retry: block and fall through to raise S3Error(response).

Condition 2 could happen if the response did not have a key data or data were empty:
https://github.com/s3tools/s3cmd/blob/master/S3/Exceptions.py#L52

Any other suggestions for debugging?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions