-
-
Notifications
You must be signed in to change notification settings - Fork 910
400 RequestTimeout - s3cmd is not re-trying #737
Description
s3cmd version 1.6.1
SunOS cloudbackup 5.11 joyent_20160428T170316Z i86pc i386 i86pc Solaris
When doing an initial sync of about 100k files to S3:
s3cmd sync /path/to/photos s3://photo-archive-bucket/
.. plenty of successes for > 12 hours (averaging 600 kB/sec) ...
then:
WARNING: photos/Photos/20 - Mexico - Group1.jpg: Owner groupname not known. Storing GID=100 instead.
upload: 'photos/Photos/20 - Mexico - Group1.jpg' -> 's3://photo-archive-bucket/photos/Photos/20 - Mexico - Group1.jpg' [15078 of 53083]
313221 of 313221 100% in 33s 9.00 kB/s done
ERROR: S3 error: 400 (RequestTimeout): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
I found the following (very old) issue in AWS' official repository:
aws/aws-sdk-js#281
It seems like in most cases the S3 error 400 RequestTimeout Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed. is related to incorrect content-length header. However, I had not run with --debug and so cannot confirm.
Importantly, and the reason I am filing this issue, is that I also found the following on the s3tools mailing list:
https://sourceforge.net/p/s3tools/mailman/message/25713745/
This indicates that the code had been fixed to retry on 400 status codes.
Indeed, looking at the s3cmd source code:
https://github.com/s3tools/s3cmd/blob/master/S3/S3.py#L1289
It should be retrying (up to 5 times total as the S3 class _max_retries is initialized at 5), but it is not. Why not?
The 9.00 kB/s (my other files all transmitted at > 600 kB/sec) suggest perhaps I had an intermittent internet connectivity issue. EVEN SO, s3cmd should have retried several additional times.
Looking at the code path this could happen if
- retries is initialized as 0 -- I don't see this happening, but I could have missed it
if err.code in [ 'BadDigest', 'OperationAborted', 'TokenRefreshRequired', 'RequestTimeout' ]:evaluates to false, which would then skip theif try_retry:block and fall through toraise S3Error(response).
Condition 2 could happen if the response did not have a key data or data were empty:
https://github.com/s3tools/s3cmd/blob/master/S3/Exceptions.py#L52
Any other suggestions for debugging?