Skip to content

Feature requests: --ignore-warning= mode (for WARNINGS) and --max-retries= / --wait= #819

@dxdc

Description

@dxdc

First off, this is a terrific utility and I've written a script to perform a weekly backup on my server. I have two suggestions for features that would be of immense benefit.

I am running s3cmd version 1.6.1, so not sure if some of these issues have been resolved in a more current release, but based on my review of the code I don't think so.

  1. I have probably 30 GB of user-uploads, which I have an automated cron script to run to back these up on a routine basis. Due to the number of files, it just takes a single bad connection to cause a failure with these kinds of warning messages. Then, I get a cron notification and have to restart the process manually.
Cannot retrieve any response status before encountering an EPIPE or ECONNRESET exception
WARNING: Upload failed: /srv/html/uploads/548/88919fd4bec2bf.pdf ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...

It would be helpful to hardcode in some kind of retries so that it will not just bail after a single upload. Perhaps even an increasing wait period between retries? There could be a retry limit per file and a total retry limit as well, which can be tuned based on network performance.

  1. When running this as a cron job, warnings about symbolic links, etc. are piped to STDERR causing it to fail with exit status.

Some of these warnings (for example, about symbolic links) should be ignored. Currently, I can achieve this with some advanced bash scripting, piped through grep, but it would be better to handle this differently. Specifying which warnings to silence would be very useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions