-
Notifications
You must be signed in to change notification settings - Fork 1.7k
High download bandwidth usage on prune #3049
Copy link
Copy link
Closed
Description
I believe what I list as separate issues, are related and might be the same issue.
Output of restic version
restic 0.10.0 compiled with go1.15.2 on linux/amd64
Also happens on 0.9.5
How did you run restic exactly?
export RESTIC_PASSWORD='1234'
export RESTIC_REPOSITORY='rclone:restictest:backups/'
export RCLONE_CONFIG_RESTICTEST_TYPE=sftp
export RCLONE_CONFIG_RESTICTEST_HOST='127.0.0.1'
export RCLONE_CONFIG_RESTICTEST_PORT=2222
export RCLONE_CONFIG_RESTICTEST_USER=${USER}
export RCLONE_CONFIG_RESTICTEST_KEY_FILE="${HOME}/.ssh/id_rsa"
restic init
restic backup backup_test_source/
restic prune
restic prune
restic pruneWhat backend/server/service did you use to store the repository?
SFTP:
version: '3.7'
services:
sftp:
image: atmoz/sftp:alpine
container_name: sftp
ports:
- '2222:22'
volumes:
- "${HOME}/.ssh/id_rsa.pub:${HOME}/.ssh/keys/id_rsa.pub:ro"
- "./backups:${HOME}/backups/"
command:
- "${USER?Required USER}::${UID?Required UID}"Expected behavior
- Do nothing when running
pruneback-to-back. - Minimize the amount of
downdata that is required to be downloaded - for many storage providers, downloading data is expensive.
Actual behavior
- Currently my test repository gets rewritten each time I run
prune, which requires restic to download (in my case) 175MB of data and upload 270MB of data. - Depending on the data being backed up, restic seems to upload not much more data than it downloads in a month if each backup is followed by
forget --pruneof oldest backup. A few numbers from dedicated backup host (no restoring, single restic target):
| month | rx (received from restic) | tx (downloaded by restic) | total |
| Sep | 1.18 TiB | 1.11 TiB | 2.30 TiB |
| Oct | 775.25 GiB | 737.19 GiB | 1.48 TiB |
Data gathered with vnstat.
Steps to reproduce the behavior
Issue with back-to-back prune:
- Start attached docker-compose service
- Following my steps above - export variables, initialize the repo and create a backup. Might use
head -c 500M </dev/urandom >| test-fileto generate random file - Install vnstat:
docker-compose exec sftp apk add vnstat - Watch vnstat
docker-compose exec sftp vnstat -l - Start running prune commands, restarting
vnstatafter each one - Notice every prune uses up a significant amount of bandwidth.
Issue with high prune bandwith post forget:
As above - just for step 2 get some more "real" backup data, and for step 5/6 forget one of the older snapshots.
With some data, restic seems to download almost twice as much as it uploads during such prune (304.59MiB upload, 576.88MiB download).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels