Skip to content

High download bandwidth usage on prune #3049

@Prof-Bloodstone

Description

@Prof-Bloodstone

I believe what I list as separate issues, are related and might be the same issue.

Output of restic version

restic 0.10.0 compiled with go1.15.2 on linux/amd64
Also happens on 0.9.5

How did you run restic exactly?

export RESTIC_PASSWORD='1234'
export RESTIC_REPOSITORY='rclone:restictest:backups/'
export RCLONE_CONFIG_RESTICTEST_TYPE=sftp
export RCLONE_CONFIG_RESTICTEST_HOST='127.0.0.1'
export RCLONE_CONFIG_RESTICTEST_PORT=2222
export RCLONE_CONFIG_RESTICTEST_USER=${USER}
export RCLONE_CONFIG_RESTICTEST_KEY_FILE="${HOME}/.ssh/id_rsa"

restic init
restic backup backup_test_source/
restic prune
restic prune
restic prune

What backend/server/service did you use to store the repository?

SFTP:

version: '3.7'

services:
  sftp:
    image: atmoz/sftp:alpine
    container_name: sftp
    ports:
      - '2222:22'
    volumes:
      - "${HOME}/.ssh/id_rsa.pub:${HOME}/.ssh/keys/id_rsa.pub:ro"
      - "./backups:${HOME}/backups/"
    command:
      - "${USER?Required USER}::${UID?Required UID}"

Expected behavior

  1. Do nothing when running prune back-to-back.
  2. Minimize the amount of down data that is required to be downloaded - for many storage providers, downloading data is expensive.

Actual behavior

  1. Currently my test repository gets rewritten each time I run prune, which requires restic to download (in my case) 175MB of data and upload 270MB of data.
  2. Depending on the data being backed up, restic seems to upload not much more data than it downloads in a month if each backup is followed by forget --prune of oldest backup. A few numbers from dedicated backup host (no restoring, single restic target):
month rx (received from restic) tx (downloaded by restic) total
Sep 1.18 TiB 1.11 TiB 2.30 TiB
Oct 775.25 GiB 737.19 GiB 1.48 TiB

Data gathered with vnstat.

Steps to reproduce the behavior

Issue with back-to-back prune:

  1. Start attached docker-compose service
  2. Following my steps above - export variables, initialize the repo and create a backup. Might use head -c 500M </dev/urandom >| test-file to generate random file
  3. Install vnstat: docker-compose exec sftp apk add vnstat
  4. Watch vnstat docker-compose exec sftp vnstat -l
  5. Start running prune commands, restarting vnstat after each one
  6. Notice every prune uses up a significant amount of bandwidth.

Issue with high prune bandwith post forget:

As above - just for step 2 get some more "real" backup data, and for step 5/6 forget one of the older snapshots.

With some data, restic seems to download almost twice as much as it uploads during such prune (304.59MiB upload, 576.88MiB download).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions