Skip to content

[architecture] Performance of restic snapshots with high-latency remote #523

@mappu

Description

@mappu

Output of restic version

Hi,

Current restic performs sequential roundtrips in some operations. One of these operations is snapshot list. With high-latency (e.g. 150ms transatlantic) remote server (e.g. restic-server) and unbounded repository growth (e.g. hourly backups since ~1 month = 600 snapshots), then restic requires ~2 minutes just to list available snapshots.

I have a patch that parallelizes requests in restic.LoadAllSnapshots(), and additionally a separate patch so that this code can actually be used inside cmd_snapshots.go since currently it's reimplemented. But, i think that's a stopgap measure and doesn't significantly address the issue. In the long term, what is the architectural possibility of building a kind of packfile system but for the snapshots?

Expected behavior

restic snapshots should run in time not proportional to latency*numSnapshots

Actual behavior

restic snapshots runs in time proportional to latency*numSnapshots

Steps to reproduce the behavior

  1. Make 600 or more snapshots - doesn't matter the content - to a high-latency server
  2. Observe the time needed to run restic snapshots

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions