-
Notifications
You must be signed in to change notification settings - Fork 1.7k
[architecture] Performance of restic snapshots with high-latency remote #523
Description
Output of restic version
Hi,
Current restic performs sequential roundtrips in some operations. One of these operations is snapshot list. With high-latency (e.g. 150ms transatlantic) remote server (e.g. restic-server) and unbounded repository growth (e.g. hourly backups since ~1 month = 600 snapshots), then restic requires ~2 minutes just to list available snapshots.
I have a patch that parallelizes requests in restic.LoadAllSnapshots(), and additionally a separate patch so that this code can actually be used inside cmd_snapshots.go since currently it's reimplemented. But, i think that's a stopgap measure and doesn't significantly address the issue. In the long term, what is the architectural possibility of building a kind of packfile system but for the snapshots?
Expected behavior
restic snapshots should run in time not proportional to latency*numSnapshots
Actual behavior
restic snapshots runs in time proportional to latency*numSnapshots
Steps to reproduce the behavior
- Make 600 or more snapshots - doesn't matter the content - to a high-latency server
- Observe the time needed to run
restic snapshots