This is a request for comment to improve the startup time of most CLI commands.

Background

Currently restic interleaves tree information with their data blobs in packs. File operations do a treewalk to get to the next level of directories. Access to those blobs (any blobs for that matter) is sped up by using an index which maps blob hashes to pack file/offsets. Each backup creates a new indexes which are kept around until either one of rebuild_indexes or prune cli commands is used.

Issue

At the end of each backup, a new index is created which contains the set of all previous indexes plus the new blobs written for this backup. This new index has a field, telling which other indexes it supersedes. Mainly this is done so that if 2 backups run at the roughly same time, each can write it's own index without interfering with the other backup. As a consequence one ends up with a large amount of superseded indexes, which, during startup of a cli cmd, ALL need to be read and determined for applicability.

Idea

Somewhere after reading of the available indexes, we remove indexes who are marked superseded. This removal should be non-fatal as 2 backups started at the same time, after reading all the indexes, might start removing the same indexes. Also, when reading all indexes and then failing to open them (aka, a race condition with another ReadIndexes() instance who deleted it), should be ignored, as long as in the end ReadIndexes() finds at least one non superseded index to work with.
This would not need an update of the locking strategy, a backup already has a non exclusive lock and since we are sure that the superseded indexes do not play a role in the resulting index , we do not need to upgrade our backup lock, even though it is a mutating operation.

Rationale

These superseded indexes are no longer needed as their mapping are by definition already contained any one the indexes marked as superseding them. (Note I'm only arguing to remove indexes where their superseding index is already committed to disk).

Some senseless ascii art.

A -> B(A) -> D(B,C) -> F(D,C)         -> H(F,G)
  -> C(A) ->       E(B,C)* -> G(D,E)  -|
Note that backup E took a long time, overlapping with the start of backup F. When E finishes, F has not finished so G does not see F's index.