What has changed since #820 / #998 such that restic no longer needs a blob size cache? The Index/MasterIndex structure back in restic v0.4.0 at first glance seems to be rather similar to the current state.

(Or could this have something to do with debug logging? #1549 also seems to have sped up index lookups a lot)

@MichaelEischer I have honestly no idea. I've never been able to reproduce #820. In any case, with #2788 and/or #2789, there shouldn't be a performance issue left, at least not in the single-client case.

@MichaelEischer I also don't have any idea why the blob size cache could significantly increase speed... The only issue I see is with a lot of index files as LookupBlobSize needs to loop over all index files within the master index. And there is some small overhead in the used 'Lookup', but this will be removed with #2788 or #2789 as @greatroar correctly pointed out.

Moreover, #2587 will make restic mount much more responsive and only needs to call 'LookupBlobSize` when the file is really accessed.

So I suggest to remove this memory-hungry cache and leave further speed optimizations to #2587 or to a future PR that stores all loaded index information in one single data structure.

If the performance problem that was tackled using the blob size cache still exists, then #2587 only moves the incredibly slow operation from listing the directory to opening the file. That would improve performance a bit, but whether you have to wait 12 or 9 minutes (just taking the numbers from #820) isn't that much of a difference.

I'm actually able to reproduce the performance problem (tested on macOS, #820 was reported on Linux): I've mounted a backup containing a folder with a 100 GB VM image. The first call to ls -laR . in that folder takes roughly 5 seconds (doesn't matter whether the debug log is enabled or not). The folder in #820 had roughly 900GB and the optimizations in #1549 speed-up lookups by factor 10-20x. Taken together this means that the 5 seconds are in fact the scaled down variant of the 12 minutes in #820.

I've noticed that the backup repository has 527 index files which for sure doesn't help the index performance, but that many index files are probably not too uncommon.

Does a second ls use the filesystem cache?

I prepared #2799 which merges the in-memory index into one index data structure.
If #2789 and #2799 will be merged, I do not see any reason why to have an extra blob size cache...

@greatroar Please rebase this PR to the current master. The index implementation is probably fast enough by now to avoid a (big) performance regression.

Done!