Hello, please could you test the performance with large files + 50gb. I already had problems listing large files, so the file size cache was implemented.

I have a snapshot that contains a single 58GB file. When I ls that file, I get instant results:

$ time ls
bigfile.tar.bz2

real	0m0,019s
user	0m0,003s
sys	0m0,000s

Is that enough of a test, or do you have a suggestion for further benchmarking?

See also the discussion in #1680

About your PR:

• I do agree about removing BlobSizeCache (if the index is fast enough to deliver the size).
• Your index updates are covered (and extended a lot in means of memory usage) in the change discussed in Comparison of approaches to tackle index memory usage #2523. My two cents: It doesn't help (much) to optimize here. Also MasterIndex should be optimized to prevent looping over potentially lots of index files. This leads shortly to big changes in the index implementation, this is why I proposed a re-implementation.

About the test setting: Testing one single large files is not enough as the mountperformance is IMO mainly driven by listing dirs, i.e. loading tree blobs (of which you most likely have only one.
Index performance generally is depending on index entries and number of index files. Both are comparatively small for a repo with just one big file (average blob size should be around 8MB).

...when writing my last comment I realized that I should have checked whether reading tree blob is really the slow operation for listing files in FUSE. In fact it was not...
So I made #2587 which should solve the main problem and should IMO really make caching blob sizes unnecessary because it only reads the blob sizes when a file is read.

Thanks @greatroar for leading me to the right part to look at.
And sorry for proposing a "competing" PR... :-/

I had seen #2523, but decided to do a more localized optimization for the short term. I'd missed #1680.

Re: performance on lots of small files, it's hard to get consistent measurements because the kernel/FUSE seems to be caching the responses from restic. The first ls takes seconds for a dir of 10k small files, the second .06s, both with master and this PR. That also suggests that an LRU cache, which I considered as a potential further optimization, will be useless.

@greatroar Is it possible that you test your example with 10k of small files with #2587 ?
I think there also the first ls should be already very fast...

@aawsome Here's the test. Setup code:

export RESTIC_REPOSITORY=/tmp/testrepo
export RESTIC_PASSWORD=test

restic init
mkdir /tmp/testdata
for i in $(seq 10000); do
	touch /tmp/testdata/$i
done

restic backup /tmp/testdata

Then mount, and do

time ls $mountpoint/snapshots/latest/tmp/testinput

That gives, for master:

real	0m0,770s
user	0m0,034s
sys	0m0,102s

For this PR:

real	0m0,811s
user	0m0,036s
sys	0m0,119s

For #2587:

real	0m1,068s
user	0m0,051s
sys	0m0,153s

No big changes. Still, I'm closing in favor of #2587 because its changes are more localized.