Output of restic version

restic 0.8.3 (a05bd98)
compiled with go1.10 on linux/amd64

How did you run restic exactly?

restic -r sftp:HOSTNAME:/path/to/restic/repo -p PASSWORD_FILE mount /path/to/mnt/point
And
restic -r sftp:HOSTNAME:/path/to/restic/repo -p PASSWORD_FILE --no-cache mount /path/to/mnt/point

What backend/server/service did you use to store the repository?

SFTP over Internet with high speed (100 Mbps, ping <25ms) access between the client and server

Expected behavior

Randomly read small files with a reasonable latency (e.g. within a few seconds), and consumes reasonable amount of RAM and CPU resource.

Actual behavior

With the cache, random reading small files (e.g.: cat mountpoint/snapshots/latest/path/to/small/file.ext) between a few bytes to a few MB takes many minutes because restic downloads entire pack file(s) (size between a few MB to many hundred MB) to the cache directory before serving the read request.
Without the cache, random reading small files in small directory is a lot faster. However when the directory contains millions of small files, the latency goes up to many minutes per file regardless of the files size.

Either way, the memory consummation and CPU usage are crappy, it uses all of the available memory on the client side (12.3GB out of 16GB physical RAM, the remaining memory are used by other applications), and the CPU usage often goes up to and stays at 100% for more than half a minute. The CPU usage is worse when --no-cache is given.

In addition to above, the disk usage with cache enabled is also concerning me, randomly access a few files resulted in a multi GB cache directory. I'm not sure if the cache size will go up linearly when more files are accessed.

Steps to reproduce the behavior

Backup a 12TB directory with 65 million files spread in nearly 700 subdirectories. Then use restic mount from a different server, and randomly access the files in the mount point.

This is to simulate a disaster recovery scenario, where the main server which hosts the multi TB of data fails, and to keep the business going, we use a new server to do fuse mount and temporarily serve the files from it while we are restoring files to a different server.

Do you have any idea what may have caused this?

For the memory issue, probably the golang GC? this is just a wild guess.
For the CPU issue, I think restic is decrypting (and then discarding?) extremely large amount of data.
For the latency, my guess is, with the cache restic spend time to download the entire pack files instead of only the needed part; without the cache restic spend time to download the index and decrypt it over and over again.

Do you have an idea how to solve the issue?

With cache enabled, only download (and cache) the required data for accessing a file, not the whole pack file. This is certainly doable, but will require the local cache structure change (but this does not break the backward compatibility, it's only a cache)
Without cache, do we really need to download the index over and over again? And is it necessary to download and decrypt whole index files? Can we download only part of it? (Sorry, I haven't read the code or doc about the index structure)

Also we will need a proper cache management. For example, cache eviction policy, size limit, disk full handling (which is filled up by the cache in my case, nearly 30GB cache on a 40GB root partition).

Did restic help you or made you happy in any way?

I'm not sure about this yet, we are currently looking for solutions allow us to backup more than 50TB of data with ~32% of duplications, data size increasing a few hundred GB daily (new files only, existing files are never changed). We will need to be able to recover from the main storage server failure fast enough to not affect users badly.
We've looked at a lots of different software solutions, and have only tested restic yet, because it is the only one supports B2 backend (cheap and faster retrieval time than Glacier) AND has deduplication AND allows mount.