Output of restic version

restic 0.9.5 compiled with go1.12.9 on linux/amd64

How did you run restic exactly?

See the attached test script, which I ran like this:

$ ./test 1000000

The script creates a directory containing a million empty files and then backs it up using restic. On an ext4 filesystem the directory is 22MB in size.

What backend/server/service did you use to store the repository?

Filesystem

Expected behaviour

Lower memory usage.

Actual behaviour

Higher memory usage, see the attached heap profiles with GOGC unset and
GOGC=1 taken towards the end of the restic backup process. The peak RAM usage with GOGC unset is 2.8 GB.

With real-world workloads (such as large Maildirs containing more files), the RAM usage can exceed the amount of RAM on the system, leading to swap death.

Steps to reproduce the behaviour

See the attached test script.

Do you have any idea what may have caused this?

I am guessing that restic prepares directory indexes in RAM instead of preparing them on-disk. The issue could also be inefficiently stored directory indexes.

Do you have an idea how to solve the issue?

I don't know enough about restic's internals nor Golang memory profiling.

Did restic help you or made you happy in any way?

I was happy to find a good replacement for rdiff-backup.