Output of restic version

restic 0.9.6 compiled with go1.13.4 on windows/amd64

How did you run restic exactly?

restic -r D:\restic backup C:\ G:\ --exclude-file
No enviroment variables

What backend/server/service did you use to store the repository?

External USB 3.0 HDD drive

Expected behavior

High throughput during backup, almost no fragmentation on the drive.

Actual behavior

When I get 30 MB/s write speed on the drive I'm very happy. Mostly it is < 10 MB/s while the utilisation is always > 90 % mostly stuck at 100 %.
I've found this issue with restoring #2675
I think while backup up it is the same.
Restic syncs so heavily or uses so heavily IO that my whole USB3.0 subsystem crashes after a while. The external Seagate 4 TB USB 3.0 2.5" HDD drive disappears, comes back and so on. Disconnecting and reconnecting doesn't help. I have to reboot the whole PC. I've tried a lot backup applications with this disk so far. That never happend. To get the disk work I have to reboot. Could be a general hardwareproblem as well, but just to mention in this issue.

Take a look at windows taskmanager. > 18 seconds response time of the drive. A lot of time 0 write speed

After this backup failed I ran defrag command. It showed 64 % (!) fragmentation.

This behaviour also puts high load on the drive, wearing it faster.

Don't know how exactly restic creates files, but small files (I think < 4 KB) are created on NTFS within the MFT Table. When the size gets bigger it has to be transfered out of the MFT-Table to the disk area. This puts additional load on the drive. With ressource monitor I also see a lot of traffic to MFT-Table. Could be because of creating new files or because my assumption is correct.

defrag /A D:
Microsoft-Laufwerksoptimierung
Copyright (c) Microsoft Corp.

"Analyse" wird für "Volume (D:)" aufgerufen...


Der Vorgang wurde abgeschlossen.

Post Defragmentation Report:

        Volumeinformationen:
                Volumegröße                 = 3,63 TB
                Freier Speicherplatz        = 3,38 TB
                Fragmentierter Speicherplatz insgesamt      = 64%
                Größter freier Speicherplatz     = 3,38 TB

        Hinweis: Dateifragmente, die größer als 64 MB sind, sind nicht in den Fragmentierungsstatistiken enthalten.

        Es wird empfohlen, dieses Volume zu defragmentieren.

Steps to reproduce the behavior

Use a USB HDD and make a large backup with data size of about 3 TB.

Do you have any idea what may have caused this?

The way restic creates the backup files.

Do you have an idea how to solve the issue?

Since the individual pack files are only 4-8 MB you could hold them in RAM and then write them out at once.

Did restic help you today? Did it make you happy in any way?

I like the idea and the concept behind restic, but to be honest I took me so far a lot of time to try and figure out what is happening. I know you want to make a great job and thats why I create this bug report to help the community with restic, so that we have a very powerful and fast backup tool. I'm confident that this issue will be solved fast so that restic is my new backuptool.