restic (0.17.0) on Windows silently excludes inaccessible files from new snapshots #4953
Description
Output of restic version
restic 0.17.0 compiled with go1.22.5 on windows/amd64
What backend/service did you use to store the repository?
S3 Wasabi
Problem description / Steps to reproduce
restic is used for nightly backups of several servers. The Windows servers run a batch file via the Task Scheduler.
For testing restic was called with these arguments:
restic_0.17.0.exe backup --tag DEBUG -vv --use-fs-snapshot --iexclude-file=excludes_global.txt --iexclude-file=excludes_filesrv.txt --exclude-if-present=".restic.ignore:NOBACKUP" P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken | findstr /V /B "unchanged " >>logfile_0.17.txt 2>&1I was testing the new 0.17.0 release on one of the older Windows Server 2012 R2 hosts, which holds mostly unused or rarely modified data.
While testing the backup with restic 0.17.0, I noticed that restic had issues accessing some path that is too long, but which has been backed up before, so I didn't mind the command's warning after the dry run Warning: at least one source file could not be read because I don't expect this folder to change until the server is decommissioned.
What I did notice was that even though neither version reported any removed files for the last snapshot, a restic diff between a snapshot created by 0.16.5 and 0.17.0 showed a removal of the unreadable files.
In my case this would have gone unnoticed if I hadn't run the backup interactively. Neither the runtime error nor the output of restic backup regarding the removal was written to the log files and only the read error inside the console window and the restic diff were showing any signs of a potential removal.
Expected behavior
I know the actual issue regarding the too long file path isn't caused by restic, but by the old Windows version, but I was expecting to see any errors and removals in restic's output written to my log files or that the backup would fail completely so I can react accordingly.
So if a file, that's already backed up, isn't readable anymore, let restic mention the file's removal in the verbose output of restic backup and print this fact in the summary either as "removed" or "unreadable".
Actual behavior
Manually create snapshots for both versions:
set AWS_ACCESS_KEY_ID=REDACTED
set AWS_SECRET_ACCESS_KEY=REDACTED
set RESTIC_REPOSITORY_FILE=restic_repo.txt
set RESTIC_PASSWORD_FILE=restic_pass.txt
restic_0.16.5.exe backup --tag DEBUG -vv --use-fs-snapshot --iexclude-file=excludes_global.txt --iexclude-file=excludes_filesrv.txt --exclude-if-present=".restic.ignore:NOBACKUP" P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken | findstr /V /B "unchanged " >>logfile_0.16.txt 2>&1
restic_0.17.0.exe backup --tag DEBUG -vv --use-fs-snapshot --iexclude-file=excludes_global.txt --iexclude-file=excludes_filesrv.txt --exclude-if-present=".restic.ignore:NOBACKUP" P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken | findstr /V /B "unchanged " >>logfile_0.17.txt 2>&1
restic_0.16.5.exe backup console output:
none
logfile_0.16.txt contents:
open repository
lock repository
using parent snapshot da251b32
load index files
start scan on [P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken]
start backup on [P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken]
creating VSS snapshot for [p:\]
successfully created snapshot for [p:\]
scan finished in 36.928s: 542 files, 5.287 GiB
[...snip...]
Files: 0 new, 0 changed, 542 unmodified
Dirs: 0 new, 6 changed, 64 unmodified
Data Blobs: 0 new
Tree Blobs: 7 new
Added to the repository: 2.096 KiB (1.795 KiB stored)
processed 542 files, 5.287 GiB in 0:46
snapshot e8ffd0f4 saved
restic_0.17.0.exe backup console output:
error: nodeFromFileInfo P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken\Redacted Präsentation\Englisch\1 Injection Moulding\3 Process technology\Injection compression moulding\Redacted Example Redacted Injection compression moulding LED lenses.pdf: get named security info failed with: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
[...snip x8...]
error: nodeFromFileInfo P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken\Redacted Präsentation\Englisch\1 Injection Moulding\3 Process technology\Silicone injection moulding - LSR and HTV\Redacted LSR Servo-electric demoulding device 28534-01.tif: get named security info failed with: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.
Warning: at least one source file could not be read
I don't know the exact English version of the error Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch., but it's basically Windows not being able to access a file because of a too long path.
logfile_0.17.txt contents:
open repository
using parent snapshot e8ffd0f4
load index files
start scan on [P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken]
start backup on [P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken]
creating VSS snapshot for [p:\]
successfully created snapshot for [p:\]
modified /P/UserProfiles/user.name/Documents/Schulungsunterlagen/Materialsammlung/Fremdgrafiken/Redacted Präsentation/Englisch/1 Injection Moulding/3 Process technology/Clean room technology/, saved in 0.017s (0 B added, 0 B stored, 0 B metadata)
[...snip... only "modified" entries, no removals]
modified /P/, saved in 11.377s (0 B added, 0 B stored, 0 B metadata)
Files: 0 new, 0 changed, 542 unmodified
Dirs: 0 new, 70 changed, 0 unmodified
Data Blobs: 0 new
Tree Blobs: 0 new
Added to the repository: 0 B (0 B stored)
processed 542 files, 5.287 GiB in 0:36
snapshot 8da27a4e saved
# restic snapshots --host filesrv --tag DEBUG
repository c19ce220 opened (version 2, compression level auto)
ID Time Host Tags Paths Size
-------------------------------------------------------------------------------------------------------------------------------------------------------------
e8ffd0f4 2024-07-29 10:15:21 filesrv DEBUG P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken
8da27a4e 2024-07-29 10:18:23 filesrv DEBUG P:\UserProfiles\user.name\Documents\Schulungsunterlagen\Materialsammlung\Fremdgrafiken 5.287 GiB
-------------------------------------------------------------------------------------------------------------------------------------------------------------
2 snapshots
# restic diff e8ffd0f4 8da27a4e
repository c19ce220 opened (version 2, compression level auto)
comparing snapshot e8ffd0f4 to 8da27a4e:
[0:14] 100.00% 1275 / 1275 index files loaded
- /P/UserProfiles/user.name/Documents/Schulungsunterlagen/Materialsammlung/Fremdgrafiken/Redacted Präsentation/Englisch/1 Injection Moulding/3 Process technology/Injection compression moulding/Redacted Example Redacted Injection compression moulding LED lenses.pdf
[...snip x8...]
- /P/UserProfiles/user.name/Documents/Schulungsunterlagen/Materialsammlung/Fremdgrafiken/Redacted Präsentation/Englisch/1 Injection Moulding/3 Process technology/Silicone injection moulding - LSR and HTV/Redacted LSR Servo-electric demoulding device 28534-01.tif
Files: 0 new, 10 removed, 0 changed
Dirs: 0 new, 0 removed
Others: 0 new, 0 removed
Data Blobs: 0 new, 3 removed
Tree Blobs: 71 new, 71 removed
Added: 671.646 KiB
Removed: 2.922 MiB
So in summary:
restic backup 0.16.5:
Files: 0 new, 0 changed, 542 unmodified
restic backup 0.17.0:
Files: 0 new, 0 changed, 542 unmodified
restic diff 0.16.5_snap 0.17.0_snap:
Files: 0 new, 10 removed, 0 changed
Do you have any idea what may have caused this?
I guess the new feature to read extended attributes and ACLs on Windows causes a re-read of all backed up files so the problematic path is now leading to errors. I would assume that any other reason for a re-read would've led to the same issue in the past.
Did restic help you today? Did it make you happy in any way?
restic is awesome and get's better all the time!