Windows - Reparse points / cloud files creating a lot of errors in logs #4155
Description
Output of restic version
restic 0.14.0 compiled with go1.19 on windows/amd64
How did you run restic exactly?
Target machine: Windows 10 21H2 x64 with local only NTFS filesystem.
"restic.exe" backup "c:\Users" --iexclude-file excludes/generic_excluded_extensions --iexclude-file excludes/generic_excludes --iexclude-file excludes/windows_excludes --exclude-caches --use-fs-snapshot
I've ran restic as Administrator and as System, both produced the same results.
I've filtered the logs since there are thousands of files, and left a couple of lines per error type:
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\DOSSIERS ARCHIVES UTILE\réglement intérieur\projet RI.DOC: The media is write protected.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\DOSSIERS ARCHIVES UTILE\réglement intérieur\réglement intérieur modèle.DOC: The media is write protected.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1981.doc: The cloud operation is not supported on a read-only volume.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1982- 66.doc: The cloud operation is not supported on a read-only volume.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1983 - .pdf: The cloud operation is not supported on a read-only volume.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1984- LIFT.doc: The cloud operation is not supported on a read-only volume.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1985_FRUITS.doc: The cloud operation is not supported on a read-only volume.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1986-.doc: The cloud operation is not supported on a read-only volume.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1988-.doc: The cloud operation is not supported on a read-only volume.
error: read \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy6\Users\User\Documents\foo\bar\FINANCIIER\factures\Fact 1989_.doc: The cloud operation is not supported on a read-only volume.
What backend/server/service did you use to store the repository?
rest-server 0.11
Expected behavior
The machine I'm backing up is a Windows 10 x64 with nextcloud.
The latter uses virtual file system, just as OneDrive does.
Some files are not present on the disk, and are represented by NTFS reparse points.
Those files can obviously not be backed up, and should thus be automatically excluded by restic.
Actual behavior
restic tries to backup those files, and will fail with error message
The cloud operation is not supported on a read-only volume. or The media is write protected.
Steps to reproduce the behavior
- Install nextcloud desktop client
- Synchronize a folder
- Enable virtual file system support, see fig 1
- Right click on any synchronized folder, untick "nextcloud/available offline"
Fig 1
- Run restic backup for that file
You can also achieve this using Onedrive files or probably google drive or dropbox too.
You can check whether a file is a reparse point with fsutil utility:
fsuti reparsepoint query "c:\Users\<user>\OneDrive - <org> FRANCE"`
Which would output something like:
C:\Users\<user>>fsutil reparsepoint query "c:\Users\<user>\OneDrive - <org> FRANCE"
Valeur de la balise d’analyse : 0x9000701a
Valeur de balise : Microsoft
Valeur de balise : répertoire
Analyser la longueur des données : 0x00000064
Données d’analyse :
0000: 01 00 64 00 46 65 52 70 bf 73 95 b5 60 00 00 00 ..d.FeRp.s..`...
0010: 02 00 09 00 07 00 01 00 58 00 00 00 0a 00 04 00 ........X.......
0020: 5c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 \...............
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................
0060: 76 00 00 00
"Valeur de la balise d'analyse" translates to "Reparse Tag Value", which is the reparse point attribute.
Do you have any idea what may have caused this?
restic sees reparse points as standard files.
Do you have an idea how to solve the issue?
Restic should, as it probably already does for NTFS junctions, detect reparse files and exclude them.
Reparse points are identified by file attributes.
In my example above, Googling 0x9000701a made me find winnt.h file in mingw which has a list of all reparse point types:
In https://github.com/mirror/mingw-w64/blob/master/mingw-w64-tools/widl/include/winnt.h (line 2298) :
#define IO_REPARSE_TAG_MOUNT_POINT __MSABI_LONG(0xA0000003)
#define IO_REPARSE_TAG_HSM __MSABI_LONG(0xC0000004)
#define IO_REPARSE_TAG_DRIVE_EXTENDER __MSABI_LONG(0x80000005)
#define IO_REPARSE_TAG_HSM2 __MSABI_LONG(0x80000006)
#define IO_REPARSE_TAG_SIS __MSABI_LONG(0x80000007)
#define IO_REPARSE_TAG_WIM __MSABI_LONG(0x80000008)
#define IO_REPARSE_TAG_CSV __MSABI_LONG(0x80000009)
#define IO_REPARSE_TAG_DFS __MSABI_LONG(0x8000000A)
#define IO_REPARSE_TAG_FILTER_MANAGER __MSABI_LONG(0x8000000B)
#define IO_REPARSE_TAG_SYMLINK __MSABI_LONG(0xA000000C)
#define IO_REPARSE_TAG_IIS_CACHE __MSABI_LONG(0xA0000010)
#define IO_REPARSE_TAG_DFSR __MSABI_LONG(0x80000012)
#define IO_REPARSE_TAG_DEDUP __MSABI_LONG(0x80000013)
#define IO_REPARSE_TAG_NFS __MSABI_LONG(0x80000014)
#define IO_REPARSE_TAG_FILE_PLACEHOLDER __MSABI_LONG(0x80000015)
#define IO_REPARSE_TAG_WOF __MSABI_LONG(0x80000017)
#define IO_REPARSE_TAG_WCI __MSABI_LONG(0x80000018)
#define IO_REPARSE_TAG_WCI_1 __MSABI_LONG(0x90001018)
#define IO_REPARSE_TAG_GLOBAL_REPARSE __MSABI_LONG(0xA0000019)
#define IO_REPARSE_TAG_CLOUD __MSABI_LONG(0x9000001A)
#define IO_REPARSE_TAG_CLOUD_1 __MSABI_LONG(0x9000101A)
#define IO_REPARSE_TAG_CLOUD_2 __MSABI_LONG(0x9000201A)
#define IO_REPARSE_TAG_CLOUD_3 __MSABI_LONG(0x9000301A)
#define IO_REPARSE_TAG_CLOUD_4 __MSABI_LONG(0x9000401A)
#define IO_REPARSE_TAG_CLOUD_5 __MSABI_LONG(0x9000501A)
#define IO_REPARSE_TAG_CLOUD_6 __MSABI_LONG(0x9000601A)
#define IO_REPARSE_TAG_CLOUD_7 __MSABI_LONG(0x9000701A)
#define IO_REPARSE_TAG_CLOUD_8 __MSABI_LONG(0x9000801A)
#define IO_REPARSE_TAG_CLOUD_9 __MSABI_LONG(0x9000901A)
#define IO_REPARSE_TAG_CLOUD_A __MSABI_LONG(0x9000A01A)
#define IO_REPARSE_TAG_CLOUD_B __MSABI_LONG(0x9000B01A)
#define IO_REPARSE_TAG_CLOUD_C __MSABI_LONG(0x9000C01A)
#define IO_REPARSE_TAG_CLOUD_D __MSABI_LONG(0x9000D01A)
#define IO_REPARSE_TAG_CLOUD_E __MSABI_LONG(0x9000E01A)
#define IO_REPARSE_TAG_CLOUD_F __MSABI_LONG(0x9000F01A)
#define IO_REPARSE_TAG_CLOUD_MASK __MSABI_LONG(0x0000F000)
#define IO_REPARSE_TAG_APPEXECLINK __MSABI_LONG(0x8000001B)
#define IO_REPARSE_TAG_GVFS __MSABI_LONG(0x9000001C)
#define IO_REPARSE_TAG_STORAGE_SYNC __MSABI_LONG(0x8000001E)
#define IO_REPARSE_TAG_WCI_TOMBSTONE __MSABI_LONG(0xA000001F)
#define IO_REPARSE_TAG_UNHANDLED __MSABI_LONG(0x80000020)
#define IO_REPARSE_TAG_ONEDRIVE __MSABI_LONG(0x80000021)
#define IO_REPARSE_TAG_GVFS_TOMBSTONE __MSABI_LONG(0xA0000022)
If restic already excludes NTFS junctions via an attribute filter, one could add the specific file attributes as filters.
In C, this would look like something along this:
if(*pdwFileAttributes & FILE_ATTRIBUTE_REPARSE_POINT)
sb->st_rdev=WIN32_MOUNT_POINT;
else
sb->st_rdev=0;
Sorry, I have no golang knowlegde to help more.
Matybe this could be implemented as part of #3863 which already identifies file types ?
Did restic help you today? Did it make you happy in any way?
I really enjoy restic and decided to invest some time to make it more easy to use for end users.
I am writing a nice backup wrapper and GUI for restic, which adds some missing parts like pre and post backup hooks and a function that makes a new snapshot only if no recent snapshot exists.
This will hopefully go opensource once I ironed out caveats.
Works on Windows and Linux, also has a config gui and a restore gui.
I'd love to have a solution to get rid of those errors.