Skip to content

feat(backup): add possibility to exclude macOS cloud-only files#5370

Merged
MichaelEischer merged 1 commit into
restic:masterfrom
hashier:feat/exclude-macOS-cloud-files
Nov 16, 2025
Merged

feat(backup): add possibility to exclude macOS cloud-only files#5370
MichaelEischer merged 1 commit into
restic:masterfrom
hashier:feat/exclude-macOS-cloud-files

Conversation

@hashier

@hashier hashier commented Apr 20, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR change? What problem does it solve?

Be able to exclude cloud-only files on macOS. Tested with iCloud drive. Should work with Dropbox/OneDrive too.

Some files local, some are only in the cloud

$ ls -lO%
total 2048
-rw-r--r--% 1 me  staff  compressed,dataless 1048576 Apr  9 17:34 cloud-only-file
-rw-r--r--% 1 me  staff  compressed,dataless      62 Apr 10 00:32 go.mod
-rw-r--r--% 1 me  staff  compressed,dataless     153 Apr 20 00:14 go.sum
-rw-r--r--@ 1 me  staff  -                   1048576 Apr  9 17:34 local
-rw-r--r--% 1 me  staff  compressed,dataless     501 Apr 20 00:13 main.go

restic backup will only backup the one file that are locally available (named: local)

$ /tmp/restic -r /tmp/t/ backup . -vv --exclude-cloud-files
open repository
repository e0ee1268 opened (version 2, compression level auto)
using parent snapshot 5ccf6f80
load index files
[0:00] 100.00%  17 / 17 index files loaded
start scan on [.]
start backup on [.]
scan finished in 0.021s: 2 files, 1.001 MiB
unchanged /.vscode/launch.json
unchanged /.vscode/
new       /local, saved in 0.003s (0 B added)

Files:           1 new,     0 changed,     1 unmodified
Dirs:            0 new,     0 changed,     1 unmodified
Data Blobs:      0 new
Tree Blobs:      1 new
Added to the repository: 1.066 KiB (583 B stored)

processed 2 files, 1.001 MiB in 0:00
snapshot fe7b0c99 saved

Materialize all files in the folder

$ shasum -a 256 *

$ ls -lO%
total 4120
-rw-r--r--@ 1 me  staff  - 1048576 Apr  9 17:34 cloud-only-file
-rw-r--r--@ 1 me  staff  -      62 Apr 10 00:32 go.mod
-rw-r--r--@ 1 me  staff  -     153 Apr 20 00:14 go.sum
-rw-r--r--@ 1 me  staff  - 1048576 Apr  9 17:34 local
-rw-r--r--@ 1 me  staff  -     501 Apr 20 00:13 main.go

Let's run restic backup again. The newly materialized files are going to be backed up now too that were not backed up in the last restic backup run

$ /tmp/restic -r /tmp/t/ backup . -vv --exclude-cloud-files
open repository
repository e0ee1268 opened (version 2, compression level auto)
using parent snapshot 54715f50
load index files
[0:00] 100.00%  18 / 18 index files loaded
start scan on [.]
start backup on [.]
scan finished in 0.021s: 6 files, 2.001 MiB
unchanged /.vscode/launch.json
unchanged /.vscode/
new       /go.mod, saved in 0.002s (0 B added)
new       /go.sum, saved in 0.002s (0 B added)
unchanged /local
new       /main.go, saved in 0.000s (0 B added)
new       /cloud-only-file, saved in 0.004s (0 B added)

Files:           4 new,     0 changed,     2 unmodified
Dirs:            0 new,     0 changed,     1 unmodified
Data Blobs:      0 new
Tree Blobs:      1 new
Added to the repository: 3.093 KiB (989 B stored)

processed 6 files, 2.001 MiB in 0:00
snapshot 6ff3a494 saved

And Debug Log info:

$ rm /tmp/restic-debug.log
$ restic backup …
$ rg 'rejecting online-only' /tmp/restic-debug.log
361:2025/04/20 23:38:41 archiver/exclude.go:330	main.collectRejectFuncs.RejectCloudFiles.func2	85	rejecting online-only cloud file /Users/me/Documents/tmp-test/cloud-only-file
387:2025/04/20 23:38:41 archiver/exclude.go:330	main.collectRejectFuncs.RejectCloudFiles.func2	39	rejecting online-only cloud file /Users/me/Documents/tmp-test/cloud-only-file

Note: I'm a bit surprised to see 2 log lines about cloud-only-file in the Debug Log.

Was the change previously discussed in an issue or on the forum?

Closes #5352

Checklist

  • I have added tests for all code changes.
  • I have added documentation for relevant changes (in the manual).
  • There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
  • I'm done! This pull request is ready for review.

Additional info/questions

  • What happens with this PR on HFS+ formatted file systems?
    • Since this only really works on macOS 14.0 and newer, there shouldn't be any systems with HFS+
  • This will only work on macOS 14.0 and newer source. Older versions of macOS will materialize the file when calling stat() on cloud files. Should we maybe print out a warning when using --exclude-cloud-files on older macOS versions than 14.0? (On older macOS version restic should behave the same whether --exclude-cloud-files is provided or not)

@hashier hashier marked this pull request as ready for review April 23, 2025 10:37
@hashier hashier force-pushed the feat/exclude-macOS-cloud-files branch from 52b221d to 1145076 Compare April 23, 2025 13:28
@hashier

hashier commented Apr 24, 2025

Copy link
Copy Markdown
Contributor Author

Checked MS OneDrive, it uses the same stat information for cloud only files so should just work with OneDrive too:

$ ls -lO%
-rw-r--r--% 1 me  staff  compressed,dataless 5 Apr 24 09:09 cloud
-rw-r--r--  1 me  staff  -                   5 Apr 24 09:09 local

@fronesis47

fronesis47 commented May 13, 2025

Copy link
Copy Markdown

I built this fork and did a bunch of testing on an M3 Mac running MacOS 15.4.1. Everything I tried work perfectly.

For already downloaded files, it worked just like restic always does. But cloud-only (dataless) files were consistently skipped without forcing the materialization of files.

This would be a great feature enhancement for restic. Currently I cannot use restic on the laptop I share with my wife, as we need to have the iCloud Drive "optimize" feature turned on, in order to avoid filling up the disc. This would be fantastic.

Kudos to @hashier for the great work here.

@hashier hashier mentioned this pull request Apr 13, 2025
Closed
greatroar
greatroar reviewed Jun 4, 2025
View reviewed changes
Comment thread internal/fs/stat_darwin.go
greatroar
greatroar reviewed Jun 4, 2025
View reviewed changes
Comment thread internal/fs/stat_darwin.go Outdated
MichaelEischer
MichaelEischer requested changes Oct 4, 2025
View reviewed changes

@MichaelEischer MichaelEischer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will only work on macOS 14.0 and newer source. Older versions of macOS will materialize the file when calling stat() on cloud files.

Are you sure that the old stub files trigger a download when restic accesses them? The article sounds like the use a special filetype which has to be converted by Finder or something else into a regular file.

Other than that, I've only found a few nits, see below.

Comment thread changelog/unreleased/issue-5352 Outdated
Comment thread changelog/unreleased/issue-5352 Outdated
Comment thread changelog/unreleased/issue-5352 Outdated
Comment thread internal/fs/stat_darwin.go Outdated
@hashier

hashier commented Oct 4, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

This will only work on macOS 14.0 and newer source. Older versions of macOS will materialize the file when calling stat() on cloud files.

Are you sure that the old stub files trigger a download when restic accesses them? The article sounds like the use a special filetype which has to be converted by Finder or something else into a regular file.

Can test this tomorrow on my old 2015 Mac that has an old version of macOS running.

Will resolve now the conflicts in a merge and push. Please let me know if you want this PR squashed/rebased instead.

Thanks a ton for the review, much appreciated.

@MichaelEischer

MichaelEischer commented Oct 4, 2025 •
edited
Loading

Copy link
Copy Markdown
Member

Will resolve now the conflicts in a merge and push. Please let me know if you want this PR squashed/rebased instead.

Please rebase and squash the PR. Squashing is optional, but at first glance the commits aren't structured that it's worth keeping the individual commits.

@hashier hashier force-pushed the feat/exclude-macOS-cloud-files branch from 158e39e to 9e5d48e Compare October 4, 2025 17:15
@hashier hashier force-pushed the feat/exclude-macOS-cloud-files branch from 9e5d48e to f3d9589 Compare October 4, 2025 17:23
@hashier

hashier commented Oct 4, 2025

Copy link
Copy Markdown
Contributor Author

Rebased and squashed. Should be all good now.

@hashier hashier requested a review from MichaelEischer October 5, 2025 12:16
MichaelEischer
MichaelEischer approved these changes Oct 5, 2025
View reviewed changes

@MichaelEischer MichaelEischer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

Did you test this with an older macOS version yet?

@hashier

hashier commented Oct 5, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@MichaelEischer I had time to check on my old system (macOS 12.7.6 Monterey) and you are right, there is a stub .<filename> file in the folder and cat-ing it on console just gives some plist information about the file but does not materialize it. My original comment was contradicting itself, the 2nd part said:

(On older macOS version restic should behave the same whether --exclude-cloud-files is provided or not)

I think this is true.

On new versions of macOS files get automatically materialized when acted upon them. So running restic backup in iCloud drive will materialize all files by default. (fun fact: this automatically materialization did not work for me when restic is run from LaunchAgent and the backup would give some errors for those files but would complete successfully with all files that are already local.)

@MichaelEischer MichaelEischer merged commit 157f174 into restic:master Nov 16, 2025
12 checks passed
@amrkmn amrkmn mentioned this pull request Jun 9, 2026
Closed
@BrewTestBot BrewTestBot mentioned this pull request Jun 9, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MichaelEischer MichaelEischer MichaelEischer approved these changes

+1 more reviewer

@greatroar greatroar greatroar left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Improve --exclude-cloud-files to ignore iCloud-only files on macOS

4 participants

@hashier @fronesis47 @MichaelEischer @greatroar