backup: allow excluding online-only cloud files by m-wild · Pull Request #4990 · restic/restic

m-wild · 2024-08-08T11:17:50Z

What does this PR change? What problem does it solve?

Allow excluding online-only cloud files (such as OneDrive Files On-Demand) with a switch --exclude-cloud-files

Assuming we have the following file test.txt in OneDrive, but not available on the device

Attempting to backup the file with the --exclude-cloud-files switch will not attempt to download the file locally, instead the file will be excluded.

restic -r test --verbose backup C:\Users\username\OneDrive\test.txt --exclude-cloud-files

open repository
repository e5f7dfde opened (version 2, compression level auto)
created new cache in C:\Users\username\AppData\Local\restic
no parent snapshot found, will read all files
load index files
[0:00]          0 index files loaded
start scan on [C:\Users\username\OneDrive\test.txt]
start backup on [C:\Users\username\OneDrive\test.txt]
scan finished in 0.003s: 0 files, 0 B

Files:           0 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      5 new
Added to the repository: 2.425 KiB (1.841 KiB stored)

processed 0 files, 0 B in 0:00
snapshot 5ef1ff96 saved

2024/08/08 22:39:41 archiver/tree.go:298	archiver.NewTree	1	result:
/C, root "C:\\", path "", meta "C:\\"
    /Users, root "", path "", meta "C:\\Users"
        /username, root "", path "", meta "C:\\Users\\username"
            /OneDrive, root "", path "", meta "C:\\Users\\username\\OneDrive"
                /test.txt, root "", path "C:\\Users\\username\\OneDrive\\test.txt", meta ""

...

2024/08/08 22:39:44 archiver/archiver.go:420	archiver.(*Archiver).save	40	/C/Users/username/OneDrive/test.txt target "C:\\Users\\username\\OneDrive\\test.txt", previous <nil>
2024/08/08 22:39:44 restic/exclude.go:404	main.collectRejectFuncs.rejectCloudFiles.func1	40	rejecting online-only cloud file C:\Users\username\OneDrive\test.txt
2024/08/08 22:39:44 archiver/archiver.go:443	archiver.(*Archiver).save	40	C:\Users\username\OneDrive\test.txt is excluded

Was the change previously discussed in an issue or on the forum?

Closes #3697

Checklist

I have read the contribution guidelines.
I have enabled maintainer edits.
I have added tests for all code changes.
I have added documentation for relevant changes (in the manual).
There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
I have run gofmt on the code in all commits.
All commit messages are formatted in the same style as the other commits in the repo.
I'm done! This pull request is ready for review.

cmd/restic/exclude.go

m-wild · 2024-08-08T11:33:31Z

@MichaelEischer I'm not really sure how to add tests for this one. It would require either manually setting the file attributes or somehow setting up OneDrive on Windows?

cmd/restic/cmd_backup.go

cmd/restic/exclude.go

deajan · 2024-09-07T16:28:39Z

@m-wild I really love that PR. That's exactly what I'm searching for more than a year at #4155
I'll happily make some tests with some beta builds if needed.

greatroar · 2024-10-04T13:47:15Z

There's already an option --one-file-system. Can this feature use that option, instead of introducing a new one? Or does it already have a meaning on Windows?

MichaelEischer · 2024-10-16T19:32:50Z

There's already an option --one-file-system. Can this feature use that option, instead of introducing a new one? Or does it already have a meaning on Windows?

--one-file-system currently has no effect on Windows. However, Windows does support junctions which can be used to mount other filesystems within a filesystem. My expectation would be that --one-file-system once it has an effect, addresses exactly this scenario.

Cloud files feel somewhat different: they are placeholders for files that can basically show up everywhere in the filesystem. I wouldn't really expect that --one-file-system excludes those files.

deajan · 2024-10-24T00:41:46Z

@m-wild Any news on this one perhaps ?

MichaelEischer · 2024-11-11T21:03:30Z

If someone is interested in taking over this PR, please just create a new PR, then I'll close this one.

deajan · 2024-11-12T12:04:02Z

@m-wild @MichaelEischer I'll happily sponsor whoever wants to pick this up ;)

deajan · 2024-12-17T08:15:36Z

@m-wild Do you still want to work on this ?

m-wild · 2024-12-19T12:52:42Z

@m-wild Do you still want to work on this ?

Planning to pick it up over the Xmas break

m-wild · 2024-12-24T14:38:49Z

Ho ho ho, it's a Christmas miracle, a PR ready to review.

deajan · 2024-12-24T14:40:30Z

Ho ho ho, it's a Christmas miracle, a PR ready to review.

Ho ho ho, ping me whenever this is merged so I can sponsor you a bit ;)

m-wild · 2025-01-11T21:51:57Z

Have just fixed the lint issues.

MichaelEischer

Thanks for updating the PR! I'm mostly happy with it except for a few nits.

internal/fs/file_windows.go

internal/fs/file_windows_test.go

changelog/unreleased/issue-3697

MichaelEischer

LGTM.

MichaelEischer · 2025-01-13T20:11:55Z

(Force pushed to fix the linter warning)

deajan · 2025-01-20T13:08:55Z

Can confirm that the new option --exclude-cloud-files works great with a test nextcloud setup using current restic beta binary.
Thank you for that great work.

@m-wild As I suggested earlier, I'd like to sponsor you a bit. Do you have any donation system setup perhaps ?

m-wild · 2025-01-20T13:15:41Z

@deajan thanks for testing the beta :) I have GitHub sponsorship enabled.

deajan · 2025-01-20T14:09:08Z

@m-wild Done ;) Thank you again.

m-wild commented Aug 8, 2024

View reviewed changes

cmd/restic/exclude.go Outdated Show resolved Hide resolved

m-wild force-pushed the exclude-cloud-files branch from 554229d to 1504042 Compare August 8, 2024 11:29

m-wild marked this pull request as ready for review August 8, 2024 11:31

MichaelEischer requested changes Aug 9, 2024

View reviewed changes

cmd/restic/cmd_backup.go Outdated Show resolved Hide resolved

cmd/restic/exclude.go Outdated Show resolved Hide resolved

MichaelEischer mentioned this pull request Sep 6, 2024

Refactor and cleanup fs code from internal/restic #5021

Open

11 tasks

m-wild force-pushed the exclude-cloud-files branch from 1504042 to 9b69e17 Compare December 24, 2024 14:28

m-wild requested a review from MichaelEischer December 24, 2024 14:38

m-wild force-pushed the exclude-cloud-files branch from 9b69e17 to fbe76e9 Compare January 11, 2025 21:51

MichaelEischer requested changes Jan 12, 2025

View reviewed changes

internal/fs/file_windows.go Outdated Show resolved Hide resolved

internal/fs/file_windows_test.go Outdated Show resolved Hide resolved

changelog/unreleased/issue-3697 Show resolved Hide resolved

m-wild force-pushed the exclude-cloud-files branch from fbe76e9 to a7539ff Compare January 12, 2025 16:32

m-wild requested a review from MichaelEischer January 12, 2025 16:34

MichaelEischer approved these changes Jan 13, 2025

View reviewed changes

MichaelEischer enabled auto-merge January 13, 2025 20:07

backup: allow excluding online-only cloud files

da71e77

MichaelEischer force-pushed the exclude-cloud-files branch from a7539ff to da71e77 Compare January 13, 2025 20:11

MichaelEischer merged commit bd4ce8a into restic:master Jan 13, 2025

deajan mentioned this pull request Jan 20, 2025

Windows - Reparse points / cloud files creating a lot of errors in logs #4155

Closed

m-wild deleted the exclude-cloud-files branch January 20, 2025 13:15

BrewTestBot mentioned this pull request Mar 27, 2025

restic 0.18.0 Homebrew/homebrew-core#216856

Merged

hashier mentioned this pull request Apr 10, 2025

Improve --exclude-cloud-files to ignore iCloud-only files on macOS #5352

Closed

m-wild mentioned this pull request May 15, 2025

Support --exclude-cloud-files flag creativeprojects/resticprofile#499

Closed

Conversation

m-wild commented Aug 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR change? What problem does it solve?

Was the change previously discussed in an issue or on the forum?

Checklist

Uh oh!

Uh oh!

m-wild commented Aug 8, 2024

Uh oh!

Uh oh!

Uh oh!

deajan commented Sep 7, 2024

Uh oh!

greatroar commented Oct 4, 2024

Uh oh!

MichaelEischer commented Oct 16, 2024

Uh oh!

deajan commented Oct 24, 2024

Uh oh!

MichaelEischer commented Nov 11, 2024

Uh oh!

deajan commented Nov 12, 2024

Uh oh!

deajan commented Dec 17, 2024

Uh oh!

m-wild commented Dec 19, 2024

Uh oh!

m-wild commented Dec 24, 2024

Uh oh!

deajan commented Dec 24, 2024

Uh oh!

m-wild commented Jan 11, 2025

Uh oh!

MichaelEischer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaelEischer left a comment

Choose a reason for hiding this comment

Uh oh!

MichaelEischer commented Jan 13, 2025

Uh oh!

deajan commented Jan 20, 2025

Uh oh!

m-wild commented Jan 20, 2025

Uh oh!

deajan commented Jan 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

m-wild commented Aug 8, 2024 •

edited

Loading