Enhancement: restic rewrite: allow include filters in command#5191
Enhancement: restic rewrite: allow include filters in command#5191MichaelEischer merged 6 commits intorestic:masterfrom
restic rewrite: allow include filters in command#5191Conversation
|
I have managed to crack the issue of empty subtrees. This change is incorporated in internal/walker/rewriter.go |
MichaelEischer
left a comment
There was a problem hiding this comment.
A few preliminary comments before you try to further work on this PR
|
I put this PR on hold until PR #5185 is fully resolved. |
360dc5b to
f6e0dac
Compare
MichaelEischer
left a comment
There was a problem hiding this comment.
Thanks for the changes so far! I have added quite a few comments, but those are mostly small changes and nitpicks. The general approach now looks like what I had in mind.
96cf546 to
9f63e52
Compare
9f63e52 to
2790d3b
Compare
|
I manually moved all the changed code to the new restic level 0.18.1. Otherwise all requested changes should still be incorporated, The only bit which is somewhat different is |
27358f5 to
5f36c80
Compare
|
Rebased changes onto current development tree. |
5f36c80 to
d6050b1
Compare
|
Rebased changes onto current development tree. |
restic rewrite: allow include filters in command
cmd/restic/cmd_rewrite.go: introduction of include filters for this command: - add include filters, add error checking code - add new parameter 'keepEmptyDirectoryFunc' to 'walker.NewSnapshotSizeRewriter()', so empty directories have to be kept to keep the directory structure intact - add parameter 'keepEmptySnapshot' to 'filterAndReplaceSnapshot()' to keep snapshots intact when nothing is to be included - introduce helper function 'gatherIncludeFilters()' and 'gatherExcludeFilters()' to keep code flow clean cmd/restic/cmd_rewrite_integration_test.go: add several new tests around the 'include' functionality internal/filter/include.go: this is where is include filter is defined internal/walker/rewriter.go: - struct RewriteOpts gains field 'KeepEmtpyDirectory', which is a 'NodeKeepEmptyDirectoryFunc()' which defaults to nil, so that al subdirectories are kept - function 'NewSnapshotSizeRewriter()' gains the parameter 'keepEmptyDirecoryFilter' which controls the management of empty subdirectories in case of include filters active internal/data/tree.go: gains a function Count() for checking the number if node elements in a newly built tree internal/walker/rewriter_test.go: function 'NewSnapshotSizeRewriter()' gets an additional parameter nil to keeps things happy cmd/restic/cmd_repair_snapshots.go: function 'filterAndReplaceSnapshot()' gets an additional parameter 'keepEmptySnapshot=nil' doc/045_working_with_repos.rst: gets to mention include filters changelog/unreleased/issue-4278: the usual announcement file git rebase master -i produced this restic rewrite include - keep linter happy cmd/restic/cmd_rewrite_integration_test.go: linter likes strings.Contain() better than my strings.Index() >= 0
convert exclusive lock on repository to 'no-lock'
added function Count() to the *TreeWriter methods
1e8a356 to
ee154ce
Compare
MichaelEischer
left a comment
There was a problem hiding this comment.
LGTM. I've added a few commits that include a bunch of smaller cleanups, add tests for KeepEmptyDirectory and change the node filtering to only include directories that have a chance that some file in a subdirectory matches. The latter can speed up rewrite quite a lot, when for example matching against /data or some other absolute path.
| err := testRunRewriteWithOpts(t, | ||
| RewriteOptions{ | ||
| Forget: true, | ||
| ExcludePatternOptions: filter.ExcludePatternOptions{Excludes: []string{"nothing-whatsoever"}}, |
There was a problem hiding this comment.
This test case doesn't test anything at the moment. It would have to test using pattern *. However, then the test fails as it is currently perfectly valid to just exclude all files in a snapshot. For simplicity I think we should just keep it at that and drop this testcase.
3f64687 to
901235e
Compare
|
Thanks. It was a long haul. |
What does this PR change? What problem does it solve?
This enhancement enables the use of include filters for the rewrite command. Instead of using complex logic to exclude everything else bar the files and directories which the user is interested in, it can now be defined as include "all the bits I am interested in".
File internal/filter/include.go gained an "Empty() function, and file internal/walker/rewriter.go gained logic to recognize empty sub-trees and suppresses the creation for include filtering.
cmd_rewite adds code for inclusion filters. Only sub-trees which are needed will be created.
Was the change previously discussed in an issue or on the forum?
closes #4278
Checklist
changelog/unreleased/that describes the changes for our users (see template).