Skip to content

Enhancement: restic rewrite: allow include filters in command#5191

Merged
MichaelEischer merged 6 commits intorestic:masterfrom
wplapper:cmd_rewrite_include
Feb 1, 2026
Merged

Enhancement: restic rewrite: allow include filters in command#5191
MichaelEischer merged 6 commits intorestic:masterfrom
wplapper:cmd_rewrite_include

Conversation

@wplapper
Copy link
Copy Markdown
Contributor

@wplapper wplapper commented Dec 23, 2024

What does this PR change? What problem does it solve?

This enhancement enables the use of include filters for the rewrite command. Instead of using complex logic to exclude everything else bar the files and directories which the user is interested in, it can now be defined as include "all the bits I am interested in".

File internal/filter/include.go gained an "Empty() function, and file internal/walker/rewriter.go gained logic to recognize empty sub-trees and suppresses the creation for include filtering.

cmd_rewite adds code for inclusion filters. Only sub-trees which are needed will be created.

Was the change previously discussed in an issue or on the forum?

closes #4278

Checklist

  • I have added tests for all code changes.
  • I have added documentation for relevant changes (in the manual).
  • There's a new file in changelog/unreleased/ that describes the changes for our users (see template).
  • I'm done! This pull request is ready for review.

@wplapper
Copy link
Copy Markdown
Contributor Author

I have managed to crack the issue of empty subtrees. This change is incorporated in internal/walker/rewriter.go

@wplapper wplapper marked this pull request as draft January 24, 2025 21:47
@wplapper wplapper marked this pull request as ready for review January 24, 2025 21:48
@wplapper wplapper marked this pull request as draft January 24, 2025 21:48
Copy link
Copy Markdown
Member

@MichaelEischer MichaelEischer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few preliminary comments before you try to further work on this PR

@wplapper
Copy link
Copy Markdown
Contributor Author

wplapper commented Feb 2, 2025

I put this PR on hold until PR #5185 is fully resolved.

@wplapper wplapper closed this Feb 2, 2025
@wplapper wplapper reopened this Feb 6, 2025
@wplapper wplapper marked this pull request as ready for review February 15, 2025 14:31
@wplapper wplapper force-pushed the cmd_rewrite_include branch from 360dc5b to f6e0dac Compare February 19, 2025 13:07
@wplapper wplapper changed the title Issue 4278: Enhancement: include filters in command rewrite restic rewrite: allow include filters in command Feb 19, 2025
Copy link
Copy Markdown
Member

@MichaelEischer MichaelEischer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes so far! I have added quite a few comments, but those are mostly small changes and nitpicks. The general approach now looks like what I had in mind.

@wplapper wplapper force-pushed the cmd_rewrite_include branch from 96cf546 to 9f63e52 Compare July 7, 2025 11:56
@wplapper wplapper changed the title restic rewrite: allow include filters in command restic rewrite: allow include filters in command - issue #4278. Aug 7, 2025
@wplapper wplapper force-pushed the cmd_rewrite_include branch from 9f63e52 to 2790d3b Compare October 18, 2025 07:13
@wplapper
Copy link
Copy Markdown
Contributor Author

wplapper commented Oct 18, 2025

I manually moved all the changed code to the new restic level 0.18.1. Otherwise all requested changes should still be incorporated, The only bit which is somewhat different is cmd_rewrite_integration_test.go.

@wplapper wplapper force-pushed the cmd_rewrite_include branch from 27358f5 to 5f36c80 Compare November 21, 2025 14:19
@wplapper
Copy link
Copy Markdown
Contributor Author

Rebased changes onto current development tree.

@wplapper wplapper force-pushed the cmd_rewrite_include branch from 5f36c80 to d6050b1 Compare December 28, 2025 16:53
@wplapper
Copy link
Copy Markdown
Contributor Author

Rebased changes onto current development tree.

@wplapper wplapper changed the title restic rewrite: allow include filters in command - issue #4278. Enhancement: restic rewrite: allow include filters in command Jan 10, 2026
cmd/restic/cmd_rewrite.go:
introduction of include filters for this command:
- add include filters, add error checking code
- add new parameter 'keepEmptyDirectoryFunc' to 'walker.NewSnapshotSizeRewriter()',
  so empty directories have to be kept to keep the directory structure intact
- add parameter 'keepEmptySnapshot' to 'filterAndReplaceSnapshot()' to keep snapshots
  intact when nothing is to be included
- introduce helper function 'gatherIncludeFilters()' and 'gatherExcludeFilters()' to
  keep code flow clean

cmd/restic/cmd_rewrite_integration_test.go:
add several new tests around the 'include' functionality

internal/filter/include.go:
this is where is include filter is defined

internal/walker/rewriter.go:
- struct RewriteOpts gains field 'KeepEmtpyDirectory', which is a 'NodeKeepEmptyDirectoryFunc()'
  which defaults to nil, so that al subdirectories are kept
- function 'NewSnapshotSizeRewriter()' gains the parameter 'keepEmptyDirecoryFilter' which
  controls the management of empty subdirectories in case of include filters active

internal/data/tree.go:
gains a function Count() for checking the number if node elements in a newly built tree

internal/walker/rewriter_test.go:
function 'NewSnapshotSizeRewriter()' gets an additional parameter nil to keeps things happy

cmd/restic/cmd_repair_snapshots.go:
function 'filterAndReplaceSnapshot()' gets an additional parameter 'keepEmptySnapshot=nil'

doc/045_working_with_repos.rst:
gets to mention include filters

changelog/unreleased/issue-4278:
the usual announcement file

git rebase master -i produced this

restic rewrite include - keep linter happy

cmd/restic/cmd_rewrite_integration_test.go:
linter likes strings.Contain() better than my strings.Index() >= 0
convert exclusive lock on repository to 'no-lock'
added function Count() to the *TreeWriter methods
@wplapper wplapper force-pushed the cmd_rewrite_include branch from 1e8a356 to ee154ce Compare January 31, 2026 19:59
Copy link
Copy Markdown
Member

@MichaelEischer MichaelEischer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I've added a few commits that include a bunch of smaller cleanups, add tests for KeepEmptyDirectory and change the node filtering to only include directories that have a chance that some file in a subdirectory matches. The latter can speed up rewrite quite a lot, when for example matching against /data or some other absolute path.

err := testRunRewriteWithOpts(t,
RewriteOptions{
Forget: true,
ExcludePatternOptions: filter.ExcludePatternOptions{Excludes: []string{"nothing-whatsoever"}},
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test case doesn't test anything at the moment. It would have to test using pattern *. However, then the test fails as it is currently perfectly valid to just exclude all files in a snapshot. For simplicity I think we should just keep it at that and drop this testcase.

@wplapper
Copy link
Copy Markdown
Contributor Author

wplapper commented Feb 1, 2026

Thanks. It was a long haul.

@MichaelEischer MichaelEischer merged commit 07d380d into restic:master Feb 1, 2026
27 of 33 checks passed
@wplapper wplapper deleted the cmd_rewrite_include branch February 1, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Include" switch for rewrite command

2 participants