Skip to content

prune/rebuild-index: Add warmup possibilities#2881

Closed
aawsome wants to merge 3 commits intorestic:masterfrom
aawsome:prune-warmup
Closed

prune/rebuild-index: Add warmup possibilities#2881
aawsome wants to merge 3 commits intorestic:masterfrom
aawsome:prune-warmup

Conversation

@aawsome
Copy link
Copy Markdown
Contributor

@aawsome aawsome commented Aug 7, 2020

What does this PR change? What problem does it solve?

Adds the possibility to access all packs which need a repack during prune and rebuild.index
This allows some cold storages to warm up these packs such that they are all accessable. This warm-up can also be done within a dry run such that the needed packs are available for the next prune run.
Also the --json option is now implemented for prune dry-run. This allows to build custom warm-up processes if the pure access to files doesn't do the warmup.

This can be easily extended to restore, see #2796.

Was the change discussed in an issue or in the forum before?

The idea comes from #2796.
For cold storage discussions see also #2504

I didn't see a discussion about pruning cold storage, but this PR proved to be very useful for pruning my OVH Cloud Archive repositories.

Checklist

  • I have read the Contribution Guidelines
  • I have enabled maintainer edits for this PR
  • I have added tests for all changes in this PR
  • I have added documentation for the changes (in the manual)
  • There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
  • I have run gofmt on the code in all commits
  • All commit messages are formatted in the same style as the other commits in the repo
  • I'm done, this Pull Request is ready for review

@aawsome aawsome changed the title Add warmup possibility for packs-to-repack in prune Add warmup possibilities in prune Nov 7, 2020
@aawsome aawsome changed the title Add warmup possibilities in prune prune: Add warmup possibilities Dec 29, 2020
@aawsome aawsome changed the title prune: Add warmup possibilities prune/rebuild-index: Add warmup possibilities Jan 3, 2021
@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Jan 3, 2021

I rebased to master and added to --warm-up option to rebuild-index.

@rawtaz
Copy link
Copy Markdown
Contributor

rawtaz commented Jan 3, 2021

The more options there are, the more confusing it is for users. We want to keep the number of options as small as possible. Please try to not introduce new options unless really necessary. Is it really necessary in this case, and if so why can't it be the default behavior instead?

@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Jan 3, 2021

Is it really necessary in this case, and if so why can't it be the default behavior instead?

See #3202. This is only relevant if "data pack files" cannot be accessed immediately. If this is the case, we need to distinguish the "warm-up" run and the real run, where the needed files are accessible (which can be hours later that the warm-up run).

Of course, we could decide not to add the --warm-up option at all and instead only give a verbose JSON output for --dry-run --json. This would mean that we only support the workaround where the users create scripts to do the warmup themselves and the option for warmup would be effectively the --dry-run option.

@aawsome
Copy link
Copy Markdown
Contributor Author

aawsome commented Aug 13, 2022

see also rustic-rs/rustic#121

@aawsome aawsome closed this Feb 24, 2024
@aawsome aawsome deleted the prune-warmup branch February 24, 2024 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants