Skip to content

cli --min-packsize and --file-read-concurrency#2671

Closed
madsi1m wants to merge 9 commits intorestic:masterfrom
madsi1m:master
Closed

cli --min-packsize and --file-read-concurrency#2671
madsi1m wants to merge 9 commits intorestic:masterfrom
madsi1m:master

Conversation

@madsi1m
Copy link
Copy Markdown

@madsi1m madsi1m commented Mar 26, 2020

What is the purpose of this change? What does it change?

Would like to add the following in order to tune restic rather than hardcoding it.
--min-packsize
--file-read-concurrency

Was the change discussed in an issue or in the forum before?

Nope

Checklist

  • I have read the Contribution Guidelines
  • I have enabled maintainer edits for this PR
  • I have added tests for all changes in this PR
  • I have added documentation for the changes (in the manual)
  • There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
  • I have run gofmt on the code in all commits
  • All commit messages are formatted in the same style as the other commits in the repo
  • I'm done, this Pull Request is ready for review

@dimejo
Copy link
Copy Markdown
Contributor

dimejo commented Mar 27, 2020

While I understand the need to tune such options I think they require proper documentation. People tend to tune such options to the extreme which might cause bottlenecks or performance issues and result in a lot of questions and complains on the forum. Hence the documentation should cover these questions:

  • When is it necessary to tune these options?
  • What are sane maximum values?
  • What happens if these options are set too high (e.g. --file-read-concurrency 100)?
  • Is it possible to change --min-packsize between 2 backup runs? What are the impacts?

Edit for reference: https://forum.restic.net/t/control-the-minimal-pack-files-size/617

@MichaelEischer
Copy link
Copy Markdown
Member

Does --file-read-concurrency have to be a global-option? Right now it's exclusively used by the backup command. --min-packsize is also not used by that many commands, actually just by backup, prune and forget --prune. But I guess that makes the option simliar to --limit-upload.

@madsi1m
Copy link
Copy Markdown
Author

madsi1m commented Apr 5, 2020

--file-read-concurrency doesn't need to be a global option, you are quite right it is only used by the backup command.

@@ -0,0 +1,5 @@
Feature: Addition of --min-packsize and --file-read-concurrency
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Judging from the changelog template https://github.com/restic/restic/blob/master/changelog/TEMPLATE this should read Add [...] flags

@@ -0,0 +1,5 @@
Feature: Addition of --min-packsize and --file-read-concurrency

In order to tune restic for large installs we need to be able to configure --min-packsize and --file-read-concurrency. In version 0.9.6 these numbers are hard coded, this pull request allows you to adjust these.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changelog usually doesn't refer explicitly to previous versions. The description should also be written in past tense. There's also no need to mention that this was change via a pull request. (Sorry for the nitpicking)

f.BoolVar(&globalOptions.CleanupCache, "cleanup-cache", false, "auto remove old cache directories")
f.IntVar(&globalOptions.LimitUploadKb, "limit-upload", 0, "limits uploads to a maximum rate in KiB/s. (default: unlimited)")
f.IntVar(&globalOptions.LimitDownloadKb, "limit-download", 0, "limits downloads to a maximum rate in KiB/s. (default: unlimited)")
f.UintVar(&globalOptions.MinPackSize, "min-packsize", 0, "set min pack size in MiB. (default: $RESTIC_MIN_PACKSIZE or 4)")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you put backticks '`' around the word size then cobra will use that to name the flag parameter.

@metalsp0rk
Copy link
Copy Markdown
Contributor

@madsi1m I'd love to see this included. Do you have the bandwidth to finish this up, or would you mind if I took over?

@madsi1m
Copy link
Copy Markdown
Author

madsi1m commented May 17, 2020

@metalsp0rk sadly I don't have the bandwidth at the moment. Go for it. Would be great to have this in master 👍

@metalsp0rk
Copy link
Copy Markdown
Contributor

No worries, I just didn't want to step on any toes. 👍

@metalsp0rk
Copy link
Copy Markdown
Contributor

I've submitted a new PR for this, #2750

@madsi1m
Copy link
Copy Markdown
Author

madsi1m commented May 24, 2020

cool

@aawsome
Copy link
Copy Markdown
Contributor

aawsome commented Jun 6, 2020

What about closing this now that #2750 is almost finished?

@MichaelEischer
Copy link
Copy Markdown
Member

Closing this in favor of #2750.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants