stored: change default block size to 1 MiB#1396
Merged
arogge merged 10 commits intobareos:masterfrom Jun 7, 2023
Merged
Conversation
6ca926c to
38f11b6
Compare
pstorz
requested changes
Apr 26, 2023
Member
pstorz
left a comment
There was a problem hiding this comment.
Good work! Please see comments.
...nuals/source/manually_added_config_directive_descriptions/sd-device-MinimumBlockSize.rst.inc
Outdated
Show resolved
Hide resolved
pstorz
requested changes
May 30, 2023
Member
pstorz
left a comment
There was a problem hiding this comment.
Only two points are left:
1: Adapt the documentation with your last suggestion of the sentence
2: Merge the latest docs commits into one.
Good work!
6339634 to
5cef926
Compare
pstorz
approved these changes
Jun 1, 2023
move initialization from default ctor to the class itself.
Historically Bareos uses a default block size of 63k (64.512 bytes). This usually works fine, until you use a modern tape drive. We already have a whilepaper on how to achieve good performance with tape drives, but you need to reconfigure the block sizes and there are a lot of ways you can break Bareos that way. With the default block size set to 1M (1.048.576 bytes), there is no need to configure a custom block size for tapes anymore, as this new default now provides optimum performance for recent tape drives out of the box. The default has been chosen carefully to provide optimum performance while retaining maximum interoperability. As there are still hostadapters (e.g. smartpqi) that cannot handle I/O larger than 1M and tape performance does not show significant increase when using a blocksize larger than 1M, this seems to be the reasonable choice.
Using gentestdata you can generate a pseudo-random binary data that you can use during testing. This will produce the same data every time, but it looks random and the data is neither sparse nor compressible.
The checkpoint test was relying on smaller blocksizes, so it had to be adapted to the new 1M blocksize. We now also create special testdata for that with the new gentestdata tool and use that for the tests.
When taking a backup that fits entirely in a single block (i.e. the SOS and the EOS together with at least one file are in the same block), the SD would write the EOS record with JobFiles=0. If you bscan such a job, restore will not be possible. This patch updates JobFiles immediately before writing the EOS, ensuring it is not zero if a file was actually saved.
d24b2b4 to
39c4cba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Historically Bareos uses a default block size of 63k (64.512 bytes). This usually works fine, until you use a modern tape drive. We already have a whilepaper on how to achieve good performance with tape drives, but you need to reconfigure the block sizes and there are a lot of ways you can break Bareos that way.
With the default block size set to 1M (1.048.576 bytes), there is no need to configure a custom block size for tapes anymore, as this new default now provides optimum performance for recent tape drives out of the box.
The default has been chosen carefully to provide optimum performance while retaining maximum interoperability. As there are still hostadapters (e.g. smartpqi) that cannot handle I/O larger than 1M and tape performance does not show significant increase when using a blocksize larger than 1M, this seems to be the reasonable choice.
Please check
If you have any questions or problems, please give a comment in the PR.
Helpful documentation and best practices
Checklist for the reviewer of the PR (will be processed by the Bareos team)
Make sure you check/merge the PR using
devtools/pr-toolto have some simple automated checks run and a proper changelog record added.General
Source code quality
Tests