fvec: add mmap based vector #1662
Merged
BareosBot merged 22 commits intobareos:masterfrom Feb 13, 2024
Merged
Conversation
14 tasks
e842a5e to
fe37ecf
Compare
f6bbb2b to
911f781
Compare
pstorz
approved these changes
Feb 2, 2024
911f781 to
d8ed4dc
Compare
This allows one to easily append multiple elements to the end at once.
When handling files we do not want to grow too fast or too slow since its a) wasteful or b) slow.
The restore result was not checked before
This construct enters into a deadlock if 1) the writer is waiting because the queue is full 2) the reader is waiting on the writer to quit The reason is that this status is not communicated through the queue but with a separate variable. This commit makes it so the status is communicated only through the queue itself. Special note has to be taken in the case of a timeout in the reader (res == fd->Timeout) since normally this branch does not look at the queue at all. Because of this we needed to add a function the the channel that actively checks whether the other side is still alive.
utime_t (defined in bc_types.h) is needed to use this header.
This also fixes an issue that occurs if mmap/mremap fails. In that case buffer should be reset to nullptr, so that the destructor does not try to unmap it a second time.
The test did not support each subtest getting called on its own (since they shared state). To fix this each test now gets its own file, as well as each test setting up the file how they need it to be.
we use ftruncate if posix_fallocate does not work
It turns out that posix_fallocate can become very slow on glibc if you the filesystem does not really support it since the fallback is just writing a bunch of zeroes into the file. This is not what we want so we only use ftruncate for now.
d8ed4dc to
a8cc628
Compare
We want to use huge pages if possible to reduce the number of page faults.
We first try to extend the first mapping instead of doing the whole unmark + mark step.
a8cc628 to
a51168e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for contributing to the Bareos Project!
This PR adds a utility class that resembles a vector, but it does not use regular memory but instead uses a memory mapped file instead.
This structure is needed to implement the dedup backend.
Please check
If you have any questions or problems, please give a comment in the PR.
Helpful documentation and best practices
Checklist for the reviewer of the PR (will be processed by the Bareos team)
Make sure you check/merge the PR using
devtools/pr-toolto have some simple automated checks run and a proper changelog record added.General
Source code quality
Tests