Skip to content

fvec: add mmap based vector #1662

Merged
BareosBot merged 22 commits intobareos:masterfrom
sebsura:dev/ssura/master/fvec
Feb 13, 2024
Merged

fvec: add mmap based vector #1662
BareosBot merged 22 commits intobareos:masterfrom
sebsura:dev/ssura/master/fvec

Conversation

@sebsura
Copy link
Contributor

@sebsura sebsura commented Jan 15, 2024

Thank you for contributing to the Bareos Project!

This PR adds a utility class that resembles a vector, but it does not use regular memory but instead uses a memory mapped file instead.

This structure is needed to implement the dedup backend.

Please check

  • Short description and the purpose of this PR is present above this paragraph
  • Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General
  • Is the PR title usable as CHANGELOG entry?
  • Purpose of the PR is understood
  • Commit descriptions are understandable and well formatted
  • Check backport line
  • Required backport PRs have been created
Source code quality
  • Source code changes are understandable
  • Variable and function names are meaningful
  • Code comments are correct (logically and spelling)
  • Required documentation changes are present and part of the PR
Tests
  • Decision taken that a test is required (if not, then remove this paragraph)
  • The choice of the type of test (unit test or systemtest) is reasonable
  • Testname matches exactly what is being tested
  • On a fail, output of the test leads quickly to the origin of the fault

@sebsura sebsura mentioned this pull request Jan 15, 2024
14 tasks
@sebsura sebsura force-pushed the dev/ssura/master/fvec branch 3 times, most recently from e842a5e to fe37ecf Compare January 17, 2024 09:30
@pstorz pstorz self-requested a review January 18, 2024 10:29
@pstorz pstorz self-assigned this Jan 18, 2024
@sebsura sebsura force-pushed the dev/ssura/master/fvec branch 2 times, most recently from f6bbb2b to 911f781 Compare January 30, 2024 12:45
@pstorz pstorz changed the title fvec: add file based vector fvec: add mmap based vector Feb 2, 2024
@sebsura sebsura force-pushed the dev/ssura/master/fvec branch from 911f781 to d8ed4dc Compare February 6, 2024 12:47
@sebsura sebsura mentioned this pull request Feb 6, 2024
6 tasks
This allows one to easily append multiple elements to the end at
once.
When handling files we do not want to grow too fast or too slow since
its a) wasteful or b) slow.
The restore result was not checked before
This construct enters into a deadlock if
1) the writer is waiting because the queue is full
2) the reader is waiting on the writer to quit

The reason is that this status is not communicated through the queue
but with a separate variable.

This commit makes it so the status is communicated only through the
queue itself.  Special note has to be taken in the case of a timeout
in the reader (res == fd->Timeout) since normally this branch does not
look at the queue at all.

Because of this we needed to add a function the the channel that
actively checks whether the other side is still alive.
utime_t (defined in bc_types.h) is needed to use this header.
This also fixes an issue that occurs if mmap/mremap fails.  In that
case buffer should be reset to nullptr, so that the destructor does
not try to unmap it a second time.
The test did not support each subtest getting called on its own (since
they shared state).  To fix this each test now gets its own file, as
well as each test setting up the file how they need it to be.
we use ftruncate if posix_fallocate does not work
It turns out that posix_fallocate can become very slow on glibc if you
the filesystem does not really support it since the fallback is just
writing a bunch of zeroes into the file.   This is not what we want so
we only use ftruncate for now.
@sebsura sebsura force-pushed the dev/ssura/master/fvec branch from d8ed4dc to a8cc628 Compare February 12, 2024 10:49
We want to use huge pages if possible to reduce the number of page
faults.
We first try to extend the first mapping instead of doing the whole
unmark + mark step.
@sebsura sebsura force-pushed the dev/ssura/master/fvec branch from a8cc628 to a51168e Compare February 13, 2024 08:03
@BareosBot BareosBot merged commit cd082f1 into bareos:master Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants