filed: parallel compression/reading/sending#1533
Merged
BareosBot merged 31 commits intobareos:masterfrom Nov 6, 2023
Merged
Conversation
8ea29bc to
b8bffe5
Compare
arogge
pushed a commit
to bareos-testing/bareos
that referenced
this pull request
Aug 29, 2023
If leftover snapshots exist from previous backups, they are now cleaned up before taking a new snapshot. Additionally, the snapshot cleanup after backup now works more reliable by reconnecting to the vCenter server if necessary. Code refactored to consequently use methods from pyVim.task. - fix a bug when folder=/ is used. Fixes: bareos#1533: Restoring vmware vms keeps failing, can't restore the data - Fix restore when bootOrder was set When bootOrder was set to a virtualDisk at backup time, ensure the restore works by setting the bootOrder now after disks were added to the recreated VM. - Fix restore when virtualCdrom path is invalid Restore fails when recreating a VM and the backing file of a virtual CDROM is inaccessible, this can occur when it was connected to an ISO file on a NFS datastore which is not accessible at restore time. This change will then create a disconnected virtual CDROM to allow the restore to finish successfully. Co-authored-by: Philipp Storz <philipp.storz@bareos.com>
sebsura
pushed a commit
to sebsura/bareos
that referenced
this pull request
Aug 31, 2023
If leftover snapshots exist from previous backups, they are now cleaned up before taking a new snapshot. Additionally, the snapshot cleanup after backup now works more reliable by reconnecting to the vCenter server if necessary. Code refactored to consequently use methods from pyVim.task. - fix a bug when folder=/ is used. Fixes: bareos#1533: Restoring vmware vms keeps failing, can't restore the data - Fix restore when bootOrder was set When bootOrder was set to a virtualDisk at backup time, ensure the restore works by setting the bootOrder now after disks were added to the recreated VM. - Fix restore when virtualCdrom path is invalid Restore fails when recreating a VM and the backing file of a virtual CDROM is inaccessible, this can occur when it was connected to an ISO file on a NFS datastore which is not accessible at restore time. This change will then create a disconnected virtual CDROM to allow the restore to finish successfully. Co-authored-by: Philipp Storz <philipp.storz@bareos.com> (cherry picked from commit 0e7f740)
b8bffe5 to
88b37ef
Compare
1704007 to
70e2ab5
Compare
arogge
requested changes
Sep 20, 2023
Member
arogge
left a comment
There was a problem hiding this comment.
I have a few remarks already. However, I guess I don't fully understand everything yet. So maybe some of the remarks are a bit "dull"...
70e2ab5 to
6b0f8ad
Compare
c808c61 to
ac057cf
Compare
arogge
reviewed
Oct 12, 2023
5ee59b7 to
2f2ebda
Compare
arogge
requested changes
Oct 19, 2023
Member
arogge
left a comment
There was a problem hiding this comment.
Looks pretty complicated, but great!
I have some minor remarks, am still not happy with the parameter naming and you have an endianess issue.
fea9cc3 to
1383a99
Compare
arogge
approved these changes
Nov 3, 2023
683699b to
34e6e31
Compare
Because the jcr is not fully created after new_jcr (the daemon specific pointer is still null), it is not 100% safe to register it since a lot of jcr-chain-walkers expect that pointer to be set when they walk the chain. This introduces a new register_jcr function which handles just the registering so that it can be done after the initialisation is actually over.
This should make it easier to implement parallelisation.
When we want to to this in parallel we clearly cannot use the buffers in bctx but have to allocate our own. As preperation for that we do that even now in serial mode.
Instead of reading the file into the send buffer directly, we read it into its own buffer that is shared with every worker thread. Then we either compress the file data into the send buffer or just memcpy it -- in case compression is not enabled -- in a worker thread.
Previously thread_pool took care of two different concerns: 1) Being able to reuse threads for work 2) Sharing threads for concurrent non-blocking computations To be able to support both of these concerns we needed to add a lot of complexity to the class. Splitting it into two, a thread_pool, that only lets us borrow threads for a while, and a work group, with which you can share threads to do non-blocking computations, made the code much easier to understand.
Also adds zero-copy sending to the non-compression path.
34e6e31 to
edd1b8e
Compare
arogge
reviewed
Nov 6, 2023
9cd2812 to
24cab2d
Compare
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for contributing to the Bareos Project!
Instead of serially doing read chunk -> compress chunk -> send chunk we instead try to do as much as possible in parallel.
Please check
If you have any questions or problems, please give a comment in the PR.
Helpful documentation and best practices
Checklist for the reviewer of the PR (will be processed by the Bareos team)
Make sure you check/merge the PR using
devtools/pr-toolto have some simple automated checks run and a proper changelog record added.General
Source code quality
Tests