stored: dird: add backup checkpoints that save backup metadata to the Catalog during the execution of the backup#1074
Merged
pstorz merged 46 commits intobareos:masterfrom Sep 19, 2022
Conversation
0b09d14 to
ca5487a
Compare
b004c04 to
3309146
Compare
pstorz
requested changes
Mar 7, 2022
Member
pstorz
left a comment
There was a problem hiding this comment.
Very good work.
However, I see the following things that need to be tested in a system test:
1: Kill the filedaemon during the backup and check that the last successful checkpoint can be recovered.
2: The same with killing the storage daemon.
3: The same with the director.
4: We also need to measure the impact on backup speeed with a big amount data both to disk and to tape.Depending on the results we can also define a suggested value for the checkpoint interval.
5: The documentation of the feature needs to be written based on the results of the testing..
7b9988d to
073843c
Compare
a1150c6 to
3d34d74
Compare
added 29 commits
September 14, 2022 09:46
Moved attribute check to SendAttrsToDir callers
Clarified file index checks
This tests is disabled by default, and needs to be manually adjusted to feed with custom made data
update append_test
For the moment, we only enable any type of checkpoints (on cancel, on volume change, or on time interval) when the interval value is set.
In previous commits, we had to modify the `UpdateJobEndRecord` function in cats in order to mitigate an update issue that would overwrite the last jobbytes and jobfiles values with zeros. With this commit, we can keep the original `UpdateJobEndRecord` as it is while keeping the data intact
use expect_grep fix for solaris cancel with jobid
small refactor of checkpoint function headers
remove `set -e`
s wait 60 s wesh s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When doing backup jobs, backup metadata (number of files written to volume, number of bytes written, etc.) is only saved on a successful job. On a failed or canceled job, users end up with volumes filled with backed up data, but information about this data is not available in the catalog, and therefore already backed up data before the fail cannot be restored using the restore command.
This PR addresses the issue by introducing checkpoints during a backup's execution that save necessary metadata (updating File, Job, and Jobmedia tables) that enables users to restore data even for failed or canceled jobs. Said checkpoints happen on volume switches, or after a certain interval of time set by the user in the SD configuration.
Please check
If you have any questions or problems, please give a comment in the PR.
Helpful documentation and best practices
Checklist for the reviewer of the PR (will be processed by the Bareos team)
General
Source code quality
bareos-check-sources --since-mergedoes not report any problemsgit statusshould not report modifications in the source tree after building and testingTests