Skip to content

stored: add dedupable backend#1663

Merged
BareosBot merged 47 commits intobareos:masterfrom
sebsura:dev/ssura/fvec/dedup
Sep 13, 2024
Merged

stored: add dedupable backend#1663
BareosBot merged 47 commits intobareos:masterfrom
sebsura:dev/ssura/fvec/dedup

Conversation

@sebsura
Copy link
Contributor

@sebsura sebsura commented Jan 15, 2024 •
edited by pstorz
Loading

Thank you for contributing to the Bareos Project!

The dedup backend makes it possible for deduplicating filesystems to deduplicate backed up data.

Based of pr #1662

Please check

  • Short description and the purpose of this PR is present above this paragraph
  • Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General
  • Is the PR title usable as CHANGELOG entry?
  • Purpose of the PR is understood
  • Commit descriptions are understandable and well formatted
    Check backport line
  • Required backport PRs have been created
Source code quality
  • Source code changes are understandable
  • Variable and function names are meaningful
  • Code comments are correct (logically and spelling)
  • Required documentation changes are present and part of the PR
Tests
  • Decision taken that a test is required (if not, then remove this paragraph)
  • The choice of the type of test (unit test or systemtest) is reasonable
  • Testname matches exactly what is being tested
  • On a fail, output of the test leads quickly to the origin of the fault

@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch from 5bfa1f6 to cb416e8 Compare January 17, 2024 07:10
@sebsura sebsura marked this pull request as draft January 17, 2024 07:16
@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch 2 times, most recently from 095f8f7 to 2c6f955 Compare January 17, 2024 10:20
@sebsura
Copy link
Contributor Author

sebsura commented Jan 17, 2024

One thing i was not sure about was the error reporting. Currently the device itself outputs error messages whenever they are caught and prints an Emsg with it. This makes it easier to give good error reports but there will probably be a lot of double reports ala:

dedup device : could not open dedup volume. ERR=...
normal bareos device: could not open dedup volume. ERR=...

An alternative would be to set errno appropriately (printing a dmsg instead) and let the normal bareos routines handle the error reporting.

The problem is that some problems do not have an associated error number since they might be internal logic errors so there is no way to give a helpful message in that case.

@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch 2 times, most recently from 4a0948a to 61553e0 Compare January 19, 2024 09:31
@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch 3 times, most recently from de34fb5 to 8aa5c24 Compare March 19, 2024 08:40
@arogge arogge added this to the 24.0.0 milestone Apr 2, 2024
@sebsura sebsura assigned sebsura and unassigned pstorz Jul 19, 2024
@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch from 2fe906d to 770d33b Compare August 1, 2024 06:49
@sebsura sebsura removed the nobuild label Aug 1, 2024
@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch from 30ac0cd to d6864c9 Compare August 2, 2024 06:15
@sebsura
Copy link
Contributor Author

sebsura commented Aug 19, 2024

Make the file indices etc 64 bit

pstorz
pstorz requested changes Aug 19, 2024
View reviewed changes
Copy link
Member

@pstorz pstorz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see comments. I think we should rename the device to dedupable

pstorz
pstorz requested changes Aug 22, 2024
View reviewed changes
Copy link
Member

@pstorz pstorz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check for docs changes. I also would remove the remaining sentence about the deprecated bacula format in mediaformat.rst:

From Bacula >= 1.27 and therefore Bareos, each block contains only records of a single job.

@pstorz pstorz linked an issue Aug 22, 2024 that may be closed by this pull request
update docs: "Mtime Only" may be ignored when using multiple option blocks in FileSet #1839
Closed
@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch 2 times, most recently from 41bc522 to b2ba5cb Compare August 23, 2024 09:55
@sebsura sebsura force-pushed the dev/ssura/fvec/dedup branch 2 times, most recently from c05462f to d077b5e Compare September 12, 2024 05:53
sebsura and others added 27 commits September 13, 2024 07:45
@sebsura @BareosBot
dedup-device: add relabeling support
e16d0e0
@sebsura @BareosBot
dedup-device: add d_flush support
903a5ee
@sebsura @BareosBot
dedup-device: implement d_read
139a681
@sebsura @BareosBot
dedup-volume: make dedup::data more generic
68b7b70 
Now accepts any number of datafiles.  This makes a lot of code much
cleaner since you do not need to constantly find out if a certain data
file is the "aligned" one or not.
@sebsura @BareosBot
dedup-volume: make the current block part of the state
87e7089
@sebsura @BareosBot
dedup-volume: correctly handle split records
af20c3d
@sebsura @BareosBot
dedup-device: implement eod()
f8c4ccb
@sebsura @BareosBot
volume: remove persistent config
13bac98
@sebsura @BareosBot
volume: refactor config
bff8800
@sebsura @BareosBot
dedup-config: change deserialize api
ee39d70
@sebsura @BareosBot
dedup-device: fix bad error messages
cb7afb0
@sebsura @BareosBot
dedup-device: implement d_truncate
fb3d1fa
@sebsura @BareosBot
dedup-device: correctly set eot when end is reached
1c3f4c2
@sebsura @BareosBot
dedup-device: fix handling split records
37040af
@sebsura @BareosBot
systemtests: add dedup tests
e855a3e
@sebsura @BareosBot
packaging: add dedup backend to packaging
27cda32
@sebsura @BareosBot
dedup-volume: fix reserving instead of allocating
ab50c76
@sebsura @BareosBot
dedup-volume: fix CommitBlock/AbortBlock error
0fe8762 
It was previously not possible to abort once a commit was
started (since the savestate was moved immediately on the CommitBlock
call).  This was fixed by instead waiting for the commit to finish
before issueing the move.
@sebsura @BareosBot
dedup-volume: add versioning to volume
00abc10
@sebsura @BareosBot
dedup-volume: switch record file to part file
9b13308 
This allows us to split up records into multiple parts with desirable
sizes. For example a 129k record may be split into a 128k part and a
1k part so that at least the first 128k are dedupable.

The record header now is not treated special anymore.  Its just
another (tiny) bit of data.
@sebsura @BareosBot
util: add bareos ASSERT macro
e20bf4d
@sebsura @BareosBot
docs: update mediaformat docs
ee81a22
@sebsura @BareosBot
docs: add short explanation of dedup
482b642
@pstorz @BareosBot
docs: Director.rst: warn about multiple options blocks
8bf1ce3
@pstorz @BareosBot
docs. mediaformat.rst: remove outdated bacula info
6487872
@sebsura @BareosBot
dedup: rename dedup_device -> dedupable_device
1a75c70
@BareosBot
Update CHANGELOG.md
a9cc902
@BareosBot BareosBot merged commit c5b6cd4 into bareos:master Sep 13, 2024
@bruno-at-bareos bruno-at-bareos mentioned this pull request Sep 25, 2024
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pstorz pstorz pstorz approved these changes

Assignees

@sebsura sebsura

Labels

enhancement requires backport to 23

Projects

None yet

Milestone

24.0.0

Development

Successfully merging this pull request may close these issues.

update docs: "Mtime Only" may be ignored when using multiple option blocks in FileSet

4 participants

@sebsura @pstorz @arogge @BareosBot