Skip to content

autoxflate: fix autoxflate on replication#1576

Merged
BareosBot merged 17 commits intobareos:masterfrom
SamuelBoerlin:autoxflate
Nov 10, 2023
Merged

autoxflate: fix autoxflate on replication#1576
BareosBot merged 17 commits intobareos:masterfrom
SamuelBoerlin:autoxflate

Conversation

@SamuelBoerlin
Copy link
Contributor

@SamuelBoerlin SamuelBoerlin commented Oct 14, 2023

Thank you for contributing to the Bareos Project!

While trying to set up autoxflate I ran into a bug where autoxflate doesn't seem to work with copy jobs (in Bareos 21 - and still an issue currently).

This test does the following:

  1. Create a full backup with LZ4 compression (as per SelfTest fileset config)
  2. Copy full backup to other pool and decompress (as per FileStorage2 device config)
  3. Restore copied job

Once the copy is restored I'd expect the files to be the same as the original files. While the restore job does end in success ("Restore OK"), it doesn't actually seem to restore anything, just a couple of empty files.

So, after some investigation it turns out to be a problem caused by several things:

  1. bSdEventSetupRecordTranslation is not called for migrate/copy jobs, causing autoxflate to not be able to set up inflate/deflate buffers
  2. autoxflate would in certain cases not set up inflate/deflate buffers anyways when it should depending on the configuration
  3. when LZ4 fails to decompress it seems to do so silently and just returns 0 bytes instead of logging an error, i.e., corrupted data is written to disk. This is not the case when e.g. GZIP is used, in which case an error is logged and it falls back to writing the compressed stream to disk, so data is not corrupted.

This PR fixes 1 and 2 and also adds a check to ensure the inflate buffer is set up. Autoxflate should now also work with migrate/copy jobs and the test passes.
I've also fixed a small memory leak which would occur when autoxflate detects that the deflate buffer is not set up.

Please check

  • Short description and the purpose of this PR is present above this paragraph
  • Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General
  • Is the PR title usable as CHANGELOG entry?
  • Purpose of the PR is understood
  • Commit descriptions are understandable and well formatted
    Check backport line
  • Required backport PRs have been created
Source code quality
  • Source code changes are understandable
  • Variable and function names are meaningful
  • Code comments are correct (logically and spelling)
  • Required documentation changes are present and part of the PR
Tests
  • Decision taken that a test is required (if not, then remove this paragraph)
  • The choice of the type of test (unit test or systemtest) is reasonable
  • Testname matches exactly what is being tested
  • On a fail, output of the test leads quickly to the origin of the fault

@SamuelBoerlin SamuelBoerlin changed the title systemtests: add autoxflate + copy test (currently failing due to a bug?) autoxflate: fix autoxflate with migrate/copy + systemtest Oct 16, 2023
@SamuelBoerlin SamuelBoerlin marked this pull request as ready for review October 16, 2023 06:56
Copy link
Contributor

@sebsura sebsura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution. The changes are really good. I added some notes

@SamuelBoerlin
Copy link
Contributor Author

Thanks for the feedback!

@SamuelBoerlin SamuelBoerlin requested a review from sebsura October 25, 2023 10:53
@sebsura sebsura changed the title autoxflate: fix autoxflate with migrate/copy + systemtest autoxflate: fix autoxflate on replication Oct 26, 2023
@sebsura
Copy link
Contributor

sebsura commented Oct 26, 2023

Just to give you an update: The code is great and will be merged. But i want to add some additional tests before that. Thank you for your contribution!

@SamuelBoerlin
Copy link
Contributor Author

Okay, thanks!

Copy link
Member

@pstorz pstorz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments.

@sebsura sebsura force-pushed the autoxflate branch 2 times, most recently from 208ab28 to de5afc0 Compare November 10, 2023 08:49
SamuelBoerlin and others added 11 commits November 10, 2023 11:29
The buffers have to be set up if inflate/deflate is enabled in any way
not just when used in a particular direction.
nrec would get leaked if the needed buffers were not set up
correctly.  This should fix that issue.
A bSdEventSetupRecordTranslation event should get dispatched
when reading and writing records during copy/migration.
This is done in preparation of adding more tests
If autodeflate is set, we need to ensure that an algorithm was
actually selected.
@BareosBot BareosBot merged commit a39f089 into bareos:master Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants