Add droplet compatible storage backend by arogge · Pull Request #1790 · bareos/bareos

arogge · 2024-04-24T15:51:10Z

Add a new storage backend "dplcompat" that can work as a drop-in replacement for the "droplet" backend.
This will replace the current S3 backend that uses the outdated Droplet library.

TODO:

Please check

Short description and the purpose of this PR is present above this paragraph
Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General

Is the PR title usable as CHANGELOG entry?
Purpose of the PR is understood
Commit descriptions are understandable and well formatted
~~Required backport PRs have been created~~

Source code quality

Source code changes are understandable
Variable and function names are meaningful
Code comments are correct (logically and spelling)
Required documentation changes are present and part of the PR

Tests

Decision taken that a test is required (if not, then remove this paragraph)
The choice of the type of test (unit test or systemtest) is reasonable
Testname matches exactly what is being tested
On a fail, output of the test leads quickly to the origin of the fault

bruno-at-bareos · 2024-07-15T09:25:45Z

While working on tests there's some warnings during cmake with recent cmake version

-- CPM: Adding package GSL@4.0.0 (v4.0.0)
CMake Warning (dev) at /usr/share/cmake/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(GSL) is deprecated, call
  FetchContent_MakeAvailable(GSL) instead.  Policy CMP0169 can be set to OLD
  to allow FetchContent_Populate(GSL) to be called directly for now, but the
  ability to call it with declared details will be removed completely in a
  future version.
Call Stack (most recent call first):
  cmake/CPM.cmake:1010 (FetchContent_Populate)
  cmake/CPM.cmake:804 (cpm_fetch_package)
  CMakeLists.txt:44 (CPMAddPackage)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake/Modules/FetchContent.cmake:1953 (message):
  Calling FetchContent_Populate(tl-expected) is deprecated, call
  FetchContent_MakeAvailable(tl-expected) instead.  Policy CMP0169 can be set
  to OLD to allow FetchContent_Populate(tl-expected) to be called directly
  for now, but the ability to call it with declared details will be removed
  completely in a future version.
Call Stack (most recent call first):
  cmake/CPM.cmake:1010 (FetchContent_Populate)
  cmake/CPM.cmake:804 (cpm_fetch_package)
  CMakeLists.txt:48 (CPMAddPackage)

Entering /bareos/git/bareos/third-party
CMake Deprecation Warning at third-party/CLI11/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

bruno-at-bareos · 2024-07-23T07:22:53Z

Redoing different testing today 23.07.2024) I didn't detect previous failure.
A new build to publish new binaries would be appreciated.

core/src/dird/dird_conf.cc

core/src/filed/filed_conf.cc

core/src/stored/backends/crud_storage.cc

For Bpipe on Windows we need double-quoted strings in the cmdline. As our parser that turns strings into argc/argv for Unix handles single- and double-quotes the same, we can just switch to double-quotes.

This changes the debug levels for dplcompat and crud_storage so we can get better logs as follows: dplcompat informational: 100 crud_storage informational: 110 dplcompat tracing: 120 crud_storage tracing: 130

Use ${var} instead of ${var:+x} to check if options are set. This is safe, as all option variables are initialized to a default, so they are always set.

use an ASCII-only version of std::isalnum() to check for ASCII characters, as std::isalnum() is locale-dependant. Also improves the comment explaining what names we consider valid as an environment variable name.

use a string_view where it is possible instead of copying data into a std::string.

this adds a wrapper to catch the exception from BPipeHandle's ctor and return it as tl::unexpected. That wrapper is now used instead of creating BPipeHandle objects directly and errors are propagated correctly.

instead of keeping or releasing the buffers when we swap, it turned out that chunked_device does the right thing when we keep the release flags untouched.

The new droplet compatible storage backend introduced in bareos#1790 adds two new dependencies, that were not added to the debian control files. This commit adds the appropriate packages as build dependencies. Without this cmake will try to load the dependencies by itself and will fail, because git is not available by default during the package build. Please note: If you are building for bullseye, you will either need a backport of libcli11-dev and libmsgsl-dev or you will need to have git available during the packaging.

The new droplet compatible storage backend introduced in bareos#1790 adds two new dependencies, that were not added to the debian control files. This commit adds the appropriate packages as build dependencies. Without this cmake will try to load the dependencies by itself and will fail, because git is not available by default during the package build. Please note: If you are building for bullseye, you will either need a backport of libcli11-dev and libmsgsl-dev or you will need to have git available during the packaging. (cherry picked from commit 3fb9e03)

The new droplet compatible storage backend introduced in bareos#1790 adds two new dependencies, that were not added to the debian control files. This commit adds the appropriate packages as build dependencies. Without this cmake will try to load the dependencies by itself and will fail, because git is not available by default during the package build. Please note: If you are building for bullseye, you will either need a backport of libcli11-dev and libmsgsl-dev or you will need to have git available during the packaging.

The new droplet compatible storage backend introduced in bareos#1790 adds two new dependencies, that were not added to the debian control files. This commit adds the appropriate packages as build dependencies. Without this cmake will try to load the dependencies by itself and will fail, because git is not available by default during the package build. Please note: If you are building for bullseye, you will either need a backport of libcli11-dev and libmsgsl-dev or you will need to have git available during the packaging. (cherry picked from commit 3fb9e03)

arogge added the enhancement label Apr 24, 2024

arogge added this to the 24.0.0 milestone Apr 24, 2024

arogge force-pushed the scripted-storage-backend branch 2 times, most recently from a2d86d8 to 4656423 Compare April 25, 2024 12:50

sebsura added the draft label Apr 26, 2024

sebsura assigned arogge Apr 29, 2024

arogge force-pushed the scripted-storage-backend branch from bf7d1c3 to 9529753 Compare May 2, 2024 08:09

arogge added the onhold label Jul 9, 2024

arogge force-pushed the scripted-storage-backend branch 2 times, most recently from b762fe5 to 647013e Compare July 22, 2024 14:26

arogge removed the onhold label Jul 23, 2024

arogge force-pushed the scripted-storage-backend branch 3 times, most recently from 4733297 to c188a51 Compare July 24, 2024 07:24

arogge force-pushed the scripted-storage-backend branch 2 times, most recently from 2364080 to 03e214e Compare October 9, 2024 10:14

arogge force-pushed the scripted-storage-backend branch 3 times, most recently from 08e020f to 9429f75 Compare October 30, 2024 09:48

sebsura reviewed Nov 4, 2024

View reviewed changes

arogge force-pushed the scripted-storage-backend branch from 4e8dfcc to dc5ddf0 Compare November 25, 2024 09:14

arogge mentioned this pull request Nov 25, 2024

droplet/S3 does not support TLS1.3 and/or ECDSA #2036

Closed

arogge force-pushed the scripted-storage-backend branch from 266956a to 369b9ae Compare November 25, 2024 11:41

arogge force-pushed the scripted-storage-backend branch 5 times, most recently from ec3de6e to e547d01 Compare December 3, 2024 21:45

arogge added 16 commits December 12, 2024 08:40

crud_storage: change single- to double-quotes in cmdlines

4771799

For Bpipe on Windows we need double-quoted strings in the cmdline. As our parser that turns strings into argc/argv for Unix handles single- and double-quotes the same, we can just switch to double-quotes.

docs: add description to Scripts Directory option

57ca7c9

crud_storage: improve BPipeHandle

203199f

dplcompat: use distinct debug-levels

49b6297

This changes the debug levels for dplcompat and crud_storage so we can get better logs as follows: dplcompat informational: 100 crud_storage informational: 110 dplcompat tracing: 120 crud_storage tracing: 130

dplcompat: add configuration examples

00a1d99

docs: move Droplet documentation to include-file

8b0ca08

docs: add Dplcompat documentation

79551e4

freebsd: remove unused subpkglist

d140398

docs: add dplcompat wrapper programming guide

0145ec7

scripts: simplify option-checks in s3cmd-wrapper.sh

787bf37

Use ${var} instead of ${var:+x} to check if options are set. This is safe, as all option variables are initialized to a default, so they are always set.

dplcompat: use proper ASCII alphanumeric check

4ec0970

use an ASCII-only version of std::isalnum() to check for ASCII characters, as std::isalnum() is locale-dependant. Also improves the comment explaining what names we consider valid as an environment variable name.

dplcompat: handle EINTR during up/download correctly

9466e3a

dplcompat: use chrono::seconds for timeout

efc6f4a

dplcompat: use string_view where possible

1b393cc

use a string_view where it is possible instead of copying data into a std::string.

dplcompat: rename list -> chunk_map

61fde89

dplcompat: clean up systemtest configuration

370fc91

arogge changed the title ~~droplet compatible storage backend~~ Add droplet compatible storage backend Dec 12, 2024

arogge force-pushed the scripted-storage-backend branch from 9c363fd to de4c797 Compare December 12, 2024 07:41

arogge added 2 commits December 12, 2024 11:16

dplcompat: handle OpenBpipe() errors

8107782

this adds a wrapper to catch the exception from BPipeHandle's ctor and return it as tl::unexpected. That wrapper is now used instead of creating BPipeHandle objects directly and errors are propagated correctly.

chunked_device: correctly handle buffers when swapping

6344ca3

instead of keeping or releasing the buffers when we swap, it turned out that chunked_device does the right thing when we keep the release flags untouched.

arogge force-pushed the scripted-storage-backend branch from de4c797 to 6344ca3 Compare December 12, 2024 10:16

BareosBot added 2 commits December 12, 2024 11:50

Update LICENSE.txt

0608638

Update CHANGELOG.md

31cac50

BareosBot merged commit 06cb3ea into bareos:master Dec 12, 2024

AlexanderS mentioned this pull request Jan 21, 2025

debian: Add missing build dependencies #2128

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add droplet compatible storage backend#1790

Add droplet compatible storage backend#1790
BareosBot merged 86 commits intobareos:masterfrom
arogge:scripted-storage-backend

arogge commented Apr 24, 2024 •

edited by sebsura

Loading

Uh oh!

bruno-at-bareos commented Jul 15, 2024

Uh oh!

bruno-at-bareos commented Jul 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

arogge commented Apr 24, 2024 • edited by sebsura Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Please check

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

General

Source code quality

Tests

Uh oh!

bruno-at-bareos commented Jul 15, 2024

Uh oh!

bruno-at-bareos commented Jul 23, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

arogge commented Apr 24, 2024 •

edited by sebsura

Loading