bareos-fd: add option for grpc fallback by sebsura · Pull Request #2104 · bareos/bareos

sebsura · 2025-01-06T13:16:28Z

This PR adds a new option to the bareos-fd: Grpc Module (Fd->Client).

If this option is set, and the grpc plugin is loaded, then the fd will try to use the specified grpc module to load any plugin that the fd did not already load itself.

As an example: If you instruct the fd to load only the grpc plugin, but not the python plugin (i.e. specifiying only grpc in Plugin Names (Fd->Client), and the director sends a fileset which specifies the python plugin, then the fd will use the specified grpc module (via the grpc plugin) to load the python plugin.

This PR also adds two types of tests for this:

a manually created test that checks that backups are interchangable between a client using the new grpc feature and a client using the old python plugin.
it (automatically) adds a new test for each already existing python test which causes the fd to use the grpc module instead of using the python plugin.

TODO:

write documentation
add grpc adr
rename grpc-python-module -> bareos-grpc-fd-python-bridge

Thank you for contributing to the Bareos Project!

Please check

Short description and the purpose of this PR is present above this paragraph
Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General

Is the PR title usable as CHANGELOG entry?
Purpose of the PR is understood
Commit descriptions are understandable and well formatted
Required backport PRs have been created
Correct milestone is set

Source code quality

Source code changes are understandable
Variable and function names are meaningful
Code comments are correct (logically and spelling)
Required documentation changes are present and part of the PR

Tests

Decision taken that a test is required (if not, then remove this paragraph)
The choice of the type of test (unit test or systemtest) is reasonable
Testname matches exactly what is being tested
On a fail, output of the test leads quickly to the origin of the fault

arogge · 2025-01-07T10:50:19Z

We don't need the option to select the compatibility module, as it will only be used when grpc module is loaded, but the requested module is not.

bruno-at-bareos

Nice work, eager to get it merge. Until that I've a few remarks, and would like to have second pair of eyes n filed_conf.cc and fd_plugins

docs/adr/0004-use-grpc-for-rpc-needs.md

systemtests/cmake/BareosSystemtestFunctions.cmake

systemtests/tests/auto-grpc/etc/bareos-python/bareos-fd.d/client/myself.conf.in

core/src/filed/filed_conf.cc

systemtests/tests/auto-grpc/etc/bareos/bareos-dir.d/storage/File.conf.in

systemtests/tests/grpc-fd/testrunner-python-io-in-core

systemtests/tests/grpc-fd/testrunner-test-backup

systemtests/tests/py3plug-fd-basic/etc/bareos/bareos-dir.d/fileset/CancelFileset.conf.in

docs/manuals/source/include/autogenerated/bareos-fd-config-schema.json

core/platforms/packaging/bareos.spec

sebsura · 2025-01-15T10:52:23Z

Rebased onto master because of a confict

bruno-at-bareos

Redone a bunch of tests especially those mixing grpc and native, with no troubles.
On my side I would approve the great work done.
Now pass the ball to another C++ in depth review.

systemtests/tests/py3plug-fd-postgresql/testrunner-default

systemtests/tests/grpc-fd/etc/bareos/bareos-dir.d/fileset/SelfTest.conf.in

sduehr · 2025-02-07T09:59:02Z

Thanks, tested this with the VMware plugin, works fine.

pstorz

Please see comments

docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/GrpcPlugin.rst.inc

It looks like this is blocking the merge

This will later be used to run all normal plugins through grpc

When a plugin cannot be found, the grpc plugin is loaded and a grpc compat module is defined, then we try to load that plugin with that compat module. To facilitate this we must make sure that the plugin handling is able to modify the plugin command strings both into the plugin and out of the plugin as we need to add/remove the grpc prefix. We also remove the `reverse` option of `GeneratePluginEvent` as it was unused and would mess with the (already complicated) grpc fallback logic.

Since auto grpc needs completely different plugins on the filed and director, we now have two different variables: python_module_name and dir_python_module_name for python tests. python_module_name is the module name on the fd side, whereas dir_python_module_name is the module name on the director side. This also removes the need for the python_module_suffix as the normal grpc tests can now just specify the dir module name directly.

Currently the core adds fallbacks for the python plugin, so that even if your fd has python3-fd and your fileset specifies just "python", it will still work. This also has to be supported in the grpc python module for painless transitions.

This makes the transition to grpc easier as users just have to set their plugin names to grpc.

The name did not say much and was also misleading. We chose a name that hopefully makes it clear what it actually does when it shows up in a process explorer.

This argument is used in the vmware plugin, so we cannot just ignore it.

Our systemtests will quit when `waitpid` sees a zombie process. This can happen when the system is under a lot of load. Because of this we wait for 1 second after discovering a zombie process to give the system some time to clean it up.

This test sometimes fails because scheduling sometimes takes a lot of time (~10secs), causing the first job to be executed first even though it waited for a long time.

sebsura added the enhancement label Jan 6, 2025

sebsura added this to the 25.0.0 milestone Jan 6, 2025

sebsura self-assigned this Jan 6, 2025

bruno-at-bareos self-requested a review January 6, 2025 13:28

sebsura force-pushed the dev/ssura/master/auto-grpc branch from 5af1777 to 0f8d8c4 Compare January 13, 2025 06:42

bruno-at-bareos added the requires backport to 24 label Jan 13, 2025

bruno-at-bareos previously requested changes Jan 13, 2025

View reviewed changes

sebsura force-pushed the dev/ssura/master/auto-grpc branch 3 times, most recently from dc615d8 to c0fedc9 Compare January 15, 2025 10:51

bruno-at-bareos force-pushed the dev/ssura/master/auto-grpc branch from aff470a to 035c24a Compare January 15, 2025 13:54

sebsura force-pushed the dev/ssura/master/auto-grpc branch from 2bd7130 to a627c60 Compare January 17, 2025 07:15

bruno-at-bareos self-requested a review January 21, 2025 14:31

bruno-at-bareos reviewed Jan 21, 2025

View reviewed changes

sebsura force-pushed the dev/ssura/master/auto-grpc branch from a627c60 to 298518b Compare January 22, 2025 09:01

sebsura requested a review from pstorz January 23, 2025 11:59

sebsura commented Jan 23, 2025

View reviewed changes

systemtests/tests/py3plug-fd-postgresql/testrunner-default Outdated Show resolved Hide resolved

sebsura commented Jan 23, 2025

View reviewed changes

systemtests/tests/grpc-fd/etc/bareos/bareos-dir.d/fileset/SelfTest.conf.in Outdated Show resolved Hide resolved

sebsura force-pushed the dev/ssura/master/auto-grpc branch 2 times, most recently from 8f82381 to b1672c7 Compare January 29, 2025 05:59

sebsura force-pushed the dev/ssura/master/auto-grpc branch 3 times, most recently from 3a5fae4 to 5646340 Compare February 5, 2025 13:12

pstorz approved these changes Feb 14, 2025

View reviewed changes

sebsura force-pushed the dev/ssura/master/auto-grpc branch 2 times, most recently from 6bb9967 to f369743 Compare February 20, 2025 08:37

pstorz self-requested a review February 20, 2025 08:45

pstorz requested changes Feb 20, 2025

View reviewed changes

docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/GrpcPlugin.rst.inc Outdated Show resolved Hide resolved

docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/GrpcPlugin.rst.inc Show resolved Hide resolved

sebsura mentioned this pull request Feb 27, 2025

bareos-fd: add option for grpc fallback #2193

Merged

8 tasks

sebsura requested a review from pstorz February 27, 2025 09:25

pstorz approved these changes Feb 28, 2025

View reviewed changes

sebsura force-pushed the dev/ssura/master/auto-grpc branch from 048c8a3 to 4118cad Compare March 3, 2025 06:40

sebsura and others added 21 commits March 5, 2025 09:45

grpc-fd-test: fix using same restore dir for all tests

6f7aec8

filed: add option to specify default grpc module

5d1b3cf

This will later be used to run all normal plugins through grpc

tests: add testing of automatically using grpc for plugins

712d115

grpc: fix description

52c03cd

packaging: move grpc modules into grpc subdirectory

eaeb7af

fd-plugins: unify PluginSave/PluginEstimate plugin search

ceb816d

filed: add default for grpc module

f49a07b

This makes the transition to grpc easier as users just have to set their plugin names to grpc.

adrs: fix typo

69fea86

adrs: add grpc adr

af86198

grpc-fd: rename grpc-python-module

fbfa684

The name did not say much and was also misleading. We chose a name that hopefully makes it clear what it actually does when it shows up in a process explorer.

docs: fix bad link

b52cab8

grpc: add where to restore packet

ab9663c

This argument is used in the vmware plugin, so we cannot just ignore it.

docs: enhance grpc docs

0d07edb

grpc-fd: fix killing process immediately; fix double join

60fdf1d

grpc: fix copyright dates

7e0c662

parallel-job-test: increase waiting time

853e311

This test sometimes fails because scheduling sometimes takes a lot of time (~10secs), causing the first job to be executed first even though it waited for a long time.

Update CHANGELOG.md

17548b5

BareosBot force-pushed the dev/ssura/master/auto-grpc branch from fbcf911 to 17548b5 Compare March 5, 2025 09:45

BareosBot merged commit 49cfd28 into bareos:master Mar 5, 2025

Conversation

sebsura commented Jan 6, 2025 • edited by pstorz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Thank you for contributing to the Bareos Project!

Please check

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

General

Source code quality

Tests

Uh oh!

arogge commented Jan 7, 2025

Uh oh!

bruno-at-bareos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sebsura commented Jan 15, 2025

Uh oh!

bruno-at-bareos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sduehr commented Feb 7, 2025

Uh oh!

pstorz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sebsura commented Jan 6, 2025 •

edited by pstorz

Loading