Refactor libcloud plugin to work with current Python versions#2428
Merged
BareosBot merged 7 commits intobareos:masterfrom Nov 24, 2025
Merged
Conversation
bruno-at-bareos
requested changes
Nov 20, 2025
Contributor
bruno-at-bareos
left a comment
There was a problem hiding this comment.
Awesome work! That's full of improvements.
I've let a few suggestions, but also questions and comments.
To test the whole code, I would like to see at least three buckets with one excluded, so we know we are able to parse a list of buckets, can exclude some
For the data, that would be nice also to check excluded data
(not present in restore)
testrunner: might be interesting to split them up by tasks
During testing second backup was failing here: so I suggest to test at least one incremental and accurate mode.
docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/ApacheLibcloudPlugin.rst
Outdated
Show resolved
Hide resolved
docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/ApacheLibcloudPlugin.rst
Outdated
Show resolved
Hide resolved
docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/ApacheLibcloudPlugin.rst
Outdated
Show resolved
Hide resolved
docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/ApacheLibcloudPlugin.rst
Outdated
Show resolved
Hide resolved
core/src/plugins/filed/python/libcloud/bareos_libcloud_api/mtime.py
Outdated
Show resolved
Hide resolved
core/src/plugins/filed/python/libcloud/bareos_libcloud_api/process_base.py
Outdated
Show resolved
Hide resolved
core/src/plugins/filed/python/libcloud/bareos_libcloud_api/queue_message.py
Outdated
Show resolved
Hide resolved
core/src/plugins/filed/python/libcloud/bareos_libcloud_api/queue_message.py
Show resolved
Hide resolved
core/src/plugins/filed/python/libcloud/bareos_libcloud_api/worker.py
Outdated
Show resolved
Hide resolved
Member
Author
|
Addressed all requested changes, but I don't think the effort is reasonable to add buckets_include and buckets_exclude to the systemtest, as this is quite simple code which is not prone to errors. |
40762b0 to
cbb9bd1
Compare
bruno-at-bareos
approved these changes
Nov 24, 2025
Contributor
bruno-at-bareos
left a comment
There was a problem hiding this comment.
I've a small changes we can add to emphasis the need of grpc for this plugin. But nothing that should block the merge.
Excellent work.
docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/ApacheLibcloudPlugin.rst.inc
Outdated
Show resolved
Hide resolved
docs/manuals/source/TasksAndConcepts/Plugins/FileDaemonPlugins/ApacheLibcloudPlugin.rst
Outdated
Show resolved
Hide resolved
Major change is to use threading instead of multiprocessing as that does not work with Bareos plugins and current Pyhton versions. Unfortunately libcloud is not fully thread safe, when libcloud calls run into timeouts throwing exceptions, which can happen on bad network connections, next calls get stuck within the worker threads. So the default for the fail_on_download_error is now yes and the plugin code tries to detect unresponsive worker threads and terminates the job.
Adapt packaging as BareosFdPluginLibcloud.py was removed.
Add plugin option libcloud_timout, this is passed to the libcloud driver initialization to make this configurable.
Adapt Apache Libcloud Plugin documentation to the latest changes.
ea5d329 to
58750b0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adaption and refactoring of the libcloud plugin to work with current Python versions.
Unfortunately libcloud is not fully thread safe, this can cause unresponsive worker
threads on bad network connections. The plugin code tries to handle that in a reasonable
way.
Please check
If you have any questions or problems, please give a comment in the PR.
Helpful documentation and best practices
Checklist for the reviewer of the PR (will be processed by the Bareos team)
Make sure you check/merge the PR using
devtools/pr-toolto have some simple automated checks run and a proper changelog record added.General
Required backport PRs have been createdSource code quality
Tests