rgw/bucket-logging: support EC pools by nbalacha · Pull Request #65900 · ceph/ceph

nbalacha · 2025-10-12T12:37:36Z

Log buckets can now be created within erasure-coded (EC) pools.
To support append operations, a temporary log record object is initially
created in the replicated default.rgw.log pool. This object is then copied
to the EC pool upon log record commitment.
All implicit log commit operations will execute asynchronously. A new
BucketLoggingManager class is responsible for processing these pending
commits at set intervals. Explicit commit operations, however, will
continue to be performed synchronously.

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

nbalacha · 2025-10-13T10:42:42Z

@yuvalif , please take a look at the approach and let me know what you think.
This is not ready for review.

Pending tasks:

Check if the log bucket is on a EC pool and use that information to determine where to create the log record object.
Cleanup the bucket logging list objects when logging is disabled for a bucket
Remove the debug "NITHYA" log messages.

nbalacha · 2025-10-30T05:34:56Z

Pending tasks:

Code refactoring and cleanup (file names, logs, namespaces)
EC pool tests
Additional error handling in the BucketLoggingManager

anthonyeleven · 2025-10-31T16:05:11Z

Please select a line under the Tracker section of the checklist.
If you rebase the RTD failure should resolve.

Log buckets can now be created within erasure-coded (EC) pools. To support append operations, a temporary log record object is initially created in the replicated default.rgw.log pool. This object is then copied to the EC pool upon log record commitment. All implicit log commit operations will execute asynchronously. A new BucketLoggingManager class is responsible for processing these pending commits at set intervals. Explicit commit operations, however, will continue to be performed synchronously. Fixes: https://tracker.ceph.com/issues/71365 Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>

Run the rgw bucket logging teuthology tests on an erasure coded pool. Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>

yuvalif · 2025-12-01T13:39:18Z

jenkins test submodules

yuvalif · 2025-12-01T13:47:42Z

followup work (to be done in other PRs):

remove the must_commit parameter
- in case of -EFBIG we should not return an error. this is done post rollover and new records will be written to the new obj (so going over size limit should not be an issue)
- when doing explicit rollovers (conf change or flush) we should perform the commit syncronlusly since we are expected to reply with the "last committed" name. however, these operations are not idempotent, and client retries will not fixed failed commits, so, in case od failed commit, we should not reply with an error
- optionally, we may perfrom an async commit if the sync commit failed
add bucket logging debug subsystem for all related code (e.g. rgw_bucket_logging.cc)
call read_global_logging_list() in chunks, instead of readign the entire list
use regex in parse_list_names()

yuvalif · 2025-12-02T12:33:33Z

bucket logging tests are passing: https://pulpito.ceph.com/nithyab-2025-12-01_15:29:39-rgw:bucket-logging-wip-nbalacha-71365-distro-default-smithi/
rgw regression has 19 failures: https://pulpito.ceph.com/nithyab-2025-12-01_16:01:59-rgw-wip-nbalacha-71365-distro-default-smithi/

d4n - known
multisite - known
valgrind error: SyscallParam sendmsg AsyncConnection::_try_send(bool) - known
'sudo yum -y install ceph-osd-classic'/Unable to locate package ceph-osd-classic - in centos/ubuntu looks like test infra issue
File "/home/teuthworker/src/git.ceph.com_ceph-c_05918a1c67d8532b8d46e9abfd56ca9bbffe75f6/qa/tasks/radosgw_admin.py", line 993, in task assert(len(r) == 0) - known issue

github-actions · 2025-12-02T12:35:30Z

This is an automated message by src/script/redmine-upkeep.py.

I found one or more Fixes: tags in the commit messages in

git log 822f175f606c694e72519d9b6182e01be9b1d238^..822f175f606c694e72519d9b6182e01be9b1d238

The referenced tickets are:

https://tracker.ceph.com/issues/71365

Those tickets do not reference this merged Pull Request. If this Pull Request merge resolves any of those tickets, please update the "Pull Request ID" field on each ticket. A future run of this script will appropriately update them.

Update Log: https://github.com/ceph/ceph/actions/runs/19858758035

nbalacha requested review from a team as code owners October 12, 2025 12:37

github-actions Bot added build/ops common core documentation rgw labels Oct 12, 2025

nbalacha marked this pull request as draft October 12, 2025 12:39

anthonyeleven reviewed Oct 12, 2025

View reviewed changes

nbalacha requested review from a team and yuvalif October 13, 2025 03:12

bluikko reviewed Oct 13, 2025

View reviewed changes

Comment thread src/rgw/driver/rados/rgw_bucketlogging.cc Outdated

nbalacha force-pushed the wip-nbalacha-71365 branch from 655c5cc to 07d68e1 Compare October 13, 2025 10:12

nbalacha force-pushed the wip-nbalacha-71365 branch from 07d68e1 to a77a07d Compare October 30, 2025 05:33

nbalacha force-pushed the wip-nbalacha-71365 branch 2 times, most recently from 9192f62 to 44c07a6 Compare October 31, 2025 15:34

nbalacha force-pushed the wip-nbalacha-71365 branch 3 times, most recently from 01997e0 to defee46 Compare November 4, 2025 08:55