rgw/dedup: add throttling mechanism by benhanokh · Pull Request #64730 · ceph/ceph

benhanokh · 2025-07-29T07:20:50Z

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

benhanokh · 2025-08-14T13:27:52Z

jenkins test make check

benhanokh · 2025-08-15T05:55:20Z

jenkins test make check

benhanokh · 2025-08-17T05:43:08Z

jenkins test make check

benhanokh · 2025-08-17T05:43:32Z

jenkins test make check arm64

benhanokh · 2025-08-17T05:43:36Z

jenkins test api

benhanokh · 2025-08-17T05:44:11Z

jenkins test make check arm64

benhanokh · 2025-08-17T07:40:05Z

jenkins test make check

benhanokh · 2025-08-17T08:33:28Z

jenkins test make check

benhanokh · 2025-08-17T12:11:39Z

jenkins test make check

benhanokh · 2025-08-17T14:42:18Z

jenkins test make check

benhanokh · 2025-09-06T14:38:58Z

Rebased the code to fix the GC issue.
Shaman build: https://shaman.ceph.com/builds/ceph/wip_dedup_throttle_V1/
Teuthology testing: https://pulpito.ceph.com/benhanokh-2025-09-06_14:05:12-rgw-wip_dedup_throttle_V1-distro-default-smithi/

benhanokh · 2025-09-07T09:55:58Z

53 tests passed and 11 failed:

3 failures were caused by failing to load linux code:
'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

5 more errors reported by valgrind on Objector (which is not affected by the PR)
valgrind error: Leak_PossiblyLost operator new(unsigned long) Objecter::start_tick() Objecter::start(OSDMap const*)

2 errors for rgw multisite on multi-object delete, but dedup code was inactive so should not be related
ERROR: rgw_multi.tests.test_multi_object_delete

1 error coming from test_delete_objects_version_if_match_size() failing to delete objects by version.
Unrelated to my code which doesn't touch version code and which was not activate on this system

restarted the failing test: https://pulpito.ceph.com/benhanokh-2025-09-07_09:40:40-rgw-wip_dedup_throttle_V1-distro-default-smithi/

benhanokh · 2025-09-07T12:14:49Z

Rerun solved the load-linux issues (including in rgw-dedup test) and the multi-site issue.
We still have 5 failures from valgrind error: Leak_PossiblyLost operator new(unsigned long) Objecter::start_tick() Objecter::start(OSDMap const*).
This is unrelated to my PR

yuvalif · 2025-09-08T08:50:31Z

Rerun solved the load-linux issues (including in rgw-dedup test) and the multi-site issue. We still have 5 failures from valgrind error: Leak_PossiblyLost operator new(unsigned long) Objecter::start_tick() Objecter::start(OSDMap const*). This is unrelated to my PR

all failures are known issue

yuvalif · 2025-09-08T09:02:08Z

doc/radosgw/s3_objects_dedup.rst

   Aborts an active dedup session and release all resources used by it.
 - ``radosgw-admin dedup estimate``:
   Starts a new dedup estimate session (aborting first existing session if exists).
+- ``radosgw-admin dedup throttle --max-bucket-index_ops=<count>``:


Suggested change

- ``radosgw-admin dedup throttle --max-bucket-index_ops=<count>``:

- ``radosgw-admin dedup throttle --max-bucket-index-ops=<count>``:

doc/radosgw/s3_objects_dedup.rst

github-actions · 2025-09-26T18:01:36Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

benhanokh · 2025-09-30T11:40:20Z

https://pulpito.ceph.com/benhanokh-2025-09-30_11:36:26-rgw-dedup_throttle-distro-default-smithi/

cbodley · 2025-10-02T14:15:29Z

https://pulpito.ceph.com/benhanokh-2025-09-30_11:36:26-rgw-dedup_throttle-distro-default-smithi/

lots of s3test failures because we've merged new test cases since then. can you please rebase/rerun?

benhanokh · 2025-10-03T12:48:30Z

jenkins test make check arm64

benhanokh · 2025-10-03T19:13:08Z

jenkins test make check arm64

benhanokh · 2025-10-04T11:26:10Z

jenkins test make check arm64

benhanokh · 2025-10-04T14:35:26Z

jenkins test make check arm64

benhanokh · 2025-10-04T14:45:48Z

jenkins test make check arm64

Copilot

Pull Request Overview

This PR adds a throttling mechanism to the RGW deduplication system to control the rate of operations during dedup execution. The throttling system allows administrators to limit bucket-index operations and metadata access operations per second to manage resource consumption.

Add new throttling classes and commands to control dedup operation rates
Implement throttling for bucket-index and metadata operations with configurable limits
Update command interface from dedup restart to dedup exec with backward compatibility

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/test/rgw/dedup/test_dedup.py	Updates test commands and adds throttling configuration with variable naming corrections
src/test/cli/radosgw-admin/help.t	Updates help text to reflect new dedup exec command and throttle options
src/rgw/radosgw-admin/radosgw-admin.cc	Adds throttle command parsing and implements new dedup exec command interface
src/rgw/driver/rados/rgw_dedup_utils.h	Defines Throttle class and throttling-related data structures
src/rgw/driver/rados/rgw_dedup_utils.cc	Implements throttling logic and message encoding/decoding
src/rgw/driver/rados/rgw_dedup_store.cc	Adds failure tracking for slab write operations
src/rgw/driver/rados/rgw_dedup_cluster.h	Adds new function signature for throttle control with buffer list
src/rgw/driver/rados/rgw_dedup_cluster.cc	Implements throttle control logic and reporting functionality
src/rgw/driver/rados/rgw_dedup.h	Integrates throttle objects into control structure
src/rgw/driver/rados/rgw_dedup.cc	Applies throttling to metadata and bucket index operations throughout dedup process
doc/radosgw/s3_objects_dedup.rst	Documents new throttling commands and updated dedup exec interface

Comments suppressed due to low confidence (1)

src/rgw/driver/rados/rgw_dedup_utils.cc:1

Corrected spelling of 'disbaled' to 'disabled'.

// -*- mode:C++; tab-width:8; c-basic-offset:2;

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/test/rgw/dedup/test_dedup.py

Copilot · 2025-10-05T09:12:32Z

src/rgw/driver/rados/rgw_dedup.cc

    }
    //ldpp_dout(dpp, 0) << __func__ << "::sleep for 2 seconds\n" << dendl;
    //std::this_thread::sleep_for(std::chrono::seconds(2));
+    //std::this_thread::sleep_forstd::chrono::microseconds(usec_timeout);


There's a missing space in the commented code. Should be 'sleep_for std::chrono::microseconds'.

Suggested change

//std::this_thread::sleep_forstd::chrono::microseconds(usec_timeout);

//std::this_thread::sleep_for(std::chrono::microseconds(usec_timeout));

anrao19 · 2025-10-08T07:04:13Z

jenkins test make check arm64

anrao19 · 2025-10-08T07:05:57Z

pr testing completed and approved by @ivancich. Please find the details : https://tracker.ceph.com/issues/73370
@benhanokh , if there is not further testing needed this can be merged. Please let me know

Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com> rgw/dedup: Change throttle code to work lock free and remove the atomic from the timestamp Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>

benhanokh · 2025-10-16T05:29:48Z

jenkins test make check arm64

benhanokh · 2025-10-16T15:32:13Z

https://pulpito.ceph.com/benhanokh-2025-10-15_12:33:04-rgw-wip_dedup_throttle_rebase_Z2-distro-default-smithi/

benhanokh · 2025-10-19T10:40:05Z

8553077, 8553079:
s3:get_bucket_location init_permissions on :datacachebucket[]) failed, ret=-2002

8553067, 8553087, 8553092, 8553096,8553106, 8553116, 8553118, 8553120, 8553122
valgrind issue in ceph-client

8553082, 8553094,
ulimits issue

8553066,
rgw main: Lua ERROR: failed to find luarocks

benhanokh · 2025-10-19T10:41:14Z

I don't see how can this PR cause any issue as it doesn't affect RGW code and dedup is only activated from command line
Can we please merge the PR?

benhanokh self-assigned this Jul 29, 2025

benhanokh requested a review from a team as a code owner July 29, 2025 07:20

benhanokh marked this pull request as draft July 29, 2025 07:21

github-actions bot added the rgw label Jul 29, 2025

benhanokh force-pushed the dedup_throttle branch from 9748ec7 to 67cbe59 Compare August 6, 2025 12:33

github-actions bot added the tests label Aug 6, 2025

benhanokh requested a review from yuvalif August 6, 2025 12:42

benhanokh force-pushed the dedup_throttle branch from 67cbe59 to 9fd7353 Compare August 14, 2025 10:18

github-actions bot added the documentation label Aug 14, 2025

benhanokh force-pushed the dedup_throttle branch from 9fd7353 to 7963164 Compare August 16, 2025 07:47

benhanokh added the needs-tentacle-backport PRs that need tentacle back-port label Aug 17, 2025

benhanokh mentioned this pull request Aug 26, 2025

rgw/dedup: Tech preview of full dedup support which means duplicate objects will be removed #64271

Merged

14 tasks

benhanokh force-pushed the dedup_throttle branch from 7963164 to f41b08d Compare September 6, 2025 11:49

yuvalif reviewed Sep 8, 2025

View reviewed changes

doc/radosgw/s3_objects_dedup.rst Show resolved Hide resolved

yuvalif reviewed Sep 8, 2025

View reviewed changes

doc/radosgw/s3_objects_dedup.rst Outdated Show resolved Hide resolved

ivancich removed the TESTED label Sep 25, 2025

yuvalif removed the wip-yuval-testing label Sep 25, 2025

github-actions bot added the needs-rebase label Sep 26, 2025

benhanokh force-pushed the dedup_throttle branch from c2c3db1 to 98202a7 Compare September 30, 2025 11:39

github-actions bot removed the needs-rebase label Sep 30, 2025

benhanokh force-pushed the dedup_throttle branch from 98202a7 to 086d5d4 Compare October 3, 2025 06:53

benhanokh force-pushed the dedup_throttle branch from 086d5d4 to 16ad586 Compare October 4, 2025 06:04

benhanokh requested a review from Copilot October 5, 2025 09:11

Copilot AI reviewed Oct 5, 2025

View reviewed changes

anrao19 added wip-anrao3-testing TESTED ready-to-merge and removed wip-anrao3-testing labels Oct 6, 2025

rgw/dedup: add throttling mechanism

58461e0

Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com> rgw/dedup: Change throttle code to work lock free and remove the atomic from the timestamp Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>

benhanokh force-pushed the dedup_throttle branch from 16ad586 to 58461e0 Compare October 15, 2025 09:15

yuvalif merged commit 04280e1 into ceph:main Oct 19, 2025
13 checks passed

	- ``radosgw-admin dedup throttle --max-bucket-index_ops=<count>``:
	- ``radosgw-admin dedup throttle --max-bucket-index-ops=<count>``:

	//std::this_thread::sleep_forstd::chrono::microseconds(usec_timeout);
	//std::this_thread::sleep_for(std::chrono::microseconds(usec_timeout));

Conversation

benhanokh commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

benhanokh commented Aug 14, 2025

Uh oh!

benhanokh commented Aug 15, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Aug 17, 2025

Uh oh!

benhanokh commented Sep 6, 2025

Uh oh!

benhanokh commented Sep 7, 2025

Uh oh!

benhanokh commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuvalif commented Sep 8, 2025

Uh oh!

yuvalif Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

benhanokh commented Sep 30, 2025

Uh oh!

cbodley commented Oct 2, 2025

Uh oh!

benhanokh commented Oct 3, 2025

Uh oh!

benhanokh commented Oct 3, 2025

Uh oh!

benhanokh commented Oct 4, 2025

Uh oh!

benhanokh commented Oct 4, 2025

Uh oh!

benhanokh commented Oct 4, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Oct 5, 2025

Choose a reason for hiding this comment

Uh oh!

anrao19 commented Oct 8, 2025

Uh oh!

anrao19 commented Oct 8, 2025

Uh oh!

benhanokh commented Oct 16, 2025

Uh oh!

benhanokh commented Oct 16, 2025

Uh oh!

benhanokh commented Oct 19, 2025

Uh oh!

benhanokh commented Oct 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

benhanokh commented Jul 29, 2025 •

edited

Loading

benhanokh commented Sep 7, 2025 •

edited

Loading