Project

General

Profile

Actions

Bug #41230

open

multisite: better spread multisite sync load over cooperating gateways

Added by Casey Bodley over 6 years ago. Updated 8 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Backport:
reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Fixed In:
v18.0.0-3936-g8c26848ad6
Released In:
v19.2.0~2365
Upkeep Timestamp:
2025-07-14T20:44:15+00:00

Description

each radosgw tries to lock every shard of each multisite log for processing, and can hold the locks for a long time. this generally results in one gateway hogging most of the shards


Related issues 1 (1 open0 closed)

Copied to rgw - Backport #61172: reef: multisite: better spread multisite sync load over cooperating gatewaysIn ProgressShilpa MJActions
Actions #1

Updated by Casey Bodley over 3 years ago

  • Assignee changed from J. Eric Ivancich to Or Friedmann
  • Pull request ID changed from 28119 to 45958
Actions #2

Updated by Christian Rohmann over 3 years ago

Casey Bodley wrote:

each radosgw tries to lock every shard of each multisite log for processing, and can hold the locks for a long time. this generally results in one gateway hogging most of the shards

We use this exact setup with 3 gateways on each end and with multiple endpoints explicitly configured in the zone / zonegroup.
Apart from proper "cooperation" there seems it seems that a single non-responsive RGW causes the whole replication to suffer massive slowdowns and calls of radosgw-admin for "sync status" to time out.

Actions #3

Updated by Casey Bodley almost 3 years ago

  • Status changed from In Progress to Fix Under Review
  • Assignee changed from Or Friedmann to Shilpa MJ
Actions #4

Updated by Casey Bodley almost 3 years ago

metadata sync fairness merged in https://github.com/ceph/ceph/pull/45958

@Shilpa, should we resolve this issue and start a new one for data sync, or leave this one open to track data sync? do you want to backport meta sync fairness to reef now, or wait for data sync too?

Actions #5

Updated by Shilpa MJ almost 3 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to reef

Casey Bodley wrote:

metadata sync fairness merged in https://github.com/ceph/ceph/pull/45958

@Shilpa, should we resolve this issue and start a new one for data sync, or leave this one open to track data sync? do you want to backport meta sync fairness to reef now, or wait for data sync too?

I have opened a new tracker for data sync. https://tracker.ceph.com/issues/61171
And I could backport this one first.

Actions #6

Updated by Upkeep Bot almost 3 years ago

  • Copied to Backport #61172: reef: multisite: better spread multisite sync load over cooperating gateways added
Actions #8

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to 8c26848ad68fd75ac576edad547e7304f851a0de
  • Fixed In set to v18.0.0-3936-g8c26848ad68
  • Released In set to v19.2.0~2365
  • Upkeep Timestamp set to 2025-07-09T18:56:50+00:00
Actions #9

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v18.0.0-3936-g8c26848ad68 to v18.0.0-3936-g8c26848ad6
  • Upkeep Timestamp changed from 2025-07-09T18:56:50+00:00 to 2025-07-14T20:44:15+00:00
Actions

Also available in: Atom PDF