Project

General

Profile

Actions
Copy link

Bug #66014

closed

mds: Beacon code can deadlock messenger

Added by Greg Farnum almost 2 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Patrick Donnelly
Category:
Correctness/Safety
Target version:
Ceph - v20.0.0
% Done:

100%

Source:
Development
Backport:
squid,quincy,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
57469
Tags (freeform):
Merge Commit:
eb10f3df6e5116a020011365579584449ac76f04
Fixed In:
v19.3.0-2364-geb10f3df6e
Released In:
v20.2.0~2842
Upkeep Timestamp:
2025-11-01T01:11:34+00:00

Description

The Beacon code grabs a lock on entry into every function. This includes Beacon::_send(), which calls monc->send_mon_message(); and Beacon::handle_mds_beacon, invoked via ms_fast_dispatch. That means Beacon::mutex is ordered both above and below messenger locks.

We (likely?) haven't seen this because Beacon messages are fairly rare and actually triggering them simultaneously is improbable. But it could happen!

I'm also very suspicious of how much work Beacon::notify_health() does under that same mutex. If we need to fast dispatch the Beacon (it's not clear to me from ticket history that we do), we need to reorganize this module so that fast_dispatch can proceed without grabbing locks that are shared for other purposes.

Related issues 3 (0 open — 3 closed)

Copied to CephFS - Backport #66194: reef: mds: Beacon code can deadlock messengerResolvedPatrick DonnellyActions
Copied to CephFS - Backport #66195: quincy: mds: Beacon code can deadlock messengerRejectedPatrick DonnellyActions
Copied to CephFS - Backport #66196: squid: mds: Beacon code can deadlock messengerResolvedPatrick DonnellyActions
Actions
Copy link
 #1

Updated by Greg Farnum almost 2 years ago

  • Backport set to squid, quincy, reef
Actions
Copy link
 #2

Updated by Patrick Donnelly almost 2 years ago

  • Subject changed from Beacon code can deadlock messenger to mds: Beacon code can deadlock messenger
  • Status changed from New to In Progress
  • Target version set to v20.0.0
  • Source set to Development
  • Backport changed from squid, quincy, reef to squid,quincy,reef
Actions
Copy link
 #3

Updated by Patrick Donnelly almost 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 57469
Actions
Copy link
 #4

Updated by Patrick Donnelly almost 2 years ago ยท Edited

  • Status changed from Fix Under Review to Pending Backport

Note: the commits in https://github.com/ceph/ceph/pull/57469/commits reference the wrong ticket.

Actions
Copy link
 #5

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #66194: reef: mds: Beacon code can deadlock messenger added
Actions
Copy link
 #6

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #66195: quincy: mds: Beacon code can deadlock messenger added
Actions
Copy link
 #7

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #66196: squid: mds: Beacon code can deadlock messenger added
Actions
Copy link
 #9

Updated by Konstantin Shalygin about 1 year ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100
Actions
Copy link
 #10

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to eb10f3df6e5116a020011365579584449ac76f04
  • Fixed In set to v19.3.0-2364-geb10f3df6e5
  • Upkeep Timestamp set to 2025-06-27T03:23:38+00:00
Actions
Copy link
 #11

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-2364-geb10f3df6e5 to v19.3.0-2364-geb10f3df6e
  • Upkeep Timestamp changed from 2025-06-27T03:23:38+00:00 to 2025-07-14T16:45:00+00:00
Actions
Copy link
 #12

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~2842
  • Upkeep Timestamp changed from 2025-07-14T16:45:00+00:00 to 2025-11-01T01:11:34+00:00
Actions
Copy link

Also available in: Atom PDF