Skip to content

mon/MonClient: post version request completions outside of monc_lock#65177

Merged
idryomov merged 1 commit intoceph:mainfrom
idryomov:wip-72692
Aug 22, 2025
Merged

mon/MonClient: post version request completions outside of monc_lock#65177
idryomov merged 1 commit intoceph:mainfrom
idryomov:wip-72692

Conversation

@idryomov
Copy link
Contributor

This fixes a regression that was introduced in #62770.

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

dispatch() is allowed to invoke the completion object in the current
thread, before control returns from dispatch().  This isn't desirable
when it comes to discarding version requests in MonClient::shutdown()
and MonClient::_reopen_session() because completion objects could then
be invoked under monc_lock.  In case of MonClient::_reopen_session() in
particular, this leads to an attempt to acquire monc_lock once again in
MonClient::get_version() on a retry due to monc_errc::session_reset
that is converted to errc::resource_unavailable_try_again:

  MonClient::ms_handle_reset
    < takes monc_lock >
    MonClient::_reopen_session
      < invokes the completion object via dispatch() with ec == monc_errc::session_reset >
      Objecter::CB_Objecter_GetVersion::operator() [ ec == errc::resource_unavailable_try_again ]
        Objecter::_wait_for_latest_osdmap
          MonClient::get_version
            < attempts to take monc_lock in the body of the lambda >

The end result is either a lockup or some form of undefined behavior.
The best possible outcome here is an exception (std::system_error with
"Resource deadlock avoided" error) and a successive call to
std::terminate().

This is a regression introduced in commit e81d4ea ("common/async:
Update `use_blocked` for newer asio").  Revert to posting version
request completions for the error cases in a way that is uniform with
the success case in MonClient::handle_get_version_reply().

Fixes: https://tracker.ceph.com/issues/72692
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Copy link
Contributor

@cbodley cbodley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed that we shouldn't dispatch under monc_lock 👍

@adamemerson if you really want to use dispatch here, we could potentially move version_requests into a local variable then loop over that after dropping the lock. but it's probably not a big deal, especially during shutdown

@cbodley
Copy link
Contributor

cbodley commented Aug 21, 2025

@adamemerson i think we should make sure objecter and monclient changes go through the rbd and fs suites too

@idryomov
Copy link
Contributor Author

@idryomov idryomov merged commit 4413740 into ceph:main Aug 22, 2025
14 checks passed
@idryomov idryomov deleted the wip-72692 branch August 22, 2025 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants