Problem:

1. There are certain scenarios in degraded
   stretched cluster where will try to
   go into the
   function Monitor::go_recovery_stretch_mode()
   that will lead to a ceph_assert.

2. In the function maybe_go_degraded_stretch_mode()
   when osdmon is not writeable we shouldn't go into
   trigger_degraded_stretch_mode because we will
   crash at ceph_assert(osdmon()->is_writeable()).
   The current code does not exit maybe_go_degraded_stretch_mode()
   when we are waiting for osdmon to be writeable, therefore,
   we crash.

Solution:

1. Make sure dead_mon_buckets.size() == 0
   in OSDMonitor:update_from_paxos()
   before going into Monitor::go_recovery_stretch_mode().
2. Exit the function by returning nothing after going into
   wait_for_writeable_ctx, since at that point we would have
   queued the context and all we have to do is wait for finish
   context to execute maybe_go_degraded_stretch_mode again.
   Also, added a bit of logging so that user is aware
   when osdmon and monmon are not writeable.
   We fix other parts of the monitor code that are missing
   the return after wait_for_writeable_ctx and wait_for_readable_ctx
   as well.

Fixes:
https://tracker.ceph.com/issues/57017
https://tracker.ceph.com/issues/59271

Backporting relevant commits from main PRs:

#47340
#50857

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests