mon [stretch mode]: support disable_stretch_mode by kamoltat · Pull Request #59483 · ceph/ceph

kamoltat · 2024-08-28T13:48:54Z

Problem:

Currently, Ceph lacks the ability
to exit stretch mode and move back
to normal cluster (non-stretched).

Solution:

Provide a command to allow
the user to exit stretch mode gracefully:

ceph mon disable_stretch_mode <crush_rule> --yes-i-really-mean-it

Users can either specify a crush rule that
they want all pools to move to or not specify
a rule and Ceph will use a default replicated crush rule.

Fixes: https://tracker.ceph.com/issues/67467

TODO

Code changes
Test changes that exercises all the case
Documentation changes

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

anthonyeleven · 2024-09-08T00:49:28Z

+
+.. describe:: {crush_rule}
+
+   The CRUSH rule that the user wants all the pools to move back to. If this


anthonyeleven · 2024-09-08T00:49:41Z

+.. describe:: {crush_rule}
+
+   The CRUSH rule that the user wants all the pools to move back to. If this
+   is not specified, the cluster will move back to the default CRUSH rule.


s/cluster/pools/. ?

anthonyeleven · 2024-09-08T00:51:04Z

+All pools will move its ``size`` and ``min_size``
+back to the default values it started with.
+At this point the user is responsible for scaling down the cluster
+to the desired number of OSDs.


Why would exiting stretch mode necessarily mean taking out OSDs?

One of the use cases is that the user may encounter a whole DC failure and doesn't want to bother bringing the failed DC back up and wants to continue using the surviving DC without stretch mode.

I suspected as much, but as written this might confuse a user who thinks that scaling OSDs is necessary in all cases.

@anthonyeleven
That is a good point, I added.

At this point the user is responsible for scaling down the cluster to the desired number of OSDs if they choose to operate with less number of OSDs.

Let me know if you think this clarifies the point I'm trying to make or should I just delete this sentence completely.

Thanks

kamoltat · 2024-09-08T19:38:03Z

status:

Code changes completed
Documentation first draft, need to resolve feedback provided by Anthony
Test coverage: Added a setup script, still need to write a python integration test

kamoltat · 2024-09-14T18:34:54Z

wrote python integration test

however still need to modify the command to include --yes-i-really-really-mean-it

Problem: Currently, Ceph lacks the ability to exit stretch mode and move back to normal cluster (non-stretched). Solution: Provide a command to allow the user to exit stretch mode gracefully: `ceph mon disable_stretch_mode <crush_rule> --yes-i-really-mean-it` User can either specify a crush rule that they want all pools to move to or not specify a rule and Ceph will use a default replicated crush rule. Fixes: https://tracker.ceph.com/issues/67467 Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Added documentation about exiting stretch mode. Fixes: https://tracker.ceph.com/issues/67467 Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

NitzanMordhai · 2024-09-19T04:37:22Z

+    log-ignorelist:
+      - overall HEALTH_
+      - \(OSDMAP_FLAGS\)
+      - \(OSD_
+      - \(PG_
+      - \(POOL_
+      - \(CACHE_POOL_
+      - \(OBJECT_
+      - \(SLOW_OPS\)
+      - \(REQUEST_SLOW\)
+      - \(TOO_FEW_PGS\)
+      - slow request
+      - \(POOL_APP_NOT_ENABLED\)
+      - overall HEALTH_
+      - \(MGR_DOWN\)
+      - \(MON_DOWN\)
+      - \(PG_AVAILABILITY\)
+      - \(SLOW_OPS\)


@kamoltat do you think we can eliminate a few of the log-ignorelist? Or should we split it so we have it before and after the conversion ignorelist ?
my concern is that PG_* will be ignored and we will miss important warnings after the conversion happened

That's a good point @NitzanMordhai let me remove most of the ones that can affect the integrity of the test.

@NitzanMordhai I had to put in things like MON_DOWN and PG_AVAILABILITY since my test involves a lot of failover. Honestly, this grep for warning in the cluster log adds more issues than it actually solves. Maybe worth discussing with the team about it.

kamoltat · 2024-09-20T21:57:03Z

Passed 1/1 test
https://pulpito.ceph.com/ksirivad-2024-09-20_21:29:53-rados:singleton-wip-ksirivad-exit-stretch-mode-distro-default-smithi/

Test disabling stretch mode with the following scenario: 1. Healthy Stretch Mode 2. Degraded Stretch Mode Fixes: https://tracker.ceph.com/issues/67467 Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Problem: Current dump for "removed_ranks" and "disallowed_leaders" doesn't have the correct format so the python test script can parse through these values. Solution: Modified the values such that it is in the correct format Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

kamoltat · 2024-09-23T02:56:57Z

https://pulpito.ceph.com/ksirivad-2024-09-22_17:16:54-rados:singleton-wip-ksirivad-exit-stretch-mode-distro-default-smithi/

Passed 1/1 Test after addressing Nitzan remarks on the ignore list.

kamoltat · 2024-09-23T03:03:16Z

jenkins test windows

kamoltat · 2024-09-23T03:03:19Z

jenkins test api

kamoltat · 2024-09-23T03:05:04Z

hmm something wrong with build system ...

kamoltat · 2024-09-23T19:25:33Z

jenkins test windows

kamoltat · 2024-09-23T19:26:47Z

@NitzanMordhai let me know what you think of the new change on the log-ignorelist

NitzanMordhai · 2024-09-24T05:34:48Z

@NitzanMordhai let me know what you think of the new change on the log-ignorelist

@kamoltat I have an issue with PG_DEGRADED, let's say you moved between modes, and PG gets stuck with PG_DEGRADED, how will you catch such an error? the test will ignore it. this issue is not specific to that test only, we have more suites that have that, but we can check at the last iterator if we have any warning before closing the test (if it has already cleared, we are ok with that).

kamoltat · 2024-09-24T14:17:50Z

@NitzanMordhai let me know what you think of the new change on the log-ignorelist

@kamoltat I have an issue with PG_DEGRADED, let's say you moved between modes, and PG gets stuck with PG_DEGRADED, how will you catch such an error? the test will ignore it. this issue is not specific to that test only, we have more suites that have that, but we can check at the last iterator if we have any warning before closing the test (if it has already cleared, we are ok with that).

@NitzanMordhai I understand your concern, however, in my test I do check for all PGs to be active+clean when switching between the modes, e.g.,

            ))
        # Disable stretch mode without crush rule (expect success 0)
        self.assertEqual(
            0,
            self.mgr_cluster.mon_manager.raw_cluster_cmd_result(
                'mon',
                'disable_stretch_mode',
                '--yes-i-really-mean-it'
            ))
        # Check if stretch mode is disabled correctly
        self._stretch_mode_disabled_correctly()
        # all PGs are active + clean
        self.wait_until_true_and_hold(
            lambda: self.mgr_cluster.mon_manager.pg_all_active_clean(),
            timeout=self.RECOVERY_PERIOD,
            success_hold_time=self.SUCCESS_HOLD_TIME
        )
        # write some data to the pool
        self._write_some_data(self.WRITE_PERIOD)
        # Enable stretch mode
        self.assertEqual(
            0,
            self.mgr_cluster.mon_manager.raw_cluster_cmd_result(
                'mon',
                'enable_stretch_mode',
                self.TIEBREAKER_MON_NAME,
                self.STRETCH_CRUSH_RULE,
                self.STRETCH_BUCKET_TYPE
            ))
        self._stretch_mode_enabled_correctly()
        # all PGs are active + clean
        self.wait_until_true_and_hold(
            lambda: self.mgr_cluster.mon_manager.pg_all_active_clean(),
            timeout=self.RECOVERY_PERIOD,
            success_hold_time=self.SUCCESS_HOLD_TIME
        )
        # write some data to the pool

the only time I am willing to tolerate degraded PGs is when I am testing the case where the stretch cluster is in degraded stretch mode (PGs are expected to be degraded because we loss 1 DC) and the user is disabling stretch mode and going back to a normal cluster. Therefore, I think grepping for PG_DEGRADED at the end of this test won't help us in this case.

kamoltat · 2024-09-30T15:54:13Z

@NitzanMordhai ping, let me know what you think of the comment above

NitzanMordhai · 2024-10-06T06:17:22Z

@NitzanMordhai ping, let me know what you think of the comment above

well, in that

the only time I am willing to tolerate degraded PGs is when I am testing the case where the stretch cluster is in degraded stretch mode (PGs are expected to be degraded because we loss 1 DC) and the user is disabling stretch mode and going back to a normal cluster. Therefore, I think grepping for PG_DEGRADED at the end of this test won't help us in this case.

in that case, that's perfectly fine! then we are covered and not allowing this situation, thanks for checking that!

NitzanMordhai

LGTM!

kamoltat · 2024-11-05T18:06:03Z

RADOS approved 119/153 test passed.
https://pulpito.ceph.com/ksirivad-2024-10-29_19:55:42-rados-wip-ksirivad-exit-stretch-mode-distro-default-smithi/
The remaining failures are due to infrastructure and known issues.

kamoltat added core needs-squid-backport PR needs a squid backport labels Aug 28, 2024

kamoltat self-assigned this Aug 28, 2024

kamoltat requested a review from a team as a code owner August 28, 2024 13:48

kamoltat changed the title ~~[wip] mon: support exiting stretch mode~~ [WIP] mon: support exiting stretch mode Aug 28, 2024

github-actions Bot added the mon label Aug 28, 2024

kamoltat changed the title ~~[WIP] mon: support exiting stretch mode~~ [WIP] [Stretch Mode] mon: support exiting stretch mode Aug 30, 2024

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch from e2810ca to 0ecd953 Compare September 7, 2024 22:40

kamoltat changed the title ~~[WIP] [Stretch Mode] mon: support exiting stretch mode~~ [WIP] mon [stretch mode]: support disable_stretch_mode Sep 7, 2024

kamoltat changed the title ~~[WIP] mon [stretch mode]: support disable_stretch_mode~~ mon [stretch mode]: support disable_stretch_mode Sep 7, 2024

kamoltat added the needs-reef-backport label Sep 7, 2024

kamoltat requested a review from a team as a code owner September 7, 2024 23:02

github-actions Bot added the documentation label Sep 7, 2024

anthonyeleven reviewed Sep 8, 2024

View reviewed changes

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch from f4b00cd to 5e32cfc Compare September 8, 2024 19:27

github-actions Bot added the tests label Sep 8, 2024

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch 2 times, most recently from bbbd4fa to 97f54a6 Compare September 14, 2024 18:26

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch 3 times, most recently from a120c38 to 89ecbec Compare September 17, 2024 21:48

kamoltat added 2 commits September 18, 2024 18:09

doc/rados/operations/stretch-mode.rst: Added Exitting Stretch Mode

0680f17

Added documentation about exiting stretch mode. Fixes: https://tracker.ceph.com/issues/67467 Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch from 89ecbec to 80f0c1e Compare September 18, 2024 18:09

NitzanMordhai reviewed Sep 19, 2024

View reviewed changes

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch 3 times, most recently from 18b0f24 to 4ca4b52 Compare September 19, 2024 21:15

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch 2 times, most recently from b1bfcf0 to 2d28d24 Compare September 20, 2024 21:29

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch 2 times, most recently from e2bcb8d to 768c374 Compare September 22, 2024 03:46

kamoltat added 2 commits September 22, 2024 17:12

qa: Added tests for disabling stretch mode

4d2f887

Test disabling stretch mode with the following scenario: 1. Healthy Stretch Mode 2. Degraded Stretch Mode Fixes: https://tracker.ceph.com/issues/67467 Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

kamoltat force-pushed the wip-ksirivad-exit-stretch-mode branch from 768c374 to a7f3b7b Compare September 22, 2024 17:12

NitzanMordhai approved these changes Oct 6, 2024

View reviewed changes

kamoltat added the needs-qa label Oct 28, 2024

kamoltat merged commit 28e38e3 into ceph:main Nov 5, 2024

This was referenced Nov 5, 2024

squid: mon [stretch mode]: support disable_stretch_mode #60629

Merged

reef: mon [stretch mode]: support disable_stretch_mode & qa/workunits/mon: ensure election strategy is "connectivity" for stretch mode #60630

Merged


		.. describe:: {crush_rule}

		The CRUSH rule that the user wants all the pools to move back to. If this

Conversation

kamoltat commented Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Contribution Guidelines

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kamoltat commented Sep 8, 2024

Uh oh!

kamoltat commented Sep 14, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kamoltat commented Sep 20, 2024

Uh oh!

kamoltat commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kamoltat commented Sep 23, 2024

Uh oh!

kamoltat commented Sep 23, 2024

Uh oh!

kamoltat commented Sep 23, 2024

Uh oh!

kamoltat commented Sep 23, 2024

Uh oh!

kamoltat commented Sep 23, 2024

Uh oh!

NitzanMordhai commented Sep 24, 2024

Uh oh!

kamoltat commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kamoltat commented Sep 30, 2024

Uh oh!

NitzanMordhai commented Oct 6, 2024

Uh oh!

NitzanMordhai left a comment

Choose a reason for hiding this comment

Uh oh!

kamoltat commented Nov 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kamoltat commented Aug 28, 2024 •

edited

Loading

kamoltat commented Sep 23, 2024 •

edited

Loading

kamoltat commented Sep 24, 2024 •

edited

Loading