mds: prevent scrubbing for standby-replay MDS by neesingh-rh · Pull Request #53301 · ceph/ceph

neesingh-rh · 2023-09-06T06:29:16Z

Fixes: https://tracker.ceph.com/issues/62537

Signed-off-by: Neeraj Pratap Singh neesingh@redhat.com

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

src/mds/MDSRank.cc

vshankar · 2023-09-26T15:14:34Z

https://pulpito.ceph.com/?branch=wip-vshankar-testing-20230926.081818

src/mds/MDSRank.cc

vshankar · 2023-10-04T06:25:56Z

@neesingh-rh please ping when this ready for review again.

neesingh-rh · 2023-10-05T06:33:02Z

@neesingh-rh please ping when this ready for review again.

I have updated the PR, ready for re-review.

mchangir

LGTM

* refs/pull/53301/head: mds: prevent scrub start for standby-replay MDS Reviewed-by: Venky Shankar <vshankar@redhat.com>

vshankar

@neesingh-rh LGTM -- please add a test.

neesingh-rh · 2023-10-10T10:38:09Z

@neesingh-rh LGTM -- please add a test.

Added the test. PTAL

src/mds/MDSRank.cc

qa/tasks/cephfs/test_scrub_checks.py

batrick · 2023-10-17T16:10:45Z

qa/tasks/cephfs/test_scrub_checks.py

+        # start the scrub and verify
+        with self.assertRaises(CommandFailedError) as ce:
+            self.fs.run_scrub(["start", abs_test_path, "recursive"])
+        self.assertEqual(ce.exception.exitstatus, errno.EINVAL)


Suggested change

self.assertEqual(ce.exception.exitstatus, errno.EINVAL)

self.assertEqual(ce.exception.exitstatus, errno.EINVAL)

and others below

We should be checking for the error status only after getting the command fail exit status. I guess it should be this way only. https://github.com/ceph/ceph/blob/quincy/qa/tasks/cephfs/test_scrub_checks.py#L354

ce would be available inside with .., isn't it?

with.. is an alternative for try-catch, how can the statement be executed after the exception is encountered at the run_scrub only. Pls correct me if you mean something else.

I guess this is fine too

batrick · 2023-10-17T16:12:52Z

qa/tasks/cephfs/test_scrub_checks.py

+
+        # start the scrub and verify
+        with self.assertRaises(CommandFailedError) as ce:
+            self.fs.run_scrub(["start", abs_test_path, "recursive"])


I'm puzzled. Is this test not racing with the MDS becoming active?

I also expected to see a scrub command directed at the standby-replay daemon.

Spoke to @neesingh-rh - this needs fixing. Fetch the s-r mds daemons id and do a ceph tell mds.<> scrub start to the s-r daemon.

Updated the PR with the proposed changes.

vshankar · 2023-11-28T09:45:53Z

@neesingh-rh ping?

github-actions · 2024-01-27T11:01:26Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

vshankar · 2024-07-03T09:55:47Z

jenkins test make check

vshankar · 2024-07-04T05:56:35Z

@neesingh-rh please relook the make check failure.

vshankar · 2024-07-05T16:14:24Z

@neesingh-rh I see you added a change to fix a failing dashboard test and then removed it??

neesingh-rh · 2024-07-05T16:38:35Z

@neesingh-rh I see you added a change to fix a failing dashboard test and then removed it??

Nope, I pushed a commit by mistake that wasn't part of this PR. I added it locally just for testing.

neesingh-rh · 2024-07-05T16:39:14Z

vstart_runner results:

2024-07-04 11:42:00,758.758 INFO:__main__:Stopped test: test_scrub_when_mds_is_inactive (tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive) in 63.529271s
2024-07-04 11:42:00,758.758 INFO:__main__:test_scrub_when_mds_is_inactive (tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive) ... ok
2024-07-04 11:42:00,758.758 INFO:__main__:
2024-07-04 11:42:00,758.758 INFO:__main__:----------------------------------------------------------------------
2024-07-04 11:42:00,758.758 INFO:__main__:Ran 1 test in 63.530s
2024-07-04 11:42:00,758.758 INFO:__main__:
2024-07-04 11:42:00,758.758 INFO:__main__:OK
2024-07-04 11:42:00,758.758 INFO:__main__:

vshankar · 2024-07-08T07:30:33Z

jenkins test dashboard

vshankar · 2024-07-08T07:30:37Z

jenkins test docs

Fixes: https://tracker.ceph.com/issues/62537 Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

vshankar · 2024-07-09T04:25:39Z

jenkins test make check

Comments addressed and approved.

* refs/pull/53301/head: qa: adding test for preventing scrub when mds is inactive mds: prevent scrub start for standby-replay MDS Reviewed-by: Dhairya Parmar <dparmar@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Milind Changire <mchangir@redhat.com>

github-actions bot added the cephfs Ceph File System label Sep 6, 2023

batrick requested changes Sep 12, 2023

View reviewed changes

src/mds/MDSRank.cc Outdated Show resolved Hide resolved

neesingh-rh force-pushed the wip-62537 branch 2 times, most recently from 631bb8b to c181dd8 Compare September 13, 2023 09:46

vshankar approved these changes Sep 26, 2023

View reviewed changes

vshankar added the wip-vshankar-testing3 label Sep 26, 2023

neesingh-rh requested a review from batrick September 26, 2023 11:37

dparmar18 approved these changes Sep 26, 2023

View reviewed changes

mchangir reviewed Sep 27, 2023

View reviewed changes

src/mds/MDSRank.cc Show resolved Hide resolved

neesingh-rh force-pushed the wip-62537 branch from c181dd8 to 8982199 Compare September 28, 2023 13:37

dparmar18 reviewed Sep 28, 2023

View reviewed changes

src/mds/MDSRank.cc Show resolved Hide resolved

vshankar removed the wip-vshankar-testing3 label Oct 4, 2023

mchangir approved these changes Oct 5, 2023

View reviewed changes

vshankar added a commit to vshankar/ceph that referenced this pull request Oct 7, 2023

Merge PR ceph#53301 into wip-vshankar-testing-20230926.081818

7e9ae82

* refs/pull/53301/head: mds: prevent scrub start for standby-replay MDS Reviewed-by: Venky Shankar <vshankar@redhat.com>

vshankar approved these changes Oct 9, 2023

View reviewed changes

vshankar requested changes Oct 9, 2023

View reviewed changes

github-actions bot added the tests label Oct 10, 2023

dparmar18 reviewed Oct 11, 2023

View reviewed changes

src/mds/MDSRank.cc Show resolved Hide resolved

dparmar18 reviewed Oct 11, 2023

View reviewed changes

qa/tasks/cephfs/test_scrub_checks.py Outdated Show resolved Hide resolved

neesingh-rh force-pushed the wip-62537 branch from 807985f to 5f504d4 Compare October 11, 2023 13:56

vshankar approved these changes Oct 12, 2023

View reviewed changes

vshankar added the wip-vshankar-testing4 label Oct 12, 2023

batrick previously requested changes Oct 17, 2023

View reviewed changes

vshankar removed the wip-vshankar-testing4 label Oct 20, 2023

vshankar removed the wip-vshankar-testing3 label Jul 1, 2024

neesingh-rh force-pushed the wip-62537 branch from 6940d3a to ca6c9fa Compare July 4, 2024 08:22

neesingh-rh requested a review from a team as a code owner July 4, 2024 08:22

neesingh-rh requested review from aaSharma14 and nmunet and removed request for a team July 4, 2024 08:23

github-actions bot added dashboard mgr pybind labels Jul 4, 2024

neesingh-rh force-pushed the wip-62537 branch from ca6c9fa to aff442c Compare July 4, 2024 08:24

neesingh-rh removed request for aaSharma14 and nmunet July 4, 2024 08:25

neesingh-rh removed pybind mgr dashboard labels Jul 4, 2024

qa: adding test for preventing scrub when mds is inactive

b9a2d05

Fixes: https://tracker.ceph.com/issues/62537 Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

neesingh-rh force-pushed the wip-62537 branch from aff442c to b9a2d05 Compare July 8, 2024 10:04

vshankar merged commit 69704e9 into ceph:main Jul 9, 2024

vshankar removed the ready-to-merge label Jul 9, 2024

neesingh-rh mentioned this pull request Jul 24, 2024

quincy: mds: prevent scrubbing for standby-replay MDS #58799

Merged

	self.assertEqual(ce.exception.exitstatus, errno.EINVAL)
	self.assertEqual(ce.exception.exitstatus, errno.EINVAL)

Conversation

neesingh-rh commented Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

Uh oh!

vshankar commented Sep 26, 2023

Uh oh!

Uh oh!

Uh oh!

vshankar commented Oct 4, 2023

Uh oh!

neesingh-rh commented Oct 5, 2023

Uh oh!

mchangir left a comment

Choose a reason for hiding this comment

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

neesingh-rh commented Oct 10, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vshankar Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vshankar commented Nov 28, 2023

Uh oh!

github-actions bot commented Jan 27, 2024

Uh oh!

vshankar commented Jul 3, 2024

Uh oh!

vshankar commented Jul 4, 2024

Uh oh!

vshankar commented Jul 5, 2024

Uh oh!

neesingh-rh commented Jul 5, 2024

Uh oh!

neesingh-rh commented Jul 5, 2024

Uh oh!

vshankar commented Jul 8, 2024

Uh oh!

vshankar commented Jul 8, 2024

Uh oh!

vshankar commented Jul 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

neesingh-rh commented Sep 6, 2023 •

edited

Loading

vshankar Apr 2, 2024 •

edited

Loading