qa: mds Enable multimds killpoint tests by lxbsz · Pull Request #41969 · ceph/ceph

lxbsz · 2021-06-22T09:09:53Z

This version has been impoved a lot, including hadfailover_rank(),
to make it have the same logic with hadfailover().

And also keeps retry by sleeping 5 seconds every time instead of
hard code waiting 150 seconds to speed up the test. And also some
others small fixings.

Fixes: http://tracker.ceph.com/issues/17835
Signed-off-by: Sidharth Anupkrishnan sanupkri@redhat.com
Signed-off-by: Xiubo Li xiubli@redhat.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

lxbsz · 2021-06-22T09:11:49Z

This is based @sidharthanup's previous PR #28004 and have some improvements about it.

batrick

@lxbsz please run through QA to verify it works when you're done adjusting the code.

Thanks a lot for picking this up.

batrick · 2021-06-25T22:43:48Z

qa/tasks/cephfs/filesystem.py

        #all matching
        return False

+    def hadfailover_rank(self, fscid, status, rank):


This does an implicit ceph fs dump in Filesystem.get_rank.

I think a better place for this is in FSStatus with this signature:

def had_failover_rank(self, fscid, rank, status2):

Looks good to me. Will update it.

batrick · 2021-06-25T22:45:06Z

qa/tasks/cephfs/test_exports.py

+
+        try:
+            # This should kill either or both MDS process
+            self.mount_a.setfattr("abc", "ceph.dir.pin", "1")


What's preventing the balancer from doing part of this export before reaching this point? (I think we need to pin the directory to rank 0 before populating it.)

Good point. Yeah, actually we cannot be sure the abc dir is already pinned or auth in rank 0.

@lxbsz please run through QA to verify it works when you're done adjusting the code.

Sure, will do that.

lxbsz · 2021-06-28T11:59:09Z

jenkins test make check arm64

lxbsz · 2021-06-30T11:12:28Z

Today I test it more and found several bugs in MDS migrate code, I need more time to debug and fix it in late future.

lxbsz · 2021-07-29T02:23:46Z

jenkins test make check arm64

batrick

I'm not sure mds: set session state to _CLOSED when replaying the EImportStart is correct. The MDSMap contains an export_targets set to get clients to connect to potential importers of subtrees.

lxbsz · 2021-08-06T05:24:12Z

I'm not sure mds: set session state to _CLOSED when replaying the EImportStart is correct. The MDSMap contains an export_targets set to get clients to connect to potential importers of subtrees.

Since when writing journal the session was in _CLOSED state, after that in some killpoints the session maybe still in _CLOSED state, in some later it will be _OPENED, so just set it to _OPENED is not correct IMO. When replaying it just set it to the initial state, which is the same with when the journal is flushed.

I tried that weeks ago, it didn't work for me. If the above won't work for your I will try it again next week.

github-actions · 2021-10-21T02:40:40Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

lxbsz · 2021-10-25T02:24:37Z

Rebased to the upstream and remove the first 3 commits, which have been merged in another PR.

lxbsz · 2021-11-17T08:44:02Z

jenkins retest this please

When the importer mds crashes just after the EImportStart journal was flushed, the standby mds will replay it later, and when replaying the EImportStart the standby mds will wait the client to reconnect, but actually the client may not open the session yet. So we need to make sure the export_targets to mdsmap is updated just before the EImportStart log is flushed, then in the Client side we can use this info to reconnect the export target mds. And when the exporter mds crashes and is replaced by a standby mds the export_targets in the mdsmap will be cleaned, so we need to record it by adding EExportStart logevent. Signed-off-by: Xiubo Li <xiubli@redhat.com>

Signed-off-by: Xiubo Li <xiubli@redhat.com>

This version has been impoved a lot, including hadfailover_rank(), to make it have the same logic with hadfailover(). And also keeps retry by sleeping 5 seconds every time instead of hard code waiting 150 seconds to speed up the test. And also some others small fixings. Fixes: http://tracker.ceph.com/issues/17835 Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com> Signed-off-by: Xiubo Li <xiubli@redhat.com>

github-actions · 2022-09-01T00:05:01Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2022-11-28T05:01:37Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

lxbsz · 2023-01-17T02:44:33Z

@vshankar I think we still need this to test the exporting.

vshankar · 2023-01-17T04:23:53Z

@vshankar I think we still need this to test the exporting.

I was not tracking this and I haven't gone through the changes. Any work that is pending?

lxbsz · 2023-01-17T05:02:43Z

@vshankar I think we still need this to test the exporting.

I was not tracking this and I haven't gone through the changes. Any work that is pending?

No work is pending.

github-actions · 2023-03-26T16:01:29Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2023-04-25T16:01:31Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

github-actions bot added cephfs Ceph File System tests labels Jun 22, 2021

lxbsz requested review from a team and batrick June 22, 2021 09:10

batrick mentioned this pull request Jun 25, 2021

qa: enable MDS export killpoint tests #28004

Closed

3 tasks

batrick requested changes Jun 25, 2021

View reviewed changes

lxbsz force-pushed the import_export_killpoint branch from ed2ed9f to 3273fec Compare June 28, 2021 06:33

lxbsz force-pushed the import_export_killpoint branch from 3273fec to 17618a9 Compare July 28, 2021 12:02

lxbsz mentioned this pull request Jul 29, 2021

mds/client: switch to use ceph_assert() instead of assert() #42541

Merged

3 tasks

lxbsz force-pushed the import_export_killpoint branch 3 times, most recently from 462cbe0 to 64e7eb2 Compare August 5, 2021 08:52

lxbsz requested a review from a team as a code owner August 5, 2021 08:52

github-actions bot added build/ops common crimson rbd rgw labels Aug 5, 2021

lxbsz force-pushed the import_export_killpoint branch 2 times, most recently from 1a6fafc to fe92d5c Compare August 5, 2021 11:07

batrick requested changes Aug 5, 2021

View reviewed changes

lxbsz force-pushed the import_export_killpoint branch from fe92d5c to b5cff8b Compare August 6, 2021 05:00

lxbsz force-pushed the import_export_killpoint branch 3 times, most recently from 24c9e52 to 97152c3 Compare August 9, 2021 01:22

github-actions bot added the needs-rebase label Oct 21, 2021

lxbsz force-pushed the import_export_killpoint branch from 580e292 to d274277 Compare October 25, 2021 02:22

lxbsz requested a review from vshankar October 25, 2021 02:23

github-actions bot removed the needs-rebase label Oct 25, 2021

lxbsz added 3 commits November 18, 2021 13:10

qa: allow the coredump dir to be empty

c73e832

Signed-off-by: Xiubo Li <xiubli@redhat.com>

lxbsz force-pushed the import_export_killpoint branch from d274277 to 7f1aa5f Compare November 18, 2021 05:11

tchaikov removed the crimson label Dec 22, 2021

tchaikov removed the request for review from a team December 22, 2021 23:59

djgalloway changed the base branch from master to main July 3, 2022 00:00

github-actions bot added the stale label Sep 1, 2022

github-actions bot closed this Nov 28, 2022

lxbsz removed the stale label Jan 17, 2023

lxbsz reopened this Jan 17, 2023

lxbsz requested a review from a team January 17, 2023 02:43

vshankar added the wip-vshankar-backlog Backlog of CephFS PRs to pick for testing label Jan 25, 2023

vshankar mentioned this pull request Feb 2, 2023

mds: "Failed to authpin,subtree is being exported" results in large number of blocked requests #49940

Closed

14 tasks

github-actions bot added the stale label Mar 26, 2023

github-actions bot closed this Apr 25, 2023

Conversation

lxbsz commented Jun 22, 2021

Checklist

Uh oh!

lxbsz commented Jun 22, 2021

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

batrick Jun 25, 2021

Choose a reason for hiding this comment

Uh oh!

lxbsz Jun 28, 2021

Choose a reason for hiding this comment

Uh oh!

batrick Jun 25, 2021

Choose a reason for hiding this comment

Uh oh!

lxbsz Jun 28, 2021

Choose a reason for hiding this comment

Uh oh!

lxbsz Jun 28, 2021

Choose a reason for hiding this comment

Uh oh!

lxbsz commented Jun 28, 2021

Uh oh!

lxbsz commented Jun 30, 2021

Uh oh!

lxbsz commented Jul 29, 2021

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

lxbsz commented Aug 6, 2021

Uh oh!

github-actions bot commented Oct 21, 2021

Uh oh!

lxbsz commented Oct 25, 2021

Uh oh!

lxbsz commented Nov 17, 2021

Uh oh!

github-actions bot commented Sep 1, 2022

Uh oh!

github-actions bot commented Nov 28, 2022

Uh oh!

lxbsz commented Jan 17, 2023

Uh oh!

vshankar commented Jan 17, 2023

Uh oh!

lxbsz commented Jan 17, 2023

Uh oh!

github-actions bot commented Mar 26, 2023

Uh oh!

github-actions bot commented Apr 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants