mds: abort fragment/export when quiesced by batrick · Pull Request #57059 · ceph/ceph

batrick · 2024-04-23T17:51:32Z

Fixes: https://tracker.ceph.com/issues/65603

Still working on a more clear commit message and maybe a dedicated test.

Checklist

Tracker (select at least one)
- References tracker ticket
Component impact
- No impact that needs to be tracked
Documentation (select at least one)
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

batrick · 2024-04-24T02:23:13Z

I've constructed a unit test to reproduce the issue. I'll finish this up tomorrow.

batrick · 2024-04-25T01:45:28Z

This PR is under test in https://tracker.ceph.com/issues/65661.

leonid-s-usov

Looks great! Canceling the fragmentation due to a quiesce is a good solution, and thanks for the test! Does it cover both fragmenting and exporting?

As for the is_wrlocked(quiescelock), I'd prefer to avoid exposing that implementation detail. Isn't that check equivalent to asking whether it's quiesced? Being xlocked is the only reason for a local wrlock to fail, isn't it?

src/mds/Migrator.cc

src/mds/MDCache.cc

qa/tasks/cephfs/test_quiesce.py

batrick · 2024-04-29T15:17:07Z

Fixed issue observed in

/teuthology/leonidus-2024-04-26_10:54:14-fs-wip-lusov-quiescer-fixes-distro-default-smithi/7674683/teuthology.log

batrick · 2024-04-29T16:20:52Z

jenkins test make check

* refs/pull/57059/head: mds: do not try fragmenting or exporting a quiesced directory mds: pass bypassfreezing to parent auth pin req qa: add quiesce tests during fragmentation qa: translate empty output from rank_tell to empty dict

batrick · 2024-04-29T17:10:50Z

jenkins test windows

batrick · 2024-04-29T17:10:56Z

jenkins test make check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Not all commands return JSON, like `dirfrag split`. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Reproduces: https://tracker.ceph.com/issues/65603 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Credit to Leonid for first noticing this. Fixes: https://tracker.ceph.com/issues/65603 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

This is an optimization to obviate repeated calls to acquire_locks. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

And handle inode becoming quiesced after op is created. Fixes: https://tracker.ceph.com/issues/65603 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick · 2024-04-29T21:09:27Z

This PR is under test in https://tracker.ceph.com/issues/65694.

batrick · 2024-04-30T01:41:02Z

jenkins test make check

batrick · 2024-04-30T01:41:07Z

jenkins test api

batrick · 2024-04-30T16:15:29Z

https://tracker.ceph.com/issues/65694#note-2

needs further review for https://pulpito.ceph.com/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680431/ failure

batrick · 2024-04-30T16:27:46Z

jenkins test api

batrick · 2024-04-30T16:27:52Z

jenkins test make check

batrick · 2024-04-30T17:11:36Z

https://tracker.ceph.com/issues/65716

Not a blocker to merge this in my opinion; it is a related but different type of failure.

batrick · 2024-04-30T17:13:02Z

jenkins test make check arm64

batrick · 2024-04-30T20:45:29Z

jenkins test api

batrick · 2024-04-30T20:45:35Z

jenkins test make check arm64

github-actions bot added the cephfs Ceph File System label Apr 23, 2024

batrick force-pushed the i65603 branch from ccb0fe3 to 28066b5 Compare April 23, 2024 17:53

batrick force-pushed the i65603 branch from 0a09fb8 to 014e6f8 Compare April 24, 2024 20:16

github-actions bot added the tests label Apr 24, 2024

batrick force-pushed the i65603 branch 2 times, most recently from 41ba94d to 3b57d8c Compare April 25, 2024 01:32

batrick closed this Apr 25, 2024

batrick deleted the i65603 branch April 25, 2024 01:32

batrick restored the i65603 branch April 25, 2024 01:32

batrick reopened this Apr 25, 2024

batrick requested a review from leonid-s-usov April 25, 2024 01:33

batrick added the wip-pdonnell-testing2 label Apr 25, 2024

batrick changed the title ~~mds: pass bypassfreezing to parent auth pin req~~ mds: abort fragment/export when quiesced Apr 25, 2024

batrick force-pushed the i65603 branch from 3b57d8c to d050f9f Compare April 25, 2024 01:58

leonid-s-usov suggested changes Apr 25, 2024

View reviewed changes

src/mds/Migrator.cc Outdated Show resolved Hide resolved

src/mds/MDCache.cc Outdated Show resolved Hide resolved

src/mds/MDCache.cc Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_quiesce.py Outdated Show resolved Hide resolved

batrick force-pushed the i65603 branch 2 times, most recently from ef2d139 to bc4e33c Compare April 29, 2024 15:16

batrick force-pushed the i65603 branch from bc4e33c to 0a96be2 Compare April 29, 2024 15:27

batrick added 2 commits April 29, 2024 16:59

qa: move reqid_tostr helper

78a82ee

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: translate empty output from rank_tell to empty dict

f543379

Not all commands return JSON, like `dirfrag split`. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick force-pushed the i65603 branch from 0a96be2 to cf10002 Compare April 29, 2024 21:00

batrick added 2 commits April 29, 2024 17:08

qa: add quiesce tests during fragmentation

527f46f

Reproduces: https://tracker.ceph.com/issues/65603 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: pass bypassfreezing to parent auth pin req

ded6a1c

Credit to Leonid for first noticing this. Fixes: https://tracker.ceph.com/issues/65603 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick added 2 commits April 29, 2024 17:08

mds: set/test ALL_LOCKED on fragment_dir request

fca346d

This is an optimization to obviate repeated calls to acquire_locks. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: do not try fragmenting or exporting a quiesced directory

67925a3

And handle inode becoming quiesced after op is created. Fixes: https://tracker.ceph.com/issues/65603 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

batrick force-pushed the i65603 branch from cf10002 to 67925a3 Compare April 29, 2024 21:08

leonid-s-usov approved these changes May 1, 2024

View reviewed changes

batrick merged commit 10acaf6 into ceph:main May 1, 2024

batrick deleted the i65603 branch May 1, 2024 12:58

batrick mentioned this pull request May 1, 2024

squid: mds: abort fragment/export when quiesced #57203

Merged

Conversation

batrick commented Apr 23, 2024 • edited by leonid-s-usov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

batrick commented Apr 24, 2024

Uh oh!

batrick commented Apr 25, 2024

Uh oh!

leonid-s-usov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

batrick commented Apr 29, 2024

Uh oh!

batrick commented Apr 29, 2024

Uh oh!

batrick commented Apr 29, 2024

Uh oh!

batrick commented Apr 29, 2024

Uh oh!

batrick commented Apr 29, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

batrick commented Apr 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

batrick commented Apr 23, 2024 •

edited by leonid-s-usov

Loading