mds: blocklist clients with "bloated" session metadata by vshankar · Pull Request #52944 · ceph/ceph

vshankar · 2023-08-11T11:31:48Z

Fixes: http://tracker.ceph.com/issues/61947

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

batrick

I think a PendingReleaseNote would be appropriate.

It'd also be good if we had some kind of cluster warning about this. Maybe add a message parameter to evict_client so it can issue the cluster warning (I'm not sure SessionMap.cc can do it easily).

src/mds/SessionMap.cc

qa/tasks/cephfs/test_client_limits.py

dparmar18 · 2023-08-16T10:43:18Z

@vshankar the comments are yet to be addressed right?

vshankar · 2023-08-16T10:45:32Z

@vshankar the comments are yet to be addressed right?

Pushing an update (I mark it resolved before an update just for my sanity).

qa/tasks/cephfs/test_client_limits.py

src/mds/SessionMap.h

vshankar · 2023-08-17T05:47:02Z

Added note in PendingReleaseNotes.

src/common/options/mds.yaml.in

src/mds/SessionMap.cc

robbat2

approving as some of the original users affected by this. Minor ask, not-critical: having a perf metric about how many sessions exceeded the threshold would be valuable.

vshankar · 2023-08-21T05:00:19Z

approving as some of the original users affected by this. Minor ask, not-critical: having a perf metric about how many sessions exceeded the threshold would be valuable.

Good idea. I'll include this and push an update.

Buggy clients (or maybe a MDS bug) causes a huge buildup of `completed_requests` metadata in its session information. This could cause the MDS to go read-only when its flushing session metadata to the journal since the bloated metadata causes the ODSOp payload to exceed the maximum write size. Blocklist such clients so as to allow the MDS to continue servicing requests. Fixes: http://tracker.ceph.com/issues/61947 Signed-off-by: Venky Shankar <vshankar@redhat.com>

…data threshold being exceeded Signed-off-by: Venky Shankar <vshankar@redhat.com>

... when its session metadata is bloated due to buildup of `completed_requests`. Signed-off-by: Venky Shankar <vshankar@redhat.com>

…mds config Signed-off-by: Venky Shankar <vshankar@redhat.com>

vshankar · 2023-08-22T04:51:49Z

fixed and updated.

dparmar18

LGTM

vshankar · 2023-08-23T14:40:35Z

fs suite test results - https://pulpito.ceph.com/vshankar-2023-08-22_10:18:46-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/

will go through the run list tomorrow and do the needful.

vshankar · 2023-08-25T02:18:18Z

fs suite test results - https://pulpito.ceph.com/vshankar-2023-08-22_10:18:46-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/

will go through the run list tomorrow and do the needful.

Test run looks fine - no failures related to this change. I'll merge this shortly.

vshankar · 2023-08-25T02:18:31Z

jenkins test api

vshankar

https://tracker.ceph.com/projects/cephfs/wiki/Main#25-Aug-2023

izxl007 · 2025-08-20T01:34:01Z

@vshankar Hi~
There is another scenario regarding this issue.
When the total size of all sessions exceeds the limit value of the OSD, it will also cause that the MDS goes read-only.
My idea is to flash the sessions in batches, and the size of each batch should not exceed the limit value of the OSD.
I don't know if this idea has any other impacts. Could you give me some suggestions?

vshankar added the cephfs Ceph File System label Aug 11, 2023

vshankar requested a review from a team August 11, 2023 11:31

github-actions bot added common tests labels Aug 11, 2023

batrick requested changes Aug 11, 2023

View reviewed changes

vshankar force-pushed the wip-61947 branch from e5093e8 to a4de150 Compare August 16, 2023 10:52

batrick approved these changes Aug 16, 2023

View reviewed changes

batrick added the needs-qa label Aug 16, 2023

dparmar18 reviewed Aug 16, 2023

View reviewed changes

qa/tasks/cephfs/test_client_limits.py Show resolved Hide resolved

dparmar18 reviewed Aug 17, 2023

View reviewed changes

src/mds/SessionMap.h Show resolved Hide resolved

vshankar force-pushed the wip-61947 branch from a4de150 to 2e54884 Compare August 17, 2023 05:38

github-actions bot added the documentation label Aug 17, 2023

vshankar force-pushed the wip-61947 branch from 2e54884 to 4e1de47 Compare August 17, 2023 06:20

robbat2 reviewed Aug 17, 2023

View reviewed changes

src/common/options/mds.yaml.in Outdated Show resolved Hide resolved

robbat2 reviewed Aug 17, 2023

View reviewed changes

src/mds/SessionMap.cc Show resolved Hide resolved

robbat2 approved these changes Aug 18, 2023

View reviewed changes

vshankar added 4 commits August 22, 2023 00:47

mds: add perf counter to track number of sessions evicted due to meta…

59dd587

…data threshold being exceeded Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add test to verify that a buggy client is blocklisted

84df4b3

... when its session metadata is bloated due to buildup of `completed_requests`. Signed-off-by: Venky Shankar <vshankar@redhat.com>

PendingReleaseNotes: add a note for mds_session_metadata_threshold …

ac3ab1a

…mds config Signed-off-by: Venky Shankar <vshankar@redhat.com>

vshankar force-pushed the wip-61947 branch from 4e1de47 to ac3ab1a Compare August 22, 2023 04:47

vshankar added the wip-vshankar-testing5 label Aug 22, 2023

dparmar18 approved these changes Aug 22, 2023

View reviewed changes

vshankar commented Aug 25, 2023

View reviewed changes

vshankar merged commit e9f8be4 into ceph:main Aug 25, 2023

vshankar removed the wip-vshankar-testing5 label Aug 25, 2023

vshankar mentioned this pull request Sep 25, 2023

pacific: mds: blocklist clients with "bloated" session metadata #53634

Merged

Conversation

vshankar commented Aug 11, 2023 • edited by dparmar18 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dparmar18 commented Aug 16, 2023

Uh oh!

vshankar commented Aug 16, 2023

Uh oh!

Uh oh!

Uh oh!

vshankar commented Aug 17, 2023

Uh oh!

Uh oh!

Uh oh!

robbat2 left a comment

Choose a reason for hiding this comment

Uh oh!

vshankar commented Aug 21, 2023

Uh oh!

vshankar commented Aug 22, 2023

Uh oh!

dparmar18 left a comment

Choose a reason for hiding this comment

Uh oh!

vshankar commented Aug 23, 2023

Uh oh!

vshankar commented Aug 25, 2023

Uh oh!

vshankar commented Aug 25, 2023

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

izxl007 commented Aug 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vshankar commented Aug 11, 2023 •

edited by dparmar18

Loading