mds: add mds_bal_rank_mask config option by yongseokoh · Pull Request #43284 · ceph/ceph

yongseokoh · 2021-09-23T11:13:08Z

This PR introduces mds_bal_rank_mask option to rebalance subtrees within certain active MDSs. With this option, some active ranks are used for statically pinned subdirs, whereas the rest ranks are for subtrees to be rebalanced dynamically by MDBalancer.

fixes: https://tracker.ceph.com/issues/52720

Signed-off-by: Yongseok Oh yongseok.oh@linecorp.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

yongseokoh · 2021-10-08T02:35:09Z

@varshar16 Ping?

jtlayton · 2021-10-08T12:55:24Z

I think this needs an explanation of the benefit it offers. When would I set this option, and how? Is this for testing, or some other purpose? It's not clear to me how you expect this to be used.

yongseokoh · 2021-10-13T04:45:36Z

@jtlayton The main benefit of this feature is that it allows administrators to manage dynamic subtree partitioning and static pining schemes in separate active MDS ranks. Specifically, in our production cluster, a lot of sub-volumes are allotted to multiple VMs. For management convenience, most subvolumes are not accessed frequently, so dynamic balancing is considered, and some subvolumes that require high performance are managed by static partitioning. However, the existing dynamic balancing spreads evenly metadata workloads to all active MDS ranks. Performances of static pinned subvolumes inevitably may be degraded. Therefore, if this approach is applied, static/dynamic partitioning can be applied separately to different MDS ranks. For instance, assume that we have four active MDS ranks, 0 and 1 active ranks can only be used for dynamic balancing and the remaining 2 and 3 active ranks can be used for static pinning.

Active 0 and 1 ranks are used for dynamic balancing with the below command line.
ceph config set mds mds_bal_rank_mask 0x3

if there are no pined subdirs, metadata requests are submitted to only active 0 and 1 ranks. You can see it with ceph fs status.

yongseokoh · 2021-11-01T05:23:05Z

Ping?

github-actions · 2021-11-01T19:15:46Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar · 2021-11-02T04:59:04Z

@jtlayton The main benefit of this feature is that it allows administrators to manage dynamic subtree partitioning and static pining schemes in separate active MDS ranks. Specifically, in our production cluster, a lot of sub-volumes are allotted to multiple VMs. For management convenience, most subvolumes are not accessed frequently, so dynamic balancing is considered, and some subvolumes that require high performance are managed by static partitioning. However, the existing dynamic balancing spreads evenly metadata workloads to all active MDS ranks. Performances of static pinned subvolumes inevitably may be degraded. Therefore, if this approach is applied, static/dynamic partitioning can be applied separately to different MDS ranks. For instance, assume that we have four active MDS ranks, 0 and 1 active ranks can only be used for dynamic balancing and the remaining 2 and 3 active ranks can be used for static pinning.

Active 0 and 1 ranks are used for dynamic balancing with the below command line. ceph config set mds mds_bal_rank_mask 0x3

if there are no pined subdirs, metadata requests are submitted to only active 0 and 1 ranks. You can see it with ceph fs status.

This functionality looks useful for certain use-cases as you mention. Thanks for explaining.

Could you rebase this PR please?

yongseokoh · 2021-11-02T12:32:40Z

jenkins test make check

yongseokoh · 2021-11-02T13:02:19Z

This PR has been rebased. Could you please take a look at it?

vshankar · 2021-11-24T04:27:52Z

thanks @yongseokoh - I'll review and put it to test next week.

yongseokoh · 2021-11-26T09:52:17Z

@vshankar I am looking forward to your feedback.

vshankar

I'm still playing with this change. Providing my initial set of comments.

@yongseokoh It would be good add some tests too. Also, FWIW, have you looked or tried using mantle0 to replicate this functionality?

src/mds/MDBalancer.cc

github-actions · 2021-12-01T06:52:36Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar · 2021-12-03T09:46:03Z

@yongseokoh - I'll take a look. Thanks!

BTW, have you looked at mantle (https://docs.ceph.com/en/latest/cephfs/mantle/) as an alternate way to achieve this functionality?

yongseokoh · 2021-12-04T01:37:51Z

@vshankar Thanks for your valuable recommendation. I will go over its feasibility.

yongseokoh · 2021-12-22T01:52:53Z

@vshankar Do you have any comments?

vshankar · 2022-01-06T09:57:27Z

@vshankar Do you have any comments?

Hey @yongseokoh - I was on year end PTO, sorry for the delay.

Were you able to look into my comment - #43284 (comment) and check its feasibility?

batrick

The idea is good and I think I recall getting asked for this at least once. Probably by @dvanders

src/common/options/mds.yaml.in

src/mds/MDBalancer.h

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de>

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de> (cherry picked from commit 7b8def5) Conflicts: src/mds/MDSMap.cc: The change has been backported as it is but the lines surrounding the change are different in Quincy compared to main branch. src/mon/MonCommands.h: Same as above -- the change has been backported as it is but the lines surrounding the change are different in Quincy compared to main branch.

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de> (cherry picked from commit 7b8def5)

That introduces the ceph.dir.bal.mask vxattr, which is an option to rebalance a subtree within specific active MDSs. Similar to the CPU mask, this feature enables load balancing of specific directories across multiple MDS ranks. It is especially useful for fine-tuning and improving performance in various scenarios. Previously, the bal_rank_mask in ceph#43284 supports isolating unpinned subtrees under the root directory ('/') to a specific MDS rank. However, with this new option vxattr, it becomes possible to isolate specific subdirectories to designated MDS ranks. By introducing the ceph.dir.bal.mask vxattr, this PR empowers Ceph administrators with enhanced control and flexibility for optimizing performance and fine-tuning their deployments. trakcer: https://tracker.ceph.com/issues/61777 Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de> (cherry picked from commit 7b8def5) Conflicts: src/mds/MDSMap.cc: The change has been backported as it is but the lines surrounding the change are different in Quincy compared to main branch. src/mon/MonCommands.h: Same as above -- the change has been backported as it is but the lines surrounding the change are different in Quincy compared to main branch.

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de> (cherry picked from commit 7b8def5) src/mds/MDSMap.cc - The change has been backported as it is but the lines surrounding the change are different in Quincy compared to main branch. src/mon/MonCommands.h - The change has been backported as it is but the lines surrounding the change are different in Quincy compared to main branch.

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de> (cherry picked from commit 7b8def5)

That introduces the ceph.dir.bal.mask vxattr, which is an option to rebalance a subtree within specific active MDSs. Similar to the CPU mask, this feature enables load balancing of specific directories across multiple MDS ranks. It is especially useful for fine-tuning and improving performance in various scenarios. Previously, the bal_rank_mask in ceph#43284 supports isolating unpinned subtrees under the root directory ('/') to a specific MDS rank. However, with this new option vxattr, it becomes possible to isolate specific subdirectories to designated MDS ranks. By introducing the ceph.dir.bal.mask vxattr, this PR empowers Ceph administrators with enhanced control and flexibility for optimizing performance and fine-tuning their deployments. trakcer: https://tracker.ceph.com/issues/61777 Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de> (cherry picked from commit 7b8def5)

This is required for a reef client to work with a higher revision MDS, since essentially, this happens: reef(client): if (version >=17) { decode(bal_rank_mask, p); } and higher-revision MDS (say, upcoming squid): version = 17 encode(version, bl); ... ... encode(max_xattr_size, bl); encode(bal_rank_mask, bl); The client incorrectly decodes max_xattr_size (type: uint64_t) into bal_rank_mask (type: string). This situation ended up due to a couple of reasons: * the kclient patchset hanlding `max_xattr_size` was merged early on and another MDS side change that bumped the MDSMap encoding version to 17 got merged in the midst (PR ceph#43284). Details in comment: ceph#46357 (comment) * The reef backport for PR ceph#46357 got delayed (and, reef branched out). Which means reef(18.2.0) user-space clients are broken with higher version MDSs. Fixes: https://tracker.ceph.com/issues/63713 Signed-off-by: Venky Shankar <vshankar@redhat.com>

This is required for a reef client to work with a higher revision MDS, since essentially, this happens: reef(client): if (version >=17) { decode(bal_rank_mask, p); } and higher-revision MDS (say, upcoming squid): version = 17 encode(version, bl); ... ... encode(max_xattr_size, bl); encode(bal_rank_mask, bl); The client incorrectly decodes max_xattr_size (type: uint64_t) into bal_rank_mask (type: string). This situation ended up due to a couple of reasons: * the kclient patchset hanlding `max_xattr_size` was merged early on and another MDS side change that bumped the MDSMap encoding version to 17 got merged in the midst (PR ceph#43284). Details in comment: ceph#46357 (comment) * The reef backport for PR ceph#46357 got delayed (and, reef branched out). Which means reef(18.2.0) user-space clients are broken with higher version MDSs. Fixes: https://tracker.ceph.com/issues/63713 Signed-off-by: Venky Shankar <vshankar@redhat.com> (cherry picked from commit 36ee8e7)

That introduces the ceph.dir.bal.mask vxattr, which is an option to rebalance a subtree within specific active MDSs. Similar to the CPU mask, this feature enables load balancing of specific directories across multiple MDS ranks. It is especially useful for fine-tuning and improving performance in various scenarios. Previously, the bal_rank_mask in ceph#43284 supports isolating unpinned subtrees under the root directory ('/') to a specific MDS rank. However, with this new option vxattr, it becomes possible to isolate specific subdirectories to designated MDS ranks. By introducing the ceph.dir.bal.mask vxattr, this PR empowers Ceph administrators with enhanced control and flexibility for optimizing performance and fine-tuning their deployments. trakcer: https://tracker.ceph.com/issues/61777 Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>

This new configuration option will allow to define the maximum size for a filesystem xattrs blob. This is a filesystem-wide knob that will replace the per-MDS mds_max_xattr_pairs_size option. Note: The kernel client patch to handle this new configuration was merged before the corresponding ceph-side pull-request. This was unfortunate because in the meantime PR ceph#43284 was merged and the encoding/decoding of 'bal_rank_mask' got in between. Hence the 'max_xattr_size' is being encoding/decoded before 'bal_rank_mask'. URL: https://tracker.ceph.com/issues/55725 Signed-off-by: Luís Henriques <lhenriques@suse.de> (cherry picked from commit 7b8def5)

That introduces the ceph.dir.bal.mask vxattr, which is an option to rebalance a subtree within specific active MDSs. Similar to the CPU mask, this feature enables load balancing of specific directories across multiple MDS ranks. It is especially useful for fine-tuning and improving performance in various scenarios. Previously, the bal_rank_mask in ceph#43284 supports isolating unpinned subtrees under the root directory ('/') to a specific MDS rank. However, with this new option vxattr, it becomes possible to isolate specific subdirectories to designated MDS ranks. By introducing the ceph.dir.bal.mask vxattr, this PR empowers Ceph administrators with enhanced control and flexibility for optimizing performance and fine-tuning their deployments. trakcer: https://tracker.ceph.com/issues/61777 Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>

github-actions bot added cephfs Ceph File System common documentation labels Sep 23, 2021

varshar16 requested review from a team and batrick September 28, 2021 01:13

varshar16 added the needs-review label Sep 28, 2021

github-actions bot added the needs-rebase label Nov 1, 2021

yongseokoh force-pushed the feat-mds-rank-mask branch from cdd5cae to f608fad Compare November 2, 2021 10:28

github-actions bot removed the needs-rebase label Nov 2, 2021

vshankar reviewed Nov 30, 2021

View reviewed changes

src/mds/MDBalancer.cc Outdated Show resolved Hide resolved

src/mds/MDBalancer.cc Outdated Show resolved Hide resolved

src/mds/MDBalancer.cc Outdated Show resolved Hide resolved

src/mds/MDBalancer.cc Outdated Show resolved Hide resolved

src/mds/MDBalancer.cc Outdated Show resolved Hide resolved

github-actions bot added the needs-rebase label Dec 1, 2021

yongseokoh force-pushed the feat-mds-rank-mask branch from f608fad to caeb592 Compare December 3, 2021 08:44

github-actions bot removed the needs-rebase label Dec 3, 2021

yongseokoh force-pushed the feat-mds-rank-mask branch from caeb592 to a141541 Compare December 3, 2021 08:45

batrick requested changes Jun 21, 2022

View reviewed changes

src/common/options/mds.yaml.in Show resolved Hide resolved

src/mds/MDBalancer.h Outdated Show resolved Hide resolved

yongseokoh mentioned this pull request Jul 10, 2023

mds: add ceph.dir.bal.mask vxattr for MDS Balancer #52373

Open

14 tasks

vshankar mentioned this pull request Dec 1, 2023

mds: encode bal_rank_mask using a higher (next) version #54745

Merged

14 tasks

Conversation

yongseokoh commented Sep 23, 2021

Checklist

Uh oh!

yongseokoh commented Oct 8, 2021

Uh oh!

jtlayton commented Oct 8, 2021

Uh oh!

yongseokoh commented Oct 13, 2021

Uh oh!

yongseokoh commented Nov 1, 2021

Uh oh!

github-actions bot commented Nov 1, 2021

Uh oh!

vshankar commented Nov 2, 2021

Uh oh!

yongseokoh commented Nov 2, 2021

Uh oh!

yongseokoh commented Nov 2, 2021

Uh oh!

vshankar commented Nov 24, 2021

Uh oh!

yongseokoh commented Nov 26, 2021

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 1, 2021

Uh oh!

vshankar commented Dec 3, 2021

Uh oh!

yongseokoh commented Dec 4, 2021

Uh oh!

yongseokoh commented Dec 22, 2021

Uh oh!

vshankar commented Jan 6, 2022

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants