mon,mds: map mds daemons to a particular fs by liewegas · Pull Request #32015 · ceph/ceph

liewegas · 2019-12-04T21:11:01Z

add mds_fs option to map an mds to a particular fs
report it via the beacons
update fsmap and mon logic to respect the setting
update cephadm to set this for each set of mds daemons it deploys

sebastian-philipp · 2019-12-04T21:41:57Z

On the other hand, would it work to treat the mds as something unrelated to the fs? Similar to the relationship between osds and pools?

liewegas · 2019-12-05T15:18:29Z

On the other hand, would it work to treat the mds as something unrelated to the fs? Similar to the relationship between osds and pools?

I think that would work if the MDS could multiplex multiple file systems and manage resources accordingly, but it can't do that. Today, daemons for different volumes will be given different amounts of memory and may be tuned differently, have different settings applied, etc.

liewegas · 2019-12-05T15:18:56Z

retest this please

sebastian-philipp · 2019-12-05T15:20:50Z

On the other hand, would it work to treat the mds as something unrelated to the fs? Similar to the relationship between osds and pools?

I think that would work if the MDS could multiplex multiple file systems and manage resources accordingly, but it can't do that. Today, daemons for different volumes will be given different amounts of memory and may be tuned differently, have different settings applied, etc.

Then, 👍 for pinning to a particular mds

batrick · 2019-12-10T19:24:28Z

Also, this approach maps standby to an fscid. This has the unfortunate side-effect that we lose the preference if the mds comes up before the fs is created, since the mds requests an fs by name. We could modify the standby_daemons_fscid map to use an fs name instead... WDYT?

I think the fscid mapping is right. The fix should be that the periodic beacons may update standby_daemon_fscid. Either the fs is now available (or maybe deleted and recreated) or the mds_fs config was changed at runtime.

batrick · 2019-12-10T19:26:28Z

For posterity, the CDM etherpad: https://pad.ceph.com/p/mds-affinity

batrick · 2019-12-10T19:27:34Z

src/common/options.cc

    .set_flag(Option::FLAG_NO_MON_UPDATE)
    .set_description("path to MDS data and keyring"),

+    Option("mds_fs", Option::TYPE_STR, Option::LEVEL_ADVANCED)


I think this option name is not clear enough. What about mds_affinity_fs?

mds_prefer_fs? mds_force_fs? That way it reads as verb+noun and not a string of 3 nouns.

How about mds_join_fs which is consistent with other names we use like the joinable fs flag.

ah, just saw this. works for me!

src/common/options.cc

batrick · 2019-12-10T19:29:27Z

src/messages/MMDSBeacon.h

  void set_health(const MDSHealth &h) { health = h; }

+  const string& get_fs() const { return fs; }
+  void set_fs(const string& s) { fs = s; }


void set_fs(std::string_view s)

liewegas · 2019-12-13T21:49:23Z

repushed

the mon can now update the standby fscid if it changes (and the mds is still standby)
if the mds asks for an fs that doesn't exist, we register it as NONE (meaning, don't use this mds for anything). if there is no entry in the map, the mds has no preference and can be used for anything.

...except we don't actually factor this in yet when assigning the standby. I didn't rename the option yet either, what say ye?

liewegas · 2019-12-18T18:45:26Z

@batrick this should now be complete, ready for review

batrick

One missing piece I see is the forced replacement of actives if a standby is available for that fscid. Are you planning to do that in this PR or save it for a followup?

src/mds/FSMap.cc

src/mds/FSMap.h

batrick · 2019-12-18T19:28:38Z

src/common/options.cc

    .set_flag(Option::FLAG_NO_MON_UPDATE)
    .set_description("path to MDS data and keyring"),

+    Option("mds_fs", Option::TYPE_STR, Option::LEVEL_ADVANCED)


liewegas · 2019-12-18T19:46:35Z

One missing piece I see is the forced replacement of actives if a standby is available for that fscid. Are you planning to do that in this PR or save it for a followup?

Can you point me to the path you mean?

Signed-off-by: Sage Weil <sage@redhat.com>

batrick · 2019-12-18T19:51:04Z

One missing piece I see is the forced replacement of actives if a standby is available for that fscid. Are you planning to do that in this PR or save it for a followup?

Can you point me to the path you mean?

I'm referring to this proposal in the etherpad:

If new standby becomes available with stronger affinity, replacement occurs.

This PR simply prefers standbys with mds_join_fs == fscid during replacement only. I think we should also do the replacement if a standby has stronger affinity than a current active rank (but not degraded rank).

Signed-off-by: Sage Weil <sage@redhat.com>

batrick · 2019-12-18T19:52:04Z

This PR should also add a pending release note and small blurb in doc/cephfs/standby.rst

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas · 2019-12-18T20:01:06Z

Updated. Leaving the preemption of active MDSs for later.

* refs/pull/32015/head: doc/cephfs/standby: document mds_join_fs mgr/cephadm: map mds daemons to a particular fs mon/MDSMonitor: respect mfs fscid preference mon/MDSMonitor: assign standbys to their preferred fscid mds/FSMap: track preferred fscid for standby daemons mds: add mds_join_fs option; pass via MMDSBeacon Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

liewegas requested a review from a team as a code owner December 4, 2019 21:11

liewegas requested a review from batrick December 4, 2019 21:11

liewegas added the cephfs Ceph File System label Dec 4, 2019

batrick requested changes Dec 10, 2019

View reviewed changes

liewegas force-pushed the wip-ssh-mds-fs branch from b7af858 to cf499e8 Compare December 13, 2019 21:47

liewegas force-pushed the wip-ssh-mds-fs branch from cf499e8 to 4d79dad Compare December 18, 2019 18:43

liewegas changed the title ~~RFC: map mds daemons to a particular fs~~ mon,mds: map mds daemons to a particular fs Dec 18, 2019

liewegas requested a review from batrick December 18, 2019 18:44

liewegas added the feature label Dec 18, 2019

liewegas force-pushed the wip-ssh-mds-fs branch from 4d79dad to e99a8e4 Compare December 18, 2019 18:45

batrick requested changes Dec 18, 2019

View reviewed changes

liewegas force-pushed the wip-ssh-mds-fs branch from e99a8e4 to cb5201e Compare December 18, 2019 19:45

mds: add mds_join_fs option; pass via MMDSBeacon

5e0571c

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-ssh-mds-fs branch from cb5201e to 7f6b71e Compare December 18, 2019 19:48

liewegas added 4 commits December 18, 2019 13:51

mds/FSMap: track preferred fscid for standby daemons

40f8521

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MDSMonitor: assign standbys to their preferred fscid

ad4ee63

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MDSMonitor: respect mfs fscid preference

e5013b4

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: map mds daemons to a particular fs

ffe1aca

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-ssh-mds-fs branch from 7f6b71e to ffe1aca Compare December 18, 2019 19:57

doc/cephfs/standby: document mds_join_fs

9aa25d7

Signed-off-by: Sage Weil <sage@redhat.com>

batrick approved these changes Dec 18, 2019

View reviewed changes

batrick added needs-qa wip-pdonnell-testing labels Dec 19, 2019

batrick merged commit 9aa25d7 into ceph:master Jan 6, 2020

batrick mentioned this pull request Jan 16, 2020

Add mds_join_fs config to MDS pods rook/rook#4707

Closed

Conversation

liewegas commented Dec 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sebastian-philipp commented Dec 4, 2019

Uh oh!

liewegas commented Dec 5, 2019

Uh oh!

liewegas commented Dec 5, 2019

Uh oh!

sebastian-philipp commented Dec 5, 2019

Uh oh!

batrick commented Dec 10, 2019

Uh oh!

batrick commented Dec 10, 2019

Uh oh!

batrick Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

liewegas Dec 13, 2019

Choose a reason for hiding this comment

Uh oh!

batrick Dec 16, 2019

Choose a reason for hiding this comment

Uh oh!

batrick Dec 18, 2019

Choose a reason for hiding this comment

Uh oh!

liewegas Dec 18, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

batrick Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

liewegas commented Dec 13, 2019

Uh oh!

liewegas commented Dec 18, 2019

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

batrick Dec 18, 2019

Choose a reason for hiding this comment

Uh oh!

liewegas commented Dec 18, 2019

Uh oh!

batrick commented Dec 18, 2019

Uh oh!

batrick commented Dec 18, 2019

Uh oh!

liewegas commented Dec 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

liewegas commented Dec 4, 2019 •

edited

Loading