mgr/cephadm: add RGW support for NFS ganesha by mgfritch · Pull Request #37600 · ceph/ceph

mgfritch · 2020-10-08T14:27:46Z

create an RGW keyring for NFS daemon access
generate RGW FSAL in ganesha.conf

Fixes: https://tracker.ceph.com/issues/43686
Signed-off-by: Michael Fritch mfritch@suse.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

varshar16 · 2020-10-09T13:04:29Z

src/pybind/mgr/cephadm/templates/services/nfs/ganesha.conf.j2

+RGW {
+        cluster = "ceph";
+        name = "client.{{ rgw_user }}";
+}
+


Instead of adding RGW block by default. It would be good to have export type option in ceph orch apply nfs or let higher level interfaces modify the ganesha conf according to export type. And we have cluster type option in volumes/nfs interface ceph nfs cluster create <type> <clusterid> [<placement>] for the purpose of modifying minimal ganesha conf according to export type. @batrick @jtlayton

generally agree our goal is to minimize ganesha.conf into a very simple bootstrap config, but we do need to manage the keyrings (both add/remove) and also mount the keyring into each daemon container.

I think the problem is that once defined here you can't add other options to the RGW section from another RADOS config. cephadm can setup the keys but I think it'd be better for the higher level config to write the RGW section.

I've been able to overwrite this block from the higher level config, but the entire block is replaced, which is problematic because this would require the higher level workflow to also have knowledge of the keyring entity.

batrick · 2020-10-09T18:04:39Z

src/cephadm/cephadm

        mounts[os.path.join(data_dir, 'etc/ganesha')] = '/etc/ganesha:z'
+        if self.rgw:
+            cluster = self.rgw.get('cluster', 'ceph')
+            rgw_user = self.rgw.get('user', 'admin')


Do we do this elsewhere? I don't think we should use the admin credential as a default.

Would a NoneType be better here? At this point, we've already generated a client keyring and the input validation should have raised an exception if no keyring was provided ...

Would a NoneType be better here?

Yes, I think so.

batrick · 2020-10-09T18:10:48Z

src/pybind/mgr/cephadm/services/nfs.py

+            'prefix': 'auth get-or-create',
+            'entity': entity,
+            'caps': ['mon', 'allow r',
+                     'osd', 'allow rwx'],


Why does NFS-Ganesha need access to the OSDs when talking to RGW?

Do you have a more restrictive set of perms in mind?

I think it just needs mon "allow r"? cc @cbodley @mattbenjamin

librgw is a librados client. does that mean it needs osd caps?

I thought librgw was just talking to the monitors and not the OSDs but maybe I misunderstood.

In any case, it should not have unlimited access to the OSDs with osd "allow rwx".

librgw, when initialized, starts an entire radosgw absent only the HTTP front-ends and (by default) some of the background services such as lifecycle. By design, it accesses services in the Ceph cluster the same as an ordinary radosgw would. It needs the same permissions to execute--and even identifies itself with the same client string.

I see. Thanks for explaining @mattbenjamin. How do you normally restrict rgw instances though in terms of osd caps?

@batrick it is possible to restrict rados caps to just the specific pools that a radosgw cluster needs for index, data, log, and data-extra. The most complete example I know of is by @leseb recently in setting up rook external clusters. We could probably re-use that here.

mattbenjamin · 2020-10-11T15:19:48Z

@mgfritch (folks), where does the ganesha.conf come from? There are options here we'd like to check. Also, potentially allow selection of rgw_nfs_fast_attrs.
@kalebskeithley

batrick · 2020-10-12T13:55:26Z

@mgfritch (folks), where does the ganesha.conf come from? There are options here we'd like to check. Also, potentially allow selection of rgw_nfs_fast_attrs.

https://github.com/ceph/ceph/pull/37600/files#diff-d19145aa3232f726604da0f1f3c6bba7

We've taken to calling this the "bootstrap" config because most configuration should come from a higher layer like the volumes nfs driver.

mgfritch · 2020-10-12T15:42:04Z

a "bootstrap" ganesha.conf is rendered from within mgr/cephadm and injected into each daemon container. The bootstrap config defines a %url directive for additional cluster-wide or user-defined configuration blocks via a RADOS object.

Could we have an RGW block defined in both the bootstrap ganesha.conf and also another RGW block later defined via a RADOS conifg object. ... but I think that would require the ganesha daemon to take a union on the config values instead of the current behavior (which is replace the entire RGW block) ?

batrick · 2020-10-12T16:19:52Z

Could we have an RGW block defined in both the bootstrap ganesha.conf and also another RGW block later defined via a RADOS conifg object. ... but I think that would require the ganesha daemon to take a union on the config values instead of the current behavior (which is replace the entire RGW block) ?

As I said in the other comment, it does not take a union so if cephadm defines it then it cannot be overridden TMK.

mgfritch · 2020-10-12T18:02:56Z

Could we have an RGW block defined in both the bootstrap ganesha.conf and also another RGW block later defined via a RADOS conifg object. ... but I think that would require the ganesha daemon to take a union on the config values instead of the current behavior (which is replace the entire RGW block) ?

As I said in the other comment, it does not take a union so if cephadm defines it then it cannot be overridden TMK.

I think this was true in older versions of ganesha, but with ganesha v3.3, it does seem the that the %url directive will now replace the entire block (rather than simply ignore), but a union of config values still seems like a good way to allow multiple consumers to define/overwrite various items in the conf blocks ... for example a user-config where we only want to change a single conf value and not rewrite the entire config block.

Fixes: https://tracker.ceph.com/issues/43686 Signed-off-by: Michael Fritch <mfritch@suse.com>

- create an RGW keyring for NFS daemon access - generate RGW FSAL in ganesha.conf Fixes: https://tracker.ceph.com/issues/43686 Signed-off-by: Michael Fritch <mfritch@suse.com>

remove RGW keyring during NFS daemon `post_remove` Fixes: https://tracker.ceph.com/issues/43686 Signed-off-by: Michael Fritch <mfritch@suse.com>

Daniel-Pivonka

This functionally worked for me i was able to put files in buckets and they would show up on the nfs mount

mattbenjamin · 2020-10-23T15:36:58Z

@Daniel-Pivonka woot

mattbenjamin · 2020-10-23T15:38:48Z

@Daniel-Pivonka @kalebskeithley @dang should we have a review of the configs that can now be generated? I'd like to ensure that our preferred configurations (incl. fast-attrs, above) can be set up.

bk201

work for using Dashboard to create RGW share.
I can access files in buckets with RO mode.

jmolmo

I think we can go with this.
Probably we will need to improve things around how to setup the service in a more simple way. But this modifications are definitely a first and a big step. :-)

src/pybind/mgr/cephadm/services/nfs.py

restrict the OSD keyring caps to the `rgw` application Signed-off-by: Michael Fritch <mfritch@suse.com>

mgfritch · 2020-11-07T00:56:32Z

green: https://pulpito.ceph.com/mgfritch-2020-11-06_14:54:25-rados:cephadm-wip-mgfritch-testing-2020-11-05-1314-distro-basic-smithi/

PR ceph#37600 introduced support for both cephfs and rgw while using the same nfs-ganesha daemon Signed-off-by: Michael Fritch <mfritch@suse.com>

PR ceph#37600 introduced support for both `cephfs` and `rgw` exports to be configured using a single nfs-ganesha cluster Signed-off-by: Michael Fritch <mfritch@suse.com>

PR ceph#37600 introduced support for both `cephfs` and `rgw` exports to be configured using a single nfs-ganesha cluster Fixes: https://tracker.ceph.com/issues/50369 Signed-off-by: Michael Fritch <mfritch@suse.com>

PR ceph#37600 introduced support for both `cephfs` and `rgw` exports to be configured using a single nfs-ganesha cluster Fixes: https://tracker.ceph.com/issues/50369 Signed-off-by: Michael Fritch <mfritch@suse.com> (cherry picked from commit e9b2c38)

PR ceph#37600 introduced support for both `cephfs` and `rgw` exports to be configured using a single nfs-ganesha cluster Fixes: https://tracker.ceph.com/issues/50369 Signed-off-by: Michael Fritch <mfritch@suse.com> (cherry picked from commit e9b2c38) (cherry picked from commit 678ee63) Resolves: rhbz#1951969

At some point, upstream obviously changed the syntax of the "ceph nfs cluster create" command in octopus, thereby breaking "sesdev create pacific" and "sesdev create ses7". Later, this syntax change got reverted [1], but since this revert was made possible by a big feature [2], so far it has been backported only to pacific [3] [1] ceph/ceph#40411 [2] ceph/ceph#37600 [3] ceph/ceph#41005 Fixes: SUSE#611 Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgfritch added cephadm nfs labels Oct 8, 2020

mgfritch requested review from Daniel-Pivonka, bk201 and varshar16 October 8, 2020 14:27

mgfritch requested a review from a team as a code owner October 8, 2020 14:27

mgfritch mentioned this pull request Oct 8, 2020

Maintenance Update: Document rgw-nfs SUSE/doc-ses#507

Closed

varshar16 reviewed Oct 9, 2020

View reviewed changes

batrick added the rgw label Oct 9, 2020

batrick requested changes Oct 9, 2020

View reviewed changes

bk201 added the needs-rebase label Oct 20, 2020

mgfritch added 3 commits October 20, 2020 11:23

cephadm: mount RGW keyring in NFS container

53c6388

Fixes: https://tracker.ceph.com/issues/43686 Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add RGW support for NFS ganesha

5a2ffc3

- create an RGW keyring for NFS daemon access - generate RGW FSAL in ganesha.conf Fixes: https://tracker.ceph.com/issues/43686 Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: clean-up NFS rgw keyring

219c0c3

remove RGW keyring during NFS daemon `post_remove` Fixes: https://tracker.ceph.com/issues/43686 Signed-off-by: Michael Fritch <mfritch@suse.com>

mgfritch force-pushed the cephadm-nfs-rgw branch from 4934879 to 219c0c3 Compare October 20, 2020 17:32

mgfritch removed the needs-rebase label Oct 20, 2020

Daniel-Pivonka approved these changes Oct 23, 2020

View reviewed changes

bk201 approved these changes Oct 28, 2020

View reviewed changes

jmolmo approved these changes Oct 28, 2020

View reviewed changes

batrick requested changes Oct 28, 2020

View reviewed changes

src/pybind/mgr/cephadm/services/nfs.py Outdated Show resolved Hide resolved

mgr/cephadm: add NFS RGW keyring application tag

d400677

restrict the OSD keyring caps to the `rgw` application Signed-off-by: Michael Fritch <mfritch@suse.com>

mgfritch requested a review from batrick October 28, 2020 20:33

batrick approved these changes Oct 28, 2020

View reviewed changes

mgfritch added needs-qa wip-mgfritch-testing labels Oct 28, 2020

mgfritch merged commit 33a907a into ceph:master Nov 7, 2020

mgfritch deleted the cephadm-nfs-rgw branch November 7, 2020 01:10

sebastian-philipp mentioned this pull request Nov 18, 2020

octopus: cephadm batch backport November #38155

Merged

mgfritch mentioned this pull request Mar 25, 2021

mgr/volumes/nfs: drop type param during cluster create #40411

Merged

3 tasks

smithfarm mentioned this pull request Nov 5, 2021

templates/cephadm/deployment_day_2: fix NFS cluster create command SUSE/sesdev#612

Merged

Conversation

mgfritch commented Oct 8, 2020

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattbenjamin commented Oct 11, 2020

Uh oh!

batrick commented Oct 12, 2020

Uh oh!

mgfritch commented Oct 12, 2020

Uh oh!

batrick commented Oct 12, 2020

Uh oh!

mgfritch commented Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Daniel-Pivonka left a comment

Choose a reason for hiding this comment

Uh oh!

mattbenjamin commented Oct 23, 2020

Uh oh!

mattbenjamin commented Oct 23, 2020

Uh oh!

bk201 left a comment

Choose a reason for hiding this comment

Uh oh!

jmolmo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mgfritch commented Nov 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mgfritch commented Oct 12, 2020 •

edited

Loading