Skip to content

Enable 'ceph.conf' management by cephadm#291

Merged
ricardoasmarques merged 1 commit intoceph:masterfrom
ricardoasmarques:enable-manage-ceph-conf
Aug 10, 2020
Merged

Enable 'ceph.conf' management by cephadm#291
ricardoasmarques merged 1 commit intoceph:masterfrom
ricardoasmarques:enable-manage-ceph-conf

Conversation

@ricardoasmarques
Copy link
Contributor

@ricardoasmarques ricardoasmarques commented Jul 8, 2020 •
edited
Loading

After ceph/ceph#35576 is backported (ceph/ceph#35898)
After ceph/ceph#36286 is backported(ceph/ceph#36450)

Since ceph/ceph#35576 we can now rely on cephadm to keep /etc/ceph/ceph.conf file updated on all hosts, for instance, when MONs change.

Fixes: #199

Signed-off-by: Ricardo Marques rimarques@suse.com

@ricardoasmarques
Copy link
Contributor Author

ricardoasmarques commented Jul 8, 2020

For some reason, I'm not getting the expected behavior...

I see that mgr/cephadm/manage_etc_ceph_ceph_conf is enabled:

master:~ # ceph config-key get mgr/cephadm/manage_etc_ceph_ceph_conf
obtained 'mgr/cephadm/manage_etc_ceph_ceph_conf'
true

And my config is referencing node1 host (10.20.66.201):

master:~ # salt '*' cmd.run 'grep mon_host /etc/ceph/ceph.conf'
node3.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]
master.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]
node1.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]
node2.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]

But when I change MON to node2:

master:~ # ceph orch apply mon node2
Scheduled mon update...

My /etc/ceph/ceph.conf is not updated acordingly:

master:~ # sleep 20

master:~ # salt '*' cmd.run 'grep mon_host /etc/ceph/ceph.conf'
node3.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]
master.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]
node2.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]
node1.pacific.test:
    	mon_host = [v2:10.20.66.201:3300/0,v1:10.20.66.201:6789/0]

@sebastian-philipp @jschmid1 any idea if I'm doing something wrong here, or if it's a bug on ceph/ceph#35576?

@ricardoasmarques ricardoasmarques mentioned this pull request Jul 8, 2020
Closed
@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Jul 22, 2020

I just verified this on a cluster:

master:~ # cat /etc/ceph/ceph.conf 
# minimal ceph.conf for 5acf8c52-cbf3-11ea-914d-525400481a1a
[global]
        fsid = 5acf8c52-cbf3-11ea-914d-525400481a1a
        mon_host = [v2:192.168.0.150:3300/0,v1:192.168.0.150:6789/0]
master:~ # ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true
master:~ # ceph -W cephadm
  cluster:
    id:     5acf8c52-cbf3-11ea-914d-525400481a1a
    health: HEALTH_WARN
            failed to probe daemons or devices
 
  services:
    mon: 3 daemons, quorum mon.monitor3,mon.monitor1,mon.monitor2 (age 4m)
    mgr: mgr.monitor1.veewkq(active, since 5m), standbys: monitor2.uuseco, mgr.monitor3.ktlure, mgr.monitor2.uqympm
    osd: 10 osds: 10 up (since 12m), 10 in (since 20m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   10 GiB used, 90 GiB / 100 GiB avail
    pgs:     1 active+clean
 

^Cmaster:~ #at /etc/ceph/ceph.conf ^C
master:~ # cat /etc/ceph/ceph.conf 
# minimal ceph.conf for 5acf8c52-cbf3-11ea-914d-525400481a1a
[global]
        fsid = 5acf8c52-cbf3-11ea-914d-525400481a1a
        mon_host = [v2:192.168.0.150:3300/0,v1:192.168.0.150:6789/0] [v2:192.168.0.151:3300/0,v1:192.168.0.151:6789/0] [v2:192.168.0.152:3300/0,v1:192.168.0.152:6789/0]

@ricardoasmarques
Copy link
Contributor Author

ricardoasmarques commented Jul 22, 2020

master:~ # ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true

Ok, I'm not setting the config correctly.
I'm doing: ceph config-key set mgr/cephadm/manage_etc_ceph_ceph_conf true
instead of: ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true

I'll fix it and test again.

@ricardoasmarques ricardoasmarques force-pushed the enable-manage-ceph-conf branch 4 times, most recently from 231f580 to 26e3ba7 Compare July 24, 2020 13:58
@ricardoasmarques
Copy link
Contributor Author

ricardoasmarques commented Jul 24, 2020

I'm trying to set this configs during bootstrap, but apparently cephadm is ignoring it:

node1:~ # cat /tmp/bootstrap-ceph.conf 
[mgr]
        mgr/cephadm/manage_etc_ceph_ceph_conf = true

        
node1:~ # CEPHADM_IMAGE=registry.opensuse.org/filesystems/ceph/master/upstream/images/ceph/ceph \
cephadm --verbose bootstrap \
        --mon-ip 10.20.108.201 \
        --config /tmp/bootstrap-ceph.conf
...
INFO:cephadm:Bootstrap complete.


node1:~ # ceph config get mgr mgr/cephadm/manage_etc_ceph_ceph_conf
false

@ricardoasmarques
Copy link
Contributor Author

ricardoasmarques commented Jul 24, 2020

As an alternative, I can execute ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true.
But if I do it immediately after the bootstrap I get the following MGR error:

Jul 24 16:47:17 node1 bash[31656]: debug 2020-07-24T14:47:17.500+0000 7fc9f438b700 -1 mgr load Traceback (most recent call last):
Jul 24 16:47:17 node1 bash[31656]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 271, in __init__
Jul 24 16:47:17 node1 bash[31656]:     self.config_notify()
Jul 24 16:47:17 node1 bash[31656]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 525, in config_notify
Jul 24 16:47:17 node1 bash[31656]:     self.config_notify_one(what)
Jul 24 16:47:17 node1 bash[31656]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 531, in config_notify_one
Jul 24 16:47:17 node1 bash[31656]:     self.cache.distribute_new_etc_ceph_ceph_conf()
Jul 24 16:47:17 node1 bash[31656]: AttributeError: 'CephadmOrchestrator' object has no attribute 'cache'

@ricardoasmarques ricardoasmarques force-pushed the enable-manage-ceph-conf branch from 26e3ba7 to 1a6bc2b Compare July 24, 2020 15:51
@ricardoasmarques ricardoasmarques changed the title [DMN] Enable 'ceph.conf' management by cephadm [After ceph/ceph#36286] Enable 'ceph.conf' management by cephadm Jul 24, 2020
@ricardoasmarques
Copy link
Contributor Author

ricardoasmarques commented Jul 24, 2020

As an alternative, I can execute ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true.
But if I do it immediately after the bootstrap I get the following MGR error:

Jul 24 16:47:17 node1 bash[31656]: debug 2020-07-24T14:47:17.500+0000 7fc9f438b700 -1 mgr load Traceback (most recent call last):
Jul 24 16:47:17 node1 bash[31656]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 271, in __init__
Jul 24 16:47:17 node1 bash[31656]:     self.config_notify()
Jul 24 16:47:17 node1 bash[31656]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 525, in config_notify
Jul 24 16:47:17 node1 bash[31656]:     self.config_notify_one(what)
Jul 24 16:47:17 node1 bash[31656]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 531, in config_notify_one
Jul 24 16:47:17 node1 bash[31656]:     self.cache.distribute_new_etc_ceph_ceph_conf()
Jul 24 16:47:17 node1 bash[31656]: AttributeError: 'CephadmOrchestrator' object has no attribute 'cache'

Will be fixed by ceph/ceph#36286

@ricardoasmarques ricardoasmarques force-pushed the enable-manage-ceph-conf branch 2 times, most recently from 346d10e to 42ef753 Compare July 31, 2020 08:32
@ricardoasmarques ricardoasmarques changed the title [After ceph/ceph#36286] Enable 'ceph.conf' management by cephadm [After ceph/ceph#36450] Enable 'ceph.conf' management by cephadm Aug 6, 2020
@ricardoasmarques ricardoasmarques force-pushed the enable-manage-ceph-conf branch 2 times, most recently from 2759336 to 8748c50 Compare August 10, 2020 11:13
@ricardoasmarques ricardoasmarques changed the title [After ceph/ceph#36450] Enable 'ceph.conf' management by cephadm Enable 'ceph.conf' management by cephadm Aug 10, 2020
@ricardoasmarques ricardoasmarques added Add To Changelog Marks the PR to be included in the changelog of the next version release and removed blocked-by-cephadm labels Aug 10, 2020
@ricardoasmarques
Enable 'ceph.conf' management by cephadm
645e7ce 
Fixes: ceph#199

Signed-off-by: Ricardo Marques <rimarques@suse.com>
@ricardoasmarques ricardoasmarques force-pushed the enable-manage-ceph-conf branch from 8748c50 to 645e7ce Compare August 10, 2020 15:34
@ricardoasmarques ricardoasmarques merged commit e9717ca into ceph:master Aug 10, 2020
@ricardoasmarques ricardoasmarques removed the Add To Changelog Marks the PR to be included in the changelog of the next version release label Aug 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

potentially invalid ceph.conf after moving monitors

2 participants

@ricardoasmarques @sebastian-philipp