Skip to content

mgr/cephadm: fix and improve osd draining#39536

Merged
sebastian-philipp merged 6 commits intoceph:masterfrom
liewegas:cephadm-drain-weight
Feb 24, 2021
Merged

mgr/cephadm: fix and improve osd draining#39536
sebastian-philipp merged 6 commits intoceph:masterfrom
liewegas:cephadm-drain-weight

Conversation

@liewegas
Copy link
Member

@liewegas liewegas commented Feb 17, 2021 •
edited
Loading

  • when purging and osd, the osd was removed from teh osdmap before teh daemon was removed, resulting in the daemon not getting cleaned up.
  • when purging, we should set the crush weight to 0 instead of marking the osd out (fixes https://tracker.ceph.com/issues/49339)
  • clean up the log output (chattiness and formatting)

@liewegas
mgr/cephadm: fix 'orch daemon add osd ...'
e864327 
When adding an osd daemon explicitly, there is no created timestamp
for the spec, and we should never not apply it.

Fixes: b129c13
Signed-off-by: Sage Weil <sage@newdream.net>
@liewegas
mgr/cephadm: less log noise from osd drain code
e2f0e56 
Signed-off-by: Sage Weil <sage@newdream.net>
@liewegas
mgr/cephadm: make drain adjust crush weight if not replacing
4fc1309 
If we are replacing an OSD, we should mark it out and then back in
again when a new device shows up.  However, if we are going to
destroy an OSD, we should just weight it to 0 in crush, so that data
doesn't move again once the OSD is purged.

Signed-off-by: Sage Weil <sage@newdream.net>
@liewegas
mgr/cephadm: simplify OSD __str__ for drain
ca4050b 
Signed-off-by: Sage Weil <sage@newdream.net>
@liewegas
mgr/cephadm: remove daemon before osd destroy/purge
b5eab0d 
Otherwise it doesn't work!

Drop the fullname property: it is always "osd.{self.osd_id}".

Signed-off-by: Sage Weil <sage@newdream.net>
@liewegas
mgr/cephadm: fix up the strings reporting osd ids
a1ff3a9 
Signed-off-by: Sage Weil <sage@newdream.net>
@liewegas liewegas requested a review from a team as a code owner February 17, 2021 21:40
@sebastian-philipp sebastian-philipp added the wip-swagner-testing My Teuthology tests label Feb 18, 2021
@liewegas liewegas mentioned this pull request Feb 21, 2021
Closed
3 tasks
@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Feb 24, 2021

https://pulpito.ceph.com/swagner-2021-02-22_13:51:42-rados:cephadm-wip-swagner-testing-2021-02-22-1115-distro-basic-smithi/

@sebastian-philipp sebastian-philipp merged commit 6913065 into ceph:master Feb 24, 2021
@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Feb 24, 2021

oops. this broke the upgrade test:

 [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished

https://pulpito.ceph.com/swagner-2021-02-22_13:54:50-rados:cephadm-wip-swagner4-testing-2021-02-22-1119-distro-basic-smithi/

@liewegas
Copy link
Member Author

liewegas commented Feb 24, 2021 •
edited
Loading

1.15.120:0/2476104201' entity='mgr.x' cmd='[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]': finished
audit [INF] from='mgr.14646 172.21.15.120:0/2476104201' entity='mgr.x' cmd=[{"prefix":"config-key set","key":"mgr/cephadm/osd_remove_queue","val":"[]"}]: dispatch
audit [INF] from='mgr.14646 172.2

I don't think this is related to my pull request. The process_removal_queue() method is called at the top of the serve loop and it unconditionally saves the list. (also, that run was a pass)

@liewegas liewegas deleted the cephadm-drain-weight branch February 24, 2021 18:43
@sebastian-philipp sebastian-philipp mentioned this pull request Mar 3, 2021
Merged
@strenuous-life
Copy link

strenuous-life commented Jul 5, 2021 •
edited
Loading

@sebastian-philipp @liewegas Whether this pr will backport to octopus? In the issue #49339, the backport is described as 'pacific,octopus'.

@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Jul 6, 2021

do you want to do the backport? That's mainly a git cherry-pick -x for all the commits here

This was referenced Jun 13, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@sebastian-philipp sebastian-philipp sebastian-philipp approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cephadm pybind wip-sage-testing wip-swagner-testing My Teuthology tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liewegas @sebastian-philipp @strenuous-life