tested on mira ssh working from container:

$ ssh ubuntu@mira097.front.sepia.ceph.com

add ssh key :

ssh-copy-id -f -i /etc/ceph/ceph.pub ubuntu@mira097.front.sepia.ceph.com

failure when using /etc/ceph/ceph.pub because of:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/etc/ceph/ceph.pub' are too open.

resolved when made the fule only readable:

#tip: use get-private-key method: ceph config-key get mgr/cephadm/ssh_identity_key

hmm..

key are the same pair generated one… seems normal

aah the host check seems to be the issue:

$ ceph -W cephadm
cluster:
id: 9875053a-b737-11ea-b4da-0025900809c6
health: HEALTH_WARN
1 hosts fail cephadm check
Reduced data availability: 1 pg inactive
OSD count 0 < osd_pool_default_size 3

$ ceph cephadm check-host senta02
Error ENOENT: Failed to connect to senta02 (senta02).
Check that the host is reachable and accepts connections using the cephadm SSH key

#tip: always perform check hosts before executing a command so that we know the root cause is cephadm failure.

Doc: https://docs.ceph.com/docs/master/cephadm/operations/ says no workaround remove host which will be remove cluster in this case as there is only one host 😦 back to square 1!

why did it went stale btw …

#tip: prior to add host add command to generate ssh key

finally a clean cluster has achieved the feat! 🙂

#tip: check-host was the key to diagonose, we must have some kind of help msg in that case:

Warning: one or more hosts down, if the primary mon+mgr host is down…

shutdown the cluster/ produce error in case of low mon mgr availability.