Just adjusted this pretty heavily to add a force flag to ok-to-stop rather than just adding notice messages for monitoring daemons.

Things to note:

• changed _host_ok_to_stop to run through all daemons rather than returning on first failure. The idea here being if a user wanted to put a host in maintenance that had two daemons that would cause availability issues we want to show both of them rather than just showing one and then having a subsequent call with the force flag override both.
• Did not include full use of this flag in places other than host maintenance. orch stop and orch daemon stop should likely make use of ok-to-stop and therefore this flag. I've left them alone for now because the way daemon actions is implemented through scheduling involves storing the action in a [host][daemon] -> action (str) mapping. I think that data structure would have to be adjusted (and therefore everywhere that uses the scheduled daemon actions as well) to pass the information that the force flag was set along. I could include that in this PR but I think it could also be its own change.
• applying service specs also makes use of ok-to-stop. I currently have it default to force=True there as that maintains the old functionality. The apply command could theoretically also make use of the force flag. Again, not sure if we want something like that in this PR.
• Currently, all the monitoring stack ok-to-stop functions just use an error code of 1 if they're going to block. I'm open to suggestions for better values to use. Swapped to EBUSY

As a general rule of thumb: Please don't use Optional[bool] for parameter annotations.

The difference is:

def my_func(param: Optionl[bool]): ...

declares a variable as nullable. If you use Optionl[bool], param can be either True or False or None and you have to check that in the code. param = True or False is not identical to param = True or False or None. And on the other hand:

def my_func(param: bool = False): ...

declares param as optional when calling the function.

cc @ceph/orchestrators

https://pulpito.ceph.com/swagner-2021-01-18_11:41:36-rados:cephadm-wip-swagner-testing-2021-01-15-1448-distro-basic-smithi/

Failures

https://tracker.ceph.com/issues/48894
https://tracker.ceph.com/issues/48715
https://tracker.ceph.com/issues/48142

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

queued: https://pulpito.ceph.com/swagner-2021-01-26_11:54:26-rados:cephadm-wip-swagner-testing-2021-01-25-1228-distro-basic-smithi/

https://pulpito.ceph.com/swagner-2021-01-26_11:54:26-rados:cephadm-wip-swagner-testing-2021-01-25-1228-distro-basic-smithi/

lots of podman failures due to containers/podman#9096