Ceph » crimson

See: https://tracker.ceph.com/issues/73970:

With https://github.com/ceph/ceph/pull/65782 merged. `ceph-osd` is no longer overriden when Crimson is enabled:

While https://github.com/ceph/ceph/pull/66458 might be a good intermidate step to allow us to test containerized builds -
Adding a Crimson OSD should be more "friendly". We would need for cephadm to support adding crimson osds either on a per
OSD basis or cluster wide.

Copying Sam's review on 66458 - which further summarizes this tracker:

This PR is reasonable as is and is a good way to make progress testing crimson with cephadm.

I think the next step (future PRs) would be:

Add a osd_type: crimson|classic (classic default) flag to the osd spec definition for users to use instead of explicitly specifying extra_container_args. Internally, it's perfectly reasonable for cephadm to just add that flag, I just don't want users to need to interact with it. Users should simply need to specify that an osd should be crimson without needing to know what binaries to replace.
Documentation -- crimson documentation needs updated instructions.