Hmm, if the DEPRECATED macros and MASK stuff doesn't make it "do the right thing" for random clients, I think feature bits which are client-visible are just...not reusable or able to be retired.

@athanatos @jdurgin do either of you remember much of the planning around feature bit retirement? I don't think I got involved in its development, but I think I would have noticed if we went from "all clients meeting the admin-configurable min_compat_client" to "3 versions, and incredibly messy for the kernel". I'm sure @idryomov has thoughts about that.

Hrm, I don't remember, I'd need to take a bit of time and think it through.

Client-facing feature bits can't really be retired in an orderly fashion, so any N-<something> rule is bound to not work, exactly because

we have little control over kernel versions and user-space libraries within third party's containers (old librbd, ceph-fuse and so on).

We have done this only once before, slurping up all argonaut-ish feature bits. I suppose we could do this again for pre-hammer feature bits some time soon but this would have to be an ad-hoc, review each feature bit sort of thing. If the intent is to document the procedure, I would rather document that there is no procedure and go with something like

For feature bits that were anyhow exposed to clients (even through a common part like OSDMap): don't touch the ones that are already there, think hard before introducing a new one, assume a lifetime of at least 10 years.

Thanks for the input, Ilya!

My intention is just to document the state of the thing – discover & document. I'm aware that another views (i.e. n - 3 as as hard guarantee) also exist. I think it would be good to describe also how mon_osd_initial_require_min_compat_client interplays here.

CC: @gregsfortytwo, @athanatos, @jdurgin.

require_min_compat_client is only tangentially related: it's a setting that is intended to prevent an administrator from accidentally enabling features that older clients would choke on (i.e. features that would prevent older clients from connection to the cluster). The fact that it's set to luminous by default doesn't mean much: in the end it's based on the same feature bits, many of which can be manipulated in isolation.

To be clear: it's possible to have e.g. hammer-era clients connect to a reef cluster after making some adjustments (mostly regarding CRUSH tunables) no matter what require_min_compat_client is set to.