DEP: in support of greater forbearance in bumping optional dependencies, beyond SPEC 0 #18967
Description
I'm seeing a tendency to drop support for any dependency as soon as SPEC 0 allows for it, wether mandatory or optional.
I would like to make a plea for hardening our policy beyond what adhering to SPEC 0 allows us to do, specifically about optional dependencies, also known as extras.
Recent examples of bumping optional dependencies early include
Aggressive upgrades of this sort may create problems for end users; because the dependencies are optional, we cannot provide any guarantee they are installed at the same time as astropy, using the proper extra specification (typically as, e.g., pip install 'astropy[recommended]'). Crucially, and in contrast with direct dependencies such as numpy, we cannot actually restrict what versions of, e.g., scipy or matplotlib are installed with astropy, save for the most disciplined part of our user base who
- know about extras
- how to use them
- ... and when to use them.
I would argue that this is probably a very small fraction of the population, but the point is, many, if not most of our users should not be expected to be knowledgeable in this area. Considering most scientific Python users still run long-lived Python environments (and who can blame them ? alternative workflows are only starting to emerge now), it is especially likely that someone might, for instance, install matplotlib 3.6, then astropy 7.2.0 (which, at the time of opening, is at a rc stage, and -optionally- requires matplotlib 3.8), and end up with a broken environment, even though we had everything needed to support this valid use case, but decided to remove the backward compatibility layer
To be clear, I am not pointing fingers at the people who on-boarded the aforementionned bumps. I mean to point to a systematic failure to recognize optional dependencies as a different kind of contract with our users: we should only be allowed to make a dependency optional, or to aggressively bumping our requirements on it, but not both.
The solution to this, is to
- apply forbearance when bumping our optional dependencies (a backward compatibility layer is temporary code any way, and not necessarily a burden to carry around for a bit longer)
- promote or demote the ones we need to bump to either hard requirements, or dev-only dependencies
Incidentally, I believes this clarifies an existing, but so far unclear incentive for #17623
An immediate action could be to revert #18164 and #18492 for the time being. Tentatively triaging this issue to the 7.2.0 milestone with this point in mind, but I expect the ticket not to be closed without a broader discussion and possibly with some policy change.