Okay, I've reviewed this, I'm open to more simplification but I don't currently see anywhere obvious.

@henryiii I think you asked me at the packaging summit if there had been any discussion on this, unfortunately the discussion on interpreting the pre-release specification comes from many scattered sources.

My ultimate motivation here is that I would like a standards compliant fast resolver writen in Python. And there are certain logical operations you need to be able to do with specifiers to write fast resolvers, such as taking the intersection of two specifiers, or having a canonical normal form of a specifier.

Because of pre-releases these operations aren't obvious even when following the spec, but because packaging doesn't consistently follow the spec it's currently impossible to implement these with packaging right now. There are a couple of discussions at:

• Add a method to simplify a SpecifierSet #776
• And at: https://discuss.python.org/t/proposal-intersect-and-disjoint-operations-for-python-version-specifiers/71888

These conversations always get stuck because there is a confusion about how SHOULD pre-releases behave, so I have carefully spent time to understand what pre-releases SHOULD do. Feel free to ask any questions about this.

But this does also fix a bunch of pip issues (pypa/pip#7579, pypa/pip#9121, pypa/pip#12469).

BTW, here is a specific invariant that this will fix that is required for certain resolution algorithms:

• Given specifier A results in no available versions, then specifier A and specifier B also results in no available versions

This does not currently hold for packaging SpecifierSets:

>>> list(SpecifierSet('<=2').filter(['1.0a1']))
[]
>>> list(SpecifierSet('<=2,<=4b1').filter(['1.0a1']))
['1.0a1']

But it SHOULD according to this line in the spec:

accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier

I'm going to ping @pradyunsg who asked by Brett to review my previous PR #794 that affected pre-releases, in case they have any input.

Also in case they have any thoughts, I'm going to ping @pfmoore, as I was very much inspired to raise this PR by our discussion in this thread: https://discuss.python.org/t/proposal-intersect-and-disjoint-operations-for-python-version-specifiers/71888

I will also note, for tools using packaging to determine if a pre-release should be selected this will make that more likely, such as in pip, and such tools may want to give control to the user to prevent pre-releases (for pip I have an open issue pypa/pip#13221). When this lands in pip I will do a careful review of how pip is using Specifier and SpecifierSet as there are likely code workarounds from legacy behavior.

I'm going to ping @pfmoore, as I was very much inspired to raise this PR by our discussion in this thread

I'll note that although I was arguing in that thread that the existing spec isn't ambiguous, and tools that don't implement the spec as written are wrong (as opposed to having an "interpretation" of ambiguous wording), I'm not actually a big fan of the current spec¹. The whole pre-release handling behaviour is far too much a case of "do what I mean" for my liking. I think we would be much better off with a simpler, clearer spec, even if that meant people had to be more explicit about whether they wanted pre-releases. Of course, that would require a new PEP, rather than just a PR to the packaging project...

All of which is to say that if this change causes incompatible behaviour changes in pip, I'd be very sympathetic with users who objected. "The spec made us do it" is a bit of a cop-out IMO. But let's wait and see what the review comes up with - I don't want to start a panic for no reason.

I think we would be much better off with a simpler, clearer spec, even if that meant people had to be more explicit about whether they wanted pre-releases. Of course, that would require a new PEP, rather than just a PR to the packaging project...

I would happily support someone else to work on that 😉. I'm of the opinion that having a spec become more mathematically simple would be a big breaking change (namely < V would have to change so it could select a pre-release version of V) for all installers / resolvers , and therefore would need buy-in from all popular package installers.

At the moment, packaging (and by extension pip) almost follows the spec, Poetry does follow the spec, uv has made design choices here that are explictly not following the spec. So I would rather just fix packaging to follow the spec, and deal with future spec changes independently.

All of which is to say that if this change causes incompatible behaviour changes in pip, I'd be very sympathetic with users who objected. "The spec made us do it" is a bit of a cop-out IMO. But let's wait and see what the review comes up with - I don't want to start a panic for no reason.

I agree with that this should not be motivated by implementing the spec for specs sake.

But this causes real problems in pip (pypa/pip#7579, pypa/pip#9121) and packaging (#854), and as mentioned it breaks important invariants for implementing more advanced resolution algorithms (#897 (comment)).

Further, packaging implements the spec correctly in some places (Specifier.filter and empty SpecifierSet.filter) but not others ( Specifier.contains, SpecifierSet.contains, and non-empty SepcifierSet.filter) e.g. #856 and #917. So depending on how the downstream tool (like pip) uses packaging, they may end up following the spec, not following the spec, or following it some times but not others.

I think these all build a strong case for changing the behavior, I also think the real world impact will be fairly small, there was very little fall out from #794.

Hi all, I'm back from a short vacation, any thoughts on my response? Anyone wanting to block this?

I thought a bit more about the important invariant for resolution and realized I'd worded it a little weirdly:

Given specifier A results in no available versions, then specifier A and specifier B also results in no available versions

I think it's better worded:

Given some version v is not satisfied by specifier A, then v should not be satisfied by specifiers A and B

And as discussed, that's not currently true when you add to a SpecifierSet:

>>> SpecifierSet('<=2').contains('1.0a1')
False
>>> SpecifierSet('<=2,<=4b1').contains('1.0a1')
True

This PR makes both true, by more correctly follows the spec.

And hopefully not to tire everyone with how this makes things more consistent, but another example of how this makes contains and filter in sync with each other, which they are not now:

>>> Specifier('<=2').contains('1.0a1')
False
>>> list(Specifier('<=2').filter(['1.0a1']))
['1.0a1']

Will become:

>>> Specifier('<=2').contains('1.0a1')
True
>>> list(Specifier('<=2').filter(['1.0a1']))
['1.0a1']

Still not seeing any objections. I'll merge tomorrow unless there are.

Thanks for all your work on this!

Thanks @henryiii , I appreciate turning up to a project and saying you've been interpreting this aspect of the spec wrong for many years is not an easy situation.

I will be around if users do complain and do my best to empathetically and patiently justify the changes.

Though with the last change I made to pre-release only one issue was posted as far as I'm aware, and it was on the pip side and it was more of an informational interaction, they accepted the change.