Skip to content

fix: make Specifier / SpecifierSet pickle-safe#1168

Merged
henryiii merged 8 commits intopypa:mainfrom
henryiii:henryiii/fix/specset
Apr 23, 2026
Merged

fix: make Specifier / SpecifierSet pickle-safe#1168
henryiii merged 8 commits intopypa:mainfrom
henryiii:henryiii/fix/specset

Conversation

@henryiii
Copy link
Copy Markdown
Contributor

This is a followup to #1163. I've had an model apply the same changes/logic in that PR to Specifier/SpecifierSet. I think this is likely in better shape, due to not referncing a missing module, but should be better long term.

I also had it generate similar tests for several packaging versions.

🤖 Assisted by both OpenCode w/ Kimi-K2.5 & Copilot CLI w/ (new for CLI) auto model selection (claude-sonnet-4.6).

CC @eachimei.

@henryiii henryiii force-pushed the henryiii/fix/specset branch from 8e44fef to 0a5742b Compare April 20, 2026 04:08
@eachimei
Copy link
Copy Markdown
Contributor

Looks good! 👍
One observation: the indexed-tuple approach optimizes for compactness and performance, at the cost of relying on shape-sniffing (isinstance, len, nested type checks) to distinguish formats in __setstate__. That's perfectly reasonable for this and Version use cases, just noting that there's no explicit format tag or version marker, so future format changes will need another heuristic branch rather than a clean dispatch. Not a current risk, and if it does become a problem in the future, a format tag can then be introduced at that point...

@henryiii
Copy link
Copy Markdown
Contributor Author

Yes, I usually start the tuple with an "format version" integer. But I don't think this will be likely to change much, and simply adding a value is safe. Longer term, we could probably use pattern matching here if it's not too much slower.

@eachimei
Copy link
Copy Markdown
Contributor

Indeed, sounds good

@henryiii henryiii force-pushed the henryiii/fix/specset branch from 3954b39 to 3c2603d Compare April 20, 2026 12:55
@henryiii henryiii marked this pull request as ready for review April 20, 2026 12:59
@henryiii henryiii force-pushed the henryiii/fix/specset branch 3 times, most recently from 3799a8a to 4cca7ec Compare April 21, 2026 14:22
@henryiii henryiii requested a review from Copilot April 21, 2026 14:22
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the pickle-safety/backward-compatibility work from #1163 to Specifier and SpecifierSet, aiming to ensure pickles remain loadable across packaging versions and don’t persist cache/internal state.

Changes:

  • Add __getstate__/__setstate__ to Specifier and SpecifierSet, using compact tuple state and discarding caches on restore while supporting older pickle formats.
  • Add pickle roundtrip tests for Specifier/SpecifierSet.
  • Add fixture-based tests to load historical pickle payloads (packaging 25.0, 26.0, 26.1, 26.2-style).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
src/packaging/specifiers.py Implements compact pickle state + legacy restore logic for Specifier and SpecifierSet, clearing caches on restore.
tests/test_specifiers.py Adds roundtrip and historical-pickle loading tests for specifiers/specifiersets across multiple packaging versions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/packaging/specifiers.py Outdated
Comment thread src/packaging/specifiers.py Outdated
Comment thread tests/test_specifiers.py
@henryiii
Copy link
Copy Markdown
Contributor Author

henryiii commented Apr 21, 2026

The copilot review boils down to one thing: protect against malformed pickles. I've added it (generated tests, but threw away the model-implemented solution and added my own), but I'm not sure it's really needed to always check this (auto-generated pickle code doesn't have such checks), but I think it's probably a good idea for the "old" pickles. I've added it for everything, but I'd also be fine to drop it for new-style pickles (outer tuple). It's not too expensive, though, and it does make typing happy.

Pattern matching would be so nice here. :)

@henryiii henryiii force-pushed the henryiii/fix/specset branch from 47b00d5 to cab5c2c Compare April 21, 2026 15:45
@henryiii
Copy link
Copy Markdown
Contributor Author

henryiii commented Apr 21, 2026

Also, I've followed #1163 in generating old pickles and storing them, but we could also just store the intermediate state, which wouldn't be a somewhat opaque binary string, and would unit test the same. I think the current one is a bit better, but if there's a desire to avoid binary strings, we could make something work.

Comment thread src/packaging/specifiers.py
henryiii and others added 7 commits April 22, 2026 22:35
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: Copilot:claude-sonnet-4.6
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: OpenCode:Kimi-K2.5
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: Copilot:claude-sonnet-4.6
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
@henryiii henryiii force-pushed the henryiii/fix/specset branch from aa7f4c2 to 75f29b3 Compare April 23, 2026 02:35
@henryiii henryiii force-pushed the henryiii/fix/specset branch from 75f29b3 to 04ad48f Compare April 23, 2026 03:00
@henryiii henryiii merged commit 4bed32d into pypa:main Apr 23, 2026
57 checks passed
@henryiii henryiii deleted the henryiii/fix/specset branch April 23, 2026 03:34
@eachimei
Copy link
Copy Markdown
Contributor

Cheers, thank you! 🍻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants