Skip to content

Don't convert versions to strings when using importlib.metadata backend#13660

Merged
notatallshaw merged 3 commits intopypa:mainfrom
notatallshaw:assume-packaging-Versions-on-python-3.11+
Nov 22, 2025
Merged

Don't convert versions to strings when using importlib.metadata backend#13660
notatallshaw merged 3 commits intopypa:mainfrom
notatallshaw:assume-packaging-Versions-on-python-3.11+

Conversation

@notatallshaw
Copy link
Copy Markdown
Member

@notatallshaw notatallshaw commented Nov 21, 2025 •
edited
Loading

If I am understanding the following notice correctly as of Python 3.11 pip will no longer use pkg_resources._vendor.packaging.version.Version in any situation (it doesn't seem to by default anyway), and can only use packaging.version.Version:

pip/src/pip/_internal/metadata/__init__.py

Lines 63 to 67 in e18a82f

def _emit_pkg_resources_deprecation_if_needed() -> None:
if sys.version_info < (3, 11):
# All pip versions supporting Python<=3.11 will support pkg_resources,
# and pkg_resources is the default for these, so let's not bother users.
return

Skipping stringification and Version creation saves a non-trivial amount of time when heavily backtracking. In the pathological resolver benchmark I'm using to find hotspots and bottlenecks this change reduces the number of Version objects created from ~3.37 million to ~2.30 million.

Before and after call graphs:

Details

Before:

image

After:

image

@notatallshaw notatallshaw added the skip news Does not need a NEWS file entry (eg: trivial changes) label Nov 21, 2025
@notatallshaw notatallshaw force-pushed the assume-packaging-Versions-on-python-3.11+ branch from 60b96af to 8a45605 Compare November 21, 2025 03:29
@ichard26
Copy link
Copy Markdown
Member

ichard26 commented Nov 21, 2025

While the importlib-metadata backend is indeed the default on Python 3.11+, the legacy pkg_resources backend can still be force enabled until Python 3.14 with either an envvar or a module constant:

pip/src/pip/_internal/metadata/__init__.py

Lines 30 to 46 in e18a82f

def _should_use_importlib_metadata() -> bool:
"""Whether to use the ``importlib.metadata`` or ``pkg_resources`` backend.
By default, pip uses ``importlib.metadata`` on Python 3.11+, and
``pkg_resources`` otherwise. Up to Python 3.13, This can be
overridden by a couple of ways:
* If environment variable ``_PIP_USE_IMPORTLIB_METADATA`` is set, it
dictates whether ``importlib.metadata`` is used, for Python <3.14.
* On Python 3.11, 3.12 and 3.13, Python distributors can patch
``importlib.metadata`` to add a global constant
``_PIP_USE_IMPORTLIB_METADATA = False``. This makes pip use
``pkg_resources`` (unless the user set the aforementioned environment
variable to *True*).
On Python 3.14+, the ``pkg_resources`` backend cannot be used.
"""

I suspect that the number of users actually using pkg_resources on Python 3.11 or higher is extremely low1 though.

Footnotes

  1. The main reason I saw for the envvar was because the importlib backend was slower when first introduced, but those performance issues have been since fixed. ↩

@notatallshaw
Copy link
Copy Markdown
Member Author

notatallshaw commented Nov 21, 2025

While the importlib-metadata backend is indeed the default on Python 3.11+, the legacy pkg_resources backend can still be force enabled until Python 3.14

😢

Do you know if there is a way we can detect which backend is being used?

@sbidoul
Copy link
Copy Markdown
Member

sbidoul commented Nov 21, 2025

Note we have a deprecation in place to remove the pkg_resources backend in 26.3 (when we drop python 3.10 actually).

@notatallshaw
Copy link
Copy Markdown
Member Author

notatallshaw commented Nov 21, 2025

Do you know if there is a way we can detect which backend is being used?

Okay, this was actually simple, you can directly look at select_backend().NAME, and the function call is already cached so there's almost no cost. Updated PR.

@notatallshaw notatallshaw force-pushed the assume-packaging-Versions-on-python-3.11+ branch from 3fe9c2f to bb0d5f5 Compare November 21, 2025 23:16
@notatallshaw notatallshaw changed the title Assume packaging Versions on Python 3.11+ Don't convert versions to strings when using importlib.metadata backend Nov 22, 2025
@notatallshaw notatallshaw merged commit f215c5e into pypa:main Nov 22, 2025
28 checks passed
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 7, 2025
@ichard26 ichard26 added the type: performance Commands take too long to run label Feb 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@pfmoore pfmoore pfmoore approved these changes

@ichard26 ichard26 ichard26 approved these changes

Assignees

No one assigned

Labels

skip news Does not need a NEWS file entry (eg: trivial changes) type: performance Commands take too long to run

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@notatallshaw @ichard26 @sbidoul @pfmoore