Install build dependencies in-process by ichard26 · Pull Request #13450 · pypa/pip

ichard26 · 2025-06-27T19:19:27Z

Towards #7294, #9081, and its associated issues.

This PR adds an alternative build environment dependency installer that installs in-process, using a stripped down reimplementation of the install command. It's disabled by default. To use the in-process installer, pass --use-feature inprocess-build-deps.

Building pip from source with setuptools cached, inprocess-build-deps shaves ~300 ms (1850ms -> 1530ms).

pip wheel . --no-index --find-links tests/data/common_wheels --use-feature inprocess-build-deps

I'll note that the plan would be to:

Merge the new installer as an opt-in feature
Flip the default some time later (and keep the subprocess installer available as a deprecated feature)
Remove the legacy subprocess installer once the legacy resolver is removed (we can also simplify the package finder, build tracker, and other parts of the codebase that exist only to support the legacy installer)

Known behavioural differences

More options are inherited generally in accordance to this proposal: Rethink pip's PEP-517 mechanism #9081 (comment)
- Key differences are that --implementation, --abi, --platform (--python-version as well?) are inherited through reusing the finder. It may be not be a good idea to inherit these though...?
- In addition, --require-hashes and --constraints are not inherited as planned as they need a proper transition plan. They will be handled as follow-up items.
- Since there is no longer a subprocess, environment variables cannot be used to directly configure the build environment. Environment variables will only affect the build environment if the associated options are inherited in code. PIP_CONSTRAINTS. This feature will automatically enable --use-feature build-constraint as well.
The finder/in-memory caches are shared across all build environments, thus if a new version of a build dependency is released in the middle of a long install, it won't be picked up.

Sample error messages

Important

I am unhappy with the output and error handling. It is messy and could do with improvement. However, it is in a workable and releasable state. Unless you have suggestions for consolidating and rationalizing things here, I will leave this as a follow up item.

Diagnostic error (normal + verbose)

Note: In verbose mode, coloured logs are printed while installing build dependencies. Otherwise, they are printed without colour.

Other pip error (normal VS verbose)

Errors that do not inherit from PipError

ichard26 · 2025-06-27T19:39:58Z

I should probably be explicit with what I want from people right now. I simply want feedback on the general idea. Is this something we want to pursue given the inherent additional complexity? If so, I'll add tests, write proper commit messages, and break this up into smaller PRs so this is easier to review.

For the rational, please see #9081 (TL;DR, it will be faster, fix a bunch of bugs with passing the right arguments down, allow for better error handling, and clear the way to remove some ugly hacks in the codebase)

If not, I'll abandon this.

ichard26 · 2025-06-27T19:58:47Z

To help put the extra code into context, I've prototyped what the removal of the legacy installer would look like. In the 10 minutes I spent on this, this is the result:

$ git sdiff
 src/pip/__pip-runner__.py                          | 50 -------------
 src/pip/_internal/build_env.py                     | 82 +---------------------
 src/pip/_internal/commands/install.py              |  4 +-
 src/pip/_internal/commands/wheel.py                |  4 +-
 src/pip/_internal/index/package_finder.py          | 31 --------
 .../_internal/operations/build/build_tracker.py    | 75 ++------------------
 6 files changed, 10 insertions(+), 236 deletions(-)

pfmoore · 2025-06-27T20:04:33Z

+1 from me on the general idea.

notatallshaw · 2025-06-27T20:47:42Z

FYI, I was planning to get back to working on #13300 soonish, but looking over this PR I see no issue with working on both at the same time, there will only be a small overlap, just at the point of passing the flags to the build dependency installation command.

notatallshaw · 2025-06-27T21:55:46Z

Talking with @ichard26, out of band, my above comment is completely wrong, and I think we need to make a choice on this.

The current behavior: --constraint is not passed to the build environment, to make build constraints you must use the PIP_CONSTRAINT env flag or config options, this gives the ability to set constraints in the regular dependency resolution and separate constraints in the build environments dependency resolution.

This PRs ~~current behavior~~ (edit: what I thought this PR was doing): The build environment inherits the existing environments behavior, so it is not possible to set separate constraints between the regular dependency resolution and in the build environments dependency resolution.

Now, while I understand that the current behavior was probably not intentional I do think it's advantageous to be able to have build constraints and regular constraints as separate. And I am concerned with this as a breaking change.

But if this is going to be a breaking change I have a proposal to clean everything up:

Match uv's behavior, add a build constraint flag, and have regular constraints NEVER apply to the build environments, and have build constraints apply to ALL build environments:

Requirement Level	`PIP_CONSTRAINT` / `--constraint`	`PIP_BUILD_CONSTRAINT` / `--build-constraint`
Install	Applied	Not Applied
Build	Not Applied	Applied
Build of Build (etc.)	Not Applied	Applied

From a user's perspective this will be much more clean and obvious as to what each flag is doing, the new --build-constraint flag will only work with the new inprocess-build-deps so it never has to go through a behavior change.

I am willing to figure out how to wrangle the constraints vs build constraints and have it as a separate PR to land after the inprocess-build-deps lands, if @ichard26 is not up for that work.

Whether this proposal is agreed to or not, I think it's important that a constraint design is agreed upon before merging this.

ichard26 · 2025-06-27T22:02:10Z

This PRs current behavior: The build environment inherits the existing environments behavior, so it is not possible to set separate constraints between the regular dependency resolution and in the build environments dependency resolution.

This is also not exactly correct. The current behaviour is that neither --constraints nor PIP_CONSTRAINTS are respected for build dependencies because I simply didn't include constraints in the stripped pip install implementation for InprocessBuildEnvironmentInstaller.

I agree that this would be a good opportunity to clean up the constraint handling though.

ichard26 · 2025-06-28T03:09:16Z

Alrighty. I've broken this up into a handful of smaller PRs. Reviews on the following would be appreciated :)

notatallshaw · 2025-07-03T15:28:41Z

Do those 3 PRs cover everything in this PR or are intended to be the building steps before actually adding the feature that users can enable?

ichard26 · 2025-07-05T02:46:15Z

Building blocks. The actual feature will be added in a separate PR (or more likely, I'll force push to this PR) once all of the prerequisite building blocks have been landed.

notatallshaw · 2025-10-14T03:16:47Z

Moving this to 26.0 unless there are objections.

ichard26 · 2025-10-14T17:43:01Z

No objections. There is no way I'm going to have time to work on this until the holidays.

ichard26 · 2025-12-16T03:06:08Z

Update: this PR is now ready for review. I've updated the PR description to reflect current reality and the updated error handling. As a reminder, this feature is gated behind --use-feature inprocess-build-deps. It's expected that it will take a few pip releases until this stabilizes and is flipped on by default. Anyhow, this depends on --use-feature build-constraint becoming the default first.

notatallshaw

I've run this against a lot of real world scenarios and done some contrived tests and not found an issue with it.

With a merge (or rebase) and conflicts fixed I am happy to see this land.

ichard26 · 2026-01-09T22:10:51Z

Thank you @notatallshaw for the review! I am very happy to see this land. 🎉

This patch adds an alternative build environment dependency installer that installs in-process, using a stripped down re-implementation of the install command. It's disabled by default. To use the in-process installer, pass --use-feature inprocess-build-deps. This new mechanism is faster, supports --no-clean and --no-cache-dir reliably, and supports prompting for authentication. Building pip from source with setuptools cached, inprocess-build-deps shaves ~300 ms (1850ms -> 1530ms, before we switched to flit-core). The plan is to flip the default in a future release and eventually remove the legacy subprocess-based installer once the new installer has stabilized. In general, the in-process installer inherits more options. Notably, since the finder is shared, --implementation, --abi, --platform (and possibly --python-version) are inherited. It may be not be a good idea to inherit these, but we'll see if it becomes a problem. Since there is no longer a subprocess, environment variables cannot be used to directly configure the build environment. Environment variables will only affect the build environment if the associated options are inherited in code. Thus, build-constraint is mandated when this feature is enabled. Please note that the in-memory (finder) caches are shared across all build environments, thus if a new version of a build dependency is released in _the middle of_ a long install, it will no longer be picked up. The error handling story leaves much to be desired, but it can be improved in a follow up.

This was referenced Jun 27, 2025

Add rich based spinner #13451

Merged

refactor: Add BuildEnvironmentInstaller protocol #13452

Merged

Convert installable wheel build error to diagnostic #13453

Merged

ichard26 added this to the 25.3 milestone Jul 26, 2025

ichard26 added 4 commits July 29, 2025 13:56

Add in-process build dependency installer

cc074cd

Rework arguments

20f7b67

Rework arguments, pt. 2

8a0cb71

wip

10ebd8f

notatallshaw mentioned this pull request Aug 9, 2025

Add build constraints #13534

Merged

ichard26 mentioned this pull request Aug 14, 2025

Release 25.3 (October) #13522

Closed

notatallshaw modified the milestones: 25.3, 26.0 Oct 14, 2025

ichard26 mentioned this pull request Oct 14, 2025

PEP 517 only #13602

Merged

ichard26 mentioned this pull request Nov 4, 2025

Release 26.0 (January) #13641

Closed

Fix installer with latest changes on main

bf31979

ichard26 mentioned this pull request Dec 15, 2025

refactor: Store resume retries value on PipSession #13705

Merged

ichard26 added 3 commits December 15, 2025 20:38

Rework error handling one more time

ff0a8a8

Add smoke test for nested builds

5de8374

Support --build-constraint

740061d

ichard26 force-pushed the feat/inprocess-build-deps branch from b55b0fd to 621d1d6 Compare December 16, 2025 02:06

ichard26 added 3 commits December 15, 2025 21:47

Add news entry

8b73ac8

Warn about PIP_CONSTRAINT if present

b8ce779

Fix resume retries after pulling in main

794bc78

ichard26 force-pushed the feat/inprocess-build-deps branch from 621d1d6 to 794bc78 Compare December 16, 2025 02:52

psf-chronographer Bot added the bot:chronographer:provided label Dec 16, 2025

ichard26 marked this pull request as ready for review December 16, 2025 03:06

notatallshaw self-requested a review December 16, 2025 03:56

ichard26 mentioned this pull request Dec 31, 2025

Speedup startup time #4768

Open

notatallshaw approved these changes Jan 9, 2026

View reviewed changes

ichard26 added 2 commits January 9, 2026 16:05

Merge branch 'main' into feat/inprocess-build-deps

445f6b7

Use local trove-classifiers and calvar sdists

7b365ea

ichard26 enabled auto-merge (squash) January 9, 2026 22:07

ichard26 merged commit 7abcae8 into pypa:main Jan 9, 2026
28 checks passed

github-actions Bot locked as resolved and limited conversation to collaborators Jan 25, 2026

Conversation

ichard26 commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ichard26 commented Jun 27, 2025

Uh oh!

ichard26 commented Jun 27, 2025

Uh oh!

pfmoore commented Jun 27, 2025

Uh oh!

notatallshaw commented Jun 27, 2025

Uh oh!

notatallshaw commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ichard26 commented Jun 27, 2025

Uh oh!

ichard26 commented Jun 28, 2025

Uh oh!

notatallshaw commented Jul 3, 2025

Uh oh!

ichard26 commented Jul 5, 2025

Uh oh!

notatallshaw commented Oct 14, 2025

Uh oh!

ichard26 commented Oct 14, 2025

Uh oh!

ichard26 commented Dec 16, 2025

Uh oh!

notatallshaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ichard26 commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ichard26 commented Jun 27, 2025 •

edited

Loading

notatallshaw commented Jun 27, 2025 •

edited

Loading