Skip to content

extend fast-deps to sdists (and allow downloading foreign sdists) (working prototype!) #8929

@cosmicexplorer

Description

@cosmicexplorer

What's the problem this feature will solve?

  1. --use-feature=fast-deps is awesome. After Download wheels in batch at the end of .prepare_linked_requirements_more() #8896, the stage is set to try downloading all the lazy deps in parallel at once! However, we still run into a mild slowdown by downloading, unpacking, then doing some further processing on every non-wheel dependency! It turns out this isn't necessary.
  2. It is not possible to invoke pip to resolve transitive dependencies for pip download or for a --dry-run resolve as per Add pip install --dry-run or similar, to get resolution result #53 if any of those dependencies are only provided as sdists.

Describe the solution you'd like
Extend fast-deps to sdists by extracting the single PKG-INFO file, without any further processing. This file existed in every sdist download I was able to quickly check on pypi (~5000 projects).

  • This should improve the parallelism available when building sdists in pip wheel, and downloading dists in pip download, once that feature is implemented.
  • As described in Add pip install --dry-run or similar, to get resolution result #53 (comment), it would be wonderful to perform a "dry run" resolve without having to incur the full download costs at all. The use case described in that comment is a PEX file which resolves its dependencies when first executed, allowing the deployed PEX to be much much smaller. Right now, it would still be impossible to deploy such an "ipex" file to a foreign platform if there were any sdist-only dependencies, even if Add pip install --dry-run or similar, to get resolution result #53 was otherwise fully implemented.

Alternative Solutions
Pex invokes pip for all download, wheel-building, and installation tasks, so its API for resolving wheels at least is predicated upon any limitations in pip. In this case, it would be lovely to improve download performance as well as enable dry-run resolves in Pex, but it's not clear to me whether there is any workaround without upstream pip changes.

Additional context
This is prototyped in the branch https://github.com/cosmicexplorer/pip/tree/fast-deps-sdists (on top of #8896) and appears to work really smoothly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions