I am truly sorry for not clarifying this earlier in a discussion point and ended up causing a decent amount of confusion 😅
Originally posted by @McSinyx in #8638 (comment)
Regarding my GSoC project on download parallelization, now we have:
In the current code base, when fast-deps is used, RequirementPreparer.prepare_linked_requirement may skip the actual preparation (including some checks, downloading the whole wheel and moving it around) and save it for right before the new resolver returning the requirement set. Essentially the download preparation is what we want to execute in parallel, however to obtain progress data for display, it is not a good idea to map_multithreading the method directly.
With that in mind, we may want to invoke the download before the final preparation calls (the following snippet).
|
for actual_req in req_set.all_requirements: |
|
self.factory.preparer.prepare_linked_requirement_more(actual_req) |
|
|
|
return req_set |
@pradyunsg and I came to a conclusion that it might be a good idea to have the path to download files stored in the InstallRequirement to be able to cleanly skip the call to unpack_url.
This comes out as the output of my discussion with @pradyunsg after a few unsuccessful attempts of mine to solve the issue. He suggested that a issue should be filed to gather opinions from other maintainers (as well interested contributors) to construct the most complete view on the matter. Please feel free to share what's in your mind!
Originally posted by @McSinyx in #8638 (comment)
Regarding my GSoC project on download parallelization, now we have:
fast-deps(thank you @chrahunt for the clean up at Move fast-dep handling to RequirementPreparer #8685)In the current code base, when
fast-depsis used, RequirementPreparer.prepare_linked_requirement may skip the actual preparation (including some checks, downloading the whole wheel and moving it around) and save it for right before the new resolver returning the requirement set. Essentially the download preparation is what we want to execute in parallel, however to obtain progress data for display, it is not a good idea tomap_multithreadingthe method directly.With that in mind, we may want to invoke the download before the final preparation calls (the following snippet).
pip/src/pip/_internal/resolution/resolvelib/resolver.py
Lines 162 to 165 in ee4371c
@pradyunsg and I came to a conclusion that it might be a good idea to have the path to download files stored in the InstallRequirement to be able to cleanly skip the call to unpack_url.
This comes out as the output of my discussion with @pradyunsg after a few unsuccessful attempts of mine to solve the issue. He suggested that a issue should be filed to gather opinions from other maintainers (as well interested contributors) to construct the most complete view on the matter. Please feel free to share what's in your mind!