Skip to content

Speed up CircleCI jobs by replacing pip with uv pip#5255

Closed
cclauss wants to merge 6 commits intopyodide:mainfrom
cclauss:uv
Closed

Speed up CircleCI jobs by replacing pip with uv pip#5255
cclauss wants to merge 6 commits intopyodide:mainfrom
cclauss:uv

Conversation

@cclauss
Copy link
Contributor

@cclauss cclauss commented Dec 14, 2024

DRAFT: Highly experimental!!

Description

Fixes #5228

Creating Pyodide pull requests often requires several commits to get things working correctly and all tests passing. Unfortunately, there are numerous, long test runs on GitHub Actions and CircleCI so waiting for feedback can slow innovation. Pip installing numerous, large dependencies seems to be a substantial portion of the time required to run all these CI jobs.

Could some CI jobs be substantially accelerated using uv pip instead of pip?

https://docs.astral.sh/uv -- `10-100x faster than pip` sounds like a useful improvement to me.
https://github.com/astral-sh/uv/blob/main/BENCHMARKS.md

If there is interest in trying this and a maintainer can recommend one CI job to start with, I would like to create a proof-of-concept pull request to benchmark the performance improvement of using uv for that job.

Related to:

#5018 (comment)
#5004 (comment)

Let's start just with the instances of pip install in one file: .circleci/config.yml


When using uv, the virtual environment does not need to be activated. uv will find a virtual environment (named .venv) in the working directory or any parent directories. -- https://docs.astral.sh/uv/reference/cli/#uv-venv

@dhruvmanila Any advice on how to avoid repeated source .venv/bin/activate calls inside complex CircleCI jobs? https://docs.astral.sh/uv does not yet mention CircleCI.

Checklists

  • Add a CHANGELOG entry
  • Add / update tests
  • Add new / update outdated documentation

@cclauss
Copy link
Contributor Author

cclauss commented Dec 14, 2024

This would be more reliable and easier to understand if uv was installed in Dockerfile as recommended at
https://docs.astral.sh/uv/guides/integration/docker/#installing-uv -->

@dhruvmanila
Copy link

@dhruvmanila Any advice on how to avoid repeated source .venv/bin/activate calls inside complex CircleCI jobs? docs.astral.sh/uv does not yet mention CircleCI.

You don't need to activate the virtual environments at least for uv as running uv pip install will find the virtual environment and install the package in it. Refer to https://docs.astral.sh/uv/pip/environments/#using-a-virtual-environment.

@agriyakhetarpal
Copy link
Member

I've personally found some troubles with using the uv action on GHA without having a virtual environment set: astral-sh/setup-uv#124. A workaround is to set the PATH to the venv folder so that later jobs can make use of it.

@cclauss
Copy link
Contributor Author

cclauss commented Dec 16, 2024

Thanks for your assistance @dhruvmanila. I believe each CircleCI job step is a new invocation of bash so source .venv/bin/activate seems to be required in each job step that will touch the Python dependencies. Perhaps #5255 will help by installing uv and its venv from the start in Dockerfile.

@cclauss cclauss changed the title Speed up CI jobs by replacing pip with uv pip Speed up CircleCI jobs by replacing pip with uv pip Dec 16, 2024
@agriyakhetarpal agriyakhetarpal marked this pull request as ready for review December 18, 2024 01:58
@agriyakhetarpal agriyakhetarpal marked this pull request as draft December 18, 2024 01:59
@agriyakhetarpal
Copy link
Member

Apologies, I was looking at it through the GitHub Mobile app and inadvertently pressed that button.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants