Skip to content

python: create venv's --without-pip#15792

Merged
MikeMcQuaid merged 1 commit intoHomebrew:masterfrom
branchv:venv-no-pip
Jul 31, 2023
Merged

python: create venv's --without-pip#15792
MikeMcQuaid merged 1 commit intoHomebrew:masterfrom
branchv:venv-no-pip

Conversation

@branchv
Copy link
Copy Markdown
Member

@branchv branchv commented Jul 30, 2023 •
edited
Loading

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes? Here's an example.
  • Have you successfully run brew style with your changes locally?
  • Have you successfully run brew typecheck with your changes locally?
  • Have you successfully run brew tests with your changes locally?

Currently, virtualenv_install_with_resources creates a venv with a full copy of pip. This is pretty wasteful, since we really only need it at build time to install resources. Instead, we can:

  1. Create a venv using --without-pip
  2. Use python@x.y's pip to install resources into this venv, using the --python option
  3. Tangentially, remove all shell activate scripts (which admittedly doesn't save much but aren't necessary)

As a result, this saves ~7MB of disk space per venv:

$ brew install black
...
==> Summary
🍺  /opt/homebrew/Cellar/black/23.7.0_1: 1,121 files, 13.5MB
$ brew reinstall -s black
...
==> Summary
🍺  /opt/homebrew/Cellar/black/23.7.0_1: 558 files, 6.6MB, built in 41 seconds

Note I don't foresee this causing any issues, but I've left an opt-out to be safe.

MikeMcQuaid
MikeMcQuaid reviewed Jul 31, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me! Could you add a commented-out odeprecated for the opt-out so we can remove it if it's unused? Thanks!

@branchv
Copy link
Copy Markdown
Member Author

branchv commented Jul 31, 2023

Done!

@MikeMcQuaid
Copy link
Copy Markdown
Member

MikeMcQuaid commented Jul 31, 2023

Thanks again @branchvincent!

@MikeMcQuaid MikeMcQuaid merged commit d1f60ae into Homebrew:master Jul 31, 2023
@branchv branchv deleted the venv-no-pip branch July 31, 2023 18:21
@Bo98 Bo98 mentioned this pull request Aug 5, 2023
Closed
6 tasks
@cangussu
Copy link
Copy Markdown

cangussu commented Aug 11, 2023

I have a bunch of machines running different brew versions and a formula that uses virtualenv_create and relies on the fact that pip is available. What is the proper way to deal with this change? It broke my formula with new version of brew but if I add the extra parameter it will break on older versions.

@branchv
Copy link
Copy Markdown
Member Author

branchv commented Aug 11, 2023

Sorry about that!

running different brew versions

may be a silly question, but is updating to the latest version not an option?

relies on the fact that pip is available

in what way? there may be a simple change you could make, like Homebrew/homebrew-core#138661 (comment)

@MikeMcQuaid
Copy link
Copy Markdown
Member

MikeMcQuaid commented Aug 14, 2023

I have a bunch of machines running different brew versions and a formula that uses virtualenv_create and relies on the fact that pip is available.

@cangussu Which versions? Note we do not support running any Homebrew versions beyond the latest.

@bebound bebound mentioned this pull request Aug 17, 2023
Closed
@bebound
Copy link
Copy Markdown

bebound commented Aug 17, 2023

This also removes setuptools. Some namespace packages may rely on it:

linuxbrew@794b47da48b2:/down$ az -v
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/linuxbrew/.linuxbrew/Cellar/azure-cli/2.51.0/libexec/lib/python3.10/site-packages/azure/__init__.py", line 1, in <module>
    __import__('pkg_resources').declare_namespace(__name__)
ModuleNotFoundError: No module named 'pkg_resources'

@branchv
Copy link
Copy Markdown
Member Author

branchv commented Aug 17, 2023

Since setuptools is included with our python, you'll only run into trouble if you've disabled system_site_packages

Also note if you use setuptools at runtime, you really should add it to your requirements as Python 3.12 will no longer include it either python/cpython#95299 pypa/pip#12032

This was referenced Aug 17, 2023
Merged
Merged
@cangussu
Copy link
Copy Markdown

cangussu commented Sep 6, 2023

@branchvincent I followed your suggestion to use python -m pip and it worked for our use case, tks!

in what way? there may be a simple change you could make, like Homebrew/homebrew-core#138661 (comment)

@github-actions github-actions bot added the outdated PR was locked due to age label Oct 7, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@MikeMcQuaid MikeMcQuaid MikeMcQuaid approved these changes

Assignees

No one assigned

Labels

outdated PR was locked due to age

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@branchv @MikeMcQuaid @cangussu @bebound