Enable linking multiple Python formulae by carlocab · Pull Request #107517 · Homebrew/homebrew-core

carlocab · 2022-08-07T06:24:16Z

python@3.9: use altinstall target
python@3.10: link into HOMEBREW_PREFIX
python@3.9: cleanup
python@3.10: cleanup
Update Python aliases
versioned_keg_only_allowlist: add Python 3.10

The primary change (see first commit) uses the altinstall target in the Python Makefile to make sure that Python3.9 only installs files that contain Python's major and minor version. That is, python@3.9 will now install only python3.9, python3.9-config, etc., instead of python3 and python3-config.

This will allow users to have both python@3.9 and python@3.10 linked into HOMEBREW_PREFIX simultaneously. Users who wish to call python3.9 as python3 can add libexec/"bin" to their PATH.

Going forward, I believe this will help simplify migrating to newer Python versions, since there will be no need to make the newer version keg-only while most dependent formulae still depend on the older version of Python.

I've made a lot of changes here. For ease of review, I placed substantive changes in the first two commits. The third and fourth commits are refactoring, and the ~~last two~~ fifth and sixth update non-formula files.

I have not tested this on Linux. This may require a few more adjustments there.

See discussion at Homebrew/discussions#2776 and documentation at https://github.com/python/cpython#installing-multiple-versions.

Closes #87075.

Have you followed the guidelines for contributing?
Have you ensured that your commits follow the commit style guide?
Have you checked that there aren't other open pull requests for the same formula update/change?
Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
Does your build pass brew audit --strict <formula> (after doing brew install --build-from-source <formula>)? If this is a new formula, does it pass brew audit --new <formula>?

Formula/python@3.10.rb

Formula/python@3.9.rb

Formula/python@3.10.rb

Formula/python@3.9.rb

Rylan12

Doesn't this mean that if you run brew install python@3.9 (no alias so no auto upgrade to python@3.10) and then have a script that relies on /opt/homebrew/bin/python3 it will no longer work unless you manually install python@3.10?

carlocab · 2022-08-07T16:00:29Z

Doesn't this mean that if you run brew install python@3.9 (no alias so no auto upgrade to python@3.10) and then have a script that relies on /opt/homebrew/bin/python3 it will no longer work unless you manually install python@3.10?

Yes. But the same thing happens whenever we make python@3.9 keg-only and link python@3.10, so the way we've been handling this so far has the same problem.

carlocab · 2022-08-08T01:42:38Z

Formula/python@3.9.rb

@Rylan12 this should help a bit with users who really just want python3 (of whatever version) but did brew install python@3.9.

It won't make it go away, since not everyone reads the caveats, but it should help a bit.

Formula/python@3.10.rb

carlocab · 2022-08-10T17:03:36Z

carlocab · 2022-08-10T17:25:20Z

swift test error is:

  'swiftpmtest': error: invalidManifestFormat("<unknown>:0: error: module \'_Builtin_stddef_max_align_t\' is defined in both \'/var/folders/_0/bwq744sj55n9b7yxj53j1rxw0000gq/C/clang/ModuleCache/OVVET4YQAF2U/_Builtin_stddef_max_align_t-2S6J4W9HHDZWV.pcm\' and \'/var/folders/_0/bwq744sj55n9b7yxj53j1rxw0000gq/C/clang/ModuleCache/OVVET4YQAF2U/_Builtin_stddef_max_align_t-E8YDO9UD74YG.pcm\'\n/private/tmp/swift-test-20220809-75938-1pkfhpt/swiftpmtest/Package.swift:4:8: error: failed to build module \'PackageDescription\' for importation due to the errors above; the textual interface may be broken by project issues or a compiler bug\nimport PackageDescription\n       ^", diagnosticFile: nil)

https://github.com/Homebrew/homebrew-core/runs/7748711279?check_suite_focus=true#step:10:6888

Seems unrelated to Python, but not certain. Any ideas, @Bo98?

Bo98 · 2022-08-10T17:50:59Z

Probably a bad file in that machine's module cache. That's annoying. It shouldn't affect ephemeral at least.

I do pass a custom -module-cache-path in the tests to avoid issues like that, but we probably need it for the SwiftPM tests too. We could abuse this: https://github.com/apple/swift-package-manager/blob/release/5.6/Sources/PackageLoading/ManifestLoader.swift#L775

Before the SwiftPM test, try ENV["SWIFTPM_MODULECACHE_OVERRIDE"] = module_cache.

carlocab · 2022-08-11T05:52:54Z

I plan to rebase, update resources, and re-run this as soon as #107645 is merged. The other fixes look like they'll take a bit longer.

This makes sure that Python does not install major-versioned files (e.g. `python3`, `python3-config`) and instead only installs files with both a major and minor version (e.g. `python3.9`, `python3.9-config`). This will allow us to have multiple versions of Python linked into `HOMEBREW_PREFIX` without conflicts. If users want to call `python3.9` as `python3`, then they can add `libexec/"bin"` to their `PATH`. Going forward, this will help simplify migrating to newer versions of Python, since we will not need to make the newer version keg-only while dependent formulae are migrated. See discussion at Homebrew/discussions#2776, and documentation at https://github.com/python/cpython#installing-multiple-versions Closes Homebrew#87075.

The previous commit minimises the conflict between `python@3.9` and `python@3.10`. The rest is handled by the `link_overwrite` lines.

Previous changes allow Python 3.9 and Python 3.10 to be linked at the same time.

Co-authored-by: XuehaiPan <XuehaiPan@pku.edu.cn>

carlocab · 2022-08-11T10:12:07Z

This is ready for review.

Some dependents will still fail, but all failures should have fixes in progress. See #107517 (comment).

I hope to be able to merge this once CI completes.

MikeMcQuaid

Makes sense to me. One small question.

MikeMcQuaid · 2022-08-11T10:23:21Z

Formula/python@3.10.rb

+  link_overwrite "Frameworks/Python.framework/Headers"
+  link_overwrite "Frameworks/Python.framework/Python"
+  link_overwrite "Frameworks/Python.framework/Resources"
+  link_overwrite "Frameworks/Python.framework/Versions/Current"


Does this mean whichever version is linked last will be Current?

I contemplated that, but found it too confusing, so decided against it.

In this PR, only the newest Python in Homebrew/core (i.e. python@3.10) will have these symlinks.

I would like to find a way to do it so that the newest Python you have installed will have these symlinks, so that the behaviour is consistent and predictable, but im punting on that for now.

I contemplated that, but found it too confusing, so decided against it.

Agreed, I think it's better this way.

In this PR, only the newest Python in Homebrew/core (i.e. python@3.10) will have these symlinks.

How do we ensure these get updated/migrated on a new Python version? Audit? Could do something like fail if the alias doesn't point to python.

How do we ensure these get updated/migrated on a new Python version? Audit? Could do something like fail if the alias doesn't point to python.

Yup, we should add an audit for this. Will look at one soon.

@carlocab Sounds great, doesn't need to block this PR 👍🏻

BrewTestBot · 2022-08-12T03:50:32Z

@carlocab has triggered a merge.

BrewTestBot added the automerge-skip `brew pr-automerge` will skip this pull request label Aug 7, 2022

carlocab mentioned this pull request Aug 7, 2022

python@3.10: migrate aliases #87075

Closed

6 tasks

carlocab requested a review from a team August 7, 2022 06:26

carlocab force-pushed the link-all-pythons branch 6 times, most recently from a56c57e to 6598029 Compare August 7, 2022 09:22

carlocab commented Aug 7, 2022

View reviewed changes

Formula/python@3.10.rb Outdated Show resolved Hide resolved

carlocab force-pushed the link-all-pythons branch 2 times, most recently from b19b708 to 5a5f61a Compare August 7, 2022 09:36

carlocab commented Aug 7, 2022

View reviewed changes

Formula/python@3.10.rb Outdated Show resolved Hide resolved

Formula/python@3.9.rb Outdated Show resolved Hide resolved

carlocab commented Aug 7, 2022

View reviewed changes

Formula/python@3.10.rb Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

carlocab commented Aug 7, 2022

View reviewed changes

Formula/python@3.9.rb Outdated Show resolved Hide resolved

carlocab force-pushed the link-all-pythons branch from 5a5f61a to e0a4730 Compare August 7, 2022 13:12

carlocab added long build Set a long timeout for formula testing CI-no-fail-fast Continue CI tests despite failing GitHub Actions matrix builds. CI-linux-self-hosted Build on Linux self-hosted runner labels Aug 7, 2022

Rylan12 reviewed Aug 7, 2022

View reviewed changes

carlocab force-pushed the link-all-pythons branch from e0a4730 to 7472b2d Compare August 8, 2022 01:41

carlocab commented Aug 8, 2022

View reviewed changes

Formula/python@3.10.rb Outdated Show resolved Hide resolved

This was referenced Aug 10, 2022

pyqt@5: use Python's libexec/"bin" #107734

Merged

six: use Python's libexec/"bin" #107735

Merged

cython: migrate to python@3.10 #107736

Closed

python-tk@3.9: call python3.9 instead of python3. #107743

Closed

carlocab mentioned this pull request Aug 10, 2022

root: migrate to python@3.10 #107745

Closed

6 tasks

carlocab mentioned this pull request Aug 10, 2022

swift: override default swiftpm modulecache #107770

Merged

carlocab and others added 10 commits August 11, 2022 18:09

python@3.10: link into HOMEBREW_PREFIX

03ff32c

The previous commit minimises the conflict between `python@3.9` and `python@3.10`. The rest is handled by the `link_overwrite` lines.

python@3.9: cleanup

8f7fd8e

python@3.10: cleanup

94b0341

Update Python aliases

bc76de1

versioned_keg_only_allowlist: add Python 3.10

43b9241

Previous changes allow Python 3.9 and Python 3.10 to be linked at the same time.

python@3.9: use python3 helper method

e8f2f78

Co-authored-by: XuehaiPan <XuehaiPan@pku.edu.cn>

python@3.10: use python3 helper method

e8aec98

Co-authored-by: XuehaiPan <XuehaiPan@pku.edu.cn>

python@3.9: update resources

04573e7

python@3.10: update resources

f16e655

MikeMcQuaid approved these changes Aug 11, 2022

View reviewed changes

This was referenced Aug 11, 2022

python@3.8: update resources, link into HOMEBREW_PREFIX #107804

Closed

python@3.7: update resources, link into HOMEBREW_PREFIX #107816

Closed

This was referenced Aug 12, 2022

gitfs: fix test failure #107867

Merged

Fix python3 references in python@3.10-dependent formulae #108008

Closed

jstvz mentioned this pull request Aug 23, 2022

Specify Python minor version in homebrew formula SFDO-Tooling/CumulusCI#3344

Merged

carlocab mentioned this pull request Sep 6, 2022

{Packaging} Update Homebrew template to use python3.10 executable Azure/azure-cli#23788

Merged

3 tasks

carlocab mentioned this pull request Oct 28, 2022

python: make python@3.11 the new default for python3 #114154

Closed

6 tasks

Uh oh!

Conversation

carlocab commented Aug 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Rylan12 left a comment

Choose a reason for hiding this comment

Uh oh!

carlocab commented Aug 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlocab Aug 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carlocab commented Aug 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlocab commented Aug 10, 2022

Uh oh!

Bo98 commented Aug 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlocab commented Aug 11, 2022

Uh oh!

carlocab commented Aug 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MikeMcQuaid left a comment

Choose a reason for hiding this comment

Uh oh!

MikeMcQuaid Aug 11, 2022

Choose a reason for hiding this comment

Uh oh!

carlocab Aug 11, 2022

Choose a reason for hiding this comment

Uh oh!

MikeMcQuaid Aug 11, 2022

Choose a reason for hiding this comment

Uh oh!

carlocab Aug 11, 2022

Choose a reason for hiding this comment

Uh oh!

MikeMcQuaid Aug 11, 2022

Choose a reason for hiding this comment

Uh oh!

BrewTestBot commented Aug 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

carlocab commented Aug 7, 2022 •

edited

Loading

carlocab commented Aug 7, 2022 •

edited

Loading

carlocab commented Aug 10, 2022 •

edited

Loading

Bo98 commented Aug 10, 2022 •

edited

Loading

carlocab commented Aug 11, 2022 •

edited

Loading