Skip to content

Fix clingo source bootstrapping on M1 macs#30894

Closed
tgamblin wants to merge 1 commit intodevelopfrom
m1-python-bootstrap
Closed

Fix clingo source bootstrapping on M1 macs#30894
tgamblin wants to merge 1 commit intodevelopfrom
m1-python-bootstrap

Conversation

@tgamblin
Copy link
Copy Markdown
Member

@tgamblin tgamblin commented May 27, 2022

Follow-on to #30834.

This tweaks the CMake arguments for clingo-bootstrap so that CMake finds the host python's sys.executable and sys.prefix. We know when bootstrapping that we are building against the host python.

One question I am not completely sure about is why the logic in the python package doesn't get this right already. We should probaby make that work rather than doing this, but I'm putting this up as a demonstrator.

@adamjstewart thoughts on that last point?

This tweaks the CMake arguments for `clingo-bootstrap` so that CMake finds the host
python's `sys.executable` and `sys.prefix`. We know when bootstrapping that we are
building against the host python.

One question I am not completely sure about is why the logic in the `python` package
doesn't get this right already. We should probaby make that work rather than doing this,
but I'm putting this up as a demonstrator.
@tgamblin tgamblin force-pushed the m1-python-bootstrap branch from d5c8821 to fe34bb1 Compare May 27, 2022 11:40
@tgamblin
Copy link
Copy Markdown
Member Author

@certik said #30834 fixed things for him, but I believe he’s using a conda python install. This is specifically trying to fix things for a system python.

One caveat is that I have the XCode CLI tools installed, which I where I believe the python framework comes from (it has headers etc.) We may need to bootstrap that too (I noticed that homebrew bootstraps it).

@adamjstewart want to try this?

@certik
Copy link
Copy Markdown
Contributor

certik commented May 27, 2022

I just clarified in #28190 (comment) that Spack now works for me without Conda as well now, using system Python. (Things have always worked with Conda, that was never an issue.)

Update: I just noticed I have this in my .zshrc:

export SPACK_PYTHON=$HOME/mambaforge/envs/spack/bin/python

So I guess I am still using Conda's Python. Sorry about that. (I did remove ~/.spack though.)

@adamjstewart adamjstewart self-assigned this May 27, 2022
@adamjstewart
Copy link
Copy Markdown
Member

I can confirm that this PR works. I'm using the full XCode suite + CLI tools and export SPACK_PYTHON=/usr/bin/python3. Now the next question is why it works...

Looking at the flags we pass to cmake, there are two differences:

  1. Python_EXECUTABLE is different
  2. we switched from Python_INCLUDE_DIR to Python_ROOT_DIR

I tried only switching 2 but that didn't help, so I believe that 1 is the change that matters. Using the system Python:

$ /usr/bin/python3
Python 3.8.9 (default, Apr 13 2022, 08:48:06) 
[Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import sys
>>> sys.executable
'/Applications/Xcode.app/Contents/Developer/usr/bin/python3'
>>> os.path.realpath('/Applications/Xcode.app/Contents/Developer/usr/bin/python3')
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8'

The former is what is used in this PR and works. The latter is what is used on develop and doesn't work.

So the question I have is, if I use SPACK_PYTHON=/usr/bin/python3, how and where does Spack determine the self.spec.prefix for Python during bootstrapping? I don't see Python listed in ~/.spack/bootstrap/config/packages.yaml.

@certik
Copy link
Copy Markdown
Contributor

certik commented May 27, 2022

I want to test this too --- is it safe to move ~/.spack to ~/.spack_backup, test this with a fresh clone of spack (new directory), and then move ~/.spack_backup back to ~/.spack? I don't want to mess up my current installation which took couple hours to build everything.

@adamjstewart
Copy link
Copy Markdown
Member

Yes, that should work

@certik
Copy link
Copy Markdown
Contributor

certik commented May 27, 2022

Ok, I can confirm that this PR works with just command line tools, system Python and no Xcode. spack spec zlib successfully bootstraps.

@certik
Copy link
Copy Markdown
Contributor

certik commented May 27, 2022

I can also confirm that the develop branch, commit 1485931 also bootstraps. So either I am doing something wrong, or this PR doesn't "fix" anything for me since things are already working in develop. Either way, no complains from me! Thank you all for fixing this.

@adamjstewart
Copy link
Copy Markdown
Member

@tgamblin is this PR still necessary?

@tgamblin
Copy link
Copy Markdown
Member Author

superseded by #30834

@tgamblin tgamblin closed this Nov 22, 2022
@adamjstewart adamjstewart deleted the m1-python-bootstrap branch November 22, 2022 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants