[WIP] py-build: bootstrap without pip#35326
Conversation
|
Hi @adamjstewart! I noticed that the following package(s) don't yet have maintainers:
Are you interested in adopting any of these package(s)? If so, simply add the following to the package class: maintainers = ["adamjstewart"]If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with $ spack blame py-pyproject-hooksThank you for your help! Please don't add maintainers without their consent. You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer. |
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: isort, black, flake8, mypy
==> Modified files
var/spack/repos/builtin/packages/py-attrs/package.py
var/spack/repos/builtin/packages/py-build/package.py
var/spack/repos/builtin/packages/py-colorama/package.py
var/spack/repos/builtin/packages/py-flit-core/package.py
var/spack/repos/builtin/packages/py-importlib-metadata/package.py
var/spack/repos/builtin/packages/py-installer/package.py
var/spack/repos/builtin/packages/py-packaging/package.py
var/spack/repos/builtin/packages/py-pyparsing/package.py
var/spack/repos/builtin/packages/py-pyproject-hooks/package.py
var/spack/repos/builtin/packages/py-pytoml/package.py
var/spack/repos/builtin/packages/py-six/package.py
var/spack/repos/builtin/packages/py-toml/package.py
var/spack/repos/builtin/packages/py-tomli/package.py
var/spack/repos/builtin/packages/py-typing-extensions/package.py
var/spack/repos/builtin/packages/py-zipp/package.py
==> Running isort checks
isort checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/py-colorama/package.py
reformatted var/spack/repos/builtin/packages/py-attrs/package.py
reformatted var/spack/repos/builtin/packages/py-flit-core/package.py
reformatted var/spack/repos/builtin/packages/py-importlib-metadata/package.py
reformatted var/spack/repos/builtin/packages/py-installer/package.py
reformatted var/spack/repos/builtin/packages/py-pyparsing/package.py
reformatted var/spack/repos/builtin/packages/py-packaging/package.py
reformatted var/spack/repos/builtin/packages/py-pyproject-hooks/package.py
reformatted var/spack/repos/builtin/packages/py-pytoml/package.py
reformatted var/spack/repos/builtin/packages/py-six/package.py
reformatted var/spack/repos/builtin/packages/py-toml/package.py
reformatted var/spack/repos/builtin/packages/py-tomli/package.py
reformatted var/spack/repos/builtin/packages/py-zipp/package.py
reformatted var/spack/repos/builtin/packages/py-typing-extensions/package.py
All done! ✨ 🍰 ✨
14 files reformatted, 1 file left unchanged.
black checks were clean
==> Running flake8 checks
flake8 checks were clean
==> Running mypy checks
Success: no issues found in 576 source files
mypy checks were clean
==> spack style checks were clean
I've updated the branch with style fixes. |
|
At the moment, this feels prohibitively intrusive just to install build without pip. We may want to consider waiting until build vendors its dependencies (seems unlikely given pypa/build#470) or abandon build support until a tool with fewer dependencies inevitably comes along (given that we have 10+ build backends, why not 10+ build frontends?). |
|
It's frustrating that this needs 3 different toml parsers, but not much we can do about that. Is it preferable to install from wheel's (opaque binary) or from git (transparent source code, patches work)? I don't understand all of Python packaging, but I think pure Python packages need only be copied into the installation prefix. |
|
For downloading, we prefer URLs over git because they can be checksummed. For sdist (source) vs. bdist (wheel), we currently prefer sdist since that's what we're used to from other languages and it makes patching and development easy, but there's no reason we can't use wheels. We've talked about moving away from URLs and towards wheels for the majority of pure-Python packages but that's a lot of work and may not be trivial. We've also talked about removing pure-Python packages from Spack entirely and instead using pip to handle those, but there's a lot of work that would need to be done on PyPI's side of things to make this possible. |
|
Other alternative is to require pip to install build/installer, and then use build/installer for everything else. This seems silly to me, but is at least less disruptive. |
After reading pypa/build#470, I think the "require |
|
Let's close this. I don't love this PR in its current state. I think we'll likely go with one of the following options instead:
I may reevaluate this after Python 3.7 support is dropped. |
Spack currently uses pip to install all Python libraries. However, pip is an entire package manager, and provides many features we don't need. We would like to move towards using build/installer instead of pip to simplify the installation process. This PR is the first step in that direction. We need to be able to bootstrap build/installer without pip in order to do that.
Unfortunately, build has the worst cyclic dependency problem I've ever seen. In order to build build, build needs to be installed (this isn't that bad), but all of builds dependencies also need to be installed. All of the following packages need to be installed from wheels in order to get a working build:
It's also worth noting that installer only supports Python 3.7+, which means none of these packages can be built for EOL Python versions. This is relevant because Spack currently supports bootstrapping its style dependencies with Python 3.6. With this PR, that will no longer be supported. Curious how @tgamblin and @alalazo feel about this.
If we really do need to be able to install these with Python 3.6, we would need to add a new
WheelPackagebase class that supports installing with pip (Python 2+) or installer (Python 3.7+). This would also require 2 new builders. I was hoping to try to tackle this in a follow-up PR, but I can turn this into one massive PR if needed.References: