Skip to content

[WIP] py-build: bootstrap without pip#35326

Closed
adamjstewart wants to merge 10 commits into
spack:developfrom
adamjstewart:packages/build
Closed

[WIP] py-build: bootstrap without pip#35326
adamjstewart wants to merge 10 commits into
spack:developfrom
adamjstewart:packages/build

Conversation

@adamjstewart

@adamjstewart adamjstewart commented Feb 3, 2023

Copy link
Copy Markdown
Member

Spack currently uses pip to install all Python libraries. However, pip is an entire package manager, and provides many features we don't need. We would like to move towards using build/installer instead of pip to simplify the installation process. This PR is the first step in that direction. We need to be able to bootstrap build/installer without pip in order to do that.

Unfortunately, build has the worst cyclic dependency problem I've ever seen. In order to build build, build needs to be installed (this isn't that bad), but all of builds dependencies also need to be installed. All of the following packages need to be installed from wheels in order to get a working build:

  • installer (build-time only)
  • flit-core (build-time only)
    • toml (3.1–3.3 only)
    • pytoml (–3.0 only)
  • packaging
    • pyparsing (–21 only)
    • six (–20.7 only)
    • attrs (19.1 only)
  • pyproject-hooks
  • colorama (Windows only)
  • importlib-metadata (Python 3.7 only)
    • zipp
    • typing-extensions (Python 3.7 only)
  • tomli (Python 3.7–3.10 only)

It's also worth noting that installer only supports Python 3.7+, which means none of these packages can be built for EOL Python versions. This is relevant because Spack currently supports bootstrapping its style dependencies with Python 3.6. With this PR, that will no longer be supported. Curious how @tgamblin and @alalazo feel about this.

If we really do need to be able to install these with Python 3.6, we would need to add a new WheelPackage base class that supports installing with pip (Python 2+) or installer (Python 3.7+). This would also require 2 new builders. I was hoping to try to tackle this in a follow-up PR, but I can turn this into one massive PR if needed.

References:

@spackbot-app

spackbot-app Bot commented Feb 3, 2023

Copy link
Copy Markdown

Hi @adamjstewart! I noticed that the following package(s) don't yet have maintainers:

  • py-pyproject-hooks

Are you interested in adopting any of these package(s)? If so, simply add the following to the package class:

    maintainers = ["adamjstewart"]

If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with spack blame:

$ spack blame py-pyproject-hooks

Thank you for your help! Please don't add maintainers without their consent.

You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer.

@adamjstewart

Copy link
Copy Markdown
Member Author

@spackbot fix style

@spackbot-app

spackbot-app Bot commented Feb 3, 2023

Copy link
Copy Markdown

Let me see if I can fix that for you!

@spackbot-app

spackbot-app Bot commented Feb 3, 2023

Copy link
Copy Markdown

I was able to run spack style --fix for you!

spack style --fix
==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  var/spack/repos/builtin/packages/py-attrs/package.py
  var/spack/repos/builtin/packages/py-build/package.py
  var/spack/repos/builtin/packages/py-colorama/package.py
  var/spack/repos/builtin/packages/py-flit-core/package.py
  var/spack/repos/builtin/packages/py-importlib-metadata/package.py
  var/spack/repos/builtin/packages/py-installer/package.py
  var/spack/repos/builtin/packages/py-packaging/package.py
  var/spack/repos/builtin/packages/py-pyparsing/package.py
  var/spack/repos/builtin/packages/py-pyproject-hooks/package.py
  var/spack/repos/builtin/packages/py-pytoml/package.py
  var/spack/repos/builtin/packages/py-six/package.py
  var/spack/repos/builtin/packages/py-toml/package.py
  var/spack/repos/builtin/packages/py-tomli/package.py
  var/spack/repos/builtin/packages/py-typing-extensions/package.py
  var/spack/repos/builtin/packages/py-zipp/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/py-colorama/package.py
reformatted var/spack/repos/builtin/packages/py-attrs/package.py
reformatted var/spack/repos/builtin/packages/py-flit-core/package.py
reformatted var/spack/repos/builtin/packages/py-importlib-metadata/package.py
reformatted var/spack/repos/builtin/packages/py-installer/package.py
reformatted var/spack/repos/builtin/packages/py-pyparsing/package.py
reformatted var/spack/repos/builtin/packages/py-packaging/package.py
reformatted var/spack/repos/builtin/packages/py-pyproject-hooks/package.py
reformatted var/spack/repos/builtin/packages/py-pytoml/package.py
reformatted var/spack/repos/builtin/packages/py-six/package.py
reformatted var/spack/repos/builtin/packages/py-toml/package.py
reformatted var/spack/repos/builtin/packages/py-tomli/package.py
reformatted var/spack/repos/builtin/packages/py-zipp/package.py
reformatted var/spack/repos/builtin/packages/py-typing-extensions/package.py
All done! ✨ 🍰 ✨
14 files reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 576 source files
  mypy checks were clean
==> spack style checks were clean
Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

@adamjstewart adamjstewart changed the title py-build: bootstrap without pip [WIP] py-build: bootstrap without pip Feb 3, 2023
@adamjstewart

Copy link
Copy Markdown
Member Author

At the moment, this feels prohibitively intrusive just to install build without pip. We may want to consider waiting until build vendors its dependencies (seems unlikely given pypa/build#470) or abandon build support until a tool with fewer dependencies inevitably comes along (given that we have 10+ build backends, why not 10+ build frontends?).

@charmoniumQ

charmoniumQ commented Feb 3, 2023

Copy link
Copy Markdown
Contributor

It's frustrating that this needs 3 different toml parsers, but not much we can do about that.

Is it preferable to install from wheel's (opaque binary) or from git (transparent source code, patches work)? I don't understand all of Python packaging, but I think pure Python packages need only be copied into the installation prefix.

@adamjstewart

Copy link
Copy Markdown
Member Author

For downloading, we prefer URLs over git because they can be checksummed. For sdist (source) vs. bdist (wheel), we currently prefer sdist since that's what we're used to from other languages and it makes patching and development easy, but there's no reason we can't use wheels. We've talked about moving away from URLs and towards wheels for the majority of pure-Python packages but that's a lot of work and may not be trivial. We've also talked about removing pure-Python packages from Spack entirely and instead using pip to handle those, but there's a lot of work that would need to be done on PyPI's side of things to make this possible.

@adamjstewart

Copy link
Copy Markdown
Member Author

Other alternative is to require pip to install build/installer, and then use build/installer for everything else. This seems silly to me, but is at least less disruptive.

@rgommers

rgommers commented Feb 9, 2023

Copy link
Copy Markdown
Contributor

Other alternative is to require pip to install build/installer, and then use build/installer for everything else. This seems silly to me, but is at least less disruptive.

After reading pypa/build#470, I think the "require pip to install build/installer" option seems very reasonable to me.

@adamjstewart

Copy link
Copy Markdown
Member Author

Let's close this. I don't love this PR in its current state. I think we'll likely go with one of the following options instead:

  1. build (and its deps) can only be built with pip, every other package can be built with either (easy)
  2. add a WheelPackage base class and convert all pure Python packages to wheels (hard)
  3. all pure Python packages should be removed from Spack and pip should be used to install them (hardest)

I may reevaluate this after Python 3.7 support is dropped.

@adamjstewart adamjstewart deleted the packages/build branch February 20, 2023 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants