[build] modernize build-backend: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta` by XuehaiPan · Pull Request #155998 · pytorch/pytorch

XuehaiPan · 2025-06-14T17:45:32Z

Stack from ghstack (oldest at bottom):

[build] bootstrap git repo for build for non-git-clone archive #157432
[build] remove upper version pin for setuptools<80.0 #156049
[build] modernize build-frontend: python setup.py develop/install -> [uv ]pip install --no-build-isolation [-e ]. #156027
-> [build] modernize build-backend: setuptools.build_meta:__legacy__ -> setuptools.build_meta #155998
[AOTI] Fix AOT inductor CMake build dependency order #157557

Change build-system.build-backend: setuptools.build_meta:__legacy__ -> setuptools.build_meta. Also, move static package info from setup.py to pyproject.toml.

Now the repo can be installed from source via pip command instead of python setup.py develop:

python -m pip install --verbose --editable .

python -m pip install --verbose --no-build-isolation --editable .

In addition, the SDist is also buildable:

python -m build --sdist
python -m install dist/torch-*.tar.gz  # build from source using SDist

Note that we should build the SDist with a fresh git clone if we will upload the output to PyPI. Because all files under third_party will be included in the SDist. The SDist file will be huge if the git submodules are initialized.

cc @seemethere @malfet @osalpekar @atalman

[ghstack-poisoned]

pytorch-bot · 2025-06-14T17:45:34Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155998

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit fc23e9c with merge base a6fab82 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pyproject.toml

Skylion007 · 2025-06-14T17:54:53Z

MANIFEST.in

+include WORKSPACE
+include requirements.txt
+include version.txt
+include .gitmodules


Buillding from sdist works with only .gitmodules?

I guess .gitmodules is not required because we will need to build under a git clone anyway.

One solution is to ship .gitmodules in the SDist and initialize an empty git folder to allow submodule cloning:

# in setup.py if not os.path.exists(os.path.join(cwd, '.git')): subprocesses.check_call(['git', 'init'], cwd=cwd) subprocesses.check_call(['git', 'add', '.'], cwd=cwd) subprocesses.check_call(['git', 'commit', '-m', 'init for building from sdist'], cwd=cwd) subprocesses.check_call(['git', 'submodule', 'update', '--init', '--recursive'], cwd=cwd)

The sdist should include all submodules. It looks to me like this is already included (since third_party is explicitly included) in gh-152098. .gitmodules should then not be required.

MANIFEST.in

XuehaiPan · 2025-06-14T17:57:28Z

MANIFEST.in

+graft torchgen
+# FIXME: torch-xla codegen build failure if include this file in wheel
+exclude torchgen/BUILD.bazel


torch_xla/csrc/BUILD genrule command (from target //torch_xla/csrc:gen_lazy_tensor) cannot import torchgen if this file is present in site-packages/torchgen/BUILD.bazel. cc @qihqi

The following fix will not resolve the import issue.

Fix torchgen imports in codegen xla#9310

CI failure before pytorch/xla#9310: https://hud.pytorch.org/pr/pytorch/pytorch/153538#43672158222
CI failure after pytorch/xla#9310: https://hud.pytorch.org/pr/pytorch/pytorch/153538#44066727613

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: 6dbd393 Pull-Request: pytorch#155998

Skylion007 · 2025-06-14T18:20:35Z

@XuehaiPan We are sure this doesn't break s3cache, right (with the temporary build dir)?

Skylion007 · 2025-06-14T18:21:42Z

uv also support --no-build-isolation flag in pyproject.toml (unlike pip). Maybe we should add an entry for it.

Skylion007 · 2025-06-14T18:22:58Z

pyproject.toml

 ]
-# Use legacy backend to import local packages in setup.py
-build-backend = "setuptools.build_meta:__legacy__"
+build-backend = "setuptools.build_meta"


Ultimately, this still builds under build isolation which is a big gotcha for the CI. Can we migrate to legacy scikit-build? That at least does a lot of stuff to effectively disable build isolation by default.

I'm not sure I understand this comment. It doesn't seem like a good idea to migrate to another legacy tool. Build isolation can be confusing for users when they're not that experienced, but for better or worse it is how Python packaging works. I don't see a reason to try hard to subvert that.

Scikit-build-core works too, but I'm not sure how stable the setup tools shim is for it.

Skylion007 · 2025-06-14T18:23:33Z

setup.py

-BUILD_LIBTORCH_WHL = os.getenv("BUILD_LIBTORCH_WHL", "0") == "1"
-BUILD_PYTHON_ONLY = os.getenv("BUILD_PYTHON_ONLY", "0") == "1"
-
+# Also update `project.requires-python` in pyproject.toml when changing this


It would be great if we could detect build isolation and raise a warning

I don't think it's possible, nor is it desirable - it's perfectly valid for example for a packager to build with python -m build --wheel.

[ghstack-poisoned]

Skylion007 · 2025-07-02T14:36:42Z

@rgommers Main reason to subvert build isolation is the multi-hour build times of pytorch when building from src.

rgommers · 2025-07-02T16:00:14Z

@rgommers Main reason to subvert build isolation is the multi-hour build times of pytorch when building from src.

I'm not sure I understand what you mean. Yes building PyTorch is quite slow, but whether or not one uses build isolation or not has no impact on build time.

[ghstack-poisoned]

XuehaiPan · 2025-07-02T18:51:03Z

@pytorchbot merge

pytorchmergebot · 2025-07-02T18:54:00Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-02T19:04:48Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Lint / toc / linux-job

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

XuehaiPan · 2025-07-02T19:29:16Z

@pytorchbot merge

pytorchmergebot · 2025-07-02T19:40:15Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-03T01:29:38Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

XuehaiPan · 2025-07-03T02:58:42Z

@pytorchbot merge

pytorchmergebot · 2025-07-03T03:00:41Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2025-07-03T16:45:16Z

@pytorchbot revert -m "Broke inductor_cpp, wrapper see https://hud.pytorch.org/hud/pytorch/pytorch/e472daa80963aae389089f9dc324b04261e2a5ef/1?per_page=50&name_filter=inductor_cpp_" -c nosignal

pytorchmergebot · 2025-07-03T16:46:53Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-07-03T16:47:10Z

@XuehaiPan your PR has been successfully reverted.

[ghstack-poisoned]

XuehaiPan · 2025-07-04T15:47:08Z

@pytorchbot merge

pytorchmergebot · 2025-07-04T15:49:13Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Update

79390a1

[ghstack-poisoned]

XuehaiPan mentioned this pull request Jun 14, 2025

[BE][Easy] pin setuptools>=77 for PEP-639 support #155999

Closed

pytorchbot added the open source label Jun 14, 2025

XuehaiPan requested review from albanD, atalman, ezyang, malfet and seemethere June 14, 2025 17:49

XuehaiPan self-assigned this Jun 14, 2025

Skylion007 reviewed Jun 14, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

Skylion007 reviewed Jun 14, 2025

View reviewed changes

XuehaiPan commented Jun 14, 2025

View reviewed changes

Skylion007 reviewed Jun 14, 2025

View reviewed changes

XuehaiPan mentioned this pull request Jun 15, 2025

[BE] add a minimal linter to check pyproject.toml consistency #156017

Closed

XuehaiPan added 6 commits June 30, 2025 17:32

Update

1171918

[ghstack-poisoned]

Update

c54239e

[ghstack-poisoned]

Update

59a3855

[ghstack-poisoned]

Update

3c13503

[ghstack-poisoned]

Update

37ff888

[ghstack-poisoned]

Update

69ad54a

[ghstack-poisoned]

XuehaiPan mentioned this pull request Jul 2, 2025

[build] bootstrap git repo for build for non-git-clone archive #157432

Closed

Update

ee8769a

[ghstack-poisoned]

Update

36cdde7

[ghstack-poisoned]

XuehaiPan mentioned this pull request Jul 3, 2025

[AOTI] Fix AOT inductor CMake build dependency order #157557

Closed

XuehaiPan added 4 commits July 4, 2025 15:39

Update

fbd31d9

[ghstack-poisoned]

Update

973de48

[ghstack-poisoned]

Update

8b6d37a

[ghstack-poisoned]

Update

fc23e9c

[ghstack-poisoned]

XuehaiPan mentioned this pull request Aug 13, 2025

[build] pin setuptools>=77 to enable PEP 639 #158104

Open

Conversation

XuehaiPan commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155998

✅ No Failures

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XuehaiPan Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Skylion007 commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Skylion007 commented Jun 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Skylion007 commented Jul 2, 2025

Uh oh!

rgommers commented Jul 2, 2025

Uh oh!

XuehaiPan commented Jul 2, 2025

Uh oh!

pytorchmergebot commented Jul 2, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 2, 2025

Merge failed

Uh oh!

XuehaiPan commented Jul 2, 2025

Uh oh!

pytorchmergebot commented Jul 2, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 3, 2025

Uh oh!

XuehaiPan commented Jul 3, 2025

Uh oh!

pytorchmergebot commented Jul 3, 2025

Merge started

Uh oh!

malfet commented Jul 3, 2025

Uh oh!

pytorchmergebot commented Jul 3, 2025

Uh oh!

pytorchmergebot commented Jul 3, 2025

Uh oh!

XuehaiPan commented Jul 4, 2025

Uh oh!

pytorchmergebot commented Jul 4, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

XuehaiPan commented Jun 14, 2025 •

edited

Loading

pytorch-bot bot commented Jun 14, 2025 •

edited

Loading

XuehaiPan Jun 14, 2025 •

edited

Loading

Skylion007 commented Jun 14, 2025 •

edited

Loading