Skip to content

[BE]: Add PEP621 project section to pyproject.toml#153055

Closed
Skylion007 wants to merge 10 commits intopytorch:mainfrom
Skylion007:skylion007/add-project-pyproject-2025-05-07
Closed

[BE]: Add PEP621 project section to pyproject.toml#153055
Skylion007 wants to merge 10 commits intopytorch:mainfrom
Skylion007:skylion007/add-project-pyproject-2025-05-07

Conversation

@Skylion007
Copy link
Collaborator

@Skylion007 Skylion007 commented May 7, 2025 •
edited by pytorch-bot bot
Loading

Follow up to @ezyang's PR #153020 , but better uses PEP621 to reduce redundant fields and pass through metadata better to uv, setuptools, poetry and other tooling.

  • Enables modern tooling like uv sync and better support for tools like poetry.
  • Also allows us to set project wide settings that are respected by linters and IDE (in this example we are able centralize the minimum supported python version).
  • Currently most of the values are dynamically fetched from setuptools, eventually we can migrate all the statically defined values to pyproject.toml and they will be autopopulated in the setuptool arguments.
  • This controls what additional metadata shows up on PyPi . Special URL Names are listed here for rendering on pypi: https://packaging.python.org/en/latest/specifications/well-known-project-urls/#well-known-labels

These also clearly shows us what fields will need to be migrated to pyproject.toml over time from setup.py per #152276. Static fields be fairly easy to migrate, the dynamically built ones like requirements are a bit more challenging.

Without this, uv sync complains:

error: No `project` table found in: `pytorch/pyproject.toml`

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k

@pytorch-bot
Copy link

pytorch-bot bot commented May 7, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153055

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit eb3b4d3 with merge base d22c403 (image):

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label May 7, 2025
@Skylion007 Skylion007 requested review from albanD, ezyang, malfet and ngimel May 7, 2025 14:49
@albanD
Copy link
Collaborator

albanD commented May 7, 2025

What is this file used for?

@Skylion007
Copy link
Collaborator Author

Skylion007 commented May 7, 2025 •
edited
Loading

What is this file used for?

It's used for resolving metadata in build-backend agnostic way. In this case, it just dispatches those resolutions to setuptools/wheel builtin metadata. This allows using more features of modern tooling like uv. For instance, now we can do uv sync which will automatically synchronize all the dependencies of PyTorch with those of the virtual environment (as well as installing minimum and maximum supported versions in those ranges).

It also provides a standard way of getting project wide settings for various tooling/linters (ie ruff, black, and isort will now autodetect the minimum supported Python version from this python-requires field and setuptools will now also enforce it even without hitting syntax errors for unsupported syntax in imports etc).

See https://peps.python.org/pep-0621/ for more info

@Skylion007 Skylion007 force-pushed the skylion007/add-project-pyproject-2025-05-07 branch 2 times, most recently from c42dbb3 to 0280085 Compare May 7, 2025 15:30
@Skylion007 Skylion007 requested a review from atalman May 7, 2025 15:31
@Skylion007 Skylion007 marked this pull request as ready for review May 7, 2025 15:31
Skylion007
Skylion007 commented May 7, 2025
View reviewed changes
pyproject.toml Outdated
Copy link
Collaborator Author

@Skylion007 Skylion007 May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These values are now inferred from requires-python

@Skylion007 Skylion007 added the better-engineering Relatively self-contained tasks for better engineering contributors label May 7, 2025
@Skylion007 Skylion007 force-pushed the skylion007/add-project-pyproject-2025-05-07 branch from 0280085 to f5bff4f Compare May 7, 2025 15:36
Skylion007
Skylion007 commented May 7, 2025
View reviewed changes
pyproject.toml Outdated
Copy link
Collaborator Author

@Skylion007 Skylion007 May 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are filled from reading the logs / warnings from uv

Copy link
Collaborator Author

@Skylion007 Skylion007 May 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a follow up PR I can migrate the static fields from setuptools to the pyproject.toml, setuptools will then automatically popuate them from the pyproject.toml

@Skylion007 Skylion007 mentioned this pull request May 7, 2025
Closed
@Skylion007 Skylion007 changed the title [BE]: Apply PEP621 to pyproject.toml [BE]: Apply PEP621 project ection to pyproject.toml May 7, 2025
@Skylion007 Skylion007 changed the title [BE]: Apply PEP621 project ection to pyproject.toml [BE]: Add PEP621 project section to pyproject.toml May 7, 2025
@janeyx99 janeyx99 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label May 7, 2025
@Skylion007
Copy link
Collaborator Author

Skylion007 commented May 8, 2025

@pytorchbot merge -i

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 8, 2025
@Skylion007 Skylion007 added release notes: releng release notes category and removed ciflow/trunk Trigger trunk jobs on your pull request topic: not user facing topic category labels May 8, 2025
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented May 8, 2025

Merge started

Your change will be merged while ignoring the following 2 checks: pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 3, 3, ephemeral.linux.2xlarge), pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 2, 3, ephemeral.linux.2xlarge)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label May 8, 2025
@pytorchmergebot pytorchmergebot force-pushed the skylion007/add-project-pyproject-2025-05-07 branch from c0156e8 to f282d78 Compare May 9, 2025 12:47
@Skylion007 Skylion007 mentioned this pull request May 9, 2025
Closed
pytorchmergebot pushed a commit that referenced this pull request May 9, 2025
@Skylion007 @pytorchmergebot
[Ez][BE]: Ensure matplotlib remains optional dependency via fake_quan…
287b1ca 
…tize (#153244)

Unblocks #153055 and ensure that matplotlib should always be optional in PyTorch.
Pull Request resolved: #153244
Approved by: https://github.com/albanD
@Skylion007 Skylion007 force-pushed the skylion007/add-project-pyproject-2025-05-07 branch from f282d78 to 0c6ca59 Compare May 9, 2025 19:21
@pytorch-bot pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (checkpoint) labels May 10, 2025
@Skylion007 Skylion007 force-pushed the skylion007/add-project-pyproject-2025-05-07 branch 3 times, most recently from 6d64f35 to f9cf974 Compare May 10, 2025 18:05
@Skylion007 Skylion007 force-pushed the skylion007/add-project-pyproject-2025-05-07 branch 2 times, most recently from f94d4ce to 63e7507 Compare May 10, 2025 18:46
@Skylion007 Skylion007 force-pushed the skylion007/add-project-pyproject-2025-05-07 branch from 63e7507 to 792a4b9 Compare May 10, 2025 18:47
@Skylion007
Copy link
Collaborator Author

Skylion007 commented May 11, 2025

@pytorchbot revert -m "And failures seems related to this change, but I don't know how, see for example https://hud.pytorch.org/hud/pytorch/pytorch/7cb5c751c3f76129c95aec479647679fbba4e7ba/1?per_page=50&name_filter=macos&mergeEphemeralLF=true" -c ignoredsignal

Figured it out, it's a change in semantics of setuptools which allows namespace packages to be found by default. Hence, all these files had broken Python files under minimum requirements / import and are now getting flagged.

@Skylion007 Skylion007 force-pushed the skylion007/add-project-pyproject-2025-05-07 branch from fc881d4 to eb3b4d3 Compare May 11, 2025 20:56
@Skylion007
Copy link
Collaborator Author

Skylion007 commented May 12, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented May 12, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ezyang ezyang ezyang approved these changes

@malfet malfet Awaiting requested review from malfet

@albanD albanD Awaiting requested review from albanD

@ngimel ngimel Awaiting requested review from ngimel

@atalman atalman Awaiting requested review from atalman

Assignees

No one assigned

Labels

better-engineering Relatively self-contained tasks for better engineering contributors ci-no-td Do not run TD on this PR ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue open source release notes: distributed (checkpoint) release notes: releng release notes category Reverted topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Skylion007 @albanD @pytorchmergebot @malfet @ezyang @pytorchbot @janeyx99