Make distributed modules importable even when backend not built by ezyang · Pull Request #159889 · pytorch/pytorch

ezyang · 2025-08-05T18:42:58Z

Stack from ghstack (oldest at bottom):

This PR is greatly simplified now that it stacked on top of a PR that builds with distributed always. We only need to stub functions that may not be defined due to a backend not being enabled.

Signed-off-by: Edward Yang ezyang@meta.com

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim

[ghstack-poisoned]

pytorch-bot · 2025-08-05T18:43:01Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/159889

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 0227778 with merge base 32911ff ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 4, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
inductor/test_fxir_backend.py::AOTFxirTestCase::test_aoti_fx_const

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 15a7a5a Pull-Request: #159889

ezyang · 2025-08-06T15:41:42Z

People were generally positive in chat so I'm going to push on this

[ghstack-poisoned]

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 9d066a4 Pull-Request: #159889

ezyang · 2025-08-07T13:15:45Z

The patch is now dramatically improved and you can review it IGNORING whitespace. However, to do this I had to remove indentation which will also make it merge conflictly.

The operator registrations are kind of sus, that's the last thing for me to review.

[ghstack-poisoned]

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: ea2f6df Pull-Request: #159889

[ghstack-poisoned]

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 5648df9 Pull-Request: #159889

…ild" I did this PR with claude code. No big deal if we decide to reject it. The goal is to get ar.py (checked in, needs to be removed) to run on a CPU build of PyTorch. The general strategy is to generate stubs for all of the things from the c10d C extension when it's not available. There's a new module torch/distributed/_distributed_c10d.py that mediates all of the C extension imports so that we generate all the stubs in Python in this case. If we agree that we are willing to do this, I will do a careful code review and clean up problems before subsequent review. The PR here is just to give you the flavor of the change. Signed-off-by: Edward Yang <ezyangmeta.com> cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta [ghstack-poisoned]

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 2628bc2 Pull-Request: #159889

albanD

Sounds pretty good!

albanD · 2025-08-07T17:39:14Z

test/distributed/tensor/test_fake.py

@@ -0,0 +1,41 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates
+# Owner(s): ["oncall: distributed"]


Does this need to be first line for it to work properly?

Well, if it needs to be, y'all should fix the linter lol

ezyang-mac:pytorch-tmp3 ezyang$ git grep -n Owner\(s | grep 2: | wc -l 114

torch/distributed/_distributed_c10d.py

albanD · 2025-08-07T17:41:52Z

torch/distributed/_op_stubs.py

+
+if not HAS_DISTRIBUTED:
+    # Define missing c10d operators
+    with torch.library._scoped_library("c10d", "DEF") as lib_def:


This shouldn't be a scoped lib but a regular library if we want to def to remain right?

I find this very confusing, because I see uses of scoped library that seemingly seem to imply that it is not gc'ed at the end. And empirically these library defs were enough to stop errors

with torch.library._scoped_library("aten", "IMPL") as aten: aten.impl("_nested_get_jagged_dummy", _nested_get_jagged_dummy, "CPU") aten.impl("_nested_get_jagged_dummy", _nested_get_jagged_dummy, "CUDA") aten.impl("_nested_get_jagged_dummy", _nested_get_jagged_dummy, "Meta")

the lib is definitely cleaned up... @zou3519 maybe remembers from the top of your head?

Fortunately, in the new version of this PR, I don't need any of this!

scoped library does gc at the end. Any of the uses that don't are either bugs or people doing copy-pasta or wanting to be extra safe.

albanD · 2025-08-07T17:43:37Z

torch/distributed/device_mesh.py

-
-else:
-    from torch._C._distributed_c10d import Backend as C10dBackend
+if True:  # just to temporarily avoid reindentation


Ok when looking without whitespaces

Yeah it's just to avoid conflicting

torch/distributed/distributed_c10d.py

wconstab · 2025-08-08T16:24:50Z

torch/distributed/_distributed_c10d.py

+
+if HAS_DISTRIBUTED:
+    # Distributed components are available - import from C extension
+    from torch._C._distributed_c10d import (  # Basic components; Additional distributed_c10d components


would it be possible to refactor this to

put all the stubs into torch.distributed._C_stubs.py

conditionalize this line like
src_module = "torch._C._distributed_c10d" if HAS_DISTRIBUTED else "torch.distributed._C_stubs"

from src_module import (
...
)

Well we can't literally do this. Hmm, what is idiomatic for this

I propose we switch this into a wildcard import

From _C import *? Hmm not sure if there is really a problem with that other than discoverability of where things come from. Would be a lot cleaner!

Can't wildcard the C module as it doesn't set __all__ correctly

[ghstack-poisoned]

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: dc77fbd Pull-Request: #159889

pytorchmergebot · 2025-09-04T13:13:39Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

…lt (#159889)" This reverts commit 4ae57d4. Reverted #159889 on behalf of https://github.com/jeanschmidt due to Failing internal tests, probably typechecks. See D81588399 ([comment](#159889 (comment)))

pytorchmergebot · 2025-09-04T13:13:55Z

@ezyang your PR has been successfully reverted.

[ghstack-poisoned]

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 65ed8ca Pull-Request: #159889

ezyang · 2025-09-04T16:43:59Z

@pytorchbot merge

[ghstack-poisoned]

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 8cdf5e2 Pull-Request: #159889

ezyang · 2025-09-04T16:52:08Z

@pytorchbot merge

pytorchmergebot · 2025-09-04T16:54:18Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[ghstack-poisoned]

ezyang · 2025-09-04T16:59:20Z

@pytorchbot merge

pytorchmergebot · 2025-09-04T16:59:43Z

Merge failed

Reason: New commits were pushed while merging. Please rerun the merge command.

Details for Dev Infra team

Raised by workflow job

pytorchmergebot · 2025-09-04T17:02:48Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-09-05T18:58:51Z

@ezyang your PR has been reverted as part of the stack under #160449.

ezyang · 2025-09-05T20:13:07Z

@pytorchbot merge -f "diff train forgot to get the companion diff"

pytorchmergebot · 2025-09-05T20:14:37Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-09-08T07:04:39Z

@ezyang your PR has been reverted as part of the stack under #160449.

[ghstack-poisoned]

ezyang · 2025-09-08T13:37:00Z

@pytorchbot merge

pytorchmergebot · 2025-09-08T13:39:28Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

ezyang · 2025-09-10T04:16:11Z

@pytorchbot revert -c "parent commit has a bad bug"

pytorch-bot · 2025-09-10T04:16:14Z

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: argument -c/--classification: invalid choice: 'parent commit has a bad bug' (choose from 'nosignal', 'ignoredsignal', 'landrace', 'weird', 'ghfirst')

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Try @pytorchbot --help for more info.

ezyang · 2025-09-10T04:17:05Z

@pytorchbot revert -c weird -m "parent commit has a bad fbcode only change, need to revert this too to avoid problems"

pytorchmergebot · 2025-09-10T04:18:42Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-09-10T04:18:53Z

Reverting PR 159889 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit a0d026688cd69583d5a4e0c6f3e5fda141a7f4a9 returned non-zero exit code 1

Auto-merging .ci/pytorch/macos-test.sh
CONFLICT (modify/delete): torch/distributed/_distributed_c10d.py deleted in parent of a0d026688cd (Make distributed modules importable even when backend not built (#159889)) and modified in HEAD.  Version HEAD of torch/distributed/_distributed_c10d.py left in tree.
Auto-merging torch/distributed/_functional_collectives.py
Auto-merging torch/distributed/device_mesh.py
error: could not revert a0d026688cd... Make distributed modules importable even when backend not built (#159889)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

Update

d918efa

[ghstack-poisoned]

pytorch-bot bot added ciflow/h100-symm-mem ciflow/inductor oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels Aug 5, 2025

ezyang added a commit that referenced this pull request Aug 5, 2025

[POC] Make distributed modules importable without distributed build

b9a3f41

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 15a7a5a Pull-Request: #159889

github-actions bot requested review from SherlockNoMad, albanD, antoniojkim, bdhirsh and miladm August 5, 2025 18:43

Update

0f55934

[ghstack-poisoned]

ezyang added a commit that referenced this pull request Aug 7, 2025

[POC] Make distributed modules importable without distributed build

8980bf0

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 9d066a4 Pull-Request: #159889

ezyang changed the title ~~[POC] Make distributed modules importable without distributed build~~ Make distributed modules importable without distributed build Aug 7, 2025

ezyang requested review from XilunWu, wconstab and zpcore August 7, 2025 13:16

Update

3817cbf

[ghstack-poisoned]

ezyang added a commit that referenced this pull request Aug 7, 2025

[POC] Make distributed modules importable without distributed build

14939da

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: ea2f6df Pull-Request: #159889

Update

f736996

[ghstack-poisoned]

ezyang added a commit that referenced this pull request Aug 7, 2025

[POC] Make distributed modules importable without distributed build

b72a42a

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 5648df9 Pull-Request: #159889

ezyang added a commit that referenced this pull request Aug 7, 2025

[POC] Make distributed modules importable without distributed build

2db204e

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 2628bc2 Pull-Request: #159889

albanD reviewed Aug 7, 2025

View reviewed changes

wconstab reviewed Aug 8, 2025

View reviewed changes

Update

f2b3abb

[ghstack-poisoned]

ezyang added a commit that referenced this pull request Aug 10, 2025

[POC] Make distributed modules importable without distributed build

b77cc02

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: dc77fbd Pull-Request: #159889

pytorchmergebot reopened this Sep 4, 2025

Update

decfbd1

[ghstack-poisoned]

ezyang added a commit that referenced this pull request Sep 4, 2025

[POC] Make distributed modules importable without distributed build

2bbc6b0

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 65ed8ca Pull-Request: #159889

Update

fc2f4ec

[ghstack-poisoned]

ezyang added a commit that referenced this pull request Sep 4, 2025

[POC] Make distributed modules importable without distributed build

8754cf7

Signed-off-by: Edward Yang <ezyang@meta.com> ghstack-source-id: 8cdf5e2 Pull-Request: #159889

Update

715055c

[ghstack-poisoned]

Update

0227778

[ghstack-poisoned]

This was referenced Sep 15, 2025

Cherry pick revert of Make distributed modules importable even when backend not built #163022

Closed

Revert "Make distributed modules importable even when backend not bui… #163024

Merged

wdvr mentioned this pull request Sep 25, 2025

Revert "[RELAND] Always build USE_DISTRIBUTED (#160449) and Make dist… #163830

Closed

		@@ -0,0 +1,41 @@
		# Copyright (c) Meta Platforms, Inc. and affiliates
		# Owner(s): ["oncall: distributed"]

Conversation

ezyang commented Aug 5, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/159889

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

ezyang commented Aug 6, 2025

Uh oh!

ezyang commented Aug 7, 2025

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pytorchmergebot commented Sep 4, 2025

Uh oh!

pytorchmergebot commented Sep 4, 2025

Uh oh!

ezyang commented Sep 4, 2025

Uh oh!

ezyang commented Sep 4, 2025

Uh oh!

pytorchmergebot commented Sep 4, 2025

Merge started

Uh oh!

ezyang commented Sep 4, 2025

Uh oh!

pytorchmergebot commented Sep 4, 2025

Merge failed

Uh oh!

pytorchmergebot commented Sep 4, 2025

Merge started

Uh oh!

pytorchmergebot commented Sep 5, 2025

Uh oh!

ezyang commented Sep 5, 2025

Uh oh!

pytorchmergebot commented Sep 5, 2025

Merge started

Uh oh!

pytorchmergebot commented Sep 8, 2025

Uh oh!

ezyang commented Sep 8, 2025

Uh oh!

ezyang commented Aug 5, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Aug 5, 2025 •

edited

Loading