make DebugMode wait on collectives before hashing outputs#169295
Open
bdhirsh wants to merge 16 commits intogh/bdhirsh/680/basefrom
Open
make DebugMode wait on collectives before hashing outputs#169295bdhirsh wants to merge 16 commits intogh/bdhirsh/680/basefrom
bdhirsh wants to merge 16 commits intogh/bdhirsh/680/basefrom
Conversation
[ghstack-poisoned]
This was referenced Dec 1, 2025
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/169295
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit 9aadcdd with merge base c49edf7 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pianpwk
reviewed
Dec 1, 2025
| # This doesn't actually test that we synchronize on collectives. | ||
| # I found this hard to test robustly since previously we would race. | ||
| # However, it does test that we do not crash. | ||
| # This doesn't actually test that we synchronize on collectives, |
yushangdi
approved these changes
Dec 1, 2025
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
Contributor
|
fyi I landed #169027! |
yushangdi
reviewed
Dec 6, 2025
torch/utils/_debug_mode.py
Outdated
| if ( | ||
| a.name == "async_op" | ||
| and type(a.type) is torch.BoolType | ||
| and i < len(args) |
Contributor
There was a problem hiding this comment.
I'm doing some testing with _allgather_base_, and I think this check might be too restrictive, because in eager, aync_op is default to True, and it doesn't show up in args, but we should wait for it
Contributor
There was a problem hiding this comment.
This worked for me:
for i, a in enumerate(func._schema.arguments):
# Annoying, async_op is not a kwarg in the non-functional collectives.
# I'm using OpOverload._schema to figure out if it exists
if a.name == "async_op" and type(a.type) is torch.BoolType:
# Determine async_op value from args, kwargs, or default
if i < len(args):
async_op_value = args[i]
elif "async_op" in kwargs:
async_op_value = kwargs["async_op"]
elif a.has_default_value():
async_op_value = a.default_value
else:
async_op_value = False
if async_op_value:
for item in result:
if isinstance(item, torch.ScriptObject) and hasattr(
item, "wait"
):
item.wait()
break
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
this is an attempt to re-land #168119 with a few tweaks: (1) for non-functional collectives, only wait on the work item with `async=True`. [See comment](#168119 (comment)) (2) For functional collectives, we can always call `wait_tensor` on the output. The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase. [ghstack-poisoned]
Collaborator
Author
|
@pytorchbot merge |
Collaborator
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Collaborator
Merge failedReason: 1 jobs have failed, first few of them are: trunk / linux-jammy-rocm-py3.10 / test (default, 2, 6, linux.rocm.gpu.gfx942.1) Details for Dev Infra teamRaised by workflow job |
Collaborator
Author
|
@pytorchbot merge -i |
Collaborator
Merge startedYour change will be merged while ignoring the following 2 checks: trunk / macos-py3-arm64 / test (mps, 1, 1, macos-m2-15), trunk / linux-jammy-rocm-py3.10 / test (default, 2, 6, linux.rocm.gpu.gfx942.1) Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Collaborator
Merge failedReason: Command Details for Dev Infra teamRaised by workflow job |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this is an attempt to re-land #168119 with a few tweaks:
(1) for non-functional collectives, only wait on the work item with
async=True. See comment(2) For functional collectives, we can always call
wait_tensoron the output.The test in this PR will probably conflict with the test in #169027, so ill wait for that PR to land first and rebase.
Stack from ghstack (oldest at bottom):