[ROCm] SDPA fix mem fault when dropout is enabled by alugorey · Pull Request #154864 · pytorch/pytorch

alugorey · 2025-06-02T18:22:02Z

Fixes issue that exhibited a device side memory access fault due to incorrect tensor life management

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @jataylo @hongxiayang @naromero77amd @pragupta @jerrymannil @xinyazhang @dllehr-amd

pytorch-bot · 2025-06-02T18:22:05Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154864

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 6 Unrelated Failures

As of commit a5a6140 with merge base ed0b1fe ():

NEW FAILURES - The following jobs have failed:

Lint / doc-redirects-check (gh)
Process completed with exit code 2.
pull / dynamo-cpython-test / test (dynamo_cpython, 1, 1, lf.linux.c7i.2xlarge) (gh)
'Test'

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / linux-jammy-py3.14t-clang15 / test (crossref, 1, 2, lf.linux.2xlarge) (gh) (similar failure)
test/test_serialization.py::TestSerialization::test_serialization_4gb_file
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 4, 6, linux.rocm.gpu.gfx942.1) (gh) (disabled by #106027 but the issue was closed recently and a rebase is needed to make it pass)
test/profiler/test_profiler.py::TestProfiler::test_profiler_cuda_sync_events
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 6, 6, linux.rocm.gpu.gfx942.1) (gh) (disabled by #107893 but the issue was closed recently and a rebase is needed to make it pass)
test/inductor/test_inductor_freezing.py::FreezingCpuTests::test_conv_with_as_strided_cpu
trunk / macos-py3-arm64 / test (default, 2, 3, macos-m1-stable) (gh) (detected as infra flaky with no log or failing log classifier)
trunk / macos-py3-arm64 / test (default, 3, 3, macos-m1-stable) (gh) (detected as infra flaky with no log or failing log classifier)
trunk / macos-py3-arm64 / test (openreg, 1, 1, macos-m1-stable) (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jeffdaily

Why did you add template <int Version = 2>?

aten/src/ATen/native/transformers/hip/flash_attn/ck/fmha_bwd.hpp

jeffdaily

approved, pending CI

jeffdaily

hipify changes to torch/csrc/Module.cpp were committed. Please revert those.

jeffdaily · 2025-08-18T15:38:09Z

Manually added trunk label. Two unrelated tests are timing out that would prevent merging, so trying trunk label for additional signal.

pruthvistony · 2025-08-19T16:49:27Z

@pytorchbot rebase

pytorchmergebot · 2025-08-19T16:51:03Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

jeffdaily · 2026-01-07T20:29:37Z

@pytorchbot merge

pytorchmergebot · 2026-01-07T20:31:42Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2026-01-07T21:08:57Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-jammy-rocm-py3.10 / test (default, 1, 6, linux.rocm.gpu.gfx942.1)

Details for Dev Infra team

Raised by workflow job

jeffdaily · 2026-01-07T23:49:56Z

@pytorchbot merge

pytorchmergebot · 2026-01-07T23:51:53Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2026-01-07T23:52:09Z

Merge failed

Reason: 1 jobs have failed, first few of them are: Meta Internal-Only Changes Check

Details for Dev Infra team

Raised by workflow job

jeffdaily · 2026-01-09T19:57:41Z

@pytorchbot merge -f "unrelated failures, Meta Internal approved by atalman"

pytorchmergebot · 2026-01-09T19:59:25Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

yangw-dev · 2026-01-13T17:23:21Z

@pytorchbot revert -m "already unland internally since it's broken internal tests, ticket: T251298521" -c ghfirst

pytorchmergebot · 2026-01-13T17:25:24Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2026-01-13T17:25:26Z

Reverting PR 154864 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit fd27eb7b00d25438af1109055a85ab203a83c949 returned non-zero exit code 1

Auto-merging torch/_C/__init__.pyi.in
Auto-merging torch/backends/cuda/__init__.py
CONFLICT (content): Merge conflict in torch/backends/cuda/__init__.py
Auto-merging torch/csrc/Module.cpp
Auto-merging torch/testing/_internal/common_cuda.py
error: could not revert fd27eb7b00d... [ROCm] SDPA fix mem fault when dropout is enabled (#154864)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

yangw-dev · 2026-01-13T17:34:15Z

@pytorchbot revert -m "already unland internally since it's broken internal tests, ticket: T251298521" -c ghfirst

pytorchmergebot · 2026-01-13T17:36:18Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2026-01-13T17:36:24Z

@alugorey your PR has been successfully reverted.

alugorey · 2026-01-16T22:20:05Z

@yangw-dev Can we please get some information as to what is breaking? this is the second time this was reverted without receiving any information as to why besides "internal breakage". Is there something we need to change on our end to get this to land?

jeffdaily · 2026-01-31T18:32:54Z

Hi @yangw-dev @drisspg. @alugorey has pushed changes that we think addresses your internal issue. Can you please import and run the test that was failing for you?

alugorey · 2026-02-04T18:24:26Z

Hi @yangw-dev @drisspg. @alugorey has pushed changes that we think addresses your internal issue. Can you please import and run the test that was failing for you?

ping @yangw-dev @drisspg

Can we please get some attention on this matter? the memory access fault we are fixing here is beginning to surface elsewhere so we need this in ASAP

alugorey · 2026-02-24T19:02:20Z

Dropping this PR for #174708

will close both when it lands

pytorchbot added the open source label Jun 2, 2025

jeffdaily requested changes Jun 2, 2025

View reviewed changes

aten/src/ATen/native/transformers/hip/flash_attn/ck/fmha_bwd.hpp Outdated Show resolved Hide resolved

soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 2, 2025

alugorey force-pushed the sdpa_mem_fault_dropout branch from 6ca22cc to dd7dfe9 Compare June 20, 2025 22:02

jeffdaily added the ciflow/rocm Trigger "default" config CI on ROCm label Jun 20, 2025

jeffdaily changed the title ~~Sdpa mem fault dropout~~ [ROCm] SDPA fix mem fault when dropout is enabled Jun 20, 2025

pytorch-bot bot added the module: rocm AMD GPU support for Pytorch label Jun 20, 2025

jeffdaily previously approved these changes Jun 20, 2025

View reviewed changes

pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Jun 20, 2025

jeffdaily added the ciflow/rocm Trigger "default" config CI on ROCm label Jun 20, 2025

pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Aug 10, 2025

alugorey force-pushed the sdpa_mem_fault_dropout branch from ca719d8 to 9c02e37 Compare August 10, 2025 03:16

jeffdaily added the ciflow/rocm Trigger "default" config CI on ROCm label Aug 11, 2025

alugorey force-pushed the sdpa_mem_fault_dropout branch from 9c02e37 to 22ff165 Compare August 11, 2025 22:36

alugorey requested review from eqy and syed-ahmed as code owners August 11, 2025 22:36

pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Aug 11, 2025

jeffdaily added the ciflow/rocm Trigger "default" config CI on ROCm label Aug 12, 2025

jeffdaily requested changes Aug 12, 2025

View reviewed changes

pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Aug 12, 2025

jeffdaily previously approved these changes Aug 12, 2025

View reviewed changes

jeffdaily added release notes: rocm mandatorylabel ciflow/rocm Trigger "default" config CI on ROCm labels Aug 12, 2025

pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Aug 12, 2025

jeffdaily added ciflow/rocm Trigger "default" config CI on ROCm ciflow/trunk Trigger trunk jobs on your pull request labels Aug 12, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 7, 2026

pytorchmergebot added the merging label Jan 7, 2026

pytorchmergebot removed the merging label Jan 7, 2026

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Jan 7, 2026

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 7, 2026

pytorchmergebot added the merging label Jan 7, 2026

yangw-dev mentioned this pull request Jan 13, 2026

Clean up existing pyrefly errors on main #172251

Closed

alugorey added 4 commits February 9, 2026 20:24

[ROCm] CK_SDPA fix mem fault when dropout is enabled

b0093a8

Fix mistakenly removed fudge_factor for grad_value

cb550ec

Fix out fudge factors after rebase

fe02d1b

Fix dimensions

a5a6140

alugorey mentioned this pull request Feb 24, 2026

[ROCm] Sdpa dropout fix #174708

Open

Conversation

alugorey commented Jun 2, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154864

❌ 2 New Failures, 6 Unrelated Failures

Uh oh!

jeffdaily left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jeffdaily left a comment

Choose a reason for hiding this comment

Uh oh!

jeffdaily left a comment

Choose a reason for hiding this comment

Uh oh!

jeffdaily commented Aug 18, 2025

Uh oh!

pruthvistony commented Aug 19, 2025

Uh oh!

pytorchmergebot commented Aug 19, 2025

Uh oh!

jeffdaily commented Jan 7, 2026

Uh oh!

pytorchmergebot commented Jan 7, 2026

Merge started

Uh oh!

pytorchmergebot commented Jan 7, 2026

Merge failed

Uh oh!

jeffdaily commented Jan 7, 2026

Uh oh!

pytorchmergebot commented Jan 7, 2026

Merge started

Uh oh!

pytorchmergebot commented Jan 7, 2026

Merge failed

Uh oh!

jeffdaily commented Jan 9, 2026

Uh oh!

pytorchmergebot commented Jan 9, 2026

Merge started

Uh oh!

yangw-dev commented Jan 13, 2026

Uh oh!

pytorchmergebot commented Jan 13, 2026

Uh oh!

pytorchmergebot commented Jan 13, 2026

Reverting PR 154864 failed

Uh oh!

yangw-dev commented Jan 13, 2026

Uh oh!

pytorchmergebot commented Jan 13, 2026

Uh oh!

pytorchmergebot commented Jan 13, 2026

Uh oh!

alugorey commented Jan 16, 2026

Uh oh!

jeffdaily commented Jan 31, 2026

Uh oh!

alugorey commented Feb 4, 2026

Uh oh!

alugorey commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

alugorey commented Jun 2, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jun 2, 2025 •

edited

Loading