[ROCm] Fix ADDMM hipBLASLt regression by naromero77amd · Pull Request #138267 · pytorch/pytorch

naromero77amd · 2024-10-17T21:24:34Z

Fixes #138067

A partial reversion of this PR: #137604

The breakage is on AMD GPUs that do not fully support hipBLASLt, e.g. gfx1100

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang

pytorch-bot · 2024-10-17T21:24:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138267

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit aec924b with merge base dbd6ada ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

naromero77amd · 2024-10-17T21:28:46Z

@pytorchbot label "topic: not user facing"

pytorch-bot · 2024-10-17T21:28:48Z

❌ 🤖 pytorchbot command failed:

Got EOF while in a quoted string```
Try `@pytorchbot --help` for more info.

naromero77amd · 2024-10-17T22:57:53Z

@pytorchbot label "topic: not user facing"

malfet

LGTM

malfet · 2024-10-20T00:17:26Z

@pytorchbot merge -f "Lint + ROCM builds are fine"

pytorchmergebot · 2024-10-20T00:19:00Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

jeffdaily · 2024-10-21T19:31:27Z

@pytorchbot revert -c nosignal -m "this PR went to far when partially reverting #137604; the env var default should be the same on ROCm and CUDA"

pytorchmergebot · 2024-10-21T19:33:07Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit 14a3e12. Reverted #138267 on behalf of https://github.com/jeffdaily due to this PR went to far when partially reverting #137604; the env var default should be the same on ROCm and CUDA ([comment](#138267 (comment)))

pytorchmergebot · 2024-10-21T19:33:16Z

@naromero77amd your PR has been successfully reverted.

This PR was reopened (likely due to being reverted), so your approval was removed. Please request another review.

naromero77amd · 2024-10-24T18:25:22Z

@pytorchbot rebase

pytorchmergebot · 2024-10-24T18:26:55Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

…Cm architecture.

…T for ROCm architecture." This reverts commit 95f2a44.

This reverts commit d09135c.

pytorchmergebot · 2024-10-24T18:27:00Z

Successfully rebased fix_addmm_hipblaslt_regression onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout fix_addmm_hipblaslt_regression && git pull --rebase)

eqy · 2024-10-25T00:52:47Z

Could we check in a repro from #138067 as a test?

Yes that is what I meant by my earlier comment that I tested it manually.

Sorry, I mean checking-in an actual test to safeguard the same failure from happening in the future

naromero77amd · 2024-10-25T17:00:22Z

Could we check in a repro from #138067 as a test?

Yes that is what I meant by my earlier comment that I tested it manually.

Sorry, I mean checking-in an actual test to safeguard the same failure from happening in the future

Thank you for bringing up testing. The right way to test this is with a gfx110x which we don't have in upstream CI yet but I have been told it is on the roadmap.

I discussed with @jeffdaily and we do need to get rid of one existing test case as it is no longer appropriate.

jeffdaily · 2024-10-28T15:59:55Z

@pytorchbot merge

pytorchmergebot · 2024-10-28T16:01:33Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Port the PR pytorch#138267 from upstream main to fix the error "RuntimeError: Attempting to use hipBLASLt on a unsupported architecture!" Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com>

naromero77amd requested review from eqy and syed-ahmed as code owners October 17, 2024 21:24

naromero77amd requested review from jeffdaily and xw285cornell October 17, 2024 21:25

naromero77amd changed the title ~~Fix ADDMM hipBLASLt regression~~ [ROCm] Fix ADDMM hipBLASLt regression Oct 17, 2024

pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm module: rocm AMD GPU support for Pytorch labels Oct 17, 2024

pytorchbot added the open source label Oct 17, 2024

pytorch-bot bot added the topic: not user facing topic category label Oct 17, 2024

drisspg added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Oct 18, 2024

malfet previously approved these changes Oct 20, 2024

View reviewed changes

pytorchmergebot added the merging label Oct 20, 2024

pytorchmergebot added the Merged label Oct 20, 2024

pytorchmergebot closed this in 14a3e12 Oct 20, 2024

pytorchmergebot removed the merging label Oct 20, 2024

pytorchmergebot added the Reverted label Oct 21, 2024

pytorchmergebot reopened this Oct 21, 2024

naromero77amd marked this pull request as draft October 21, 2024 19:43

naromero77amd added 7 commits October 24, 2024 18:26

Restore default behavior with respect to DISABLE_ADDMM_CUDA_LT for RO…

7639a8b

…Cm architecture.

Exceptions on the ROCm architecture for hipBLASLt.

8cf5e8c

Revert "Restore default behavior with respect to DISABLE_ADDMM_CUDA_L…

3a0a657

…T for ROCm architecture." This reverts commit 95f2a44.

Revert "Exceptions on the ROCm architecture for hipBLASLt."

a446ce0

This reverts commit d09135c.

Remove TORCH_WARN from helper function soite can return false.

953f4e5

Set static boolean to true when hipblaslt is not supported.

4cd19a3

Remove redundant call to helper function.

d2e7c17

pytorchmergebot force-pushed the fix_addmm_hipblaslt_regression branch from cb023ce to d2e7c17 Compare October 24, 2024 18:27

Remove hipblaslt_corner_cases_rocm test as it is no longer appropriate.

aec924b

naromero77amd requested review from IvanYashchuk, lezcano and nikitaved as code owners October 25, 2024 17:53

harkgill-amd mentioned this pull request Oct 28, 2024

[Issue]: RuntimeError: Attempting to use hipBLASLt on a unsupported architecture! ROCm/ROCm#3949

Closed

jeffdaily approved these changes Oct 28, 2024

View reviewed changes

pytorchmergebot added the merging label Oct 28, 2024

pytorchmergebot closed this in 2922b9f Oct 28, 2024

pytorchmergebot removed the merging label Oct 28, 2024

naromero77amd mentioned this pull request Jan 23, 2025

[release/2.5] Fix ADDMM hipBLASLt regression ROCm/pytorch#1855

Merged

fluidnumerics-joe mentioned this pull request Mar 3, 2025

[Issue]: Comfy_UI hipblasLT not supported for Radeon 7900XT and HSA_STATUS_ERROR_OUT_OF_REGISTERS error ROCm/ROCm#4437

Closed

garrettbyrd mentioned this pull request Mar 3, 2025

[Issue]: Attempting to use hipBLASLt on an unsupported architecture! ROCm/ROCm#4440

Closed

naromero77amd deleted the fix_addmm_hipblaslt_regression branch October 29, 2025 22:35

Conversation

naromero77amd commented Oct 17, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138267

✅ No Failures

Uh oh!

naromero77amd commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 17, 2024

Uh oh!

naromero77amd commented Oct 17, 2024

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet commented Oct 20, 2024

Uh oh!

pytorchmergebot commented Oct 20, 2024

Merge started

Uh oh!

jeffdaily commented Oct 21, 2024

Uh oh!

pytorchmergebot commented Oct 21, 2024

Uh oh!

pytorchmergebot commented Oct 21, 2024

Uh oh!

naromero77amd commented Oct 24, 2024

Uh oh!

pytorchmergebot commented Oct 24, 2024

Uh oh!

pytorchmergebot commented Oct 24, 2024

Uh oh!

eqy commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

naromero77amd commented Oct 25, 2024

Uh oh!

jeffdaily commented Oct 28, 2024

Uh oh!

pytorchmergebot commented Oct 28, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

naromero77amd commented Oct 17, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Oct 17, 2024 •

edited

Loading

naromero77amd commented Oct 17, 2024 •

edited

Loading

eqy commented Oct 25, 2024 •

edited

Loading