[AMD][gfx1100] test_decompose_mem_bound_mm.py tolerance increase by k-artem · Pull Request #165625 · pytorch/pytorch

k-artem · 2025-10-16T09:30:41Z

test_decompose_mem_bound_mm.py tolerance increase for navi3x(gfx11x)

(cherry picked from commit 03c7da0) from

Fixes for CI HUD for gfx1100

Signed-off-by: Artem Kuzmitckii artem.kuzmitckii@amd.com

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben

pytorch-bot · 2025-10-16T09:30:44Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/165625

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit ccba83d with merge base 1009790 ():

NEW FAILURES - The following jobs have failed:

rocm-navi31 / linux-jammy-rocm-py3_10 / test (default, 1, 2, linux.rocm.gpu.gfx1100) (gh)
'Test'
rocm-navi31 / linux-jammy-rocm-py3_10 / test (default, 2, 2, linux.rocm.gpu.gfx1100) (gh)
'Test'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

k-artem · 2025-10-16T09:46:20Z

@pytorchbot label "topic: not user facing"

k-artem · 2025-10-16T09:52:59Z

@pytorchbot label "release notes: rocm"

pytorch-bot · 2025-10-16T09:54:29Z

Didn't find following labels among repository labels: label:ciflow/rocm

k-artem · 2025-10-16T09:55:12Z

@pytorchbot label "ciflow/rocm"

pytorch-bot · 2025-10-16T09:55:22Z

To add these label(s) (ciflow/rocm) to the PR, please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

jeffdaily · 2025-10-16T18:42:10Z

+    def setup_tolerance(self, rtol=None, atol=None):
+        if rtol is None:
+            rtol = self.rtol
+        if atol is None:
+            atol = self.rtol


How does this work? rtol and atol local to the function. Modifying them here should not affect call sites like line 88 below. Right?

it works because setattr(self, member, value) inside decorator set correct values.

One more isssue found atol = self.rtol instead of atol = self.atol

I keep setup_tolerance (which actually can be removed) in case we need to update tolerance value at calls of compare_* functions.
Please let me know if I need to cp in back to ROCm fork or we will wait and get it via sync w/ upstream

linux-foundation-easycla · 2025-10-17T12:25:44Z

The committers listed above are authorized under a signed CLA.

✅ login: dnikolaev-amd / name: Dmitry Nikolaev (60a0174)
✅ login: iupaikov-amd / name: iupaikov-amd (e7f2289)
✅ login: jeffdaily / name: Jeff Daily (37c138f, ccba83d)
✅ login: k-artem / name: Artem Kuzmitckii (861ca48)

jithunnair-amd · 2025-10-20T17:53:13Z

/easycla

jithunnair-amd · 2025-10-20T17:53:43Z

\easycla

This should allow us to move gfx1100 workflow to a lower frequency and also allow it to be triggered on PRs via a dedicated label, for any PRs that target Navi fixes such as [this](#165630) or [this](#165625). Pull Request resolved: #165699 Approved by: https://github.com/jeffdaily

jithunnair-amd · 2025-10-20T23:57:55Z

@pytorchbot merge -f "lint failure unrelated; CLA signed but not reflecting on PR; trying to see if it is updated internally"

pytorchmergebot · 2025-10-20T23:59:40Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-10-20T23:59:54Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

EasyCLA

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

jithunnair-amd · 2025-10-21T00:06:28Z

\easycla

This should allow us to move gfx1100 workflow to a lower frequency and also allow it to be triggered on PRs via a dedicated label, for any PRs that target Navi fixes such as [this](pytorch#165630) or [this](pytorch#165625). Pull Request resolved: pytorch#165699 Approved by: https://github.com/jeffdaily

k-artem · 2025-10-21T09:03:01Z

\easycla

molssongroup · 2025-10-21T15:55:52Z

/easycla

jeffdaily · 2025-10-21T18:30:31Z

@pytorchbot rebase

pytorchmergebot · 2025-10-21T18:32:10Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

…navi3x(gfx11x) (cherry picked from commit 03c7da0) Signed-off-by: Artem Kuzmitckii <artem.kuzmitckii@amd.com>

Bug introduced by ROCm@03c7da0 (cherry picked from commit bbd0112) Signed-off-by: Artem Kuzmitckii <artem.kuzmitckii@amd.com>

Bug introduced by ROCm@03c7da0 Signed-off-by: Artem Kuzmitckii <artem.kuzmitckii@amd.com>

pytorchmergebot · 2025-10-21T18:32:15Z

Successfully rebased gfx1100_fixes onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout gfx1100_fixes && git pull --rebase)

jeffdaily · 2025-10-22T01:36:47Z

@pytorchbot merge -f "navi3 test changes only, lint is passing. too many failures still to make sense of the signal"

pytorchmergebot · 2025-10-22T01:38:27Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

This should allow us to move gfx1100 workflow to a lower frequency and also allow it to be triggered on PRs via a dedicated label, for any PRs that target Navi fixes such as [this](pytorch#165630) or [this](pytorch#165625). Pull Request resolved: pytorch#165699 Approved by: https://github.com/jeffdaily

…orch#165625) test_decompose_mem_bound_mm.py tolerance increase for navi3x(gfx11x) (cherry picked from commit 03c7da0) from Fixes for CI HUD for gfx1100 Signed-off-by: Artem Kuzmitckii <artem.kuzmitckii@amd.com> Pull Request resolved: pytorch#165625 Approved by: https://github.com/jeffdaily Co-authored-by: iupaikov-amd <Iurii.Paikov@amd.com> Co-authored-by: Dmitry Nikolaev <139769634+dnikolaev-amd@users.noreply.github.com> Co-authored-by: Jeff Daily <jeff.daily@amd.com>

k-artem requested a review from a team as a code owner October 16, 2025 09:30

pytorch-bot Bot added the module: inductor label Oct 16, 2025

pytorchbot added the open source label Oct 16, 2025

k-artem force-pushed the gfx1100_fixes branch from c1a7876 to 62cd2ac Compare October 16, 2025 09:45

pytorch-bot Bot added the topic: not user facing topic category label Oct 16, 2025

pytorch-bot Bot added the release notes: rocm mandatorylabel label Oct 16, 2025

jeffdaily reviewed Oct 16, 2025

View reviewed changes

jithunnair-amd mentioned this pull request Oct 16, 2025

[ROCm][CI] Move gfx1100 workflows to own yaml file #165699

Closed

k-artem requested a review from jeffdaily October 17, 2025 12:40

mikaylagawarecki added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Oct 17, 2025

jeffdaily approved these changes Oct 20, 2025

View reviewed changes

pytorchmergebot added the merging label Oct 20, 2025

pytorchmergebot removed the merging label Oct 20, 2025

iupaikov-amd and others added 3 commits October 21, 2025 18:32

[AMD][gfx1100] test_decompose_mem_bound_mm.py tolerance increase for …

e7f2289

…navi3x(gfx11x) (cherry picked from commit 03c7da0) Signed-off-by: Artem Kuzmitckii <artem.kuzmitckii@amd.com>

import 'Dict' to fix common_utils.py (pytorch#2181)

60a0174

Bug introduced by ROCm@03c7da0 (cherry picked from commit bbd0112) Signed-off-by: Artem Kuzmitckii <artem.kuzmitckii@amd.com>

Fix bug in setup_tolerance

861ca48

Bug introduced by ROCm@03c7da0 Signed-off-by: Artem Kuzmitckii <artem.kuzmitckii@amd.com>

pytorchmergebot force-pushed the gfx1100_fixes branch from df7c4c4 to 861ca48 Compare October 21, 2025 18:32

jeffdaily added keep-going Don't stop on first failure, keep running tests until the end ciflow/rocm-navi31 Trigger "default" config CI on ROCm Navi31 labels Oct 21, 2025

lint

37c138f

pytorch-bot Bot removed the ciflow/rocm-navi31 Trigger "default" config CI on ROCm Navi31 label Oct 21, 2025

lint

ccba83d

jeffdaily added the ciflow/rocm-navi31 Trigger "default" config CI on ROCm Navi31 label Oct 21, 2025

pytorchmergebot added the merging label Oct 22, 2025

pytorchmergebot closed this in f3b8e15 Oct 22, 2025

pytorchmergebot added Merged and removed merging labels Oct 22, 2025

k-artem deleted the gfx1100_fixes branch January 20, 2026 11:56

Conversation

k-artem commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/165625

❌ 2 New Failures

Uh oh!

k-artem commented Oct 16, 2025

Uh oh!

k-artem commented Oct 16, 2025

Uh oh!

pytorch-bot Bot commented Oct 16, 2025

Uh oh!

k-artem commented Oct 16, 2025

Uh oh!

pytorch-bot Bot commented Oct 16, 2025

Uh oh!

jeffdaily Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

k-artem Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linux-foundation-easycla Bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jithunnair-amd commented Oct 20, 2025

Uh oh!

jithunnair-amd commented Oct 20, 2025

Uh oh!

jithunnair-amd commented Oct 20, 2025

Uh oh!

pytorchmergebot commented Oct 20, 2025

Merge started

Uh oh!

pytorchmergebot commented Oct 20, 2025

Merge failed

Uh oh!

jithunnair-amd commented Oct 21, 2025

Uh oh!

k-artem commented Oct 21, 2025

Uh oh!

molssongroup commented Oct 21, 2025

Uh oh!

jeffdaily commented Oct 21, 2025

Uh oh!

pytorchmergebot commented Oct 21, 2025

Uh oh!

pytorchmergebot commented Oct 21, 2025

Uh oh!

jeffdaily commented Oct 22, 2025

Uh oh!

pytorchmergebot commented Oct 22, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

k-artem commented Oct 16, 2025 •

edited

Loading

pytorch-bot Bot commented Oct 16, 2025 •

edited

Loading

k-artem Oct 17, 2025 •

edited

Loading

linux-foundation-easycla Bot commented Oct 17, 2025 •

edited

Loading