Skip modded_nanogpt model in TorchInductor benchmark by huydhn · Pull Request #172125 · pytorch/pytorch

huydhn · 2026-01-09T21:22:56Z

This has never been working correctly #172015 (comment), so let's skip it completely. For more context, this model switches from eager_fail_to_run to fail_accuracy status recently, root cause is not clear, but it's not related to 2.10 RC

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @chauhang @amjames @Lucaskabela @jataylo

pytorch-bot · 2026-01-09T21:23:00Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/172125

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 28 Pending

As of commit 067e3d5 with merge base db4f3aa ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

zou3519 · 2026-01-09T21:25:43Z

can you file an issue saying "this has never worked correctly before" and assign it to whoever added the model originally?

atalman

lgtm!

huydhn · 2026-01-09T22:19:28Z

@pytorchbot rebase

pytorchmergebot · 2026-01-09T22:21:01Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

This has never been working correctly pytorch#172015 (comment) Signed-off-by: Huy Do <huydhn@gmail.com>

pytorchmergebot · 2026-01-09T22:21:05Z

Successfully rebased skip-modded_nanogpt onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout skip-modded_nanogpt && git pull --rebase)

huydhn · 2026-01-09T22:35:29Z

@pytorchbot drci

huydhn · 2026-01-09T22:37:49Z

@pytorchbot merge -f 'Should be ok to force merge, lint jobs are green'

huydhn · 2026-01-09T22:38:25Z

can you file an issue saying "this has never worked correctly before" and assign it to whoever added the model originally?

Yes, @atalman mentions the same thing, let me create a separate issue for that

pytorchmergebot · 2026-01-09T22:39:28Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

huydhn · 2026-01-09T22:41:01Z

@pytorchbot cherry-pick --onto release/2.10 --fixes 'Skip modded_nanogpton TorchInductor benchmark' -c regression

This has never been working correctly #172015 (comment), so let's skip it completely. For more context, this model switches from [eager_fail_to_run to fail_accuracy](https://hud.pytorch.org/benchmark/v3/dashboard/compiler_inductor?renderGroupId=main&time.start=2026-01-02T00%3A00%3A00.000Z&time.end=2026-01-09T23%3A59%3A59.999Z&filters.repo=pytorch%2Fpytorch&filters.benchmarkName=compiler&filters.backend=&filters.mode=training&filters.dtype=amp&filters.deviceName=cuda+%28h100%29&filters.device=cuda&filters.arch=h100&filters.model=modded_nanogpt&lbranch=prepare-perf-number-2.9.1&rbranch=prepare-perf-number-2.10-v2&rcommit.commit=48ac90a47e9eb80c30282ae991f2bb3399427919&rcommit.workflow_id=20837908962&rcommit.date=2026-01-09T04%3A00%3A00Z&rcommit.branch=prepare-perf-number-2.10-v2&lcommit.commit=be32f077fb50b49ad856e957a6860d37565583db&lcommit.workflow_id=20799100249&lcommit.date=2026-01-08T02%3A00%3A00Z&lcommit.branch=prepare-perf-number-2.9.1&maxSampling=110) status recently, root cause is not clear, but it's not related to 2.10 RC Pull Request resolved: #172125 Approved by: https://github.com/zou3519, https://github.com/atalman (cherry picked from commit 0cea083)

pytorchbot · 2026-01-09T22:47:14Z

Cherry picking #172125

The cherry pick PR is at #172141 and it is linked with issue Skip modded_nanogpton TorchInductor benchmark. The following tracker issues are updated:

[v.2.10.0] Release Tracker #170119 (comment)

Details for Dev Infra team

Raised by workflow job

Skip modded_nanogpt model in TorchInductor benchmark (#172125) This has never been working correctly #172015 (comment), so let's skip it completely. For more context, this model switches from [eager_fail_to_run to fail_accuracy](https://hud.pytorch.org/benchmark/v3/dashboard/compiler_inductor?renderGroupId=main&time.start=2026-01-02T00%3A00%3A00.000Z&time.end=2026-01-09T23%3A59%3A59.999Z&filters.repo=pytorch%2Fpytorch&filters.benchmarkName=compiler&filters.backend=&filters.mode=training&filters.dtype=amp&filters.deviceName=cuda+%28h100%29&filters.device=cuda&filters.arch=h100&filters.model=modded_nanogpt&lbranch=prepare-perf-number-2.9.1&rbranch=prepare-perf-number-2.10-v2&rcommit.commit=48ac90a47e9eb80c30282ae991f2bb3399427919&rcommit.workflow_id=20837908962&rcommit.date=2026-01-09T04%3A00%3A00Z&rcommit.branch=prepare-perf-number-2.10-v2&lcommit.commit=be32f077fb50b49ad856e957a6860d37565583db&lcommit.workflow_id=20799100249&lcommit.date=2026-01-08T02%3A00%3A00Z&lcommit.branch=prepare-perf-number-2.9.1&maxSampling=110) status recently, root cause is not clear, but it's not related to 2.10 RC Pull Request resolved: #172125 Approved by: https://github.com/zou3519, https://github.com/atalman (cherry picked from commit 0cea083) Co-authored-by: Huy Do <huydhn@gmail.com>

Summary: This has never been working correctly pytorch/pytorch#172015 (comment), so let's skip it completely. For more context, this model switches from [eager_fail_to_run to fail_accuracy](https://hud.pytorch.org/benchmark/v3/dashboard/compiler_inductor?renderGroupId=main&time.start=2026-01-02T00%3A00%3A00.000Z&time.end=2026-01-09T23%3A59%3A59.999Z&filters.repo=pytorch%2Fpytorch&filters.benchmarkName=compiler&filters.backend=&filters.mode=training&filters.dtype=amp&filters.deviceName=cuda+%28h100%29&filters.device=cuda&filters.arch=h100&filters.model=modded_nanogpt&lbranch=prepare-perf-number-2.9.1&rbranch=prepare-perf-number-2.10-v2&rcommit.commit=48ac90a47e9eb80c30282ae991f2bb3399427919&rcommit.workflow_id=20837908962&rcommit.date=2026-01-09T04%3A00%3A00Z&rcommit.branch=prepare-perf-number-2.10-v2&lcommit.commit=be32f077fb50b49ad856e957a6860d37565583db&lcommit.workflow_id=20799100249&lcommit.date=2026-01-08T02%3A00%3A00Z&lcommit.branch=prepare-perf-number-2.9.1&maxSampling=110) status recently, root cause is not clear, but it's not related to 2.10 RC X-link: pytorch/pytorch#172125 Approved by: https://github.com/zou3519, https://github.com/atalman Reviewed By: atalman, jeanschmidt Differential Revision: D90421963 fbshipit-source-id: 8ae5e397546afd05da131770095f600cb2bea57a

This has never been working correctly pytorch#172015 (comment), so let's skip it completely. For more context, this model switches from [eager_fail_to_run to fail_accuracy](https://hud.pytorch.org/benchmark/v3/dashboard/compiler_inductor?renderGroupId=main&time.start=2026-01-02T00%3A00%3A00.000Z&time.end=2026-01-09T23%3A59%3A59.999Z&filters.repo=pytorch%2Fpytorch&filters.benchmarkName=compiler&filters.backend=&filters.mode=training&filters.dtype=amp&filters.deviceName=cuda+%28h100%29&filters.device=cuda&filters.arch=h100&filters.model=modded_nanogpt&lbranch=prepare-perf-number-2.9.1&rbranch=prepare-perf-number-2.10-v2&rcommit.commit=48ac90a47e9eb80c30282ae991f2bb3399427919&rcommit.workflow_id=20837908962&rcommit.date=2026-01-09T04%3A00%3A00Z&rcommit.branch=prepare-perf-number-2.10-v2&lcommit.commit=be32f077fb50b49ad856e957a6860d37565583db&lcommit.workflow_id=20799100249&lcommit.date=2026-01-08T02%3A00%3A00Z&lcommit.branch=prepare-perf-number-2.9.1&maxSampling=110) status recently, root cause is not clear, but it's not related to 2.10 RC Pull Request resolved: pytorch#172125 Approved by: https://github.com/zou3519, https://github.com/atalman

- Re-enable `detectron2_maskrcnn` skip in skip.all. - Re-enable all `timm_*` model skips in skip.all. - Keep explicit upstream PR context comments for `modded_nanogpt` and `pytorch_CycleGAN_and_pix2pix`. - Remove stale expected-accuracy rows for skipped models. Relevant PRs: [1] pytorch/pytorch#120299 [2] pytorch/pytorch#164816 [3] pytorch/pytorch#172125 [4] pytorch/pytorch#175066 [5] #2306

huydhn requested review from Lucaskabela, atalman and zou3519 January 9, 2026 21:22

huydhn added the test-config/default label Jan 9, 2026

pytorch-bot bot added ciflow/inductor module: dynamo labels Jan 9, 2026

huydhn removed the request for review from zou3519 January 9, 2026 21:23

zou3519 approved these changes Jan 9, 2026

View reviewed changes

huydhn mentioned this pull request Jan 9, 2026

TorchInductor benchmark accuracy check failing for some models in 2.10 #172015

Closed

4 tasks

atalman approved these changes Jan 9, 2026

View reviewed changes

atalman added the topic: not user facing topic category label Jan 9, 2026

Skip modded_nanogpt model in TorchInductor benchmark

067e3d5

This has never been working correctly pytorch#172015 (comment) Signed-off-by: Huy Do <huydhn@gmail.com>

pytorchmergebot force-pushed the skip-modded_nanogpt branch from 7890854 to 067e3d5 Compare January 9, 2026 22:21

pytorchmergebot added the merging label Jan 9, 2026

pytorchmergebot closed this in 0cea083 Jan 9, 2026

pytorchmergebot added Merged and removed merging labels Jan 9, 2026

pytorchbot mentioned this pull request Jan 9, 2026

[v.2.10.0] Release Tracker #170119

Closed

pbielak mentioned this pull request Mar 11, 2026

Align torchbench skip lists with upstream PyTorch behavior. intel/torch-xpu-ops#3038

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip modded_nanogpt model in TorchInductor benchmark#172125

Skip modded_nanogpt model in TorchInductor benchmark#172125
huydhn wants to merge 1 commit intopytorch:mainfrom
huydhn:skip-modded_nanogpt

huydhn commented Jan 9, 2026 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Jan 9, 2026 •

edited

Loading

Uh oh!

zou3519 commented Jan 9, 2026

Uh oh!

atalman left a comment

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

pytorchmergebot commented Jan 9, 2026

Uh oh!

pytorchmergebot commented Jan 9, 2026

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

pytorchmergebot commented Jan 9, 2026

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

pytorchbot commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

huydhn commented Jan 9, 2026 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/172125

⏳ No Failures, 28 Pending

Uh oh!

zou3519 commented Jan 9, 2026

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

pytorchmergebot commented Jan 9, 2026

Uh oh!

pytorchmergebot commented Jan 9, 2026

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

pytorchmergebot commented Jan 9, 2026

Merge started

Uh oh!

huydhn commented Jan 9, 2026

Uh oh!

pytorchbot commented Jan 9, 2026

Cherry picking #172125

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

huydhn commented Jan 9, 2026 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jan 9, 2026 •

edited

Loading