Improve device info with new flops and bandwidth formula based on hardware libraries by exclamaforte · Pull Request #162245 · pytorch/pytorch

exclamaforte · 2025-09-05T04:04:23Z

Previously, DeviceInfo provided theoretical hardware information based on a hardcoded list manually created from various datasheets.

This update:

Attempting to gather the information from a hardware library like pynvml, improving accuracy and expanding support to devices that don't have entries in the datasheet list.
Adjusts flops and bw calculation based on these hardware values. For example, if the the memory or SMs are underclocked, it adjusts the theoretical max flops/bw accordingly.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo @chenyang78

pytorch-bot · 2025-09-05T04:04:26Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/162245

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 16 Unrelated Failures

As of commit ae6712f with merge base be8095b ():

NEW FAILURE - The following job has failed:

rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 3, 6, linux.rocm.gpu.gfx942.1) (gh)
'test/inductor/test_cuda_repro.py::CudaReproTests::test_epilogue_fusion_with_view'

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor / inductor-test / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'Test'
inductor / inductor-test / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
stable_diffusion_text_encoder
inductor / unit-test / inductor-test / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh) (similar failure)
'test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager'
inductor / unit-test / inductor-test / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager'
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 1, 6, linux.rocm.gpu.gfx942.1) (gh) (similar failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 2, 6, linux.rocm.gpu.gfx942.1) (gh) (disabled by #121530 but the issue was closed recently and a rebase is needed to make it pass)
functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad__batch_norm_with_update_cuda_float64

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / inductor-test / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
Process completed with exit code 134.
pull / linux-jammy-py3.10-clang12 / test (crossref, 1, 2, lf.linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-clang12 / test (default, 5, 5, lf.linux.4xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-clang12 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-clang18-asan / test (default, 4, 7, lf.linux.4xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-gcc11 / test (default, 1, 5, lf.linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.13-clang12 / test (crossref, 2, 2, lf.linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.13-clang12 / test (default, 3, 5, lf.linux.4xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.13-clang12 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 2, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops

This comment was automatically generated by Dr. CI and updates every 15 minutes.

torch/_inductor/analysis/device_info.py

v0i0

looks great, left some nits / questions

torch/_inductor/analysis/device_info.py

eellison

deferring to @v0i0 and @shunting314 but great to see this change!

…dware libraries

exclamaforte · 2025-09-10T11:11:19Z

@pytorchbot merge

pytorchmergebot · 2025-09-10T14:16:12Z

Merge failed

Reason: 1 jobs have failed, first few of them are: inductor / unit-test / inductor-test / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team

Raised by workflow job

torch/_inductor/analysis/device_info.py

exclamaforte · 2025-09-10T20:43:22Z

@pytorchbot merge

pytorchmergebot · 2025-09-10T20:45:22Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…dware libraries (pytorch#162245) Previously, DeviceInfo provided theoretical hardware information based on a hardcoded list manually created from various datasheets. This update: - Attempting to gather the information from a hardware library like `pynvml`, improving accuracy and expanding support to devices that don't have entries in the datasheet list. - Adjusts flops and bw calculation based on these hardware values. For example, if the the memory or SMs are underclocked, it adjusts the theoretical max flops/bw accordingly. Pull Request resolved: pytorch#162245 Approved by: https://github.com/v0i0, https://github.com/shunting314

facebook-github-bot · 2025-09-19T20:07:24Z

@pytorchbot revert -m="Diff reverted internally" -c="ghfirst"

This Pull Request has been reverted by a revert inside Meta. To re-land this change, please open another pull request, assign the same reviewers, fix the CI failures that caused the revert and make sure that the failing CI runs on the PR by applying the proper ciflow label (e.g., ciflow/trunk).)

pytorchmergebot · 2025-09-19T20:08:59Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-09-19T20:09:15Z

@exclamaforte your PR has been successfully reverted.

…d on hardware libraries (#162245)" This reverts commit 35d7b32. Reverted #162245 on behalf of https://github.com/facebook-github-bot due to Diff reverted internally ([comment](#162245 (comment)))

…dware libraries (pytorch#162245) Previously, DeviceInfo provided theoretical hardware information based on a hardcoded list manually created from various datasheets. This update: - Attempting to gather the information from a hardware library like `pynvml`, improving accuracy and expanding support to devices that don't have entries in the datasheet list. - Adjusts flops and bw calculation based on these hardware values. For example, if the the memory or SMs are underclocked, it adjusts the theoretical max flops/bw accordingly. Pull Request resolved: pytorch#162245 Approved by: https://github.com/v0i0, https://github.com/shunting314

…d on hardware libraries (pytorch#162245)" This reverts commit 35d7b32. Reverted pytorch#162245 on behalf of https://github.com/facebook-github-bot due to Diff reverted internally ([comment](pytorch#162245 (comment)))

…dware libraries (pytorch#162245) Previously, DeviceInfo provided theoretical hardware information based on a hardcoded list manually created from various datasheets. This update: - Attempting to gather the information from a hardware library like `pynvml`, improving accuracy and expanding support to devices that don't have entries in the datasheet list. - Adjusts flops and bw calculation based on these hardware values. For example, if the the memory or SMs are underclocked, it adjusts the theoretical max flops/bw accordingly. Pull Request resolved: pytorch#162245 Approved by: https://github.com/v0i0, https://github.com/shunting314

…d on hardware libraries (pytorch#162245)" This reverts commit 35d7b32. Reverted pytorch#162245 on behalf of https://github.com/facebook-github-bot due to Diff reverted internally ([comment](pytorch#162245 (comment)))

…dware libraries (pytorch#162245) Previously, DeviceInfo provided theoretical hardware information based on a hardcoded list manually created from various datasheets. This update: - Attempting to gather the information from a hardware library like `pynvml`, improving accuracy and expanding support to devices that don't have entries in the datasheet list. - Adjusts flops and bw calculation based on these hardware values. For example, if the the memory or SMs are underclocked, it adjusts the theoretical max flops/bw accordingly. Pull Request resolved: pytorch#162245 Approved by: https://github.com/v0i0, https://github.com/shunting314

…d on hardware libraries (pytorch#162245)" This reverts commit 35d7b32. Reverted pytorch#162245 on behalf of https://github.com/facebook-github-bot due to Diff reverted internally ([comment](pytorch#162245 (comment)))

github-actions · 2025-12-03T00:49:53Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

pytorch-bot bot added ciflow/inductor module: inductor labels Sep 5, 2025

exclamaforte force-pushed the exclamaforte/profile-diff-algo branch from 63f7d5c to 01b8dd3 Compare September 5, 2025 09:00

exclamaforte changed the title ~~Update device info with new flops formula~~ Update device info with new flops formula based on hardware libraries Sep 5, 2025

exclamaforte added the topic: not user facing topic category label Sep 5, 2025

exclamaforte force-pushed the exclamaforte/profile-diff-algo branch 2 times, most recently from 6e5f1b3 to 0178760 Compare September 6, 2025 02:39

exclamaforte added the ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 label Sep 6, 2025

exclamaforte changed the title ~~Update device info with new flops formula based on hardware libraries~~ Improve device info with new flops and bandwidth formula based on hardware libraries Sep 6, 2025

exclamaforte force-pushed the exclamaforte/profile-diff-algo branch 3 times, most recently from 358d218 to d90e00d Compare September 7, 2025 21:04

exclamaforte requested review from eellison, shunting314, v0i0 and xmfan September 8, 2025 02:57

shunting314 reviewed Sep 8, 2025

View reviewed changes

torch/_inductor/analysis/device_info.py Outdated Show resolved Hide resolved

torch/_inductor/analysis/device_info.py Outdated Show resolved Hide resolved

v0i0 approved these changes Sep 8, 2025

View reviewed changes

torch/_inductor/analysis/device_info.py Outdated Show resolved Hide resolved

torch/_inductor/analysis/device_info.py Show resolved Hide resolved

torch/_inductor/analysis/device_info.py Show resolved Hide resolved

exclamaforte force-pushed the exclamaforte/profile-diff-algo branch from 818777d to 62b819a Compare September 8, 2025 23:31

shunting314 approved these changes Sep 9, 2025

View reviewed changes

eellison reviewed Sep 10, 2025

View reviewed changes

exclamaforte added 5 commits September 10, 2025 01:09

Improve device info with new flops and bandwidth formula based on har…

e389caf

…dware libraries

switch to already supported pynvml

a623157

max clock info

dbabeea

fix tests

9022179

lint

7797807

exclamaforte force-pushed the exclamaforte/profile-diff-algo branch from 1f4b603 to 7797807 Compare September 10, 2025 10:01

fix tests

17619ef

pytorchmergebot removed the merging label Sep 10, 2025

xmfan reviewed Sep 10, 2025

View reviewed changes

torch/_inductor/analysis/device_info.py Outdated Show resolved Hide resolved

try removing torch.cuda check

ae6712f

pytorchmergebot added the merging label Sep 10, 2025

pytorchmergebot added the Merged label Sep 10, 2025

pytorchmergebot closed this in 35d7b32 Sep 10, 2025

pytorchmergebot removed the merging label Sep 10, 2025

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Sep 19, 2025

pytorchmergebot reopened this Sep 19, 2025

pytorchbot added the open source label Oct 4, 2025

github-actions bot added the Stale label Dec 3, 2025

github-actions bot closed this Jan 2, 2026

github-actions bot deleted the exclamaforte/profile-diff-algo branch February 1, 2026 02:22

Conversation

exclamaforte commented Sep 5, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/162245

❌ 1 New Failure, 16 Unrelated Failures

Uh oh!

Uh oh!

Uh oh!

v0i0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eellison left a comment

Choose a reason for hiding this comment

Uh oh!

exclamaforte commented Sep 10, 2025

Uh oh!

pytorchmergebot commented Sep 10, 2025

Merge failed

Uh oh!

Uh oh!

exclamaforte commented Sep 10, 2025

Uh oh!

pytorchmergebot commented Sep 10, 2025

Merge started

Uh oh!

facebook-github-bot commented Sep 19, 2025

Uh oh!

pytorchmergebot commented Sep 19, 2025

Uh oh!

pytorchmergebot commented Sep 19, 2025

Uh oh!

github-actions bot commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

exclamaforte commented Sep 5, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 5, 2025 •

edited

Loading