bump kineto submodule to 0035505 by scotts · Pull Request #177101 · pytorch/pytorch

scotts · 2026-03-11T00:16:11Z

Updates Kineto to hash 00355051f09eef00ba32c366326e73e8057421da from March 10, 2026. See: https://github.com/pytorch/kineto/tree/00355051f09eef00ba32c366326e73e8057421da

pytorch-bot · 2026-03-11T00:16:15Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/177101

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

⏳ 3 Pending, 1 Unrelated Failure

As of commit 1fd9c49 with merge base 5843119 ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / linux-jammy-py3.10-gcc11 / test (distributed, 2, 3, lf.linux.2xlarge) (gh) (similar failure)
test/distributed/tensor/test_dtensor_ops.py::TestUnbackedDTensorOpsCPU::test_unbacked_dtensor_op_db_kthvalue_cpu_float32

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ryanzhang22

one CI test failure but from first glance it looks unrelated

jathu

thanks!

scotts · 2026-03-11T15:48:02Z

I dug into the new failure (https://github.com/pytorch/pytorch/actions/runs/22930355625/job/66606473310?pr=177101), and I think it's unrelated to the Kineto changes as it's a test about CUDA graph capture:

distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph [rank1]:[E311 13:57:29.074453410 ProcessGroupNCCL.cpp:2126] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: operation not permitted when stream is capturing

AMD docs: https://rocm.docs.amd.com/projects/HIP/en/docs-6.3.3/reference/error_codes.html#term-hipErrorStreamCaptureUnsupported

See the test:

pytorch/test/distributed/test_c10d_ops_nccl.py

Line 291 in 42d2a45

def test_allreduce_in_cudagraph(self):

scotts · 2026-03-11T15:58:30Z

@pytorchbot merge

pytorchmergebot · 2026-03-11T16:00:47Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2026-03-11T16:01:06Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.gfx950.4)

Details for Dev Infra team

Raised by workflow job

ngimel · 2026-03-11T16:25:46Z

@pytorchbot merge -i

pytorchmergebot · 2026-03-11T16:29:45Z

Merge started

Your change will be merged while ignoring the following 2 checks: pull / linux-jammy-py3.10-gcc11 / test (distributed, 1, 2, lf.linux.2xlarge), trunk / linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.gfx950.4)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2026-03-12T19:30:41Z

@pytorchbot revert -m "I could be wrong, but look like this introduced rocm distributed failures, see https://hud.pytorch.org/hud/pytorch/pytorch/d9d7c0bef5db069c6c47a49c1472f2d3fe034aec/2?per_page=50&name_filter=trunk%20%2F%20linux-jammy-rocm-py3.10%20%2F%20test%20(dist&mergeEphemeralLF=true" -c ignoredsignal

pytorchmergebot · 2026-03-12T19:32:33Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit 17af810. Reverted #177101 on behalf of https://github.com/malfet due to I could be wrong, but look like this introduced rocm distributed failures, see https://hud.pytorch.org/hud/pytorch/pytorch/d9d7c0bef5db069c6c47a49c1472f2d3fe034aec/2?per_page=50&name_filter=trunk%20%2F%20linux-jammy-rocm-py3.10%20%2F%20test%20(dist&mergeEphemeralLF=true ([comment](#177101 (comment)))

pytorchmergebot · 2026-03-12T19:32:38Z

@scotts your PR has been successfully reverted.

jithunnair-amd · 2026-03-12T21:13:34Z

@pytorchbot revert -m "I could be wrong, but look like this introduced rocm distributed failures, see https://hud.pytorch.org/hud/pytorch/pytorch/d9d7c0bef5db069c6c47a49c1472f2d3fe034aec/2?per_page=50&name_filter=trunk%20%2F%20linux-jammy-rocm-py3.10%20%2F%20test%20(dist&mergeEphemeralLF=true" -c ignoredsignal

@malfet @scotts While it does seem like the kineto bump (and hence rocprofiler-sdk) made ROCm distributed tests run slower and hence frequently timeout, there are also other issues causing distributed timeouts on ROCm. One of them is being addressed by #176251. While we try to get this situation under control, I have marked the trunk distributed jobs as unstable: #177301

ngimel · 2026-03-12T21:55:46Z

Let's reland this then?

huydhn · 2026-03-12T22:31:16Z

@pytorchbot merge

cc @malfet in case you want to cancel the merge

pytorchmergebot · 2026-03-12T22:33:41Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

## Motivation Fixes #3962 - The `rocprofiler-sdk` shared library is not being preloaded, causing `librocprofiler-sdk.so.1` to be missing at runtime. This is because the PyTorch `kineto` submodule was bumped which switched from `roctracer` to `rocprofiler-sdk`: pytorch/pytorch#177101 - `test_mempool_expandable` was enabled on ROCm by pytorch/pytorch#173330. This test was failing as it requires the rocm[devel] packages but was causing a crash: https://github.com/ROCm/TheRock/actions/runs/23164829934/job/67321547840. This test is currently already skipped for other torch versions. - Also skip `test_mempool_empty_cache_inactive`, `test_mempool_limited_memory_with_allocator`, `test_deleted_mempool_not_used_on_oom`, and `test_mempool_ctx_multithread` as these also require building `dummy_allocator` and are skipped in other torch versions. ## Technical Details - Adds `rocprofiler-sdk` to `LINUX_LIBRARY_PRELOADS` in `build_prod_wheels.py` so that `librocprofiler-sdk.so` is loaded - Registers `rocprofiler-sdk` as a `LibraryEntry` in `_dist_info.py` so the `rocm_sdk` package can resolve the name to the actual `.so` file. ## Test Plan - Verify that ROCm builds, the nightly smoke tests pass and that running the torch tests do not crash ## Test Result - ROCm builds successfully: https://github.com/ROCm/TheRock/actions/runs/23152017500 - Smoke tests pass for torch nightly and the runner is not crashing: https://github.com/ROCm/TheRock/actions/runs/23253453219 ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

Updates Kineto to hash 00355051f09eef00ba32c366326e73e8057421da from March 10, 2026. See: https://github.com/pytorch/kineto/tree/00355051f09eef00ba32c366326e73e8057421da Pull Request resolved: pytorch#177101 Approved by: https://github.com/ryanzhang22, https://github.com/jathu

This reverts commit 17af810. Reverted pytorch#177101 on behalf of https://github.com/malfet due to I could be wrong, but look like this introduced rocm distributed failures, see https://hud.pytorch.org/hud/pytorch/pytorch/d9d7c0bef5db069c6c47a49c1472f2d3fe034aec/2?per_page=50&name_filter=trunk%20%2F%20linux-jammy-rocm-py3.10%20%2F%20test%20(dist&mergeEphemeralLF=true ([comment](pytorch#177101 (comment)))

Updates Kineto to hash 00355051f09eef00ba32c366326e73e8057421da from March 10, 2026. See: https://github.com/pytorch/kineto/tree/00355051f09eef00ba32c366326e73e8057421da Pull Request resolved: pytorch#177101 Approved by: https://github.com/ryanzhang22, https://github.com/jathu

Updated Kineto to 00355051f09eef00ba32c366326e73e8057421da

1fd9c49

pytorch-bot bot added the topic: not user facing topic category label Mar 11, 2026

scotts mentioned this pull request Mar 11, 2026

bump kineto submodule to 18154e9 #176963

Closed

scotts added ciflow/trunk Trigger trunk jobs on your pull request ci-no-td Do not run TD on this PR ciflow/rocm-mi200 Trigger "default" config CI on ROCm MI200 ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Mar 11, 2026

ryanzhang22 approved these changes Mar 11, 2026

View reviewed changes

jathu approved these changes Mar 11, 2026

View reviewed changes

pytorchmergebot added the merging label Mar 11, 2026

pytorchmergebot removed the merging label Mar 11, 2026

pytorchmergebot added the merging label Mar 11, 2026

pytorchmergebot added the Merged label Mar 11, 2026

pytorchmergebot closed this in 17af810 Mar 11, 2026

pytorchmergebot removed the merging label Mar 11, 2026

pytorchmergebot added the Reverted label Mar 12, 2026

pytorchmergebot reopened this Mar 12, 2026

malfet mentioned this pull request Mar 12, 2026

UNSTABLE trunk / linux-jammy-rocm-py3.10 / test (distributed) #177301

Closed

pytorchmergebot added the merging label Mar 12, 2026

pytorchmergebot closed this in 2ad990b Mar 13, 2026

pytorchmergebot removed the merging label Mar 13, 2026

darren-amd mentioned this pull request Mar 19, 2026

[torch] Preload rocprofiler-sdk to fix nightly smoketests ROCm/TheRock#4065

Merged

1 task

Conversation

scotts commented Mar 11, 2026

Uh oh!

pytorch-bot bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/177101

⏳ 3 Pending, 1 Unrelated Failure

Uh oh!

ryanzhang22 left a comment

Choose a reason for hiding this comment

Uh oh!

jathu left a comment

Choose a reason for hiding this comment

Uh oh!

scotts commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scotts commented Mar 11, 2026

Uh oh!

pytorchmergebot commented Mar 11, 2026

Merge started

Uh oh!

pytorchmergebot commented Mar 11, 2026

Merge failed

Uh oh!

ngimel commented Mar 11, 2026

Uh oh!

pytorchmergebot commented Mar 11, 2026

Merge started

Uh oh!

malfet commented Mar 12, 2026

Uh oh!

pytorchmergebot commented Mar 12, 2026

Uh oh!

pytorchmergebot commented Mar 12, 2026

Uh oh!

jithunnair-amd commented Mar 12, 2026

Uh oh!

ngimel commented Mar 12, 2026

Uh oh!

huydhn commented Mar 12, 2026

Uh oh!

pytorchmergebot commented Mar 12, 2026

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

pytorch-bot bot commented Mar 11, 2026 •

edited

Loading

scotts commented Mar 11, 2026 •

edited

Loading