[CUDA] Use runtime driver API for cuStreamWriteValue32 by eee4017 · Pull Request #158295 · pytorch/pytorch

eee4017 · 2025-07-14T23:28:30Z

Reopen #156097

Fixes #154073

Reference: NVIDIA/Fuser#4197

See PR #156097 and #154097

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k

pytorch-bot · 2025-07-14T23:28:34Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158295

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit c4a63b8 with merge base da05b7f ():

NEW FAILURES - The following jobs have failed:

periodic / linux-jammy-cuda12.4-py3.10-gcc11-sm89 / test (default, 3, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
inductor/test_cudagraph_trees.py::CudaGraphTreeTests::test_frozen_fn
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / build (gh)
ModuleNotFoundError: No module named 'setuptools.command.bdist_wheel'

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (similar failure)
'test/test_numba_integration.py::TestNumbaIntegration::test_from_cuda_array_interface_inferred_strides'
periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 4, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (similar failure)
'test/test_numba_integration.py::TestNumbaIntegration::test_from_cuda_array_interface_inferred_strides'
periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 7, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (similar failure)
'test/test_numba_integration.py::TestNumbaIntegration::test_from_cuda_array_interface_inferred_strides'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

huydhn · 2025-07-14T23:56:12Z

I have this PR to add the older driver check to CI #158300. We probably want to merge that first, then rebase this PR on top?

pytorch-bot · 2025-07-15T00:05:24Z

To add the ciflow label ciflow/trunk please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-07-15T00:05:45Z

To add the ciflow label ciflow/h100-distributed please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-07-15T00:05:45Z

To add the ciflow label ciflow/trunk please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

ngimel · 2025-07-15T16:31:25Z

@huydhn Since you've manually checked this PR on an older driver, I think it's ok to merge it

eqy · 2025-07-15T17:48:40Z

@pytorchmergebot rebase

pytorchmergebot · 2025-07-15T17:50:14Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-07-15T17:50:19Z

Successfully rebased main-fix-symmetric-memory-api-call-new onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout main-fix-symmetric-memory-api-call-new && git pull --rebase)

huydhn · 2025-07-16T08:46:59Z

@pytorchbot rebase -b main

huydhn · 2025-07-16T17:51:21Z

@pytorchbot merge

pytorchmergebot · 2025-07-16T17:53:19Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-16T17:58:52Z

Merge failed

Reason: 1 jobs have failed, first few of them are: s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / build

Details for Dev Infra team

Raised by workflow job

ngimel · 2025-07-16T18:01:20Z

@pytorchbot merge -i

pytorchmergebot · 2025-07-16T18:03:42Z

Merge started

Your change will be merged while ignoring the following 1 checks: s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / build

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-16T19:55:07Z

Merge failed

Reason: 1 jobs have failed, first few of them are: periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 4, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck)

Details for Dev Infra team

Raised by workflow job

ngimel · 2025-07-16T23:06:29Z

@pytorchbot merge -i

pytorchmergebot · 2025-07-16T23:08:41Z

Merge started

Your change will be merged while ignoring the following 5 checks: s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / build, periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 4, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck), periodic / linux-jammy-cuda12.4-py3.10-gcc11 / test (legacy_nvidia_driver, 3, 5, lf.linux.4xlarge.nvidia.gpu), periodic / linux-jammy-cuda12.8-py3.10-gcc9-debug / test (default, 7, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build), periodic / linux-jammy-cuda12.4-py3.10-gcc11-sm89 / test (default, 3, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-07-17T18:31:34Z

@pytorchbot cherry-pick --onto release/2.8 -c critical

Reopen #156097 Fixes #154073 Reference: NVIDIA/Fuser#4197 See PR #156097 and #154097 Pull Request resolved: #158295 Approved by: https://github.com/Skylion007, https://github.com/ngimel, https://github.com/eqy, https://github.com/huydhn Co-authored-by: Wei Wang <weiwan@nvidia.com> (cherry picked from commit a9f902a)

pytorchbot · 2025-07-17T18:36:13Z

Cherry picking #158295

The cherry pick PR is at #158585 and it is recommended to link a critical cherry pick PR with an issue. The following tracker issues are updated:

[v.2.8.0] Release Tracker #156745 (comment)

Details for Dev Infra team

Raised by workflow job

[CUDA] Use runtime driver API for cuStreamWriteValue32 (#158295) Reopen #156097 Fixes #154073 Reference: NVIDIA/Fuser#4197 See PR #156097 and #154097 Pull Request resolved: #158295 Approved by: https://github.com/Skylion007, https://github.com/ngimel, https://github.com/eqy, https://github.com/huydhn (cherry picked from commit a9f902a) Co-authored-by: Frank Lin <eee4017@gmail.com> Co-authored-by: Wei Wang <weiwan@nvidia.com>

[CUDA] Use runtime driver API for cuStreamWriteValue32 (pytorch#158295) Reopen pytorch#156097 Fixes pytorch#154073 Reference: NVIDIA/Fuser#4197 See PR pytorch#156097 and pytorch#154097 Pull Request resolved: pytorch#158295 Approved by: https://github.com/Skylion007, https://github.com/ngimel, https://github.com/eqy, https://github.com/huydhn (cherry picked from commit a9f902a) Co-authored-by: Frank Lin <eee4017@gmail.com> Co-authored-by: Wei Wang <weiwan@nvidia.com>

eee4017 requested review from eqy and syed-ahmed as code owners July 14, 2025 23:28

pytorch-bot bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Jul 14, 2025

eee4017 mentioned this pull request Jul 14, 2025

[CUDA] Use runtime driver API for cuStreamWriteValue32 #156097

Closed

Skylion007 approved these changes Jul 14, 2025

View reviewed changes

pytorchbot added the open source label Jul 14, 2025

huydhn added the topic: not user facing topic category label Jul 14, 2025

ngimel added this to the 2.8.0 milestone Jul 15, 2025

ngimel added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 15, 2025

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Jul 15, 2025

ngimel added ciflow/trunk Trigger trunk jobs on your pull request ciflow/h100-distributed labels Jul 15, 2025

pytorch-bot bot removed ciflow/h100-distributed ciflow/trunk Trigger trunk jobs on your pull request labels Jul 15, 2025

ngimel approved these changes Jul 15, 2025

View reviewed changes

eqy approved these changes Jul 15, 2025

View reviewed changes

pytorchmergebot force-pushed the main-fix-symmetric-memory-api-call-new branch from dcdc640 to 68d505b Compare July 15, 2025 17:50

huydhn approved these changes Jul 16, 2025

View reviewed changes

huydhn added test-config/default test-config/legacy_nvidia_driver labels Jul 16, 2025

pytorchmergebot force-pushed the main-fix-symmetric-memory-api-call-new branch from 95066e0 to c4a63b8 Compare July 16, 2025 17:49

pytorch-bot bot removed the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Jul 16, 2025

huydhn added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Jul 16, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 16, 2025

pytorchmergebot added the merging label Jul 16, 2025

pytorchmergebot removed the merging label Jul 16, 2025

pytorchmergebot added the merging label Jul 16, 2025

pytorchmergebot removed the merging label Jul 16, 2025

ngimel removed the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Jul 16, 2025

pytorchmergebot added the merging label Jul 16, 2025

pytorchmergebot closed this in a9f902a Jul 16, 2025

pytorchmergebot added Merged and removed merging labels Jul 16, 2025

pytorchbot mentioned this pull request Jul 17, 2025

[v.2.8.0] Release Tracker #156745

Open

atalman mentioned this pull request Jul 23, 2025

Release 2.8.0 validations checklist and cherry-picks #158939

Closed

79 tasks

guangyey mentioned this pull request Jul 29, 2025

Add DeviceAllocator as the base device allocator #138222

Closed

Conversation

eee4017 commented Jul 14, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158295

❌ 2 New Failures, 3 Unrelated Failures

Uh oh!

huydhn commented Jul 14, 2025

Uh oh!

pytorch-bot bot commented Jul 15, 2025

Uh oh!

pytorch-bot bot commented Jul 15, 2025

Uh oh!

pytorch-bot bot commented Jul 15, 2025

Uh oh!

ngimel commented Jul 15, 2025

Uh oh!

eqy commented Jul 15, 2025

Uh oh!

pytorchmergebot commented Jul 15, 2025

Uh oh!

pytorchmergebot commented Jul 15, 2025

Uh oh!

huydhn commented Jul 16, 2025

Uh oh!

huydhn commented Jul 16, 2025

Uh oh!

pytorchmergebot commented Jul 16, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 16, 2025

Merge failed

Uh oh!

ngimel commented Jul 16, 2025

Uh oh!

pytorchmergebot commented Jul 16, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 16, 2025

Merge failed

Uh oh!

ngimel commented Jul 16, 2025

Uh oh!

pytorchmergebot commented Jul 16, 2025

Merge started

Uh oh!

atalman commented Jul 17, 2025

Uh oh!

pytorchbot commented Jul 17, 2025

Cherry picking #158295

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

eee4017 commented Jul 14, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 14, 2025 •

edited

Loading