[CUDA] Use runtime driver API for cuStreamWriteValue32 by eee4017 · Pull Request #156097 · pytorch/pytorch

eee4017 · 2025-06-16T16:02:51Z

Fixes #154073

Reference: NVIDIA/Fuser#4197

See PR #154097

@nWEIdia is currently out of the office, so I’ve temporarily taken over his work.

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @ezyang @msaroufim @dcci

pytorch-bot · 2025-06-16T16:02:56Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156097

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit dabccdb with merge base 7f14b42 ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu, unstable) (gh)
MISSING REGRESSION TEST

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ngimel

Please fix lint

eee4017 · 2025-06-17T14:08:10Z

@pytorchbot merge

pytorchmergebot · 2025-06-17T14:10:13Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-06-18T21:46:55Z

@pytorchmergebot revert -c nosiganl -m "break internal tests"

pytorch-bot · 2025-06-18T21:46:57Z

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: argument -c/--classification: invalid choice: 'nosiganl' (choose from 'nosignal', 'ignoredsignal', 'landrace', 'weird', 'ghfirst')

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Try @pytorchbot --help for more info.

atalman · 2025-06-18T21:47:09Z

@pytorchmergebot revert -c nosignal -m "break internal tests"

pytorchmergebot · 2025-06-18T21:48:43Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-06-18T21:48:53Z

@eee4017 your PR has been successfully reverted.

…#156097)" This reverts commit cf90c9f. Reverted #156097 on behalf of https://github.com/atalman due to break internal tests ([comment](#156097 (comment)))

This PR was reopened (likely due to being reverted), so your approval was removed. Please request another review.

This reverts commit f510f60.

Update

eee4017 · 2025-07-12T15:29:23Z

c10/cuda/driver_api.cpp

-#undef LOOKUP_LIBCUDA_ENTRY
+#define LOOKUP_LIBCUDA_ENTRY_WITH_VERSION_OPTIONAL(name, version) \
+  r.name##_ = reinterpret_cast<decltype(&name)>(get_symbol(#name, version));
+  C10_LIBCUDA_DRIVER_API_OPTIONAL(LOOKUP_LIBCUDA_ENTRY_WITH_VERSION_OPTIONAL)


The assertion was removed to allow compatibility with older drivers.

Thank you for the update! Let me do a round of testing on Monday to confirm

Rerun the manual test with the latest change from the PR https://github.com/pytorch/pytorch/actions/runs/16276707651/job/45957270665?pr=158181

Although it's sill running, I think the test looks good now because:

It correctly uses older NVIDIA driver https://github.com/pytorch/pytorch/actions/runs/16276707651/job/45959318683?pr=158181#step:13:764

Tests are running ok instead of crashing

huydhn · 2025-07-14T21:49:15Z

@pytorchbot merge

pytorchmergebot · 2025-07-14T21:51:13Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-14T21:51:31Z

Merge failed

Reason: 1 jobs have failed, first few of them are: Meta Internal-Only Changes Check

Details for Dev Infra team

Raised by workflow job

ngimel · 2025-07-14T23:02:36Z

@eee4017 sorry can you please create a new pull request?

eee4017 · 2025-07-14T23:29:29Z

Hi, it was created here: #158295

github-actions · 2025-09-15T18:40:09Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

eee4017 · 2025-09-16T05:25:16Z

Closed with #158585

eee4017 requested review from eqy and syed-ahmed as code owners June 16, 2025 16:02

pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category labels Jun 16, 2025

pytorchbot added the open source label Jun 16, 2025

eqy added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 16, 2025

eqy requested a review from ngimel June 16, 2025 16:45

ngimel previously approved these changes Jun 16, 2025

View reviewed changes

eee4017 force-pushed the main-fix-symmetric-memory-api-call branch from 96693fe to ad432d9 Compare June 17, 2025 05:40

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Jun 17, 2025

Aidyn-A added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 17, 2025

cyyever previously approved these changes Jun 17, 2025

View reviewed changes

pytorchmergebot added the merging label Jun 17, 2025

pytorchmergebot added the Merged label Jun 17, 2025

pytorchmergebot closed this in cf90c9f Jun 17, 2025

pytorchmergebot removed the merging label Jun 17, 2025

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Jun 18, 2025

pytorchmergebot reopened this Jun 18, 2025

nWEIdia and others added 17 commits July 12, 2025 14:29

Revert "Refactoring driver api"

6444afb

This reverts commit f510f60.

Copy TE

c9ca6b8

Update

export get_symbol

5ddc3fa

Add TORCH_CHECK

78eba5e

Remove unnecessary mutex

3ae5a99

use get_symbol on all cuda drvier API

5bf1f4f

fix lint

faa9756

directly use cudaGetDriverEntryPoint

9bcc353

remove header

8caa1ff

fix log info; add version check

71346af

lint

85800dc

hardcode version

a9853af

add version to each api

7d8162f

move get_symbol into anonymous space

6f23d49

remove compile-time version check

3b73503

fix typo

17df78e

add back compile-time check and remove the assertion for 12.3 apis

61a30ad

eee4017 commented Jul 12, 2025

View reviewed changes

fix typo

dabccdb

huydhn approved these changes Jul 14, 2025

View reviewed changes

eee4017 mentioned this pull request Jul 14, 2025

[CUDA] Use runtime driver API for cuStreamWriteValue32 #158295

Closed

huydhn mentioned this pull request Jul 14, 2025

Add a periodic test for older NVIDIA driver #158300

Closed

pytorchbot mentioned this pull request Jul 17, 2025

[CUDA] Use runtime driver API for cuStreamWriteValue32 #158585

Merged

Conversation

eee4017 commented Jun 16, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/156097

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

ngimel left a comment

Choose a reason for hiding this comment

Uh oh!

eee4017 commented Jun 17, 2025

Uh oh!

pytorchmergebot commented Jun 17, 2025

Merge started

Uh oh!

atalman commented Jun 18, 2025

Uh oh!

pytorch-bot bot commented Jun 18, 2025

Uh oh!

atalman commented Jun 18, 2025

Uh oh!

pytorchmergebot commented Jun 18, 2025

Uh oh!

pytorchmergebot commented Jun 18, 2025

Uh oh!

eee4017 Jul 12, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn Jul 13, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn commented Jul 14, 2025

Uh oh!

pytorchmergebot commented Jul 14, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 14, 2025

Merge failed

Uh oh!

ngimel commented Jul 14, 2025

Uh oh!

eee4017 commented Jul 14, 2025

Uh oh!

github-actions bot commented Sep 15, 2025

Uh oh!

eee4017 commented Sep 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

eee4017 commented Jun 16, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jun 16, 2025 •

edited

Loading