Skip to content

Fix torch.accelerator api abort when passing invaild device#143550

Closed
guangyey wants to merge 5 commits intogh/guangyey/110/basefrom
gh/guangyey/110/head
Closed

Fix torch.accelerator api abort when passing invaild device#143550
guangyey wants to merge 5 commits intogh/guangyey/110/basefrom
gh/guangyey/110/head

Conversation

@guangyey
Copy link
Collaborator

@guangyey guangyey commented Dec 19, 2024 •
edited by pytorch-bot bot
Loading

Stack from ghstack (oldest at bottom):

Motivation

Fix #143543

Solution

We should raise python exception instead of aborting...

Additional Context

without this PR:

>>> import torch
>>> torch.accelerator.current_stream(torch.accelerator.device_count())
terminate called after throwing an instance of 'c10::Error'
  what():  device is out of range, device is 2, total number of device is 2.
Exception raised from check_device_index at /home/dvrogozh/git/pytorch/pytorch/c10/xpu/XPUFunctions.h:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xac (0x7f30707eb95c in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xf3 (0x7f307078fc57 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10.so)
frame #2: <unknown function> + 0x19a3e (0x7f3070c2ba3e in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10_xpu.so)
frame #3: c10::xpu::getCurrentXPUStream(signed char) + 0x2f (0x7f3070c2c83f in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10_xpu.so)
frame #4: <unknown function> + 0x1ca35 (0x7f3070c2ea35 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10_xpu.so)
frame #5: <unknown function> + 0x653f15 (0x7f3083391f15 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x39e5f2 (0x7f30830dc5f2 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libtorch_python.so)
<omitting python frames>
frame #20: <unknown function> + 0x29d90 (0x7f308b19bd90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #21: __libc_start_main + 0x80 (0x7f308b19be40 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)

with this PR:

>>> import torch
>>> torch.accelerator.current_stream(torch.accelerator.device_count())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pt-gpu/4T-4652/guangyey/stock-pytorch/torch/accelerator/__init__.py", line 123, in current_stream
    return torch._C._accelerator_getStream(device_index)
RuntimeError: The device index is out of range. It must be in [0, 2), but got 2.

cc @albanD @EikanWang

@pytorch-bot
Copy link

pytorch-bot bot commented Dec 19, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143550

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 2403bb2 with merge base 19d8bba (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@guangyey guangyey changed the title Fix torch.accelerator api abort when passing invaild device [WIP] Fix torch.accelerator api abort when passing invaild device Dec 19, 2024
@guangyey guangyey added the topic: improvements topic category label Dec 19, 2024
@pytorch-bot pytorch-bot bot added ciflow/mps Run MPS tests (subset of trunk) release notes: mps Release notes category labels Dec 19, 2024
@guangyey guangyey added ciflow/xpu Run XPU CI tasks ciflow/trunk Trigger trunk jobs on your pull request ciflow/rocm Trigger "default" config CI on ROCm release notes: xpu release notes category labels Dec 19, 2024
@guangyey guangyey mentioned this pull request Dec 19, 2024
Closed
@EikanWang
Copy link
Collaborator

EikanWang commented Dec 19, 2024

@guangyey , pls. help refine the error message a little bit as it is user-facing.

guangyey added a commit that referenced this pull request Dec 19, 2024
@guangyey
Fix torch.accelerator api abort when passing invaild device
2d0d447 
ghstack-source-id: b27b3e6
Pull Request resolved: #143550
@guangyey
Copy link
Collaborator Author

guangyey commented Dec 19, 2024

@guangyey , pls. help refine the error message a little bit as it is user-facing.

Updated in 2403bb2

@guangyey guangyey requested a review from albanD December 19, 2024 06:20
EikanWang
EikanWang approved these changes Dec 19, 2024
View reviewed changes
Copy link
Collaborator

@EikanWang EikanWang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@guangyey
Update
5dd62df 
[ghstack-poisoned]
@guangyey
Update
8eb7df7 
[ghstack-poisoned]
@guangyey
Update
75d5f32 
[ghstack-poisoned]
@guangyey
Update
046ad8f 
[ghstack-poisoned]
@guangyey
Update
2403bb2 
[ghstack-poisoned]
dvrogozh
dvrogozh approved these changes Dec 19, 2024
View reviewed changes
Copy link
Contributor

@dvrogozh dvrogozh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me to address reported #143543.

@guangyey guangyey changed the title [WIP] Fix torch.accelerator api abort when passing invaild device Fix torch.accelerator api abort when passing invaild device Dec 20, 2024
@guangyey guangyey added the module: accelerator Issues related to the shared accelerator API label Dec 20, 2024
albanD
albanD approved these changes Dec 20, 2024
View reviewed changes
Copy link
Collaborator

@albanD albanD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops !

@guangyey
Copy link
Collaborator Author

guangyey commented Dec 23, 2024

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Dec 23, 2024

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the gh/guangyey/110/head branch January 23, 2025 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@albanD albanD albanD approved these changes

@EikanWang EikanWang EikanWang approved these changes

@eqy eqy Awaiting requested review from eqy

@syed-ahmed syed-ahmed Awaiting requested review from syed-ahmed

@gujinghui gujinghui Awaiting requested review from gujinghui

@jeffdaily jeffdaily Awaiting requested review from jeffdaily

@jithunnair-amd jithunnair-amd Awaiting requested review from jithunnair-amd

@kulinseth kulinseth Awaiting requested review from kulinseth

@malfet malfet Awaiting requested review from malfet

+1 more reviewer

@dvrogozh dvrogozh dvrogozh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ciflow/mps Run MPS tests (subset of trunk) ciflow/rocm Trigger "default" config CI on ROCm ciflow/trunk Trigger trunk jobs on your pull request ciflow/xpu Run XPU CI tasks Merged module: accelerator Issues related to the shared accelerator API open source release notes: mps Release notes category release notes: xpu release notes category topic: improvements topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@guangyey @EikanWang @pytorchmergebot @albanD @dvrogozh @pytorchbot