Skip to content

Don't terminate the whole process when Compile fails#6707

Merged
will-cromar merged 1 commit intomasterfrom
wcromar/compile-exceptions
Mar 11, 2024
Merged

Don't terminate the whole process when Compile fails#6707
will-cromar merged 1 commit intomasterfrom
wcromar/compile-exceptions

Conversation

@will-cromar
Copy link
Copy Markdown
Collaborator

@will-cromar will-cromar commented Mar 9, 2024

See #6700

The error won't kill your whole notebook anymore:

>>> import torch_xla as xla
>>> import torch
>>> torch.dot(torch.tensor([2, 3], device=xla.device()), torch.tensor([2, 1], device=xla.device()))
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:libtpu.so and TPU device found. Setting PJRT_DEVICE=TPU.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1709942790.379654  501354 pjrt_api.cc:100] GetPjrtApi was found for tpu at /usr/local/lib/python3.8/site-packages/libtpu/libtpu.so
I0000 00:00:1709942790.379741  501354 pjrt_api.cc:79] PJRT_Api is set for device type tpu
I0000 00:00:1709942790.379751  501354 pjrt_api.cc:146] The PJRT plugin has PJRT API version 0.40. The framework PJRT API version is 0.40.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/torch/_tensor.py", line 464, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/usr/local/lib/python3.8/site-packages/torch/_tensor_str.py", line 697, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/usr/local/lib/python3.8/site-packages/torch/_tensor_str.py", line 432, in _str_intern
    self = self.to("cpu")
RuntimeError: Bad StatusOr access: UNIMPLEMENTED: While rewriting computation to not contain X64 element types, XLA encountered an HLO for which this rewriting is not implemented: %dot.3 = s64[] dot(s64[2]{0} %p1.2, s64[2]{0} %p0.1), lhs_contracting_dims={0}, rhs_contracting_dims={0}

@will-cromar will-cromar added runtime usability Bugs/features related to improving the usability of PyTorch/XLA labels Mar 9, 2024
@will-cromar will-cromar requested a review from JackCaoG March 9, 2024 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

runtime usability Bugs/features related to improving the usability of PyTorch/XLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants