Skip to content

[RelEng] Define BUILD_BUNDLE_PTXAS (#119750)#119988

Merged
atalman merged 1 commit intopytorch:release/2.2from
atalman:cherry_pick_ptxas
Feb 15, 2024
Merged

[RelEng] Define BUILD_BUNDLE_PTXAS (#119750)#119988
atalman merged 1 commit intopytorch:release/2.2from
atalman:cherry_pick_ptxas

Conversation

@atalman
Copy link
Contributor

@atalman atalman commented Feb 15, 2024

That would bundle PTXAS into a bin folder

When compiling for Triton, define TRITION_PTXAS_PATH if ptxas is bundled with PyTorch Needed to make PyTorch compiled against CUDA-11.8 usable with 11.8 driver, as Triton is bundled with latest (CUDA-12.3 at time of PyTorch-2.2 release) ptxas

Needs pytorch/builder@5c814e2 to produce valid binary builds

Test plan:

  • Create dummy ptxas in torch/bin folder and observe torch.compile fail with backtrace in Triton module.
  • Run following script (to be added to binary tests ) against CUDA-11.8 wheel:
import torch
import triton

@torch.compile
def foo(x: torch.Tensor) -> torch.Tensor:
  return torch.sin(x) + torch.cos(x)

x=torch.rand(3, 3, device="cuda")
print(foo(x))
# And check that CUDA versions match
cuda_version = torch.version.cuda
ptxas_version = triton.backends.nvidia.compiler.get_ptxas_version().decode("ascii")
assert cuda_version in ptxas_version, f"CUDA version mismatch: torch build with {cuda_version}, but Triton uses ptxs {ptxas_version}"

Fixes #119054

Pull Request resolved: #119750
Approved by: https://github.com/jansel, https://github.com/atalman

Fixes #ISSUE_NUMBER

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

That would bundle PTXAS into a `bin` folder

When compiling for Triton, define `TRITION_PTXAS_PATH` if `ptxas` is bundled with PyTorch Needed to make PyTorch compiled against CUDA-11.8 usable with 11.8 driver, as Triton is bundled with latest (CUDA-12.3 at time of PyTorch-2.2 release) ptxas

Needs pytorch/builder@5c814e2 to produce valid binary builds

Test plan:
- Create dummy ptxas in `torch/bin` folder and observe `torch.compile` fail with backtrace in Triton module.
- Run following script (to be added to binary tests ) against CUDA-11.8 wheel:
```python
import torch
import triton

@torch.compile
def foo(x: torch.Tensor) -> torch.Tensor:
  return torch.sin(x) + torch.cos(x)

x=torch.rand(3, 3, device="cuda")
print(foo(x))
# And check that CUDA versions match
cuda_version = torch.version.cuda
ptxas_version = triton.backends.nvidia.compiler.get_ptxas_version().decode("ascii")
assert cuda_version in ptxas_version, f"CUDA version mismatch: torch build with {cuda_version}, but Triton uses ptxs {ptxas_version}"
```

Fixes pytorch#119054

Pull Request resolved: pytorch#119750
Approved by: https://github.com/jansel, https://github.com/atalman
@pytorch-bot
Copy link

pytorch-bot bot commented Feb 15, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/119988

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit b86d77c with merge base a8bd593 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@atalman atalman merged commit 6c8c5ad into pytorch:release/2.2 Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants