[CD] Add pytorch xpu wheel build in nightly by chuanqi129 · Pull Request #129560 · pytorch/pytorch

chuanqi129 · 2024-06-26T09:09:23Z

Add pytorch xpu wheel build in nightly after the xpu build image enabling PR pytorch/builder#1879 merged. Link to #114850

pytorch-bot · 2024-06-26T09:09:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129560

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[PREEMPTIVE] CPU amd64 non-GPU instances migration to Linux AMZN2023

❌ 2 New Failures

As of commit daf563e with merge base 46c5266 ():

NEW FAILURES - The following jobs have failed:

linux-binary-manywheel / manywheel-py3_12-xpu-test (gh)
File was unable to be removed Error: EACCES: permission denied, unlink '/home/jenkins/actions-runner-6/_work/pytorch/pytorch/builder/test/smoke_test/__pycache__/smoke_test.cpython-39.pyc'
windows-binary-conda / conda-py3_9-cuda11_8-test (gh)
Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

atalman · 2024-06-27T20:25:33Z

.github/scripts/generate_binary_build_matrix.py


 ROCM_ARCHES = ["6.0", "6.1"]

+XPU_ARCHES = ["xpu"]


@chuanqi129 separate this PR into 2.

First build triton-xpu wheel. The changes in this file and generated-linux-binary-manywheel-nightly.yml are not required at this point

Second PR actually add manywheel build. At this point add changes to this file and generated-linux-binary-manywheel-nightly.yml

Thanks @atalman. Make sense, have split the triton-xpu wheel build in PR #129730.

chuanqi129 · 2024-07-03T10:06:23Z

The xpu 3.8 wheel build failed on numpy 1.15 installation should be fixed by PR pytorch/builder#1909

EikanWang · 2024-07-06T01:40:58Z

.github/workflows/generated-linux-binary-manywheel-nightly.yml

+      PACKAGE_TYPE: manywheel
+      # TODO: This is a legacy variable that we eventually want to get rid of in
+      #       favor of GPU_ARCH_VERSION
+      DESIRED_CUDA: xpu


What's the meaning of DESIRED_CUDA?

Same here. And the DESIRED_CUDA shared with all devices, including cpu, rocm etc.

EikanWang · 2024-07-06T01:51:47Z

.github/workflows/generated-linux-binary-manywheel-nightly.yml

      conda-pytorchbot-token-test: ${{ secrets.CONDA_PYTORCHBOT_TOKEN_TEST }}
    uses: ./.github/workflows/_binary-upload.yml

+  manywheel-py3_9-xpu-build:


Most of the code is copy-past. Compared with Py3.8, is the python version the only difference? I'm just wondering if we can reuse the code (https://yaml.org/spec/1.2.2/#rule-c-ns-anchor-property)

Hi @EikanWang , this workflow is generated by script .github/scripts/generate_ci_workflows.py based on template .github/templates/linux_binary_build_workflow.yml.j2

chuanqi129 · 2024-07-09T19:20:58Z

The build crash will be fixed by PR #130333

chuanqi129 · 2024-07-11T07:58:27Z

Hi @atalman , all PR changes related test jobs are passed, especially for the XPU manylinux wheel build and test jobs. But there are some unrelated test job failures caused by conda, could you please help to double check and review the PR again? Thanks

.circleci/scripts/binary_linux_test.sh

chuanqi129 · 2024-07-11T15:08:10Z

@pytorchbot merge

pytorchmergebot · 2024-07-11T15:09:52Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2024-07-11T15:47:08Z

@pytorchmergebot merge -f "lint is successful, previous run was fully green"

pytorchmergebot · 2024-07-11T15:47:27Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

pytorchmergebot · 2024-07-11T15:48:55Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Torch XPU nightly wheel build has been enabled with #pytorch/pytorch#129560 landed, add XPU to binary build generation. Works for pytorch/pytorch#114850 ```bash > python tools/scripts/generate_binary_build_matrix.py --with-xpu enable { "include": [ { "python_version": "3.8", "gpu_arch_type": "cpu", "gpu_arch_version": "", "desired_cuda": "cpu", "container_image": "pytorch/manylinux-builder:cpu", "package_type": "manywheel", "build_name": "manywheel-py3_8-cpu", "validation_runner": "linux.2xlarge", "installation": "pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu", "channel": "nightly", "upload_to_base_bucket": "no", "stable_version": "2.3.1", "use_split_build": false }, { "python_version": "3.8", "gpu_arch_type": "cuda", "gpu_arch_version": "11.8", "desired_cuda": "cu118", "container_image": "pytorch/manylinux-builder:cuda11.8", "package_type": "manywheel", "build_name": "manywheel-py3_8-cuda11_8", "validation_runner": "linux.g5.4xlarge.nvidia.gpu", "installation": "pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118", "channel": "nightly", "upload_to_base_bucket": "no", "stable_version": "2.3.1", "use_split_build": false }, { "python_version": "3.8", "gpu_arch_type": "cuda", "gpu_arch_version": "12.1", "desired_cuda": "cu121", "container_image": "pytorch/manylinux-builder:cuda12.1", "package_type": "manywheel", "build_name": "manywheel-py3_8-cuda12_1", "validation_runner": "linux.g5.4xlarge.nvidia.gpu", "installation": "pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121", "channel": "nightly", "upload_to_base_bucket": "no", "stable_version": "2.3.1", "use_split_build": false }, { "python_version": "3.8", "gpu_arch_type": "cuda", "gpu_arch_version": "12.4", "desired_cuda": "cu124", "container_image": "pytorch/manylinux-builder:cuda12.4", "package_type": "manywheel", "build_name": "manywheel-py3_8-cuda12_4", "validation_runner": "linux.g5.4xlarge.nvidia.gpu", "installation": "pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124", "channel": "nightly", "upload_to_base_bucket": "no", "stable_version": "2.3.1", "use_split_build": false }, { "python_version": "3.8", "gpu_arch_type": "rocm", "gpu_arch_version": "6.0", "desired_cuda": "rocm6.0", "container_image": "pytorch/manylinux-builder:rocm6.0", "package_type": "manywheel", "build_name": "manywheel-py3_8-rocm6_0", "validation_runner": "linux.2xlarge", "installation": "pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.0", "channel": "nightly", "upload_to_base_bucket": "no", "stable_version": "2.3.1", "use_split_build": false }, { "python_version": "3.8", "gpu_arch_type": "rocm", "gpu_arch_version": "6.1", "desired_cuda": "rocm6.1", "container_image": "pytorch/manylinux-builder:rocm6.1", "package_type": "manywheel", "build_name": "manywheel-py3_8-rocm6_1", "validation_runner": "linux.2xlarge", "installation": "pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.1", "channel": "nightly", "upload_to_base_bucket": "no", "stable_version": "2.3.1", "use_split_build": false }, { "python_version": "3.8", "gpu_arch_type": "xpu", "gpu_arch_version": "", "desired_cuda": "xpu", "container_image": "pytorch/manylinux2_28-builder:xpu", "package_type": "manywheel", "build_name": "manywheel-py3_8-xpu", "validation_runner": "linux.2xlarge", "installation": "pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/xpu", "channel": "nightly", "upload_to_base_bucket": "no", "stable_version": "2.3.1", "use_split_build": false } ] } ```

Add pytorch xpu wheel build in nightly after the xpu build image enabling PR pytorch/builder#1879 merged Pull Request resolved: pytorch#129560 Approved by: https://github.com/atalman

pytorch-bot bot added the topic: not user facing topic category label Jun 26, 2024

chuanqi129 added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Jun 26, 2024

pytorchbot added the open source label Jun 26, 2024

chuanqi129 force-pushed the xpu_nightly branch from 3c9cb76 to f3a6aca Compare June 26, 2024 15:57

chuanqi129 requested review from EikanWang and atalman June 26, 2024 18:24

chuanqi129 force-pushed the xpu_nightly branch 2 times, most recently from 70eec88 to 092324e Compare June 27, 2024 16:15

chuanqi129 mentioned this pull request Jun 27, 2024

[PyTorch Upstream] Triton xpu manylinux build failed in Pytorch XPU CD enabling intel/intel-xpu-backend-for-triton#1505

Closed

atalman reviewed Jun 27, 2024

View reviewed changes

chuanqi129 force-pushed the xpu_nightly branch from 092324e to 6e5ed64 Compare June 28, 2024 05:06

chuanqi129 force-pushed the xpu_nightly branch from 6e5ed64 to 087d40e Compare July 4, 2024 16:13

chuanqi129 marked this pull request as ready for review July 5, 2024 08:44

chuanqi129 requested a review from a team as a code owner July 5, 2024 08:44

chuanqi129 requested a review from atalman July 5, 2024 09:16

EikanWang reviewed Jul 6, 2024

View reviewed changes

zou3519 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jul 8, 2024

chuanqi129 force-pushed the xpu_nightly branch from 459e3ab to 4918bff Compare July 9, 2024 01:52

chuanqi129 added 5 commits July 10, 2024 14:07

[CD] Add pytorch xpu wheel build in nightly

aa584e5

[CD] Add triton xpu in binary populate env

25de34a

fix lint error

a5dfc6e

fix source xpu bundle env failure

3652b43

disable python 3.13 for xpu nightly build

afc8420

chuanqi129 force-pushed the xpu_nightly branch from 4918bff to afc8420 Compare July 10, 2024 21:14

atalman approved these changes Jul 11, 2024

View reviewed changes

atalman reviewed Jul 11, 2024

View reviewed changes

.circleci/scripts/binary_linux_test.sh Outdated Show resolved Hide resolved

Add issue link

daf563e

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 11, 2024

pytorchmergebot added the merging label Jul 11, 2024

pytorchmergebot added the Merged label Jul 11, 2024

pytorchmergebot closed this in ca023f7 Jul 11, 2024

pytorchmergebot removed the merging label Jul 11, 2024

henrylhtsang mentioned this pull request Jul 17, 2024

[aoti] Unskip some aot inductor tests #130973

Closed

chuanqi129 mentioned this pull request Jul 22, 2024

Add XPU to binary build generation pytorch/test-infra#5489

Merged

chuanqi129 mentioned this pull request Jul 28, 2024

Add xpu manylinux 2.28 wheel build pytorch/builder#1879

Merged

Conversation

chuanqi129 commented Jun 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129560

❗ 1 Active SEVs

❌ 2 New Failures

Uh oh!

atalman Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chuanqi129 Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

chuanqi129 commented Jul 3, 2024

Uh oh!

EikanWang Jul 6, 2024

Choose a reason for hiding this comment

Uh oh!

chuanqi129 Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

EikanWang Jul 6, 2024

Choose a reason for hiding this comment

Uh oh!

chuanqi129 Jul 9, 2024

Choose a reason for hiding this comment

Uh oh!

chuanqi129 commented Jul 9, 2024

Uh oh!

chuanqi129 commented Jul 11, 2024

Uh oh!

Uh oh!

chuanqi129 commented Jul 11, 2024

Uh oh!

pytorchmergebot commented Jul 11, 2024

Merge started

Uh oh!

atalman commented Jul 11, 2024

Uh oh!

pytorchmergebot commented Jul 11, 2024

Uh oh!

pytorchmergebot commented Jul 11, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

chuanqi129 commented Jun 26, 2024 •

edited

Loading

pytorch-bot bot commented Jun 26, 2024 •

edited

Loading

atalman Jun 27, 2024 •

edited

Loading