Use Manylinux2_28 for wheel builds by atalman · Pull Request #138732 · pytorch/pytorch

atalman · 2024-10-23T19:12:36Z

Fixes #123649
Use Manylinux 2_28 Docker builds for PyTorch Nightly builds

This moves the wheels to a Docker image that uses : quay.io/pypa/manylinux_2_28_x86_64 as a base rather then centos:7 which is EOL on June 30, 2024.

Information:
https://github.com/pypa/manylinux#manylinux_2_28-almalinux-8-based

manylinux_2_28 (AlmaLinux 8 based)
Toolchain: GCC 13
Built wheels are also expected to be compatible with other distros using glibc 2.28 or later, including:
Debian 10+
Ubuntu 18.10+
Fedora 29+
CentOS/RHEL 8+

This migration should enable us to migrate to latest CUDNN version, and land this PR: #137978

pytorch-bot · 2024-10-23T19:12:43Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138732

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[DomainsOnly] Jobs fail with GLIBC version not found

❌ 10 New Failures, 1 Unrelated Failure

As of commit 20e5128 with merge base c1bf714 ():

NEW FAILURES - The following jobs have failed:

linux-aarch64-binary-manywheel / manywheel-py3_10-cpu-aarch64-build / build (gh)
linux-aarch64-binary-manywheel / manywheel-py3_10-cuda-aarch64-build / build (gh)
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
linux-aarch64-binary-manywheel / manywheel-py3_11-cpu-aarch64-build / build (gh)
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
linux-aarch64-binary-manywheel / manywheel-py3_11-cuda-aarch64-build / build (gh)
linux-aarch64-binary-manywheel / manywheel-py3_12-cpu-aarch64-build / build (gh)
linux-aarch64-binary-manywheel / manywheel-py3_12-cuda-aarch64-build / build (gh)
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.
linux-aarch64-binary-manywheel / manywheel-py3_13-cpu-aarch64-build / build (gh)
linux-aarch64-binary-manywheel / manywheel-py3_13-cuda-aarch64-build / build (gh)
linux-aarch64-binary-manywheel / manywheel-py3_9-cpu-aarch64-build / build (gh)
linux-aarch64-binary-manywheel / manywheel-py3_9-cuda-aarch64-build / build (gh)
docker: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

windows-binary-wheel / wheel-py3_10-xpu-build (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Enabling Manywheel builds here: #138732 During the build I observe the failure with cuda jobs: ``` -- Compiler does not support SVE extension. Will not build perfkernels. -- Found CUDA: /usr/local/cuda (found version "11.8") -- The CUDA compiler identification is unknown CMake Error at cmake/public/cuda.cmake:47 (enable_language): No CMAKE_CUDA_COMPILER could be found. Tell CMake where to find the compiler by setting either the environment variable "CUDACXX" or the CMake cache entry CMAKE_CUDA_COMPILER to the full path to the compiler, or to the compiler name if it is in the PATH. Call Stack (most recent call first): cmake/Dependencies.cmake:44 (include) CMakeLists.txt:851 (include) ``` While correct sequence suppose to be: ``` -- Found CUDA: /usr/local/cuda (found version "11.8") -- The CUDA compiler identification is NVIDIA 11.8.89 -- Detecting CUDA compiler ABI info -- Detecting CUDA compiler ABI info - done -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped -- Detecting CUDA compile features -- Detecting CUDA compile features - done -- Found CUDAToolkit: /usr/local/cuda/include (found version "11.8.89") ``` Issue found to be missing PATH setting in 2_28 Docker file. This section exist in CentOS Docker file here: https://github.com/pytorch/pytorch/blob/main/.ci/docker/manywheel/Dockerfile#L174-L175 (Please Note these Docker images are not used yet. The #138732 should enable using these images) Pull Request resolved: #139631 Approved by: https://github.com/malfet, https://github.com/huydhn

… builds (#139636) Install setuptools and wheel dependencies for cp312, cp313, cp313t on Manylinux 2_28 images. This should resolve ``` ModuleNotFoundError: No module named 'setuptools' ``` On PR: #138732 This issue was addressed on XPU images already. We should apply the same fix for the rest of the images instead of keeping it XPU specific. Pull Request resolved: #139636 Approved by: https://github.com/huydhn, https://github.com/chuanqi129

malfet

LGTM

malfet · 2024-11-05T17:19:37Z

@pytorchbot merge -f "Binary builds looks fine"

pytorchmergebot · 2024-11-05T17:21:14Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Followup after pytorch/pytorch#138732

atalman · 2024-11-06T19:11:25Z

@pytorchmergebot revert -c nosignal -m "Reverting for now will be relanding"

pytorchmergebot · 2024-11-06T19:12:48Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit 5860c8e. Reverted #138732 on behalf of https://github.com/atalman due to Reverting for now will be relanding ([comment](#138732 (comment)))

pytorchmergebot · 2024-11-06T19:12:55Z

@atalman your PR has been successfully reverted.

jondea · 2024-11-08T10:07:12Z

I think #137696 is necessary when building for AArch64 using Manylinux2_28

atalman · 2024-11-14T00:23:58Z

@pytorchmergebot merge -f "lint is green"

pytorchmergebot · 2024-11-14T00:25:36Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Accompanying PR from Pytorch: pytorch/pytorch#138732

Use https://hub.docker.com/r/pytorch/manylinux2_28_aarch64-builder/tags Similar to #138732 Pull Request resolved: #140743 Approved by: https://github.com/malfet

Accompanying PR from Pytorch: pytorch/pytorch#138732

Fixes pytorch#123649 Use Manylinux 2_28 Docker builds for PyTorch Nightly builds This moves the wheels to a Docker image that uses : ``quay.io/pypa/manylinux_2_28_x86_64`` as a base rather then ``centos:7`` which is EOL on June 30, 2024. Information: https://github.com/pypa/manylinux#manylinux_2_28-almalinux-8-based manylinux_2_28 (AlmaLinux 8 based) Toolchain: GCC 13 Built wheels are also expected to be compatible with other distros using glibc 2.28 or later, including: Debian 10+ Ubuntu 18.10+ Fedora 29+ CentOS/RHEL 8+ This migration should enable us to migrate to latest CUDNN version, and land this PR: pytorch#137978 Pull Request resolved: pytorch#138732 Approved by: https://github.com/Skylion007, https://github.com/malfet

This reverts commit 5860c8e. Reverted pytorch#138732 on behalf of https://github.com/atalman due to Reverting for now will be relanding ([comment](pytorch#138732 (comment)))

Fixes pytorch#123649 Use Manylinux 2_28 Docker builds for PyTorch Nightly builds This moves the wheels to a Docker image that uses : ``quay.io/pypa/manylinux_2_28_x86_64`` as a base rather then ``centos:7`` which is EOL on June 30, 2024. Information: https://github.com/pypa/manylinux#manylinux_2_28-almalinux-8-based manylinux_2_28 (AlmaLinux 8 based) Toolchain: GCC 13 Built wheels are also expected to be compatible with other distros using glibc 2.28 or later, including: Debian 10+ Ubuntu 18.10+ Fedora 29+ CentOS/RHEL 8+ This migration should enable us to migrate to latest CUDNN version, and land this PR: pytorch#137978 Pull Request resolved: pytorch#138732 Approved by: https://github.com/Skylion007, https://github.com/malfet, https://github.com/huydhn

Use https://hub.docker.com/r/pytorch/manylinux2_28_aarch64-builder/tags Similar to pytorch#138732 Pull Request resolved: pytorch#140743 Approved by: https://github.com/malfet

atalman requested a review from a team as a code owner October 23, 2024 19:12

pytorch-bot bot added the topic: not user facing topic category label Oct 23, 2024

atalman added the ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR label Oct 23, 2024

This was referenced Nov 4, 2024

Fix cuda Manylinux 2_28 docker images PATH setting #139631

Closed

Add setuptools and wheel to cp312, cp313 and cp313t for Manylinux2_28 builds #139636

Closed

atalman force-pushed the test_manywheel2_28_images branch from 9ed8215 to 91efbbf Compare November 5, 2024 12:43

Skylion007 approved these changes Nov 5, 2024

View reviewed changes

malfet approved these changes Nov 5, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 5, 2024

pytorchmergebot added the merging label Nov 5, 2024

pytorchmergebot added the Merged label Nov 5, 2024

pytorchmergebot closed this in 5860c8e Nov 5, 2024

pytorchmergebot removed the merging label Nov 5, 2024

atalman mentioned this pull request Nov 6, 2024

Use Manylinux2_28 for wheel builds pytorch/test-infra#5866

Merged

atalman added a commit to pytorch/test-infra that referenced this pull request Nov 6, 2024

Use Manylinux2_28 for wheel builds (#5866)

bc5a826

Followup after pytorch/pytorch#138732

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Nov 6, 2024

pytorchmergebot reopened this Nov 6, 2024

fix

20e5128

atalman force-pushed the test_manywheel2_28_images branch from e74c907 to 20e5128 Compare November 13, 2024 22:02

huydhn approved these changes Nov 14, 2024

View reviewed changes

pytorchmergebot added the merging label Nov 14, 2024

pytorchmergebot closed this in 70acf02 Nov 14, 2024

pytorchmergebot removed the merging label Nov 14, 2024

atalman mentioned this pull request Nov 14, 2024

[DomainsOnly] Jobs fail with GLIBC version not found #140631

Closed

atalman added a commit to pytorch/test-infra that referenced this pull request Nov 14, 2024

Use Manylinux2_28 for wheel builds v2 (#5909)

f5a0474

Accompanying PR from Pytorch: pytorch/pytorch#138732

This was referenced Nov 14, 2024

[RFC] PyTorch next wheel build platform: manylinux-2.28 #123649

Closed

Use Manylinux 2.28 for aarch64 CPU workflows #140743

Closed

NicolasHug pushed a commit to NicolasHug/test-infra that referenced this pull request Nov 18, 2024

Use Manylinux2_28 for wheel builds v2 (pytorch#5909)

73d3021

Accompanying PR from Pytorch: pytorch/pytorch#138732

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Manylinux2_28 for wheel builds#138732

Use Manylinux2_28 for wheel builds#138732
atalman wants to merge 5 commits intopytorch:mainfrom
atalman:test_manywheel2_28_images

atalman commented Oct 23, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Oct 23, 2024 •

edited

Loading

Uh oh!

malfet left a comment

Uh oh!

malfet commented Nov 5, 2024

Uh oh!

pytorchmergebot commented Nov 5, 2024

Uh oh!

atalman commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Uh oh!

jondea commented Nov 8, 2024

Uh oh!

atalman commented Nov 14, 2024

Uh oh!

pytorchmergebot commented Nov 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

atalman commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138732

❗ 1 Active SEVs

❌ 10 New Failures, 1 Unrelated Failure

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet commented Nov 5, 2024

Uh oh!

pytorchmergebot commented Nov 5, 2024

Merge started

Uh oh!

atalman commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Uh oh!

jondea commented Nov 8, 2024

Uh oh!

atalman commented Nov 14, 2024

Uh oh!

pytorchmergebot commented Nov 14, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

atalman commented Oct 23, 2024 •

edited

Loading

pytorch-bot bot commented Oct 23, 2024 •

edited

Loading