Use gcc13 in Manylinux 2.28 images by atalman · Pull Request #152825 · pytorch/pytorch

atalman · 2025-05-05T14:31:24Z

Related to: #152426

pytorch-bot · 2025-05-05T14:31:28Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/152825

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 5 Pending, 1 Unrelated Failure

As of commit 6778233 with merge base 2f09e79 ():

NEW FAILURES - The following jobs have failed:

linux-binary-manywheel / manywheel-py3_10-rocm6_4-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_11-rocm6_4-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_12-rocm6_4-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_13-rocm6_4-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_13t-rocm6_4-build / build (gh)
ninja: build stopped: subcommand failed
linux-binary-manywheel / manywheel-py3_9-rocm6_4-build / build (gh)
ninja: build stopped: subcommand failed
linux-s390x-binary-manywheel / manywheel-py3_11-cpu-s390x-build / build (gh)
The process '/usr/bin/git' failed with exit code 128
Test check_binary / Test check_binary.sh for Linux CPU / linux-job (gh)
RuntimeError: Command docker exec -t 4f8b7fc9fb445664541eb979b8f1a3a79711980345c2c1a3962116d1f4373b61 /exec failed with exit code 1
Test check_binary / Test check_binary.sh for Linux CUDA / linux-job (gh)
RuntimeError: Command docker exec -t e53c32fea4c8a02df9f0c7d6e4373ff9d49e6468f668103dbb001699cb42b57d /exec failed with exit code 1

FLAKY - The following job failed but was likely due to flakiness present on trunk:

linux-binary-manywheel / manywheel-py3_13-cpu-test / test (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tinglvv · 2025-05-06T20:58:10Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-05-06T20:59:39Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-05-06T20:59:42Z

Successfully rebased fixes_gcc14 onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout fixes_gcc14 && git pull --rebase)

atalman · 2025-05-07T15:57:55Z

+        DOCKER_GPU_BUILD_ARG=" --build-arg DEVTOOLSET_VERSION=13 --build-arg NINJA_VERSION=1.12.1"
        MANY_LINUX_VERSION="2_28_aarch64"
        ;;
    manylinuxcxx11-abi-builder:cpu-cxx11-abi)


Do we want to do anything about this build ? Looks like its still using gcc=9

Looks like currently it is using gcc11? https://github.com/pytorch/pytorch/blob/main/.ci/docker/manywheel/Dockerfile_2_28_aarch64#L4
We may need a PR to test the docker image before merging 🤔

malfet

Sure, though I though that manylinux standard mandates minimum compler version, isn't it?

atalman · 2025-05-07T16:44:46Z

Hi @malfet as far as I know its not mandated, glibc version is. I believe you commended on the issue earlier: #114232 (comment)

atalman · 2025-05-08T20:02:49Z

@pytorchmergebot merge -f "all required tests and lint are green"

pytorchmergebot · 2025-05-08T20:04:31Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…el for GCC 12 and above (#158117) This PR disables `strict-aliasing` GCC C++ optimization flag on all AArch64 cpus for GCC versions 12 and above. Pull Request #152825 upgraded gcc version from 11 to 13 in manywheel which caused several segmentation faults in unit tests ( not visible in CI workflows because the jammy gcc version has not been updated yet ). We Identified the problem also exists in GCC12 hence the ` __GNUC__ >= 12` Fixes #157626 fixes these tests failures when pytorch is built in GCC12 and above ``` test_ops.py::TestCommonCPU::test_noncontiguous_samples_grid_sampler_2d_cpu_float32 Fatal Python error: Segmentation fault test_ops.py::TestCommonCPU::test_dtypes_grid_sampler_2d_cpu Fatal Python error: Segmentation fault test_ops.py::TestMathBitsCPU::test_neg_view_nn_functional_grid_sample_cpu_float64 free(): invalid next size (fast) test_ops.py::TestCompositeComplianceCPU::test_backward_grid_sampler_2d_cpu_float32 Fatal Python error: Segmentation fault test_ops.py::TestCommonCPU::test_dtypes_nn_functional_grid_sample_cpu Fatal Python error: Segmentation fault ``` Pull Request resolved: #158117 Approved by: https://github.com/malfet

This is needed because manylinux uses GCC-13 since #152825 As a result of the current compiler version mismatches, we've seen tests passing jammy-aarch64 pre-commit CI, but failing for wheels built in manylinux Related to: #166736 ghstack-source-id: a68de47 Pull-Request: #166849

This is needed because manylinux uses GCC-13 since #152825 As a result of the current compiler version mismatches, we've seen tests passing jammy-aarch64 pre-commit CI, but failing for wheels built in manylinux Related to: #166736 ghstack-source-id: 9d18d85 Pull-Request: #166849

This is needed because manylinux uses GCC-13 since #152825 As a result of the current compiler version mismatches, we've seen tests passing jammy-aarch64 pre-commit CI, but failing for wheels built in manylinux Related to: #166736 ghstack-source-id: 36be71d Pull-Request: #166849

This is needed because manylinux uses GCC-13 since #152825 As a result of the current compiler version mismatches, we've seen tests passing jammy-aarch64 pre-commit CI, but failing for wheels built in manylinux Related to: #166736 Pull Request resolved: #166849 Approved by: https://github.com/robert-hardwick, https://github.com/malfet, https://github.com/Skylion007, https://github.com/atalman

atalman requested a review from jeffdaily as a code owner May 5, 2025 14:31

pytorch-bot Bot added the topic: not user facing topic category label May 5, 2025

atalman added the ciflow/binaries Trigger all binary build and upload jobs on the PR label May 5, 2025

pytorchmergebot force-pushed the fixes_gcc14 branch from 0b4d42b to 8e180a5 Compare May 6, 2025 20:59

atalman changed the title ~~Use gcc14 in Manylinux 2.28 images~~ Use gcc13 in Manylinux 2.28 images May 6, 2025

atalman added 9 commits May 7, 2025 06:09

Use gcc14 in Manylinux 2.28 images

de8e7c7

gcc14

131776a

manylinux

a09aa80

install_devtoolchain

006c724

fix

642cd13

fix

1b24de7

use_gcc13

5435f68

devtoolset13

8e715b9

check_binary_and_more

bd187d4

atalman force-pushed the fixes_gcc14 branch from 8e180a5 to bd187d4 Compare May 7, 2025 13:09

atalman requested a review from a team as a code owner May 7, 2025 13:09

fix

c20dcb8

atalman commented May 7, 2025

View reviewed changes

malfet approved these changes May 7, 2025

View reviewed changes

atalman added 2 commits May 7, 2025 12:17

fix

8400400

Add rocm to check_binary

6778233

pytorchmergebot added the merging label May 8, 2025

pytorchmergebot closed this in 6ae7730 May 8, 2025

pytorchmergebot added Merged and removed merging labels May 8, 2025

dilililiwhy mentioned this pull request Jun 21, 2025

Looking for valid compiling option for extension based on torch-2.1.0+cpu.cxx11.abi #143780

Open

robert-hardwick mentioned this pull request Jul 4, 2025

Segmentation faults in test_ops.py tests with gcc13 on AArch64 (v1) #157626

Closed

atalman mentioned this pull request Jul 7, 2025

Migrate pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11 -> pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc13 #157748

Closed

robert-hardwick mentioned this pull request Jul 11, 2025

Fix AArch64 segfaults by disabling strict-aliasing in GridSamplerKernel for GCC 12 and above #158117

Closed

robert-hardwick mentioned this pull request Jul 16, 2025

[cherry-pick] Fix AArch64 segfaults by disabling strict-aliasing in GridSamplerKernel for GCC 12 and above #158445

Merged

fadara01 mentioned this pull request Nov 3, 2025

[ci][cpu] Update compiler to GCC-13 in jammy-aarch64 #166849

Closed

fadara01 mentioned this pull request Nov 3, 2025

[ci][cpu] Update AArch64 manylinux compiler to GCC14 #166876

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use gcc13 in Manylinux 2.28 images#152825

Use gcc13 in Manylinux 2.28 images#152825
atalman wants to merge 12 commits intopytorch:mainfrom
atalman:fixes_gcc14

atalman commented May 5, 2025 •

edited

Loading

Uh oh!

pytorch-bot Bot commented May 5, 2025 •

edited

Loading

Uh oh!

tinglvv commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Uh oh!

atalman May 7, 2025

Uh oh!

tinglvv May 7, 2025

Uh oh!

malfet left a comment

Uh oh!

atalman commented May 7, 2025

Uh oh!

atalman commented May 8, 2025

Uh oh!

pytorchmergebot commented May 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

atalman commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/152825

❌ 9 New Failures, 5 Pending, 1 Unrelated Failure

Uh oh!

tinglvv commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Uh oh!

atalman May 7, 2025

Choose a reason for hiding this comment

Uh oh!

tinglvv May 7, 2025

Choose a reason for hiding this comment

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

atalman commented May 7, 2025

Uh oh!

atalman commented May 8, 2025

Uh oh!

pytorchmergebot commented May 8, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

atalman commented May 5, 2025 •

edited

Loading

pytorch-bot Bot commented May 5, 2025 •

edited

Loading