Skip to content

[ci][cpu] Update AArch64 manylinux compiler to GCC14#166876

Open
fadara01 wants to merge 32 commits intogh/fadara01/9/basefrom
gh/fadara01/9/head
Open

[ci][cpu] Update AArch64 manylinux compiler to GCC14#166876
fadara01 wants to merge 32 commits intogh/fadara01/9/basefrom
gh/fadara01/9/head

Conversation

@fadara01
Copy link
Copy Markdown
Collaborator

@fadara01 fadara01 commented Nov 3, 2025

Stack from ghstack (oldest at bottom):

Realted to: #152426
Fixes: #168288

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jerryzh168 @aditew01 @gujinghui @PenghuiCheng @jianyuh @min-jean-cho @yanbing-j @Guobing-Chen @Xia-Weiwen @snadampal @milpuz01 @nikhil-arm @nWEIdia

[ghstack-poisoned]
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot Bot commented Nov 3, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166876

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 18 New Failures, 3 Unrelated Failures

As of commit c4c0643 with merge base b33d536 (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot Bot added the topic: not user facing topic category label Nov 3, 2025
fadara01 added a commit that referenced this pull request Nov 3, 2025
@fadara01 fadara01 added module: cpu CPU specific problem (e.g., perf, algorithm) module: arm Related to ARM architectures builds of PyTorch. Includes Apple M1 ciflow/linux-aarch64 linux aarch64 CI workflow labels Nov 3, 2025
@fadara01
Copy link
Copy Markdown
Collaborator Author

fadara01 commented Nov 3, 2025

This is blocked on fixing the failures from the xnnpack version update in #166873

@fadara01
Copy link
Copy Markdown
Collaborator Author

fadara01 commented Nov 3, 2025

In a similar PR (#152825) which updated AArch64 manylinux to GCC13, I saw some ABI compatibility checks in .ci/pytorch/check_binary.sh which no longer exists. Are these no longer needed or have they moved to another place? I couldn't find any references to that in pytorch.

cc: @atalman

[ghstack-poisoned]
@fadara01 fadara01 requested review from a team and jeffdaily as code owners November 5, 2025 17:49
fadara01 added a commit that referenced this pull request Nov 5, 2025
[ghstack-poisoned]
fadara01 added a commit that referenced this pull request Nov 5, 2025
[ghstack-poisoned]
fadara01 added a commit that referenced this pull request Nov 5, 2025
@robert-hardwick
Copy link
Copy Markdown
Collaborator

robert-hardwick commented Nov 20, 2025

I meant to raise this issue last week but we have a couple of unit tests with internal compiler error that is fixed in GCC14
#168288

@malfet you should be able to use this again, as per description ( #166687 ) after we are on GCC14. Not familiar with the workaround you landed but thought it should mention.

AT_DISPATCH_ALL_TYPES_AND(kHalf,

[ghstack-poisoned]
fadara01 added a commit that referenced this pull request Nov 26, 2025
@fadara01 fadara01 added ciflow/linux-aarch64 linux aarch64 CI workflow and removed ciflow/linux-aarch64 linux aarch64 CI workflow labels Nov 29, 2025
[ghstack-poisoned]
fadara01 added a commit that referenced this pull request Dec 3, 2025
Realted to: #152426
Fixes: #168288

- Updates manylinux compiler to gcc14 as per manylinux requirements: https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux_2_28-almalinux-8-based
- Updates ubuntu for non-manylinux builds from jammy to noble, and updates gcc for these builds to gcc14

ghstack-source-id: e931b48
Pull-Request: #166876
@fadara01
Copy link
Copy Markdown
Collaborator Author

fadara01 commented Dec 3, 2025

We still get similar failures if we disable USE_PRIORITIZED_TEXT_FOR_LD

Comment thread .ci/docker/manywheel/build.sh
@robert-hardwick
Copy link
Copy Markdown
Collaborator

@pytorchbot rebase

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict gh/fadara01/9/orig returned non-zero exit code 1

warning: skipped previously applied commit 38f18d59ef3
hint: use --reapply-cherry-picks to include skipped commits
hint: Disable this message with "git config set advice.skippedCherryPicks false"
Rebasing (1/1)
Auto-merging .ci/docker/build.sh
Auto-merging .ci/docker/manywheel/build.sh
Auto-merging .github/workflows/docker-builds.yml
Auto-merging .github/workflows/inductor-perf-test-nightly-aarch64.yml
CONFLICT (content): Merge conflict in .github/workflows/inductor-perf-test-nightly-aarch64.yml
Auto-merging .github/workflows/linux-aarch64.yml
CONFLICT (content): Merge conflict in .github/workflows/linux-aarch64.yml
Auto-merging .github/workflows/operator_benchmark.yml
CONFLICT (content): Merge conflict in .github/workflows/operator_benchmark.yml
error: could not apply 6cd4bb5cac6... [ci][cpu] Update AArch64 manylinux compiler to GCC14
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Could not apply 6cd4bb5cac6... # [ci][cpu] Update AArch64 manylinux compiler to GCC14

Raised by https://github.com/pytorch/pytorch/actions/runs/20228907567

@robert-hardwick
Copy link
Copy Markdown
Collaborator

Hi @fadara01 , can you rebase this locally and resync? Theres some merge conflicts so i don't think this will merge to main.

Also, i've done some debugging and it look like it only happens with CI because BUILD_STATIC_RUNTIME_BENCHMARK=ON for CI and not for manywheel, which creates a huge static executable which triggers the linking error.

So can you try commenting out these lines to confirm yourself
https://github.com/pytorch/pytorch/blob/main/.ci/pytorch/build.sh#L240 ?

I'm looking into the linker problem itself, looks like a bug with veneers in ld to me as it's not happening with gold linker.

[ghstack-poisoned]
fadara01 added a commit that referenced this pull request Dec 15, 2025
Realted to: #152426
Fixes: #168288

- Updates manylinux compiler to gcc14 as per manylinux requirements: https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux_2_28-almalinux-8-based
- Updates ubuntu for non-manylinux builds from jammy to noble, and updates gcc for these builds to gcc14

ghstack-source-id: d5fbe62
Pull-Request: #166876
[ghstack-poisoned]
fadara01 added a commit that referenced this pull request Dec 15, 2025
Realted to: #152426
Fixes: #168288

- Updates manylinux compiler to gcc14 as per manylinux requirements: https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux_2_28-almalinux-8-based
- Updates ubuntu for non-manylinux builds from jammy to noble, and updates gcc for these builds to gcc14

ghstack-source-id: 64799a5
Pull-Request: #166876
@robert-hardwick
Copy link
Copy Markdown
Collaborator

@pytorchbot rebase -r viable/strict

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot Bot commented Jan 8, 2026

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: unrecognized arguments: -r viable/strict

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick} ...

Try @pytorchbot --help for more info.

@robert-hardwick
Copy link
Copy Markdown
Collaborator

@pytorchbot rebase

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[ghstack-poisoned]
pytorchmergebot pushed a commit that referenced this pull request Jan 8, 2026
Realted to: #152426
Fixes: #168288

- Updates manylinux compiler to gcc14 as per manylinux requirements: https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux_2_28-almalinux-8-based
- Updates ubuntu for non-manylinux builds from jammy to noble, and updates gcc for these builds to gcc14

ghstack-source-id: 542569b
Pull-Request: #166876
@pytorchmergebot
Copy link
Copy Markdown
Collaborator

Successfully rebased gh/fadara01/9/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/166876)

@fadara01
Copy link
Copy Markdown
Collaborator Author

fadara01 commented Jan 9, 2026

The R_AARCH64_CALL26 errors with GCC14 and Noble are fixed by a new binutils patch release.
Shoutout to @robert-hardwick for verifying this!

@robert-hardwick
Copy link
Copy Markdown
Collaborator

The R_AARCH64_CALL26 errors with GCC14 and Noble are fixed by a new binutils patch release. Shoutout to @robert-hardwick for verifying this!

I understand the fix will make it into the next binutils ( 2.46 ) release.

@github-actions
Copy link
Copy Markdown
Contributor

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[ghstack-poisoned]
fadara01 added a commit that referenced this pull request Mar 20, 2026
Realted to: #152426
Fixes: #168288

- Updates manylinux compiler to gcc14 as per manylinux requirements: https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux_2_28-almalinux-8-based
- Updates ubuntu for non-manylinux builds from jammy to noble, and updates gcc for these builds to gcc14

ghstack-source-id: 3fe3633
Pull-Request: #166876
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/linux-aarch64 linux aarch64 CI workflow ciflow/trunk Trigger trunk jobs on your pull request module: arm Related to ARM architectures builds of PyTorch. Includes Apple M1 module: cpu CPU specific problem (e.g., perf, algorithm) module: mkldnn Related to Intel IDEEP or oneDNN (a.k.a. mkldnn) integration open source topic: not user facing topic category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants