[CI] Use sccache-0.8.2 for CUDA builds by malfet · Pull Request #140614 · pytorch/pytorch

malfet · 2024-11-13T21:31:03Z

Instead of an ancient prebuilt binary

This is a followup from #121323
For some reason, newer sccache does not work when gcc is invoked with -E option, so one have to special-case -E case in /opt/ccache/bin/gcc wrapper, which had to be special cased to work with nvcc by checking whether -E is passed not only as first or second, but as 3rd argument as well(to be followed up by a generic #142813 ), i.e. to generate following wrapper:

#!/bin/sh

if [ "$1" = "-E" ] || [ "$2" = "-E" ] || [ "$3" = "-E" ]; then
  exec /usr/bin/gcc "$@"
elif [ $(env -u LD_PRELOAD ps -p $PPID -o comm=) != sccache ]; then
  exec sccache /usr/bin/gcc "$@"
else
  exec /usr/bin/gcc "$@"
fi

Without it sccache nvcc hello.cu failed with no-descriptive

    sccache: error: failed to execute compile
    sccache: caused by: Compiler not supported: ""

pytorch-bot · 2024-11-13T21:31:06Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140614

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit 8f911c4 with merge base e52a534 ():

NEW FAILURES - The following jobs have failed:

Build manywheel docker images for s390x / build-docker-cpu-s390x (gh)
no space left on device
pull / linux-jammy-py3-clang12-executorch / build (gh)
curl: (22) The requested URL returned error:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3-clang12-executorch) (gh) (similar failure)
curl: (22) The requested URL returned error:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

malfet · 2024-11-18T21:39:10Z

@pytorchbot merge -r

pytorchmergebot · 2024-11-18T21:40:39Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-11-18T21:40:42Z

Successfully rebased malfet-patch-28 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout malfet-patch-28 && git pull --rebase)

pytorchmergebot · 2024-11-18T21:41:56Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-11-18T22:24:49Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / libtorch-linux-focal-cuda12.4-py3.7-gcc9-debug / build

Details for Dev Infra team

Raised by workflow job

malfet · 2024-11-20T23:17:28Z

@pytorchbot rebase

pytorchmergebot · 2024-11-20T23:19:04Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-11-20T23:19:08Z

Successfully rebased malfet-patch-28 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout malfet-patch-28 && git pull --rebase)

removes sccache from bazel builds. Will move bazel builds to periodic if build succeed CUDA bazel test succeeded, moving to periodic Pull Request resolved: #142241 Approved by: https://github.com/malfet

wdvr · 2024-12-07T12:06:09Z

@pytorchbot rebase

pytorchmergebot · 2024-12-07T12:07:39Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-12-07T12:07:42Z

Successfully rebased malfet-patch-28 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout malfet-patch-28 && git pull --rebase)

pytorchmergebot · 2024-12-10T01:35:11Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-12-10T01:35:14Z

Successfully rebased malfet-patch-28 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout malfet-patch-28 && git pull --rebase)

malfet · 2024-12-11T00:32:49Z

@pytorchbot merge -f "All builds are green"

pytorchmergebot · 2024-12-11T00:34:24Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Follow up of TODO in #140614 It was found experimentally, that for one GPU architecture, `sccache` passes `-E` as 1st, 2nd or 3rd argument, but it's much better to do this if `-E` is passed as any argument No need to worry about exit or elif chains, as `exec` aborts script execution Pull Request resolved: #142813 Approved by: https://github.com/malfet Co-authored-by: Nikita Shulga <nshulga@meta.com>

ZainRizvi · 2024-12-11T18:15:50Z

Does this mean that for ggc -E commands we do not use sccache at all @malfet ? What kind of a perf hit are we expecting when building the jobs?

ZainRizvi · 2024-12-11T18:37:55Z

@pytorchmergebot revert -c nosignal -m "Some build jobs are timing out, I'm guessing because they aren't using the cache. GH job link HUD commit link"

pytorchmergebot · 2024-12-11T18:39:22Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2024-12-11T18:39:26Z

Reverting PR 140614 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit b94a20641459b295573617b97916a09b284dccfd returned non-zero exit code 1

Auto-merging .ci/docker/common/install_cache.sh
CONFLICT (content): Merge conflict in .ci/docker/common/install_cache.sh
error: could not revert b94a2064145... [CI] Use sccache-0.8.2 for CUDA builds (#140614)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

wdvr · 2024-12-11T18:46:19Z

@ZainRizvi yes, but that was already like that, sccache doesn't play nicely with the preprocessor.

malfet · 2024-12-11T18:50:32Z

@pytorchmergebot revert -c nosignal -m "Some build jobs are timing out, I'm guessing because they aren't using the cache. GH job link HUD commit link"

Those timeouts have resolved itself..

pytorch-bot bot added the topic: not user facing topic category label Nov 13, 2024

malfet changed the title ~~Use sccache-0.8.2 everywhere~~ [CI] Use sccache-0.8.2 for CUDA builds Nov 13, 2024

wdvr approved these changes Nov 15, 2024

View reviewed changes

malfet marked this pull request as ready for review November 18, 2024 21:38

malfet requested a review from jeffdaily as a code owner November 18, 2024 21:38

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 18, 2024

pytorchmergebot force-pushed the malfet-patch-28 branch from 2acb323 to 7084e46 Compare November 18, 2024 21:40

pytorchmergebot added the merging label Nov 18, 2024

pytorchmergebot removed the merging label Nov 18, 2024

malfet added the no-runner-experiments Bypass Meta/LF runner determinator label Nov 19, 2024

pytorchmergebot force-pushed the malfet-patch-28 branch from 7084e46 to 81f8824 Compare November 20, 2024 23:19

malfet force-pushed the malfet-patch-28 branch from 81f8824 to 8085be3 Compare November 26, 2024 21:15

wdvr added a commit that referenced this pull request Dec 6, 2024

remove sccache from bazel, to go together with #140614

94f312b

pytorchmergebot force-pushed the malfet-patch-28 branch from 8085be3 to dd31785 Compare December 7, 2024 12:07

malfet and others added 2 commits December 10, 2024 01:35

Use sccache-0.8.2 everywhere

e1dbc85

update to sccache 0.9.0 that shoudl support nvcc

e4f4f36

pytorchmergebot force-pushed the malfet-patch-28 branch from 8986e22 to e4f4f36 Compare December 10, 2024 01:35

malfet changed the title ~~[CI] Use sccache-0.8.2 for CUDA builds~~ [CI] Use sccache-0.9.0 for CPU and CUDA builds Dec 10, 2024

More -E magic

8f911c4

malfet changed the title ~~[CI] Use sccache-0.9.0 for CPU and CUDA builds~~ [CI] Use sccache-0.8.2 for CUDA builds Dec 10, 2024

wdvr mentioned this pull request Dec 10, 2024

allow -E to be in any spot in the compiler command #142813

Closed

pytorchmergebot added the merging label Dec 11, 2024

pytorchmergebot added the Merged label Dec 11, 2024

pytorchmergebot closed this in b94a206 Dec 11, 2024

pytorchmergebot removed the merging label Dec 11, 2024

wdvr added ciflow/trunk Trigger trunk jobs on your pull request and removed ciflow/trunk Trigger trunk jobs on your pull request labels Dec 11, 2024

malfet deleted the malfet-patch-28 branch December 12, 2024 22:29

Conversation

malfet commented Nov 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140614

❌ 2 New Failures, 1 Unrelated Failure

Uh oh!

malfet commented Nov 18, 2024

Uh oh!

pytorchmergebot commented Nov 18, 2024

Uh oh!

pytorchmergebot commented Nov 18, 2024

Uh oh!

pytorchmergebot commented Nov 18, 2024

Merge started

Uh oh!

pytorchmergebot commented Nov 18, 2024

Merge failed

Uh oh!

malfet commented Nov 20, 2024

Uh oh!

pytorchmergebot commented Nov 20, 2024

Uh oh!

pytorchmergebot commented Nov 20, 2024

Uh oh!

wdvr commented Dec 7, 2024

Uh oh!

pytorchmergebot commented Dec 7, 2024

Uh oh!

pytorchmergebot commented Dec 7, 2024

Uh oh!

pytorchmergebot commented Dec 10, 2024

Uh oh!

pytorchmergebot commented Dec 10, 2024

Uh oh!

malfet commented Dec 11, 2024

Uh oh!

pytorchmergebot commented Dec 11, 2024

Merge started

Uh oh!

ZainRizvi commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZainRizvi commented Dec 11, 2024

Uh oh!

pytorchmergebot commented Dec 11, 2024

Uh oh!

pytorchmergebot commented Dec 11, 2024

Reverting PR 140614 failed

Uh oh!

wdvr commented Dec 11, 2024

Uh oh!

malfet commented Dec 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

malfet commented Nov 13, 2024 •

edited

Loading

pytorch-bot bot commented Nov 13, 2024 •

edited

Loading

ZainRizvi commented Dec 11, 2024 •

edited

Loading