Enable CUDA 12.6 OSS CI by tinglvv · Pull Request #140793 · pytorch/pytorch

tinglvv · 2024-11-15T07:46:34Z

referred to #121956 and #139407

related to #138440

pytorch-bot · 2024-11-15T07:46:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140793

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 15 Unrelated Failures

As of commit 4f7b945 with merge base aade4fb ():

NEW FAILURES - The following jobs have failed:

BC Lint / bc_linter (gh)
Process completed with exit code 128.
inductor / cuda12.6-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
crossvit_9_240
pull / linux-focal-cuda12.6-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu) (gh)
test_foreach.py::TestForeachCUDA::test_pointwise_op_with_tensor_of_scalarlist_overload__foreach_addcmul_is_fastpath_True_cuda_bfloat16
trunk / libtorch-linux-focal-cuda12.6-py3.10-gcc9-debug / build (gh)
undefined reference to std::__throw_bad_array_new_length()'`

CANCELLED JOB - The following job was cancelled. Please retry:

inductor-periodic / cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
##[error]The operation was canceled.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor-periodic / cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'Test'
inductor-periodic / cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'Test'
inductor-periodic / cuda12.4-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'Test'
inductor-periodic / cuda12.4-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
hrnet_w18
inductor-periodic / cuda12.4-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
'Test'
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_aot_inductor_amp_freezing_torchbench, 2, 2, linux.8xlarge.amx) (gh) (detected as infra flaky with no log or failing log classifier)
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_amp_freezing_torchbench, 2, 2, linux.8xlarge.amx) (gh) (similar failure)
speech_transformer
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_timm, 2, 2, linux.8xlarge.amx) (gh) (matched linux rule in flaky-rules.json)
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_freezing_torchbench, 2, 2, linux.8xlarge.amx) (gh) (similar failure)
speech_transformer
inductor-periodic / linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (cpu_inductor_freezing_avx2_torchbench, 2, 2, linux.10xlarge.avx2) (gh) (similar failure)
speech_transformer
inductor-periodic / rocm6_3-py3_10-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
torch_multimodal_clip
inductor-periodic / rocm6_3-py3_10-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
hrnet_w18
inductor-periodic / rocm6_3-py3_10-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 2, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
torch_multimodal_clip
inductor-periodic / rocm6_3-py3_10-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.rocm.gpu.mi300.2) (gh) (similar failure)
torch_multimodal_clip

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

trunk / libtorch-linux-focal-cuda12.4-py3.10-gcc9-debug / build (gh) (#148495)
undefined reference to std::__throw_bad_array_new_length()'`

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot · 2024-11-18T23:19:37Z

Warning: Unknown label ciflow/inductor-cu126.
Currently recognized labels are

ciflow/binaries
ciflow/binaries_conda
ciflow/binaries_libtorch
ciflow/binaries_wheel
ciflow/inductor
ciflow/inductor-periodic
ciflow/inductor-rocm
ciflow/inductor-perf-compare
ciflow/inductor-micro-benchmark
ciflow/inductor-micro-benchmark-cpu-x86
ciflow/inductor-cu124
ciflow/linux-aarch64
ciflow/mps
ciflow/nightly
ciflow/periodic
ciflow/rocm
ciflow/s390
ciflow/slow
ciflow/trunk
ciflow/unstable
ciflow/xpu
ciflow/torchbench
ciflow/autoformat

Please add the new label to .github/pytorch-probot.yml

.ci/docker/build.sh

netlify · 2024-12-06T18:59:12Z

✅ Deploy Preview for chimerical-cranachan-793287 ready!

Name	Link
🔨 Latest commit	`8c1e2d4`
🔍 Latest deploy log	https://app.netlify.com/sites/chimerical-cranachan-793287/deploys/67534d69f9595400083cecac
😎 Deploy Preview	https://deploy-preview-140793--chimerical-cranachan-793287.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

atalman · 2024-12-06T19:59:37Z

Hi @tinglvv please fix lint

pytorchmergebot · 2025-02-07T00:52:07Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-02-07T00:52:11Z

Successfully rebased inductor-126-test onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout inductor-126-test && git pull --rebase)

tinglvv · 2025-02-28T22:16:41Z

@pytorchbot rebase

pytorchmergebot · 2025-02-28T22:18:10Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-02-28T22:18:12Z

Successfully rebased inductor-126-test onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout inductor-126-test && git pull --rebase)

tinglvv · 2025-03-04T00:38:14Z

@pytorchbot rebase

pytorchmergebot · 2025-03-04T00:39:40Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-03-04T00:39:43Z

Successfully rebased inductor-126-test onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout inductor-126-test && git pull --rebase)

atalman · 2025-03-05T21:35:32Z

Trunk failures are related to: #148495

trunk / libtorch-linux-focal-cuda12.6-py3.10-gcc9-debug / build (gh)
undefined reference to std::__throw_bad_array_new_length()'`

) #145570 breaking #140793 into eager and inductor benchmarks to unblock Pull Request resolved: #148602 Approved by: https://github.com/atalman, https://github.com/malfet Co-authored-by: atalman <atalman@fb.com>

#145570 breaking #140793 into eager and inductor benchmarks to unblock Seems many inductor yml are added after initial change was prepared. Pull Request resolved: #148612 Approved by: https://github.com/nWEIdia, https://github.com/atalman Co-authored-by: atalman <atalman@fb.com>

pytorch-bot bot added the topic: not user facing topic category label Nov 15, 2024

pytorchbot added the open source label Nov 15, 2024

tinglvv changed the title ~~add inductor cuda 12.6 test~~ Enable CUDA 12.6 OSS CI Nov 15, 2024

atalman added the ciflow/inductor-cu126 label Nov 18, 2024

tinglvv mentioned this pull request Nov 20, 2024

add inductor-cu126 tag #141110

Closed

tinglvv added ciflow/inductor-cu126 and removed ciflow/inductor-cu126 labels Dec 2, 2024

tinglvv marked this pull request as ready for review December 3, 2024 23:40

tinglvv requested review from a team and jeffdaily as code owners December 3, 2024 23:40

cpuhrsch requested review from malfet and seemethere December 4, 2024 00:01

cpuhrsch added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Dec 4, 2024

nWEIdia reviewed Dec 5, 2024

View reviewed changes

.ci/docker/build.sh Outdated Show resolved Hide resolved

tinglvv force-pushed the inductor-126-test branch from 1aa42b8 to e69e393 Compare December 6, 2024 18:58

tinglvv added ciflow/trunk Trigger trunk jobs on your pull request ciflow/inductor-periodic labels Dec 6, 2024

atalman mentioned this pull request Dec 12, 2024

Enable CUDA 12.6 CI/CD , Disable CUDA 12.1 #138440

Closed

31 tasks

tinglvv force-pushed the inductor-126-test branch from 8c1e2d4 to 6d65abb Compare December 13, 2024 00:21

tinglvv force-pushed the inductor-126-test branch from 9e21efd to 1d5b30c Compare January 15, 2025 18:48

tinglvv mentioned this pull request Jan 16, 2025

[aarch64] multiple inductor test failures related to vec128_bfloat16 #144818

Closed

tinglvv force-pushed the inductor-126-test branch from ed63a51 to cc70b6b Compare January 23, 2025 18:24

pytorch-bot bot temporarily deployed to upload-benchmark-results January 23, 2025 20:00 Inactive

pytorch-bot bot temporarily deployed to upload-benchmark-results January 24, 2025 23:11 Inactive

pytorchmergebot force-pushed the inductor-126-test branch from f1e40f2 to fe7c5a6 Compare February 7, 2025 00:52

pytorchmergebot force-pushed the inductor-126-test branch from fe7c5a6 to 616f465 Compare February 28, 2025 22:18

tinglvv added 12 commits March 4, 2025 00:39

add inductor cuda 12.6 test

5a7fb16

correct 124 with 126

f309ba6

fix trailing space

44bc452

use 12.6.3

b175a71

update trunk.yml after rebase

09f1b31

reformat the inductor-cu126 tests for lint

71fe82c

Update to latest

0a699dc

Update to latest

c95e2fc

resolve conflict

6868b50

update

1023e66

fix lint

485f478

fix lint

4f7b945

pytorchmergebot force-pushed the inductor-126-test branch from 616f465 to 4f7b945 Compare March 4, 2025 00:39

nWEIdia added the keep-going Don't stop on first failure, keep running tests until the end label Mar 5, 2025

tinglvv mentioned this pull request Mar 5, 2025

[CI][CUDA] Move away from cuda12.4, Add cuda12.6 eager CI tests #148602

Closed

tinglvv mentioned this pull request Mar 5, 2025

[CI] [inductor] Add cu126 inductor jobs and move away cu124 #148612

Closed

tinglvv closed this Mar 7, 2025

Conversation

tinglvv commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/140793

❌ 4 New Failures, 1 Cancelled Job, 15 Unrelated Failures

Uh oh!

pytorch-bot bot commented Nov 18, 2024

Uh oh!

Uh oh!

netlify bot commented Dec 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for chimerical-cranachan-793287 ready!

Uh oh!

atalman commented Dec 6, 2024

Uh oh!

pytorchmergebot commented Feb 7, 2025

Uh oh!

pytorchmergebot commented Feb 7, 2025

Uh oh!

tinglvv commented Feb 28, 2025

Uh oh!

pytorchmergebot commented Feb 28, 2025

Uh oh!

pytorchmergebot commented Feb 28, 2025

Uh oh!

tinglvv commented Mar 4, 2025

Uh oh!

pytorchmergebot commented Mar 4, 2025

Uh oh!

pytorchmergebot commented Mar 4, 2025

Uh oh!

atalman commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tinglvv commented Nov 15, 2024 •

edited

Loading

pytorch-bot bot commented Nov 15, 2024 •

edited

Loading

netlify bot commented Dec 6, 2024 •

edited

Loading

atalman commented Mar 5, 2025 •

edited

Loading