[Reland] Upgrade NCCL to 2.29.3 by kwen2501 · Pull Request #176424 · pytorch/pytorch

kwen2501 · 2026-03-04T12:49:12Z

Trying to reland after original PR's revert #174338

pytorch-bot · 2026-03-04T12:49:17Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/176424

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6d17806 with merge base ad92985 ():

NEW FAILURE - The following job has failed:

vLLM Benchmark / Run vLLM benchmarks (linux.dgx.b200.8, openai/gpt-oss-120b) / benchmark (gh)
Process completed with exit code 1.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

trunk / linux-jammy-rocm-py3.10 / test (default, 1, 6, linux.rocm.gpu.gfx950.1) (gh) (disabled by #106027 but the issue was closed recently and a rebase is needed to make it pass)
test/profiler/test_profiler.py::TestProfiler::test_profiler_cuda_sync_events
trunk / linux-jammy-rocm-py3.10 / test (default, 5, 6, linux.rocm.gpu.gfx950.1) (gh) (disabled by #107893 but the issue was closed recently and a rebase is needed to make it pass)
test/inductor/test_inductor_freezing.py::FreezingCpuTests::test_conv_with_as_strided_cpu

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

inductor / inductor-cpu-test / test (cpu_inductor_torchbench, 1, 2, linux.2xlarge.amx, unstable) (gh) (#174929)
detectron2_maskrcnn_r_50_fpn

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kwen2501 · 2026-03-04T12:50:33Z

@huydhn can you please rerun vLLM CI with NCCL_DEBUG=INFO? Thanks!

Skylion007 · 2026-03-04T17:10:11Z

We should probably update nvshmem too soon

huydhn · 2026-03-04T23:21:16Z

Link to the run https://github.com/pytorch/pytorch/actions/runs/22694184506

huydhn

The mock benchmark looks good https://github.com/pytorch/pytorch/actions/runs/22694184506/job/65806064390. It means that whatever caused this to fail before was on vLLM and had been fixed

huydhn · 2026-03-05T02:20:43Z

@pytorchbot merge

pytorchmergebot · 2026-03-05T02:22:54Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2026-03-05T03:05:40Z

Merge failed

Reason: 1 jobs have failed, first few of them are: vLLM Benchmark / Run vLLM benchmarks (linux.dgx.b200.8, openai/gpt-oss-120b) / benchmark

Details for Dev Infra team

Raised by workflow job

huydhn · 2026-03-05T03:07:56Z

The gpt-oss-120b benchmark failure is from trunk and not related to this change

huydhn · 2026-03-05T03:08:01Z

@pytorchbot merge -i

pytorchmergebot · 2026-03-05T03:10:05Z

Merge started

Your change will be merged while ignoring the following 2 checks: inductor / inductor-cpu-test / test (cpu_inductor_torchbench, 1, 2, linux.2xlarge.amx, unstable), vLLM Benchmark / Run vLLM benchmarks (linux.dgx.b200.8, openai/gpt-oss-120b) / benchmark

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

kwen2501 · 2026-03-05T06:48:01Z

Thanks a lot @huydhn

kwen2501 · 2026-03-05T12:59:39Z

@pytorchbot cherry-pick --onto release/2.11 -c release

pytorchbot · 2026-03-05T13:05:23Z

Cherry picking #176424

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x a2ab1624d05251ca75f57ecb967ab8aaf8ae356b returned non-zero exit code 1

Auto-merging .github/scripts/generate_binary_build_matrix.py
Auto-merging .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml
CONFLICT (content): Merge conflict in .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml
Auto-merging .github/workflows/generated-linux-binary-manywheel-nightly.yml
CONFLICT (content): Merge conflict in .github/workflows/generated-linux-binary-manywheel-nightly.yml
error: could not apply a2ab1624d05... [Reland] Upgrade NCCL to 2.29.3 (#176424)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

Skylion007 · 2026-03-05T18:55:02Z

When are we updating to 2.29.7 now? :)

kwen2501 · 2026-03-06T05:30:43Z

When are we updating to 2.29.7 now? :)

Here is the 2.29.7 PR (which you approved, thanks!):
#176299

This 2.29.3 PR is for cherry-picking back into 2.11 release. (It was previously reverted by the time of the cut).

[Reland] Upgrade NCCL to 2.29.3

6d17806

kwen2501 requested review from a team and jeffdaily as code owners March 4, 2026 12:49

pytorch-bot bot added ci-no-td Do not run TD on this PR ciflow/inductor topic: not user facing topic category labels Mar 4, 2026

kwen2501 requested a review from huydhn March 4, 2026 12:49

pytorchbot added the open source label Mar 4, 2026

Skylion007 approved these changes Mar 4, 2026

View reviewed changes

huydhn approved these changes Mar 5, 2026

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 5, 2026

pytorchmergebot added the merging label Mar 5, 2026

pytorchmergebot removed the merging label Mar 5, 2026

pytorchmergebot added the merging label Mar 5, 2026

pytorchmergebot added the Merged label Mar 5, 2026

pytorchmergebot closed this in a2ab162 Mar 5, 2026

pytorchmergebot removed the merging label Mar 5, 2026

This was referenced Mar 6, 2026

Reland NCCL 2.29.3 in 2.11 #176699

Open

[v.2.11.0] Release Tracker #175093

Open

Conversation

kwen2501 commented Mar 4, 2026

Uh oh!

pytorch-bot bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/176424

❌ 1 New Failure, 3 Unrelated Failures

Uh oh!

kwen2501 commented Mar 4, 2026

Uh oh!

Skylion007 commented Mar 4, 2026

Uh oh!

huydhn commented Mar 4, 2026

Uh oh!

huydhn left a comment

Choose a reason for hiding this comment

Uh oh!

huydhn commented Mar 5, 2026

Uh oh!

pytorchmergebot commented Mar 5, 2026

Merge started

Uh oh!

pytorchmergebot commented Mar 5, 2026

Merge failed

Uh oh!

huydhn commented Mar 5, 2026

Uh oh!

huydhn commented Mar 5, 2026

Uh oh!

pytorchmergebot commented Mar 5, 2026

Merge started

Uh oh!

kwen2501 commented Mar 5, 2026

Uh oh!

kwen2501 commented Mar 5, 2026

Uh oh!

pytorchbot commented Mar 5, 2026

Cherry picking #176424

Uh oh!

Skylion007 commented Mar 5, 2026

Uh oh!

kwen2501 commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pytorch-bot bot commented Mar 4, 2026 •

edited

Loading