[BE] Update CUDNN to 9.4.0.58 by Skylion007 · Pull Request #127925 · pytorch/pytorch

Skylion007 · 2024-06-04T16:55:45Z

Updates to CUDNN 9.4.0.58 https://docs.nvidia.com/deeplearning/cudnn/latest/release-notes.html#cudnn-9-4-0
5-30% performance increase for CUDNN attention (likely by integrating FA3 improvements).
Updates to CUDNN 9.3.0.75 https://docs.nvidia.com/deeplearning/cudnn/latest/release-notes.html#cudnn-9-3-0
Updates CUDNN to 9.2.1.18 to further improve CUDNN flash attention and fix some heuristics. This also has a lot of bugfixes to fix edge cases in the CUDNN Flash Attention and also in very large convolutions.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @rec

pytorch-bot · 2024-06-04T16:55:48Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127925

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 13 New Failures, 1 Cancelled Job, 2 Unrelated Failures

As of commit 6a28122 with merge base f8a5b71 ():

NEW FAILURES - The following jobs have failed:

Build libtorch docker images / build-docker-cuda (12.1) (gh)
Process completed with exit code 1.
docker-builds / docker-build (linux.12xlarge, pytorch-linux-focal-py3-clang10-onnx) (gh)
Process completed with exit code 1.
inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
ERROR: failed to solve: process "/bin/sh -c if [ -n \"${TRITON}\" ]; then bash ./install_triton.sh; fi" did not complete successfully: exit code: 1
inductor / cuda12.1-py3.10-gcc9-sm86 / test (aot_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
ERROR: failed to solve: process "/bin/sh -c if [ -n \"${TRITON}\" ]; then bash ./install_triton.sh; fi" did not complete successfully: exit code: 1
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
ERROR: failed to solve: process "/bin/sh -c if [ -n \"${TRITON}\" ]; then bash ./install_triton.sh; fi" did not complete successfully: exit code: 1
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
#49 135.2 urllib.error.HTTPError: HTTP Error 520:
inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
#49 139.2 urllib.error.HTTPError: HTTP Error 520:
linux-binary-manywheel / manywheel-py3_9-cuda11_8-test / test (gh)
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu11==9.4.0.58; platform_system == "Linux" and platform_machine == "x86_64" (from torch) (from versions: 8.7.0.84, 8.9.6.50, 8.9.7.29, 9.1.0.70)
linux-binary-manywheel / manywheel-py3_9-cuda12_1-test / test (gh)
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu12==9.4.0.58; platform_system == "Linux" and platform_machine == "x86_64" (from torch) (from versions: 8.8.1.3, 8.9.2.26, 8.9.7.29, 9.1.0.70, 9.3.0.75)
linux-binary-manywheel / manywheel-py3_9-cuda12_4-test / test (gh)
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu12==9.4.0.58; platform_system == "Linux" and platform_machine == "x86_64" (from torch) (from versions: 8.9.7.29, 9.1.0.70)
linux-binary-manywheel-split / manywheel-py3_9-cuda11_8-test / test (gh)
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu11==9.4.0.58; platform_system == "Linux" and platform_machine == "x86_64" (from torch-no-python) (from versions: 8.7.0.84, 8.9.6.50, 8.9.7.29, 9.1.0.70)
linux-binary-manywheel-split / manywheel-py3_9-cuda12_1-test / test (gh)
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu12==9.4.0.58; platform_system == "Linux" and platform_machine == "x86_64" (from torch-no-python) (from versions: 8.8.1.3, 8.9.2.26, 8.9.7.29, 9.1.0.70)
linux-binary-manywheel-split / manywheel-py3_9-cuda12_4-test / test (gh)
ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu12==9.4.0.58; platform_system == "Linux" and platform_machine == "x86_64" (from torch-no-python) (from versions: 8.9.7.29, 9.1.0.70)

CANCELLED JOB - The following job was cancelled. Please retry:

periodic / linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu) (gh)
##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

docker-builds / docker-build (linux.12xlarge, pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks) (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

periodic / win-vs2019-cuda11.8-py3 / test (default, 1, 4, lf.windows.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
test_linalg.py::TestLinalgCUDA::test_matmul_offline_tunableop_cuda_float16

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Skylion007 · 2024-06-04T16:56:59Z

In the off chance we can squeeze this into to 2.4 or if it solves any inductor issues we are having.

Skylion007 · 2024-06-07T14:03:23Z

@pytorchbot rebase

pytorchmergebot · 2024-06-07T14:04:57Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-06-07T14:05:00Z

Successfully rebased skylion007/update-cudnn-9-1-1-17 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout skylion007/update-cudnn-9-1-1-17 && git pull --rebase)

Skylion007 · 2024-08-13T14:38:53Z

@atalman Updated it again to the latest CUDNN version

eqy · 2024-08-13T17:45:13Z

We have a few internal (conv?) tests that seem to newly failiying with 9.3.0.75, will try to provide public reproducers that we can test...

Skylion007 · 2024-09-28T16:33:38Z

@eqy Are these issues fixed in the newly released 9.4?

eqy · 2024-10-03T01:15:22Z

Tentatively signing off on this, I think we are OK for now

nWEIdia · 2024-10-10T20:47:31Z

Have inductor tests been run with this PR yet? It would be good to know whether certain models would have fluctuations regarding the accuracy.

Skylion007 · 2024-10-14T16:16:46Z

@nWEIdia Updated benchmarks, tinynet_a now passes inductor accuracy tests

nWEIdia

Looks great to me!

Skylion007 · 2024-10-14T16:31:37Z

@nWEIdia I need to get the binaries uploaded to the pytorch_builder repo, that's the main blocker.

nWEIdia · 2024-10-14T16:33:45Z

@nWEIdia I need to get the binaries uploaded to the pytorch_builder repo, that's the main blocker.

Do you mean uploading the cudnn PYPI dependency packages the S3 (download.pytorch.org)?
cc @atalman for help.

Skylion007 requested review from albanD, eqy, ezyang and malfet June 4, 2024 16:55

Skylion007 requested a review from a team as a code owner June 4, 2024 16:55

pytorch-bot bot added the topic: not user facing topic category label Jun 4, 2024

Skylion007 added the better-engineering Relatively self-contained tasks for better engineering contributors label Jun 4, 2024

pytorchbot added the open source label Jun 4, 2024

Skylion007 added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Jun 4, 2024

ezyang approved these changes Jun 5, 2024

View reviewed changes

ezyang requested a review from atalman June 5, 2024 14:36

pytorchmergebot force-pushed the skylion007/update-cudnn-9-1-1-17 branch from 0557788 to 159c0c8 Compare June 7, 2024 14:05

ezyang approved these changes Jun 7, 2024

View reviewed changes

Skylion007 added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 7, 2024

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch from 159c0c8 to decaa27 Compare July 14, 2024 19:00

Skylion007 changed the title ~~[BE] Update CUDNN to 9.1.1.17~~ [BE] Update CUDNN to 9.2.1.18 Jul 14, 2024

eqy approved these changes Jul 15, 2024

View reviewed changes

albanD removed their request for review July 16, 2024 21:34

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch from decaa27 to a5f9cfe Compare August 2, 2024 14:30

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch from a5f9cfe to 0d6ad43 Compare August 13, 2024 14:37

Skylion007 changed the title ~~[BE] Update CUDNN to 9.2.1.18~~ [BE] Update CUDNN to 9.3.0.75 Aug 13, 2024

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch from 0d6ad43 to 299ca52 Compare August 13, 2024 14:40

Skylion007 requested a review from jeffdaily as a code owner August 13, 2024 14:45

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch 5 times, most recently from 75d27aa to c1d95f1 Compare September 28, 2024 16:48

Skylion007 changed the title ~~[BE] Update CUDNN to 9.3.0.75~~ [BE] Update CUDNN to 9.4.0.58 Sep 28, 2024

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch 2 times, most recently from c2c03a1 to 2e46d77 Compare September 28, 2024 17:14

Skylion007 mentioned this pull request Oct 10, 2024

Updating cuda binary build to get cusparselt from PYPI #137653

Closed

eqy added the ciflow/inductor label Oct 10, 2024

pytorch-bot bot added the module: dynamo label Oct 14, 2024

Skylion007 added 5 commits October 14, 2024 09:08

[BE] Update CUDNN to 9.1.1.17

8773721

Fix conflict

0ece56e

Update CUDNN to 9.3.0.75

34a100f

Update docker too

aa419b3

Try to update cudnn 9.4

28cdafb

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch 2 times, most recently from 892e328 to ef99783 Compare October 14, 2024 16:14

nWEIdia approved these changes Oct 14, 2024

View reviewed changes

Skylion007 mentioned this pull request Oct 14, 2024

Any plan to support flash attention 3 for hopper GPUs? #137901

Open

Update inductor benchmarks

6a28122

Skylion007 force-pushed the skylion007/update-cudnn-9-1-1-17 branch from ef99783 to 6a28122 Compare October 14, 2024 16:30

Skylion007 closed this Dec 6, 2024

atalman mentioned this pull request Dec 12, 2024

Enable CUDA 12.6 CI/CD , Disable CUDA 12.1 #138440

Closed

31 tasks

Conversation

Skylion007 commented Jun 4, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127925

❌ 13 New Failures, 1 Cancelled Job, 2 Unrelated Failures

Uh oh!

Skylion007 commented Jun 4, 2024

Uh oh!

Skylion007 commented Jun 7, 2024

Uh oh!

pytorchmergebot commented Jun 7, 2024

Uh oh!

pytorchmergebot commented Jun 7, 2024

Uh oh!

Skylion007 commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eqy commented Aug 13, 2024

Uh oh!

Skylion007 commented Sep 28, 2024

Uh oh!

eqy commented Oct 3, 2024

Uh oh!

nWEIdia commented Oct 10, 2024

Uh oh!

Skylion007 commented Oct 14, 2024

Uh oh!

nWEIdia left a comment

Choose a reason for hiding this comment

Uh oh!

Skylion007 commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nWEIdia commented Oct 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Skylion007 commented Jun 4, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jun 4, 2024 •

edited

Loading

Skylion007 commented Aug 13, 2024 •

edited

Loading

Skylion007 commented Oct 14, 2024 •

edited

Loading