Skip to content

Add CUDA 12.4 workflows#121684

Closed
nWEIdia wants to merge 22 commits intopytorch:mainfrom
nWEIdia:workflows_cu124
Closed

Add CUDA 12.4 workflows#121684
nWEIdia wants to merge 22 commits intopytorch:mainfrom
nWEIdia:workflows_cu124

Conversation

@nWEIdia
Copy link
Copy Markdown
Collaborator

@nWEIdia nWEIdia commented Mar 11, 2024 •
edited by pytorch-bot bot
Loading

@nWEIdia nWEIdia requested a review from a team as a code owner March 11, 2024 23:23
@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Mar 11, 2024
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot bot commented Mar 11, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/121684

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 456757f with merge base a03b9a2 (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Aidyn-A
Aidyn-A reviewed Mar 12, 2024
View reviewed changes
ptrblck
ptrblck reviewed Mar 12, 2024
View reviewed changes
ptrblck
ptrblck reviewed Mar 12, 2024
View reviewed changes
ptrblck
ptrblck reviewed Mar 12, 2024
View reviewed changes
ptrblck
ptrblck reviewed Mar 12, 2024
View reviewed changes
@johnnynunez
Copy link
Copy Markdown
Contributor

johnnynunez commented Mar 13, 2024 •
edited
Loading

@ptrblck @nWEIdia why not with cudnn 9.0? It's major improve on flash attention, and I saw that headers were updated to cudnn 9.0
https://docs.nvidia.com/deeplearning/cudnn/release-notes.html

@ptrblck
Copy link
Copy Markdown
Collaborator

ptrblck commented Mar 13, 2024

@johnnynunez It's not available as described in the review: #121684 (comment)

@johnnynunez
Copy link
Copy Markdown
Contributor

johnnynunez commented Mar 13, 2024

@johnnynunez It's not available as described in the review: #121684 (comment)

I see.. thanks

@drisspg drisspg added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Mar 13, 2024
eqy
eqy reviewed Mar 14, 2024
View reviewed changes
@ptrblck ptrblck added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Mar 15, 2024
@johnnynunez
Copy link
Copy Markdown
Contributor

johnnynunez commented Mar 17, 2024

when will it merged? :)

@johnnynunez
Copy link
Copy Markdown
Contributor

johnnynunez commented Apr 4, 2024

12.4 update 1 is out:
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/

@johnnynunez
Copy link
Copy Markdown
Contributor

johnnynunez commented Apr 9, 2024

@ptrblck @nWEIdia nvidia cudnn9 now is available nvidia-cudnn-cu12 9.0.0.312
https://pypi.org/project/nvidia-cudnn-cu12/

nWEIdia added a commit to nWEIdia/builder that referenced this pull request Apr 11, 2024
@nWEIdia
Enable CUDA 12.4 builds
bf63b1a 
GHA results show this is needed to fix errors in pytorch/pytorch#121684
Reference: pytorch#1374
@nWEIdia nWEIdia mentioned this pull request Apr 11, 2024
Merged
@atalman
Copy link
Copy Markdown
Contributor

atalman commented Apr 11, 2024 •
edited
Loading

@nWEIdia while windows AMI is not yet in place we would need to add only Linux part of things.

malfet pushed a commit to pytorch/builder that referenced this pull request Apr 12, 2024
@nWEIdia
Enable CUDA 12.4 builds (#1785)
4e10974 
GHA results show this is needed to fix errors in pytorch/pytorch#121684 

Reference: #1374
@malfet malfet added ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR and removed ciflow/binaries Trigger all binary build and upload jobs on the PR labels Apr 16, 2024
@nWEIdia
Copy link
Copy Markdown
Collaborator Author

nWEIdia commented Apr 16, 2024

Hi @malfet, for pip wheels, the build seems successful, but the test job failed: installing it required the presence of cu124/ AWS directory. https://github.com/pytorch/pytorch/actions/runs/8701283769/job/23866760262
Could you please help prepare the cu124 directory? https://download.pytorch.org/whl/nightly/cu124 Thanks!

nWEIdia and others added 11 commits April 30, 2024 10:26
@nWEIdia
update driver for cu124
85c22ac 
reference: https://docs.nvidia.com/cuda/archive/12.4.0/cuda-toolkit-release-notes/index.html#id6

                 Linux x86_64 Driver Version | Windows x86_64 Driver Version

CUDA 12.4 GA        >=550.54.14              |     >=551.61
@nWEIdia
Update driver to 550.54.15 according to https://docs.nvidia.com/datac…
9818814 
…enter/tesla/tesla-release-notes-550-54-15/index.html
@nWEIdia
Revert "Update driver to 550.54.15 according to https://docs.nvidia.c…
f8c3d42 
…om/datacenter/tesla/tesla-release-notes-550-54-15/index.html"

This reverts commit 7ec3934.
@nWEIdia
Revert "update driver for cu124"
1ac29ad 
This reverts commit aaa24b9.
@nWEIdia
Add CI test and bazel test using new driver with setup-nvidia action
457e6e8 
driver-version: "550.54.15"
@nWEIdia
Revert "Disable CUDA 12.4 build for Windows"
427ca8c 
This reverts commit 0e23765.
@nWEIdia
Revert "Change builder repo/branch to test this PR and pytorch/builde…
1f2af3a 
…r#1786"

This reverts commit a1e3a36.
@nWEIdia
Running ./regenerate.sh again after reverting the disabling of Windows
0a20a79 
CU124
@nWEIdia
Reverting driver version changes as the default should already be
44cb1fe 
550.54.15 since
    pytorch/test-infra@d5695df
is in main branch of pytorch/test-infra
@nWEIdia @atalman
Update .github/scripts/generate_binary_build_matrix.py
6a94cb7 
We need to keep 12.1 here still since this is the default wheel we will be uploading to pypi 12.4 should still be experimental build for now.

Co-authored-by: Andrey Talman <atalman@fb.com>
@nWEIdia
Run ./regenerate after 12.4 -> 12.1 change in PYTORCH_EXTRA_INSTALL_R…
6d474e5 
…EQUIREMENTS
@atalman
Copy link
Copy Markdown
Contributor

atalman commented Apr 30, 2024

@pytorchmergebot merge -f "All required jobs are passing"

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Apr 30, 2024

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

pytorch-bot bot pushed a commit that referenced this pull request May 3, 2024
@nWEIdia @atalman
Add CUDA 12.4 workflows (#121684)
58e18ef 
Reference: #98492

Co-authored-by: Andrey Talman <atalman@fb.com>
Pull Request resolved: #121684
Approved by: https://github.com/atalman
@nWEIdia nWEIdia mentioned this pull request Jul 31, 2024
Closed
nWEIdia added a commit to nWEIdia/pytorch that referenced this pull request Jul 31, 2024
@nWEIdia
Update cuda 12.4.1 main component version, consulting: https://docs.n…
46c2b60 
…vidia.com/cuda/archive/12.4.1/cuda-toolkit-release-notes/index.html

Related pytorch#121684
nWEIdia added a commit to nWEIdia/pytorch that referenced this pull request Aug 5, 2024
@nWEIdia
Update cuda 12.4.1 main component version, consulting: https://docs.n…
c18ac47 
…vidia.com/cuda/archive/12.4.1/cuda-toolkit-release-notes/index.html

Related pytorch#121684
nWEIdia added a commit to nWEIdia/pytorch that referenced this pull request Aug 6, 2024
@nWEIdia
Update cuda 12.4.1 main component version, consulting: https://docs.n…
0d334b3 
…vidia.com/cuda/archive/12.4.1/cuda-toolkit-release-notes/index.html

Related pytorch#121684
pytorch-bot bot pushed a commit that referenced this pull request Aug 15, 2024
@nWEIdia @pytorchmergebot
Update cuda 12.4.1 main component version, consulting: https://docs.n…
ac187d0 
…vidia.com/cuda/archive/12.4.1/cuda-toolkit-release-notes/index.html

Related #121684
pytorchmergebot pushed a commit that referenced this pull request Aug 20, 2024
@nWEIdia @pytorchmergebot
Enable CUDA 12.4.1 (#132202)
333890b 
Trying to keep a record of the steps before I lose track of it.

- 1st Commit: Similar to pytorch/builder#1720
- 2nd Commit:  Update CUDA 12.4 CI CUDA versions from 12.4.0 to 12.4.1 mapping to changes in https://github.com/pytorch/pytorch/pull/125944/files
- 3rd Commit: update for aarch64 install_cuda_aarch64.sh docker step
- 4th Commit: aaa456e Related #121684
- Synchronization point: Meta helps uploading pypi cuda dependencies specified in .github/scripts/generate_binary_build_matrix.py
- The above pypi upload is done (thanks Andrey!), restarted jobs like https://github.com/pytorch/pytorch/actions/runs/10188203670/job/28369471321
- 7753234, use temporary docker containers (generated from a previous successful container build). If merged, these containers would be rebuilt, therefore testing them now.  (5th commit)
- 6th commit 5f93c62: revert the 5th commit. Update, done but have to debug seemingly irrelevant failures (rocm/xpu/mps)

Pull Request resolved: #132202
Approved by: https://github.com/Skylion007, https://github.com/eqy, https://github.com/atalman
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eqy eqy eqy left review comments

@atalman atalman atalman approved these changes

@ptrblck ptrblck ptrblck left review comments

@Aidyn-A Aidyn-A Aidyn-A left review comments

Assignees

No one assigned

Labels

ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR ciflow/trunk Trigger trunk jobs on your pull request Merged open source skip-pr-sanity-checks topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.