Advance nightly docker to 11.6 by atalman · Pull Request #86941 · pytorch/pytorch

atalman · 2022-10-13T21:22:34Z

Fixes following:
https://github.com/pytorch/pytorch/actions/runs/3242695506/jobs/5316334351
crash in Docker builds introduced by: #82682

The PR seems to introduce some changes not compatible with cuda 11.3 which is used by our Docker builds

pytorch-bot · 2022-10-13T21:22:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86941

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 Failures, 1 Pending

As of commit ccdbb27:

The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

.github/scripts/build_publish_nightly_docker.sh

weiwangmeta · 2022-10-13T23:29:49Z

Dockerfile

Does pytorch conda channel contain pytorch-cuda packages?

These 2 channels should contain the right packages:

ARG CUDA_CHANNEL=nvidia ARG INSTALL_CHANNEL=pytorch-nightly

Nightly contains pytorch-cuda
Pytorch not yet

Docker builds fix Fix docker file Testing Advance Docker builds to cuda 11.6 Fix typo Fix typo Fix cuda version test Tune substitution test test fix test test test test testing

seemethere · 2022-10-14T16:08:42Z

docker.Makefile

 CUDA_CHANNEL              = nvidia
 # The conda channel to use to install pytorch / torchvision
-INSTALL_CHANNEL          ?= pytorch
+INSTALL_CHANNEL          ?= pytorch-nightly


oh wait we shouldn't change the defaults for this

There might be users who rely on the default behavior so changing this could be very problematic

The issue with pytorch channel is that it does not contain pytorch-cuda yet. I think publishing pytorch-cuda package to pytorch channel should fix this issue.

seemethere · 2022-10-14T16:10:28Z

docker.Makefile

 CUDA_CHANNEL              = nvidia
 # The conda channel to use to install pytorch / torchvision
-INSTALL_CHANNEL          ?= pytorch
+INSTALL_CHANNEL          ?= pytorch-nightly


There might be users who rely on the default behavior so changing this could be very problematic

.github/scripts/build_publish_nightly_docker.sh

docker.Makefile

atalman · 2022-10-17T21:53:59Z

Please note current failure of Build Official Docker Image is because of this issue:

ContinuumIO/anaconda-issues#13073

atalman · 2022-10-19T21:25:24Z

@pytorchmergebot merge -f "some build failures expected"

pytorchmergebot · 2022-10-19T21:26:49Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2022-10-20T13:14:58Z

@pytorchmergebot revert -c "Workflow is passing but installs CUDA 11.3 PyTorch rather then 11.6"

pytorch-bot · 2022-10-20T13:15:00Z

❌ 🤖 pytorchbot command failed:

@pytorchbot revert: error: argument -c/--classification: invalid choice: 'Workflow is passing but installs CUDA 11.3 PyTorch rather then 11.6' (choose from 'nosignal', 'ignoredsignal', 'landrace', 'weird', 'ghfirst')

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Try @pytorchbot --help for more info.

atalman · 2022-10-20T13:15:40Z

@pytorchmergebot revert -m "Workflow is passing but installs CUDA 11.3 PyTorch rather then 11.6" -c weird

pytorchmergebot · 2022-10-20T13:17:06Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2022-10-20T13:17:15Z

@atalman your PR has been successfully reverted.

This reverts commit c5de535. Reverted #86941 on behalf of https://github.com/atalman due to Workflow is passing but installs CUDA 11.3 PyTorch rather then 11.6

Fixes following: https://github.com/pytorch/pytorch/actions/runs/3242695506/jobs/5316334351 crash in Docker builds introduced by: #82682 The PR seems to introduce some changes not compatible with cuda 11.3 which is used by our Docker builds This is a reland of original pr: #86941 (Created this new PR to start fresh) Which was reverted because conda install, installed wrong version of pytorch. It installed pytorch for cuda 11.3 still rather then 11.6 This should be fixed now with Release 1.13 Pull Request resolved: #87858 Approved by: https://github.com/seemethere, https://github.com/malfet, https://github.com/izaitsevfb

Fixes following: https://github.com/pytorch/pytorch/actions/runs/3242695506/jobs/5316334351 crash in Docker builds introduced by: pytorch#82682 The PR seems to introduce some changes not compatible with cuda 11.3 which is used by our Docker builds This is a reland of original pr: pytorch#86941 (Created this new PR to start fresh) Which was reverted because conda install, installed wrong version of pytorch. It installed pytorch for cuda 11.3 still rather then 11.6 This should be fixed now with Release 1.13 Pull Request resolved: pytorch#87858 Approved by: https://github.com/seemethere, https://github.com/malfet, https://github.com/izaitsevfb

atalman requested a review from a team as a code owner October 13, 2022 21:22

pytorch-bot bot added the topic: not user facing topic category label Oct 13, 2022

weiwangmeta reviewed Oct 13, 2022

View reviewed changes

.github/scripts/build_publish_nightly_docker.sh Outdated Show resolved Hide resolved

weiwangmeta reviewed Oct 13, 2022

View reviewed changes

atalman force-pushed the test_cuda_116_doker branch 2 times, most recently from b93eb95 to 2cbb202 Compare October 14, 2022 00:29

Advance nightly docker to 11.6

7302d2f

Docker builds fix Fix docker file Testing Advance Docker builds to cuda 11.6 Fix typo Fix typo Fix cuda version test Tune substitution test test fix test test test test testing

atalman force-pushed the test_cuda_116_doker branch from 33bcc85 to 7302d2f Compare October 14, 2022 01:11

Use nightly by default

aab4e9c

seemethere approved these changes Oct 14, 2022

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 14, 2022

seemethere reviewed Oct 14, 2022

View reviewed changes

seemethere requested changes Oct 14, 2022

View reviewed changes

malfet reviewed Oct 15, 2022

View reviewed changes

.github/scripts/build_publish_nightly_docker.sh Outdated Show resolved Hide resolved

malfet reviewed Oct 15, 2022

View reviewed changes

docker.Makefile Outdated Show resolved Hide resolved

malfet reviewed Oct 15, 2022

View reviewed changes

docker.Makefile Outdated Show resolved Hide resolved

malfet reviewed Oct 15, 2022

View reviewed changes

docker.Makefile Outdated Show resolved Hide resolved

Use pytorch channel and cuda version 11.6.2

d7d273e

malfet approved these changes Oct 17, 2022

View reviewed changes

atalman added 2 commits October 18, 2022 13:32

Include update conda

7d71526

Fix conda update

ccdbb27

pytorchmergebot added the Merged label Oct 19, 2022

pytorchmergebot closed this in c5de535 Oct 19, 2022

pytorchmergebot added the Reverted label Oct 20, 2022

atalman reopened this Oct 27, 2022

atalman closed this Oct 27, 2022

atalman mentioned this pull request Oct 27, 2022

Advance nightly docker to 11.6 #87858

Closed

Conversation

atalman commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86941

❌ 3 Failures, 1 Pending

Uh oh!

Uh oh!

weiwangmeta Oct 13, 2022

Choose a reason for hiding this comment

Uh oh!

atalman Oct 13, 2022

Choose a reason for hiding this comment

Uh oh!

seemethere Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

seemethere Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

atalman Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

seemethere Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

atalman commented Oct 17, 2022

Uh oh!

atalman commented Oct 19, 2022

Uh oh!

pytorchmergebot commented Oct 19, 2022

Merge started

Uh oh!

atalman commented Oct 20, 2022

Uh oh!

pytorch-bot bot commented Oct 20, 2022

Uh oh!

atalman commented Oct 20, 2022

Uh oh!

pytorchmergebot commented Oct 20, 2022

Uh oh!

pytorchmergebot commented Oct 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

atalman commented Oct 13, 2022 •

edited

Loading

pytorch-bot bot commented Oct 13, 2022 •

edited

Loading