CUDA11 builders for Windows by zasdfgbnm · Pull Request #463 · pytorch/builder

zasdfgbnm · 2020-06-18T22:45:24Z

cc: @peterjc123 #450 is deprecated and your comments in #450 will be resolved here. It is now WIP, not complete yet, and I don't have a chance to test it yet.

zasdfgbnm · 2020-06-18T23:11:20Z

@peterjc123 How do I debug these scripts? Do I just create azure pipelines on my fork with windows/azure-pipelines.yml and it will run everything?

peterjc123 · 2020-06-19T02:19:16Z

We use CircleCI instead of Azure Pipelines now. I guess you'll need to modify https://github.com/pytorch/builder/blob/master/.circleci/regenerate.py. You can test that by removing the filter conditions in the builder jobs at PyTorch.

peterjc123

LGTM. But there are several things missing including

CircleCI configs in .circle/regenerate.py
Conda configs in conda/build_pytorch.sh & conda/pytorch-nightly/bld.bat
MAGMA with CUDA 11 for Windows (I can do that for you if you want)

peterjc123 · 2020-06-19T02:20:46Z

windows/internal/smoke_test.bat

 pushd %SRC_DIR%\..

 if "%CUDA_VERSION%" == "102" call internal\driver_update.bat
+if "%CUDA_VERSION%" == "110" call internal\driver_update.bat


Have you checked that the driver is up to date?

Thanks for reminding me, and you are right that it is not up to date. It requires >=451.22. I will update this.

We can use this link http://us.download.nvidia.com/tesla/451.82/451.82-tesla-desktop-winserver-2019-2016-international.exe, or you can upload it to S3 so I can use S3 :)

peterjc123 · 2020-06-19T02:22:05Z

windows/internal/cuda_install.bat

+)
+
+if not exist "%SRC_DIR%\temp_build\cudnn-11.0-windows-x64-v8.0.0.180.zip" (
+    curl -k -L https://developer.download.nvidia.com/compute/redist/cudnn/v8.0.0/cudnn-11.0-windows-x64-v8.0.0.180.zip --output "%SRC_DIR%\temp_build\cudnn-11.0-windows-x64-v8.0.0.180.zip"


This link requires the credentials. So we have to upload it to S3.

I updated this link to use the latest CUDA 11 and cuDNN release. ~~These links should be working without login, but feel free to upload to S3 to keep consistent with other links.~~

Edit: I lied, this link is not working. Please do upload it to S3. :)

robert-baldwin · 2020-06-29T03:59:14Z

I'm just getting started with pytorch on Ubuntu on WSL2. Will this allow me to conda install pytorch torchvision cudatoolkit=11.0 -c pytorch?

peterjc123 · 2020-06-29T04:08:52Z

I'm just getting started with pytorch on Ubuntu on WSL2. Will this allow me to conda install pytorch torchvision cudatoolkit=11.0 -c pytorch?

Nope, I don't think there will be binaries for CUDA 11 recently because the conda package cudatoolkit=11.0 is not released yet. But you may eventually be able do that no matter you are using WSL or not on Windows.

robert-baldwin · 2020-06-29T19:17:15Z

Understood, thank you. I have one follow-up question if you don't mind.

I tried cloning the pytorch repository and building from source from the master branch which successfully built and loaded in python, but torch.cuda.is_available() #=> False although prior to running python setup.py install I had conda install -c nvidia cudatoolkit which gave me version 11.0.171. Would this branch potentially fix the local build issue?

robert-baldwin · 2020-06-29T19:48:46Z

I appreciate the follow up @zasdfgbnm ! I am on the Windows Insider Preview Build 20152.1000 using Ubuntu 18.04 on WSL2 and have followed NVidia's CUDA Toolkit Documentation here: https://docs.nvidia.com/cuda/wsl-user-guide/index.html

I've verified that my GPU is reachable by watching Windows Task Manager while running scripts from the Cuda Samples in https://github.com/NVIDIA/cuda-samples within WSL2

One thing I've been uncertain about is whether installing CUDA Toolkit via apt as outlined in NVidia's somehow interferes w/ the version installed via conda -- I suspect so, since I saw "cycles in the constraint graph" warnings when I had both installed -- and opted to run the pytorch source build w/ only conda installed version only.

At this point I'm not quite sure where to look to discover whether where my issue may be.

robert-baldwin · 2020-06-29T19:53:22Z

I'm also happy to open an issue or move this conversation to a more appropriate place to get support based on your guidance. I very much appreciate your time and am conscious about unintentionally monopolizing it 😄

zasdfgbnm · 2020-06-29T19:57:24Z

@robert-baldwin I actually don't know, because I haven't tried WSL yet. But if I were you, I would try

nightly build, and
compile from source with CUDA 10.2

peterjc123 · 2020-06-30T17:15:10Z

Understood, thank you. I have one follow-up question if you don't mind.

I tried cloning the pytorch repository and building from source from the master branch which successfully built and loaded in python, but torch.cuda.is_available() #=> False although prior to running python setup.py install I had conda install -c nvidia cudatoolkit which gave me version 11.0.171. Would this branch potentially fix the local build issue?

No, it is only a runtime package, not a source package.

zasdfgbnm · 2020-08-03T19:32:00Z

@peterjc123 This PR is merged now #461, could you please use this to build magma for me? Thanks :)

zasdfgbnm · 2020-08-03T23:05:23Z

This PR is tested by pytorch/pytorch#42482

peterjc123 · 2020-08-03T23:45:43Z

You could refer to the PR I mentioned below. Also magma is already built. See pytorch/pytorch#42420

zasdfgbnm · 2020-08-03T23:49:25Z

@peterjc123 WOW, amazing! I was not aware of that PR. Thanks for working on it!

…10-win

peterjc123 · 2020-08-04T00:01:34Z

windows/internal/cuda_install.bat


 if not exist "%SRC_DIR%\temp_build\cudnn-11.0-windows10-x64-v8.0.2.39.zip" (
-    curl -k -L https://developer.download.nvidia.com/compute/redist/cudnn/v8.0.2/cudnn-11.0-windows10-x64-v8.0.2.39.zip --output "%SRC_DIR%\temp_build\cudnn-11.0-windows10-x64-v8.0.2.39.zip"
+    curl -k -L https://ossci-windows.s3.amazonaws.com/cudnn-11.0-windows-x64-v8.0.2.39.zip --output "%SRC_DIR%\temp_build\cudnn-11.0-windows10-x64-v8.0.2.39.zip"


Do you mean the windows vs windows10? Looks like it is the same here: https://github.com/pytorch/pytorch/pull/42420/files#diff-3fde3b44a644efc229244fe515ea7d39R9

Ok. But the output name is windows10, right? It is confusing.

peterjc123

The changes LGTM now, but would you mind opening a PR to test it, like what you do in pytorch/pytorch#42482?

zasdfgbnm · 2020-08-04T00:10:58Z

@peterjc123 I will reuse pytorch/pytorch#42482 by rerunning the build. This shouldn't be merged until pytorch/pytorch#42482 is green.

peterjc123 · 2020-08-04T00:11:32Z

windows/build_pytorch.bat

 set MAGMA_VERSION=2.5.2
-if "%CUDA_VERSION%" == "80" set MAGMA_VERSION=2.4.0
-if "%CUDA_VERSION%" == "90" set MAGMA_VERSION=2.5.0
+if "%CUDA_VERSION%" == "110" set MAGMA_VERSION=2.5.3


I think you can just update all to 2.5.3. Leaving 92 and 100 to 2.5.2.

peterjc123 · 2020-08-04T00:18:59Z

windows/internal/cuda_install.bat

+)
+
+if not exist "%SRC_DIR%\temp_build\cudnn-11.0-windows10-x64-v8.0.2.39.zip" (
+    curl -k -L https://ossci-windows.s3.amazonaws.com/cudnn-11.0-windows10-x64-v8.0.2.39.zip --output "%SRC_DIR%\temp_build\cudnn-11.0-windows10-x64-v8.0.2.39.zip"


So it should be https://ossci-windows.s3.amazonaws.com/cudnn-11.0-windows-x64-v8.0.2.39.zip.

Yeah, I see. I just checked our cuDNN release page, for 10.2 there is a windows10 and windows, but for 11.0, there is only a windows. Sorry for being confusing, I will change the link and output names to windows.

zasdfgbnm · 2020-08-04T00:55:45Z

The build/install logic should be OK, but I see some compilation errors that I believe fixed in your pytorch/pytorch#42420. I will wait for the merge of your PR and rerun the test again.

zasdfgbnm · 2020-08-05T21:39:36Z

@malfet This should be ready too. This is tested at pytorch/pytorch#42482, which is failing with the same cuSparse error.

malfet · 2020-08-06T17:16:07Z

windows/templates/build_task.yml

+        PY3.5_110:
+          DESIRED_PYTHON: 3.5
+          CUDA_VERSION: 110


We no longer support Python3.5 but I guess it can be done in a separate PR

zasdfgbnm added 20 commits June 8, 2020 15:22

magma for 110

170c6bb

more

0349f2d

more

ffe79b7

Windows

e3e76ed

fix line get

f22ea87

fix copy-paste

f5c09f2

skip cudnn8 now

385d246

install cudnn

cdb8ef0

uncomment

d3ca931

sm80

78f622d

Update Dockerfile

20026d7

Update install_magma.sh

5d02ced

Merge branch 'master' of github.com:pytorch/builder into cuda110

f769225

Merge branch 'cuda110' of github.com:zasdfgbnm/builder into cuda110

17185cc

fix merge

b64340c

revert-some

e3c5244

revert unrelated changes

ba4d66b

revert upstream

85253b6

revert more

f764eaf

update driver for 102

dbc5c92

zasdfgbnm mentioned this pull request Jun 18, 2020

[WIP] CUDA 11.0 support #450

Closed

cudnn

b683b18

zasdfgbnm added 2 commits June 18, 2020 16:13

cuda110.bat

06b6081

revert default

ff7c53a

peterjc123 reviewed Jun 19, 2020

View reviewed changes

zasdfgbnm added 2 commits August 3, 2020 12:27

update link

b8df65c

Merge branch 'master' of github.com:pytorch/builder into cuda110-win

69e6a35

zasdfgbnm mentioned this pull request Aug 3, 2020

[DO NOT MERGE] Test binary build for windows pytorch/pytorch#42482

Closed

Update cuda_install.bat

c69584d

zasdfgbnm added 2 commits August 3, 2020 17:00

Driver, link

0bc9a4f

Merge branch 'cuda110-win' of github.com:zasdfgbnm/builder into cuda1…

de8542c

…10-win

peterjc123 reviewed Aug 4, 2020

View reviewed changes

windows10

cd2509c

peterjc123 approved these changes Aug 4, 2020

View reviewed changes

magma

44237c2

peterjc123 reviewed Aug 4, 2020

View reviewed changes

magma version

b79b115

peterjc123 reviewed Aug 4, 2020

View reviewed changes

windows10 -> windows

2ed8dc4

zasdfgbnm changed the title ~~[WIP] CUDA11 builders for Windows~~ CUDA11 builders for Windows Aug 5, 2020

malfet approved these changes Aug 6, 2020

View reviewed changes

malfet merged commit 492acfa into pytorch:master Aug 6, 2020

zasdfgbnm deleted the cuda110-win branch August 6, 2020 17:19

zasdfgbnm mentioned this pull request Aug 6, 2020

Add missing nvprof_11.0 for Windows builder #483

Merged

Conversation

zasdfgbnm commented Jun 18, 2020

Uh oh!

zasdfgbnm commented Jun 18, 2020

Uh oh!

peterjc123 commented Jun 19, 2020

Uh oh!

peterjc123 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zasdfgbnm Aug 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zasdfgbnm Aug 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-baldwin commented Jun 29, 2020

Uh oh!

peterjc123 commented Jun 29, 2020

Uh oh!

robert-baldwin commented Jun 29, 2020

Uh oh!

robert-baldwin commented Jun 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robert-baldwin commented Jun 29, 2020

Uh oh!

zasdfgbnm commented Jun 29, 2020

Uh oh!

peterjc123 commented Jun 30, 2020

Uh oh!

zasdfgbnm commented Aug 3, 2020

Uh oh!

zasdfgbnm commented Aug 3, 2020

Uh oh!

peterjc123 commented Aug 3, 2020

Uh oh!

zasdfgbnm commented Aug 3, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterjc123 left a comment

Choose a reason for hiding this comment

Uh oh!

zasdfgbnm commented Aug 4, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zasdfgbnm commented Aug 4, 2020

Uh oh!

zasdfgbnm commented Aug 5, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

peterjc123 left a comment •

edited

Loading

zasdfgbnm Aug 3, 2020 •

edited

Loading

zasdfgbnm Aug 3, 2020 •

edited

Loading

robert-baldwin commented Jun 29, 2020 •

edited

Loading