Skip to content

[Docker Release] Test if pytorch was compiled with CUDA before pushing to repo#128852

Closed
atalman wants to merge 3 commits intopytorch:mainfrom
atalman:test_cuda_cpu_docker
Closed

[Docker Release] Test if pytorch was compiled with CUDA before pushing to repo#128852
atalman wants to merge 3 commits intopytorch:mainfrom
atalman:test_cuda_cpu_docker

Conversation

@atalman
Copy link
Contributor

@atalman atalman commented Jun 17, 2024 •
edited
Loading

Related to: #125879
Would check if we are compiled with CUDA before publishing CUDA Docker nightly image

Test

#18 [conda-installs 5/5] RUN IS_CUDA=$(python -c 'import torch ; print(torch.cuda._is_compiled())');    echo "Is torch compiled with cuda: ${IS_CUDA}";     if test "${IS_CUDA}" != "True" -a ! -z "12.4.0"; then 	exit 1;     fi
#18 1.656 Is torch compiled with cuda: False
#18 ERROR: process "/bin/sh -c IS_CUDA=$(python -c 'import torch ; print(torch.cuda._is_compiled())');    echo \"Is torch compiled with cuda: ${IS_CUDA}\";     if test \"${IS_CUDA}\" != \"True\" -a ! -z \"${CUDA_VERSION}\"; then \texit 1;     fi" did not complete successfully: exit code: 1
------
 > [conda-installs 5/5] RUN IS_CUDA=$(python -c 'import torch ; print(torch.cuda._is_compiled())');    echo "Is torch compiled with cuda: ${IS_CUDA}";     if test "${IS_CUDA}" != "True" -a ! -z "12.4.0"; then 	exit 1;     fi:
1.656 Is torch compiled with cuda: False
------
Dockerfile:80
--------------------
  79 |     RUN /opt/conda/bin/pip install torchelastic
  80 | >>> RUN IS_CUDA=$(python -c 'import torch ; print(torch.cuda._is_compiled())');\
  81 | >>>     echo "Is torch compiled with cuda: ${IS_CUDA}"; \
  82 | >>>     if test "${IS_CUDA}" != "True" -a ! -z "${CUDA_VERSION}"; then \
  83 | >>> 	exit 1; \
  84 | >>>     fi
  85 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c IS_CUDA=$(python -c 'import torch ; print(torch.cuda._is_compiled())');    echo \"Is torch compiled with cuda: ${IS_CUDA}\";     if test \"${IS_CUDA}\" != \"True\" -a ! -z \"${CUDA_VERSION}\"; then \texit 1;     fi" did not complete successfully: exit code: 1
(base) [ec2-user@ip-172-30-2-248 pytorch]$ docker buildx build --progress=plain  --platform="linux/amd64"  --target official -t ghcr.io/pytorch/pytorch:2.5.0.dev20240617-cuda12.4-cudnn9-devel --build-arg BASE_IMAGE=nvidia/cuda:12.4.0-devel-ubuntu22.04 --build-arg PYTHON_VERSION=3.11 --build-arg CUDA_VERSION= --build-arg CUDA_CHANNEL=nvidia --build-arg PYTORCH_VERSION=2.5.0.dev20240617 --build-arg INSTALL_CHANNEL=pytorch --build-arg TRITON_VERSION= --build-arg CMAKE_VARS="" .
#0 building with "default" instance using docker driver

Please note looks like we are installing from pytorch rather then nighlty channel on PR hence cuda 12.4 is failing since its not in pytorch channel yet:
https://github.com/pytorch/pytorch/actions/runs/9555354734/job/26338476741?pr=128852

@atalman atalman requested a review from a team as a code owner June 17, 2024 17:02
@pytorch-bot
Copy link

pytorch-bot bot commented Jun 17, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/128852

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures

As of commit 758deb0 with merge base a87d82a (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jun 17, 2024
@atalman
test
9cc60d8 
test

test

test

test

test

test

test

test

test

test

test

test

test

fix

test

fix
@atalman atalman force-pushed the test_cuda_cpu_docker branch from ab75991 to 9cc60d8 Compare June 17, 2024 21:40
@atalman atalman changed the title [Docker Release] Test if pytorch was compiled with CUDA before publishing latest tag [Docker Release] Test if pytorch was compiled with CUDA before pushing to repo Jun 17, 2024
malfet
malfet reviewed Jun 17, 2024
View reviewed changes
Dockerfile
esac && \
/opt/conda/bin/conda clean -ya
RUN /opt/conda/bin/pip install torchelastic
RUN IS_CUDA=$(python -c 'import torch ; print(torch.cuda._is_compiled())'); \
Copy link
Contributor

@malfet malfet Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if user wants to build CPU docker version? Should we somehow map IS_CUDA to say INSTALL_CHANNEL to figure out whether we want cu or cpu versions?

Copy link
Contributor Author

@atalman atalman Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added check

if test "${IS_CUDA}" != "True" -a ! -z "${CUDA_VERSION}"; then \
        exit 1; \
fi

@atalman
Copy link
Contributor Author

atalman commented Jun 17, 2024

@pytorchmergebot merge -f "lint is green, cuda 12.4 failures are expected"

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jun 17, 2024

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malfet malfet malfet approved these changes

Assignees

No one assigned

Labels

Merged topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@atalman @pytorchmergebot @malfet