Add CI workflow for tests that requires pytorch CUDA. by vanbasten23 · Pull Request #7073 · pytorch/xla

vanbasten23 · 2024-05-16T22:40:45Z

This PR adds a new CI workflow that build pytorch with CUDA enabled from source, build pytorch/xla with CUDA enabled from source, then run tests. The intention is to run tests that requires pytorch with CUDA.

In detail, this PR add 2 more jobs to .github/workflows/build_and_test.yml

build-torch-with-cuda-xla-with-cuda: build pytorch with cuda and build pytorch/xla (It's ok to build torch_xla with cuda enabled now because it takes 2537s to build pytorch and takes 352s to build torch_xla).
test-cuda-with-pytorch-cuda-enabled: only run the tests that requires pytorch CUDA.

vanbasten23 · 2024-05-21T17:13:55Z

Note to myself: it seems that BUILD XLA CUDA plugin requires the env var USE_CUDA=0. Otherwise, it'll fails with an error

      [5,777 / 9,258] Compiling tsl/platform/default/logging.cc; 0s local, remote-cache ... (17 actions, 2 running)
      ERROR: /github/home/.cache/bazel/_bazel_root/197a057057a49e5811107144e2d78508/external/xla/xla/service/gpu/BUILD:4486:13: Compiling xla/service/gpu/buffer_comparator.cu.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command (from target @xla//xla/service/gpu:buffer_comparator_kernel) external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/xla/xla/service/gpu/_objs/buffer_comparator_kernel/buffer_comparator.cu.pic.d ... (remaining 65 arguments skipped)
      nvcc fatal   : Unsupported gpu architecture 'compute_35'
      Target @xla//xla/pjrt/c:pjrt_c_api_gpu_plugin.so failed to build
      Use --verbose_failures to see the command lines of failed build steps.
      INFO: Elapsed time: 40.997s, Critical Path: 1.96s
      INFO: 5806 processes: 949 remote cache hit, 4857 internal.
      FAILED: Build did NOT complete successfully
      FAILED: Build did NOT complete successfully
      INFO: Streaming build results to: https://source.cloud.google.com/results/invocations/bba75106-7be5-4ad8-9a83-b3dd6222c3e1
      Traceback (most recent call last):
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-e97mw0k_/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-e97mw0k_/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-e97mw0k_/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 11, in <module>
        File "/__w/xla/xla/pytorch/xla/plugins/cuda/../../build_util.py", line 67, in bazel_build
          subprocess.check_call(bazel_argv, stdout=sys.stdout, stderr=sys.stderr)
        File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['bazel', 'build', '@xla//xla/pjrt/c:pjrt_c_api_gpu_plugin.so', '--symlink_prefix=/__w/xla/xla/pytorch/xla/plugins/cuda/bazel-', '--config=remote_cache', '--jobs=16', '--config=cuda', '--remote_default_exec_properties=cache-silo-key=dev']' returned non-zero exit status 1.
      error: subprocess-exited-with-error

vanbasten23 · 2024-05-23T15:15:46Z

Seems without installing the cuda plugin, the tests would fail with error https://gist.github.com/vanbasten23/7dd6ddeaad93843e57653990c43cf476

vanbasten23 · 2024-05-23T21:02:34Z

    cuda_t = torch.arange(25, device=torch.device('cuda')).reshape(5, 5)

    t1 = cuda_t.t()
+    print('xw32 t1.device=', t1.device)


will-cromar · 2024-05-23T21:12:00Z

+      BAZEL_REMOTE_CACHE: 1
+      BUILD_CPP_TESTS: 1
+    steps:
+      - name: Setup gcloud


This setup is repetitive. I was already on the fence about encapsulating it in a new action (since it already appears the build action, test actions, and the docs push. If we add another copy, it really should be encapsulated so we don't have to update a bunch of places at once.

will-cromar · 2024-05-23T21:14:50Z

+          name: torch-with-cuda-xla-with-cuda-wheels
+          path: /tmp/wheels/
+          pattern: torch-*.whl 
+      - name: Fetch CUDA plugin


This setup is repetitive. I was already on the fence about encapsulating it in a new action (since it already appears the build action, test actions, and the docs push). If we add another copy that's more-or-less identical, it really should be encapsulated so we don't have to update a bunch of places at once.

Stepping back, is there a way to merge this with _test.yml? You would need to add a parameter for some of the test groups to install the torch CUDA build.

I can see your point. Let me give it a try.

will-cromar · 2024-05-23T21:18:03Z

+        shell: bash
+        run: |
+          cd pytorch/xla/infra/ansible
+          ansible-playbook playbook.yaml -vvv -e "stage=build arch=amd64 accelerator=cuda cuda_compute_capabilities=5.2,7.5 src_root=${GITHUB_WORKSPACE} build_cpp_tests=1 git_versioned_xla_build=1 cache_suffix=-ci build_pytorch_with_cuda=1" --skip-tags=fetch_srcs,install_deps


I know this goes against my philosophy of "put everything in ansible", but what if you just build the PyTorch CUDA wheel directly here? I don't think we should build multiple copies of torch_xla and torchvision.

Building PyTorch with CUDA support is only part of our CI workflow, and it will never be part of our release workflow. It's okay in my mind to just directly USE_CUDA=1 python setup.py bdist_wheel here and upload only the torch GPU wheel as an artifact.

The test workflow can then use the same torch-xla, torch-xla-cuda-plugin, and torchvision.

bhavya01 · 2024-05-25T00:28:15Z

Thanks for working on this. Added myself as a reviewer as I also need this for Triton tests.

This reverts commit 792fd61.

This reverts commit 59cabde.

…flow.

… test.

…-artifact@v4's pattern functionality does not work so a latter actions/download-artifact@v4 override the previous torch cuda wheel.

vanbasten23 · 2024-05-30T15:01:03Z

close it in favor of #7140

vanbasten23 mentioned this pull request May 16, 2024

Add dlpack support #7025

Merged

vanbasten23 force-pushed the xiowei/addCIWorkflow branch 3 times, most recently from 5f27b23 to f79aee7 Compare May 21, 2024 04:50

vanbasten23 force-pushed the xiowei/addCIWorkflow branch 5 times, most recently from 9a251c7 to 91abb38 Compare May 22, 2024 23:43

vanbasten23 force-pushed the xiowei/addCIWorkflow branch 2 times, most recently from 9c59552 to 723f9d1 Compare May 23, 2024 17:06

vanbasten23 marked this pull request as ready for review May 23, 2024 17:07

vanbasten23 requested review from JackCaoG, lsy323 and will-cromar as code owners May 23, 2024 17:07

vanbasten23 requested a review from ysiraichi May 23, 2024 17:12

vanbasten23 commented May 23, 2024

View reviewed changes

will-cromar reviewed May 23, 2024

View reviewed changes

vanbasten23 force-pushed the xiowei/addCIWorkflow branch 2 times, most recently from 7576407 to bdb37d7 Compare May 23, 2024 23:27

bhavya01 self-requested a review May 25, 2024 00:27

vanbasten23 force-pushed the xiowei/addCIWorkflow branch from cbd190a to 1e910fb Compare May 29, 2024 00:14

vanbasten23 added 5 commits May 29, 2024 13:56

add workflow for test that requires pytorch CUDA.

e6cf5cb

fix error Failed to detect a default CUDA architecture.

dba97ce

errofix r Fai:led to detect a default CUDA architecture.

d43152e

fix again

998a3ed

fix the test

88d1d8e

vanbasten23 added 26 commits May 29, 2024 13:56

fix syntax

74cb998

fix syntax

c0ea174

change build file

15f6e80

chnage build_and_test.yml

3ce6603

Revert "chnage build_and_test.yml"

4c0464c

This reverts commit 792fd61.

Revert "Revert "chnage build_and_test.yml""

f3b4433

This reverts commit 59cabde.

fix error: require-pytorch-cuda is not defined in the referenced work…

cdc1082

…flow.

fix more

93a2cfe

syntax error

e89ea83

not set build_cpp_tests=1

ca6cb96

enable cpp test

f1d2851

In the last commit, the new workflow is green. This commit set up the…

adc3539

… test.

try to fix the cuda plugin build

5664610

use option --gpus all when running the docker container

9e8385d

check if cuda is available

1275b92

print if cuda is available in the build phase

293c0d9

not use the non-cuda torch_xla wheel at all. Somehow actions/download…

df6534b

…-artifact@v4's pattern functionality does not work so a latter actions/download-artifact@v4 override the previous torch cuda wheel.

removed unwanted

d4859bb

fix a test

d706a46

fix comment

e0a9c8e

merge the test script

450f7f2

install torch cuda

5306fec

fix error: ls: cannot access '/dist/': No such file or directory

d852468

fix error

c461010

fix error that there are 2 torch wheels

285fb1d

add a new env

bcd007c

vanbasten23 force-pushed the xiowei/addCIWorkflow branch from 1e910fb to bcd007c Compare May 29, 2024 13:56

vanbasten23 closed this May 30, 2024

vanbasten23 mentioned this pull request May 30, 2024

Add a CI workflow for tests that requires pytorch CUDA. #7140

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI workflow for tests that requires pytorch CUDA.#7073

Add CI workflow for tests that requires pytorch CUDA.#7073
vanbasten23 wants to merge 34 commits intomasterfrom
xiowei/addCIWorkflow

vanbasten23 commented May 16, 2024 •

edited

Loading

Uh oh!

vanbasten23 commented May 21, 2024 •

edited

Loading

Uh oh!

vanbasten23 commented May 23, 2024

Uh oh!

vanbasten23 May 23, 2024

Uh oh!

will-cromar May 23, 2024

Uh oh!

will-cromar May 23, 2024

Uh oh!

vanbasten23 May 23, 2024

Uh oh!

will-cromar May 23, 2024

Uh oh!

bhavya01 commented May 25, 2024

Uh oh!

vanbasten23 commented May 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

vanbasten23 commented May 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanbasten23 commented May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanbasten23 commented May 23, 2024

Uh oh!

vanbasten23 May 23, 2024

Choose a reason for hiding this comment

Uh oh!

will-cromar May 23, 2024

Choose a reason for hiding this comment

Uh oh!

will-cromar May 23, 2024

Choose a reason for hiding this comment

Uh oh!

vanbasten23 May 23, 2024

Choose a reason for hiding this comment

Uh oh!

will-cromar May 23, 2024

Choose a reason for hiding this comment

Uh oh!

bhavya01 commented May 25, 2024

Uh oh!

vanbasten23 commented May 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vanbasten23 commented May 16, 2024 •

edited

Loading

vanbasten23 commented May 21, 2024 •

edited

Loading