Skip to content

[CI][CD] Fix install_nvshem function#159907

Closed
malfet wants to merge 2 commits intomainfrom
malfet-patch-6
Closed

[CI][CD] Fix install_nvshem function#159907
malfet wants to merge 2 commits intomainfrom
malfet-patch-6

Conversation

@malfet
Copy link
Contributor

@malfet malfet commented Aug 5, 2025 •
edited
Loading

When one builds CD docker, all CUDA dependencies must be installed into /usr/local/cuda/ folder

Test plan: Looks at the binary build logs, for example here:

2025-08-06T05:58:00.7347471Z -- NVSHMEM_HOME set to:  ''
2025-08-06T05:58:00.7348378Z -- NVSHMEM wheel installed at:  ''
2025-08-06T05:58:00.7392528Z -- NVSHMEM_HOST_LIB:  '/usr/local/cuda/lib64/libnvshmem_host.so'
2025-08-06T05:58:00.7393251Z -- NVSHMEM_DEVICE_LIB:  '/usr/local/cuda/lib64/libnvshmem_device.a'
2025-08-06T05:58:00.7393792Z -- NVSHMEM_INCLUDE_DIR:  '/usr/local/cuda/include'
2025-08-06T05:58:00.7394252Z -- NVSHMEM found, building with NVSHMEM support

@malfet
[CI][CD] Fix install_nvshem function
ea7c28c 
When one builds CD docker, all CUDA dependencies must be installed into `/usr/local/cuda/` folder
@malfet malfet requested a review from jeffdaily as a code owner August 5, 2025 22:07
@pytorch-bot
Copy link

pytorch-bot bot commented Aug 5, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/159907

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ You can merge normally! (1 Unrelated Failure)

As of commit 4076d96 with merge base 64cc6f0 (image):

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

  • pull / linux-jammy-py3_9-clang9-xla / test (xla, 1, 1, lf.linux.12xlarge, unstable) (gh) (#158876)
    /var/lib/jenkins/workspace/xla/torch_xla/csrc/runtime/BUILD:476:14: Compiling torch_xla/csrc/runtime/xla_util_test.cpp failed: (Exit 1): gcc failed: error executing CppCompile command (from target //torch_xla/csrc/runtime:xla_util_test) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 229 arguments skipped)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@malfet malfet requested review from Skylion007 and removed request for jeffdaily August 5, 2025 22:07
@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Aug 5, 2025
@malfet malfet added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 5, 2025
malfet
malfet commented Aug 6, 2025
View reviewed changes
@malfet
Copy link
Contributor Author

malfet commented Aug 6, 2025

@pytorchbot merge -f "Lint + binary builds are green"

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Aug 6, 2025

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@ngimel ngimel mentioned this pull request Aug 6, 2025
Closed
This was referenced Aug 12, 2025
Closed
Closed
@kwen2501
Copy link
Collaborator

kwen2501 commented Aug 12, 2025 •
edited
Loading

Thanks for the fix! Indeed I wouldn't know CD has a different Docker build way than CI.

Just a minor thing -- I am actually a bit surprised that CMake can still find the header file after this move.
The following is the CMake scripts for the header search:

pytorch/caffe2/CMakeLists.txt

Lines 1014 to 1017 in 8e6a313

find_path(NVSHMEM_INCLUDE_DIR
NAMES nvshmem.h
HINTS $ENV{NVSHMEM_HOME}/include ${NVSHMEM_PY_DIR}/include
DOC "The location of NVSHMEM headers.")

Both NVSHMEM_HOME and NVSHMEM_PY_DIR are empty in your test above.
Yet CMake is still able to find the header under /usr/local/cuda. That seems to indicate that /usr/local/cuda is now one of the default search path of find_path?

@github-actions github-actions bot deleted the malfet-patch-6 branch September 12, 2025 02:07
markc-614 pushed a commit to markc-614/pytorch that referenced this pull request Sep 17, 2025
@malfet @markc-614
[CI][CD] Fix install_nvshem function (pytorch#159907)
2d3ff0d 
When one builds CD docker, all CUDA dependencies must be installed into `/usr/local/cuda/` folder

Test plan: Looks at the binary build logs, for example [here](https://github.com/pytorch/pytorch/actions/runs/16768141521/job/47477380147?pr=159907):
```
2025-08-06T05:58:00.7347471Z -- NVSHMEM_HOME set to:  ''
2025-08-06T05:58:00.7348378Z -- NVSHMEM wheel installed at:  ''
2025-08-06T05:58:00.7392528Z -- NVSHMEM_HOST_LIB:  '/usr/local/cuda/lib64/libnvshmem_host.so'
2025-08-06T05:58:00.7393251Z -- NVSHMEM_DEVICE_LIB:  '/usr/local/cuda/lib64/libnvshmem_device.a'
2025-08-06T05:58:00.7393792Z -- NVSHMEM_INCLUDE_DIR:  '/usr/local/cuda/include'
2025-08-06T05:58:00.7394252Z -- NVSHMEM found, building with NVSHMEM support
```

Pull Request resolved: pytorch#159907
Approved by: https://github.com/Skylion007, https://github.com/ngimel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Skylion007 Skylion007 Skylion007 approved these changes

@ngimel ngimel ngimel approved these changes

Assignees

No one assigned

Labels

ciflow/h100-symm-mem ciflow/trunk Trigger trunk jobs on your pull request Merged topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@malfet @pytorchmergebot @kwen2501 @Skylion007 @ngimel