Skip to content

[SymmMem] Avoid library mismatch in CMake search#157836

Closed
kwen2501 wants to merge 1 commit intogh/kwen2501/193/basefrom
gh/kwen2501/193/head
Closed

[SymmMem] Avoid library mismatch in CMake search#157836
kwen2501 wants to merge 1 commit intogh/kwen2501/193/basefrom
gh/kwen2501/193/head

Conversation

@kwen2501
Copy link
Collaborator

@kwen2501 kwen2501 commented Jul 8, 2025 •
edited
Loading

Stack from ghstack (oldest at bottom):

Before, if NVSHMEM is installed at BOTH system location (e.g. /usr/local) and conda location (e.g. /path/to/conda/lib/python3.10/site-packages/nvidia/nvshmem, there can be a mismatch in where host lib and device lib are found:

-- NVSHMEM_HOME set to:  ''
-- NVSHMEM wheel installed at:  '.conda/envs/pytorch-3.10/lib/python3.10/site-packages/nvidia/nvshmem'
-- NVSHMEM_HOST_LIB:  '/usr/local/lib/libnvshmem_host.so'
-- NVSHMEM_DEVICE_LIB:  '.conda/envs/pytorch-3.10/lib/python3.10/site-packages/nvidia/nvshmem/lib/libnvshmem_device.a'
-- NVSHMEM_INCLUDE_DIR:  '.conda/envs/pytorch-3.10/lib/python3.10/site-packages/nvidia/nvshmem/include'

The reason is that CMake prioritize name search over dir search. In the script below, CMake will search all locations for libnvshmem_host.so first, before it searches for .so.3.

find_library(NVSHMEM_HOST_LIB
      # In pip install case, the lib suffix is `.so.3` instead of `.so`
      NAMES nvshmem_host nvshmem_host.so.3
      HINTS $ENV{NVSHMEM_HOME} ${NVSHMEM_PY_DIR}
      PATH_SUFFIXES lib lib64 cuda/lib cuda/lib64 lib/x64)

This PR adds the NAMES_PER_DIR flag, according to CMake's doc:

The NAMES_PER_DIR option tells this command to consider one directory at a time and search for all names in it.

After this PR:

-- NVSHMEM_HOME set to:  ''
-- NVSHMEM wheel installed at:  '.conda/envs/pytorch-3.10/lib/python3.10/site-packages/nvidia/nvshmem'
-- NVSHMEM_HOST_LIB:  '.conda/envs/pytorch-3.10/lib/python3.10/site-packages/nvidia/nvshmem/lib/libnvshmem_host.so.3'
-- NVSHMEM_DEVICE_LIB:  '.conda/envs/pytorch-3.10/lib/python3.10/site-packages/nvidia/nvshmem/lib/libnvshmem_device.a'
-- NVSHMEM_INCLUDE_DIR:  '.conda/envs/pytorch-3.10/lib/python3.10/site-packages/nvidia/nvshmem/include'

@kwen2501
Update
cfa5e4d 
[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Jul 8, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157836

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit cfa5e4d with merge base a6fab82 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jul 8, 2025
kwen2501 added a commit that referenced this pull request Jul 8, 2025
@kwen2501
[SymmMem] Avoid library mismatch in CMake search
d6e61d6 
ghstack-source-id: 352bf85
Pull-Request-resolved: #157836
@kwen2501 kwen2501 added release notes: distributed (symm_mem) release note label for symmetric memory and removed topic: not user facing topic category labels Jul 8, 2025
@kwen2501 kwen2501 requested review from Skylion007 and fduwjj July 8, 2025 20:51
@kwen2501 kwen2501 added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 8, 2025
@fduwjj
Copy link
Contributor

fduwjj commented Jul 10, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 10, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 10, 2025

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

@fduwjj
Copy link
Contributor

fduwjj commented Jul 11, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 11, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 12, 2025

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

@kwen2501
Copy link
Collaborator Author

kwen2501 commented Jul 14, 2025

@pytorchbot merge -f "timeout"

@pytorch-bot
Copy link

pytorch-bot bot commented Jul 14, 2025

You need to provide a reason for using force merge, in the format @pytorchbot merge -f 'Explanation'.
The explanation needs to be clear on why this is needed. Here are some good examples:

  • Bypass checks due to unrelated upstream failures from ...
  • This is a minor fix to ..., which shouldn't break anything
  • This is pre-tested in a previous CI run
  • Bypass flaky ... check

@kwen2501
Copy link
Collaborator Author

kwen2501 commented Jul 14, 2025

@pytorchbot merge -f "all tests passed but merge timed out"

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jul 14, 2025

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the gh/kwen2501/193/head branch August 14, 2025 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fegin fegin fegin approved these changes

@fduwjj fduwjj fduwjj approved these changes

@Skylion007 Skylion007 Awaiting requested review from Skylion007

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: distributed (symm_mem) release note label for symmetric memory

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kwen2501 @fduwjj @pytorchmergebot @fegin