Skip to content

Cray support: use linux platform for newer craype versions#29392

Merged
tgamblin merged 7 commits intodevelopfrom
features/use-linux-for-newer-craype
Nov 4, 2022
Merged

Cray support: use linux platform for newer craype versions#29392
tgamblin merged 7 commits intodevelopfrom
features/use-linux-for-newer-craype

Conversation

@becker33
Copy link
Copy Markdown
Member

@becker33 becker33 commented Mar 7, 2022

Newer versions of the CrayPE for EX systems have standalone compiler executables for CCE and compiler wrappers for Cray MPICH. With those, we can treat the cray systems as part of the linux platform rather than having a separate cray platform.

This PR:

  • Changes cray platform detection to ignore EX systems with Craype version 21.10 or later
  • Changes the cce compiler to be detectable via paths
  • Changes the spack compiler wrapper to understand the executable names for the standalone cce compiler (craycc, crayCC, crayftn).

@haampie
Copy link
Copy Markdown
Member

haampie commented Mar 7, 2022

FWIW, there are still many issues and unresolved tickets with Cray PE's without modules.

This is 21.12:
$ ./libtree_x86_64 /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpich.so /opt/cray/pe/mpich/8.1.13/ofi/nvidia/20.7/lib/libmpich.so /opt/cray/pe/mpich/8.1.13/ofi/cray/10.0/lib/libmpich.so /opt/cray/pe/mpich/8.1.13/ofi/aocc/3.0/lib/libmpich.so /opt/cray/pe/mpich/8.1.13/ofi/intel/19.0/lib/libmpich.so
libmpi_gnu_91.so.12 
├── libatomic.so.1 [runpath]
├── libquadmath.so.0 [runpath]
├── libgfortran.so.5 [runpath]
│   └── libquadmath.so.0 [runpath]
├── libpmi.so.0 [ld.so.conf]
│   └── libpmi2.so.0 [ld.so.conf]
│       └── libpals.so.0 [ld.so.conf]
├── libpmi2.so.0 [ld.so.conf]
├── librt.so.1 [default path]
└── libfabric.so.1 not found
    ┊ Paths considered in this order:
    ┊ 1. rpath is skipped because runpath was set
    ┊ 2. LD_LIBRARY_PATH was not set
    ┊ 3. runpath:
    ┊    /opt/cray/pe/gcc-libs
    ┊    /opt/cray/gcc-libs
    ┊ 4. ld config files:
    ┊    /usr/local/lib64
    ┊    /usr/local/lib
    ┊    /opt/cray/pe/lib64
    ┊    /opt/cray/pe/lib64/cce
    ┊    /opt/cray/xpmem/default/lib64
    ┊ 5. Standard paths:
    ┊    /lib
    ┊    /lib64
    ┊    /usr/lib
    ┊    /usr/lib64
Error [/opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpich.so]: Not all dependencies were found
libmpi_nvidia.so.12 
├── libatomic.so.1 [runpath]
├── libpmi2.so.0 [runpath]
│   └── libresolv.so.2 [default path]
├── libpmi.so.0 [runpath]
│   ├── libslurm_pmi.so [runpath]
│   │   └── libresolv.so.2 [default path]
│   └── libresolv.so.2 [default path]
├── librt.so.1 [default path]
├── libfabric.so.1 not found
├── libnvomp.so not found
├── libnvc.so not found
├── libnvcpumath.so not found
└── libnvf.so not found
    ┊ Paths considered in this order:
    ┊ 1. rpath is skipped because runpath was set
    ┊ 2. LD_LIBRARY_PATH was not set
    ┊ 3. runpath:
    ┊    /opt/cray/pe/gcc-libs
    ┊    /opt/cray/gcc-libs
    ┊    /workspace/pebuildenv/opt/nvidia/Linux_x86_64/20.7/compilers/lib
    ┊    /usr/lib64/gcc/x86_64-suse-linux/7/../../../../lib64
    ┊ 4. ld config files:
    ┊    /usr/local/lib64
    ┊    /usr/local/lib
    ┊    /opt/cray/pe/lib64
    ┊    /opt/cray/pe/lib64/cce
    ┊    /opt/cray/xpmem/default/lib64
    ┊ 5. Standard paths:
    ┊    /lib
    ┊    /lib64
    ┊    /usr/lib
    ┊    /usr/lib64
Error [/opt/cray/pe/mpich/8.1.13/ofi/nvidia/20.7/lib/libmpich.so]: Not all dependencies were found
libmpi_cray.so.12 
├── libpmi2.so.0 [ld.so.conf]
├── libpmi.so.0 [ld.so.conf]
├── libcsup.so.1 [ld.so.conf]
│   └── librt.so.1 [default path]
├── libfi.so.1 [ld.so.conf]
│   ├── libmodules.so.1 [runpath]
│   │   ├── libfi.so.1 [runpath]
│   │   ├── libu.so.1 [runpath]
│   │   │   ├── libcsup.so.1 [runpath]
│   │   │   ├── libquadmath.so.0 [runpath]
│   │   │   └── librt.so.1 [default path]
│   │   ├── libf.so.1 [runpath]
│   │   │   ├── libu.so.1 [runpath]
│   │   │   ├── libcsup.so.1 [runpath]
│   │   │   └── librt.so.1 [default path]
│   │   └── librt.so.1 [default path]
│   ├── libquadmath.so.0 [runpath]
│   ├── libcraymath.so.1 [runpath]
│   │   ├── libquadmath.so.0 [runpath]
│   │   ├── libu.so.1 [runpath]
│   │   ├── libgfortran.so.5 [runpath]
│   │   └── librt.so.1 [default path]
│   ├── libu.so.1 [runpath]
│   ├── libf.so.1 [runpath]
│   └── librt.so.1 [default path]
├── libmodules.so.1 [ld.so.conf]
├── libquadmath.so.0 [ld.so.conf]
├── libcraymath.so.1 [ld.so.conf]
├── libf.so.1 [ld.so.conf]
├── libu.so.1 [ld.so.conf]
├── librt.so.1 [default path]
├── libatomic.so.1 [default path]
└── libfabric.so.1 not found
    ┊ Paths considered in this order:
    ┊ 1. rpath:
    ┊ 2. LD_LIBRARY_PATH was not set
    ┊ 3. runpath was not set
    ┊ 4. ld config files:
    ┊    /usr/local/lib64
    ┊    /usr/local/lib
    ┊    /opt/cray/pe/lib64
    ┊    /opt/cray/pe/lib64/cce
    ┊    /opt/cray/xpmem/default/lib64
    ┊ 5. Standard paths:
    ┊    /lib
    ┊    /lib64
    ┊    /usr/lib
    ┊    /usr/lib64
Error [/opt/cray/pe/mpich/8.1.13/ofi/cray/10.0/lib/libmpich.so]: Not all dependencies were found
libmpi_aocc.so.12 
├── libatomic.so.1 [runpath]
├── libquadmath.so.0 [runpath]
├── libpgmath.so [runpath]
├── libflang.so [runpath]
│   ├── libquadmath.so.0 [ld.so.conf]
│   ├── librt.so.1 [default path]
│   └── libflangrti.so not found
│       ┊ Paths considered in this order:
│       ┊ 1. rpath:
│       ┊ 2. LD_LIBRARY_PATH was not set
│       ┊ 3. runpath was not set
│       ┊ 4. ld config files:
│       ┊    /usr/local/lib64
│       ┊    /usr/local/lib
│       ┊    /opt/cray/pe/lib64
│       ┊    /opt/cray/pe/lib64/cce
│       ┊    /opt/cray/xpmem/default/lib64
│       ┊ 5. Standard paths:
│       ┊    /lib
│       ┊    /lib64
│       ┊    /usr/lib
│       ┊    /usr/lib64
├── libompstub.so [runpath]
│   └── libquadmath.so.0 [ld.so.conf]
├── libflangrti.so [runpath]
│   ├── libquadmath.so.0 [ld.so.conf]
│   ├── libomp.so not found
│   └── libpgmath.so not found
│       ┊ Paths considered in this order:
│       ┊ 1. rpath:
│       ┊ 2. LD_LIBRARY_PATH was not set
│       ┊ 3. runpath was not set
│       ┊ 4. ld config files:
│       ┊    /usr/local/lib64
│       ┊    /usr/local/lib
│       ┊    /opt/cray/pe/lib64
│       ┊    /opt/cray/pe/lib64/cce
│       ┊    /opt/cray/xpmem/default/lib64
│       ┊ 5. Standard paths:
│       ┊    /lib
│       ┊    /lib64
│       ┊    /usr/lib
│       ┊    /usr/lib64
├── libpmi.so.0 [ld.so.conf]
├── libpmi2.so.0 [ld.so.conf]
├── librt.so.1 [default path]
└── libfabric.so.1 not found
    ┊ Paths considered in this order:
    ┊ 1. rpath is skipped because runpath was set
    ┊ 2. LD_LIBRARY_PATH was not set
    ┊ 3. runpath:
    ┊    /opt/cray/pe/gcc-libs
    ┊    /opt/cray/gcc-libs
    ┊    /opt/AMD/aocc-compiler-3.0.0/bin/../lib
    ┊ 4. ld config files:
    ┊    /usr/local/lib64
    ┊    /usr/local/lib
    ┊    /opt/cray/pe/lib64
    ┊    /opt/cray/pe/lib64/cce
    ┊    /opt/cray/xpmem/default/lib64
    ┊ 5. Standard paths:
    ┊    /lib
    ┊    /lib64
    ┊    /usr/lib
    ┊    /usr/lib64
Error [/opt/cray/pe/mpich/8.1.13/ofi/aocc/3.0/lib/libmpich.so]: Not all dependencies were found
libmpi_intel.so.12 
├── libpmi2.so.0 [ld.so.conf]
├── libpmi.so.0 [ld.so.conf]
├── librt.so.1 [default path]
├── libatomic.so.1 [default path]
├── libfabric.so.1 not found
├── libifcoremt.so.5 not found
├── libintlc.so.5 not found
├── libsvml.so not found
├── libifport.so.5 not found
└── libimf.so not found
    ┊ Paths considered in this order:
    ┊ 1. rpath:
    ┊ 2. LD_LIBRARY_PATH was not set
    ┊ 3. runpath was not set
    ┊ 4. ld config files:
    ┊    /usr/local/lib64
    ┊    /usr/local/lib
    ┊    /opt/cray/pe/lib64
    ┊    /opt/cray/pe/lib64/cce
    ┊    /opt/cray/xpmem/default/lib64
    ┊ 5. Standard paths:
    ┊    /lib
    ┊    /lib64
    ┊    /usr/lib
    ┊    /usr/lib64
Error [/opt/cray/pe/mpich/8.1.13/ofi/intel/19.0/lib/libmpich.so]: Not all dependencies were found
Any clue if they are being resolved?

@becker33 becker33 force-pushed the features/use-linux-for-newer-craype branch from 696d751 to 5c32958 Compare March 7, 2022 23:28
@spackbot-app spackbot-app bot added the core PR affects Spack core functionality label Aug 29, 2022
@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 29, 2022

It looks like you had an issue with style checks! I can help with that if you ask me! Just say:

@spackbot fix style

... and I'll try to fix style and push a commit to your fork with the fix.

Alternatively, you can run:

$ spack style --fix

And then update the pull request here.

1 similar comment
@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 29, 2022

It looks like you had an issue with style checks! I can help with that if you ask me! Just say:

@spackbot fix style

... and I'll try to fix style and push a commit to your fork with the fix.

Alternatively, you can run:

$ spack style --fix

And then update the pull request here.

@becker33
Copy link
Copy Markdown
Member Author

@spackbot fix style

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 29, 2022

Let me see if I can fix that for you!

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 29, 2022

I was able to run spack style --fix for you!

spack style --fix
==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/compilers/cce.py
  lib/spack/spack/platforms/cray.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
Success: no issues found in 558 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/compilers/cce.py
reformatted lib/spack/spack/platforms/cray.py
All done! ✨ 🍰 ✨
2 files reformatted.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/compilers/cce.py:6: [F401] 'os' imported but unused
  flake8 found errors
Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with isort fixes.

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 29, 2022

It looks like you had an issue with style checks! I can help with that if you ask me! Just say:

@spackbot fix style

... and I'll try to fix style and push a commit to your fork with the fix.

Alternatively, you can run:

$ spack style --fix

And then update the pull request here.

1 similar comment
@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 29, 2022

It looks like you had an issue with style checks! I can help with that if you ask me! Just say:

@spackbot fix style

... and I'll try to fix style and push a commit to your fork with the fix.

Alternatively, you can run:

$ spack style --fix

And then update the pull request here.

@spackbot-app spackbot-app bot added the tests General test capability(ies) label Aug 30, 2022
@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 30, 2022

It looks like you had an issue with style checks! I can help with that if you ask me! Just say:

@spackbot fix style

... and I'll try to fix style and push a commit to your fork with the fix.

Alternatively, you can run:

$ spack style --fix

And then update the pull request here.

1 similar comment
@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 30, 2022

It looks like you had an issue with style checks! I can help with that if you ask me! Just say:

@spackbot fix style

... and I'll try to fix style and push a commit to your fork with the fix.

Alternatively, you can run:

$ spack style --fix

And then update the pull request here.

@becker33
Copy link
Copy Markdown
Member Author

@spackbot fix style

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 30, 2022

Let me see if I can fix that for you!

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Aug 30, 2022

I was able to run spack style --fix for you!

spack style --fix
==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/compilers/cce.py
  lib/spack/spack/platforms/cray.py
  lib/spack/spack/test/architecture.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
Success: no issues found in 558 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/test/architecture.py
All done! ✨ 🍰 ✨
1 file reformatted, 2 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean
Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with isort fixes.

tgamblin
tgamblin previously approved these changes Nov 2, 2022
@becker33 becker33 force-pushed the features/use-linux-for-newer-craype branch from 3bcb5f6 to 1194eaa Compare November 2, 2022 21:17
@becker33
Copy link
Copy Markdown
Member Author

becker33 commented Nov 3, 2022

@spackbot fix style

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Nov 3, 2022

Let me see if I can fix that for you!

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Nov 3, 2022

I was able to run spack style --fix for you!

spack style --fix
==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/spack/compilers/cce.py
  lib/spack/spack/platforms/cray.py
  lib/spack/spack/test/architecture.py
  var/spack/repos/builtin/packages/aluminum/package.py
  var/spack/repos/builtin/packages/esmf/package.py
  var/spack/repos/builtin/packages/mpich/package.py
  var/spack/repos/builtin/packages/mvapich2/package.py
  var/spack/repos/builtin/packages/opencoarrays/package.py
  var/spack/repos/builtin/packages/scorep/package.py
  var/spack/repos/builtin/packages/xsdk/package.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
Success: no issues found in 566 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/compilers/cce.py
reformatted lib/spack/spack/test/architecture.py
reformatted var/spack/repos/builtin/packages/scorep/package.py
All done! ✨ 🍰 ✨
3 files reformatted, 7 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean
Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

@becker33
Copy link
Copy Markdown
Member Author

becker33 commented Nov 3, 2022

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Nov 3, 2022

I've started that pipeline for you!

@tgamblin
Copy link
Copy Markdown
Member

tgamblin commented Nov 4, 2022

Pipeline failure is spurious and this PR doesn't affect that part of the pipeline.

@tgamblin tgamblin merged commit 53fbaa5 into develop Nov 4, 2022
@tgamblin tgamblin deleted the features/use-linux-for-newer-craype branch November 4, 2022 21:52
charmoniumQ pushed a commit to charmoniumQ/spack that referenced this pull request Nov 19, 2022
Newer versions of the CrayPE for EX systems have standalone compiler executables for CCE and compiler wrappers for Cray MPICH. With those, we can treat the cray systems as part of the linux platform rather than having a separate cray platform.

This PR:
- [x] Changes cray platform detection to ignore EX systems with Craype version 21.10 or later
- [x] Changes the cce compiler to be detectable via paths
- [x] Changes the spack compiler wrapper to understand the executable names for the standalone cce compiler (`craycc`, `crayCC`, `crayftn`).
@quellyn quellyn mentioned this pull request Jan 6, 2023
4 tasks
amd-toolchain-support pushed a commit to amd-toolchain-support/spack that referenced this pull request Feb 16, 2023
Newer versions of the CrayPE for EX systems have standalone compiler executables for CCE and compiler wrappers for Cray MPICH. With those, we can treat the cray systems as part of the linux platform rather than having a separate cray platform.

This PR:
- [x] Changes cray platform detection to ignore EX systems with Craype version 21.10 or later
- [x] Changes the cce compiler to be detectable via paths
- [x] Changes the spack compiler wrapper to understand the executable names for the standalone cce compiler (`craycc`, `crayCC`, `crayftn`).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

architecture compilers core PR affects Spack core functionality dependencies patch tests General test capability(ies) update-package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants