Skip to content

Pin git sha to git ref based versions#44319

Closed
psakievich wants to merge 43 commits intospack:developfrom
psakievich:psakiev/f/pin
Closed

Pin git sha to git ref based versions#44319
psakievich wants to merge 43 commits intospack:developfrom
psakievich:psakiev/f/pin

Conversation

@psakievich
Copy link
Copy Markdown
Contributor

@psakievich psakievich commented May 22, 2024

This PR will make it so the concretizer assigns the latest commit sha for branch based versions of packages. Right now this is done after the solve. If we try to move it inside the solve I believe we'd have to run git ls-remote on every possible branch-version that could appear in the graph. Seems wasteful makes the solve dependent on the speed of that operation.

- [x] Add a config option, and CLI option to the common solve arguments

  • Add tests for re-use of an exact hash match
  • Add functionality for air-gapped systems
  • Extend branch implementation to all git ref checkouts
  • Add docs

Depends on #43859
Fixes #4674

This PR will make it so the concretizer assigns the latest commit
sha for branch based versions of packages.
@spackbot-app spackbot-app bot added core PR affects Spack core functionality new-version tests General test capability(ies) labels May 22, 2024
@psakievich
Copy link
Copy Markdown
Contributor Author

Effect of this PR:

$ spack solve exawind@master ^nalu-wind@master+tioga ^amr-wind@main
==> Best of 13 considered solutions.
==> Optimization Criteria:
  Priority  Criterion                                            Installed  ToBuild
  1         number of packages to build (vs. reuse)                      -       21
  2         number of nodes from the same package                        -        0
  3         requirement weight                                           0        0
  4         deprecated versions used                                     0        0
  5         version badness (roots)                                      0        0
  6         number of non-default variants (roots)                       0        0
  7         preferred providers for roots                                0        0
  8         default values of variants not being used (roots)            0        0
  9         number of non-default variants (non-roots)                  13        0
  10        preferred providers (non-roots)                              0        0
  11        compiler mismatches that are not from CLI                    0        0
  12        compiler mismatches that are not from CLI                    0        0
  13        non-preferred OS's                                          36        0
  14        version badness (non roots)                                500       32
  15        default values of variants not being used (non-roots)        2       19
  16        non-preferred compilers                                    108        0
  17        target mismatches                                           27        0
  18        non-preferred targets                                      180        0
  19        compiler mismatches (runtimes)                               0        0
  20        version badness (runtimes)                                   0        0
  21        non-preferred targets (runtimes)                             0        0
  22        edge wiring                                               2699        0

 -   exawind@git.0069668964b844420182084ad024f78f85b0a1f7=master%apple-clang@15.0.0~amr_wind_gpu~cuda~gpu-aware-mpi~ipo~nalu_wind_gpu~rocm~sycl build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -       ^amr-wind@git.6ba000b628aa3178545cdbbea508cc2cb2e5c76c=main%apple-clang@15.0.0~ascent~cuda~gpu-aware-mpi~hdf5~helics~hypre~ipo~masa+mpi+netcdf~openfast~openmp~rocm+shared~sycl+tests+tiny_profile~umpire~waves2amr build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -           ^netcdf-c@4.9.2%apple-clang@15.0.0+blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize+parallel-netcdf+pic+shared+szip+zstd build_system=autotools patches=0161eb8,4e10547 arch=darwin-sonoma-m1
 -               ^autoconf@2.72%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -               ^automake@1.16.5%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -               ^bzip2@1.0.8%apple-clang@15.0.0~debug~pic+shared build_system=generic arch=darwin-ventura-aarch64
 -                   ^diffutils@3.10%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -               ^c-blosc@1.21.5%apple-clang@15.0.0+avx2~ipo build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -                   ^lz4@1.9.4%apple-clang@15.0.0+pic build_system=makefile libs=shared,static arch=darwin-sonoma-m1
 -                   ^snappy@1.1.10%apple-clang@15.0.0~ipo+pic+shared build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -               ^libaec@1.0.6%apple-clang@15.0.0~ipo+shared build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -               ^m4@1.4.19%apple-clang@15.0.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=darwin-ventura-aarch64
 -                   ^libsigsegv@2.14%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -               ^zstd@1.5.6%apple-clang@15.0.0+programs build_system=makefile compression=none libs=shared,static arch=darwin-ventura-aarch64
 -       ^cmake@3.27.9%apple-clang@15.0.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=darwin-ventura-aarch64
 -           ^curl@8.7.1%apple-clang@15.0.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=secure_transport arch=darwin-ventura-aarch64
 -               ^nghttp2@1.61.0%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -               ^perl@5.38.2%apple-clang@15.0.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-ventura-aarch64
 -           ^ncurses@6.5%apple-clang@15.0.0~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=darwin-ventura-aarch64
 -           ^zlib-ng@2.1.6%apple-clang@15.0.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=darwin-ventura-aarch64
 -       ^gmake@4.4.1%apple-clang@15.0.0~guile build_system=generic arch=darwin-ventura-aarch64
 -       ^nalu-wind@git.83c6597c0b7afd042cb510615af286e9fae30b29=master%apple-clang@15.0.0~boost~catalyst~cuda~fftw+fsi~gpu-aware-mpi+hypre~ipo+openfast+pic~rocm+shared+tioga+trilinos-solvers~umpire~wind-utils abs_tol=1e-15 build_system=cmake build_type=Release generator=make rel_tol=1e-12 arch=darwin-sonoma-m1
 -           ^hypre@2.31.0%apple-clang@15.0.0~caliper~complex~cublas~cuda~debug+fortran~gptune~gpu-aware-mpi~int64~internal-superlu~magma~mixedint+mpi~openmp~rocblas~rocm~shared~superlu-dist~sycl~umpire~unified-memory build_system=autotools arch=darwin-sonoma-m1
 -           ^nccmp@1.9.1.0%apple-clang@15.0.0~ipo build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -       ^openfast@git.5468fbf641853ef17c8598f594be5f39bf58f4ed=develop%apple-clang@15.0.0+cxx+dll-interface+double-precision~ipo+netcdf~openmp+pic~rosco+shared build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -           ^hdf5@1.14.3%apple-clang@15.0.0+cxx~fortran+hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make patches=82088c8 arch=darwin-sonoma-m1
 -           ^libxml2@2.10.3%apple-clang@15.0.0+pic~python+shared build_system=autotools arch=darwin-ventura-aarch64
 -               ^libiconv@1.17%apple-clang@15.0.0 build_system=autotools libs=shared,static arch=darwin-ventura-aarch64
 -               ^xz@5.4.6%apple-clang@15.0.0~pic build_system=autotools libs=shared,static arch=darwin-ventura-aarch64
 -           ^openblas@0.3.26%apple-clang@15.0.0~bignuma~consistent_fpcsr+dynamic_dispatch~fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=darwin-ventura-aarch64
 -       ^openmpi@5.0.3%apple-clang@15.0.0~atomics~cuda~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~static+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=darwin-ventura-aarch64
 -           ^autoconf@2.72%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -           ^automake@1.16.5%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -           ^gnuconfig@2022-09-17%apple-clang@15.0.0 build_system=generic arch=darwin-ventura-aarch64
 -           ^hwloc@2.9.1%apple-clang@15.0.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl~pci~rocm build_system=autotools libs=shared,static arch=darwin-ventura-aarch64
 -           ^libevent@2.1.12%apple-clang@15.0.0+openssl build_system=autotools arch=darwin-ventura-aarch64
 -               ^openssl@3.3.0%apple-clang@15.0.0~docs+shared build_system=generic certs=mozilla arch=darwin-ventura-aarch64
 -                   ^ca-certificates-mozilla@2023-05-30%apple-clang@15.0.0 build_system=generic arch=darwin-ventura-aarch64
 -           ^libtool@2.4.7%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -               ^findutils@4.9.0%apple-clang@15.0.0 build_system=autotools patches=440b954 arch=darwin-ventura-aarch64
 -           ^openssh@9.7p1%apple-clang@15.0.0+gssapi build_system=autotools patches=3505c58,d886b98 arch=darwin-ventura-aarch64
 -               ^krb5@1.21.2%apple-clang@15.0.0+shared build_system=autotools arch=darwin-ventura-aarch64
 -                   ^bison@3.8.2%apple-clang@15.0.0~color build_system=autotools arch=darwin-ventura-aarch64
 -                   ^gettext@0.22.5%apple-clang@15.0.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=darwin-ventura-aarch64
 -                       ^tar@1.34%apple-clang@15.0.0 build_system=autotools zip=pigz arch=darwin-ventura-aarch64
 -                           ^pigz@2.8%apple-clang@15.0.0 build_system=makefile arch=darwin-ventura-aarch64
 -               ^libedit@3.1-20230828%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -               ^libxcrypt@4.4.35%apple-clang@15.0.0~obsolete_api build_system=autotools patches=4885da3 arch=darwin-ventura-aarch64
 -           ^perl@5.38.0%apple-clang@15.0.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=darwin-ventura-aarch64
 -               ^berkeley-db@18.1.40%apple-clang@15.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-ventura-aarch64
 -               ^gdbm@1.23%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -                   ^readline@8.2%apple-clang@15.0.0 build_system=autotools patches=bbf97f1 arch=darwin-ventura-aarch64
 -           ^pkgconf@2.2.0%apple-clang@15.0.0 build_system=autotools arch=darwin-ventura-aarch64
 -           ^pmix@5.0.1%apple-clang@15.0.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=darwin-ventura-aarch64
 -       ^tioga@git.73a690616d4ca6478a3968fb6b6de3acd92bc71b=develop%apple-clang@15.0.0~ipo~nodegid+pic~shared~stats~timers build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -       ^trilinos@15.1.1%apple-clang@15.0.0~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings+exodus+explicit_template_instantiation~float+fortran+gtest+hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~pamgen~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec+shards~shared~shylu+stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~test~thyra+tpetra~trilinoscouplings~wrapper~x11+zoltan~zoltan2 build_system=cmake build_type=Release cxxstd=17 generator=make gotype=long arch=darwin-sonoma-m1
 -           ^cgns@4.4.0%apple-clang@15.0.0~base_scope~fortran+hdf5~int64~ipo~legacy~mem_debug+mpi~pic+scoping+shared~static~testing~tools build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1
 -           ^kokkos@4.2.01%apple-clang@15.0.0~aggressive_vectorization~compiler_warnings~cuda~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~openmptarget~pic~rocm+serial+shared~sycl~tests~threads~tuning~wrapper build_system=cmake build_type=Release cxxstd=17 generator=make intel_gpu_arch=none arch=darwin-sonoma-m1
 -           ^matio@1.5.26%apple-clang@15.0.0+hdf5+shared+zlib build_system=autotools arch=darwin-sonoma-m1
 -           ^metis@5.1.0%apple-clang@15.0.0~gdb~int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903 arch=darwin-sonoma-m1
 -           ^parallel-netcdf@1.12.3%apple-clang@15.0.0~burstbuffer+cxx+fortran+pic+shared build_system=autotools arch=darwin-sonoma-m1
 -           ^parmetis@4.0.3%apple-clang@15.0.0~gdb~int64~ipo+shared build_system=cmake build_type=Release generator=make patches=4f89253,50ed208,704b84f arch=darwin-sonoma-m1
 -       ^yaml-cpp@0.6.3%apple-clang@15.0.0~ipo+pic+shared~tests build_system=cmake build_type=Release generator=make arch=darwin-sonoma-m1

@psakievich psakievich changed the title Pin git sha to branch based versions Pin git sha to git ref based versions Jul 3, 2024
@psakievich psakievich added the snl-core-team Issue for SNL Spack developers label Dec 3, 2024
@becker33
Copy link
Copy Markdown
Member

becker33 commented Dec 5, 2024

I think this fixes #4674, @psakievich can you edit the description to say this closes/fixes that issue?

@psakievich psakievich closed this by deleting the head repository May 14, 2025
@tgamblin tgamblin removed this from the v1.0.0 stretch goals milestone Jun 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core PR affects Spack core functionality defaults fetching new-version shell-support snl-core-team Issue for SNL Spack developers tests General test capability(ies) update-package versions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store commit hash along with tag or branch

3 participants