Replies: 5 comments 8 replies
-
|
Would appreciate some feedback from @psakievich, @alecbcs, @becker33, @haampie, @alalazo, @adamjstewart, @michaelkuhn, @wdconinc, and anyone else who'd like to weigh in here. |
Beta Was this translation helpful? Give feedback.
-
|
This is great. I like all of the functionality outlined in this proposal and I think it'll go a long way to improving Spack's understand-ability for newer and less experienced users. |
Beta Was this translation helpful? Give feedback.
-
|
When above we say:
do we mean that it is already in the The other point is that, if we consider splitting the current
The rationale is that, if a command is meant to update specs, I would find it weird if by default it does a very partial update, and needs further instructions to do a full update. Not sure this is a mainstream opinion though. |
Beta Was this translation helpful? Give feedback.
-
+1. I'd prefer upgrade to be this hybrid |
Beta Was this translation helpful? Give feedback.
-
Not exactly. I think the semantic of Instead, I'd prefer if
I don't agree 100% with this. Consider these 3 variations: spack:
specs:
- hdf5 ^openmpi
###
spack:
specs:
- hdf5
packages:
mpi:
require: openmpi
###
spack:
specs:
- hdf5
- openmpiAccording to what you say the latter cares about
I don't get this intuition... I think it depends on how old the environment is, and the deps. Probably combining fairly old deps with new roots causes failures too. |
Beta Was this translation helpful? Give feedback.
-
We should be catching this in the |
Beta Was this translation helpful? Give feedback.
-
This makes sense to me. As an application developer if I specify an environment as, spack:
specs:
- hdf5 ^openmpiWhat I'm saying here is that I want |
Beta Was this translation helpful? Give feedback.
-
Sure, but all these are double way arguments. By the same argument, we should catch and fix build failures in To be clear, I'm not strongly opposed to any of the above, just gave my personal opinion on the default of The only note I have is that a few claims above seems more like other personal preferences. They are fine, but as I said I don't immediately see why having Or why this: spack:
specs:
- hdf5 ^openmpimeans I don't care about spack:
specs:
- hdf5
- openmpimeans I do (to me all the environments above are kind of equivalent). |
Beta Was this translation helpful? Give feedback.
-
|
@tgamblin how do you imagine |
Beta Was this translation helpful? Give feedback.
-
|
So if I'm understanding correctly, you're proposing the following two changes:
If so, I'm fine with this behavior. It makes more sense linguistically. Some caveats to think about:
|
Beta Was this translation helpful? Give feedback.
-
|
Both
Given both of those projects are extremely popular and often people's first introduction to a package manager, I'd be a big proponent of us adopting/reusing a similar syntax in Spack. Also linking to #37549 which I think could be a nice complement to this |
Beta Was this translation helpful? Give feedback.
-
|
I'm supportive of the concept. I think I like the defaults @alalazo proposed of I like the idea of basically one syntax being conservative (default of current There is definitely a large union between the two based on the proposed flags/options which I don't think is terrible. However, spack's historical adoption of many syntaxes to accomplish the same thing means there is some analysis paralysis each time I spawn a new automated process rooted in spack. Polar opposite defaults make the intentions more clear, easier to communicate to users, and reduces the number of flags required to get what you want. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
TL;DR: We're reaching the limits of how many jobs we entrust to a single
spack installcommand and have it still make sense, so I'm proposing to simplify things by breaking it up into two commands (like many other systems do):spack install: your favorite spack command: tried and mostly true.spack installshould--reuseexisting build caches and installations by defaultspack upgrade: a new command to bring existing installations up to dateGoal
We want to make consumer and builder workflows easy.
Motivation
I'll try to give a little history here to describe how we ended up where we are.
1. The beginning: fresh installs
Originally, every installation in Spack was an update, because we would do a fresh concretization for everything, without considering installed packages. If there was any reuse, it was because something concretized to exactly the same hash as a previous installation.
spack install --fresh.2. Adding
--reusePossibly everyone's biggest complaint about Spack for many years was the number of rebuilds needed to do any sort of installation.
--freshgets you the latest, and it's deterministic (for the same Spack commit) but it's slow and often not what users want. So:v0.17, we added--reusein concretizer: reuse existing packages #25310.v0.18, we made--freshan option in#28468and reuse the default in Reuse by default in Spack #303963. The backlash:
reuse: dependenciesFolks who were used to the olden days were not entirely pleased with the new semantics, as now by default if you said
spack install foo, andfoowas already installed, Spack would simply reuse the old version offooand say it was installed. This is probably what most people wantinstallto do, but for development workflows and for upgrades, it's not ideal.--freshis a blunt instrument: when you specify it, nothing is reused, and devs likely want their dependencies to stay put while iterating on some root package.spack install --freshupgrades everything, including dependencies.In response to this, we (probably too hastily):
reuse:dependenciesthe default in concretizer: add mode to reuse dependencies only #30990.This makes Spack reuse only dependencies but roots (here,
foo) are fresh by default andspack install foowill reinstall whatever the latestfoois. in an environment,spack installwill always upgrade all the roots.4. More issues: build caches and upstreams
Per the description of #41302, we've seen a lot of very confusing use cases where people have tried to push binaries to a build cache and have a friend install them directly from there. Without
--reuse, this is a major pain, because you have to get your local configuration just right to ensure that specs from he cache are actually used. This also makes it hard to use Spack in a pure binary workflow - i.e., where you have one dev producing binaries for a team, and most users are installing the produced binaries.So now, we've:
reuse:default to True #41302Splitting
installandupgradeI think the solution here is to split what is now
spack installand a bunch of concretization options intospack installandspack upgrade. See the first section above for why.Specifics
There are a lot of cases to hear so I'm enumerating them for others' consideration. I'm hoping we can settle on something that allows both consumer workflows and developer/upgrade workflows to be intuitive.
installspack installspack install --freshspack install SPECspack install --fresh SPECupgradespack upgradedeconcretizeroots* reconcretize all, reusing same deps
* install result
spack upgrade --freshdeconcretizeroots* reconcretize all, no reuse
* install result
spack upgrade SPEC* also reuse deps
* install result
* reconcretize SPEC reusing deps
* install resulting SPEC
spack upgrade --fresh SPEC* no reuse
* install result
* reconcretize SPEC, no reuse
* install resulting SPEC
This is as far as I've gotten so would appreciate some feedback on the semantics. Basically you can think of
spack upgradeasspack install, but:spack upgradeis:deconcretizeupgraded specs so that they're removed from an env's lockfile. This forces the thing you ask for to be upgraded in an env.Beta Was this translation helpful? Give feedback.
All reactions