concretizer: don't change concrete environments without --force#37438
Merged
concretizer: don't change concrete environments without --force#37438
--force#37438Conversation
Member
|
I haven't looked at the code in detail, but it feels like a solid improvement to me, especially when you have a long list of specs already and just need to add a tiny package on top of it 👍 |
Member
Author
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: isort, black, flake8, mypy
==> Modified files
lib/spack/spack/environment/environment.py
lib/spack/spack/test/cmd/env.py
==> Running isort checks
isort checks were clean
==> Running black checks
reformatted lib/spack/spack/environment/environment.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
black checks were clean
==> Running flake8 checks
flake8 checks were clean
==> Running mypy checks
Success: no issues found in 577 source files
mypy checks were clean
==> spack style checks were clean
I've updated the branch with style fixes. |
63695b2 to
db9b860
Compare
Member
|
@alalazo this is a really big improvement, IMO. Can you take a look at this for 0.20? @becker33: I consolidated your original (duplicated) logic into a method and tweaked the docs and error messages a bit, but this looks great. See what you think of the changes. I think this will make people way happier with Spack environments. IMO it's ready to go pending a confirming review. |
tgamblin
previously approved these changes
May 13, 2023
--force
Member
|
One thought on this (as a follow-on): I think it's actually good for Spack to fail fast if it would have to change any existing concrete specs in an env, but I also think that when using |
alalazo
approved these changes
May 13, 2023
Member
alalazo
left a comment
There was a problem hiding this comment.
Code LGTM. Test driven with the following example:
tgamblin
pushed a commit
that referenced
this pull request
May 16, 2023
…7681) Two bugs came in from #37438 1. `unify: when_possible` was broken, because of an incorrect assertion. abstract/concrete spec pairs were compared against the results that were in the process of being computed, rather than against the previous results. 2. `unify: true` had an ordering bug that could mix the association between abstract and concrete specs - [x] 1 is resolved by creating a lookup from old concrete specs to old abstract specs, and we use that to associate the "new" concrete specs that happen to be the old ones with their abstract specs (since those are stripped out for concretization - [x] 2 is resolved by combining the new and old abstract as lists instead of combining them as sets. This is important because `set() | set()` does not make any ordering promises, even though set ordering is otherwise guaranteed in `python@3.7:`
alalazo
pushed a commit
that referenced
this pull request
May 16, 2023
…7681) Two bugs came in from #37438 1. `unify: when_possible` was broken, because of an incorrect assertion. abstract/concrete spec pairs were compared against the results that were in the process of being computed, rather than against the previous results. 2. `unify: true` had an ordering bug that could mix the association between abstract and concrete specs - [x] 1 is resolved by creating a lookup from old concrete specs to old abstract specs, and we use that to associate the "new" concrete specs that happen to be the old ones with their abstract specs (since those are stripped out for concretization - [x] 2 is resolved by combining the new and old abstract as lists instead of combining them as sets. This is important because `set() | set()` does not make any ordering promises, even though set ordering is otherwise guaranteed in `python@3.7:`
alalazo
pushed a commit
that referenced
this pull request
May 16, 2023
…7681) Two bugs came in from #37438 1. `unify: when_possible` was broken, because of an incorrect assertion. abstract/concrete spec pairs were compared against the results that were in the process of being computed, rather than against the previous results. 2. `unify: true` had an ordering bug that could mix the association between abstract and concrete specs - [x] 1 is resolved by creating a lookup from old concrete specs to old abstract specs, and we use that to associate the "new" concrete specs that happen to be the old ones with their abstract specs (since those are stripped out for concretization - [x] 2 is resolved by combining the new and old abstract as lists instead of combining them as sets. This is important because `set() | set()` does not make any ordering promises, even though set ordering is otherwise guaranteed in `python@3.7:`
haampie
pushed a commit
that referenced
this pull request
May 16, 2023
…7681) Two bugs came in from #37438 1. `unify: when_possible` was broken, because of an incorrect assertion. abstract/concrete spec pairs were compared against the results that were in the process of being computed, rather than against the previous results. 2. `unify: true` had an ordering bug that could mix the association between abstract and concrete specs - [x] 1 is resolved by creating a lookup from old concrete specs to old abstract specs, and we use that to associate the "new" concrete specs that happen to be the old ones with their abstract specs (since those are stripped out for concretization - [x] 2 is resolved by combining the new and old abstract as lists instead of combining them as sets. This is important because `set() | set()` does not make any ordering promises, even though set ordering is otherwise guaranteed in `python@3.7:`
alalazo
pushed a commit
that referenced
this pull request
May 18, 2023
…7681) Two bugs came in from #37438 1. `unify: when_possible` was broken, because of an incorrect assertion. abstract/concrete spec pairs were compared against the results that were in the process of being computed, rather than against the previous results. 2. `unify: true` had an ordering bug that could mix the association between abstract and concrete specs - [x] 1 is resolved by creating a lookup from old concrete specs to old abstract specs, and we use that to associate the "new" concrete specs that happen to be the old ones with their abstract specs (since those are stripped out for concretization - [x] 2 is resolved by combining the new and old abstract as lists instead of combining them as sets. This is important because `set() | set()` does not make any ordering promises, even though set ordering is otherwise guaranteed in `python@3.7:`
Contributor
|
@becker33 @alalazo I found an issue with this PR and I am not sure it is expected or not. It seems that it prevents me to install beta snapshots of the package (the tags that are not included into package.py). For example, following command but if use the code without these changes, it installs the package without any issue. Anyway, I need to have this capability since we are testing different versions of the ESMF package and it would be hard to add all the beta snapshots to the package.py. So, is there any other way to workaround the issue. Is this a bug? Any idea? |
Contributor
3 tasks
vjranagit
pushed a commit
to vjranagit/spack
that referenced
this pull request
Jan 18, 2026
…ack#37681) Two bugs came in from spack#37438 1. `unify: when_possible` was broken, because of an incorrect assertion. abstract/concrete spec pairs were compared against the results that were in the process of being computed, rather than against the previous results. 2. `unify: true` had an ordering bug that could mix the association between abstract and concrete specs - [x] 1 is resolved by creating a lookup from old concrete specs to old abstract specs, and we use that to associate the "new" concrete specs that happen to be the old ones with their abstract specs (since those are stripped out for concretization - [x] 2 is resolved by combining the new and old abstract as lists instead of combining them as sets. This is important because `set() | set()` does not make any ordering promises, even though set ordering is otherwise guaranteed in `python@3.7:`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Users generally don't want to change large chunks of their existing concrete environment, but Spack currently does that when using
concretizer:unify:true(the default) and ``concretizer:unify:when_possible`. Reconcretizing large environments can also be slow, as Spack has to start from scratch and solve everything again.We can do for
unify:trueandunify:when_possiblewhat we already do forunify:false: preserve the existing concrete environment state. This PR adds that functionality, and it lets you invalidate the existing concrete state withspack concretize -f. Now,spack concretizewill only concretize the new portions of the environment andspack concretize -freconcretizes the entire environment.