mpi4py: do not re-generate cython sources that already exist in mpi4py release tarball#39661
Closed
mpi4py: do not re-generate cython sources that already exist in mpi4py release tarball#39661
Conversation
Member
|
I thought we moved towards cythonizing everything nonetheless, cause old cython may generate code that is not forward compatible with python (it relies on python internals that change over time). Should this change be accompanied with upperbounds on python for older releases? |
Contributor
Author
|
The cython@3.0 dependency issues appear to be a bit wider than just mpi4py. I noticed numpy also has a similar issue [no version that supprots cython-3.0] - so the concretizer resolves to using balay@xsdk:/data/balay/spack>./bin/spack spec py-numpy |egrep '(python|cython|numpy)@'
- py-numpy@1.25.2%gcc@11.3.1+blas+lapack build_system=python_pip patches=873745d arch=linux-rocky9-cascadelake
- ^py-cython@0.29.36%gcc@11.3.1 build_system=python_pip arch=linux-rocky9-cascadelake
- ^python@3.10.12%gcc@11.3.1+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-rocky9-cascadelake
balay@xsdk:/data/balay/spack>./bin/spack spec py-numpy ^py-cython@3.0.0 |egrep '(python|cython|numpy)@'
- ^py-cython@3.0.0
- py-numpy@1.17.5%gcc@11.3.1+blas+lapack build_system=python_pip patches=cf407c1 arch=linux-rocky9-cascadelake
[+] ^py-cython@3.0.0%gcc@11.3.1 build_system=python_pip arch=linux-rocky9-cascadelake
[+] ^python@3.8.17%gcc@11.3.1+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,f2fd060 arch=linux-rocky9-cascadelakeSo I'm not sure what the best way to proceed here is.. |
Member
|
@alalazo has a pull request to support multiple versions of the same build dep per dag |
Contributor
Author
what is the max version of python that cython@0.29.36 is compatible with? |
Python 3.12, which is in release candidate stage (API frozen). |
3 tasks
Contributor
Author
To clarify - the incompatibly will start with python-3.13 (and not 3.12)? |
Yes, exactly. |
Member
Yes, probably
#38447 has been merged, but it's opt-in at the moment, so the spec would still fail to resolve unless the user knows what they're doing. #38671 is also waiting on this to become the default behavior. |
alalazo
added a commit
to alalazo/spack
that referenced
this pull request
Sep 20, 2023
For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set.
alalazo
added a commit
to alalazo/spack
that referenced
this pull request
Sep 26, 2023
For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set.
alalazo
added a commit
to alalazo/spack
that referenced
this pull request
Sep 27, 2023
For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set.
alalazo
added a commit
to alalazo/spack
that referenced
this pull request
Oct 5, 2023
For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set.
alalazo
added a commit
to alalazo/spack
that referenced
this pull request
Oct 5, 2023
* Allow branching out of the "generic build" unification set For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems.
alalazo
added a commit
that referenced
this pull request
Oct 6, 2023
* Allow branching out of the "generic build" unification set For cases like the one in #39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems. * Fixed a few issues in packages py-gevent: restore dependency on py-cython@3 jsoncpp: fix typo in build dependency ecp-data-vis-sdk: update spack.yaml and cmake recipe py-statsmodels: add v0.13.5 * Make dependency on "blt" of type "build"
RikkiButler20
pushed a commit
to RikkiButler20/spack
that referenced
this pull request
Nov 2, 2023
* Allow branching out of the "generic build" unification set For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems. * Fixed a few issues in packages py-gevent: restore dependency on py-cython@3 jsoncpp: fix typo in build dependency ecp-data-vis-sdk: update spack.yaml and cmake recipe py-statsmodels: add v0.13.5 * Make dependency on "blt" of type "build"
RikkiButler20
pushed a commit
to RikkiButler20/spack
that referenced
this pull request
Jan 11, 2024
* Allow branching out of the "generic build" unification set For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems. * Fixed a few issues in packages py-gevent: restore dependency on py-cython@3 jsoncpp: fix typo in build dependency ecp-data-vis-sdk: update spack.yaml and cmake recipe py-statsmodels: add v0.13.5 * Make dependency on "blt" of type "build"
RikkiButler20
pushed a commit
to RikkiButler20/spack
that referenced
this pull request
Jan 31, 2024
* Allow branching out of the "generic build" unification set For cases like the one in spack#39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems. * Fixed a few issues in packages py-gevent: restore dependency on py-cython@3 jsoncpp: fix typo in build dependency ecp-data-vis-sdk: update spack.yaml and cmake recipe py-statsmodels: add v0.13.5 * Make dependency on "blt" of type "build"
Member
|
I believe this is not needed anymore (we default to multiple versions of build deps), but feel free to reopen if I'm wrong. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(that depend on cython < 0.29.36)
This avoids conflict with petsc4py versions that have a dependency on cython@3.0.0
ref: mpi4py/mpi4py#386 (comment)