Autogenerate environment.yml file from pyproject.toml by drammock · Pull Request #12914 · mne-tools/mne-python

drammock · 2024-10-22T19:49:19Z

Easiest to review a few specific commits:

a3f2a0e adds the hook
1a0d7b9 does nothing except sort environment.yml entries so that subsequent hook-induced changes to that file are easier to see
ef61f28 shows the actual changes caused by the hook on first run (plus a fake "aaaaaaa" dep added to trigger the hook, which is removed in subsequent commit 4e99502)

Note that this does not address #12903 (comment) --- I figured easier to get this one in / tested and then deal later with cross-repo syncing.

…st-extra)

.conda-extra-dependencies.yml

pyproject.toml

tools/hooks/update_environment_file.py

This reverts commit 44a08e4.

drammock · 2024-10-24T16:58:36Z

windows-latest mamba job will continue to fail until the fix for mamba-org/mamba#3467 has been released.

environment.yml

larsoner

It occurred to me that any autogenerated file should have a comment so I just pushed a tiny commit to add this to the top of the env file:

# THIS FILE IS AUTO-GENERATED BY tools/hooks/update_environment_file.py AND WILL BE OVERWRITTEN

similar to what we have in doc/sphinxext/credit_tools.py

larsoner · 2024-10-24T18:06:47Z

Ughhhh looks like I was wrong about openblas, will push a commit

for more information, see https://pre-commit.ci

larsoner · 2024-10-24T19:31:17Z

I am a bit mystified by both the pip-pre failure and the conda one. Looking at a diff of the sorted main YML and the one in this PR I don't see anything that should break linalg results as seems to be happening in the conda install now.

drammock · 2024-10-24T20:30:05Z

I am a bit mystified by both the pip-pre failure and the conda one. Looking at a diff of the sorted main YML and the one in this PR I don't see anything that should break linalg results as seems to be happening in the conda install now.

I'm scratching my head too 😕

drammock · 2024-10-24T20:44:43Z

the numpy config on windows-latest mamba is:

 Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/runneradmin/micromamba/envs/mne/Library/include
    lib directory: C:/Users/runneradmin/micromamba/envs/mne/Library/lib
    name: blas
    openblas configuration: unknown
    pc file directory: D:\bld\numpy_1707225570061\_h_env\Library\lib\pkgconfig
    version: 3.9.0
  lapack:
    detection method: internal
    found: true
    include directory: unknown
    lib directory: unknown
    name: dep2598594164480
    openblas configuration: unknown
    pc file directory: unknown
    version: 1.26.4

The line blas: openblas configuration: unknown is different on my local machine, which has USE_64BITINT=1 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= HASWELL MAX_THREADS=2. The version string also looks fishy (I see 0.3.23.dev locally, CI has 3.9.0?? AFAIK openblas versions are all still 0.x)

drammock · 2024-10-24T20:49:41Z

comparing that to the windows pip-pre job:

Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-rk8_4lij/cp312-win_amd64/build/venv/Lib/site-packages/scipy_openblas64/include
    lib directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-rk8_4lij/cp312-win_amd64/build/venv/Lib/site-packages/scipy_openblas64/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.28  USE64BITINT DYNAMIC_ARCH NO_AFFINITY
      Haswell MAX_THREADS=24
    pc file directory: D:/a/numpy/numpy/.openblas
    version: 0.3.28
  lapack:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-rk8_4lij/cp312-win_amd64/build/venv/Lib/site-packages/scipy_openblas64/include
    lib directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-rk8_4lij/cp312-win_amd64/build/venv/Lib/site-packages/scipy_openblas64/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.28  USE64BITINT DYNAMIC_ARCH NO_AFFINITY
      Haswell MAX_THREADS=24
    pc file directory: D:/a/numpy/numpy/.openblas
    version: 0.3.28

here we see a reasonable version number, known configuration strings, and a different name ("scipy-openblas" instead of "blas")

drammock · 2024-10-24T21:15:45Z

ok, yep: windows-latest mamba is getting an MKL version of numpy for some reason:

 + numpy                1.26.4 (MKL 2024.2.2-Product with 1 thread)

drammock · 2024-10-24T22:33:39Z

disabling auto-merge because I'm not certain if 69a790d will actually be necessary anymore.

This reverts commit 69a790d.

drammock · 2024-10-25T01:25:13Z

tools/azure_dependencies.sh

 elif [ "${TEST_MODE}" == "pip-pre" ]; then
 	${SCRIPT_DIR}/install_pre_requirements.sh
-	python -m pip install $STD_ARGS --pre -e .[test]
+	python -m pip install $STD_ARGS --pre -e .[test_extra]


the corresponding branch in github_actions_dependencies.sh uses test_extra too, and it's needed for utils/tests/test_config.py (I think previously it was passing on windows by chance)

larsoner · 2024-10-25T14:28:53Z

Thanks @drammock !

drammock · 2024-10-25T14:30:46Z

@larsoner do you have any idea why we were suddenly getting MKL? I never did track down a cause.

larsoner · 2024-10-25T14:51:23Z

No idea. Could be unfortunate timing with some conda change. Or something in the solver changing some logic. Or a new package. But I like the nomkl solution!

* upstream/main: eegbci api: allow downloading multiple subjects (mne-tools#12918) DOC: Document Linux desktop workaround (mne-tools#12900) Allow exporting edf where a channel contains only constant values (mne-tools#12911) Autogenerate environment.yml file from pyproject.toml (mne-tools#12914)

drammock added 9 commits October 18, 2024 16:42

additions to pyproject.toml

8b14553

add VTK pin from YAML

ee45879

spaces around version spec operators

8a3e78a

TOML-sort changes induced by spaces around operators

4cf4f3c

remove unhelpful comment (lots of deps duplicated between full and te…

a5da48d

…st-extra)

sorting and spacing of environment.yml

1a0d7b9

add env auto-gen hook and associated files

a3f2a0e

proof of concept: add fake dependency to trigger hook

ef61f28

remove fake dep used for proof-of-concept

4e99502

drammock added MAINT no-changelog-entry-needed labels Oct 22, 2024

drammock requested a review from larsoner as a code owner October 22, 2024 19:49

Merge branch 'main' into autogen-env-yaml

cdd25e8

larsoner reviewed Oct 22, 2024

View reviewed changes

.conda-extra-dependencies.yml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

tools/hooks/update_environment_file.py Outdated Show resolved Hide resolved

drammock added 5 commits October 22, 2024 16:12

eric's review; don't assume space-separator in source file

5826fa0

pytest pin

0bff690

workaround micromamba issue

44a08e4

Revert "workaround micromamba issue"

536558b

This reverts commit 44a08e4.

avoid excluding pyside6==6.7.0 (not found on conda-forge)

1fb1e27

drammock force-pushed the autogen-env-yaml branch from ebc9fc2 to 1fb1e27 Compare October 24, 2024 15:54

avoid backslash in f-string

0fc4250

larsoner reviewed Oct 24, 2024

View reviewed changes

environment.yml Outdated Show resolved Hide resolved

drammock and others added 2 commits October 24, 2024 12:11

workaround micromamba ci bug

32d5768

FIX: Comment

e499ed0

larsoner approved these changes Oct 24, 2024

View reviewed changes

larsoner enabled auto-merge (squash) October 24, 2024 17:19

larsoner and others added 2 commits October 24, 2024 14:07

FIX: Openblas

fa07a66

[pre-commit.ci] auto fixes from pre-commit.com hooks

ab1b403

for more information, see https://pre-commit.ci

drammock added 2 commits October 24, 2024 15:07

missed whitespace change

f255010

make sure imageio installed in pip-pre job

69a790d

add nomkl to micromamba init

0e3b018

drammock disabled auto-merge October 24, 2024 22:32

drammock added 2 commits October 24, 2024 20:22

make azure_deps match actions_deps for pip-pre (use test_extra)

1ed4490

Revert "make sure imageio installed in pip-pre job"

9385229

This reverts commit 69a790d.

drammock commented Oct 25, 2024

View reviewed changes

FIX: More

2f21782

larsoner merged commit 0755ef0 into mne-tools:main Oct 25, 2024

drammock deleted the autogen-env-yaml branch October 25, 2024 14:29

Uh oh!

Conversation

drammock commented Oct 22, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drammock commented Oct 24, 2024

Uh oh!

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

larsoner commented Oct 24, 2024

Uh oh!

larsoner commented Oct 24, 2024

Uh oh!

drammock commented Oct 24, 2024

Uh oh!

drammock commented Oct 24, 2024

Uh oh!

drammock commented Oct 24, 2024

Uh oh!

drammock commented Oct 24, 2024

Uh oh!

drammock commented Oct 24, 2024

Uh oh!

drammock Oct 25, 2024

Choose a reason for hiding this comment

Uh oh!

larsoner commented Oct 25, 2024

Uh oh!

drammock commented Oct 25, 2024

Uh oh!

larsoner commented Oct 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants