Skip to content

numpy version 2.0 compatibility#6776

Merged
jni merged 14 commits intonapari:mainfrom
GenevieveBuckley:numpy-v2-compat
May 15, 2024
Merged

numpy version 2.0 compatibility#6776
jni merged 14 commits intonapari:mainfrom
GenevieveBuckley:numpy-v2-compat

Conversation

@GenevieveBuckley
Copy link
Copy Markdown
Contributor

@GenevieveBuckley GenevieveBuckley commented Mar 27, 2024

References and relevant issues

Description

This PR:

  1. Adds a ruff linting rule to check compatibility with numpy v2.0, and
  2. Updates existing napari code to ensure compatibility with numpy v2.0

@github-actions github-actions bot added tests Something related to our tests qt Relates to qt labels Mar 27, 2024
@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

GenevieveBuckley commented Mar 27, 2024

Status:

  • Done: Added ruff linting rule for numpy 2.0 compatibility check
  • Done: Removed references to numpy.sctypes, which is removed in numpy v2.0
  • Done: Replaced np.set_string_function(None) --> np.set_printoptions() (since numpy.set_string_function is removed in numpy v2.0) EDIT: nope, this did not reset the whole thing like I expected, it only resets the formatter kwarg values. it's better to use np.set_printoptions as a context manager instead.
  • Need help: How do I replace np.set_string_function(lambda arr: f'{type(arr)} {arr.shape} {arr.dtype}') with an equivalent np.set_printoptions command? This is the numpy.set_printoptions docs, and these are the old numpy.set_string_function docs. I can change the formatting of individual numpy array elements, but I can't figure out how to make it return a single summary string for the whole array.

As expected, the new ruff linting check is failing until we fix this last np.set_string_function problem:

napari/utils/_tracebacks.py:37:13: NPY201 `np.set_string_function` will be removed in NumPy 2.0. Use `np.set_printoptions` for custom printing of NumPy objects.
napari/utils/_tracebacks.py:91:17: NPY201 `np.set_string_function` will be removed in NumPy 2.0. Use `np.set_printoptions` for custom printing of NumPy objects.
napari/utils/_tracebacks.py:137:17: NPY201 `np.set_string_function` will be removed in NumPy 2.0. Use `np.set_printoptions` for custom printing of NumPy objects.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.41%. Comparing base (f195690) to head (0bd57fc).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6776      +/-   ##
==========================================
- Coverage   92.45%   92.41%   -0.04%     
==========================================
  Files         614      614              
  Lines       55164    55162       -2     
==========================================
- Hits        51001    50979      -22     
- Misses       4163     4183      +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

cc @JoOkuma - maybe you have a suggestion for this? It'd be nice to keep something similar to the print formatting style you introduced in #2910, but I'm not sure how to do that using the options available in numpy v2.0

@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

I've looked into the numpy.set_printoptions docs some more, and I don't think we can reproduce the old behaviour exactly. But I can reduce how verbose the array is in the print output. Is this sufficient, or does anyone have a better plan?

arr = np.random.random((512,512))

Old:

<class 'numpy.ndarray'> (512, 512) float64

New (with np.printoptions(precision=5, threshold=80, edgeitems=2):)

[[0.98454 0.93896 ... 0.54017 0.85307]
 [0.48407 0.53626 ... 0.83477 0.55798]
 ...
 [0.13319 0.35231 ... 0.89343 0.93346]
 [0.15881 0.52209 ... 0.04007 0.71324]]

Default (no printoptions settings)

array([[0.98453704, 0.9389569 , 0.4197145 , ..., 0.89444065, 0.54017021,
        0.85306717],
       [0.48406665, 0.53625644, 0.58014584, ..., 0.0319012 , 0.83477052,
        0.55798271],
       [0.68847506, 0.41386651, 0.55052169, ..., 0.28323962, 0.66602634,
        0.86895102],
       ...,
       [0.21836951, 0.1869048 , 0.7513639 , ..., 0.51765584, 0.53707556,
        0.97249828],
       [0.13318953, 0.35231209, 0.7720745 , ..., 0.11958483, 0.89343037,
        0.9334581 ],
       [0.15881113, 0.52209381, 0.24165429, ..., 0.69698159, 0.04007437,
        0.7132449 ]])

@GenevieveBuckley GenevieveBuckley marked this pull request as ready for review March 27, 2024 05:19
@JoOkuma
Copy link
Copy Markdown
Contributor

JoOkuma commented Mar 27, 2024

Hi @GenevieveBuckley, I couldn't find how to reproduce the old behavior in numpy 2.0.0.

There's a formatter argument, but it seems to be applied per value.

Changing the options to have a smaller output, as you did, might be a good compromise.

@JoOkuma
Copy link
Copy Markdown
Contributor

JoOkuma commented Mar 27, 2024

This PR looks good to me.

@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

Hi @GenevieveBuckley, I couldn't find how to reproduce the old behavior in numpy 2.0.0.

There's a formatter argument, but it seems to be applied per value.

Changing the options to have a smaller output, as you did, might be a good compromise.

Yeah, that's what I found too.
It's good to hear you think that's a reasonable compromise.

@GenevieveBuckley GenevieveBuckley added this to the 0.4 milestone May 10, 2024
@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

Ready for review

Copy link
Copy Markdown
Collaborator

@Czaki Czaki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we relax constraint <2 on NumPy version (line 52 in pyproject.toml) and run pre-test, or is there a known problem why we cannot test against numpy 2.0?

@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

Could we relax constraint <2 on NumPy version (line 52 in pyproject.toml) and run pre-test, or is there a known problem why we cannot test against numpy 2.0?

Sure, we can do that. I've pushed this branch to napari/napari, and set the --pre test workflow running here: https://github.com/napari/napari/actions/runs/9055845357

@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

There's one failing job from --pre test: windows-latest py3.12 pyqt6 --pre

I've run the test a second time, same Windows access error. But windows-latest with pyqt5 passes fine, it's only windows with pyqt6 that has this problem. I'll look into it a bit more if I have some time.

  napari\_vispy\_tests\test_vispy_labels_layer.py ........Windows fatal exception: access violation
  Current thread 0x00001368 (most recent call first):
    File "D:\a\napari\napari\napari\utils\_testsupport.py", line 318 in make_napari_viewer
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\fixtures.py", line 893 in _teardown_yield_fixture
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\fixtures.py", line 1009 in finish
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 544 in teardown_exact
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 188 in pytest_runtest_teardown
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 241 in <lambda>
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 341 in from_call
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 240 in call_and_report
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 136 in runtestprotocol
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 116 in pytest_runtest_protocol
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 364 in pytest_runtestloop
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 339 in _main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 285 in wrap_session
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 332 in pytest_cmdline_main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\config\__init__.py", line 178 in main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\config\__init__.py", line 206 in console_main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pytest\__main__.py", line 7 in <module>
    File "<frozen runpy>", line 88 in _run_code
    File "<frozen runpy>", line 198 in _run_module_as_main
  py312-windows-pyqt6-no_cov: exit 3221225477 (195.62 seconds) D:\a\napari\napari> python -m pytest --color=yes --basetemp=\tmp\.tox\py312-windows-pyqt6-no_cov\tmp --ignore tools --maxfail=5 --json-report --json-report-file=D:\a\napari\napari/report-py312-windows-pyqt6-no_cov.json --save-leaked-object-graph pid=1004
  py312-windows-pyqt6-no_cov: FAIL code 3221225477 (384.62=setup[188.95]+cmd[0.05,195.62] seconds)
  evaluation failed :( (385.03 seconds)
Error: The process 'C:\hostedtoolcache\windows\Python\3.12.3\x64\python.exe' failed with exit code 4294967295
The process 'C:\hostedtoolcache\windows\Python\3.12.3\x64\python.exe' failed with exit code 4294967295

@github-actions github-actions bot added the maintenance PR with maintance changes, label May 13, 2024
@GenevieveBuckley
Copy link
Copy Markdown
Contributor Author

I'm not sure if the Windows access error seen above is just a flaky test. I can't reproduce the problem on my fork, and it seems to disappear here when I add the -v verbose flag to pytest in the tox.ini file. Perhaps this is not an actual problem, and we can merge as is.

@Czaki
Copy link
Copy Markdown
Collaborator

Czaki commented May 13, 2024

It is a known problem with pyqt6 and our cleaning mechanism.

@jni jni modified the milestones: 0.4, 0.5.0 May 13, 2024
@jni jni added the ready to merge Last chance for comments! Will be merged in ~24h label May 13, 2024
@jni jni merged commit c74f20b into napari:main May 15, 2024
@jni
Copy link
Copy Markdown
Member

jni commented May 15, 2024

Thank you very much for getting this done @GenevieveBuckley!!! I kept seeing the NumPy announcements thinking, really should get onto that... 😅 😬 😂 So I'm super grateful!!!

@github-actions github-actions bot removed the ready to merge Last chance for comments! Will be merged in ~24h label May 15, 2024
jni pushed a commit that referenced this pull request Jun 13, 2024
# References and relevant issues
I don't see a tracking issue for numpy 2.0 support, but here are some
previous PRs I've found:
#6932
#6776

And a zulip thread:

https://napari.zulipchat.com/#narrow/stream/212875-general/topic/handling.20the.20numpy.202.2E0.20release/near/381330412

# Description
This fixes a few more numpy-2.0 related issues. I will comment on some
of them inline.

I tested locally using [a wheel from my vispy
PR](https://github.com/vispy/vispy/actions/runs/9424977460?pr=2599). I
also uninstalled `tensorstore` to skip related tests (see
google/tensorstore#165).

There are still a few test failures in
`napari/layers/image/_tests/test_image.py` that look possibly related to
Xarray:
```
napari/layers/image/_tests/test_image.py:649 test_image_scale[scale5] - DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and… [1012/3861]
```

I also get a failure on
`napari/_qt/widgets/_tests/test_qt_tooltip.py::test_qt_tooltip_label`,
but I think this may be a macOS thing (it passes if I mouse over it).

Other than that, local tests with `pytest napari` are all green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance PR with maintance changes, qt Relates to qt tests Something related to our tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add NumPy 2.0 linting

4 participants