Hello! I'm currently interning at Quansight, with @ngoldbaum as my mentor. We are working on making NumPy's test suite more thread safe in an effort to make NumPy have better support for free-threaded Python.

When running NumPy's test suite with pytest-run-parallel, there are numerous failures, with many thread-unsafe methods and modules causing tests to fail for various reasons (such as AssertionErrors due to global state being modified between tests, or the most common: AttributeErrors due to pytest's xunit setup and teardown methods being thread-unsafe). I have identified all these thread-unsafe tests, and have begun fixing some of these issues.

The thread-unsafe tests are documented in this spreadsheet, as well as my progress on fixing them in my own fork. Feel free to take a look, and feel free to modify the spreadsheet with your own findings if you decide to tackle some of these.

Progress

• Update classic pytest xunit setup and teardown methods into regular methods or fixtures. These methods are thread-unsafe, often not running before the tests and resulting in AttributeErrors. This is the large majority of failures found, and will likely expose more thread safety issues once fixed.
• Some tests require a lot of memory, and will crash when ran on multiple threads. Marking them with pytest.mark.parallel_threads(n) to only run on n threads should help.
• Update tests that use np.random.seed() to use Generator or an instance of RandomState instead. np.random is global state, which often results in AssertionErrors, with random number generation not acting as expected.
• For tests that use tmp_path and tmpdir, ensure files generated by tests on parallel threads are unique. Could utilize uuids? pytest-run-parallel is currently looking into patching tmp_path to be thread safe.
• Fix up tests that race to modify parameterized values, potentially use a fixture or method generator instead.
• Fix or mark tests that use pytest fixtures that are thread-unsafe (capfd, recwarn).
• Mark or remove unittest.mock.patch.
• Fix tests that seem to have thread safety issues with warnings.
• Mark tests that have unreliable results with hypothesis.
• Mark tests that use sys.getrefcount() and produce unexpected results.
• Mark tests that use assert_no_gc_cycle.
• Adjust numpy/random/tests/test_smoke.py to utilize bit generation per class/test, currently bit generation is modifying global state.
• Fix tests that have file races.
• Mark tests that utilize stdout.
• Mark tests that utilize thread-unsafe features of NumPy (such as np.genfromtext(), which generates unexpected results).
• Mark any tests that purposefully modify global state or engage in thread-unsafe techniques that cannot be changed (such as tests that purposefully modify doc strings).
• Get pytest-run-parallel running for the entire test suite
• Get a CI job implemented into NumPy that runs the test suite under parallel threads.

Additional Notes

• monkeypatch is thread-unsafe, and is setup for all tests with this fixture in conftest.py. Because of this, almost every test will be marked thread-unsafe because of monkeypatch. As of TST: delete global env_setup fixture #29648 this has been fixed.
• To prevent pytest.warns from being marked thread-unsafe:
  • Set the PYTHON_CONTEXT_AWARE_WARNINGS environment variable or -X context_aware_warnings command to 1.
  • Set the PYTHON_THREAD_INHERIT_CONTEXT environment variable -X thread_inherit_context command to 1.
  • Both of these are set to 1 by default when using a free-thread version of Python