Skip to content

BUG: Running test suite in parallel causes ThreadSanitizer data races #30085

@bwhitt7

Description

@bwhitt7

Describe the issue:

When running the test suite under pytest-run-parallel and ThreadSanitizer, many tests cause data races. The affected tests (~80 test) are listed in this spreadsheet. Below are some tests I thought were notable.

Tests cause a crash, with no output (some of these overlap with ASan errors I found previously, and likely have to do with the tests crashing due to memory issues):

  • numpy/_core/tests/test_arrayprint.py::test_multithreaded_array_printing
  • numpy/_core/tests/test_multiarray.py::TestCreation::test_zeros_big
  • numpy/_core/tests/test_regression.py::TestRegression::test_to_ctypes
  • numpy/_core/tests/test_shape_base.py::TestConcatenate::test_huge_list_error

Segmentation fault after data race

  • numpy/_core/tests/test_custom_dtypes.py::TestSFloat::test_wrapped_and_wrapped_reductions

RandomState data race (may be a linux-only bug)

  • numpy/linalg/tests/test_linalg.py::TestCond::test_stacked_singular
  • numpy/_core/tests/test_umath.py::TestLog::test_log_strides
  • numpy/linalg/tests/test_linalg.py::TestCond::test_stacked_singular

dec_addstatus data races

  • numpy/_core/tests/test_umath.py::TestRationalFunctions::test_decimal
  • numpy/lib/tests/test_function_base.py::TestAverage::test_object_dtype
  • numpy/lib/tests/test_function_base.py::TestAverage::test_object_no_weights

Caused test failure

  • numpy/lib/tests/test_io.py::TestSaveTxt::test_large_zip
  • numpy/testing/tests/test_utils.py::TestAssertNoGcCycles::test_asserts

Steps to reproduce

  • Use Python 3.14t with ThreadSanitizer enabled.
    • Steps on how to do this can be found here.
    • Currently using these TSAN options export TSAN_OPTIONS=suppressions=/numpy_suppressions.txt:allocator_may_return_null=1:halt_on_error=0
    • Using the suppression file from here
  • Build NumPy using clang, ThreadSanitizer and without openBLAS
    • spin build --clean -- -Db_sanitize=thread -Dbuildtype=debugoptimized -Dallow-noblas=true
  • Run test suite under pytest-run-parallel
    • spin test -p auto -sv

Error message:

I have the traceback files for all TSan warnings I ran into. Don't want to post them all here and make a mess, but I can provide any of them if needed.

Python and NumPy Versions:

NumPy: 2.4.0.dev0+git20251003.da5c632
Python: 3.14.0rc3+ free-threading build (heads/3.14:460ad1aa1ba, Sep 22 2025, 23:51:20) [Clang 18.1.3 (1ubuntu1)]

Runtime Environment:

[{'numpy_version': '2.4.0.dev0+git20251003.da5c632',
'python': '3.14.0rc3+ free-threading build (heads/3.14:460ad1aa1ba, Sep 22 '
'2025, 23:51:20) [Clang 18.1.3 (1ubuntu1)]',
'uname': uname_result(system='Linux', node='SPACE', release='6.6.87.2-microsoft-standard-WSL2', version='#1 SMP PREEMPT_DYNAMIC Thu Jun 5 18:30:46 UTC 2025', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Haswell',
'filepath': '/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so',
'internal_api': 'openblas',
'num_threads': 24,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.26'}]

Context for the issue:

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    00 - Bug39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions