After merging this to main, the debug test runs started segfaulting. @seberg any ideas?

https://github.com/numpy/numpy/actions/workflows/build_test.yml

Originally posted by @mattip in #20729 (comment)