ENH: add locking to python lapacklite interface#26687
ENH: add locking to python lapacklite interface#26687ngoldbaum wants to merge 1 commit intonumpy:mainfrom
Conversation
|
Thanks Nathan!
xref #26157 (comment) where we discussed this. I'm fine with this deviation if it's still necessary after this PR. But if the locking fixes it, I'd prefer not to do this, since it will cause other problems when building numpy from source in downstream CI. |
|
This doesn't look right, unfortunately. I don't know why we even have this The locks need to be either:
My guess is that the second one is simpler. |
|
IIRC I looked at completely removing
I think so too. |
|
I think we can just delete (or deprecate) it, the only test that looks important is for
Now, it would be nice to use our |
|
Thanks for the pointer, this is my first time touching the linalg module so the structure wasn't clear initially. I think I'll close this and do this in two followups. First, right now there isn't a way to tell from inside the source code what BLAS is being used, so I need to add some meson logic to generate a macro that is set when lapacklite is being used. Next, I can add a new global static lock in the umath_linalg module and then acquire and release it in that module. I might end up putting the locking in the functions exposed to Python rather than around every BLAS call for the sake of simplicity. |
Fixes #22509.
lapack-lite is not thread safe, with tons of static global state.
This PR adds a global lock to every call into the thread-unsafe low-level lapack-lite code. I'm sure it adds some overhead but we can ameliorate that in the future leveraging PyMutex. lapack-lite is thread-unsafe even in the standard python build, it just doesn't get hit very often, and usually in CI under odd build configurations.
With this change applied the scikit-image tests referred to in #24512, which still fail under lapacklite, pass. Ping @stefanv in case scikit-image is going out of its way to avoid lapacklite in tests.
I don't have a simple numpy-only reproducer that crashes without this change. Does anyone have a simple way to crash a non-thread-safe lapack so I can add a test?
I might make it harder to install lapack-lite in the free-threaded build as a followup.