-
-
Notifications
You must be signed in to change notification settings - Fork 12.2k
Support for the new Limited API and Stable ABI in Python 3.15 (PEP 803/809) #30704
Description
Note: PEP 803 and PEP 809 - which are two competing PEPs with large overlap to define a new Stable ABI that will work with both GIL-enabled and free-threaded interpreters - are still in Draft state. This issue tries to summarize what would be needed if one of them got accepted.
NumPy doesn't use the Limited C API and Stable ABI itself, but does support other projects targeting the Stable ABI while using the NumPy headers. Support for that was implemented about two years ago in gh-25531, shipped in NumPy 2.0. It isn't widely used yet, but seems to work fine.
I worked on PyWavelets using the Stable ABI recently (PyWavelets/pywt#828), and decided to try that with PEP 803 and CPython 3.15a5, given that there are branches of Cython (cython#7399) and meson-python (mgorny/meson-python#freethreading-limited-api).
After doing the plumbing to make PyWavelets use the new abi3t, I got an unpleasant surprise from NumPy:
....
[24/32] Compiling C object pywt/_extensions/_pywt.abi3.so.p/meson-generated__pywt.c.o
samu: job failed: cc -Ipywt/_extensions/_pywt.abi3.so.p [snip] D_Py_OPAQUE_PYOBJECT -DPy_LIMITED_API=0x030f0000 -o pywt/_extensions/_pywt.abi3.so.p/meson-generated__pywt.c.o -c pywt/_extensions/_pywt.abi3.so.p/_pywt.c
In file included from /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
from /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
from pywt/_extensions/_pywt.abi3.so.p/_pywt.c:1164:
/home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/ndarraytypes.h:654:9: error: unknown type name 'PyObject_HEAD'; did you mean 'PyObject_DEL'?
PyObject_HEAD went missing - and it's pretty widely used in NumPy:
$ rg PyObject_HEAD | wc -l
78
Which shouldn't have been so surprising, given it's spelled out in https://peps.python.org/pep-0803/#opaque-pyobject. But I didn't quite connect the dots between that section of the PEP and how widely the now-disappearing macros are used in NumPy internals:
$ rg PyObject_HEAD | wc -l
78
$ rg PyObject_EXTRA_INIT | wc -l
0
$ rg PyObject_HEAD_INIT | wc -l
5
$ rg PyObject_VAR_HEAD | wc -l
3
$ rg Py_SET_TYPE | wc -l
38
0
$ rg "\->ob_type" | wc -l
22
$ # sizeof(PyObject) may also be annoying, can't easily grep for that
EDIT: for other projects to use the Limited C API with the NumPy headers, we should count only in include/numpy/, which doesn't look as bad:
$ rg PyObject_HEAD | wc -l
36
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_EXTRA_INIT | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_HEAD_INIT | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_VAR_HEAD | wc -l
1
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg Py_SET_TYPE | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg "\->ob_type" | wc -l
1
And there's the "They [opaque PyObjects] cannot be embedded in other structures." That applies to PyArrayObject I'd think. That would need to all be replaced by PEP 697 APIs, which is probably not great for performance. And seems to require the work for heap types that we (as maintainers) wanted to avoid doing ourselves for subinterpreters - see gh-24755, the discussion there is probably relevant.
The PR to allow use of the Limited API with NumPy headers from 2 years ago was pretty straightforward, however this looks like quite the project, with potential performance implications.
Taking a step back, the main two changes that are common between PEP 803 and 809 and affect NumPy are:
- Use
PyModExportas the new entry point for C extension modules, see PEP 793. - Support opaque
PyObject(no separate PEP, see https://peps.python.org/pep-0803/#opaque-pyobject for details).
(1) seems straightforward, (2) not so much.