Skip to content

Support for the new Limited API and Stable ABI in Python 3.15 (PEP 803/809) #30704

@rgommers

Description

@rgommers

Note: PEP 803 and PEP 809 - which are two competing PEPs with large overlap to define a new Stable ABI that will work with both GIL-enabled and free-threaded interpreters - are still in Draft state. This issue tries to summarize what would be needed if one of them got accepted.

NumPy doesn't use the Limited C API and Stable ABI itself, but does support other projects targeting the Stable ABI while using the NumPy headers. Support for that was implemented about two years ago in gh-25531, shipped in NumPy 2.0. It isn't widely used yet, but seems to work fine.

I worked on PyWavelets using the Stable ABI recently (PyWavelets/pywt#828), and decided to try that with PEP 803 and CPython 3.15a5, given that there are branches of Cython (cython#7399) and meson-python (mgorny/meson-python#freethreading-limited-api).

After doing the plumbing to make PyWavelets use the new abi3t, I got an unpleasant surprise from NumPy:

     ....
     [24/32] Compiling C object pywt/_extensions/_pywt.abi3.so.p/meson-generated__pywt.c.o
      samu: job failed: cc -Ipywt/_extensions/_pywt.abi3.so.p [snip] D_Py_OPAQUE_PYOBJECT -DPy_LIMITED_API=0x030f0000 -o pywt/_extensions/_pywt.abi3.so.p/meson-generated__pywt.c.o -c pywt/_extensions/_pywt.abi3.so.p/_pywt.c
      In file included from /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
                       from /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
                       from pywt/_extensions/_pywt.abi3.so.p/_pywt.c:1164:
      /home/rgommers/code/tmp/python-builds/py315t-21jan/lib/python3.15t/site-packages/numpy/_core/include/numpy/ndarraytypes.h:654:9: error: unknown type name 'PyObject_HEAD'; did you mean 'PyObject_DEL'? 

PyObject_HEAD went missing - and it's pretty widely used in NumPy:

$ rg PyObject_HEAD | wc -l
78

Which shouldn't have been so surprising, given it's spelled out in https://peps.python.org/pep-0803/#opaque-pyobject. But I didn't quite connect the dots between that section of the PEP and how widely the now-disappearing macros are used in NumPy internals:

$ rg PyObject_HEAD | wc -l
78
$ rg PyObject_EXTRA_INIT | wc -l
0
$ rg PyObject_HEAD_INIT | wc -l
5
$ rg PyObject_VAR_HEAD | wc -l
3
$ rg Py_SET_TYPE | wc -l
38
0
$ rg "\->ob_type" | wc -l
22
$ # sizeof(PyObject) may also be annoying, can't easily grep for that

EDIT: for other projects to use the Limited C API with the NumPy headers, we should count only in include/numpy/, which doesn't look as bad:

$ rg PyObject_HEAD | wc -l
36
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_EXTRA_INIT | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_HEAD_INIT | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg PyObject_VAR_HEAD | wc -l
1
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg Py_SET_TYPE | wc -l
0
(base) ~/code/numpy/numpy/_core/include/numpy (main)$ rg "\->ob_type" | wc -l
1

And there's the "They [opaque PyObjects] cannot be embedded in other structures." That applies to PyArrayObject I'd think. That would need to all be replaced by PEP 697 APIs, which is probably not great for performance. And seems to require the work for heap types that we (as maintainers) wanted to avoid doing ourselves for subinterpreters - see gh-24755, the discussion there is probably relevant.

The PR to allow use of the Limited API with NumPy headers from 2 years ago was pretty straightforward, however this looks like quite the project, with potential performance implications.


Taking a step back, the main two changes that are common between PEP 803 and 809 and affect NumPy are:

  1. Use PyModExport as the new entry point for C extension modules, see PEP 793.
  2. Support opaque PyObject (no separate PEP, see https://peps.python.org/pep-0803/#opaque-pyobject for details).

(1) seems straightforward, (2) not so much.

Metadata

Metadata

Assignees

No one assigned

    Labels

    39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)63 - C APIChanges or additions to the C API. Mailing list should usually be notified.component: numpy._core

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions