PEP 803: Stable ABI for Free-Threaded Builds

[emmatyping (Emma Smith)]

Thank you for drafting this PEP Petr! I am excited to see a path to a stable ABI for free-threaded.

@encukou

And would you prefer the bump to come at the beginning of the transition period, or at the end? :‍)

I know this was originally asked of Paul, but I think it is important to have the (eventually) incompatible, future ABI available as early as possible into the transition to free-threading being the default. The community needs time to test and adopt these changes.

@encukou

For PEP 803, an important premise is that building an extension twice (or rather, a “small” number of times), for different ABIs, is not an issue for extension authors.
If that premise is wrong, let’s do things differently.

Cython and nanobind, two of the more popular binding generators, only relatively recently gained support for targeting the stable ABI. I expect with changes to the limited API and the ABI, these projects and others like them will need significant work to be made compatible with the new API/ABI and also keep compatibility with generating bindings for older versions. So while “building one more wheel” is generally not that hard, I expect it will be a lot harder and take a fair bit of time to get changes into the hands of many extension authors.

@Jelle

@pf_moore

I don’t think that these days anyone associates “abi3” with “Python 3”

For what it’s worth, I do associate it with Python 3. (But I understand that using “abi4” for free-threading is a reasonable choice.)

I agree with Jelle here, I also associate “abi3” with “any Python 3.x”, and I think the Python documentation on the stable ABI pretty much says the current stable ABI applies to any (later) Python 3.x release:

To enable this, Python provides a Stable ABI: a set of symbols that will remain ABI-compatible across Python 3.x versions.

Furthermore, PEP 425 (compatibility tags) states that “abi3” means “the Stable ABI”:

The CPython stable ABI is abi3 as in the shared library suffix.

Taken together, it reads to me that “abi3” means “works across Python 3 versions.” I think it would be a major departure from what is specified and documented to change the compatibility story of what “abi3” means (aka, there is some future Python 3 version where an abi3 wheel won’t work).

@encukou

@philthompson

Things would be easier (but not impossible) if that ABI
supported the existing ABI for a couple of years or so.

It should be possible to make a single ABI compatible with 3.11+ (non-free-threaded) and 3.15+ (all builds).

Personally, I’d be happy to do it – please convince the SC that it’s a good idea! Currently it seems the complexity in testing and supporting this is not worth it.
It could be done as an “add-on” to PEP 803 – basically, extending the compatibility to lower versions.

I absolutely would love if this were feasible! I don’t know enough of the nitty gritty ABI details to know whether or not it is, but I think having a single ABI across existing supported versions would be a huge step to making migration to free-threaded Python more palatable to the community. One of the biggest complaints of 2->3 from my recollection was it was hard to be compatible with both versions and incrementally move to supporting 3.x from a big Python 2.x codebase. That became a lot easier as features were added to Python 3 like the u"" prefix and I think if we make it more or less drop in for users to move to the new ABI (and incrementally adopt the new limited API), people will be happy to do so. 3.10 is EOL when 3.15 is released, so timing wise it also seems like things line up nicely for this plan.