There are newly documented restrictions on tp_traverse:
The traversal function must not have any side effects. It must not modify the reference counts of any Python objects nor create or destroy any Python objects.
But, whether a C-API function modifies some reference count is an implementation detail that we can change at any time. Core devs can add logging, an audit event, or a warning, to virtually any function.
This implies that you can't safely use any C API function in tp_traverse, unless it guarantees that it has no side effects.
I propose to add such side-effect-free versions of some functions useful in tp_traverse, with a _DuringGC suffix -- namely:
PyObject_GetTypeData_DuringGCPyObject_GetItemData_DuringGCPyType_GetModuleState_DuringGCPyModule_GetState_DuringGCPyModule_GetToken_DuringGCPyType_GetBaseByToken_DuringGCPyType_GetModule_DuringGCPyType_GetModuleByToken_DuringGC
Compared to their non-suffixed variants, the main difference is that we promise that these will remain safe to use in tp_traverse. The other differences are that these functions cannot fail, and (where applicable) they return borrowed references rather than strong ones.
Also, Py_VISIT, Py_SIZE, PyObject_VisitManagedDict as well as the visitproc that Python passes to tp_traverse will be documented to work like this.
Users that call any _DuringGC function (including some type's tp_traverse) must manually must ensure that the type, MRO, and layout of any argument doesn't concurrently change. That means only calling them from tp_traverse.
See the discussion (and preferably join that if you have input): https://discuss.python.org/t/adding-c-api-for-use-in-tp-traverse/105331/13
(Note that I agree that preferring all functions to be fallible makes things less ergonomic. But being consistent is important, and ergonomics issues are exactly when we add suffixed variants of functions -- _DuringGC in this case.)
There are newly documented restrictions on tp_traverse:
But, whether a C-API function modifies some reference count is an implementation detail that we can change at any time. Core devs can add logging, an audit event, or a warning, to virtually any function.
This implies that you can't safely use any C API function in
tp_traverse, unless it guarantees that it has no side effects.I propose to add such side-effect-free versions of some functions useful in
tp_traverse, with a_DuringGCsuffix -- namely:PyObject_GetTypeData_DuringGCPyObject_GetItemData_DuringGCPyType_GetModuleState_DuringGCPyModule_GetState_DuringGCPyModule_GetToken_DuringGCPyType_GetBaseByToken_DuringGCPyType_GetModule_DuringGCPyType_GetModuleByToken_DuringGCCompared to their non-suffixed variants, the main difference is that we promise that these will remain safe to use in
tp_traverse. The other differences are that these functions cannot fail, and (where applicable) they return borrowed references rather than strong ones.Also,
Py_VISIT,Py_SIZE,PyObject_VisitManagedDictas well as thevisitprocthat Python passes totp_traversewill be documented to work like this.Users that call any
_DuringGCfunction (including some type'stp_traverse) must manually must ensure that the type, MRO, and layout of any argument doesn't concurrently change. That means only calling them fromtp_traverse.See the discussion (and preferably join that if you have input): https://discuss.python.org/t/adding-c-api-for-use-in-tp-traverse/105331/13
(Note that I agree that preferring all functions to be fallible makes things less ergonomic. But being consistent is important, and ergonomics issues are exactly when we add suffixed variants of functions --
_DuringGCin this case.)