Do you have a self-contained example that demonstrates the problem? I quickly checked using PyObjC on macOS (which basically does this when starting threads using the Cocoa interface), and that code did not deadlock.

Do you have a self-contained example that demonstrates the problem? I quickly checked using PyObjC on macOS (which basically does this when starting threads using the Cocoa interface), and that code did not deadlock.

Ok, I investigated a little deeper and admittedly I have a bit more going on in the environment where I found this. Tried duplicating until I found the difference: tracemalloc. When tracemalloc is imported and started the function flow is as shown in the header of this issue. When it is not, then after obmalloc:PyMem_RawCalloc() it goes to obmalloc:_PyMem_DebugRawCalloc() instead of _tracemalloc:_PyMem_Raw.calloc() which avoids the recursive call to PyGILState_Ensure() and works fine. I will see about getting some poc module code here tomorrow or after.

I can reproduce using PyObjC by using python3.11 -X tracemalloc script.py to start the PyObjC using script I mentioned earlier.

@pablogsal is this a release blocker?

@pablogsal is this a release blocker?

Indeed it is. Thanks for pinging me.

@ronaldoussoren Can you share the reproducer?

Or alternatively, could you bisect the issue?

Or alternatively, could you bisect the issue?

In previous versions the memory for the new PyThreadState was allocated before the HEAD_LOCK() in pystate:new_threadstate(), if you revert to that behavior the bug is fixed.

The reproducer needs PyObjC (from v8.5-branch in https://github.com/ronaldoussoren/pyobjc because the latest release does not work with Python 3.11 yet).

The reproducer script:

from Cocoa import NSObject, NSThread
import time

class Runner(NSObject):
   def doit_(self, arg):
       print("hello world")
       print(NSThread.currentThread().isMainThread())


runner = Runner.alloc().init()

thread = NSThread.alloc().initWithTarget_selector_object_(runner, b"doit:", None)
thread.start()
print(thread)
time.sleep(1)
while thread.isExecuting():
    time.sleep(1)

This script hangs when using "python3.11 -X tracemalloc script.pyand works fine when using an older version of Python or leaving out-X tracemalloc``.

It is probably easier to reproduce this using a small C extension that uses pthread_create to launch a thread that call PyGILState_Ensure.

I can look into this during the weekend at the earliest, but also need to get a release of PyObjC out to fix Python 3.11 support (which is needed due to documented API changes, nothing to worry about there).

I will prepare a patch tomorrow. I have enough information to produce a fix.

Minimal reproducer:

#include "Python.h"
#include "pthread.h"

static struct PyModuleDef deadlockmodule = {
    PyModuleDef_HEAD_INIT,
    "deadlock",
    NULL,
    -1,
    NULL,
    NULL,
    NULL,
    NULL,
    NULL
};

static void *func(void *arg)
{
    printf("in func\n");
    PyGILState_STATE state = PyGILState_Ensure();
    printf("called PyGILState_Ensure\n");
    PyGILState_Release(state);
    printf("called PyGILState_Release\n");
    return NULL;
}

PyMODINIT_FUNC
PyInit_deadlock(void)
{
    Py_BEGIN_ALLOW_THREADS
    pthread_t thread;
    pthread_create(&thread, NULL, func, NULL);
    pthread_join(thread, NULL);
    Py_END_ALLOW_THREADS
    return PyModule_Create(&deadlockmodule);
}

@pablogsal I have a fix for this #96107

Thanks for the eyes on this, everyone, and for the fixes.

I've merged the fix into main. Now I'm wondering if it would make sense to also add a test that would have caught the bug. That wouldn't be too hard, would it @kumaraditya303?