Serhiy Storchaka reported that Python 3.6 crashs earlier than Python 3.5 on calling json.dumps() when sys.setrecursionlimit() is increased.

I tested the script he wrote. Results on Python built in release mode:

Python 3.7:

...
58100 116204
Segmentation fault (core dumped)

Python 3.6:

...
74800 149604
Segmentation fault (core dumped)

Python 3.5:

...
74700 149404
Segmentation fault (core dumped)

Oh, it seems like Python 3.7 does crash earlier.

But to be clear, it's hard to control the usage of the C stack.

Oh, I didn't understand that the regression was introduced by the revision b9c9691c72c5. The purpose of this revision was to *reduce* the memory usage of the C stack!?

It seems like _PyObject_CallArg1() uses more stack memory than PyObject_CallFunctionObjArgs(). PyObject_CallFunctionObjArgs() allocates 4O bytes (5*sizeof(PyObject*)) on the stack.

At least, I can say that when the crash occurs, _PyObject_FastCallDict() is not the gdb backtrace.

Yes, that is why I asked you to revert your changes.

In additional, they introduced compiler warnings.

New changeset d35fc6e58a70 by Victor Stinner in branch 'default':
Backed out changeset b9c9691c72c5
https://hg.python.org/cpython/rev/d35fc6e58a70

> Serhiy Storchaka reported that Python 3.6 crashs earlier than Python 3.5 on calling json.dumps() when sys.setrecursionlimit() is increased.

Reference: http://bugs.python.org/issue23507#msg282190 (issue #23507).


Serhiy Storchaka: "Yes, that is why I asked you to revert your changes."

Sorry, I misunderstood your comments. So yes, my change b9c9691c72c5 introduced a regression. Sorry, I didn't have time before now to revert my  change. I just pushed the change d35fc6e58a70 which reverts b9c9691c72c5.

The question is how replacing PyObject_CallFunctionObjArgs() with _PyObject_CallArg1() increases the usage of the C stack. I wrote my change to reduce the usage of the C stack.

PyObject_CallFunctionObjArgs() allocates 5 "PyObject *", so 40 bytes, on the C stack. Maybe using _PyObject_CallArg1() increases the usage of C stack in the *caller*.


> In additional, they introduced compiler warnings.

This one was fixed by Benjamin Peterson in the issue #28855 (change 96245d4af0ca).

Thanks Victor.

Following script includes several examples of achieving a stack overflow (most are real world examples or can be used in real world examples). It measures maximal deep for every type of recursion.

When I wrote the _PyObject_CallArg1(), it looks as a cool hack:

#define _PyObject_CallArg1(func, arg) \
    _PyObject_FastCall((func), (PyObject **)&(arg), 1)

It hacks the declaration of an explicit "stack" like:

   PyObject *stack[1];
   stack[0] = arg;
   res = _PyObject_FastCall(func, stack, 1);

And I expected that the C compiler magically computes the memory address of the argument. But it seems like requesting the memory address of an argument allocates something on the C stack.

On x86_64, first function arguments are passed with CPU registers. Maybe requesting the memory address of an argument requires to allocate a local variable, copy the register into the variable, to get the address of the local variable?

So, I suggest to *remove* the _PyObject_CallArg1() macro, and use existing functions like PyObject_CallFunctionObjArgs().

What do you think Serhiy?

That was my initial preference. Mainly because this doesn't add code churn.

But I don't understand how PyObject_CallFunctionObjArgs() that uses _PyObject_CallArg1() and has many local variables can consume less stack than _PyObject_CallArg1() itself.

New changeset 4171cc26272c by Victor Stinner in branch 'default':
Issue #28858: Remove _PyObject_CallArg1() macro
https://hg.python.org/cpython/rev/4171cc26272c

The changeset 4171cc26272c "Remove _PyObject_CallArg1() macro" fixed the initial bug report, so I now close the issue.

Serhiy: If you see further enhancements, please open a new issue. Your second stack_overflow.py script is interesting, but I don't see any obvious possible changes to enhance results.

Thanks Serhiy for digging into this issue ;-)