6

Given this simple snippet, why does the first anext() call work but the second anext() results in a strange error?

Reading the documentation of anext(), I would assume that the supplied default would be used when there is no "next" value available:

If default is given, it is returned if the iterator is exhausted, otherwise StopAsyncIteration is raised.

Running this in an (interactive) Python 3.12 interpreter on MacOS Apple M3 Pro,

import asyncio

async def generator(it=None):
    if it is not None:
        yield (it, it)

async def my_func():
    # results in a=1 b=1
    a, b = await anext(generator(1), (2, 3))
    
    # results in no printing
    async for a, b in generator():
        print(a, b)
    
    # raises exception
    a, b = await anext(generator(), (2, 3))

loop = asyncio.new_event_loop()
loop.run_until_complete(my_func())

results in this exception:

StopAsyncIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/manuel/.pyenv/versions/3.12.0/lib/python3.12/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "<stdin>", line 6, in my_func
SystemError: <class 'StopIteration'> returned a result with an exception set

My snippet fails in Python 3.11 & 3.13 too with the same exception message.

1
  • It's suspicious that the exception is StopIteration, not StopAsyncIteration. Commented Dec 17, 2024 at 21:28

2 Answers 2

6

This is a Python bug, and should be reported.


When the anext-with-default-value implementation hits StopAsyncIteration, it calls _PyGen_SetStopIterationValue to replace the exception with a StopIteration. I'm not quite sure whether this is happening in __next__ or send, but both methods do essentially the same thing in this case. Here's __next__'s handling:

    if (PyErr_ExceptionMatches(PyExc_StopAsyncIteration)) {
        _PyGen_SetStopIterationValue(obj->default_value);
    }

_PyGen_SetStopIterationValue looks like this:

int
_PyGen_SetStopIterationValue(PyObject *value)
{
    PyObject *e;

    if (value == NULL ||
        (!PyTuple_Check(value) && !PyExceptionInstance_Check(value)))
    {
        /* Delay exception instantiation if we can */
        PyErr_SetObject(PyExc_StopIteration, value);
        return 0;
    }
    /* Construct an exception instance manually with
     * PyObject_CallOneArg and pass it to PyErr_SetObject.
     *
     * We do this to handle a situation when "value" is a tuple, in which
     * case PyErr_SetObject would set the value of StopIteration to
     * the first element of the tuple.
     *
     * (See PyErr_SetObject/_PyErr_CreateException code for details.)
     */
    e = PyObject_CallOneArg(PyExc_StopIteration, value);
    if (e == NULL) {
        return -1;
    }
    PyErr_SetObject(PyExc_StopIteration, e);
    Py_DECREF(e);
    return 0;
}

This function needs to perform special handling when the argument is a tuple or an exception object. In your case, the argument is a tuple (2, 3). But the special handling has an issue.

There is already an active exception set, the StopAsyncIteration. It is not safe to call arbitrary Python APIs with an exception set. All the calls in the usual-case handling are okay, but this line in the special-case handling:

e = PyObject_CallOneArg(PyExc_StopIteration, value);

is unsafe. Python detects an inconsistent state, resulting in the error you see.

To fix this issue, either _PyGen_SetStopIterationValue or the code that calls it should use PyErr_Clear to cancel the current exception before the PyObject_CallOneArg(PyExc_StopIteration, value) call happens.

Sign up to request clarification or add additional context in comments.

2 Comments

Well done. After some debugging in Python I couldn't even get the proper traceback so I was 75% sure it's a bug in CPython implementation but I don't have this much experience with C to find the reason. I deleted my answer after reading yours.
@user2357112 I created an issue with reference to your analysis.
-3

i solved the issue by replacing the tuple in the default parameter with a list, so [2,3] in stead of (2,3). i cannot tell you why it doesn't work with a tuple, but this works for me.

a, b = await anext(generator(), [2, 3])

4 Comments

Re: edit -- compare to the upvoted answer, which does actually have a proper explanation. A workaround is not an answer to a "why" question, nor a general-purpose fix.
@CharlesDuffy that answer came 3 hours after my answer, and doesn't directly provide a solution, although it describes the problem well. i don't get why this makes me deserve a downvote, when i actually put a solution in there, whilst admitting that i don't know why a tupple doesn't work and 3 hours prior to any explanation showing up, i think with or without anyone showing up to explain the why my answer could be helpful to the person who asked the question (and people ending up here from google). certainly with what was available at the point of posting it. but also now.
Again, the question asked was "why" -- not how to fix it, but why it happens. This makes no effort at all to answer that question. (And while no solution was asked for, the other answer does provide a solution -- it's a solution that requires patching the Python interpreter, but that's how it goes sometimes).
It's not my downvote. I'm just fulfilling your request for an explanation for why someone attuned to local standards and practices might consider it worth one. (If you were online right now I could prove that by adding and then withdrawing my own downvote, thus demonstrating that I hadn't already done so previously -- but since a downvote that's added can't be withdrawn without the content being edited after a time window, that's not a demonstration that can be done asynchronously)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.