What is the use case for future.add_done_callback()?

Question

I understand how to add a callback method to a future and have it called when the future is done. But why is this helpful when you can already call functions from inside coroutines?

Callback version:

def bar(future):
    # do stuff using future.result()
    ...

async def foo(future):
    await asyncio.sleep(3)
    future.set_result(1)

loop = asyncio.get_event_loop()
future = loop.create_future()
future.add_done_callback(bar)
loop.run_until_complete(foo(future))

Alternative:

async def foo():
    await asyncio.sleep(3)
    bar(1)

loop = asyncio.get_event_loop()
loop.run_until_complete(foo())

When would the second version not be available/suitable?

user4815162342 · Accepted Answer · 2024-08-10 07:47:25Z

30

In the code as shown, there is no reason to use an explicit future and add_done_callback, you could always await. A more realistic use case is if the situation were reversed, if bar() spawned foo() and needed access to its result:

def bar():
    fut = asyncio.create_task(foo())
    def when_finished(_fut):
        print("foo returned", fut.result())
    fut.add_done_callback(when_finished)

If this reminds you of "callback hell", you are on the right track - Future.add_done_callback is a rough equivalent of the then operator of JavaScript promises. (Details differ because then() is a combinator that returns another promise, but the basic idea is the same.)

A large part of asyncio is implemented in this style, using non-async functions that orchestrate async futures. That basic layer of transports and protocols feels like a modernized version of Twisted, with the coroutines and streams implemented as a separate layer on top of it, a higher-level sugar. Application code written using the basic toolset looks like this.

Even when working with non-coroutine callbacks, there is rarely a good reason to use add_done_callback, other than inertia or copy-paste. For example, the above function could be trivially transformed to use await:

def bar():
    async def coro():
        ret = await foo()
        print("foo returned", ret)
    asyncio.create_task(coro())

This is more readable than the original, and much much easier to adapt to more complex awaiting scenarios. It is similarly easy to plug coroutines into the lower-level asyncio plumbing.

So, what then are the use cases when one needs to use the Future API and add_done_callback? I can think of several:

Writing new combinators.
Connecting coroutines code with code written in the more traditional callback style, such as this or this.
Writing Python/C code where async def is not readily available.

To illustrate the first point, consider how you would implement a function like asyncio.gather(). It must allow the passed coroutines/futures to run and wait until all of them have finished. Here add_done_callback is a very convenient tool, allowing the function to request notification from all the futures without awaiting them in series. In its most basic form that ignores exception handling and various features, gather() could look like this:

async def gather(*awaitables):
    futs = [asyncio.ensure_future(aw) for aw in awaitables]
    remaining = len(futs)
    finished = asyncio.get_event_loop().create_future()
    def fut_done(fut):
        nonlocal remaining
        remaining -= 1
        if not remaining:
            finished.set_result(None)  # wake up
    for fut in futs:
        fut.add_done_callback(fut_done)
    await finished
    # all awaitables done, we can return the results
    return tuple(f.result() for f in futs)

Even if you never use add_done_callback, it's a good tool to understand and know about for that rare situation where you actually need it.

edited Aug 10, 2024 at 7:47

answered Dec 10, 2018 at 17:19

user4815162342

159k22 gold badges353 silver badges420 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user2232305 Over a year ago

Why is awaiting on the awaitables not a good idea in this situation? rs = [] ; for x in xs: rs.append(await x) ; return rs

Mesco Feb 26 at 12:29

because this way you implementaed synchronous chain of awaiting asynchronous futures. One by one. gather in this implementaion awaits for all of them at once, code of each x can interlace each other

user4815162342 Feb 26 at 12:42

That will depend on the contents of xs. If xs contains futures (like the futs list in the code in the answer), then this pattern will actually await them in parallel, because during each individual await all futures will run. But if xs contains coroutines that haven't yet been submitted to the event loop, then awaiting them in sequence will indeed serialize them. This is why I responded that whether this is a good idea depends on details that weren't shown. And of course, if unsure, it's best to use gather() and friends.

user4815162342 Over a year ago

@user2232305 I don't know that it's not a good idea, but it might depend on the details you haven't shown. Since the issue is unrelated to this answer, please ask it as a separate question. If you do so, be sure to include the context of that you're trying to do.

auxsvr · Accepted Answer · 2023-11-22 08:39:16Z

2

add_done_callback has an important use case that is often neglected: there must be a (strong) reference to a task, otherwise it may be garbage-collected before it is finished, so add_done_callback can be used with clean-up code that removes completed tasks from the container of tasks; cf. the documentation in https://docs.python.org/3/library/asyncio-task.html for create_task.

answered Nov 22, 2023 at 8:39

auxsvr

3266 silver badges10 bronze badges

Collectives™ on Stack Overflow

What is the use case for future.add_done_callback()?

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related