asyncio throws runtime error with exception ignored

Question

Below is a simple program to collect URL lengths.

import aiohttp
import asyncio
from time import perf_counter


URLS = ['http://www.cnn.com', 'http://www.huffpost.com', 'http://europe.wsj.com',
        'http://www.bbc.co.uk', 'http://failfailfail.com']

async def async_load_url(url, session):
    try:
        async with session.get(url) as resp:
            content = await resp.read()
        print(f"{url!r} is {len(content)} bytes")
    except IOError:
        print(f"failed to load {url}")        

async def main():

    async with aiohttp.ClientSession() as session:
        tasks = [async_load_url(url, session) for url in URLS]
        await asyncio.wait(tasks)

if __name__ == "__main__":
    start = perf_counter()
    asyncio.run(main())
    elapsed = perf_counter() - start
    print(f"\nTook {elapsed} seconds")

Why is the following code failing with a runtime error with exception ignored in python 3.9? How to fix it?

Traceback is: RuntimeError: Event loop is closed specifically with Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x000001F8A7A713A0>

Is it a runtime error? Or is it a warning? It probably wants you to do something with the tasks returned by asyncio.wait. — dirn
– dirn, Commented Jun 24, 2021 at 23:15
Looks like you're on Windows, in which case this is a known issue in aiohttp github.com/aio-libs/aiohttp/issues/4324 — Matt Fowler
– Matt Fowler, Commented Jun 25, 2021 at 17:05

Matt Fowler · Accepted Answer · 2021-06-25 23:03:27Z

10

This is caused by a known issue in aiohttp on Windows, for details check out the bug at https://github.com/aio-libs/aiohttp/issues/4324

There are a couple of hacks that silence this error. The first way is to get the event loop and call run_until_complete instead of asyncio.run(main()) like so:

asyncio.get_event_loop().run_until_complete(main())

Alternatively, changing the event loop policy to WindowsSelectorEventLoopPolicy before calling asyncio.run(main()) also works as the problem seems to occur when using WindowsProtractorEventLoopPolicy.

asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(main())

Of course, the second solution will make your code platform specific, so be careful.

answered Jun 25, 2021 at 23:03

Matt Fowler

2,7932 gold badges18 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jsbueno · Accepted Answer · 2021-06-25 21:01:34Z

-1

You are using asyncio.await to run all your tasks, but you are not checking the completed tasks for exceptions. await returns two sequences: one with the completed tasks, one with the pending tasks - you have to query the done tasks for exceptions:

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.task(async_load_url(url, session), name=url) for url in URLS]
        done, pending = await asyncio.wait(tasks)
        for task in done:
            try:
                task.exception() # in task execption is re-raised
            except Exception as exc:
                print (f"Exception loading url {task.name}:\n {exc}")

If it is a long process and you want to handle exceptions as they happen, asyncio.wait offers an interface that facilitates that - just tell it when it should return:

        
    async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.task(async_load_url(url, session), name=url) for url in URLS]
        while tasks:
            done, tasks = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
            for task in done:
                try:
                    task.exception() # in task execption is re-raised
                except Exception as exc:
                    print (f"Exception loading url {task.name}:\n {exc}")

answered Jun 25, 2021 at 21:01

jsbueno

114k11 gold badges160 silver badges241 bronze badges

1 Comment

Martijn Pieters Over a year ago

The exception is not raised by the tasks. They are raised when the event loop is closed and the event loop objects are deleted when their reference count drops to zero. Arguably this is a bug in the Protractor loop __del__ implementation for not verifying that it is already closed before trying to execute any remaining callbacks but no amount of task exception checking will fix this.

Collectives™ on Stack Overflow

asyncio throws runtime error with exception ignored

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related