Here is a stripped sample of something similar to what I've used. I can also think of examples in IPC in which state is encoded in the response and you need to send part of the response back in order to perform your next request. Something like a Redis "SCAN" operation. I have no actual implements doing that with asyncio, but just a thought.

import aiohttp
import asyncio
import random
from bs4 import BeautifulSoup

# Galleries would normally be fetched too. But for simplicity sake
# hardcode them.
GALLERIES = [
    'https://example.com/gallery/1',
    'https://example.com/gallery/2',
    'https://example.com/gallery/3',
    'https://example.com/gallery/4',
]

async def extract_image_srcs(session, gallery_url):
    async with session.get(gallery_url) as gallery_response:
        gallery_soup = BeautifulSoup(await gallery_response.text(), features='lxml')
        return [img.attrs['src'] for img in gallery_soup.findAll('img')]

async def get_image_data(session, image_src):
    async with session.get(image_url) as image_response:
        return await image_response.read()

async def fetch_gallery(gallery_url):
    session = aiohttp.ClientSession()
    image_sources = await extract_image_srcs(session, gallery_url)
    for image_src in image_sources:
        # This could technically be "gathered" but await is used to artifically
        # reduced the number of potential requests. We can't guarantee any
        # specific number, but it'll definitely be less than doing everything
        # at once.
        image_data = await get_image_data(session, image_src)
        with open(str(random.randint(1, 255)), 'wb') as fd:
            fd.write(image_data)
        # Optionally include additional waiting if we want to be extra nice.
        await asyncio.sleep(random.randint(1, 10))
    await session.close()

async def bootstrap():
    await asyncio.gather(*[fetch_gallery(gallery_url) for gallery_url in GALLERIES])

asyncio.run(bootstrap())

Uh oh!

Forbid await in for loop #1600

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Forbid `await` in `for` loop #1600