Python Asynchronous Requests a Web Scraping Tutorial

Why asynchronous requests matter for scraping

Web scraping is overwhelmingly an I/O-bound task. Your code is not crunching numbers; it is waiting on networks. A traditional synchronous loop using the popular requests library blocks on every call, so total runtime is roughly the sum of every individual wait. Asynchronous code instead uses a single event loop that can juggle hundreds of in-flight requests, switching to whichever one is ready. The result is dramatically higher throughput on the same hardware.

It is worth being clear about the limits. Async does not make any single request faster, and it does not parallelise CPU-heavy parsing in the way multiprocessing does. Its strength is overlapping wait time. For most scraping jobs, where the bottleneck is network latency rather than computation, that is exactly the win you want.

The core building blocks: asyncio and aiohttp

Python ships asyncio in its standard library, providing the event loop, coroutines and scheduling. For HTTP, the widely used companion is aiohttp, an asynchronous client built to cooperate with that loop. Together they let you define coroutines with async def and pause them at network boundaries using await.

import asyncio
import aiohttp

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, u) for u in urls]
        return await asyncio.gather(*tasks)

urls = ["https://example.com/page/%d" % i for i in range(1, 50)]
results = asyncio.run(main(urls))

Here a single shared ClientSession manages connection pooling. The asyncio.gather call schedules every fetch concurrently and collects the results once they all complete. This is the heart of asynchronous scraping in just a few lines.

Controlling concurrency so you stay polite

Firing fifty requests at once is fine for a demo, but firing five thousand can overwhelm a target server, trip rate limits, or get your IP blocked. A semaphore caps how many requests run simultaneously, keeping your scraper fast yet considerate.

sem = asyncio.Semaphore(10)

async def fetch(session, url):
    async with sem:
        async with session.get(url) as response:
            return await response.text()

Practical tuning tips

• Start with a modest concurrency limit and raise it while watching for errors.
• Add small randomised delays between requests to avoid an obvious machine-gun pattern.
• Respect each site's robots file and published rate guidance.
• Set sensible timeouts so one slow server cannot stall the whole batch.

Adding proxies and rotation

Even a well-behaved async scraper sends many requests in a short window, and from a single IP that pattern stands out quickly. Routing requests through proxies spreads traffic across many addresses, reducing the chance any one IP is throttled or banned. With aiohttp you pass a proxy per request, which makes rotation simple.

async def fetch(session, url, proxy):
    async with session.get(url, proxy=proxy) as response:
        return await response.text()

Rotating residential or datacenter proxies pair naturally with async concurrency: as you scale the number of simultaneous requests, a larger and more diverse pool keeps each address from being overused. When choosing a provider, weigh pool size, location coverage and reliability against price. A strong value-focused option worth considering is Cheapest Proxies, our featured value pick for keeping per-request costs low while you experiment with concurrency.

Handling errors and retries gracefully

At high concurrency, transient failures are normal rather than exceptional. Connections reset, servers return temporary error codes, and proxies occasionally drop. Robust async scrapers wrap fetches in try/except blocks, retry with a backoff delay, and rotate to a fresh proxy on repeated failure rather than abandoning the URL.

• Catch network exceptions and retry a small, fixed number of times.
• Use exponential backoff so retries do not pile pressure on a struggling server.
• Log which URLs ultimately fail so you can revisit them later.

When async is not the right tool

If your job involves heavy parsing, image processing or other CPU work per page, the event loop can become the bottleneck because that work blocks the single thread. In those cases pairing async I/O with a process pool, or stepping up to a full framework like Scrapy, may serve you better. Choose the simplest approach that meets your throughput needs.

Choosing between aiohttp, httpx and Trio-based stacks

The base walkthrough centres on aiohttp, but it is not the only option and the choice has real consequences. httpx exposes nearly identical sync and async APIs, so a team can prototype with the blocking client and flip to async with minimal rewriting; it also speaks HTTP/2, which some targets prefer. aiohttp tends to win on raw throughput for large fan-outs and has a mature connector model. For people who find asyncio's edges sharp, Trio and the anyio compatibility layer offer stricter structured concurrency that makes cancellation and error propagation easier to reason about. None is universally best; pick by whether you value a single dual-mode library, peak speed, or safety guarantees.

Connection pools, limits and the hidden bottleneck

A semaphore caps how many coroutines run your fetch logic, but it does not directly cap how many TCP sockets open. In aiohttp the TCPConnector carries its own limit and limit_per_host settings, and if you forget them the library defaults quietly govern your real concurrency. Two scrapers with the same semaphore value can behave very differently because one left the connector wide open and the other throttled it. The practical rule is to set the connector limit and your semaphore in agreement, and to use limit_per_host so a single slow domain cannot soak your whole socket budget.

Settings worth pinning explicitly

• Total connector limit, so sockets never exceed what you intend.
• Per-host limit, so one domain cannot crowd out the rest of a multi-site crawl.
• DNS cache TTL, to avoid hammering resolvers on huge URL lists.
• Keep-alive timeout, balancing reuse against holding idle sockets open.

Backpressure: producing URLs faster than you consume them

Calling asyncio.gather on a million tasks builds a million coroutine objects in memory before a single one finishes. At small scale that is fine; at large scale it is how scrapers quietly exhaust RAM. The mature pattern is a bounded producer-consumer design using an asyncio.Queue with a fixed maximum size: a producer feeds URLs in, a fixed pool of worker coroutines pulls them out, and the queue's bound applies natural backpressure so you never materialise the whole workload at once. This also lets you write each parsed result to disk or a database immediately, keeping memory flat regardless of job size.

Observability so a long crawl is not a black box

A scrape that runs for hours needs to tell you how it is doing. Track requests attempted, succeeded, retried and abandoned, plus a rolling success rate per proxy and per host. A sudden collapse in success rate usually means you have crossed a rate limit or burned a batch of IPs, and seeing it live lets you dial concurrency down before damage spreads. Lightweight counters and periodic log lines are enough; the point is that high-concurrency async hides individual failures inside gather, so you must surface aggregate health yourself.

Keep comparing with these closely related pages.