How To Make A Scraper Faster: Concurrency And Async (2026)

The first scraper I was ever proud of ran all night to do a job that should have taken twenty minutes. It walked a catalog one page at a time: ask for a page, wait for the answer, save it, ask for the next one. Patient, polite, and painfully slow. When I finally looked at where the time actually went, the machine was doing almost nothing. It was sitting idle, waiting for a server on the other side of the world to answer, thousands of times in a row.

That is the whole story of scraper performance, and it is why concurrency is the single biggest lever you have. This is how you fetch many things at once so your scraper stops spending its life waiting.

Why a scraper is slow

A slow scraper is almost never a slow computer. The bottleneck in scraping is hardly ever your processor working too hard. It is your program standing still, waiting on the network.

You send a request, and then there is a long pause while it crosses the network, the server thinks, and the answer travels back. During that pause your code is just holding its breath. If you do one request at a time, you sit through every one of those pauses back to back, and they add up to the whole night. Concurrency is the trick of overlapping those waits, so that while one request is out waiting, twenty others are out waiting at the same time.

Concurrency is not pulling harder

Draw a clean line here, because this is where people get into trouble. Going concurrent is not the same as pulling harder. The goal is not to hit the site with more force, it is to stop wasting the idle time you already had.

That distinction matters. The moment concurrency turns into a flood aimed at one site, you are back in the world of rate limits, getting throttled and blocked. Done right, concurrency makes you faster without making you louder to any single target.

The three models

There are really only three ways to run many requests at once, and each one fits a different bottleneck.

The async event loop is built for exactly this job. Picture a single thread that never sits still. It fires off a request and, instead of waiting for the answer, immediately turns to the next request and fires that one too, keeping hundreds or thousands of them in the air. When an answer comes back, the loop picks it up and deals with it. Because the work of a scraper is waiting rather than computing, one core running one loop can happily manage thousands of open requests. In Python this is the world of asyncio and the async HTTP libraries.

Threads are the simpler picture, and for a lot of scrapers they are more than enough. You keep a pool of, say, twenty workers, and each one does the plain blocking thing. You may have heard that Python threads cannot really run at the same time because of the global interpreter lock, and for heavy computation that is true. But it does not hurt you here, because your threads are waiting on the network, not computing, and a thread that is waiting steps aside and lets the others run.

Processes are the heavy option, and you reach for them only when your bottleneck moves to parsing: huge pages, heavy pattern matching, tangled documents that take real processor time to pick apart. For most scrapers the parsing is cheap and the waiting is the whole cost, so you rarely need processes just to fetch.

Bounded concurrency is the number that matters

The beginner move, the day you discover you can fetch many at once, is to fire all ten thousand URLs at the same instant. Do not. That is a self inflicted disaster. You open ten thousand sockets, exhaust your own memory and file handles, your machine falls over, and the site sees a wall of traffic arrive in one heartbeat and blocks you on the spot.

What you want is bounded concurrency: a fixed number of requests allowed in flight at once, say ten or twenty, with the rest waiting their turn. A semaphore, or a worker pool of a fixed size, is how you enforce it. This one number is the difference between a fast scraper and a self inflicted outage.

Reuse your connections

Before you even touch concurrency, there is a free speedup most people leave on the table: reusing your connections. Every time you open a brand new request from scratch, you pay for a fresh TCP and TLS handshake, a little round trip of setup before any real work happens.

If you reuse one session and let it keep the connection alive between requests, you skip that setup on every call after the first. It makes you faster, and it is genuinely lighter on the site, fewer connections to accept. One session, reused, is the cheapest win there is.

Cap concurrency per host

Here is the subtlety that separates a scraper that survives from one that gets banned in an hour. Your concurrency should be counted per site, not as one global number. Twenty requests in flight spread across twenty different sites is gentle, two at each. Twenty requests in flight all aimed at one site is a hammer.

So cap how many you run against any single host, keep that per site number low, and let your overall speed come from working many hosts at once. That is how you stay fast in total while staying polite to each individual target.

Where proxies come in

Concurrency and your address are linked. If you run twenty or fifty requests at once all leaving from a single IP, you have concentrated a burst of traffic on one address, and that is exactly the pattern that gets an address flagged and shown the door.

The fix is to spread your concurrent requests across a healthy pool of addresses, so no single one carries the whole burst. Concurrency multiplies your footprint, and a good proxy pool is what distributes it. This is where a clean mobile proxy setup earns its keep on heavy concurrent runs.

Timeouts, queues, and retries

Once you have many requests in flight, three details keep the whole thing from quietly falling apart.

Timeouts stop being optional. If one host goes slow and never answers, that request holds its slot forever, and enough of them will occupy every slot in your fixed pool. Your scraper looks like it is running, but nothing is moving. Give every request a hard timeout so a hung connection gives up and frees its slot.

A bounded queue keeps memory sane. You have one side producing URLs and another side of workers consuming them. If the producer races ahead, it loads a million URLs into memory before a single one is done. A bounded queue applies back pressure, so a fast producer cannot drown a careful fetcher.

Retries have to share the budget. A failed request should try again after a short backoff, but a retry is still a request and still costs an in flight slot. Count retries against the same concurrency budget, or a bad patch where everything is failing will silently double your load at the worst possible moment.

Measure throughput, not requests per second

Raw requests per second is the wrong number to chase. The number that matters is useful throughput: good rows or good pages collected per minute, measured right next to your block rate.

It is easy to turn concurrency up, watch requests per second climb, and feel great, while your block rate climbs right alongside it and half those requests come back as block pages. That is not progress, it is going faster into a wall. So tune it like an experiment: start low, raise the limit while real throughput climbs and blocks stay flat, and settle in just under the point where throughput plateaus or blocks tick up. Every site has its own sweet spot, and the only honest way to find it is to measure your way there.

The honest limits

Faster is not the same as allowed more. Concurrency is about not wasting the idle time you already had, never about overwhelming a site into giving up more than it wants to. Public data stays the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how many hosts you work at once.

I run this in production every day, real scrapers pulling real volume on a schedule, and getting the concurrency model right is the difference between a job that finishes before breakfast and one that runs all night and still gets blocked. The full written picks I actually use, the exact limits, the async fetch loop, and the per host caps, live at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.