Rate Limits and Retries: The Polite Scraper’s Playbook

The fastest scraper is almost never the one that wins. The one that wins is the one still running next month, and the difference between the two is almost entirely about how it handles rate. A scraper that pulls as hard as it can gets noticed, throttled, and blocked, usually within hours. A scraper that moves at a pace the site can absorb keeps going and going. This is the playbook for that second kind: how to handle rate limits, retries, and backoff so your collection stays alive.

I run production scrapers for a living, the kind that have to keep working for months without becoming somebody else’s incident, so this is the part I care about most. It isn’t glamorous. Nobody makes a flashy demo about waiting politely between requests. But it’s the actual skill that separates a toy that runs once from infrastructure that runs for a year. And almost all of it comes down to one idea: take only what the site can comfortably give, and back off the moment it signals otherwise.

What a rate limit really is

Start from the site’s side, because that’s who sets the rate. A rate limit is simply the site saying you may make this many requests in this much time, and no more. It exists to protect the site from being overwhelmed, whether by an attack, a bug, or a scraper pulling too hard. A rate limit isn’t an insult or a challenge to beat. It’s the site telling you exactly how much load it’s willing to carry from you. The polite move, and the durable one, is to listen to that number and stay under it.

The status code that says slow down

Sites have a standard way of telling you that you’ve gone too fast. When you cross a limit, a well behaved server responds with a status code, most commonly 429, which means too many requests. Often it comes with a header that literally tells you how long to wait before trying again. This is the site handing you the answer. The worst thing a scraper can do is ignore that signal and keep hammering, because that’s exactly the behavior that turns a temporary slowdown into a permanent block. Read the code, honor the wait.

Backoff is the core move

The central technique is backoff. When a request fails or gets throttled, you don’t retry instantly. You wait, and you wait longer each time it keeps failing. The first retry might wait a second, the next a few seconds, the next longer still. This gives the site room to recover and signals that you’re a well behaved client, not a battering ram. A naive retry loop that fires again the instant it fails is the single most common way people turn a recoverable hiccup into a hard ban. Backoff is what makes retries safe instead of dangerous.

Why the waits grow

The reason the wait grows each time is worth understanding. A single failure might be random noise, so a short wait and a retry usually clears it. But repeated failures mean something is actually wrong: the site is struggling or actively throttling you, and the right response is to pull back harder, not keep pushing at the same rate. Growing the delay each time means a brief blip costs you almost nothing, while a real problem makes you gracefully retreat instead of piling on. The pattern matches your pressure to the site’s actual state.

Add jitter or you stampede

Here’s a subtle one people miss. If you run many workers and they all fail at the same moment and all back off by the same amount, they’ll all retry at the same instant, producing a synchronized wave that hits the site like a hammer. The fix is jitter: adding a small random amount to each wait so the retries spread out instead of bunching up. It’s a tiny change with a big effect. Without jitter, your polite backoff can accidentally become a coordinated stampede, and the site feels a spike exactly when you meant to ease off.

A backoff in a few lines

The shape of it is small. You catch the failure, compute a growing delay with a little randomness, and try again up to a limit. That’s exponential backoff with jitter, the whole idea in a handful of lines, and it will carry you a long way.

The token bucket idea

Beyond reacting to failures, you want to control your rate before you ever trip a limit. A clean way to think about this is a token bucket. You get a steady supply of tokens, one per allowed request, and each request spends one. When the bucket is empty, you wait for it to refill. This smooths your traffic into an even, predictable stream instead of bursts, and it lets you set your own ceiling comfortably below whatever the site allows. Shaping your rate proactively is far better than sprinting until the site slams the door and then reacting.

Concurrency is a dial, not a maximum

People treat concurrency as a number to maximize, and that’s the mistake. Running more requests at once feels like progress, but past a point it just increases the load the site feels from you and the chance you get flagged. Treat concurrency as a dial you set deliberately, low enough that the site never strains, not as high as your hardware could technically push. I routinely run well below what my machines could handle, on purpose, because the bottleneck I care about is the site’s tolerance, not my throughput. Slower and alive beats fast and banned, every time.

Respect the crawl delay

Many sites publish their preferences in a robots file, and that file often includes a crawl delay, a request to wait a certain amount of time between hits. Honor it. Treating that number as a real instruction rather than a suggestion is both the courteous thing and the smart thing, because a site that publishes a crawl delay is telling you the exact pace at which it will tolerate you. Staying at or under that pace keeps you invisible in the best way: a small, steady stream of requests the site has explicitly said it can carry. Ignoring it is asking for trouble you were warned about.

Cache so you never ask twice

The cheapest request is the one you never make. A huge amount of scraper load is wasteful, fetching the same page again because the pipeline wasn’t tracking what it already had. So cache aggressively. Store what you pull, and before you request anything, check whether you already have a fresh copy. This cuts your load on the site dramatically, speeds up your own job, and shrinks the surface where anything can go wrong. Every page you serve from your own cache is a page you didn’t have to ask the site for, and politeness and efficiency point the same direction here.

Spread the load across time

If you have a big job, don’t try to finish it in one aggressive burst. Spread it across time. A hundred thousand pages pulled gently over a day is nearly invisible, while the same hundred thousand pulled as fast as possible in an hour is a spike that any monitoring will catch. The total work is identical, but the shape the site feels is completely different. Patience is a feature here. A job that isn’t urgent should be spread wide and thin, because the flatter your traffic, the less reason the site ever has to look at you twice.

The circuit breaker

There’s one more pattern worth having, borrowed from resilient systems, called a circuit breaker. The idea is that when failures pile up past a threshold, you stop entirely for a while instead of continuing to retry. If a site is returning errors on nearly everything, that’s a clear sign to pause the whole job, not to keep probing at it. The breaker trips, you wait a good while, and then you test cautiously before resuming. This stops a struggling site from turning into a hard block, and it keeps you from wasting effort against a wall that isn’t going to move right now.

Read the response, not just pass or fail

Politeness also means paying attention to what the site is telling you beyond the raw status. Rising latency, subtle warnings, a challenge appearing where there wasn’t one before, these are all signals that you’re pushing harder than the site wants. A good scraper watches those and eases off before it gets to an outright block. By the time you’re seeing hard failures, you’ve already been rude for a while. The skill is reading the soft signals early and slowing down on your own, so the site never has to make you.

Retry only what is safe to repeat

One caution about retries: they must be safe to repeat. If a request has a side effect, submitting a form or changing something, retrying it can do that twice, and you end up with duplicate or corrupt results. Keep retries to operations that only read, and design your storage so writing the same record twice does no harm. Safe retries are what let backoff work without fear, because you can try again knowing a repeat costs you nothing.

Why polite scrapers last

Put it all together and the philosophy is simple. The durable scraper is the considerate one. It stays under the limit, backs off when told, spreads its load, caches what it can, and reads the site’s signals instead of bulldozing through them. Done this way, most of the blocking machinery never even has a reason to fire, because you aren’t producing the load that triggers it. You’re just a small, steady, well behaved stream of requests, and a site has very little reason to fight a client that isn’t causing it any trouble in the first place.

The honest limits

Let me be straight about the boundaries. Handling rate well keeps a scraper healthy and welcome, but it doesn’t change what you’re allowed to collect. Public data, a robots file respected, an official API or bulk feed preferred where one exists. Politeness isn’t a trick to sneak past anything, it’s the absence of the load that gets you noticed, and it works precisely because it’s genuine restraint rather than a disguise. And no amount of careful pacing makes it right to take data that was never yours, or to ignore what a site has clearly asked you not to do.

To recap: a rate limit is the site telling you how much load it will carry, so listen to it. Honor the slow down signals, retry with backoff that grows each time, add jitter so your workers don’t stampede, and shape your rate proactively with something like a token bucket. Treat concurrency as a deliberate dial, respect the crawl delay, cache so you never ask twice, spread big jobs across time, and trip a circuit breaker when a site is clearly struggling. Be the considerate client, and you get to keep running.

For more breakdowns on scraping infrastructure, proxy setups, and pipeline design straight from production use, visit the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.