Retrying a failed request without hammering a site

Why naive retries break things

The simplest retry logic looks like this: request fails, request again, immediately, in a loop, until it succeeds or you give up. It works fine in a demo. In production, against a real site, it’s one of the fastest ways to get a proxy pool burned and an IP range added to a blocklist.

Here’s the mechanism. A failed request usually means something is already under strain, either your proxy, your network path, or the target’s backend. If you retry instantly, you’re adding load to a system that just told you it couldn’t handle the load. Multiply that by a few hundred concurrent workers in a scraping pipeline, and a single upstream hiccup turns into a retry storm: hundreds of requests per second hitting the same endpoint at the exact moment it’s least able to handle them. From the target’s side, that pattern is indistinguishable from an attack. From your side, it just looks like your job is running normally, until every proxy in the pool gets rate-limited or banned within minutes.

The fix isn’t complicated, but it has to be deliberate. You need to decide what counts as retryable, how long to wait between attempts, how many attempts to allow, and how to stop the whole pipeline from retrying in lockstep.

Not every failure deserves a retry

The first decision point is classifying the failure. Lumping every non-200 response into “retry it” is the root cause of most retry storms.

Transient, worth retrying:
– Connection timeouts and DNS failures
– 502, 503, 504 (the server or its upstream is overloaded or restarting)
– 429 (rate limited, explicitly telling you to slow down)

Terminal, don’t retry the same way:
– 403 and 401 (you’re blocked or unauthorized, and hitting it again with the same identity changes nothing)
– 404 (the resource isn’t there, retrying won’t create it)
– 400 (your request is malformed, retrying sends the same malformed request)

A 403 in particular deserves its own path. If a site is returning 403 to a specific proxy or session, retrying with that same identity just confirms to the target that the block is working and gives it more data to fingerprint your traffic pattern by. The correct response to a block is to stop, log it, and route future requests through a different identity and a longer cool-down, not to hammer the same door.

Exponential backoff, and why linear delays aren’t enough

Once you’ve decided a failure is retryable, the question is how long to wait. A fixed delay, say, retry every 5 seconds, has a subtle failure mode: if the underlying cause is server overload, all your queued retries converge on the same 5-second cadence and keep hitting the server in synchronized waves, never giving it room to recover.

Exponential backoff fixes this by doubling (or otherwise increasing) the wait time with each attempt:

delay = base_delay * (2 ^ attempt_number)

So a base delay of 1 second gives you 1s, 2s, 4s, 8s, 16s across five attempts. This spreads load out over time instead of concentrating it, and it naturally gives an overloaded server more room to recover the longer the outage lasts.

Cap it. Unbounded exponential growth means a job that started failing at 9am is still trying every four hours the next week. A max delay (commonly 30 seconds to a few minutes for scraping workloads, longer for background jobs) and a max attempt count (3 to 5 is typical) keep the backoff from running forever on a URL that’s permanently gone.

Jitter: the part people skip

Backoff alone isn’t enough if you’re running concurrent workers. If ten workers all hit a timeout on the same target at the same second, and they all use identical exponential backoff, they’ll all retry at 1s, then all retry again at 2s, then at 4s, still synchronized, just spaced further apart. The server sees the same thundering-herd pattern, just at a lower frequency.

Jitter breaks the synchronization by adding randomness to the delay:

delay = random(0, base_delay * (2 ^ attempt_number))

This is “full jitter.” There are variants (adding a random offset to a fixed exponential value instead of randomizing the whole range), but the goal is the same: spread retries out so concurrent workers don’t converge on the same timestamp. AWS’s architecture blog has a widely cited writeup on this exact problem in the context of distributed systems, and the same logic applies directly to a scraper fleet hitting the same domain from many workers.

Respect Retry-After when a site gives it to you

A 429 or 503 response sometimes comes with a Retry-After header, either a number of seconds or an HTTP date. That header is the target telling you exactly how long it wants you to wait before trying again. Ignoring it and applying your own backoff schedule instead is a good way to keep getting 429s, because you’re guessing at a number the server already gave you directly.

Parse it, honor it, and only fall back to your own backoff calculation when it’s absent. If a site is consistently sending Retry-After values, that’s also a signal worth logging separately from generic timeouts, since it usually means you’re bumping against an explicit, enforced rate limit rather than incidental server load.

Circuit breakers: stopping retries at the fleet level

Per-request backoff handles one failing request. It doesn’t handle the case where a target has gone down entirely, or has started blocking your whole IP range, and every worker in your pipeline is independently discovering that fact and independently retrying it.

A circuit breaker sits above the retry logic, at the level of “requests to this domain” rather than “this one request.” It tracks the failure rate for a target over a rolling window. Past a threshold (for example, more than half of requests to a domain failing in the last minute), it trips: new requests to that domain get short-circuited immediately, without even attempting the network call, for a cool-down period. After the cool-down, it lets a small number of test requests through (the “half-open” state) to check if the target has recovered, and only resumes full traffic if those succeed.

This is standard practice in distributed systems generally rather than anything scraping-specific, but it maps directly onto the problem: it stops hundreds of independent workers from each rediscovering the same outage the hard way, and it caps the total load your pipeline puts on a struggling target regardless of how many workers are running.

Idempotency matters more than people expect

Retries assume that repeating a request is safe. For read-only scraping, GET requests, that’s almost always true. It gets riskier the moment your pipeline does anything that changes state: submitting a form, triggering a search that costs the target compute, or hitting an endpoint that logs or counts the request server-side.

If a request might have partially succeeded before it timed out (the target received it and started processing, but the response never made it back to you), a blind retry can duplicate whatever that request caused. This is a basic correctness question more than an ethics one: know which of your requests are safe to repeat and which aren’t, and don’t apply the same retry policy uniformly across both.

What this actually protects you from

None of this makes a scraper undetectable or guarantees it won’t get blocked. Bot detection systems don’t only look at retry patterns, they look at headers, TLS fingerprints, timing, navigation patterns, and a lot more that backoff logic has no bearing on. What proper backoff and jitter do is remove one specific, avoidable failure mode: your own pipeline turning a normal transient error into a load pattern that looks like an attack, burning through proxies and getting IPs blocked for reasons that had nothing to do with detection sophistication and everything to do with a while True: retry() loop.

It’s infrastructure hygiene, not an evasion technique. A pipeline that backs off politely still gets blocked by sites that don’t want to be scraped. It just doesn’t create that outcome for itself on infrastructure that would otherwise have kept working.

If you’re setting up retry and backoff logic for your own pipeline and want to see how we structure orchestration, proxy rotation, and failure handling in practice, take a look at what we’ve published on Data Research Tools.

Get new guides and videos first — join the Telegram channel.