How Many Proxies Does Your Scraper Actually Need

The question I get more than almost any other is some version of “how many proxies do I need for this scrape?” People want a round number back, a clean fifty or a thousand they can drop into a budget. The honest answer is that it’s arithmetic, and the number falls out of two things: how fast you want to pull, and how fast the target will quietly tolerate each address. Get that math wrong one way and every address you rent gets burned in an afternoon. Get it wrong the other way and you’re paying for a huge pool you never needed, sometimes making yourself easier to spot in the process.

I run proxy infrastructure and production scrapers, so I size pools for real jobs with real budgets, not off a vendor’s marketing page. Here’s the actual math I use, the formula that turns a target rate into an address count, and the parts most people skip.

Sizing is a rate problem, not a count problem

Throw out the idea that this is a count problem. You’re not really asking how many addresses you need, you’re asking how much traffic you want to push and how thinly you have to spread it so no single address gets noticed. The pool exists to divide your total load across enough addresses that each one stays under the rate a site will accept from a single visitor. Once you see it as spreading a rate, the number almost calculates itself.

The two numbers you actually need

You need two numbers before you can size anything.

The first is your target request rate, how many requests per second you want the whole job to make. That comes from how much data you need and how quickly you need it. You choose it.

The second is the safe rate per ip, how many requests a single address can make against this particular site before that site starts treating it as suspect. You don’t choose this one. The site decides it, and you have to go and measure it. It’s a property of the target, not of your proxies, which is why you can’t read it off anyone’s spec sheet. You find it by running a small test against the real site: one address, slowly raising the rate, watching for the point where the block rate climbs or challenges start appearing. Then you set your safe rate comfortably under that ceiling.

The formula

The formula is simple enough to hold in your head:

ip count is roughly your target request rate divided by the safe rate per ip.

That’s the whole thing. If you want the job to make some number of requests per second, and each address can safely make some smaller number per second, then the first divided by the second tells you how many addresses it takes to carry that load without any one of them going too fast. Everything else is just adjusting that result.

A worked example

Say you want the whole scrape to run at 20 requests per second. And say you’ve measured that this site tolerates about one request every five seconds from a single address before it gets unhappy. That’s a safe rate of roughly 0.2 requests per second per ip.

Divide 20 by 0.2 and you get 100. So you need on the order of 100 addresses to push 20 requests per second while keeping each one at a pace the site accepts. That’s your starting number, not your final one.

Add headroom

100 is the floor, because reality is messier than the math. Some addresses in any pool are already burned before you touch them. Some go stale partway through the job. Some fail for reasons that have nothing to do with you. So pad the clean number, usually by 30 to 50 percent, so the job still hits its rate when a chunk of the pool is unusable. 100 becomes closer to 150 in practice. Round up, never down.

Concurrency is a different dial, and people tangle the two. Concurrency is how many requests are in flight at once. Rate is how many you complete per second over time. The formula is about rate. Don’t size your pool off parallel connections, size it off the sustained rate you push through them.

More proxies is not always safer

Here’s the part that surprises people. A bigger pool is not automatically safer, and past the number the math calls for, more addresses can hurt you.

Every address you add is one more with a reputation you didn’t build and can’t see. A huge pool spread thin is a crowd of strangers, and if a slice of it is already dirty, you’ve inherited blocks you never earned. You’re also paying for capacity you aren’t using, which on metered residential pools is real money leaking for nothing.

There’s a subtler cost too. Edge defenders don’t just watch single addresses, they watch across a whole site and across many sites at once. A sudden swarm of fresh addresses all crawling the same paths at the same speed is itself a pattern, even when no single address is going too fast. Throwing a thousand brand new addresses at a job that only needed a hundred can raise your footprint instead of lowering it. The right size is the smallest pool that carries your rate politely, not the biggest one you can afford.

Pool size and rotation are the same constraint

Rotation and sizing aren’t separate topics. Rotation just cycles you through the pool, handing out a fresh address on a schedule or per request. But rotation can only spread your load if the pool is big enough. If the pool is too small, the rotator comes back around to the same address too soon, and that address ends up carrying more than its safe rate anyway. A pool below the size the formula asks for doesn’t lower your per address rate, no matter how you rotate.

Think of each address as needing a rest between uses. If the safe rate is one hit every five seconds, then after an address is used it should sit idle for about that long before it comes up again. The pool has to be large enough to fill that gap for every address in the rotation. In effect, pool size is your rate multiplied by how long each address needs to rest.

Sticky sessions size differently

Stateful work flips the calculation. If you’re walking a login, a cart, or a multi step flow that has to stay on one visitor, you can’t rotate mid session without contradicting yourself. There you size by concurrent sessions instead of by raw rate: roughly one address per session you want running at once, plus a batch of spares for the ones that fail. It’s a different formula for a different shape of job, and mixing the two up is how people end up wildly over or under provisioned.

Proxy type changes the number, not the math

The type of proxy doesn’t change the formula, it changes the safe rate that goes into it. On a strict site a datacenter address might tolerate almost nothing before it’s flagged, so the safe rate is tiny and the math demands a lot of them. A residential address reads as an ordinary person and tolerates more, so you need fewer. Mobile is the odd one: a single mobile address is already shared behind carrier grade nat by many real subscribers, so you rarely size it by raw address count at all. You size it by concurrent sessions and rotation instead. The tier you pick moves the denominator, which moves the count, but the arithmetic underneath is the same.

The honest limits

Sizing a pool correctly keeps you polite and keeps the job alive. It does not make anything undetectable, and no address count changes what you’re allowed to collect. Public data only, the robots file respected, an official api or bulk feed preferred wherever one exists. The formula is about carrying your load without leaning on any single address too hard. It’s not a way around the rules.

And ignore the headline pool size a vendor prints on the box. Millions of addresses tells you nothing about how many you need or how many will actually work on your target. Your number comes from your own measured safe rate against your own site, run through the formula, plus headroom. I’ve watched a tiny well maintained pool beat a giant dirty one on the same job. Measure first, size from the measurement, and treat the marketing number as noise.

I run this infrastructure in production, so the sizing and the pools I write about are the ones I actually use and test. If you want the full written guides and the proxy pools I genuinely rely on, read them here. No undetectable promises, no guaranteed results, just the way this actually works.

Get new guides and videos first — join the Telegram channel.