Your cart is currently empty!
What Actually Caps Your Scrape Speed: Concurrency Limits on a Proxy Pool
What people mean when they say “concurrency limit”
Every time someone asks why their scraper stalls out at some fixed number of parallel requests, they’re usually imagining one bottleneck. In reality there are three or four, stacked on top of each other. “Concurrency limit” gets used loosely to mean requests per second, open connections, or active proxy IPs. Those are three different numbers, governed by three different systems.
Concurrency, strictly, is how many requests are in flight at the same moment. Throughput is how many complete per second. A pool can have high concurrency and low throughput if every request is slow. Or low concurrency and decent throughput, if requests are fast and short-lived. Mixing these up is the first reason people misdiagnose where their ceiling actually sits.
Layer one: the proxy provider’s own limits
Every proxy provider sets a cap on how many simultaneous connections a given port, IP, or account can hold open. This is infrastructure, not policy dressed up as policy. A gateway server has a finite number of file descriptors and upstream sockets. Let one customer open unlimited connections through a single port, and that port starves everyone sharing the same backend.
Datacenter proxies tend to publish these limits explicitly, often per port. Residential and mobile proxy pools are messier: the limit is whatever the upstream carrier or ISP tolerates on that one physical IP. It’s more than the gateway. A mobile modem sitting behind a real SIM card can only hold so many concurrent TCP sessions before the carrier’s own equipment starts dropping or delaying them, independent of anything the proxy vendor configured.
This is the layer people usually mean when they say “the proxy pool has a concurrency limit.” It’s rarely the layer that actually bites you first.
Layer two: the target site’s rate and connection limits
Separate from the proxy’s own ceiling is whatever the destination site enforces. Most production sites sit behind a WAF or reverse proxy that tracks connection rate and concurrent connections per source IP. Cross that threshold and you’ll see slower responses, connection resets, or explicit 429/503 responses. This is true whether the source IP is a residential address, a datacenter block, or a mobile carrier NAT shared by thousands of real phones.
The important part: this limit is per IP as the target site sees it, not per proxy port as your provider sees it. If you’re running ten concurrent requests through one proxy IP, and the target’s WAF caps concurrent connections per IP at four, you’ll hit that ceiling first. Your provider’s port limit never even shows up. Widening your proxy pool doesn’t fix this unless you’re actually distributing load across more distinct IPs. Even then, each new IP inherits its own history and reputation the moment you start sending traffic through it.
We build against these systems from the infrastructure side daily. The honest description is defensive: rate limiting and connection throttling exist specifically so a single source can’t monopolize backend capacity or hammer an endpoint. There’s no version of “get around it” that’s a stable long-term strategy. The limit adjusts to whatever pattern is stressing it.
Layer three: your own orchestration and worker pool
The third bottleneck is entirely on your side. It’s the one most people overlook. Your scraper or pipeline has its own concurrency setting, usually expressed as a worker count, a thread pool size, or a connection pool size in whatever HTTP client library you’re using. If that pool is capped at 20 connections, it doesn’t matter how many proxy IPs you have behind it. You will never exceed 20 in-flight requests.
This matters because raising your proxy count without raising your local pool size does nothing. We’ve watched teams buy a bigger proxy plan to “increase concurrency” and see zero change in throughput. The actual limiter was a Semaphore(10) sitting in their code the whole time. Before touching the proxy side, check:
- The max connections setting on your HTTP client (
aiohttp,requestswith a session adapter,httpx, whatever you’re on) - The worker or thread count in your job runner
- Any queue depth limit in your orchestration layer that silently drops or delays tasks past a threshold
- OS-level file descriptor limits (
ulimit -non Linux), which cap how many sockets a single process can hold open regardless of what your code asks for
This layer is also where DNS resolution and TLS handshake cost hide. Opening a fresh connection for every request instead of reusing keep-alive connections adds real latency per request. That lowers effective throughput even if your concurrency number looks fine on paper.
Rotating vs sticky sessions changes the math
Rotating proxies assign a new IP per request or per short interval. That means concurrent requests are spread across many identities automatically. This raises your effective concurrency ceiling against target-side per-IP limits, because no single IP is absorbing all the parallel load.
Sticky sessions pin a worker to one IP for a set duration, usually because the target site requires session continuity (login state, cart contents, pagination that’s IP-sensitive). Sticky sessions cap your concurrency per session to whatever that one IP can sustain. Need N concurrent sticky sessions? You need something close to N distinct IPs, not N connections through a shared pool. This is a real cost trade-off, not just a technical one. Sticky IP allocation is usually billed differently than rotating.
Reasoning about your actual ceiling
There’s no single number that’s “the” concurrency limit. It’s the minimum of all three layers above. A practical way to find your real ceiling without guessing:
- Fix your local worker pool at a known value and hold it steady.
- Ramp proxy concurrency up gradually while watching latency, not just success rate. Latency creeping up before errors appear is usually the first sign you’re leaning on a limit somewhere.
- Watch for 429s, 503s, and connection resets specifically. Log which layer they came from (proxy gateway error vs. target site response) instead of lumping all failures into one “failed” bucket.
- If failures cluster on specific IPs, it’s target-side throttling. If failures happen regardless of which IP you’re on, it’s more likely your provider’s port or account-level limit.
Monitoring dashboards that just report “requests failed,” without separating error source, are the biggest reason teams spend money on more proxies. The problem was in their own worker pool the whole time.
The honest limit is a moving target
None of these limits are fixed constants. You can’t just look them up once and rely on them forever. Target sites adjust their thresholds based on traffic patterns, provider infrastructure gets rebalanced, and carrier-level throttling on mobile IPs shifts with network load. Treat any specific concurrency number as a snapshot, not a guarantee. Build monitoring that tells you when you’ve crossed a threshold, rather than assuming a number you tested last month still holds today.
If you’re evaluating a proxy pool for a production pipeline, the concurrency ceiling is one of the few things worth testing directly instead of taking on faith. Run your own ramp test against your own target. Do that before committing to a plan sized around it.
For more breakdowns like this on proxy infrastructure, scraping pipelines, and how detection systems actually work, check out the Data Research Tools home page.
Get new guides and videos first — join the Telegram channel.
Leave a Reply