Queue backpressure and worker pools: keeping a scraper pipeline from falling over

What backpressure actually means

Backpressure is what happens when a system tells its upstream producer “slow down, I can’t keep up.” In a scraper pipeline, the producer is usually something enqueueing URLs to fetch, and the consumers are worker processes pulling from that queue to make requests, parse responses, and write results somewhere. If the producer keeps adding work faster than the workers can drain it, something downstream has to absorb the difference. Without backpressure, that something is usually your RAM, and RAM always loses.

This isn’t an abstract concern. Any pipeline that separates “decide what to scrape” from “go scrape it” has this problem built in, because those two steps rarely run at the same speed. A URL discovery step (crawling sitemaps, paginating a search API, expanding a seed list) can produce tens of thousands of URLs in seconds. Actually fetching each one involves a network round trip, a proxy handoff, and often a deliberate delay to avoid tripping rate limits. The mismatch is structural, not a bug you can code away.

Why scraping pipelines hit this harder than most systems

A typical web service has a fairly predictable consumer: a database or an internal API with known latency. A scraping pipeline’s consumer is the open internet, through a proxy pool, hitting targets that actively rate-limit, throttle, or block you. That means your effective consumption rate isn’t constant. It can drop suddenly when a target site tightens its rate limiting, when a proxy pool’s healthy IP count shrinks, or when a target starts returning more challenge pages that need to be retried.

If your producer doesn’t know any of that is happening, it keeps enqueueing at the same rate it always has. The queue grows. If that queue lives in process memory (a Python list, an in-memory asyncio.Queue with no maxsize, a Go channel with no buffer limit), you get an out-of-memory kill with no warning beyond a slow memory graph nobody was watching. If the queue lives in Redis or RabbitMQ, you don’t crash, but you do build a backlog that takes hours to work through even after the underlying slowdown resolves, because now you’re paying for both the current work and the debt.

Bounded queues are the first fix, not an optimization

The simplest form of backpressure is a bounded queue: give it a max size, and make the producer block (or explicitly handle rejection) when it’s full. A Python asyncio.Queue(maxsize=5000) will suspend the producer coroutine on await queue.put(item) once it hits capacity, and resume it only as workers pull items off. A buffered Go channel does the same thing with make(chan Job, 5000). Celery, RQ, and most Redis-backed task queues expose similar concepts through queue length checks and worker prefetch limits.

The reason this matters more than it sounds is that an unbounded queue isn’t neutral, it’s a decision to let memory become your buffer of last resort. Bounding the queue forces the slowdown to happen where you can see it and reason about it, instead of silently in the OS memory allocator. When the queue is full and the producer is blocked, that’s your pipeline telling you, correctly, that fetching is currently the bottleneck.

Worker pool sizing has a ceiling set by the proxy layer, not your CPU

It’s tempting to size a worker pool by what your machine can handle: number of cores, memory available, that kind of thing. For scraping, the real ceiling is usually somewhere else entirely: how many concurrent connections your proxy pool can sustain per target domain without pushing your error rate up, and how much concurrent load a target’s rate limiting will tolerate before it starts serving 429s or CAPTCHAs instead of real pages.

Running 200 workers against a target that only tolerates 20 concurrent connections per IP, spread across a proxy pool with a limited number of healthy exits, doesn’t get you more throughput. It gets you more of your requests coming back as errors, more retries, and a queue that looks busy but isn’t actually making progress. This is where a lot of “why is my scraper slow” debugging ends up: the bottleneck isn’t the worker count, it’s that the workers are being throttled and the pipeline doesn’t have a mechanism to notice and adjust.

A worker pool that’s coupled to real backpressure signals (queue depth, per-domain error rate, proxy pool health) can scale itself down when the target or the proxy layer is struggling, and back up when things recover. A fixed worker count can’t do either.

Signals worth listening to

A few concrete signals tend to be useful for deciding when to slow a scraper pipeline down:

• Queue depth relative to a high watermark. If the queue is consistently near its cap, you’re producing faster than you consume, and either need more consumer capacity or a slower producer.
• Per-domain or per-proxy error rate. A rising rate of timeouts, connection resets, or non-200 responses from a specific target or through a specific proxy is usually a much earlier signal than a full queue.
• Retry queue growth. If items are landing back in a retry queue faster than they’re being drained, that’s the pipeline quietly losing ground even while the main queue looks fine.
• Proxy pool health. A shrinking count of proxies passing health checks means your effective concurrency ceiling just dropped, whether or not the worker pool config changed.

None of these signals are useful in isolation, but a producer that checks queue depth and per-domain error rate before deciding whether to enqueue more work is doing real backpressure, not just hoping the queue never fills.

Retries, backoff, and dead letters need to cooperate with backpressure

Retry logic and backpressure interact in a way that’s easy to get wrong. If a request fails and gets immediately re-enqueued, and that happens across thousands of items during a target-side slowdown, the retry logic itself becomes a second producer piling more load onto an already-struggling consumer. Exponential backoff on retries (waiting longer between each successive retry of the same item) helps, but it needs to be paired with a hard cap on retry attempts and a dead-letter queue for items that keep failing, so they stop cycling back through the main pipeline indefinitely.

Without a dead-letter path, a batch of URLs that started returning consistent errors (a changed page structure, a blocked proxy range, a target that started geo-blocking a region) will keep consuming worker time on every retry cycle, which is capacity taken away from work that could actually succeed.

A pipeline shape that tends to hold up

In practice, a scraper pipeline that survives real-world target and proxy variability usually has these pieces, in this relationship:

1. A producer that checks queue depth (and ideally per-domain error rate) before enqueueing new URLs, rather than enqueueing on a fixed schedule.
2. A bounded queue, sized to hold a few minutes of work, not the entire day’s crawl.
3. A worker pool whose concurrency is tuned to the proxy pool’s sustainable per-domain rate, not to available CPU.
4. A separate retry queue with capped, backed-off retries and a dead-letter destination for items that exhaust their attempts.
5. Monitoring on queue depth, error rate, and proxy pool health, because any one of them can be the actual constraint at a given moment.

None of this is exotic. It’s the same producer-consumer discipline that shows up in any queue-backed system. What’s specific to scraping is that the consumer’s real capacity moves around constantly, driven by targets and proxy infrastructure you don’t control, which is exactly why the backpressure has to be measured and reactive rather than assumed.

If you’re building out pipeline infrastructure and want more breakdowns like this on queues, proxies, and the operational side of scraping, check out the rest of what we cover at Data Research Tools.

Get new guides and videos first — join the Telegram channel.