Choosing a queue for a scraper that runs all night

Why the queue matters more at 3am than at 3pm

A scraper you babysit for twenty minutes can get away with almost any queue, including no queue at all. A scraper running from midnight to 8am can’t. Nobody is watching it. If a worker crashes, if a proxy pool goes stale, if one target site starts throwing 403s at 2am, the queue is the only thing deciding whether the run recovers on its own or just stops producing data until someone notices in the morning.

That’s the real design question behind “scraper job queue”: not which library has the nicest API, but what happens to a job when something goes wrong while no human is looking.

Why an in-memory list doesn’t survive the night

A lot of scrapers start life as a Python list of URLs and a for loop. That’s fine for a few hundred pages run interactively. It falls apart on an overnight job for a specific reason: the list lives inside one process’s memory. If that process dies, whether from an unhandled exception, an out-of-memory kill, or the host rebooting for a routine patch, every job that hasn’t finished is gone. There’s no record it ever existed.

You can wrap the loop in more try/except blocks, but that doesn’t fix the underlying problem. The job list needs to live somewhere that survives the worker process crashing. That’s the first real requirement for an overnight queue: persistence outside the worker.

Persistence: where the job list actually lives

Once persistence is the requirement, the practical options split into a few tiers.

Redis is the common middle ground. Lists, sorted sets, or Redis Streams can all act as a queue backend, and tools like RQ, Celery (with Redis as broker), or BullMQ build retry logic, scheduling, and worker pools on top of it. Redis is fast and simple to run, but by default it’s in-memory with periodic snapshotting, so you need to actually configure AOF persistence or accept that a Redis restart at the wrong moment can lose recent jobs.

A message broker like RabbitMQ gives you durable queues, acknowledgments, and dead letter exchanges out of the box, at the cost of more moving parts to operate. It’s a reasonable choice once you have multiple worker types (fetchers, parsers, retriers) that need distinct queues and routing rules.

A managed queue like SQS removes the operational burden entirely. You don’t run anything, you get durability and visibility timeouts by default, and it scales without you thinking about it. The tradeoff is cost at high message volume and a bit less flexibility in retry logic than you get building it yourself.

A plain Postgres table is underrated for smaller overnight jobs. A jobs table with a status column, SELECT ... FOR UPDATE SKIP LOCKED to let multiple workers pull rows without colliding, and a cron-style sweep for stuck rows gets you most of what a dedicated queue gives you, without adding a new piece of infrastructure to your stack. If you already run Postgres for the scraped data, this is often the path of least resistance.

None of these is universally correct. The right one depends on how many workers you’re running, how much operational overhead you’re willing to own, and whether you already have the infrastructure in place.

Retries and the lease pattern

The reason a queue beats a list isn’t just “it survives a crash.” It’s that a real queue has an answer to: what happens to a job that a worker picked up but never finished?

Most durable queues use some version of a lease, also called a visibility timeout. When a worker pulls a job, the queue doesn’t delete it immediately, it hides it for a set window, say five minutes. If the worker finishes and acknowledges the job, it’s removed for good. If the worker crashes, times out, or never checks back in, the lease expires and the job becomes visible again for another worker to pick up.

This matters specifically for scraping because scrape jobs fail in ways that are often transient: a proxy that’s temporarily rate limited, a target site that’s slow, a connection reset. A lease-based retry means the job gets tried again automatically without you writing manual retry code, as long as you set the lease window sensibly. Too short, and jobs get double-processed while a slow-but-working request is still in flight. Too long, and a genuinely dead worker leaves the job stuck for a while before it comes back.

Not every retry should look the same

A flat “retry three times” policy treats a temporarily overloaded proxy the same as a page that returns a hard 404 or a site that’s actively blocking the request pattern. Those aren’t the same failure and shouldn’t get the same response.

A queue that supports backoff (waiting longer between each retry) handles transient network issues reasonably well. But a job that keeps failing because a target has started returning CAPTCHA pages or has changed its markup structure isn’t going to succeed on retry four. That’s what a dead letter queue is for: after N failed attempts, move the job to a separate holding queue instead of retrying forever, so it doesn’t burn worker time all night and instead surfaces in the morning as a bucket of “these need a human.” Building this in from the start saves you from a queue full of jobs quietly failing on loop from 1am to 7am.

Rate limiting belongs in the queue’s job model, not just the code

A queue that’s a plain FIFO with no concept of “how many of these can run against this target right now” will happily let fifty workers hit the same domain simultaneously the moment there’s a backlog. Overnight, with nobody watching the response codes, that’s how a scraper goes from “running” to “entirely blocked” between midnight and sunrise.

This is where the queue and the proxy pool have to work together operationally, not as separate concerns. Concurrency limits per domain, and ideally per proxy or per IP, need to be enforced at the point where jobs get dispatched to workers, not hoped for in application code that a tired engineer wrote at 11pm before the run started. Some queue libraries support per-queue or per-tag concurrency caps directly (Celery has rate limiting per task, RQ can be run with separate worker pools per queue). If your queue doesn’t support this natively, routing jobs for different targets into separate named queues, each with its own worker concurrency, is a low-tech way to get the same effect.

Ordering: FIFO isn’t always what you want

Plenty of scrapers default to FIFO because it’s the obvious behavior, and for orderly crawls of a fixed URL list, that’s fine. But if part of your pipeline is discovering new URLs while it runs, for example a crawler that queues newly found product pages as it goes, a strict FIFO queue means fresh discoveries sit behind whatever was already queued, sometimes for hours. Depending on how time sensitive the data is, a LIFO or priority queue for freshly discovered URLs can matter more than raw throughput. This is a case where the answer depends entirely on what the scrape is actually for, not a default you should copy from someone else’s stack.

Watching a queue you can’t watch

The honest failure mode for an overnight scraper isn’t usually a crash, it’s a slow silent stall: queue depth climbing because workers can’t keep up, or a target site quietly returning empty pages that count as “success” but produce no data. A queue you can query for depth, in-flight count, and dead letter count gives you something to alert on. A basic threshold alert, queue depth above some number for more than N minutes, or dead letter count above zero, catches most of what would otherwise be an unpleasant morning discovery. This is a bigger part of “choosing a queue” than it looks like up front: pick something you can actually inspect while it’s running, not just something that moves jobs from A to B.

What this comes down to

There’s no single right queue for an overnight scraper. A solo operator running a few thousand pages a night is well served by Redis plus RQ or a Postgres table with SKIP LOCKED. A team running multiple pipelines against dozens of targets probably wants RabbitMQ or SQS with proper dead letter routing and per-domain concurrency limits. What doesn’t scale to an unattended run, regardless of size, is a plain in-memory list with no lease semantics, no retry policy, and no way to see what’s stuck. The queue is the part of the system that has to keep working when you’re asleep, so it’s worth choosing on those terms rather than on which one was fastest to wire up on a Tuesday afternoon.

If you’re setting up the rest of an overnight pipeline, from proxy rotation to fingerprint handling, we cover the infrastructure side in more depth on Data Research Tools.

Get new guides and videos first — join the Telegram channel.