Building a Scraping Pipeline That Survives Failure (Dagster and Prefect)

A scrape that runs once is a script. A scrape that has to run every day, for months, without a human babysitting it, is a pipeline, and those are completely different animals. The script only has to work when the network is clean and the site behaves. The pipeline has to survive the network being flaky, the site changing overnight, and a run dying halfway through with half the data already written.

I run production scrapers for a living, so I have been paged at three in the morning enough times to build for failure on purpose. The goal is not a clever scraper, it is a boring one that recovers from problems on its own and only wakes you up when something is genuinely wrong. That reliability is not in the parsing code, it is in the structure around it, and that structure is what an orchestrator like Dagster or Prefect gives you.

What an orchestrator actually buys you

From the outside an orchestrator looks like overhead. It is not. It turns your scrape into a set of connected steps it runs on a schedule, and it remembers what happened: which steps ran, which failed, which produced data. It can retry the ones that broke without rerunning the ones that worked. That memory is the whole point. A plain cron job runs your script and forgets everything, so a failure means starting over. An orchestrator lets you recover from exactly where it broke.

Split fetch from parse from store

The first structural move is to stop writing one big function that does everything. Split the work into stages. One stage fetches raw pages and saves them untouched. A second parses those saved pages into clean records. A third writes those records to your database.

Keeping them separate means that when the site changes its layout and your parser breaks, you have not lost the fetched pages. You fix the parser and rerun only that stage against data you already have, instead of hammering the site all over again.

Idempotency is the whole game

Here is the idea everything else rests on. Every stage should be safe to run twice. Running it again with the same input should produce the same result and never create duplicates or corrupt what is there. That property is called idempotency, and it is what lets an orchestrator retry freely.

If rerunning a stage might double your records or half write a file, then retries are dangerous and you cannot automate recovery. So you design each step to be repeatable from the start, usually by keying records on a stable id and writing in a way that overwrites cleanly rather than blindly appending.

Checkpoint so you can resume

Long jobs need to remember their progress. If you are pulling a hundred thousand pages and the run dies at page sixty thousand, you do not want to start from zero. So you checkpoint: write down what you have finished as you go, and on the next run, read that and skip what is already done. An orchestrator tracks which chunks of work succeeded, so a restart naturally picks up the unfinished ones. That is the difference between a five minute recovery and a five hour one.

Retries with backoff, done right

Failures at the network layer are constant, not rare, so retries are not optional. But a naive retry that fires again instantly just hammers a struggling site and makes things worse. The right pattern is backoff: wait a moment before the first retry, then wait longer before the next, giving the site room to recover. Both Dagster and Prefect let you set retry counts and delays per step, so a single flaky request quietly succeeds on its second or third try, and only a request that fails every attempt gets escalated.

The dead letter queue

Not everything recovers, and you need a place for the failures to go. That place is a dead letter queue, which is just a list of the items that failed all their retries. Instead of crashing the whole run because ten pages out of a hundred thousand refused to load, you set those ten aside, finish everything else, and deal with them later. This keeps one stubborn page from taking down a whole night’s work, and it gives you a clean record of exactly what needs a second look in the morning.

Incremental beats full every time

Do not rescrape the world every day. Most sites change only a little between runs, so pulling everything from scratch wastes your resources and pounds the site for no reason. Run incrementally: track what you have already collected and only fetch what is new or changed since last time. It is lighter on your infrastructure, far lighter on the site, and it is the difference between a pipeline that scales and one that gets slower and more fragile as your dataset grows.

Backfills without fear

Sometimes you do need to go back and reprocess history, maybe because you fixed a parser or added a field. That is a backfill, and a good pipeline makes it painless. Because your stages are idempotent and your raw pages are saved, you can rerun the parse stage across a date range and rebuild the clean data without touching the site at all. Dagster in particular is built around reprocessing ranges of data. Designing for backfills from the start means a schema change is an afternoon, not a crisis.

Observability, or you are flying blind

You cannot fix what you cannot see. A pipeline that runs silently is one you do not actually trust, because the first sign of trouble is stale data days later. So you instrument it. Log how many pages you fetched, how many parsed cleanly, how many landed in the dead letter queue, and how long each stage took. Both orchestrators give you a dashboard of runs and their history. When the parsed count suddenly drops to zero, you want to see it that hour, not discover it when someone downstream asks where the data went.

Alert on the right thing

Do not alert on everything, or you will train yourself to ignore the alarms. A handful of failed pages is normal background noise and should never wake you. What should wake you is a signal the pipeline is actually broken: a whole run failing, the success rate falling off a cliff, or zero records produced when you expected thousands. An alert that fires constantly is worse than no alert, because you stop reading it.

Dagster or Prefect, briefly

Both are solid Python orchestrators and either will serve you well. Prefect leans lightweight and flexible, wrapping your existing functions as tasks and flows with very little ceremony, which makes it fast to adopt. Dagster leans toward treating your data as first class assets, with strong support for backfills, typing, and knowing exactly what each step produces. For a scrape heavy on reprocessing history I tend toward Dagster; for a simpler flexible flow I reach for Prefect. Neither choice is wrong, so pick one and build.

The shape of a Prefect flow is small. You mark functions as tasks, give them retries, and wire them into a flow:

from prefect import flow, task

@task(retries=3, retry_delay_seconds=[5, 30, 120])
def fetch(url: str) -> str:
    ...

@task
def parse(raw: str) -> list[dict]:
    ...

@task
def store(records: list[dict]) -> None:
    ...

@flow
def scrape(urls: list[str]) -> None:
    for url in urls:
        store(parse(fetch(url)))

That is a real pipeline skeleton, retries and all, in a handful of lines.

Validate, or you fill the database with nulls

The failure that hurts most is the one that does not throw an error. A site quietly renames a field, or starts returning an empty value where there used to be a price, and your parser keeps running and writes rows full of nulls. Nothing crashes, so nothing alerts, and you find out days later that your data is hollow. The defense is validation: check that the records coming out of the parse stage look right, that key fields are present and in a sane range, and fail loudly when they are not.

Test on a small slice first

Never point a fresh pipeline at the full job on its first run. Run it against a tiny slice, a handful of pages, and watch every stage work end to end before you scale up. It is far cheaper to find a broken selector or a bad write on ten records than on ten million. Keep a small sample run you can fire on demand, so after any change you confirm the whole pipeline still flows cleanly before you let it loose on the real volume.

Keep the site happy at the pipeline level

Politeness is not just a per request setting, it is a pipeline property. The orchestrator controls how many workers run at once and how fast the whole thing moves, so it is where you enforce a rate the site can absorb. Spread the load across time instead of firing a huge burst, cache aggressively so you never fetch the same page twice, and prefer an official api or bulk feed when the site offers one. A pipeline that runs gently is one that keeps running, because it never gives the site a reason to shut the door.

The honest limits

Good orchestration makes a scrape reliable and recoverable. It does not make it welcome where it is not, and it does not change what you are allowed to collect. Public data, a robots file respected, an official feed preferred, a gentle pace held. Structure buys you uptime and sane recovery, not permission. No pipeline survives a site that genuinely does not want to be scraped, nor should it try to, because durable data work lives on the compliant side of that line.

To recap: split fetch from parse from store, make every stage idempotent so retries are safe, checkpoint so you can resume, and retry with backoff instead of hammering. Set failures aside in a dead letter queue, run incrementally instead of rescraping the world, design for painless backfills, and watch the pipeline closely enough to alert only on real breakage. Build it that way and it keeps running while you sleep.

If you want more breakdowns like this, dataresearchtools.com has the rest of the scraping infrastructure guides, with real code and no undetectable promises.

Get new guides and videos first — join the Telegram channel.