Scheduling Scrapers: Cron vs Airflow vs Dagster

A scrape that runs once is a script. A scrape that has to run every day for a year, without anyone watching, is infrastructure, and the layer that turns one into the other is scheduling. It decides when your scraper runs, what happens when it fails, and whether you find out the day it breaks or three weeks later from stale data.

I run production scrapers, so I have used every rung of this ladder, from a single cron line on one box to a full orchestrator managing dozens of jobs. Here is how I think about the three real options, cron, Airflow, and Dagster, and how to pick one without overbuilding.

What scheduling actually has to do

It sounds like one job, run this at 6am, but the real work is underneath. A good scheduler remembers whether the last run finished. It retries a failed step without redoing the ones that worked. It stops two runs from colliding when one runs long. It tells you when a job broke, and, harder, when a job succeeded but returned nothing. And it lets you go back and reprocess history when you fix a bug.

Cron does the first sentence. Everything after it is why the heavier tools exist.

Start with cron

Cron is the scheduler already living on every unix box you own. You give it a line, a time and a command, and it runs that command forever. No service to install, no database, no dependencies. For a single scrape that pulls one source once a day into one table, cron is not the lazy choice, it is the correct one.

The trouble is everything cron does not do. It has no memory, so a failed run vanishes and the next starts fresh. It has no retries, so one network blip loses you a day. It will start a new run while the last is still going, and now two copies fight over the same table. And its output goes nowhere you will look, so a broken job surfaces days later as stale data rather than an alert.

Make cron last a little longer

If you only have a job or two, you do not have to abandon cron, you shore it up:

• Wrap the command in a script that takes a lock so runs cannot overlap.
• Make that script exit non-zero and notify you when the work fails.
• Send output to a real log file, not the void.
• Add a heartbeat, an external service you ping only on a clean finish, so a run that never happens alerts you by its silence.

That last one matters most. The scariest failure is not the job that crashes loudly, it is the job that silently never fires.

When you have outgrown cron

Move up a rung when any of these is true: you have steps that depend on each other and want to retry just the one that broke, you need to reprocess last month because you fixed a parser, you are running more than a couple of jobs and cannot remember what runs when, or a second person needs to see what is happening without reading your crontab.

The tools above cron are built around a dag, a directed acyclic graph. That just means steps with an order, where each points to the ones that must finish first: fetch before parse, parse before store. Once the scheduler understands that shape, it can run steps in order, parallelize the independent ones, retry a single failed step, and skip what already succeeded.

Airflow: dags and backfills

Airflow is what most teams reach for first. You write your dag in python, Airflow runs it on a schedule, and you get a ui where every run is a row of green and red boxes. Each task has its own retries and delays, and when one fails you click in, read the log, and rerun just that piece. It has been doing this for years, so almost every database, api, and cloud service has a ready made connector.

Its real superpower is the backfill. Airflow treats each run as owning a slice of time, so when you fix a parser and need to rebuild the last 30 days, you tell it to run that date range and it reprocesses each day in order. That only works if your steps are idempotent, but when they are, repairing history is one command instead of a manual slog.

None of that is free. Airflow is a scheduler, a metadata database, workers, and a web server, several moving parts to install, secure, upgrade, and babysit. Standing all of that up to run one daily scrape is like buying a forklift to carry your groceries. It earns its keep when you have many jobs and several people depending on them.

Dagster and Prefect: data assets and observability

Dagster and Prefect came later with a different framing. Instead of thinking only about tasks that run, Dagster thinks about the data assets those tasks produce, the actual table or dataset you care about. You declare the asset and the code that builds it, and Dagster tracks what it depends on and how fresh it is.

That framing fits scraping well. You do not really care that a task ran, you care whether your products dataset is current and correct. Dagster lets you ask exactly that, shows the lineage behind each asset, and leans on typing so a run that returns the wrong shape gets caught instead of silently writing nulls. Prefect takes a lighter path, wrapping your existing python functions as flows with little ceremony, which suits irregular, dynamic pipelines. Both give you nicer local development and clearer visibility than the older model.

Retries, idempotency, and the silent empty run

Three things matter no matter which tool you pick.

Put retries in the scheduler, not just the scrape. Your fetch function can back off on a single flaky request, but the scheduler retrying a whole failed task with a growing delay is what saves a run when a site has a bad five minutes.

Make every job idempotent. Running the same job twice must do no harm, or retries and backfills become a gamble. Key records on a stable id and write with an upsert that overwrites cleanly instead of appending.

And catch the silent empty run. The worst failure is not a crash, it is a job that finishes green with zero rows because the site changed a selector. Cron and even a naive Airflow setup call that a success. So count what you collected and compare it to normal, and if you usually pull thousands and today you got 12, fail the run on purpose. Alert on a whole run failing, a success rate falling off a cliff, zero rows where you expected thousands, and a run that never started at all.

How to choose

Match the tool to the scale and the team, and do not overbuild before you need to.

• Solo, one or two scrapes, no complex dependencies: cron with a lock and a heartbeat. Done.
• A handful of interdependent pipelines, backfills, or a small team that needs shared run history: Airflow, because it is proven and everyone can learn it.
• A data heavy setup where freshness, lineage, and typed outputs matter more than raw scheduling: Dagster, or Prefect when the work is irregular and pythonic.

The most common mistake is not too little tooling, it is standing up a whole orchestrator to run a scrape a single cron line handled fine. You can always climb the ladder when the pain is real.

The honest limits

Scheduling makes your collection reliable, not permitted. Running a scraper on time and recovering it cleanly does not change what you are allowed to take. Public data, a robots file respected, a gentle pace, and an official api or bulk feed preferred when one exists, those rules do not move because your orchestrator is nicer.

I run all three of these in production, cron for the small stuff and a real orchestrator for the pipelines that feed everything else. If you want the worked versions, the Airflow and Dagster setups, the idempotency patterns, and the alerting checks that catch an empty run, along with the tools I actually use, they are all at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.