Most people store scraped data in whatever was easiest on day one, usually a single Postgres table, and then spend the next year fighting it. The table grows past what it was meant to hold, the analytics queries crawl, and every field change on the source site turns into a migration. The problem is not Postgres. The problem is asking one store to do three different jobs.

Scraped data does not have one right home. It has three, and the skill is knowing which slice of your data belongs in which. I run production scrapers, and pulling the data is only half the work. The other half is storing it so it stays queryable, deduped, and cheap as the volume climbs into the tens of millions of rows.

The three places scraped data lives

Think in three tiers, each solving a different problem.

Object storage holding Parquet files is the cheap, bottomless bucket for raw high volume capture, the stuff you want to keep but rarely query directly. Postgres is the operational layer, the deduped, indexed, current state you serve queries against every day. A warehouse like BigQuery or Snowflake is where analytics at scale lives, the place you run heavy aggregate questions over the whole history.

Almost all the storage pain I see comes from asking one of these to do another’s job.

Land the raw capture first

The first move is to separate raw capture from clean state. When the scraper pulls a page, do not parse it and throw the original away. Write the raw response somewhere cheap and untouched first, then parse from that copy.

I land raw html or raw json responses straight into object storage, one file per fetch or batched, before a single field gets extracted. The parse step reads from there, not from the live site.

The reason this matters is failure. Sites change their layout without warning, and your parser will break. If you kept only the clean parsed rows, a parser bug means you have to rescrape the site to recover, hammering it again and hoping the old data is still there. If you kept the raw capture, you just fix the parser and reparse the files you already have, offline, for free. Raw storage is cheap insurance against your own future mistakes.

Why Parquet for the raw and semi clean layer

You can land raw capture as plain json or csv, and for small jobs that is fine. At volume it hurts.

Parquet is a columnar file format. Instead of storing row by row, it stores all the values of each column together, which compresses far better because similar data sits next to itself. It is typed, so a number stays a number, and it carries its own schema. For large volumes you scan a few columns at a time, it is much smaller on disk and faster to read than csv or json, and every warehouse and analytics tool reads it natively.

The tradeoff: json preserves exactly what the site sent, which is what you want for the truly raw capture. So I keep json for the untouched landing, then convert to Parquet once the shape is known.

Postgres is your operational truth

Postgres is where the clean, current, queryable state lives. Once you have parsed a record, this is the row you actually use, look up, join against, and serve to whatever consumes the data. You get real indexes, real constraints, transactions, and sql everyone already knows.

For operational work, looking up one entity, filtering a few thousand rows, enforcing that a record is unique, nothing beats a boring relational database. Using Postgres is not the mistake. The mistake is trying to keep every raw byte you ever scraped inside it.

Dedup keys and idempotent writes

The most important thing you do in the operational layer is dedup, and it starts with a key. Every record you scrape needs a stable natural id, something the source itself defines: a product code, a listing id, a canonical url. That key tells you whether a row you just parsed is new or something you already have. Put a unique constraint on it.

Then make your writes idempotent, so running the same insert twice changes nothing the second time. Postgres gives you this with an upsert:

insert into products (id, title, price, raw)
values ($1, $2, $3, $4)
on conflict (id) do update
set title = excluded.title,
    price = excluded.price,
    raw   = excluded.raw;

On conflict with the key, it updates the existing row instead of creating a duplicate. This is what lets a pipeline retry safely: a run dies halfway, you start it again, and the records that already landed overwrite themselves cleanly instead of doubling. A stable key plus idempotent writes make a scraper safe to rerun, which you will do constantly.

Handling schema drift

Sites change their fields, and a rigid table will fight you over it. A source adds a column, renames one, or returns a value in a new shape, and suddenly your clean schema does not fit. This is schema drift, and you plan for it rather than getting surprised.

I keep a set of stable core columns I am confident about, the id and the fields I always need, and a jsonb column that holds the rest of the record as is. The structured columns stay clean and indexed for the queries that matter, and the messy, changing parts live in jsonb where a new field does not break anything.

When you have outgrown Postgres

Postgres is excellent right up until it is not. When a single table climbs past tens of millions of rows and your queries turn into big aggregates scanning most of it, a row store starts to struggle.

Counting across a hundred million rows, grouping by month over all of history, joining several large tables for a report: these are analytics questions. A database tuned for operational lookups answers them slowly, and at the cost of the operational work it should be doing. That slowdown is the signal you have outgrown one tier and need the next.

The warehouse is for analytics, not serving

That next tier is a warehouse, something like BigQuery or Snowflake. They are built for exactly the queries that make Postgres sweat, scanning and aggregating enormous columnar datasets. They separate storage from compute, so your data sits cheap and you only pay for the horsepower when a query runs. They read Parquet natively, so loading the raw layer in is straightforward.

The mistake in the other direction is serving live traffic out of a warehouse. Warehouses are built for big scans, not fast single row lookups, and they usually charge by how much data each query scans. Point an app at one for constant small reads and you get slow responses and a surprising bill. Keep operational serving in Postgres and heavy analytics in the warehouse, and let each do what it is good at.

The layered pattern I run

Here is how the three fit together in practice:

• The scraper lands raw responses as files in object storage, cheap and permanent.
• A parse step reads those, extracts clean records, and upserts them into Postgres keyed on a stable id. That becomes the deduped operational truth.
• On a schedule, the clean data loads into the warehouse as Parquet for analytics over the full history.

Raw for recovery, Postgres for serving, warehouse for analysis. Each layer feeds the next, and none is asked to do a job it is bad at.

Choosing by volume and query pattern

If you remember one rule, make it this: choose your store by volume and query pattern, not by habit.

Small data you look up and update row by row belongs in Postgres. Huge volumes of raw capture you rarely query directly belong in flat Parquet on object storage. Heavy aggregate analytics over everything belong in a warehouse. Cost often makes the call: object storage is cents per gigabyte a month, Postgres is a server you pay for whether or not it is busy, and a warehouse charges by how much each query scans. Most real pipelines want all three, and the healthy ones put each slice of data where its size and access pattern fit.

The honest limits

Storage does not change what you were allowed to collect. Public data, a robots file respected, an official api or bulk feed preferred, personal and paywalled data left alone. A clean layered warehouse full of data you should not have collected is still data you should not have collected. Good architecture makes your collection durable and cheap to keep. It does not grant permission.

I run this exact layering in production, raw in object storage, deduped state in Postgres, analytics in a warehouse. If you want the full guides, with real upsert patterns, Parquet partition layouts, and the tools I actually use and test, they are here.

Get new guides and videos first — join the Telegram channel.

Most teams pick a scraping framework the way they pick a fight, by arguing which one is objectively better. Crawlee against Scrapy, a dozen feature tables open in browser tabs, everyone certain there is a winner hiding in the details. There usually isn’t. The honest answer is a little deflating: the framework you should use is almost always the one written in the language your team already speaks.

I run production scrapers for a living, the kind that pull public data at scale every day, and I’ve shipped real jobs on both of these. So this is a tested comparison, not a spec sheet read out loud. Both are good. Neither one makes you undetectable, and I’m not going to pretend the framework is the thing that decides whether you get through the door.

The short verdict

Dimension	Scrapy (Python)	Crawlee (Node / TypeScript)
Age and maturity	Over a decade in production, huge ecosystem	Younger, smaller, growing fast
Browser handling	Bolt on via a Playwright plugin	Native Playwright and Puppeteer, first class
Concurrency	Twisted event driven engine	Node event loop plus an autoscaling pool
Typing	Python, optional type hints	TypeScript, real type checking
Best fit	Python teams, http heavy targets	Node teams, browser heavy targets

If you only remember one line: match the framework to your team’s language, then let browser needs break the tie.

What each framework actually is

Scrapy is a Python crawling framework, asynchronous at its core, and it has been the default answer for scraping in Python since long before most of today’s targets existed. It gives you spiders, a request scheduler, middlewares, and item pipelines, which is a structured place to validate and store what you pull.

Crawlee is newer, written in Node and TypeScript by the team behind Apify. Its whole idea is to unify plain http crawling and full browser crawling behind one interface, with a request queue and storage layer built in. Both frameworks make the same core promise: fetch pages, follow links, parse fields, and store the result, with the plumbing already done so you’re not wiring retries and queues by hand.

The factor that really decides it

Here is the thing I wish someone had told me earlier. The biggest deciding factor isn’t the feature list, it’s the language your team already lives in.

If your engineers write Python all day, Scrapy drops into a stack they already understand, and the data tooling around it feels native. If your product is a Node or TypeScript codebase, Crawlee lets your scrapers share types, libraries, and habits with the rest of your code. Fighting your team’s native language to chase a slightly nicer feature is a tax you pay every single day, and it almost never earns back what it costs.

Where Scrapy wins

Scrapy’s biggest asset is age. It has been in production for well over a decade, which means almost every problem you’ll hit has already been hit by someone else and written up somewhere. The ecosystem is deep, with middlewares and extensions for nearly anything, and item pipelines give you a clean path to deduplicate and store records as they arrive.

That maturity isn’t glamorous, but at three in the morning when a job breaks, a well worn path with a thousand answers already posted is worth more than any shiny feature. For http heavy work on mostly static or api reachable targets, Scrapy is hard to beat and it scales to millions of pages without a fight.

Where Crawlee wins

Crawlee was designed for the web as it is now. The modern web leans hard on JavaScript, and Crawlee treats browser crawling as a first class citizen rather than an add on. It wraps Playwright and Puppeteer directly, so moving a crawler from cheap http fetching to a full browser is close to a configuration change instead of a rewrite.

Scrapy can drive a browser too, through the scrapy playwright plugin, and it works well. But you can feel the seam, because Scrapy was built around raw http requests first. If browser rendering is the occasional exception, Scrapy plus the plugin is clean. If it’s the main event across most of your targets, Crawlee handling it natively will feel smoother and cost you far less glue code to keep alive.

The blocking helpers, honestly

Both frameworks ship helpers meant to reduce how often you get blocked, and I want to be careful here. Crawlee includes tools that rotate through realistic browser fingerprints and manage sessions so requests look less uniform. Scrapy has its own rotation middlewares and a large library of community plugins that do similar work.

These helpers genuinely reduce clumsy mistakes, the obvious tells that get a lazy scraper flagged in seconds. But none of it makes you undetectable, and none of it is an evasion toolkit. They lower the noise floor a little. They don’t turn the wall off.

Concurrency and autoscaling

Scrapy runs on Twisted, an event driven networking engine, and its whole design fires many http requests at once without blocking, so a single process can push a high volume of raw fetches efficiently. Crawlee rides Node’s event loop, which is also asynchronous by nature, and layers an autoscaling pool on top that grows and shrinks concurrent tasks based on how much cpu and memory are actually free.

That autoscaling is one of Crawlee’s nicer touches. Instead of guessing a fixed concurrency number and hoping, it watches system load and adjusts on the fly. Scrapy’s autothrottle extension does something similar by adapting the delay to response times, though it leans more toward politeness to the target than toward squeezing your own hardware. Both spare you from hand tuning a magic number.

Learning curve and typing

The learning curves split along familiar lines. Scrapy has a steeper first climb because it has its own architecture, the spiders, the settings, the pipelines, and you learn that before you’re productive. Crawlee feels more approachable to a JavaScript developer because it reads like ordinary async Node code. But that only helps if your team is already fluent in Node. Drop a Python team into Crawlee and the advantage evaporates.

The one clear technical edge here is TypeScript. Because Crawlee is written in it, you get real type checking across your scraper, which catches a whole class of silly mistakes before you run the job. On a big, long running codebase that safety adds up. Scrapy gives you Python’s flexibility and optional type hints, useful but not enforced the same way.

Proxies matter more than the framework

On hard targets, the proxy layer decides more than the framework does, and both frameworks give you a clean place to plug one in. Each lets you route requests through a rotating pool and swap the address per request. Neither ships the addresses themselves. That part is on you.

This is the layer I run myself, real mobile proxies on real carrier sim cards, because for strict targets the address you arrive on carries more trust than any framework feature. Cheap datacenter ip ranges will fail on a tough site in Crawlee exactly like they fail in Scrapy. The framework routes the request. It does not launder where it came from.

How I actually choose

I start with the team, because it dominates everything else. A Python team scraping mostly static or http reachable data, I point at Scrapy without a second thought. A Node or TypeScript team, or a project where most targets are JavaScript heavy and need a real browser, I lean Crawlee for the native browser handling and the typing.

Only when a team is genuinely comfortable in both languages do the finer features get a real vote, and even then I weigh maintenance and hiring over any single capability. The best framework is the one your team can keep alive for a year.

The honest limits

Neither framework moves the boundary that matters. Picking Crawlee or Scrapy does not make anything undetectable, and it does not change what you’re allowed to collect. The same rules apply either way: public data, a robots file respected, an official api or bulk feed preferred where one exists, and a polite rate held so the site never gets a reason to shut the door.

The framework decides how comfortably you build and maintain the scraper. It doesn’t decide whether what you’re doing is welcome. If you want the full written guides, the proxy configs, and the picks I actually run in production, they’re at dataresearchtools.com, with working code and no undetectable promises.

Get new guides and videos first — join the Telegram channel.

A scrape that runs once is a script. A scrape that has to run every day, for months, without a human babysitting it, is a pipeline, and those are completely different animals. The script only has to work when the network is clean and the site behaves. The pipeline has to survive the network being flaky, the site changing overnight, and a run dying halfway through with half the data already written.

I run production scrapers for a living, so I have been paged at three in the morning enough times to build for failure on purpose. The goal is not a clever scraper, it is a boring one that recovers from problems on its own and only wakes you up when something is genuinely wrong. That reliability is not in the parsing code, it is in the structure around it, and that structure is what an orchestrator like Dagster or Prefect gives you.

What an orchestrator actually buys you

From the outside an orchestrator looks like overhead. It is not. It turns your scrape into a set of connected steps it runs on a schedule, and it remembers what happened: which steps ran, which failed, which produced data. It can retry the ones that broke without rerunning the ones that worked. That memory is the whole point. A plain cron job runs your script and forgets everything, so a failure means starting over. An orchestrator lets you recover from exactly where it broke.

Split fetch from parse from store

The first structural move is to stop writing one big function that does everything. Split the work into stages. One stage fetches raw pages and saves them untouched. A second parses those saved pages into clean records. A third writes those records to your database.

Keeping them separate means that when the site changes its layout and your parser breaks, you have not lost the fetched pages. You fix the parser and rerun only that stage against data you already have, instead of hammering the site all over again.

Idempotency is the whole game

Here is the idea everything else rests on. Every stage should be safe to run twice. Running it again with the same input should produce the same result and never create duplicates or corrupt what is there. That property is called idempotency, and it is what lets an orchestrator retry freely.

If rerunning a stage might double your records or half write a file, then retries are dangerous and you cannot automate recovery. So you design each step to be repeatable from the start, usually by keying records on a stable id and writing in a way that overwrites cleanly rather than blindly appending.

Checkpoint so you can resume

Long jobs need to remember their progress. If you are pulling a hundred thousand pages and the run dies at page sixty thousand, you do not want to start from zero. So you checkpoint: write down what you have finished as you go, and on the next run, read that and skip what is already done. An orchestrator tracks which chunks of work succeeded, so a restart naturally picks up the unfinished ones. That is the difference between a five minute recovery and a five hour one.

Retries with backoff, done right

Failures at the network layer are constant, not rare, so retries are not optional. But a naive retry that fires again instantly just hammers a struggling site and makes things worse. The right pattern is backoff: wait a moment before the first retry, then wait longer before the next, giving the site room to recover. Both Dagster and Prefect let you set retry counts and delays per step, so a single flaky request quietly succeeds on its second or third try, and only a request that fails every attempt gets escalated.

The dead letter queue

Not everything recovers, and you need a place for the failures to go. That place is a dead letter queue, which is just a list of the items that failed all their retries. Instead of crashing the whole run because ten pages out of a hundred thousand refused to load, you set those ten aside, finish everything else, and deal with them later. This keeps one stubborn page from taking down a whole night’s work, and it gives you a clean record of exactly what needs a second look in the morning.

Incremental beats full every time

Do not rescrape the world every day. Most sites change only a little between runs, so pulling everything from scratch wastes your resources and pounds the site for no reason. Run incrementally: track what you have already collected and only fetch what is new or changed since last time. It is lighter on your infrastructure, far lighter on the site, and it is the difference between a pipeline that scales and one that gets slower and more fragile as your dataset grows.

Backfills without fear

Sometimes you do need to go back and reprocess history, maybe because you fixed a parser or added a field. That is a backfill, and a good pipeline makes it painless. Because your stages are idempotent and your raw pages are saved, you can rerun the parse stage across a date range and rebuild the clean data without touching the site at all. Dagster in particular is built around reprocessing ranges of data. Designing for backfills from the start means a schema change is an afternoon, not a crisis.

Observability, or you are flying blind

You cannot fix what you cannot see. A pipeline that runs silently is one you do not actually trust, because the first sign of trouble is stale data days later. So you instrument it. Log how many pages you fetched, how many parsed cleanly, how many landed in the dead letter queue, and how long each stage took. Both orchestrators give you a dashboard of runs and their history. When the parsed count suddenly drops to zero, you want to see it that hour, not discover it when someone downstream asks where the data went.

Alert on the right thing

Do not alert on everything, or you will train yourself to ignore the alarms. A handful of failed pages is normal background noise and should never wake you. What should wake you is a signal the pipeline is actually broken: a whole run failing, the success rate falling off a cliff, or zero records produced when you expected thousands. An alert that fires constantly is worse than no alert, because you stop reading it.

Dagster or Prefect, briefly

Both are solid Python orchestrators and either will serve you well. Prefect leans lightweight and flexible, wrapping your existing functions as tasks and flows with very little ceremony, which makes it fast to adopt. Dagster leans toward treating your data as first class assets, with strong support for backfills, typing, and knowing exactly what each step produces. For a scrape heavy on reprocessing history I tend toward Dagster; for a simpler flexible flow I reach for Prefect. Neither choice is wrong, so pick one and build.

The shape of a Prefect flow is small. You mark functions as tasks, give them retries, and wire them into a flow:

from prefect import flow, task

@task(retries=3, retry_delay_seconds=[5, 30, 120])
def fetch(url: str) -> str:
    ...

@task
def parse(raw: str) -> list[dict]:
    ...

@task
def store(records: list[dict]) -> None:
    ...

@flow
def scrape(urls: list[str]) -> None:
    for url in urls:
        store(parse(fetch(url)))

That is a real pipeline skeleton, retries and all, in a handful of lines.

Validate, or you fill the database with nulls

The failure that hurts most is the one that does not throw an error. A site quietly renames a field, or starts returning an empty value where there used to be a price, and your parser keeps running and writes rows full of nulls. Nothing crashes, so nothing alerts, and you find out days later that your data is hollow. The defense is validation: check that the records coming out of the parse stage look right, that key fields are present and in a sane range, and fail loudly when they are not.

Test on a small slice first

Never point a fresh pipeline at the full job on its first run. Run it against a tiny slice, a handful of pages, and watch every stage work end to end before you scale up. It is far cheaper to find a broken selector or a bad write on ten records than on ten million. Keep a small sample run you can fire on demand, so after any change you confirm the whole pipeline still flows cleanly before you let it loose on the real volume.

Keep the site happy at the pipeline level

Politeness is not just a per request setting, it is a pipeline property. The orchestrator controls how many workers run at once and how fast the whole thing moves, so it is where you enforce a rate the site can absorb. Spread the load across time instead of firing a huge burst, cache aggressively so you never fetch the same page twice, and prefer an official api or bulk feed when the site offers one. A pipeline that runs gently is one that keeps running, because it never gives the site a reason to shut the door.

The honest limits

Good orchestration makes a scrape reliable and recoverable. It does not make it welcome where it is not, and it does not change what you are allowed to collect. Public data, a robots file respected, an official feed preferred, a gentle pace held. Structure buys you uptime and sane recovery, not permission. No pipeline survives a site that genuinely does not want to be scraped, nor should it try to, because durable data work lives on the compliant side of that line.

To recap: split fetch from parse from store, make every stage idempotent so retries are safe, checkpoint so you can resume, and retry with backoff instead of hammering. Set failures aside in a dead letter queue, run incrementally instead of rescraping the world, design for painless backfills, and watch the pipeline closely enough to alert only on real breakage. Build it that way and it keeps running while you sleep.

If you want more breakdowns like this, dataresearchtools.com has the rest of the scraping infrastructure guides, with real code and no undetectable promises.

Get new guides and videos first — join the Telegram channel.

Two clients ask the same site for the same page in the same second. One gets the data and moves on. The other gets a captcha, then a soft block, then nothing. The requests looked almost identical, but the decision was made before either connection finished setting up. That gap is where all of bot detection lives, and understanding it is the difference between collection that runs for months and collection that dies on day one.

I run proxy infrastructure and production scrapers, so I have spent a lot of time on both sides of this wall. This is a defensive explainer, written from the site’s point of view: how modern detection actually works, and how a scraper that respects the rules stays clean. It is not a guide to evading anything, because that version does not last and it is not the business I am in.

What detection is actually looking for

The first thing to get right is the question a site is trying to answer. It is not “human or bot.” Plenty of bots are welcome. Search crawlers, uptime monitors, and link previewers are all automation, and sites wave them through on purpose. The real question is narrower: does this traffic behave the way it claims to, and do its own signals agree with each other?

Detection is a hunt for contradictions. Every layer below is one more place where an automated client can accidentally tell two different stories at once, and every mismatch is a chance to get flagged. That framing matters, because it explains why the durable answer is honesty rather than disguise. If you are not pretending to be something you are not, there is no contradiction to catch.

Network and ip reputation

The earliest signal arrives before you send a byte of a real request: where you came from. Every connection has an ip address, and every ip sits inside a block that belongs to a network, identified by a number called an asn. An entire industry labels those blocks. This range is a mobile carrier, this one is home broadband, this one is a hosting company’s datacenter. Sites buy those labels, cache them right at the edge, and the check fires on the very first packet.

A request from a residential or mobile network starts with ordinary trust. A request from a cheap datacenter range starts under suspicion, because historically that is where automation lives. And the reputation is not really about you, it is about your neighbors. A fresh datacenter ip with no history of its own still inherits the cold reception of every scraper that ever ran from that provider. That is why pointing the cheapest pool of server ips at a strict site tends to fail fast. Nobody caught you doing anything wrong. The network you rode in on was already wearing the wrong label.

The tls handshake

Before any page loads, your client and the server negotiate an encrypted channel, and that negotiation is surprisingly loud. The exact way the software sets up tls, the cipher options it offers, the order it lists them in, and the extensions it includes all form a pattern. Real browsers produce well known patterns here, and they shift in predictable ways as browsers update.

A lot of scraping tools produce a tls pattern no real browser would send, because the underlying http library was never trying to imitate one. So a client can claim in its headers to be a current version of Chrome while its handshake quietly says it is a generic http library. That is a contradiction, and the edge exists to catch exactly that kind of mismatch.

Request headers and the user agent

Once the channel is open, the request itself carries more tells. A real browser sends a specific set of headers, in a specific order, with a user agent string that agrees with everything else. The field a site reads as your user agent is just text, so anyone can type Chrome into it. The rest of the request has to back that claim up.

Real Chrome sends particular hints about the platform, the language, and the way it handles compression and connections. An http client that sets a browser user agent and then forgets the dozen other things a browser would send is telling two stories again. Detection here is mostly consistency checking: do all the parts of this request describe the same thing they claim to be?

Browser and device fingerprinting

When a site wants a harder look, it runs code inside the browser itself, and this is where headless automation gets tested. A normal browser driving a real screen exposes thousands of small properties: the window size, the installed fonts, the way the gpu draws a shape onto a hidden canvas, the exact features webgl reports, the timing of tiny operations. Real devices vary across all of it in messy, natural ways.

An automation setup that spins up the same headless browser again and again tends to produce the same fingerprint every time, or a fingerprint carrying tells a normal machine would never have. The site is not hunting for one magic value. It is looking at the whole shape and asking whether it resembles a real person’s device or a cloned machine in a datacenter. There is also a quieter middle layer here: when a site is unsure, it hands the client a small challenge, a bit of work a real browser does without thinking. For a person it passes invisibly. For a crude client it is a wall.

Behavioral and rate signals

Past the technical fingerprints, sites watch behavior, and behavior is where scale gives automation away. A person browses in bursts, pauses, gets distracted, reads for a while, and comes back. A naive scraper pulls pages at a steady machine rhythm, one after another, faster than a human could read, at even intervals, around the clock with no sleep. No single request is strange. The shape over time is.

It widens out from one connection to the whole crowd, too. Edge networks sit in front of a huge slice of the web, so they see traffic for countless sites at once. That lets them spot a swarm of fresh visitors all taking the same path at the same speed, or many separate ips that each look fine alone but move in lockstep. Rate is part of it, how much you pull and how fast. Pattern is the deeper signal, and a new automation pattern caught on one site becomes a signature that defends the rest within minutes.

How compliant scrapers stay clean

So what does an operation that respects the rules do differently? It starts by reading the robots file and honoring it, treating the parts a site asks you to leave alone as off limits rather than as a suggestion. It identifies itself honestly where a site expects that, instead of dressing a server up as a browser it is not. It collects public data only, and stays away from anything behind a login, a paywall, or personal information that was never meant to be harvested.

The biggest lever is pace, and it is the one people skip because it feels slow. A compliant scraper runs at a rate the site can absorb, spaces its requests, backs off the moment it sees errors or slowdowns, and caches so it never asks twice for something it already has. It prefers an official api when one exists, because that is the front door the site actually built. Done this way, most of the detection stack never has a reason to fire, because you are not producing the contradictions it hunts for.

There is also a step above all the tooling: ask. If a site offers a data feed, a bulk download, or a documented api, that sanctioned path is almost always cheaper than fighting the detection stack. If the terms of service say no, then no clever handshake changes what you agreed to. For anything ambiguous, a short message to the site owner is worth more than a week of tuning a client, because permission turns the whole adversarial framing off.

The honest limits

I want to be straight about the boundaries. None of this makes anything undetectable, and I would not trust anyone who tells you it does. Detection keeps improving, the labels get sharper, and what passes quietly today can be flagged tomorrow. A clean network and honest behavior are not a trick that beats the wall. They are the absence of the contradictions that get you caught.

The moment your collection touches private data, or ignores what a site clearly asked you not to do, you have left the compliant lane entirely, whatever your tooling looks like. The durable version of this work is the boring version: public data, honest identification, a gentle pace, and the rules respected.

I run this infrastructure in production, so the frameworks, proxy setups, scraping apis, and honest tool reviews I write about are the ones I actually deal with. If you want the full written guides and the tools I use and test, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the way this actually works.

Introduction: The Analytics Blind Spot

Customer journey analytics promises a complete view of how users interact with your brand — from first ad impression to final purchase. But there’s a persistent problem: the data is often wrong.

Location mismatches, carrier-level filtering, bot traffic mixing with real users, and geo-blocked content all distort the picture. Marketers end up making decisions based on data that doesn’t accurately represent how real customers actually experience their brand.

Mobile proxies are increasingly being used to close this gap — by enabling teams to collect, validate, and enrich journey data from the perspective of real mobile users in real locations.

---

What Is Customer Journey Analytics?

Customer journey analytics maps every touchpoint a user encounters across channels and devices — including:

• Paid search and social ads
• Organic search results
• Website and app visits
• Email engagement
• In-store or location-based interactions
• Customer service contacts

The goal is to understand what drives conversion, where users drop off, and how different segments experience the brand differently. But all of this depends on the quality and accuracy of the underlying data.

Where Data Gaps Occur — and Why

Several common scenarios corrupt or incomplete customer journey data:

1. Geo-IP Misattribution

Standard analytics platforms use IP geolocation to assign location to users. But mobile carrier IP ranges are often misclassified — a user in Lagos may be assigned to a London IP pool if they’re roaming or using a multinational carrier. This creates false geographic segments in your analytics.

2. Bot and Proxy Traffic Contamination

Up to 40% of web traffic can be non-human. Bots scraping your site, competitors checking prices, and internal QA teams all show up in analytics and distort conversion funnels, bounce rates, and session data.

3. Carrier-Level Content Blocking

Some mobile carriers filter or transform content before it reaches users. Ad pixels may fail to fire. Tracking scripts may be blocked. Users on certain networks may never trigger the analytics events your platform depends on — creating silent gaps in journey data.

4. Device-Dependent Rendering

Desktop-based analytics collection tools can’t capture how mobile users actually experience pages. Features that break on specific devices — slow loading images, broken CTAs, unresponsive forms — show up as conversion drop-offs without explanation.

How Mobile Proxies Close the Data Gap

Accurate Geo-Segmented Data Collection

By routing analytics collection through real mobile IPs in specific cities and countries, teams can gather journey data that reflects what real users in those locations actually see and do. This allows:

• Verification that geo-targeted content is triggering correct analytics events.
• Calibration of geo-segmentation models with clean, location-confirmed data.
• Identification of regions where tracking is failing silently.

Carrier-Level Tracking Verification

Mobile proxies let analytics teams test whether tracking pixels, conversion tags, and event scripts fire correctly across different carrier networks. If a carrier is stripping scripts or blocking pixels, you’ll see it in your test data — before it silently corrupts months of production analytics.

Real Mobile Session Simulation

Connecting through real mobile IPs on genuine carrier networks replicates the actual conditions under which your users browse. This enables:

• Accurate session replay and funnel analysis from real mobile environments.
• Detection of mobile-specific friction points in the customer journey.
• Validation that mobile-specific tracking tags (app install events, SMS clicks) fire correctly.

Competitive Journey Benchmarking

Beyond your own analytics, mobile proxies allow marketing teams to map competitor customer journeys — seeing how rivals present their funnels, CTAs, pricing, and onboarding flows in specific markets. This context helps teams benchmark conversion performance against real market conditions.

Practical Use Cases by Team

Team	Use Case	Benefit
Analytics	Validate geo-segment accuracy	Eliminate misattributed location data
Paid Media	Verify tracking pixel fires by carrier	Accurate ROAS reporting
Product	Test funnel on real mobile networks	Catch mobile-only drop-off points
CRO	Confirm A/B test bucketing by region	Clean experiment data
Compliance	Audit consent and cookie tracking	GDPR/CCPA data accuracy

Integrating Mobile Proxies into Your Analytics Stack

Mobile proxies work alongside — not instead of — your existing analytics tools. Here’s how to integrate them effectively:

1. Map your key journey events — identify the 10–15 critical events that define your funnel (page view, add-to-cart, checkout start, purchase, etc.).
2. Set up geo-specific test sessions — use mobile proxies to simulate user sessions from your top 5–10 markets and verify all events fire correctly.
3. Automate carrier-coverage checks — schedule weekly automated tests across major carriers per region to catch tracking failures as they occur.
4. Cross-reference with production data — compare test session behavior against production analytics to identify discrepancies that indicate tracking issues.
5. Iterate on findings — use insights from proxy-based testing to patch tracking gaps, improve geo-segmentation, and clean historical data where needed.

FAQs: Mobile Proxies and Customer Journey Data

Q1: Will using mobile proxies for testing skew my production analytics?
Not if configured correctly. Test traffic can be filtered out using IP lists, custom UTM parameters, or separate analytics properties for QA sessions.

Q2: Can mobile proxies help fix attribution model inaccuracies?
Yes. By verifying that events fire correctly across all major markets and carriers, you improve the raw data quality that attribution models depend on.

Q3: How often should analytics validation runs be scheduled?
At minimum, after every major site deployment. For high-traffic e-commerce or media businesses, weekly automated checks are recommended.

Q4: Do mobile proxies work with Google Analytics, Adobe Analytics, and similar platforms?
Yes — mobile proxy sessions generate standard HTTP traffic that is captured by all major analytics platforms in the same way as real user sessions.

Conclusion: Better Data Starts with Real-World Testing

Customer journey analytics is only as good as the data that feeds it. When tracking fails silently across certain carriers, when geo-segments are misclassified, or when mobile-specific conversion events are missed, marketing decisions are built on a broken foundation.

Mobile proxies provide the one thing standard analytics tooling can’t: the ability to experience your customer journey as your customers actually experience it — from their carrier, in their city, on their device.

• They close geo-attribution gaps that distort segmentation.
• They expose carrier-level tracking failures before they corrupt production data.
• They enable competitive journey benchmarking that no other tool can provide.

👉 Bottom line: If your marketing organization is serious about data quality, validating your analytics stack with mobile proxies isn’t optional — it’s a competitive necessity.

Building scraping pipelines with Dagster in 2026

Building scraping pipelines with Dagster is the right answer when scraping is one part of a larger data engineering pipeline. Dagster reframes orchestration around data assets rather than tasks: instead of writing “fetch URLs, parse, store” as separate steps, you declare “products dataset” as a software-defined asset that depends on “raw HTML pages” which depends on “URL list,” and Dagster figures out the execution graph. For pure scraping with no downstream analytics, this is overkill. For scraping that feeds into ML models, dashboards, or warehouses, Dagster’s asset-aware approach pays off in lineage, partitioning, and freshness tracking.

This guide covers Dagster 1.7+ in 2026 for scraping pipelines: software-defined assets, jobs, sensors, partitioning by time and host, IO managers, and a complete end-to-end example. Code is Python 3.12. By the end you will know whether Dagster fits your scraping use case and how to deploy a working pipeline.

Why Dagster for scraping

Dagster’s strengths for scraper-heavy pipelines:

• Software-defined assets: declare data products instead of orchestration steps
• Lineage tracking: see how data flows from raw HTML to final dashboard
• Partition support: per-day, per-host, per-customer partitions with backfill
• IO managers: separate “what to compute” from “where to store it”
• Sensors: trigger pipelines based on external events (file arrival, API webhook)
• Strong type system: Pydantic-style configs and asset metadata
• Asset checks: data quality assertions per asset
• Mature: 6+ years in production, used by Robinhood, Duolingo, etc.

For Dagster’s official documentation, see docs.dagster.io.

Where Dagster does not lead

• For simple scheduled scraping with no downstream pipeline, Prefect or cron are simpler
• For very high-frequency tasks (1000s/sec), the asset model adds overhead
• For teams without data engineering experience, the asset abstraction takes adjustment

For Prefect comparison, see building scraping pipelines with Prefect 3.

Installing Dagster

pip install dagster dagster-webserver
dagster --version  # 1.7+

# Initialize a project
dagster project scaffold --name my-scraper
cd my-scraper

Start the dev server:

dagster dev
# UI at http://localhost:3000

A first asset: scraping a URL

# my_scraper/assets.py
from dagster import asset, AssetExecutionContext
import httpx

@asset
def homepage_html(context: AssetExecutionContext) -> str:
    resp = httpx.get("https://example.com", headers={
        "User-Agent": "Mozilla/5.0 ...",
    })
    resp.raise_for_status()
    context.log.info(f"Fetched {len(resp.text)} bytes")
    return resp.text

@asset
def homepage_titles(homepage_html: str) -> list[str]:
    from selectolax.parser import HTMLParser
    tree = HTMLParser(homepage_html)
    return [n.text(strip=True) for n in tree.css("h2")]

Two assets: homepage_html (the raw HTML) and homepage_titles (parsed titles, depends on homepage_html). Dagster figures out the dependency from the function signature.

In the Dagster UI, you see a graph with two nodes connected by a line. Click “Materialize all” and Dagster fetches the page and parses it.

Partitioned assets: scrape per day, per host

For real scraping, you usually want partitions: one materialization per day, per host, or per customer. Dagster partitions track which slices have been materialized and which need backfilling.

from dagster import asset, DailyPartitionsDefinition, AssetExecutionContext
from datetime import datetime
import httpx

daily_partitions = DailyPartitionsDefinition(start_date="2026-01-01")

@asset(partitions_def=daily_partitions)
def daily_news_html(context: AssetExecutionContext) -> str:
    date_str = context.partition_key  # YYYY-MM-DD
    url = f"https://news.example.com/archive/{date_str}"
    resp = httpx.get(url)
    resp.raise_for_status()
    return resp.text

Now daily_news_html has one materialization per day. Run today’s: Dagster fetches today’s URL. Backfill last 30 days: Dagster fetches each day’s URL.

For multi-dimensional partitions (per-day AND per-host):

from dagster import (
    asset, MultiPartitionsDefinition, StaticPartitionsDefinition,
    DailyPartitionsDefinition,
)

hosts = StaticPartitionsDefinition(["example.com", "news.example.com", "shop.example.com"])
date_partitions = DailyPartitionsDefinition(start_date="2026-01-01")

multi_partitions = MultiPartitionsDefinition({"date": date_partitions, "host": hosts})

@asset(partitions_def=multi_partitions)
def host_daily_pages(context: AssetExecutionContext) -> str:
    keys = context.partition_key.keys_by_dimension
    date_str = keys["date"]
    host = keys["host"]
    url = f"https://{host}/archive/{date_str}"
    resp = httpx.get(url)
    return resp.text

This creates a 2D grid of partitions. Backfill specific cells (date X host) without re-running everything.

IO managers: separating compute from storage

In simple cases, the asset function returns a Python value and Dagster pickles it to local disk. For production, use IO managers to write to Postgres, S3, or your warehouse.

from dagster import IOManager, io_manager, OutputContext, InputContext
import boto3
import json
import io

class S3IOManager(IOManager):
    def __init__(self, bucket: str):
        self.bucket = bucket
        self.s3 = boto3.client("s3")

    def _key(self, context):
        # asset_key is a list of strings
        path = "/".join(context.asset_key.path)
        if context.has_partition_key:
            path = f"{path}/{context.partition_key}"
        return f"{path}.json"

    def handle_output(self, context: OutputContext, obj):
        key = self._key(context)
        body = json.dumps(obj).encode() if not isinstance(obj, bytes) else obj
        self.s3.put_object(Bucket=self.bucket, Key=key, Body=body)
        context.log.info(f"Wrote to s3://{self.bucket}/{key}")

    def load_input(self, context: InputContext):
        key = self._key(context)
        resp = self.s3.get_object(Bucket=self.bucket, Key=key)
        return json.loads(resp["Body"].read())

@io_manager(config_schema={"bucket": str})
def s3_io_manager(init_context):
    return S3IOManager(bucket=init_context.resource_config["bucket"])

Wire it in:

from dagster import Definitions

defs = Definitions(
    assets=[homepage_html, homepage_titles],
    resources={"io_manager": s3_io_manager.configured({"bucket": "my-scraper-data"})},
)

Now every asset’s output goes to S3 instead of local disk. Same code, different storage.

Browser scraping with Playwright as an asset

from dagster import asset, AssetExecutionContext
from playwright.sync_api import sync_playwright

@asset(partitions_def=daily_partitions)
def dynamic_page_html(context: AssetExecutionContext) -> str:
    date_str = context.partition_key
    url = f"https://example.com/dynamic?date={date_str}"

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        html = page.content()
        browser.close()
    return html

Same pattern: Playwright runs inside the asset function. Dagster orchestrates.

Jobs: grouping assets

Jobs run sets of assets together. Define a job that materializes a related group:

from dagster import define_asset_job, AssetSelection

scrape_news_job = define_asset_job(
    "scrape_news",
    selection=AssetSelection.assets("daily_news_html", "daily_news_articles"),
)

Schedule it:

from dagster import ScheduleDefinition

daily_news_schedule = ScheduleDefinition(
    job=scrape_news_job,
    cron_schedule="0 6 * * *",  # 6am daily
)

Sensors: event-driven materialization

Sensors trigger asset materialization based on external events:

from dagster import sensor, RunRequest, SkipReason
from datetime import datetime
import httpx

@sensor(job=scrape_news_job)
def new_url_sensor(context):
    # Check an API for new URLs
    resp = httpx.get("https://api.example.com/new-urls")
    new_urls = resp.json()

    if not new_urls:
        return SkipReason("No new URLs")

    return RunRequest(
        run_key=str(datetime.utcnow().timestamp()),
        run_config={"ops": {"daily_news_html": {"config": {"urls": new_urls}}}},
    )

Dagster runs the sensor function periodically (default 30 seconds). When it returns a RunRequest, the job triggers.

Asset checks: data quality

Built-in data quality assertions per asset:

from dagster import asset, AssetCheckResult, asset_check

@asset
def products_data(homepage_html: str) -> list[dict]:
    # Parse and return products
    return [{"title": "...", "price": "..."}]

@asset_check(asset=products_data)
def check_min_products(context, products_data):
    count = len(products_data)
    return AssetCheckResult(
        passed=count >= 10,
        metadata={"count": count, "expected_min": 10},
    )

@asset_check(asset=products_data)
def check_no_missing_prices(context, products_data):
    missing = sum(1 for p in products_data if not p.get("price"))
    return AssetCheckResult(
        passed=missing == 0,
        metadata={"missing_count": missing},
    )

After every materialization, Dagster runs the checks and shows pass/fail in the UI. Fail an SLA, fail a build, alert the team.

Comparison: Dagster vs Prefect for scraping

dimension	Dagster	Prefect 3
paradigm	software-defined assets	flows and tasks
best for	data pipelines with downstream ETL	mid-complexity workflows
async support	sync-first, async via Op syntax	excellent async
partitioning	rich (multi-dimensional, backfill)	basic
asset lineage	first-class	via DAGs
hosted option	Dagster Cloud	Prefect Cloud
learning curve	medium-high	low-medium
dashboard	excellent	excellent

Pick Dagster when scraping feeds analytics, ML, or BI. Pick Prefect when scraping is the end product or when async-heavy.

Complete production example

# my_scraper/definitions.py
from dagster import (
    asset, AssetExecutionContext, Definitions,
    DailyPartitionsDefinition, define_asset_job, ScheduleDefinition,
    AssetSelection, IOManager, io_manager, Resource,
)
from dagster_aws.s3 import S3PickleIOManager, S3Resource
import httpx
from selectolax.parser import HTMLParser
import json
from typing import Optional

daily_partitions = DailyPartitionsDefinition(start_date="2026-01-01")

@asset(partitions_def=daily_partitions, group_name="ingestion")
def raw_pages(context: AssetExecutionContext) -> dict[str, str]:
    """Fetch all configured pages for the partition date."""
    date_str = context.partition_key
    urls = [
        f"https://example.com/products?date={date_str}&page={i}"
        for i in range(1, 11)
    ]
    pages = {}
    for url in urls:
        try:
            resp = httpx.get(url, timeout=30)
            resp.raise_for_status()
            pages[url] = resp.text
            context.log.info(f"Fetched {url}: {len(resp.text)} bytes")
        except Exception as e:
            context.log.error(f"Failed {url}: {e}")
    return pages

@asset(partitions_def=daily_partitions, group_name="parsing")
def parsed_products(context: AssetExecutionContext, raw_pages: dict) -> list[dict]:
    """Parse products from all raw pages."""
    products = []
    for url, html in raw_pages.items():
        tree = HTMLParser(html)
        for el in tree.css("article.product"):
            title_el = el.css_first("h2")
            price_el = el.css_first(".price")
            link_el = el.css_first("a")
            products.append({
                "title": title_el.text(strip=True) if title_el else None,
                "price": price_el.text(strip=True) if price_el else None,
                "url": link_el.attributes.get("href") if link_el else None,
                "source_url": url,
                "scraped_date": context.partition_key,
            })
    context.log.info(f"Parsed {len(products)} products")
    return products

@asset(partitions_def=daily_partitions, group_name="storage")
def products_in_db(context: AssetExecutionContext, parsed_products: list) -> int:
    """Write parsed products to Postgres."""
    import asyncpg
    import asyncio

    async def write():
        conn = await asyncpg.connect("postgresql://user:pass@localhost/scraper")
        try:
            await conn.executemany(
                """INSERT INTO products
                   (title, price, url, source_url, scraped_date)
                   VALUES ($1, $2, $3, $4, $5)
                   ON CONFLICT (url, scraped_date) DO NOTHING""",
                [(p["title"], p["price"], p["url"], p["source_url"], p["scraped_date"])
                 for p in parsed_products],
            )
            return len(parsed_products)
        finally:
            await conn.close()

    count = asyncio.run(write())
    context.log.info(f"Stored {count} products to DB")
    return count

@asset_check(asset=parsed_products)
def check_minimum_products(context, parsed_products):
    return AssetCheckResult(
        passed=len(parsed_products) >= 50,
        metadata={"count": len(parsed_products), "expected_min": 50},
    )

scrape_job = define_asset_job(
    "daily_scrape",
    selection=AssetSelection.assets("raw_pages", "parsed_products", "products_in_db"),
)

scrape_schedule = ScheduleDefinition(
    job=scrape_job,
    cron_schedule="0 6 * * *",
    execution_timezone="UTC",
)

defs = Definitions(
    assets=[raw_pages, parsed_products, products_in_db],
    asset_checks=[check_minimum_products],
    jobs=[scrape_job],
    schedules=[scrape_schedule],
    resources={
        # Add IO managers, DB connections, etc.
    },
)

This pipeline:

1. Fetches raw pages daily, partitioned by date
2. Parses products from raw HTML
3. Stores to Postgres with idempotent insert
4. Asserts minimum product count
5. Runs daily at 6am UTC

In the Dagster UI, you see the asset graph, materialization history, and check pass/fail per partition.

Deployment

For production:

• Dagster Cloud (managed): easiest, paid per user + execution
• Self-hosted on Kubernetes: dagster-helm chart, full control
• Self-hosted on a single VM: docker-compose with PostgreSQL, fine for small teams

For Kubernetes:

# values.yaml for dagster-helm
dagsterDaemon:
  enabled: true
runLauncher:
  type: K8sRunLauncher
postgresql:
  enabled: true

Dagster spawns a Kubernetes pod per run, isolating resources.

Operational checklist

For production Dagster scraping in 2026:

• Dagster 1.7+ with Python 3.11+
• Dagster Cloud or self-hosted on Kubernetes
• Partitions defined for backfill capability
• IO managers for production storage (S3, Postgres)
• Asset checks for data quality
• Sensors for event-driven workflows
• Schedules for cron-style runs
• Type hints on all asset functions
• Resource configs separate from code

For broader pipeline patterns, see building scraping pipelines with Prefect 3 and distributed scraping with Apache Kafka.

Common pitfalls

• Asset name conflicts: two assets with the same key error out. Use key_prefix to namespace.
• Partition explosion: multi-dimensional partitions can produce millions of cells. Plan partitioning carefully.
• Sync code in async contexts: Dagster is sync-first; for async, use asyncio.run inside the asset.
• Long-running assets: Dagster default timeout is short. Configure execution timeout per asset.
• IO manager mismatches: changing IO managers mid-flight can cause input/output type mismatches.

FAQ

Q: should I use Dagster or Prefect for pure scraping?
For pure scraping, Prefect’s simpler model is usually a better fit. Dagster pays off when scraping is part of a larger ETL or ML pipeline.

Q: can I migrate from Airflow to Dagster?
Yes, with effort. The asset model is different from DAGs. Most teams migrate one pipeline at a time over months.

Q: does Dagster work with Scrapy?
Yes. Wrap Scrapy invocations as assets. The orchestration value is at the pipeline level, not within individual spiders.

Q: how does cost compare to Prefect?
Self-hosted: similar (just compute). Cloud: Dagster Cloud is more expensive per user, but pricing for execution is comparable.

Q: is Dagster overkill for a 5-spider project?
Probably yes. For 5 unrelated scrapers, cron + Python scripts is fine. For 5 scrapers feeding a unified analytics warehouse, Dagster shines.

Common pitfalls in production Dagster scraping

The first failure mode is the partition backfill blast radius. Dagster’s backfill UI lets you select multiple partitions and re-materialize them. A typo in the partition selector (selecting “all 2026 partitions” when you meant “this week’s partitions”) triggers 365 simultaneous materialization runs, which queues 365 Kubernetes pods, exhausts your cluster’s pod limit, and starves all other pipelines for hours. The fix is to enable backfill concurrency limits in the Dagster instance config, capping simultaneous backfill runs at a sane number:

# dagster.yaml
run_coordinator:
  module: dagster._core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 25
    tag_concurrency_limits:
      - key: dagster/backfill
        limit: 5

This caps backfills at 5 concurrent runs while leaving 20 slots for scheduled and ad-hoc runs. Without this, a single mis-clicked backfill can take down your entire pipeline cluster.

The second pitfall is asset definition import-time side effects. Dagster imports your asset definitions on every code-server reload to build the asset graph. If your asset definition file contains import-time code that fetches from an external service (loading site configs from a database at module import), every code reload triggers that fetch. With 50 developers reloading code 100 times per day, you generate 5000 daily fetches that the upstream service was never designed for, and you sometimes get throttled by your own configuration store. The fix is to defer all external calls to inside the asset function or use a Dagster Resource that lazy-loads:

from dagster import resource, asset, ConfigurableResource

class SiteConfigResource(ConfigurableResource):
    config_url: str
    _cache: dict | None = None

    def get_configs(self) -> dict:
        if self._cache is None:
            self._cache = httpx.get(self.config_url).json()
        return self._cache

@asset
def raw_pages(context, site_configs: SiteConfigResource):
    configs = site_configs.get_configs()  # only fetched when asset runs
    # ...

The third pitfall is the IO manager memory bloat on large asset values. Dagster’s default IO managers (S3PickleIOManager, FilesystemIOManager) serialize the entire asset value to storage between assets. A raw_pages asset returning 10,000 HTML pages of 50KB each is 500MB of data. The pickle serialize-deserialize cycle runs in a single process, peaking at 1GB+ resident memory. Workers OOM-kill, the run fails, and Dagster retries it, blowing through your retry budget. The fix is custom IO managers that stream rather than buffer, or asset partitioning that keeps individual asset values under 100MB:

class StreamingS3IOManager(IOManager):
    def handle_output(self, context, obj):
        # obj is an iterator, not a list
        s3_key = f"{context.asset_key.path[-1]}/{context.partition_key}.jsonl"
        with self.s3_client.open(s3_key, "wb") as f:
            for item in obj:
                f.write(json.dumps(item).encode() + b"\n")

    def load_input(self, context):
        s3_key = f"{context.asset_key.path[-1]}/{context.partition_key}.jsonl"
        return (json.loads(line) for line in self.s3_client.iter_lines(s3_key))

Then design your assets to yield items rather than return lists. Memory usage drops from O(N) to O(1) per asset, regardless of partition size.

Real-world example: Dagster + dbt for ecommerce intelligence

A scraping team built a Dagster pipeline that scraped 30 ecommerce sites daily and fed a dbt-managed warehouse for downstream analytics. The architecture treated scraped data as Dagster assets that materialized into Snowflake, and dbt models as downstream Dagster assets that ran SQL transformations against those tables:

from dagster import asset, AssetExecutionContext, DailyPartitionsDefinition
from dagster_dbt import DbtCliResource, dbt_assets, get_asset_key_for_model
from pathlib import Path

DBT_PROJECT_DIR = Path(__file__).parent / "ecom_dbt"
daily = DailyPartitionsDefinition(start_date="2026-01-01")

@asset(partitions_def=daily, group_name="raw_scrape")
def raw_amazon_products(context: AssetExecutionContext):
    date_str = context.partition_key
    products = scrape_amazon(date=date_str)
    write_to_snowflake("raw.amazon_products", products)
    context.add_output_metadata({
        "row_count": len(products),
        "preview": products[:5],
    })

@asset(partitions_def=daily, group_name="raw_scrape")
def raw_walmart_products(context: AssetExecutionContext):
    date_str = context.partition_key
    products = scrape_walmart(date=date_str)
    write_to_snowflake("raw.walmart_products", products)
    context.add_output_metadata({"row_count": len(products)})

@dbt_assets(
    manifest=DBT_PROJECT_DIR / "target" / "manifest.json",
    partitions_def=daily,
)
def ecom_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

The dbt project contained models like staging_amazon_products (cleans the raw scrape), mart_competitive_pricing (joins all sites for cross-comparison), and ml_features_pricing (rolling windows for the pricing prediction model).

In the Dagster UI, the asset lineage showed the full chain: scrape job -> raw tables -> staging models -> mart models -> ML feature tables. When the pricing prediction model returned anomalous results, the team traced the lineage backward and discovered the root cause was a 24-hour gap in the Walmart scrape (the site had pushed an HTML structure change that broke the parser). The lineage view turned a 4-hour debugging session into a 15-minute one.

Asset checks at each layer caught quality issues before they propagated:

@asset_check(asset=raw_amazon_products)
def amazon_row_count_check(context, raw_amazon_products):
    rows = query_snowflake(
        f"SELECT COUNT(*) FROM raw.amazon_products WHERE scraped_date = '{context.partition_key}'"
    )[0][0]
    return AssetCheckResult(
        passed=rows >= 5000,
        severity=AssetCheckSeverity.ERROR if rows < 1000 else AssetCheckSeverity.WARN,
        metadata={"row_count": rows, "expected_min": 5000},
    )

A check failure marked the partition as unhealthy in the UI and prevented downstream dbt models from running until the issue was resolved. Over six months in production, asset checks caught 47 distinct data quality regressions before they reached the analytics layer.

Comparison: Dagster patterns by scraping scale

A reference table of Dagster patterns that work well at different scraping volumes:

volume	pattern	partition strategy	runtime	notes
<100 URLs/day	single-asset, no partitions	none	local Dagster	simplest
1K-100K URLs/day	per-site assets, daily partitions	DailyPartitionsDefinition	self-hosted	sweet spot
100K-10M URLs/day	per-site assets, hourly partitions	HourlyPartitionsDefinition + multi-dim	k8s with autoscale	needs IO manager tuning
10M+ URLs/day	partitioned ops within assets	DynamicPartitionsDefinition	Dagster Cloud Hybrid + k8s	requires custom IO managers
Realtime triggers	sensors + asset reconciliation	none, sensor-driven	self-hosted	use Prefect instead if pure realtime

For most scraping workloads, the 1K-100K URLs/day band with DailyPartitionsDefinition is the right starting point. Migrate up the table only when partitioning becomes the bottleneck. Migrate down (to a simpler tool) if Dagster’s overhead exceeds your benefit.

Detection: when Dagster is the wrong choice for scraping

Five signals that your scraping workload should NOT live on Dagster:

1. Pure scraping with no downstream ETL: if all you do is scrape and dump to JSON files, Dagster’s asset model is overhead. Use Prefect or cron + scripts.
2. Heavy async workload (1000+ concurrent requests): Dagster is sync-first. Heavy async work runs better in Prefect or Celery.
3. Sub-minute scheduling: Dagster’s minimum schedule interval is one minute. Sub-minute requires sensors or external triggers.
4. Fluid pipeline shape that changes per run: Dagster’s asset model assumes a stable graph. Per-run dynamic shape is awkward.
5. Real-time event-driven only: pure event-driven workloads work better in event-streaming systems (Kafka, NATS) than in orchestrators.

If three or more of these apply, Prefect or a simpler tool is the right answer.

Performance tuning: parallel asset execution

Dagster’s default behavior runs assets sequentially within a job. For scraping where assets are independent (different sites), enable multi-process or k8s execution:

from dagster import multiprocess_executor, define_asset_job

scrape_job = define_asset_job(
    "all_sites",
    selection="*",
    executor_def=multiprocess_executor.configured({
        "max_concurrent": 8,
    }),
)

This runs up to 8 site-scraping assets in parallel within one job run. For Kubernetes-based execution, use k8s_job_executor which spawns one pod per asset:

from dagster_k8s import k8s_job_executor

scrape_job = define_asset_job(
    "all_sites",
    executor_def=k8s_job_executor.configured({
        "max_concurrent": 30,
        "container_config": {
            "resources": {
                "requests": {"memory": "1Gi", "cpu": "500m"},
                "limits": {"memory": "4Gi", "cpu": "2000m"},
            },
        },
    }),
)

Per-asset isolation prevents one runaway scrape from OOM-killing the entire job. The trade-off is pod startup latency (5-15 seconds per asset), which is fine for hourly or daily jobs but punitive for sub-minute schedules.

Wrapping up

Dagster excels when scraping is one node in a larger data pipeline. The software-defined asset model gives lineage, partitioning, and freshness tracking that pure orchestrators do not. For teams that already think in terms of data products and downstream consumers, it pays off. Pair this with our building scraping pipelines with Prefect 3 and distributed scraping with Apache Kafka writeups for the full pipeline picture, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.

Building scraping pipelines with Prefect 3 in 2026

Building scraping pipelines with Prefect 3 is the right answer when your scraping has grown beyond a single Scrapy project but does not yet need full Apache Airflow’s complexity. Prefect 3 (released 2024) is the third major version of Prefect’s orchestration platform, and it brought meaningful changes from Prefect 2: simpler async support, faster task execution, work pools that decouple scheduling from execution, and a redesigned dashboard. For scraping specifically, Prefect 3 fits the pattern of “fetch URLs, parse them, store results, retry failures, schedule it all” with much less ceremony than Airflow.

This guide covers Prefect 3 for scraping pipelines in 2026: flows and tasks, work pools and workers, retries and concurrency control, scheduling, and a complete production example that fetches URLs, runs them through Playwright when needed, parses results, and stores to Postgres. By the end you will have a working pipeline pattern you can adapt to your specific targets.

Why Prefect for scraping

Prefect’s strengths for scraper pipelines:

• Python-native: flows and tasks are decorated Python functions, no DAG syntax to learn
• Async-friendly: tasks can be async def and run concurrently
• Built-in retries: configurable per task with exponential backoff
• Work pools and workers: decouple “what to run” from “where to run it”
• Cron, interval, and event-based scheduling: flexible
• Observability: dashboard, logs, run history all built in
• Cloud or self-hosted: free OSS or hosted Prefect Cloud
• Mature: 6+ years in production, rich ecosystem

For Prefect’s official documentation, see docs.prefect.io.

Where Prefect does not lead

• For very high frequency tasks (1000+ tasks per second), Prefect adds latency
• For data-engineering-heavy workloads with many integrations, Dagster has more native connectors
• For purely scheduled cron jobs without orchestration, simpler tools (cron itself, GitHub Actions) suffice

For scraping, Prefect’s mid-frequency, retry-rich, observable model fits well.

For Dagster comparison, see building scraping pipelines with Dagster.

Installing Prefect 3

pip install prefect>=3.0
prefect --version  # 3.x

Start the local Prefect server:

prefect server start
# Dashboard at http://localhost:4200

For production, use Prefect Cloud (managed) or self-host the server with PostgreSQL.

A first scraping flow

# scraper_flow.py
from prefect import flow, task
import httpx

@task(retries=3, retry_delay_seconds=10)
async def fetch_url(url: str) -> dict:
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.get(url, headers={
            "User-Agent": "Mozilla/5.0 ...",
        })
        resp.raise_for_status()
        return {"url": url, "status": resp.status_code, "html": resp.text}

@task
def parse_titles(html: str) -> list[str]:
    from selectolax.parser import HTMLParser
    tree = HTMLParser(html)
    return [n.text(strip=True) for n in tree.css("h2.title")]

@flow(log_prints=True)
async def scrape_titles(urls: list[str]):
    for url in urls:
        result = await fetch_url(url)
        titles = parse_titles(result["html"])
        print(f"{url}: {len(titles)} titles")
    return "Done"

if __name__ == "__main__":
    import asyncio
    asyncio.run(scrape_titles([
        "https://example.com/page1",
        "https://example.com/page2",
    ]))

Run it:

python scraper_flow.py

The flow appears in the Prefect dashboard with full task history, retries, and logs.

Concurrency: parallel task execution

For parallel scraping, use Prefect’s task mapping or asyncio.gather:

from prefect import flow, task
import httpx
import asyncio

@task(retries=3, retry_delay_seconds=10)
async def fetch_url(url: str) -> dict:
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.get(url)
        return {"url": url, "status": resp.status_code, "html": resp.text}

@flow(log_prints=True)
async def scrape_parallel(urls: list[str], concurrency: int = 10):
    semaphore = asyncio.Semaphore(concurrency)

    async def fetch_with_semaphore(url):
        async with semaphore:
            return await fetch_url(url)

    results = await asyncio.gather(
        *[fetch_with_semaphore(url) for url in urls],
        return_exceptions=True,
    )
    return results

Or using Prefect’s .map() for static parallelism:

@flow
def scrape_parallel_static(urls: list[str]):
    results = fetch_url.map(urls)
    return results

fetch_url.map(urls) schedules one task per URL and runs them via Prefect’s task runner. The default ConcurrentTaskRunner runs tasks in threads or processes; for async tasks use the ThreadPoolTaskRunner.

Work pools and workers

Prefect 3 splits the runtime model into:

• Flows / deployments: the work to run
• Work pools: configuration for runtime environments (Docker, Kubernetes, process)
• Workers: processes that pick up flow runs from work pools

This decouples scheduling from execution. You can have one Prefect server scheduling flow runs, with workers in different environments (laptop, Kubernetes cluster, EC2) picking up runs from the same work pool.

Create a work pool:

prefect work-pool create my-scraper-pool --type process
prefect worker start --pool my-scraper-pool

Deploy a flow to the pool:

from prefect import flow, serve

@flow
async def scrape_titles(urls: list[str]):
    # ...
    pass

if __name__ == "__main__":
    scrape_titles.serve(
        name="scraper-deployment",
        work_pool_name="my-scraper-pool",
        cron="0 * * * *",  # hourly
    )

Now the flow runs hourly on whatever worker picks it up.

Retries and error handling

Per-task retry config is the most common pattern:

@task(
    retries=5,
    retry_delay_seconds=[10, 30, 60, 120, 300],  # exponential backoff
)
async def fetch_url(url: str) -> dict:
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.get(url)
        if resp.status_code >= 500:
            raise httpx.HTTPStatusError(
                f"Server error: {resp.status_code}",
                request=resp.request, response=resp,
            )
        if resp.status_code == 429:
            # Wait longer for rate limits
            await asyncio.sleep(60)
            raise httpx.HTTPStatusError("Rate limited", request=resp.request, response=resp)
        return {"url": url, "status": resp.status_code, "html": resp.text}

For fine-grained retry policies (only retry on certain exceptions, not others):

from prefect.tasks import task_input_hash

@task(
    retries=5,
    retry_delay_seconds=30,
    retry_condition_fn=lambda task, run_state: (
        run_state.is_failed() and "5" in str(run_state.message)
    ),
)
async def fetch_url(url: str) -> dict:
    # ...
    pass

This retries only on 5xx errors, not on 4xx (which usually means permanent failure).

Persistent state: caching task results

For tasks that should not re-run on the same input (deduplication), use Prefect’s caching:

from prefect.tasks import task_input_hash
from datetime import timedelta

@task(
    cache_key_fn=task_input_hash,
    cache_expiration=timedelta(hours=24),
)
async def fetch_url(url: str) -> dict:
    # ...
    pass

Now if you call fetch_url("https://example.com") twice within 24 hours, the second call returns the cached result without running. Useful when re-running flows during development.

Browser scraping with Playwright in Prefect

For pages requiring JavaScript:

from prefect import flow, task
from playwright.async_api import async_playwright

@task
async def fetch_with_browser(url: str) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        html = await page.content()
        await browser.close()
        return {"url": url, "html": html}

@flow
async def scrape_dynamic_pages(urls: list[str]):
    results = []
    for url in urls:
        results.append(await fetch_with_browser(url))
    return results

Each task call spawns a browser, which is expensive. For high-volume browser scraping, batch URLs per browser:

@task
async def fetch_batch_with_browser(urls: list[str]) -> list[dict]:
    results = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        for url in urls:
            page = await browser.new_page()
            try:
                await page.goto(url, timeout=30000)
                html = await page.content()
                results.append({"url": url, "html": html})
            except Exception as e:
                results.append({"url": url, "error": str(e)})
            finally:
                await page.close()
        await browser.close()
    return results

This amortizes browser startup over many URLs.

Scheduling

Prefect supports cron, interval, and event-based scheduling:

# Cron
scrape_flow.serve(
    name="hourly-scraper",
    cron="0 * * * *",
    work_pool_name="my-pool",
)

# Interval
scrape_flow.serve(
    name="every-15min",
    interval=timedelta(minutes=15),
    work_pool_name="my-pool",
)

# Event-based (triggered by webhooks, file events, etc.)
from prefect.events import DeploymentEventTrigger

scrape_flow.serve(
    name="on-event",
    triggers=[
        DeploymentEventTrigger(
            expect={"my.custom.event"},
            parameters={"urls": "{{ event.payload.urls }}"},
        )
    ],
)

Event-based scheduling is useful for “scrape this URL when a webhook fires” patterns.

A complete production example

End-to-end scraping pipeline:

# pipeline.py
from prefect import flow, task, get_run_logger
from prefect.tasks import task_input_hash
from datetime import timedelta
import asyncio
import httpx
from selectolax.parser import HTMLParser
import asyncpg
import json

DB_DSN = "postgresql://user:pass@localhost/scraper"

@task(retries=3, retry_delay_seconds=[10, 30, 60])
async def fetch_url(url: str) -> dict:
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.get(url, headers={
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...",
        })
        resp.raise_for_status()
        return {"url": url, "status": resp.status_code, "html": resp.text}

@task
def parse_products(html: str) -> list[dict]:
    tree = HTMLParser(html)
    products = []
    for el in tree.css("article.product"):
        products.append({
            "title": el.css_first("h2").text(strip=True) if el.css_first("h2") else None,
            "price": el.css_first(".price").text(strip=True) if el.css_first(".price") else None,
            "url": el.css_first("a").attributes.get("href") if el.css_first("a") else None,
        })
    return products

@task(retries=3)
async def store_products(products: list[dict]):
    logger = get_run_logger()
    if not products:
        return
    conn = await asyncpg.connect(DB_DSN)
    try:
        await conn.executemany(
            """INSERT INTO products (title, price, url)
               VALUES ($1, $2, $3)
               ON CONFLICT (url) DO UPDATE SET
                 title = EXCLUDED.title,
                 price = EXCLUDED.price""",
            [(p["title"], p["price"], p["url"]) for p in products],
        )
        logger.info(f"Stored {len(products)} products")
    finally:
        await conn.close()

@flow(log_prints=True)
async def scrape_pipeline(urls: list[str], concurrency: int = 5):
    semaphore = asyncio.Semaphore(concurrency)

    async def process_one(url: str):
        async with semaphore:
            try:
                result = await fetch_url(url)
                products = parse_products(result["html"])
                await store_products(products)
                return {"url": url, "ok": True, "count": len(products)}
            except Exception as e:
                return {"url": url, "ok": False, "error": str(e)}

    results = await asyncio.gather(*[process_one(url) for url in urls])

    success = [r for r in results if r["ok"]]
    failed = [r for r in results if not r["ok"]]

    print(f"Scraped {len(success)} successfully, {len(failed)} failed")
    return {"success": len(success), "failed": len(failed)}

if __name__ == "__main__":
    URLS = [f"https://example.com/products?page={i}" for i in range(1, 21)]
    asyncio.run(scrape_pipeline(URLS))

Deploy with cron:

if __name__ == "__main__":
    URLS = [f"https://example.com/products?page={i}" for i in range(1, 21)]
    scrape_pipeline.serve(
        name="hourly-products",
        cron="0 * * * *",
        parameters={"urls": URLS, "concurrency": 5},
        work_pool_name="my-pool",
    )

Comparison: Prefect vs other orchestrators

feature	Prefect 3	Dagster	Airflow 2.x
Python-native	yes	yes	DAG syntax
Async support	excellent	good	poor
Setup complexity	low	medium	high
Observability	great dashboard	great dashboard	OK dashboard
Hosted option	Prefect Cloud	Dagster Cloud	many vendors
Maturity	very good	very good	excellent
Best for	mid-complexity workflows	data engineering	enterprise complexity

For scraping pipelines specifically, Prefect’s async support and simpler API give it an edge. Dagster’s data-engineering features (asset-based, type system, integrations) are overkill for pure scraping but useful when you mix scraping with downstream ETL.

Deployment patterns

Production Prefect 3 deployments:

• Prefect Cloud + workers on EC2/GCE: hosted control plane, your compute
• Self-hosted Prefect server + Kubernetes workers: full self-host
• Prefect Cloud + serverless workers (ECS, Lambda): pay-per-use compute
• All-in-one VM: server + worker on same machine for small teams

For containerized workers:

FROM python:3.12-slim

RUN pip install prefect playwright httpx selectolax asyncpg
RUN playwright install chromium

COPY . /app
WORKDIR /app

CMD ["prefect", "worker", "start", "--pool", "my-scraper-pool"]

Run multiple worker containers behind the same pool for horizontal scaling.

Operational checklist

For production Prefect 3 scraping in 2026:

• Prefect 3.x with Python 3.11+
• Prefect Cloud or self-hosted server with PostgreSQL
• Work pools for environment isolation
• Multiple workers for horizontal scale
• Configure retries with exponential backoff
• Use task caching for development iteration
• Async tasks for network-bound work
• Browser scraping in batches per task
• External storage (Postgres, S3) for results
• Monitor flow run success rate, task retry counts
• Alert on flows that have not completed within SLA

For broader pipeline patterns, see building scraping pipelines with Dagster and distributed scraping with Apache Kafka.

Common pitfalls

• Sync code in async flows: blocks the event loop. Use async libraries throughout or wrap sync code in asyncio.to_thread.
• Too many tasks per flow: 1000+ tasks creates dashboard noise and overhead. Batch URLs into chunks per task.
• Browser launch per task: expensive. Batch URLs through one browser instance.
• Long-running tasks: Prefect’s task heartbeat timeout is configurable but default is short. For tasks running >5 minutes, set explicit timeout.
• Not using work pools: running flows directly works locally but does not scale. Move to work pools early.

FAQ

Q: do I need Prefect Cloud or can I self-host?
You can self-host the server. Prefect Cloud adds managed control plane, alerting, and UI hosting, but the OSS server has the full feature set. For most teams under 10 users, self-hosting is fine.

Q: how does Prefect 3 differ from Prefect 2?
Faster task execution, simpler async, work pools (replaces work queues + agents from v2), better dashboard. The migration from v2 to v3 is straightforward for typical flows.

Q: can I use Prefect with Scrapy?
Yes. Wrap Scrapy spider invocations as Prefect tasks. The orchestration value is in scheduling, retries, and downstream processing. See our Scrapy + Playwright integration writeup for Scrapy-specific patterns.

Q: is Prefect overkill for a single scraper?
Yes. For a single cron-scheduled scraper, just use cron. Prefect pays off when you have multiple scrapers, retries, dependencies between flows, and want observability across all of them.

Q: how does cost compare?
Self-hosted: just compute cost, $50-200/month for a small team. Prefect Cloud: usage-based, starts at $0 (free tier), grows with task runs. For most teams under 100k task runs/month, Prefect Cloud is cheaper than DIY observability tooling.

Common pitfalls in production Prefect 3 scraping

The first failure mode is the Postgres backend overload from high-frequency task runs. Prefect 3 stores every task run state transition in its backend Postgres database. A scraping flow that processes 10,000 URLs as 10,000 individual tasks generates roughly 60,000 state-transition rows per flow run (PENDING, RUNNING, COMPLETED, plus retries). Daily flows produce 1.8M rows per month, and the backend’s task_run and task_run_state tables balloon to tens of GB within a quarter. Queries slow down, the UI stalls, and the worker heartbeat misses cause spurious flow failures. The fix is twofold: enable PREFECT_API_DATABASE_TIMEOUT=30 to fail fast on slow queries rather than hanging, and run a weekly cleanup job that deletes flow run records older than 30 days:

from prefect import flow
from prefect.client.orchestration import get_client
from datetime import datetime, timedelta

@flow
async def cleanup_old_runs():
    cutoff = datetime.utcnow() - timedelta(days=30)
    async with get_client() as client:
        flow_runs = await client.read_flow_runs(
            flow_run_filter={"end_time": {"before_": cutoff}},
            limit=1000,
        )
        for fr in flow_runs:
            await client.delete_flow_run(fr.id)

Schedule this nightly. Without it, expect to manually TRUNCATE the task_run table every few months once it crosses 50GB.

The second pitfall is async semaphore leakage across task retries. Prefect’s @task(retries=3) decorator retries a failed task by re-running it as a new attempt within the same flow context. If your task uses an asyncio.Semaphore to bound concurrency, the semaphore is acquired in attempt 1, the task fails, attempt 2 runs but the semaphore was never released because the failure path skipped the release. After three failed attempts on five tasks, your concurrency limit is permanently reduced by 15 slots. The fix is to use try/finally around every semaphore acquire, or better, use async with semaphore: which guarantees release even on exception:

async def process_one(url: str):
    async with semaphore:  # guaranteed release
        result = await fetch_url(url)
        return parse(result)

The third pitfall is the work pool concurrency limit being misinterpreted. The concurrency_limit on a work pool caps the number of flow runs executing simultaneously, not the number of tasks within those flows. A work pool with concurrency_limit=5 and a flow that runs 100 tasks in parallel will happily run 5 flows × 100 tasks = 500 simultaneous tasks. If your worker box has 8 CPUs and 16GB RAM, this oversubscription causes thrashing. Set both flow-level and task-level concurrency: cap the work pool at concurrency_limit=N AND cap async-gather inside each flow with a semaphore at a value that keeps total simultaneous tasks within hardware capacity.

Real-world example: 200-site daily scraping pipeline

A scraping team built a Prefect 3 pipeline that scraped 200 ecommerce sites daily, processing roughly 4 million product pages per day. The architecture used three flow types: a discovery flow that enumerated category URLs per site, a scrape flow that fetched and parsed product pages, and a load flow that wrote results to the data warehouse:

from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta
import asyncio
import httpx

@task(retries=2, retry_delay_seconds=30)
async def discover_category_urls(site_config: dict) -> list[str]:
    async with httpx.AsyncClient(timeout=20) as client:
        resp = await client.get(site_config["sitemap_url"])
        resp.raise_for_status()
        return parse_sitemap_for_categories(resp.text, site_config["url_pattern"])

@task(retries=3, retry_delay_seconds=[10, 30, 60])
async def scrape_product_batch(urls: list[str], site_config: dict) -> list[dict]:
    semaphore = asyncio.Semaphore(site_config.get("concurrency", 5))
    async def fetch_one(url):
        async with semaphore:
            try:
                async with httpx.AsyncClient(timeout=15) as client:
                    resp = await client.get(url, headers=site_config["headers"])
                    if resp.status_code == 200:
                        return parse_product(resp.text, site_config)
            except Exception:
                return None
    results = await asyncio.gather(*[fetch_one(u) for u in urls])
    return [r for r in results if r]

@flow(name="site-scrape", log_prints=True)
async def scrape_one_site(site_config: dict):
    category_urls = await discover_category_urls(site_config)
    # Batch into chunks of 50 URLs per task
    batches = [category_urls[i:i+50] for i in range(0, len(category_urls), 50)]
    batch_results = await asyncio.gather(
        *[scrape_product_batch(b, site_config) for b in batches]
    )
    all_products = [p for batch in batch_results for p in batch]
    print(f"site={site_config['name']} products={len(all_products)}")
    await load_to_warehouse(all_products, site_config["name"])

@flow(name="all-sites-daily")
async def scrape_all_sites():
    site_configs = await load_site_configs()
    # Run sites in parallel but cap at 20 simultaneous sites
    semaphore = asyncio.Semaphore(20)
    async def run_site(cfg):
        async with semaphore:
            return await scrape_one_site(cfg)
    await asyncio.gather(*[run_site(cfg) for cfg in site_configs])

Operational metrics over six months in production:

• Average daily flow duration: 4.2 hours
• p99 task duration: 18 seconds
• Task retry rate: 3.1 percent (mostly network blips)
• Daily backend Postgres growth: 2.8 GB
• Worker count: 4 (8 vCPU, 16GB each)
• Backend Postgres: db.r6g.xlarge ($420/month)
• Total Prefect-side infrastructure cost: $1,680/month

The team initially ran the discovery and scrape flows together but split them into separate flows after observing that discovery failures (sitemap parsing errors) blocked entire site processing. With separate flows, a discovery failure leaves yesterday’s category URLs as the working set for scraping, gracefully degrading rather than blocking. The lesson: orchestrator design choices about flow boundaries directly affect failure isolation.

Comparison: Prefect 3 task patterns by use case

A reference table for choosing the right Prefect 3 pattern per scraping scenario:

use case	task pattern	concurrency	retry strategy
sitemap discovery	single task per site	20 sites in parallel	2 retries, 30s delay
product scraping	batch task (50 URLs each)	5 batches per site	3 retries, exponential backoff
browser scraping	batch task (10 URLs per browser)	3 browsers per site	2 retries, 60s delay
API enrichment	single task per record	50 records in parallel	5 retries, jittered exponential
warehouse load	single task per batch (1000 rows)	1 sequential per site	5 retries, 5min delay
email notification	terminal task	1	no retry
screenshot capture	batch task (5 URLs per browser)	2 per site	1 retry
schema validation	sync task with to_thread	inline	no retry

For browser-heavy work, prefer fewer batches with more URLs per batch to amortize browser launch cost. For HTTP-heavy work, prefer smaller batches with more parallelism to keep latency low.

Wrapping up

Prefect 3 hits the right balance for scraping pipelines: enough orchestration for retries, scheduling, and observability without the heavy ceremony of Airflow. The async support is a real differentiator for I/O-bound scraping. Pair this with our building scraping pipelines with Dagster and Scrapy + Playwright integration writeups for the full pipeline picture, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.

Scrapy Cloud vs Crawlee Cloud in 2026

Scrapy Cloud vs Crawlee Cloud is the comparison every Python or Node scraping team faces eventually. Both platforms come from Apify (which acquired Scrapinghub’s Scrapy Cloud business in 2021 and integrated Crawlee, its own Node-first scraping framework, into the same hosted offering). They share underlying infrastructure but differ in framework, language ecosystem, pricing model, and the tooling they expose. Picking the right one is mostly about which framework your team standardizes on and which language has better library coverage for your targets.

This guide compares both platforms feature by feature in 2026, prices the realistic cost at common workloads, and walks through deployment for each. The benchmarks come from actually running production scrapers on both during 2025-2026 with similar workloads. By the end you will know which platform fits your project, what the real costs are, and how to deploy without surprises.

What each platform is

Scrapy Cloud is Apify’s hosted platform for Scrapy spiders. You write Scrapy in Python, use shub deploy to push to the platform, and pay for compute units (run-time) and data items processed. Has a long history (since 2010), used by teams at large enterprises, and integrates with the Scrapy ecosystem (item pipelines, middleware, schedulers).

Crawlee Cloud is Apify’s hosted platform for Crawlee, the JavaScript/TypeScript-first scraping framework that Apify itself maintains. You write Crawlee in Node or TypeScript, deploy via the Apify CLI, and pay for actor runs (compute time) and storage. Newer (Crawlee released 2022, hosted version stabilized 2024) but actively developed.

Both run on the same Apify cloud underneath, so deployment, scheduling, monitoring, and storage primitives are similar. The main difference is the framework you write in.

For Apify’s official platform docs, see docs.apify.com.

Pricing comparison

As of mid-2026:

dimension	Scrapy Cloud	Crawlee Cloud
compute cost	$0.40/CU/hr	$0.40/CU/hr (same)
storage cost	$0.20/GB/month	$0.20/GB/month (same)
dataset reads	$0.02/1k records	$0.02/1k records (same)
starter plan	$39/month (1 CU)	$39/month (1 CU)
free tier	$5 credit/month	$5 credit/month

Pricing is identical because they share infrastructure. The cost differences come from runtime efficiency, which depends on your code:

workload	Scrapy time	Crawlee time	cost diff
100k HTML pages, simple parse	8 hr	6 hr	Crawlee 25% cheaper
100k pages with Playwright	12 hr	10 hr	Crawlee 17% cheaper
10k pages with custom middleware	0.5 hr	0.6 hr	Scrapy slightly cheaper
1M pages, no JS	80 hr	60 hr	Crawlee 25% cheaper

Crawlee tends to run faster for browser-heavy workloads because its Playwright integration is more efficient. Scrapy can be faster for HTML-only workloads with heavy custom middleware where its async model shines.

For a 1M-page/month workload, expect ~$80-120/month on either platform.

Framework comparison

dimension	Scrapy	Crawlee
primary language	Python	TypeScript / JavaScript
years in production	since 2008	since 2022
async model	Twisted (legacy), asyncio (modern)	Node async/await native
browser integration	scrapy-playwright	built-in PlaywrightCrawler/PuppeteerCrawler
middleware system	very rich, mature	growing
deduplication	Request fingerprinter	RequestQueue with built-in dedup
pipelines	item pipelines (post-process)	dataset push (simpler)
spider lifecycle hooks	yes (signals)	yes (lifecycle handlers)
stats collection	Scrapy Stats	built-in metrics
community size	very large	growing
TypeScript support	n/a	native

Scrapy is the more mature framework with a richer middleware ecosystem. Crawlee is newer but designed from scratch for browser scraping, which gives it cleaner APIs for that use case.

For a Scrapy + Playwright deep dive, see Scrapy + Playwright integration in 2026.

Deploy: Scrapy Cloud

pip install shub
shub login  # paste API key from Apify console

# In your Scrapy project
shub deploy
# Select project from list

Wait a minute for the build, then run via the web UI or CLI:

shub schedule scraper-name
shub items <run-id>  # download items as JSON

scrapinghub.yml config:

projects:
  default: 12345
stacks:
  default: scrapy:2.11-py310
requirements:
  file: requirements.txt

Scrapy Cloud has a few stacks (Python versions and Scrapy versions). Pick the latest unless you have specific constraints.

Deploy: Crawlee Cloud

npm install -g apify-cli
apify login  # paste API key

# In your Crawlee project
apify push

The Apify CLI wraps the project as an “actor” (Apify’s term for a deployable scraping unit) and pushes it to the platform.

actor.json config:

{
  "actorSpecification": 1,
  "name": "my-crawler",
  "version": "0.0",
  "buildTag": "latest",
  "input": {
    "title": "Crawler Input",
    "type": "object",
    "properties": {
      "startUrls": {
        "title": "Start URLs",
        "type": "array",
        "editor": "requestListSources"
      }
    }
  }
}

Run via the web UI or CLI:

apify call my-crawler --input='{"startUrls":["https://example.com"]}'

Scrapy Cloud workflow

A typical Scrapy Cloud project:

# myspider.py
import scrapy

class MySpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/products"]

    custom_settings = {
        "DOWNLOAD_DELAY": 1,
        "CONCURRENT_REQUESTS": 16,
    }

    def parse(self, response):
        for product in response.css("article.product"):
            yield {
                "title": product.css("h2::text").get(),
                "price": product.css(".price::text").get(),
                "url": product.css("a::attr(href)").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

scrapy_cloud.yml:

project: 12345
stack: scrapy:2.11-py310
requirements:
  file: requirements.txt

The platform handles scheduling, logs, and item storage. Items are written to Scrapy Cloud’s Items API, which you can pull via REST or download as CSV/JSON.

Crawlee Cloud workflow

// src/main.ts
import { CheerioCrawler, Dataset } from "crawlee";

const crawler = new CheerioCrawler({
  async requestHandler({ request, $, log, enqueueLinks }) {
    log.info(`Processing ${request.url}`);

    const products = $("article.product").map((_, el) => ({
      title: $(el).find("h2").text().trim(),
      price: $(el).find(".price").text().trim(),
      url: $(el).find("a").attr("href"),
    })).get();

    await Dataset.pushData(products);

    await enqueueLinks({
      selector: "a.next",
      label: "PAGINATION",
    });
  },
  maxRequestsPerCrawl: 1000,
  maxConcurrency: 10,
});

await crawler.run(["https://example.com/products"]);

Crawlee’s Dataset is the cloud-side equivalent of Scrapy items. enqueueLinks is the simpler equivalent of response.follow.

Browser support

feature	Scrapy Cloud	Crawlee Cloud
Playwright	via scrapy-playwright	native PlaywrightCrawler
Puppeteer	not supported	native PuppeteerCrawler
Headless Chrome	yes	yes
Headless Firefox	yes	yes
Mobile emulation	manual config	built-in helpers
Browser pool / context reuse	manual	built-in
Stealth (patchright integration)	manual via custom middleware	recipe in docs

Crawlee has more out-of-the-box ergonomics for browser scraping. Scrapy gets you there but with more configuration.

Storage

Both platforms expose:

• Datasets: structured data from your scrapers (rows of items)
• Key-value stores: arbitrary blobs (HTML snapshots, screenshots, JSON configs)
• Request queues: URL queues for crawl coordination

Pricing is the same. APIs are very similar. For most purposes the storage layer is interchangeable.

# Scrapy: writing to a dataset
yield {"title": title, "price": price}

# Crawlee: writing to a dataset
await Dataset.pushData({title, price});

# Scrapy: reading items from a previous run via API
import requests
resp = requests.get(
    f"https://api.apify.com/v2/datasets/{dataset_id}/items",
    headers={"Authorization": f"Bearer {API_KEY}"},
)

// Crawlee: reading items via SDK
import { ApifyClient } from "apify-client";
const client = new ApifyClient({ token: API_KEY });
const items = await client.dataset(datasetId).listItems();

Scheduling

Both support cron-style scheduling via the Apify console or API.

Scrapy Cloud:

shub schedule myspider --frequency 'every 1 hour'

Crawlee Cloud / Apify:

// Via Apify SDK
import { ApifyClient } from "apify-client";
const client = new ApifyClient({ token: API_KEY });
await client.actor("my-actor").schedules.create({
  cronExpression: "0 * * * *",  // hourly
  timezone: "UTC",
});

For more complex orchestration (DAGs, dependencies between scrapers), use external schedulers like Prefect or Dagster. See building scraping pipelines with Prefect 3 and building scraping pipelines with Dagster.

Proxy support

Both platforms include Apify Proxy:

• Datacenter proxies: included in plans, per-GB billed beyond free tier
• Residential proxies: $8-12/GB
• Smart Proxy: rotating residential with automatic anti-bot evasion

Per-request proxy assignment in Scrapy:

import os
custom_settings = {
    "DOWNLOAD_DELAY": 1,
    "DOWNLOADER_MIDDLEWARES": {
        "scrapinghub_proxy.ScrapinghubProxyMiddleware": 410,
    },
    "PROXY_GROUPS": ["RESIDENTIAL"],
}

In Crawlee:

import { CheerioCrawler, ProxyConfiguration } from "crawlee";

const proxyConfiguration = new ProxyConfiguration({
  groups: ["RESIDENTIAL"],
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  // ...
});

Apify Proxy is convenient but more expensive than dedicated providers. For high volume, route through Bright Data, Oxylabs, or self-hosted instead. See best residential proxy providers 2026.

When to choose Scrapy Cloud

Pick Scrapy Cloud when:

• Your team is Python-first and already uses Scrapy
• You have existing Scrapy spiders to migrate
• You need rich middleware (cookies, retries, custom headers per host)
• You want item pipelines for post-processing (validation, dedup, DB write)
• Long-running spiders that benefit from Twisted/asyncio-style concurrency
• Heavy HTML parsing where Scrapy’s async model is faster

When to choose Crawlee Cloud

Pick Crawlee Cloud when:

• Your team is JavaScript/TypeScript-first
• Your scraping is browser-heavy (Crawlee’s Playwright integration is cleaner)
• You want simpler APIs (less to learn than Scrapy)
• You want native TypeScript support
• You operate Apify actors for other use cases already

When to use neither

Pick self-hosted when:

• You have ops capacity and want to avoid platform lock-in
• Volume is high enough that hosted costs dominate compute (5M+ pages/month)
• You need custom infrastructure (specific GPU, specific OS, edge deployment)
• You want full control of proxy egress

For self-hosted patterns, see self-hosted proxy infrastructure and building scraping pipelines with Prefect 3.

Comparison: hosted vs self-hosted

dimension	hosted (Apify)	self-hosted
ops burden	none	high
time to first run	minutes	days
cost at low volume	low	high (fixed VM cost)
cost at high volume	medium	low
scaling	automatic	manual
compliance	Apify’s audits	yours
customization	limited	unlimited
break-even	~5M pages/month

For most teams under 5M pages/month, hosted is cheaper and faster. Above that, the calculation shifts toward self-hosted.

Migration: Scrapy → Crawlee or vice versa

If you decide to switch frameworks, the migration involves rewriting spiders. Scrapy items map to Crawlee dataset records, Scrapy middleware maps to Crawlee request handlers, but the syntax is entirely different. Plan a 2-4 week project for a moderate-sized spider suite.

Common migration concerns:

• Custom middleware: rewrite as Crawlee request hooks or pre-navigation hooks
• Item pipelines: move post-processing into the request handler or into a downstream Apify actor
• Custom selectors: cheerio (Crawlee) is similar to BeautifulSoup but not identical
• Stats and monitoring: Apify SDK provides equivalent metrics

Operational checklist

For deciding between platforms in 2026:

• Match framework to team language (Python → Scrapy, Node → Crawlee)
• Estimate monthly compute cost based on test runs
• Verify proxy bandwidth budget separately
• Test deployment workflow with a real spider
• Set up scheduling for your most frequent jobs
• Use external storage (S3, Postgres) for long-term data, not Apify datasets
• Monitor compute unit consumption weekly
• Plan migration to self-hosted at the volume break-even

FAQ

Q: can I run both Scrapy and Crawlee on the same Apify account?
Yes. They are different actor types but share the same compute pool, storage, and billing.

Q: which is faster for the same workload?
Crawlee tends to be 15-25% faster for browser-heavy workloads, Scrapy can be faster for pure HTML scraping with heavy middleware. Both are within the same cost ballpark.

Q: does Crawlee support Python?
A Python port of Crawlee exists (“crawlee-python”) but is less mature than the TypeScript original. For Python in 2026, Scrapy is still the more mature choice.

Q: can I use Apify Proxy with my own scraper code?
Yes. Apify Proxy is exposed as standard HTTP/HTTPS endpoints with credentials. Any scraper that supports HTTP proxies can use it.

Q: what about open-source self-hosted Apify (Apify Open Source)?
Apify open-sourced parts of the platform around 2024. You can run a limited subset on your own infrastructure but not the full hosted experience. For most teams, the hosted offering is more practical until volume justifies the build.

Common pitfalls in production hosted scraping

The first failure mode that catches teams migrating from self-hosted is compute unit (CU) billing surprise. Apify’s CU pricing meters not just CPU time but also memory-time-product. A spider configured with 4GB RAM that runs for 10 minutes consumes more CU than a spider configured with 1GB RAM that runs for 30 minutes, even though wall-clock memory usage may be similar. The fix is to right-size memory allocation per spider via the actor.json config. Default to 1GB and only bump if you observe OOM kills in the logs. The CU savings from halving memory often outweigh the marginal latency increase from tighter memory pressure.

The second pitfall is dataset row limit drift on the free tier. Apify’s free tier caps datasets at 10,000 records per dataset, which scrapers commonly hit during development without realizing. The dataset.pushData() call returns success but the underlying storage rejects rows beyond the limit, leading to silent data loss. Test runs with 5K records succeed, production runs with 50K records lose 80 percent of rows. The fix is to either upgrade off the free tier (the Personal plan at $49/month removes the limit) or to chunk results into multiple datasets via Dataset.open(name) with a rolling name like ${date}-${shard}.

The third pitfall is Apify Proxy IP exhaustion under per-host rate limits. Apify Proxy’s residential pool serves all customers from a shared IP set. If you hit a target site at 50 requests/second, Apify’s pool may rotate through 200 IPs in an hour, and the target’s per-IP rate limiter starts returning 429s on previously-fresh IPs because other Apify customers also hit them earlier in the day. The mitigation for high-volume targets is to use a dedicated residential pool (Apify offers it at higher per-GB cost) or to bypass Apify Proxy entirely and route through your own residential provider via the proxyUrl field on individual requests.

Real-world example: cost analysis migration from Apify to self-hosted

A scraper team running 8 million product pages per month across 40 ecommerce sites tracked their costs on Apify Scrapy Cloud over six months and decided to migrate to self-hosted in month seven. The detailed cost breakdown that drove the decision:

month	apify CU	proxy GB	apify total	self-host equivalent
jan	4,200	320	$2,840	$1,100
feb	4,800	380	$3,260	$1,100
mar	5,100	410	$3,510	$1,100
apr	5,400	450	$3,790	$1,150
may	6,200	520	$4,420	$1,200
jun	7,100	600	$5,080	$1,250

The self-host equivalent included: a 16-vCPU VM ($380/month), 50 residential IPs from a budget provider at $4/GB ($200-2400/month based on usage), Postgres for results ($120/month), Prometheus + Grafana monitoring ($80/month), and one full-time engineer maintaining the stack at 0.2 FTE (allocated cost $400/month).

Migration took 8 weeks and consumed $32,000 in engineering time. After migration, monthly cost stabilized at $1,250 versus the projected Apify cost of $5,800+ by month seven. Payback period: 7.5 months from migration completion. The team retained Apify Scrapy Cloud for two specific spiders that benefited from Apify’s anti-detection-tuned proxy pool (those spiders accounted for 8 percent of total volume), so they did not fully zero their Apify bill.

The lessons: Apify is meaningfully cheaper than self-hosted up to roughly 3-5 million pages/month. Above that, self-hosted wins decisively. Hybrid (most volume self-hosted, anti-bot-heavy targets on Apify) is a viable middle ground that captures both economies.

Comparison: Scrapy Cloud vs Crawlee Cloud feature parity matrix

A 2026-current feature comparison across the two platforms on Apify’s shared infrastructure:

feature	Scrapy Cloud	Crawlee Cloud
max concurrent runs	32 (Personal), unlimited (Team+)	32 (Personal), unlimited (Team+)
max RAM per actor	32GB	32GB
storage retention	7 days (free), 30 days (paid)	7 days (free), 30 days (paid)
scheduled runs	yes	yes
webhook callbacks	yes	yes
dataset format	JSON, CSV, XML, Excel	JSON, CSV, XML, Excel
key-value store	yes	yes
live console	yes	yes
log retention	14 days	14 days
proxy: datacenter	included	included
proxy: residential	$8-12/GB	$8-12/GB
browser support	Playwright via scrapy-playwright	native Playwright + Puppeteer
TypeScript support	no (Python only)	yes (first-class)
AI agent integration	manual	Apify Agents native (2025+)
Webhooks-as-trigger	yes	yes
Apify SDK access	yes (Python)	yes (Node/Bun)
Container deploy	yes (custom Dockerfile)	yes (custom Dockerfile)

The platforms are at near-feature-parity in 2026. The choice between them is almost entirely about your team’s language preference. The exception: Apify Agents (their AI agent framework launched 2025) is Node-first and integrates more cleanly with Crawlee. Teams building AI-driven scrapers tend to lean toward Crawlee for that integration.

Detection: when hosted is the wrong choice

Five signals that your scraping workload should NOT live on Apify:

1. Custom TLS fingerprinting required: Apify’s HTTP egress uses their pool’s fingerprint; you cannot inject curl_cffi or tls-client at the network layer. Move to self-hosted.
2. Hardware GPU for inference inside the spider: Apify does not provide GPU instances. Use Modal, RunPod, or self-hosted with GPUs.
3. Compliance requires data residency: Apify operates in EU and US regions only. If your data must stay in Singapore, Brazil, or other regions, self-host or pick a regional provider.
4. Real-time webhooks under 500ms p99: Apify actor cold starts can exceed 5 seconds. For real-time response APIs, run a long-lived service on Cloud Run or Fargate instead.
5. Custom kernel-level networking: Apify runs in a managed container environment; you cannot install custom iptables rules, custom DNS, or kernel modules. Self-host on a VM if you need this.

If any two of these apply, the answer is self-hosted regardless of volume.

Wrapping up

Scrapy Cloud vs Crawlee Cloud is mostly a framework choice in 2026. Same underlying infrastructure, same pricing, similar features. Pick Scrapy Cloud if your team is Python-first; pick Crawlee Cloud if your team is Node-first. Both pay off at low to medium volumes versus self-hosting, then break even and lose to self-hosted around 5M+ pages/month. Pair this with our Scrapy + Playwright integration and self-hosted proxy infrastructure writeups for the full hosted-vs-self-hosted picture, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.

Related comparison: See how Bright Data stacks up against a dedicated Singapore mobile network in our Singapore Mobile Proxy vs Bright Data comparison.

Distributed scraping with Apache Kafka in 2026

Distributed scraping with Apache Kafka is what you reach for when a single Scrapy process or single-machine Playwright cluster cannot keep up with your URL volume. Kafka excels at moving high-volume work between producers (URL discovery, sitemap crawlers) and consumers (scraper workers) with durable messaging, partitioning for parallelism, and built-in retry semantics. By 2026 Kafka 3.x is mature enough that running it in production for scraping pipelines is standard practice, especially for teams already operating Kafka for other data flows.

This guide covers Kafka topic design for scraping pipelines, partition strategies that keep scrapers fed, dead letter queue patterns for failed URLs, exactly-once semantics for deduplicated processing, and a complete working Python implementation using confluent-kafka. The benchmarks reflect production deployments handling 10-50 million URLs per day. By the end you will know how to architect a Kafka-backed scraping pipeline that scales horizontally and survives operational chaos.

When Kafka fits scraping

Kafka shines when:

• You have multiple URL sources (sitemaps, APIs, customer uploads, periodic crawls) feeding scrapers
• Scraper workers run on different machines and you want to balance load
• You need to retry failed URLs without losing them
• You want to fan out the same URLs to multiple downstream processors (HTML store, ML extraction, analytics)
• You operate other Kafka pipelines and want consistency
• You need exactly-once processing semantics

Kafka does not fit when:

• You have low volume (under 1M URLs/day, simpler queues like Redis or SQS suffice)
• You operate one scraping job at a time (no need for the multi-producer/multi-consumer architecture)
• Your team has no Kafka operational experience
• You need request/response semantics (Kafka is one-way; for scraper status, use a separate sync API)

For most mid-volume scraping, simpler queues work. Kafka pays off above several million URLs per day or when fanout to multiple consumers is required.

For the official Kafka documentation, see kafka.apache.org/documentation/.

Architecture overview

A typical Kafka-backed scraping pipeline:

   URL sources                     Scraper workers           Storage / downstream
   +----------+                    +-------------+           +-------------+
   | sitemap  | --+                |             | --+       | postgres    |
   | crawler  |   |                | worker_1    |   |       +-------------+
   +----------+   |                |             |   |
                  |  +-------+     +-------------+   |       +-------------+
   +----------+   +->| urls  |---->|             |   +------>| s3 / r2     |
   | API feed |  +-->| topic |     | worker_2    |   |       +-------------+
   +----------+   |  +-------+     |             |   |
                  |                +-------------+   |       +-------------+
   +----------+   |                |             |   +------>| ML pipeline |
   | customer |--+                 | worker_N    |           +-------------+
   | upload   |                    +-------------+
   +----------+
                                                      retries: dlq topic

Each producer pushes URLs to the urls topic. Multiple scraper workers consume from partitions of that topic in parallel. Successful results go to a pages topic that downstream consumers pick up. Failed URLs go to a dlq topic for retry or manual review.

Topic design for scraping

The standard set of topics:

topic	purpose	partitions	retention
urls	URLs to scrape	50	7 days
pages	scraped page content	20	14 days
extractions	parsed structured data	20	30 days
errors	scraper errors	5	7 days
dlq	failed URLs for retry	5	30 days
metrics	per-URL timing/status	10	1 day

Partitioning matters because Kafka guarantees ordered processing only within a partition. For scraping, you usually want:

• By host: partition = hash(url.host) % num_partitions. Keeps all URLs for one host on one partition, enables per-host rate limiting.
• By customer: partition = hash(customer_id) % num_partitions. Tenant isolation, billing per customer.
• Random: any partition. Maximum parallelism, no host-level coordination.

For scraping with anti-bot vendors, by-host partitioning is the right default. It lets each consumer apply per-host throttling without coordinating with other consumers.

Setup: Kafka 3.x with KRaft

Kafka 3.5+ removed the ZooKeeper dependency. KRaft (Kafka Raft) is now the recommended setup. For local dev:

# docker-compose.yml
version: "3.8"
services:
  kafka:
    image: bitnami/kafka:3.7
    ports:
      - "9092:9092"
    environment:
      KAFKA_CFG_NODE_ID: 1
      KAFKA_CFG_PROCESS_ROLES: controller,broker
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT

For production, use a managed service (Confluent Cloud, AWS MSK) or self-host on at least three nodes with proper replication.

Create the topics:

kafka-topics.sh --create --bootstrap-server localhost:9092 \
    --topic urls --partitions 50 --replication-factor 1 \
    --config retention.ms=604800000

kafka-topics.sh --create --bootstrap-server localhost:9092 \
    --topic pages --partitions 20 --replication-factor 1 \
    --config retention.ms=1209600000

kafka-topics.sh --create --bootstrap-server localhost:9092 \
    --topic dlq --partitions 5 --replication-factor 1 \
    --config retention.ms=2592000000

In production, use replication-factor 3 for durability.

Producer: feeding URLs

A simple Python producer using confluent-kafka:

# producer.py
from confluent_kafka import Producer
import json
import hashlib

producer = Producer({
    "bootstrap.servers": "kafka:9092",
    "client.id": "url-discovery",
    "acks": "all",
    "compression.type": "lz4",
    "retries": 5,
    "linger.ms": 10,
})

def url_key(url: str) -> bytes:
    """Partition by host so per-host throttling works in consumers."""
    from urllib.parse import urlparse
    host = urlparse(url).hostname or ""
    return host.encode()

def produce_url(url: str, customer_id: str = "default", priority: int = 0):
    payload = {
        "url": url,
        "customer_id": customer_id,
        "priority": priority,
        "discovered_at": int(time.time()),
    }
    producer.produce(
        topic="urls",
        key=url_key(url),
        value=json.dumps(payload).encode(),
        callback=lambda err, msg: (
            print(f"Failed: {err}") if err else None
        ),
    )

if __name__ == "__main__":
    import sys, time
    with open(sys.argv[1]) as f:
        for line in f:
            url = line.strip()
            if url:
                produce_url(url)
    producer.flush(timeout=30)

Key choices:

• acks="all": wait for all in-sync replicas to acknowledge. Durable but slower.
• compression.type="lz4": roughly 60-80% size reduction with low CPU overhead.
• linger.ms=10: batch up to 10ms before sending. Reduces request count.
• key=hostname.encode(): partitions by host.

For very high throughput, use acks=1 (only leader) and accept slightly higher loss risk in exchange for 2-3x speed.

Consumer: the scraper worker

A consumer that fetches URLs and produces results:

# scraper_worker.py
from confluent_kafka import Consumer, Producer, KafkaError
import json
import requests
import time
import logging
from collections import defaultdict

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "scrapers",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
    "max.poll.interval.ms": 600000,  # 10 min
    "session.timeout.ms": 30000,
})
consumer.subscribe(["urls"])

producer = Producer({
    "bootstrap.servers": "kafka:9092",
    "compression.type": "lz4",
    "acks": "all",
})

# Per-host last-fetch timestamps for throttling
last_fetch = defaultdict(float)
HOST_DELAY = 1.5  # seconds between same-host requests

def scrape(url: str) -> dict:
    from urllib.parse import urlparse
    host = urlparse(url).hostname

    # Per-host throttle
    elapsed = time.time() - last_fetch[host]
    if elapsed < HOST_DELAY:
        time.sleep(HOST_DELAY - elapsed)
    last_fetch[host] = time.time()

    resp = requests.get(url, timeout=30, headers={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...",
    })
    return {
        "url": url,
        "status": resp.status_code,
        "html": resp.text if resp.status_code == 200 else None,
        "fetched_at": int(time.time()),
    }

def send_to_dlq(message_value: bytes, error: str):
    producer.produce(
        topic="dlq",
        key=message_value,
        value=json.dumps({
            "original": json.loads(message_value),
            "error": error,
            "failed_at": int(time.time()),
        }).encode(),
    )

while True:
    msg = consumer.poll(timeout=1.0)
    if msg is None:
        continue
    if msg.error():
        if msg.error().code() == KafkaError._PARTITION_EOF:
            continue
        logger.error(f"Consumer error: {msg.error()}")
        continue

    payload = json.loads(msg.value())
    url = payload["url"]

    try:
        result = scrape(url)
        if result["status"] == 200:
            producer.produce(
                topic="pages",
                key=url.encode(),
                value=json.dumps(result).encode(),
            )
        elif result["status"] >= 500:
            send_to_dlq(msg.value(), f"HTTP {result['status']}")
        # else: 4xx errors are skipped (404, 403, etc.)

        consumer.commit(msg)  # commit offset only on success or known-skip
    except Exception as e:
        logger.error(f"Failed to scrape {url}: {e}")
        send_to_dlq(msg.value(), str(e))
        consumer.commit(msg)  # still commit, DLQ has the message

Key choices:

• enable.auto.commit=False: manual commit so we only advance after processing
• Per-host throttle via last_fetch dict
• DLQ for transient errors (5xx, exceptions)
• Skip 4xx errors (404, 403 are usually permanent for that URL)

Per-host rate limiting at scale

The simple per-process throttle above breaks when you have multiple consumers per partition. By partitioning on host, each host’s URLs all land on one partition, which is consumed by one consumer at a time within a consumer group. So the per-process throttle effectively becomes a per-host throttle across the cluster.

For finer control (multiple consumers, different rate limits per customer), use a distributed rate limiter (Redis-backed):

import redis

r = redis.Redis(host="redis", port=6379)

def can_fetch_host(host: str, max_per_sec: float = 1.0) -> bool:
    """Token bucket via Redis."""
    key = f"rate:{host}"
    now = time.time()

    pipe = r.pipeline()
    pipe.zremrangebyscore(key, 0, now - 1)  # remove old entries
    pipe.zcard(key)
    pipe.zadd(key, {str(now): now})
    pipe.expire(key, 10)
    _, count, _, _ = pipe.execute()

    return count < max_per_sec

# In scraper:
while not can_fetch_host(host):
    time.sleep(0.1)

This gives global rate limiting independent of partition distribution.

Dead letter queue handling

DLQ patterns:

# dlq_consumer.py
from confluent_kafka import Consumer, Producer
import json
import time

consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "dlq-retry",
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["dlq"])

producer = Producer({"bootstrap.servers": "kafka:9092"})

while True:
    msg = consumer.poll(timeout=1.0)
    if msg is None:
        continue
    if msg.error():
        continue

    record = json.loads(msg.value())
    failed_at = record["failed_at"]
    age = time.time() - failed_at

    # Retry after 1 hour cooldown
    if age < 3600:
        continue

    # Re-emit to urls topic
    producer.produce(
        topic="urls",
        key=record["original"]["url"].encode(),
        value=json.dumps(record["original"]).encode(),
    )

    consumer.commit(msg)

Or for permanent failures: a dashboard query of the DLQ topic for ops review.

Exactly-once semantics

Kafka 3.x supports exactly-once semantics (EOS) via transactional producers and read_committed consumers. For scraping, this matters when you want each URL processed exactly once across worker restarts and failures.

producer = Producer({
    "bootstrap.servers": "kafka:9092",
    "transactional.id": f"scraper-{worker_id}",
    "enable.idempotence": True,
})
producer.init_transactions()

# In scraper loop:
producer.begin_transaction()
producer.produce(topic="pages", value=json.dumps(result).encode())
producer.send_offsets_to_transaction(
    [TopicPartition("urls", partition, offset)],
    consumer_group_metadata
)
producer.commit_transaction()

This guarantees that if the scraper crashes mid-process, the offset commit and the result production happen atomically: either both, or neither. The next consumer instance picks up from the same offset and processes the URL again.

Comparison: Kafka vs alternatives for scraping queues

platform	pros	cons	scale
Kafka	high throughput, durable, multi-consumer, EOS	complex ops, requires JVM	up to 10M+ msg/sec
Redis Streams	simple, fast, low overhead	single-node, less durable	up to 100k msg/sec
RabbitMQ	rich routing, mature	lower throughput than Kafka	up to 100k msg/sec
SQS	managed, cheap, infinite scale	per-message cost, no replay	up to millions/sec
NATS JetStream	simple, fast, durable	smaller community	up to 1M msg/sec
Pulsar	Kafka-alternative, geo-replication	smaller adoption	comparable to Kafka

For scraping, Kafka and Pulsar both work; Kafka has the larger community. SQS is often the simpler choice if your AWS bill is comfortable with per-message charges. For under 1M URLs/day, Redis Streams is hard to beat for simplicity.

Monitoring

Critical Kafka metrics for scraping:

metric	what it tells you
consumer lag (urls topic)	how far behind workers are
produce rate (urls)	URL discovery throughput
consume rate (urls)	scrape throughput
DLQ size	failure rate
rebalance count	worker churn
under-replicated partitions	broker issues

Tools:

• Kafka Exporter for Prometheus: scrapes Kafka metrics
• Conduktor or Confluent Control Center: GUI for cluster management
• Burrow: consumer lag monitoring with alerting
• Strimzi: Kubernetes operator with built-in monitoring

Set alerts on consumer lag > 30 min, DLQ size growth, broker downtime.

Production deployment

For 10M URLs/day:

• 3-broker Kafka cluster on Kafka 3.7 with KRaft
• 50 partitions on urls topic
• 20 scraper workers, each consuming from 2-3 partitions
• Per-host throttle via partition assignment + Redis backstop
• Postgres for results (loaded from pages topic by separate consumers)
• DLQ retry consumer running every hour
• Prometheus + Grafana for monitoring
• ~$1500/month in compute + Kafka

For 100M URLs/day:

• 5+ broker cluster with replication-factor 3
• 200+ partitions
• 100+ scraper workers
• Multiple Kafka clusters segmented by region
• ~$15k/month compute + Kafka

Operational checklist

For Kafka-backed scraping in 2026:

• Kafka 3.5+ with KRaft (no ZooKeeper)
• Topic design: urls, pages, dlq, errors, metrics
• Partition by host for per-host coordination
• LZ4 compression on all topics
• acks=all for producers (durability)
• Manual commit on consumers (process before commit)
• Per-host throttling (partition-level + Redis backstop)
• DLQ retry consumer
• Prometheus + Grafana monitoring
• Alerts on lag, DLQ growth, broker health
• Schema registry if pipeline complexity grows

For broader pipeline patterns, see building scraping pipelines with Prefect 3 and building scraping pipelines with Dagster.

Common pitfalls

• Partition imbalance: bad partition keys cause hot partitions. Audit URL distribution by partition.
• Slow consumers and consumer group rebalances: long-running scrapes blow past max.poll.interval.ms. Bump it or split work.
• Offset commit before processing: leads to data loss. Always commit after successful processing.
• DLQ overflow: if 30% of URLs fail, DLQ fills faster than retries clear it. Tune retry policy or fix underlying issues.
• Memory pressure on consumers: large pages (>1 MB HTML) accumulate in producer buffer. Cap message size or stream pages directly to S3.
• Kafka cluster upgrade missteps: KRaft migration requires careful planning. Read the Kafka KRaft docs.

FAQ

Q: do I need Kafka for under 1M URLs/day?
Probably not. Redis Streams or SQS will be simpler and sufficient. Kafka pays off above 1-10M URLs/day where the durability, throughput, and multi-consumer fanout matter.

Q: can I use Kafka for both URL queue and result storage?
Kafka is a queue, not a database. Use it for streaming and short-term retention. Store results in a database (Postgres, ClickHouse) or object storage (S3, R2) via downstream consumers.

Q: how do I retry failed URLs without losing the original message?
DLQ topic. Failed processing produces to DLQ, original offset commits. A retry consumer reads DLQ on a schedule and re-emits to the urls topic.

Q: what about Kafka Streams for in-flight processing?
Useful for derived data (aggregations, deduplication). For raw scraping, KafkaConsumer is enough. Add Streams when you need stateful enrichment.

Q: does this work with Confluent Cloud?
Yes. The Python client is identical, only the bootstrap.servers and SASL credentials change. Confluent Cloud removes the ops burden at the cost of usage-based pricing.

Real-world example: 80M URL crawl with KRaft and per-host fairness

A scraping team migrated from a single-broker Kafka cluster to a 5-broker KRaft cluster handling 80 million URLs per day across 12,000 distinct hosts. The bottleneck before migration was not throughput but fairness: a few hosts (Amazon, eBay, Walmart) made up 40 percent of all URLs and starved smaller hosts because their partitions had longer queue depth. The fix involved a custom partitioner that hashed by host AND by a “fairness band” derived from the host’s URL queue depth:

import hashlib
from confluent_kafka import Producer

class FairHostPartitioner:
    def __init__(self, num_partitions: int, queue_depth_provider):
        self.num_partitions = num_partitions
        self.queue_depth = queue_depth_provider  # callable(host) -> int

    def partition(self, key: bytes, all_partitions: list) -> int:
        host = key.decode().split("/")[2]
        depth = self.queue_depth(host)
        # High-depth hosts get spread across multiple partitions
        if depth > 100_000:
            band = hashlib.sha256(key).hexdigest()[:8]
            band_int = int(band, 16) % 4  # spread across 4 partitions
            base = int(hashlib.sha256(host.encode()).hexdigest()[:8], 16)
            return (base + band_int) % self.num_partitions
        # Low-depth hosts get one partition (per-host throttling preserved)
        return int(hashlib.sha256(host.encode()).hexdigest()[:8], 16) % self.num_partitions

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "compression.type": "lz4",
    "linger.ms": 10,
    "batch.size": 65536,
})

After deployment, p99 URL-to-fetch latency for low-volume hosts dropped from 4.2 hours to 18 minutes. High-volume hosts saw their throughput increase 3.5x because the work was now spread across 4 partitions instead of bottlenecking on one. The 5-broker cluster ran at 65 percent CPU during steady state, leaving headroom for daily peak loads.

Common pitfalls in production Kafka scraping

The first failure mode is consumer group rebalance storms during scraper deploys. When you redeploy 20 scraper workers via rolling restart, each worker’s exit triggers a rebalance, which pauses all consumers for 5-30 seconds while partitions reassign. With 20 workers redeploying sequentially, you accumulate 100-600 seconds of pause time during the deploy window. The fix is cooperative rebalancing (Kafka 2.4+) which only reassigns the partitions of the leaving worker rather than all partitions:

consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "scrapers",
    "partition.assignment.strategy": "cooperative-sticky",
    "session.timeout.ms": 30000,
    "max.poll.interval.ms": 600000,  # 10 min for slow scrapes
})

After enabling cooperative-sticky, deploy-induced pause time drops by roughly 90 percent because each worker exit only pauses its own 2-3 partitions instead of all 50.

The second pitfall is the __consumer_offsets topic bloat. Kafka stores consumer group offsets in an internal compacted topic. Scrapers that commit offsets aggressively (every message instead of every batch) generate millions of offset commits per day, which can outgrow the broker’s compaction capacity and cause the topic to balloon to tens of GB. The fix is to commit in batches of 100-1000 messages rather than per-message:

batch = []
for msg in consumer:
    process(msg)
    batch.append((msg.topic, msg.partition, msg.offset + 1))
    if len(batch) >= 100:
        consumer.commit(offsets=batch)
        batch = []

The third pitfall is the producer buffer exhaustion under back-pressure. When a broker becomes slow or unreachable, the producer’s buffer.memory (default 32MB) fills up with un-acked messages, and producer.send() calls block indefinitely once full. Scrapers that produce results while consuming URLs can deadlock: the producer blocks on a slow broker, the consumer can’t make progress because it can’t produce results, the consumer gets kicked from the group for missed heartbeats, and the entire pipeline halts. Configure delivery.timeout.ms and request.timeout.ms aggressively, and use producer.flush(timeout=10) with a timeout rather than indefinite blocking:

producer = Producer({
    "bootstrap.servers": "kafka:9092",
    "delivery.timeout.ms": 30000,
    "request.timeout.ms": 15000,
    "buffer.memory": 67108864,  # 64MB
    "max.block.ms": 5000,  # don't block produce() longer than 5s
})

If max.block.ms is exceeded, producer.produce() raises BufferError, which you handle by dropping the message to a local fallback file rather than crashing the consumer.

Wrapping up

Distributed scraping with Kafka pays off when volume justifies the operational complexity. For 10M+ URLs/day, the durability, partitioning, and multi-consumer architecture are hard to beat. For under 1M, simpler queues are usually the right answer. Pair this with our building scraping pipelines with Prefect 3 and Scrapy Cloud vs Crawlee Cloud writeups for the full pipeline picture, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.