A scrape that runs once is a script. A scrape that has to run every day for a year, without anyone watching, is infrastructure, and the layer that turns one into the other is scheduling. It decides when your scraper runs, what happens when it fails, and whether you find out the day it breaks or three weeks later from stale data.

I run production scrapers, so I have used every rung of this ladder, from a single cron line on one box to a full orchestrator managing dozens of jobs. Here is how I think about the three real options, cron, Airflow, and Dagster, and how to pick one without overbuilding.

What scheduling actually has to do

It sounds like one job, run this at 6am, but the real work is underneath. A good scheduler remembers whether the last run finished. It retries a failed step without redoing the ones that worked. It stops two runs from colliding when one runs long. It tells you when a job broke, and, harder, when a job succeeded but returned nothing. And it lets you go back and reprocess history when you fix a bug.

Cron does the first sentence. Everything after it is why the heavier tools exist.

Start with cron

Cron is the scheduler already living on every unix box you own. You give it a line, a time and a command, and it runs that command forever. No service to install, no database, no dependencies. For a single scrape that pulls one source once a day into one table, cron is not the lazy choice, it is the correct one.

The trouble is everything cron does not do. It has no memory, so a failed run vanishes and the next starts fresh. It has no retries, so one network blip loses you a day. It will start a new run while the last is still going, and now two copies fight over the same table. And its output goes nowhere you will look, so a broken job surfaces days later as stale data rather than an alert.

Make cron last a little longer

If you only have a job or two, you do not have to abandon cron, you shore it up:

• Wrap the command in a script that takes a lock so runs cannot overlap.
• Make that script exit non-zero and notify you when the work fails.
• Send output to a real log file, not the void.
• Add a heartbeat, an external service you ping only on a clean finish, so a run that never happens alerts you by its silence.

That last one matters most. The scariest failure is not the job that crashes loudly, it is the job that silently never fires.

When you have outgrown cron

Move up a rung when any of these is true: you have steps that depend on each other and want to retry just the one that broke, you need to reprocess last month because you fixed a parser, you are running more than a couple of jobs and cannot remember what runs when, or a second person needs to see what is happening without reading your crontab.

The tools above cron are built around a dag, a directed acyclic graph. That just means steps with an order, where each points to the ones that must finish first: fetch before parse, parse before store. Once the scheduler understands that shape, it can run steps in order, parallelize the independent ones, retry a single failed step, and skip what already succeeded.

Airflow: dags and backfills

Airflow is what most teams reach for first. You write your dag in python, Airflow runs it on a schedule, and you get a ui where every run is a row of green and red boxes. Each task has its own retries and delays, and when one fails you click in, read the log, and rerun just that piece. It has been doing this for years, so almost every database, api, and cloud service has a ready made connector.

Its real superpower is the backfill. Airflow treats each run as owning a slice of time, so when you fix a parser and need to rebuild the last 30 days, you tell it to run that date range and it reprocesses each day in order. That only works if your steps are idempotent, but when they are, repairing history is one command instead of a manual slog.

None of that is free. Airflow is a scheduler, a metadata database, workers, and a web server, several moving parts to install, secure, upgrade, and babysit. Standing all of that up to run one daily scrape is like buying a forklift to carry your groceries. It earns its keep when you have many jobs and several people depending on them.

Dagster and Prefect: data assets and observability

Dagster and Prefect came later with a different framing. Instead of thinking only about tasks that run, Dagster thinks about the data assets those tasks produce, the actual table or dataset you care about. You declare the asset and the code that builds it, and Dagster tracks what it depends on and how fresh it is.

That framing fits scraping well. You do not really care that a task ran, you care whether your products dataset is current and correct. Dagster lets you ask exactly that, shows the lineage behind each asset, and leans on typing so a run that returns the wrong shape gets caught instead of silently writing nulls. Prefect takes a lighter path, wrapping your existing python functions as flows with little ceremony, which suits irregular, dynamic pipelines. Both give you nicer local development and clearer visibility than the older model.

Tool	Best for	Gives you	Costs you
cron	one or two simple jobs	zero setup, runs anywhere	no memory, retries, or visibility
Airflow	scheduled dags and backfills	per-task retries, a run ui, huge connector ecosystem	a scheduler, database, workers, and web server to run
Dagster / Prefect	typed data assets, freshness, lineage	asset observability, typing, strong local dev	newer, another system to learn and host

Retries, idempotency, and the silent empty run

Three things matter no matter which tool you pick.

Put retries in the scheduler, not just the scrape. Your fetch function can back off on a single flaky request, but the scheduler retrying a whole failed task with a growing delay is what saves a run when a site has a bad five minutes.

Make every job idempotent. Running the same job twice must do no harm, or retries and backfills become a gamble. Key records on a stable id and write with an upsert that overwrites cleanly instead of appending.

And catch the silent empty run. The worst failure is not a crash, it is a job that finishes green with zero rows because the site changed a selector. Cron and even a naive Airflow setup call that a success. So count what you collected and compare it to normal, and if you usually pull thousands and today you got 12, fail the run on purpose. Alert on a whole run failing, a success rate falling off a cliff, zero rows where you expected thousands, and a run that never started at all.

How to choose

Match the tool to the scale and the team, and do not overbuild before you need to.

• Solo, one or two scrapes, no complex dependencies: cron with a lock and a heartbeat. Done.
• A handful of interdependent pipelines, backfills, or a small team that needs shared run history: Airflow, because it is proven and everyone can learn it.
• A data heavy setup where freshness, lineage, and typed outputs matter more than raw scheduling: Dagster, or Prefect when the work is irregular and pythonic.

The most common mistake is not too little tooling, it is standing up a whole orchestrator to run a scrape a single cron line handled fine. You can always climb the ladder when the pain is real.

The honest limits

Scheduling makes your collection reliable, not permitted. Running a scraper on time and recovering it cleanly does not change what you are allowed to take. Public data, a robots file respected, a gentle pace, and an official api or bulk feed preferred when one exists, those rules do not move because your orchestrator is nicer.

I run all three of these in production, cron for the small stuff and a real orchestrator for the pipelines that feed everything else. If you want the worked versions, the Airflow and Dagster setups, the idempotency patterns, and the alerting checks that catch an empty run, along with the tools I actually use, they are all at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

Most scrapers reach for a headless browser by default, and it is the most expensive habit in this work. They spin up a full Chromium for a page that would have handed the same data to a plain http request, then pay for that browser on every page, forever. The decision that matters isn’t which browser tool to use. It’s whether you need a browser at all, and most of the time you don’t.

I run production scrapers, so I’ve paid the browser bill when I didn’t have to and watched it show up in the monthly invoice. This is how I decide, where the real cost hides, and the hybrid I run so a job stays cheap where it can and only pays for a browser where a site forces one.

The one test that decides it

Open the page in your browser, view the raw source, and search that source for the thing you want to collect. If the price, the title, the number is already sitting in the html the server sent, you almost never need a browser. If the source is a near empty shell and the content only fills in after the scripts run, then you need something that executes those scripts.

That one look at the raw source settles the large majority of these calls, and it takes about ten seconds. Everything below is really just the consequences of getting that answer right or wrong.

What an http client does, and what it costs

An http client is the simple side. It opens a connection, sends a request, and hands you back exactly the bytes the server returned. It doesn’t run javascript, it doesn’t build a page, it doesn’t know what a button is. It’s just the raw response.

That’s why it’s so light. There’s no browser eating memory, no rendering, no waiting for scripts, so one modest machine can run hundreds of these requests at once. When the data lives in the server html, an http client is the whole job, and it’s fast and cheap. Libraries like requests and httpx do this, and a framework like Scrapy is built on the same idea at scale.

What a headless browser does, and what it costs

A headless browser is the heavy side. It launches a real Chromium engine with no window drawn on screen, loads the page the way a person’s browser would, runs all the javascript, builds the dom, and lets you read the page after it has fully rendered. Playwright, Puppeteer, and Selenium all drive a browser like this.

That full render is the entire reason it exists, and it’s the entire reason it costs so much. A raw http request costs a few milliseconds and a sliver of memory. A full browser page costs hundreds of megabytes of memory, real cpu, and seconds instead of milliseconds.

The gap in real numbers

The difference isn’t small. Per page, a browser routinely costs ten to fifty times the resources of a raw request, and that gap compounds at scale.

	http client	headless browser
Runs javascript	no	yes
Memory per page	a few mb	hundreds of mb
Time per page	milliseconds	seconds
Pages at once, one box	hundreds to thousands	a handful
Breaks when	html layout changes	browser updates, timing, memory

Put a job on it. A million pages with a raw http client on a modest server finishes in hours for a few dollars. The same million pages with a full browser each can need a fleet of machines and a bill many times larger, plus the constant work of keeping all those browsers healthy.

Check for a background api first

Before you assume a page needs a browser, open the developer tools and watch the network tab while the page loads. Very often the content you want arrives as a clean json response from an api the page quietly calls in the background.

When that’s happening, you don’t need to render anything. You call that json endpoint directly with a plain http request and skip the browser entirely. This one habit has saved me more compute than any tool choice, because the cheapest page to collect is the one you never render.

The hybrid: http first, browser only where forced

In practice it’s rarely all browser or all client. I run a fast http client across the whole site for the pages that hand over their data plainly, and I fall back to a browser only for the specific pages that come up empty. So the heavy cost lands on the handful of pages that genuinely need a render, and everything else stays cheap.

A browser really is the only door in a few cases: when the content is rendered on the client and never appears in the raw source, when the data hides behind an interaction like a click or an infinite scroll, or when the page is a single page app that builds itself entirely from javascript. In those cases an http client sees the empty shell and there’s nothing to parse. The rule I follow is simple: crawl wide and cheap with raw requests, render narrow and expensive only where the html arrives empty.

If you do land on a browser, the tool matters less than people think. Playwright and Puppeteer both drive Chromium cleanly, Selenium works but feels older, and I default to Playwright. The expensive part is never which one you pick, it’s the browser itself.

The maintenance burden of browsers at scale

The cost that doesn’t show up in a benchmark shows up at three in the morning. An http client is boring in the best way: it breaks when a site changes its html, and the fix is a quick selector update.

A browser breaks in far more ways. Chromium updates and a flag you relied on changes. A page gets slower and a timing assumption that held last month starts flaking. Memory leaks creep in over a long run until the process bloats and dies. Running browsers at scale becomes its own operations job: managing a pool of them, restarting them before they leak, killing the ones that hang, catching zombie processes, pinning a browser version, and wrapping every page in a timeout so it can’t hang forever. None of that exists on the http side.

There’s a subtler tax too. An http response either arrives or it doesn’t, so you know when you’re done. A browser page is never obviously finished, because scripts keep firing and content keeps arriving, so you write waits, and every wait is time you spend on every single page.

The detection angle, honestly

People ask which side is harder to detect, and it’s mostly the wrong question. Sites decide on the network you arrive from, the handshake your client sends, and the rhythm of your behavior, and both a client and a browser can be loud or quiet depending on how you run them. A real browser engine does produce a more browser like fingerprint, which is one honest reason browser automation exists.

But none of that makes anything undetectable, and this isn’t a guide to evading anything. It cuts the other way too: a headless browser in its default mode leaves its own tells, from the automation flags it sets to the too clean, repeatable shape of its fingerprint. A browser isn’t automatically safer than a client, it just fails in different places. The durable move on either side is to collect public data at a polite rate, respect the robots file, prefer an official api, and stay off the adversarial path entirely.

How I decide, every time

The ladder is short. View the raw source and check whether the data is already in the server html. If it is, use a plain http client and stop there. If it isn’t, open the network tab and look for a background json api you can call directly, and if it exists, use that. Only if both of those fail, when the data truly only exists after the browser renders it, reach for a headless browser, and even then render only the pages that need it.

Whichever side you land on, validate what comes back. The failure that hurts most is the one that doesn’t throw an error, where a client gets a soft block page or a browser reads the dom too early and hands you rows full of nulls. Confirm the fields you expect are present and in a sane range, and fail loudly when they aren’t.

I run this exact setup in production, so the frameworks and tradeoffs here are the ones I actually deal with, not theory. If you want the full written guides, the hybrid configs, and the picks I use and test, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the cheapest tool that gets the data.

Get new guides and videos first — join the Telegram channel.

Three services, one pitch: send a URL, get the page back, and let someone else own the proxies, the browsers, and the retries. ScraperAPI, Zyte, and Bright Data Web Unlocker all sell that promise, and on their landing pages they read almost the same. They aren’t the same. I paid for all three and ran them against the same targets on real jobs, and the differences show up the moment a site fights back.

I run my own proxy infrastructure and production scrapers, so this is a tested comparison, not a vendor roundup. None of these makes scraping undetectable, and none of them makes anything legal that wasn’t already legal. What they sell is convenience behind one endpoint. The only question worth answering is whether that convenience earns its price on your particular job.

The verdict, up front

Tool	Best for	Pricing shape	Hard-target success	Latency	Watch out for
ScraperAPI	Ordinary and medium sites, least setup	Credits per request (more for render or premium)	Lowest of the three on my hardest target	Fastest on simple fetches	Retries on hard sites eat credits
Zyte	Scraping as a standing operation	Closer to per successful request	Strong, a step behind Web Unlocker	Middle	Steeper learning curve
Bright Data Web Unlocker	The sites that beat everything else	Successful request, bandwidth driven	Highest on my hardest target	Slowest per page	Bandwidth pricing climbs fast at scale

Read that table as one test on one set of targets over a few days. Success rates move as sites change their defenses. What holds is the shape: the more specialized and expensive the tool, the better it did on the one site that fought hardest.

What these APIs actually do

Start with what you’re renting, because the branding hides it. A managed scraping API is the infrastructure layer you’d otherwise build, wrapped behind a single URL. You send a request, and it picks a proxy from a pool, spins up a headless browser if the page needs JavaScript, retries when a fetch fails, and hands back finished HTML or parsed fields. You’re not buying magic. You’re paying someone else to run the proxy farm and the browser fleet. Everything below is about how differently these three do it.

The pricing shapes are the real difference

The three don’t even bill the same unit, and that’s where people get surprised.

ScraperAPI mostly sells credits per request, and a request costs more credits when it needs rendering or premium proxies. Zyte bills closer to per successful request, so a failed attempt doesn’t always cost you. Web Unlocker charges for successful requests too, but the network underneath is priced around bandwidth, so a heavy page costs more than a light one.

Three pricing shapes means the cheapest option flips depending on your target. You can’t rank them on sticker price alone, which is exactly why I measure cost per successful record on my own target instead of trusting the pricing page.

ScraperAPI

ScraperAPI is the friendliest to start with. The API is simple, the credit model is easy to reason about, and for static and moderately defended sites it just works. Flip a render parameter when a page needs JavaScript, a country parameter when you need a region, and you get your HTML back. On my easy and medium targets it cleared them at a high rate and stayed cheap.

The honest part: on my hardest target, the kind of site that throws a real challenge and watches behavior, its success rate dropped and the retries started burning credits. It didn’t fail outright, but the cost per successful page climbed enough that it stopped being the cheap option there. That’s positioning, not a flaw. It’s tuned to be affordable across the broad middle of the web, not to win a brawl with the two or three most defended sites you care about.

Zyte

Zyte comes at this from the data-company angle. Its API is paired with a proxy layer that adjusts how hard it works a request, and it leans toward charging for results rather than attempts. That shape matters on a hard target, because a site that needs several tries per page won’t silently multiply your bill the way a pure per-request model can. There’s more of a data platform around it too, for teams that treat scraping as an ongoing operation rather than a weekend script.

Where Zyte asks more of you is the learning curve and reasoning about cost under load. It’s less flip-a-switch than ScraperAPI. On my tests it held a strong success rate on the medium and hard sites and landed in the middle on cost, which is roughly where its whole design points. It’s what I reach for when scraping is a standing part of the operation.

Bright Data Web Unlocker

Web Unlocker is the specialist. It isn’t a general framework, it’s an unblocking endpoint built on Bright Data’s large residential and mobile network, aimed at the sites that beat everything cheaper. You hand it a URL, it works the problem behind the scenes, and it returns the page. On the single hardest target in my set, this cleared it most reliably, and by a clear margin.

That capability isn’t free. Because the network underneath is priced around bandwidth, a heavy page rendered through it costs real money, and a large crawl adds up fast. It’s also not the quickest. The mistake I see people make is reaching for the strongest unblocker for their whole job when only a handful of pages actually need it. Use the specialist where it’s required and its price stays contained.

How I tested

Same set of targets for all three: an easy static site, a medium JavaScript site, and one genuinely hard target. A few thousand requests through each service against each. Then I measured the three numbers that matter: the real success rate on the hard target, the latency per returned page, and the true cost per successful record rather than the advertised cost per request.

On the easy and medium targets, all three cleared them at high rates and the differences barely mattered. The hard target separated them, in the order the table shows. Latency tracked difficulty: the simple fetches came back quickest, and the heavy unblocking calls were noticeably slower per page. If you’re pulling large volume, that per-page latency multiplies into hours, and that’s a real cost, not a footnote.

When to buy, and when to build

Buy when the target is genuinely hard and would otherwise eat weeks, when the volume is modest, or when time is the scarce thing. A service that works this afternoon beats a build that works in three weeks, and if scraping isn’t your core business, buying the boring part is usually right.

Build when the volume is large, when you need full control over how each request is made, or when the targets are simple. At real scale on ordinary targets, your own stack wins on cost by a wide margin, and the proxy layer you put in front of it decides whether it works. In practice I run a hybrid: my own mobile proxies on real carrier SIM cards for the predictable bulk, and a specialist API in reserve for the few sites that fight back hardest.

The honest limits

None of these three makes anything undetectable, whatever the landing page implies, and none of them makes it legal to collect data that was never yours. The same rules apply: public data, the robots file respected, an official API or bulk feed preferred where one exists, a polite rate held. A service absorbs the infrastructure work, but not the responsibility for what you scrape.

I run both sides of this in production, my own stack and these APIs on real jobs. If you want the full reviews with the measured numbers and the picks I actually use, they’re at dataresearchtools.com, tested on real targets with no undetectable promises.

Get new guides and videos first — join the Telegram channel.

The question I get more than almost any other is some version of “how many proxies do I need for this scrape?” People want a round number back, a clean fifty or a thousand they can drop into a budget. The honest answer is that it’s arithmetic, and the number falls out of two things: how fast you want to pull, and how fast the target will quietly tolerate each address. Get that math wrong one way and every address you rent gets burned in an afternoon. Get it wrong the other way and you’re paying for a huge pool you never needed, sometimes making yourself easier to spot in the process.

I run proxy infrastructure and production scrapers, so I size pools for real jobs with real budgets, not off a vendor’s marketing page. Here’s the actual math I use, the formula that turns a target rate into an address count, and the parts most people skip.

Sizing is a rate problem, not a count problem

Throw out the idea that this is a count problem. You’re not really asking how many addresses you need, you’re asking how much traffic you want to push and how thinly you have to spread it so no single address gets noticed. The pool exists to divide your total load across enough addresses that each one stays under the rate a site will accept from a single visitor. Once you see it as spreading a rate, the number almost calculates itself.

The two numbers you actually need

You need two numbers before you can size anything.

The first is your target request rate, how many requests per second you want the whole job to make. That comes from how much data you need and how quickly you need it. You choose it.

The second is the safe rate per ip, how many requests a single address can make against this particular site before that site starts treating it as suspect. You don’t choose this one. The site decides it, and you have to go and measure it. It’s a property of the target, not of your proxies, which is why you can’t read it off anyone’s spec sheet. You find it by running a small test against the real site: one address, slowly raising the rate, watching for the point where the block rate climbs or challenges start appearing. Then you set your safe rate comfortably under that ceiling.

The formula

The formula is simple enough to hold in your head:

ip count is roughly your target request rate divided by the safe rate per ip.

That’s the whole thing. If you want the job to make some number of requests per second, and each address can safely make some smaller number per second, then the first divided by the second tells you how many addresses it takes to carry that load without any one of them going too fast. Everything else is just adjusting that result.

A worked example

Say you want the whole scrape to run at 20 requests per second. And say you’ve measured that this site tolerates about one request every five seconds from a single address before it gets unhappy. That’s a safe rate of roughly 0.2 requests per second per ip.

Divide 20 by 0.2 and you get 100. So you need on the order of 100 addresses to push 20 requests per second while keeping each one at a pace the site accepts. That’s your starting number, not your final one.

Add headroom

100 is the floor, because reality is messier than the math. Some addresses in any pool are already burned before you touch them. Some go stale partway through the job. Some fail for reasons that have nothing to do with you. So pad the clean number, usually by 30 to 50 percent, so the job still hits its rate when a chunk of the pool is unusable. 100 becomes closer to 150 in practice. Round up, never down.

Concurrency is a different dial, and people tangle the two. Concurrency is how many requests are in flight at once. Rate is how many you complete per second over time. The formula is about rate. Don’t size your pool off parallel connections, size it off the sustained rate you push through them.

More proxies is not always safer

Here’s the part that surprises people. A bigger pool is not automatically safer, and past the number the math calls for, more addresses can hurt you.

Every address you add is one more with a reputation you didn’t build and can’t see. A huge pool spread thin is a crowd of strangers, and if a slice of it is already dirty, you’ve inherited blocks you never earned. You’re also paying for capacity you aren’t using, which on metered residential pools is real money leaking for nothing.

There’s a subtler cost too. Edge defenders don’t just watch single addresses, they watch across a whole site and across many sites at once. A sudden swarm of fresh addresses all crawling the same paths at the same speed is itself a pattern, even when no single address is going too fast. Throwing a thousand brand new addresses at a job that only needed a hundred can raise your footprint instead of lowering it. The right size is the smallest pool that carries your rate politely, not the biggest one you can afford.

Pool size and rotation are the same constraint

Rotation and sizing aren’t separate topics. Rotation just cycles you through the pool, handing out a fresh address on a schedule or per request. But rotation can only spread your load if the pool is big enough. If the pool is too small, the rotator comes back around to the same address too soon, and that address ends up carrying more than its safe rate anyway. A pool below the size the formula asks for doesn’t lower your per address rate, no matter how you rotate.

Think of each address as needing a rest between uses. If the safe rate is one hit every five seconds, then after an address is used it should sit idle for about that long before it comes up again. The pool has to be large enough to fill that gap for every address in the rotation. In effect, pool size is your rate multiplied by how long each address needs to rest.

Sticky sessions size differently

Stateful work flips the calculation. If you’re walking a login, a cart, or a multi step flow that has to stay on one visitor, you can’t rotate mid session without contradicting yourself. There you size by concurrent sessions instead of by raw rate: roughly one address per session you want running at once, plus a batch of spares for the ones that fail. It’s a different formula for a different shape of job, and mixing the two up is how people end up wildly over or under provisioned.

Proxy type changes the number, not the math

The type of proxy doesn’t change the formula, it changes the safe rate that goes into it. On a strict site a datacenter address might tolerate almost nothing before it’s flagged, so the safe rate is tiny and the math demands a lot of them. A residential address reads as an ordinary person and tolerates more, so you need fewer. Mobile is the odd one: a single mobile address is already shared behind carrier grade nat by many real subscribers, so you rarely size it by raw address count at all. You size it by concurrent sessions and rotation instead. The tier you pick moves the denominator, which moves the count, but the arithmetic underneath is the same.

The honest limits

Sizing a pool correctly keeps you polite and keeps the job alive. It does not make anything undetectable, and no address count changes what you’re allowed to collect. Public data only, the robots file respected, an official api or bulk feed preferred wherever one exists. The formula is about carrying your load without leaning on any single address too hard. It’s not a way around the rules.

And ignore the headline pool size a vendor prints on the box. Millions of addresses tells you nothing about how many you need or how many will actually work on your target. Your number comes from your own measured safe rate against your own site, run through the formula, plus headroom. I’ve watched a tiny well maintained pool beat a giant dirty one on the same job. Measure first, size from the measurement, and treat the marketing number as noise.

I run this infrastructure in production, so the sizing and the pools I write about are the ones I actually use and test. If you want the full written guides and the proxy pools I genuinely rely on, read them here. No undetectable promises, no guaranteed results, just the way this actually works.

Get new guides and videos first — join the Telegram channel.

Most people store scraped data in whatever was easiest on day one, usually a single Postgres table, and then spend the next year fighting it. The table grows past what it was meant to hold, the analytics queries crawl, and every field change on the source site turns into a migration. The problem is not Postgres. The problem is asking one store to do three different jobs.

Scraped data does not have one right home. It has three, and the skill is knowing which slice of your data belongs in which. I run production scrapers, and pulling the data is only half the work. The other half is storing it so it stays queryable, deduped, and cheap as the volume climbs into the tens of millions of rows.

The three places scraped data lives

Think in three tiers, each solving a different problem.

Object storage holding Parquet files is the cheap, bottomless bucket for raw high volume capture, the stuff you want to keep but rarely query directly. Postgres is the operational layer, the deduped, indexed, current state you serve queries against every day. A warehouse like BigQuery or Snowflake is where analytics at scale lives, the place you run heavy aggregate questions over the whole history.

Almost all the storage pain I see comes from asking one of these to do another’s job.

Land the raw capture first

The first move is to separate raw capture from clean state. When the scraper pulls a page, do not parse it and throw the original away. Write the raw response somewhere cheap and untouched first, then parse from that copy.

I land raw html or raw json responses straight into object storage, one file per fetch or batched, before a single field gets extracted. The parse step reads from there, not from the live site.

The reason this matters is failure. Sites change their layout without warning, and your parser will break. If you kept only the clean parsed rows, a parser bug means you have to rescrape the site to recover, hammering it again and hoping the old data is still there. If you kept the raw capture, you just fix the parser and reparse the files you already have, offline, for free. Raw storage is cheap insurance against your own future mistakes.

Why Parquet for the raw and semi clean layer

You can land raw capture as plain json or csv, and for small jobs that is fine. At volume it hurts.

Parquet is a columnar file format. Instead of storing row by row, it stores all the values of each column together, which compresses far better because similar data sits next to itself. It is typed, so a number stays a number, and it carries its own schema. For large volumes you scan a few columns at a time, it is much smaller on disk and faster to read than csv or json, and every warehouse and analytics tool reads it natively.

The tradeoff: json preserves exactly what the site sent, which is what you want for the truly raw capture. So I keep json for the untouched landing, then convert to Parquet once the shape is known.

Postgres is your operational truth

Postgres is where the clean, current, queryable state lives. Once you have parsed a record, this is the row you actually use, look up, join against, and serve to whatever consumes the data. You get real indexes, real constraints, transactions, and sql everyone already knows.

For operational work, looking up one entity, filtering a few thousand rows, enforcing that a record is unique, nothing beats a boring relational database. Using Postgres is not the mistake. The mistake is trying to keep every raw byte you ever scraped inside it.

Dedup keys and idempotent writes

The most important thing you do in the operational layer is dedup, and it starts with a key. Every record you scrape needs a stable natural id, something the source itself defines: a product code, a listing id, a canonical url. That key tells you whether a row you just parsed is new or something you already have. Put a unique constraint on it.

Then make your writes idempotent, so running the same insert twice changes nothing the second time. Postgres gives you this with an upsert:

insert into products (id, title, price, raw)
values ($1, $2, $3, $4)
on conflict (id) do update
set title = excluded.title,
    price = excluded.price,
    raw   = excluded.raw;

On conflict with the key, it updates the existing row instead of creating a duplicate. This is what lets a pipeline retry safely: a run dies halfway, you start it again, and the records that already landed overwrite themselves cleanly instead of doubling. A stable key plus idempotent writes make a scraper safe to rerun, which you will do constantly.

Handling schema drift

Sites change their fields, and a rigid table will fight you over it. A source adds a column, renames one, or returns a value in a new shape, and suddenly your clean schema does not fit. This is schema drift, and you plan for it rather than getting surprised.

I keep a set of stable core columns I am confident about, the id and the fields I always need, and a jsonb column that holds the rest of the record as is. The structured columns stay clean and indexed for the queries that matter, and the messy, changing parts live in jsonb where a new field does not break anything.

When you have outgrown Postgres

Postgres is excellent right up until it is not. When a single table climbs past tens of millions of rows and your queries turn into big aggregates scanning most of it, a row store starts to struggle.

Counting across a hundred million rows, grouping by month over all of history, joining several large tables for a report: these are analytics questions. A database tuned for operational lookups answers them slowly, and at the cost of the operational work it should be doing. That slowdown is the signal you have outgrown one tier and need the next.

The warehouse is for analytics, not serving

That next tier is a warehouse, something like BigQuery or Snowflake. They are built for exactly the queries that make Postgres sweat, scanning and aggregating enormous columnar datasets. They separate storage from compute, so your data sits cheap and you only pay for the horsepower when a query runs. They read Parquet natively, so loading the raw layer in is straightforward.

The mistake in the other direction is serving live traffic out of a warehouse. Warehouses are built for big scans, not fast single row lookups, and they usually charge by how much data each query scans. Point an app at one for constant small reads and you get slow responses and a surprising bill. Keep operational serving in Postgres and heavy analytics in the warehouse, and let each do what it is good at.

The layered pattern I run

Here is how the three fit together in practice:

• The scraper lands raw responses as files in object storage, cheap and permanent.
• A parse step reads those, extracts clean records, and upserts them into Postgres keyed on a stable id. That becomes the deduped operational truth.
• On a schedule, the clean data loads into the warehouse as Parquet for analytics over the full history.

Raw for recovery, Postgres for serving, warehouse for analysis. Each layer feeds the next, and none is asked to do a job it is bad at.

Choosing by volume and query pattern

If you remember one rule, make it this: choose your store by volume and query pattern, not by habit.

Small data you look up and update row by row belongs in Postgres. Huge volumes of raw capture you rarely query directly belong in flat Parquet on object storage. Heavy aggregate analytics over everything belong in a warehouse. Cost often makes the call: object storage is cents per gigabyte a month, Postgres is a server you pay for whether or not it is busy, and a warehouse charges by how much each query scans. Most real pipelines want all three, and the healthy ones put each slice of data where its size and access pattern fit.

The honest limits

Storage does not change what you were allowed to collect. Public data, a robots file respected, an official api or bulk feed preferred, personal and paywalled data left alone. A clean layered warehouse full of data you should not have collected is still data you should not have collected. Good architecture makes your collection durable and cheap to keep. It does not grant permission.

I run this exact layering in production, raw in object storage, deduped state in Postgres, analytics in a warehouse. If you want the full guides, with real upsert patterns, Parquet partition layouts, and the tools I actually use and test, they are here.

Get new guides and videos first — join the Telegram channel.

Most teams pick a scraping framework the way they pick a fight, by arguing which one is objectively better. Crawlee against Scrapy, a dozen feature tables open in browser tabs, everyone certain there is a winner hiding in the details. There usually isn’t. The honest answer is a little deflating: the framework you should use is almost always the one written in the language your team already speaks.

I run production scrapers for a living, the kind that pull public data at scale every day, and I’ve shipped real jobs on both of these. So this is a tested comparison, not a spec sheet read out loud. Both are good. Neither one makes you undetectable, and I’m not going to pretend the framework is the thing that decides whether you get through the door.

The short verdict

Dimension	Scrapy (Python)	Crawlee (Node / TypeScript)
Age and maturity	Over a decade in production, huge ecosystem	Younger, smaller, growing fast
Browser handling	Bolt on via a Playwright plugin	Native Playwright and Puppeteer, first class
Concurrency	Twisted event driven engine	Node event loop plus an autoscaling pool
Typing	Python, optional type hints	TypeScript, real type checking
Best fit	Python teams, http heavy targets	Node teams, browser heavy targets

If you only remember one line: match the framework to your team’s language, then let browser needs break the tie.

What each framework actually is

Scrapy is a Python crawling framework, asynchronous at its core, and it has been the default answer for scraping in Python since long before most of today’s targets existed. It gives you spiders, a request scheduler, middlewares, and item pipelines, which is a structured place to validate and store what you pull.

Crawlee is newer, written in Node and TypeScript by the team behind Apify. Its whole idea is to unify plain http crawling and full browser crawling behind one interface, with a request queue and storage layer built in. Both frameworks make the same core promise: fetch pages, follow links, parse fields, and store the result, with the plumbing already done so you’re not wiring retries and queues by hand.

The factor that really decides it

Here is the thing I wish someone had told me earlier. The biggest deciding factor isn’t the feature list, it’s the language your team already lives in.

If your engineers write Python all day, Scrapy drops into a stack they already understand, and the data tooling around it feels native. If your product is a Node or TypeScript codebase, Crawlee lets your scrapers share types, libraries, and habits with the rest of your code. Fighting your team’s native language to chase a slightly nicer feature is a tax you pay every single day, and it almost never earns back what it costs.

Where Scrapy wins

Scrapy’s biggest asset is age. It has been in production for well over a decade, which means almost every problem you’ll hit has already been hit by someone else and written up somewhere. The ecosystem is deep, with middlewares and extensions for nearly anything, and item pipelines give you a clean path to deduplicate and store records as they arrive.

That maturity isn’t glamorous, but at three in the morning when a job breaks, a well worn path with a thousand answers already posted is worth more than any shiny feature. For http heavy work on mostly static or api reachable targets, Scrapy is hard to beat and it scales to millions of pages without a fight.

Where Crawlee wins

Crawlee was designed for the web as it is now. The modern web leans hard on JavaScript, and Crawlee treats browser crawling as a first class citizen rather than an add on. It wraps Playwright and Puppeteer directly, so moving a crawler from cheap http fetching to a full browser is close to a configuration change instead of a rewrite.

Scrapy can drive a browser too, through the scrapy playwright plugin, and it works well. But you can feel the seam, because Scrapy was built around raw http requests first. If browser rendering is the occasional exception, Scrapy plus the plugin is clean. If it’s the main event across most of your targets, Crawlee handling it natively will feel smoother and cost you far less glue code to keep alive.

The blocking helpers, honestly

Both frameworks ship helpers meant to reduce how often you get blocked, and I want to be careful here. Crawlee includes tools that rotate through realistic browser fingerprints and manage sessions so requests look less uniform. Scrapy has its own rotation middlewares and a large library of community plugins that do similar work.

These helpers genuinely reduce clumsy mistakes, the obvious tells that get a lazy scraper flagged in seconds. But none of it makes you undetectable, and none of it is an evasion toolkit. They lower the noise floor a little. They don’t turn the wall off.

Concurrency and autoscaling

Scrapy runs on Twisted, an event driven networking engine, and its whole design fires many http requests at once without blocking, so a single process can push a high volume of raw fetches efficiently. Crawlee rides Node’s event loop, which is also asynchronous by nature, and layers an autoscaling pool on top that grows and shrinks concurrent tasks based on how much cpu and memory are actually free.

That autoscaling is one of Crawlee’s nicer touches. Instead of guessing a fixed concurrency number and hoping, it watches system load and adjusts on the fly. Scrapy’s autothrottle extension does something similar by adapting the delay to response times, though it leans more toward politeness to the target than toward squeezing your own hardware. Both spare you from hand tuning a magic number.

Learning curve and typing

The learning curves split along familiar lines. Scrapy has a steeper first climb because it has its own architecture, the spiders, the settings, the pipelines, and you learn that before you’re productive. Crawlee feels more approachable to a JavaScript developer because it reads like ordinary async Node code. But that only helps if your team is already fluent in Node. Drop a Python team into Crawlee and the advantage evaporates.

The one clear technical edge here is TypeScript. Because Crawlee is written in it, you get real type checking across your scraper, which catches a whole class of silly mistakes before you run the job. On a big, long running codebase that safety adds up. Scrapy gives you Python’s flexibility and optional type hints, useful but not enforced the same way.

Proxies matter more than the framework

On hard targets, the proxy layer decides more than the framework does, and both frameworks give you a clean place to plug one in. Each lets you route requests through a rotating pool and swap the address per request. Neither ships the addresses themselves. That part is on you.

This is the layer I run myself, real mobile proxies on real carrier sim cards, because for strict targets the address you arrive on carries more trust than any framework feature. Cheap datacenter ip ranges will fail on a tough site in Crawlee exactly like they fail in Scrapy. The framework routes the request. It does not launder where it came from.

How I actually choose

I start with the team, because it dominates everything else. A Python team scraping mostly static or http reachable data, I point at Scrapy without a second thought. A Node or TypeScript team, or a project where most targets are JavaScript heavy and need a real browser, I lean Crawlee for the native browser handling and the typing.

Only when a team is genuinely comfortable in both languages do the finer features get a real vote, and even then I weigh maintenance and hiring over any single capability. The best framework is the one your team can keep alive for a year.

The honest limits

Neither framework moves the boundary that matters. Picking Crawlee or Scrapy does not make anything undetectable, and it does not change what you’re allowed to collect. The same rules apply either way: public data, a robots file respected, an official api or bulk feed preferred where one exists, and a polite rate held so the site never gets a reason to shut the door.

The framework decides how comfortably you build and maintain the scraper. It doesn’t decide whether what you’re doing is welcome. If you want the full written guides, the proxy configs, and the picks I actually run in production, they’re at dataresearchtools.com, with working code and no undetectable promises.

Get new guides and videos first — join the Telegram channel.

A scrape that runs once is a script. A scrape that has to run every day, for months, without a human babysitting it, is a pipeline, and those are completely different animals. The script only has to work when the network is clean and the site behaves. The pipeline has to survive the network being flaky, the site changing overnight, and a run dying halfway through with half the data already written.

I run production scrapers for a living, so I have been paged at three in the morning enough times to build for failure on purpose. The goal is not a clever scraper, it is a boring one that recovers from problems on its own and only wakes you up when something is genuinely wrong. That reliability is not in the parsing code, it is in the structure around it, and that structure is what an orchestrator like Dagster or Prefect gives you.

What an orchestrator actually buys you

From the outside an orchestrator looks like overhead. It is not. It turns your scrape into a set of connected steps it runs on a schedule, and it remembers what happened: which steps ran, which failed, which produced data. It can retry the ones that broke without rerunning the ones that worked. That memory is the whole point. A plain cron job runs your script and forgets everything, so a failure means starting over. An orchestrator lets you recover from exactly where it broke.

Split fetch from parse from store

The first structural move is to stop writing one big function that does everything. Split the work into stages. One stage fetches raw pages and saves them untouched. A second parses those saved pages into clean records. A third writes those records to your database.

Keeping them separate means that when the site changes its layout and your parser breaks, you have not lost the fetched pages. You fix the parser and rerun only that stage against data you already have, instead of hammering the site all over again.

Idempotency is the whole game

Here is the idea everything else rests on. Every stage should be safe to run twice. Running it again with the same input should produce the same result and never create duplicates or corrupt what is there. That property is called idempotency, and it is what lets an orchestrator retry freely.

If rerunning a stage might double your records or half write a file, then retries are dangerous and you cannot automate recovery. So you design each step to be repeatable from the start, usually by keying records on a stable id and writing in a way that overwrites cleanly rather than blindly appending.

Checkpoint so you can resume

Long jobs need to remember their progress. If you are pulling a hundred thousand pages and the run dies at page sixty thousand, you do not want to start from zero. So you checkpoint: write down what you have finished as you go, and on the next run, read that and skip what is already done. An orchestrator tracks which chunks of work succeeded, so a restart naturally picks up the unfinished ones. That is the difference between a five minute recovery and a five hour one.

Retries with backoff, done right

Failures at the network layer are constant, not rare, so retries are not optional. But a naive retry that fires again instantly just hammers a struggling site and makes things worse. The right pattern is backoff: wait a moment before the first retry, then wait longer before the next, giving the site room to recover. Both Dagster and Prefect let you set retry counts and delays per step, so a single flaky request quietly succeeds on its second or third try, and only a request that fails every attempt gets escalated.

The dead letter queue

Not everything recovers, and you need a place for the failures to go. That place is a dead letter queue, which is just a list of the items that failed all their retries. Instead of crashing the whole run because ten pages out of a hundred thousand refused to load, you set those ten aside, finish everything else, and deal with them later. This keeps one stubborn page from taking down a whole night’s work, and it gives you a clean record of exactly what needs a second look in the morning.

Incremental beats full every time

Do not rescrape the world every day. Most sites change only a little between runs, so pulling everything from scratch wastes your resources and pounds the site for no reason. Run incrementally: track what you have already collected and only fetch what is new or changed since last time. It is lighter on your infrastructure, far lighter on the site, and it is the difference between a pipeline that scales and one that gets slower and more fragile as your dataset grows.

Backfills without fear

Sometimes you do need to go back and reprocess history, maybe because you fixed a parser or added a field. That is a backfill, and a good pipeline makes it painless. Because your stages are idempotent and your raw pages are saved, you can rerun the parse stage across a date range and rebuild the clean data without touching the site at all. Dagster in particular is built around reprocessing ranges of data. Designing for backfills from the start means a schema change is an afternoon, not a crisis.

Observability, or you are flying blind

You cannot fix what you cannot see. A pipeline that runs silently is one you do not actually trust, because the first sign of trouble is stale data days later. So you instrument it. Log how many pages you fetched, how many parsed cleanly, how many landed in the dead letter queue, and how long each stage took. Both orchestrators give you a dashboard of runs and their history. When the parsed count suddenly drops to zero, you want to see it that hour, not discover it when someone downstream asks where the data went.

Alert on the right thing

Do not alert on everything, or you will train yourself to ignore the alarms. A handful of failed pages is normal background noise and should never wake you. What should wake you is a signal the pipeline is actually broken: a whole run failing, the success rate falling off a cliff, or zero records produced when you expected thousands. An alert that fires constantly is worse than no alert, because you stop reading it.

Dagster or Prefect, briefly

Both are solid Python orchestrators and either will serve you well. Prefect leans lightweight and flexible, wrapping your existing functions as tasks and flows with very little ceremony, which makes it fast to adopt. Dagster leans toward treating your data as first class assets, with strong support for backfills, typing, and knowing exactly what each step produces. For a scrape heavy on reprocessing history I tend toward Dagster; for a simpler flexible flow I reach for Prefect. Neither choice is wrong, so pick one and build.

The shape of a Prefect flow is small. You mark functions as tasks, give them retries, and wire them into a flow:

from prefect import flow, task

@task(retries=3, retry_delay_seconds=[5, 30, 120])
def fetch(url: str) -> str:
    ...

@task
def parse(raw: str) -> list[dict]:
    ...

@task
def store(records: list[dict]) -> None:
    ...

@flow
def scrape(urls: list[str]) -> None:
    for url in urls:
        store(parse(fetch(url)))

That is a real pipeline skeleton, retries and all, in a handful of lines.

Validate, or you fill the database with nulls

The failure that hurts most is the one that does not throw an error. A site quietly renames a field, or starts returning an empty value where there used to be a price, and your parser keeps running and writes rows full of nulls. Nothing crashes, so nothing alerts, and you find out days later that your data is hollow. The defense is validation: check that the records coming out of the parse stage look right, that key fields are present and in a sane range, and fail loudly when they are not.

Test on a small slice first

Never point a fresh pipeline at the full job on its first run. Run it against a tiny slice, a handful of pages, and watch every stage work end to end before you scale up. It is far cheaper to find a broken selector or a bad write on ten records than on ten million. Keep a small sample run you can fire on demand, so after any change you confirm the whole pipeline still flows cleanly before you let it loose on the real volume.

Keep the site happy at the pipeline level

Politeness is not just a per request setting, it is a pipeline property. The orchestrator controls how many workers run at once and how fast the whole thing moves, so it is where you enforce a rate the site can absorb. Spread the load across time instead of firing a huge burst, cache aggressively so you never fetch the same page twice, and prefer an official api or bulk feed when the site offers one. A pipeline that runs gently is one that keeps running, because it never gives the site a reason to shut the door.

The honest limits

Good orchestration makes a scrape reliable and recoverable. It does not make it welcome where it is not, and it does not change what you are allowed to collect. Public data, a robots file respected, an official feed preferred, a gentle pace held. Structure buys you uptime and sane recovery, not permission. No pipeline survives a site that genuinely does not want to be scraped, nor should it try to, because durable data work lives on the compliant side of that line.

To recap: split fetch from parse from store, make every stage idempotent so retries are safe, checkpoint so you can resume, and retry with backoff instead of hammering. Set failures aside in a dead letter queue, run incrementally instead of rescraping the world, design for painless backfills, and watch the pipeline closely enough to alert only on real breakage. Build it that way and it keeps running while you sleep.

If you want more breakdowns like this, dataresearchtools.com has the rest of the scraping infrastructure guides, with real code and no undetectable promises.

Get new guides and videos first — join the Telegram channel.

Two clients ask the same site for the same page in the same second. One gets the data and moves on. The other gets a captcha, then a soft block, then nothing. The requests looked almost identical, but the decision was made before either connection finished setting up. That gap is where all of bot detection lives, and understanding it is the difference between collection that runs for months and collection that dies on day one.

I run proxy infrastructure and production scrapers, so I have spent a lot of time on both sides of this wall. This is a defensive explainer, written from the site’s point of view: how modern detection actually works, and how a scraper that respects the rules stays clean. It is not a guide to evading anything, because that version does not last and it is not the business I am in.

What detection is actually looking for

The first thing to get right is the question a site is trying to answer. It is not “human or bot.” Plenty of bots are welcome. Search crawlers, uptime monitors, and link previewers are all automation, and sites wave them through on purpose. The real question is narrower: does this traffic behave the way it claims to, and do its own signals agree with each other?

Detection is a hunt for contradictions. Every layer below is one more place where an automated client can accidentally tell two different stories at once, and every mismatch is a chance to get flagged. That framing matters, because it explains why the durable answer is honesty rather than disguise. If you are not pretending to be something you are not, there is no contradiction to catch.

Network and ip reputation

The earliest signal arrives before you send a byte of a real request: where you came from. Every connection has an ip address, and every ip sits inside a block that belongs to a network, identified by a number called an asn. An entire industry labels those blocks. This range is a mobile carrier, this one is home broadband, this one is a hosting company’s datacenter. Sites buy those labels, cache them right at the edge, and the check fires on the very first packet.

A request from a residential or mobile network starts with ordinary trust. A request from a cheap datacenter range starts under suspicion, because historically that is where automation lives. And the reputation is not really about you, it is about your neighbors. A fresh datacenter ip with no history of its own still inherits the cold reception of every scraper that ever ran from that provider. That is why pointing the cheapest pool of server ips at a strict site tends to fail fast. Nobody caught you doing anything wrong. The network you rode in on was already wearing the wrong label.

The tls handshake

Before any page loads, your client and the server negotiate an encrypted channel, and that negotiation is surprisingly loud. The exact way the software sets up tls, the cipher options it offers, the order it lists them in, and the extensions it includes all form a pattern. Real browsers produce well known patterns here, and they shift in predictable ways as browsers update.

A lot of scraping tools produce a tls pattern no real browser would send, because the underlying http library was never trying to imitate one. So a client can claim in its headers to be a current version of Chrome while its handshake quietly says it is a generic http library. That is a contradiction, and the edge exists to catch exactly that kind of mismatch.

Request headers and the user agent

Once the channel is open, the request itself carries more tells. A real browser sends a specific set of headers, in a specific order, with a user agent string that agrees with everything else. The field a site reads as your user agent is just text, so anyone can type Chrome into it. The rest of the request has to back that claim up.

Real Chrome sends particular hints about the platform, the language, and the way it handles compression and connections. An http client that sets a browser user agent and then forgets the dozen other things a browser would send is telling two stories again. Detection here is mostly consistency checking: do all the parts of this request describe the same thing they claim to be?

Browser and device fingerprinting

When a site wants a harder look, it runs code inside the browser itself, and this is where headless automation gets tested. A normal browser driving a real screen exposes thousands of small properties: the window size, the installed fonts, the way the gpu draws a shape onto a hidden canvas, the exact features webgl reports, the timing of tiny operations. Real devices vary across all of it in messy, natural ways.

An automation setup that spins up the same headless browser again and again tends to produce the same fingerprint every time, or a fingerprint carrying tells a normal machine would never have. The site is not hunting for one magic value. It is looking at the whole shape and asking whether it resembles a real person’s device or a cloned machine in a datacenter. There is also a quieter middle layer here: when a site is unsure, it hands the client a small challenge, a bit of work a real browser does without thinking. For a person it passes invisibly. For a crude client it is a wall.

Behavioral and rate signals

Past the technical fingerprints, sites watch behavior, and behavior is where scale gives automation away. A person browses in bursts, pauses, gets distracted, reads for a while, and comes back. A naive scraper pulls pages at a steady machine rhythm, one after another, faster than a human could read, at even intervals, around the clock with no sleep. No single request is strange. The shape over time is.

It widens out from one connection to the whole crowd, too. Edge networks sit in front of a huge slice of the web, so they see traffic for countless sites at once. That lets them spot a swarm of fresh visitors all taking the same path at the same speed, or many separate ips that each look fine alone but move in lockstep. Rate is part of it, how much you pull and how fast. Pattern is the deeper signal, and a new automation pattern caught on one site becomes a signature that defends the rest within minutes.

How compliant scrapers stay clean

So what does an operation that respects the rules do differently? It starts by reading the robots file and honoring it, treating the parts a site asks you to leave alone as off limits rather than as a suggestion. It identifies itself honestly where a site expects that, instead of dressing a server up as a browser it is not. It collects public data only, and stays away from anything behind a login, a paywall, or personal information that was never meant to be harvested.

The biggest lever is pace, and it is the one people skip because it feels slow. A compliant scraper runs at a rate the site can absorb, spaces its requests, backs off the moment it sees errors or slowdowns, and caches so it never asks twice for something it already has. It prefers an official api when one exists, because that is the front door the site actually built. Done this way, most of the detection stack never has a reason to fire, because you are not producing the contradictions it hunts for.

There is also a step above all the tooling: ask. If a site offers a data feed, a bulk download, or a documented api, that sanctioned path is almost always cheaper than fighting the detection stack. If the terms of service say no, then no clever handshake changes what you agreed to. For anything ambiguous, a short message to the site owner is worth more than a week of tuning a client, because permission turns the whole adversarial framing off.

The honest limits

I want to be straight about the boundaries. None of this makes anything undetectable, and I would not trust anyone who tells you it does. Detection keeps improving, the labels get sharper, and what passes quietly today can be flagged tomorrow. A clean network and honest behavior are not a trick that beats the wall. They are the absence of the contradictions that get you caught.

The moment your collection touches private data, or ignores what a site clearly asked you not to do, you have left the compliant lane entirely, whatever your tooling looks like. The durable version of this work is the boring version: public data, honest identification, a gentle pace, and the rules respected.

I run this infrastructure in production, so the frameworks, proxy setups, scraping apis, and honest tool reviews I write about are the ones I actually deal with. If you want the full written guides and the tools I use and test, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the way this actually works.

Introduction: The Analytics Blind Spot

Customer journey analytics promises a complete view of how users interact with your brand — from first ad impression to final purchase. But there’s a persistent problem: the data is often wrong.

Location mismatches, carrier-level filtering, bot traffic mixing with real users, and geo-blocked content all distort the picture. Marketers end up making decisions based on data that doesn’t accurately represent how real customers actually experience their brand.

Mobile proxies are increasingly being used to close this gap — by enabling teams to collect, validate, and enrich journey data from the perspective of real mobile users in real locations.

---

What Is Customer Journey Analytics?

Customer journey analytics maps every touchpoint a user encounters across channels and devices — including:

• Paid search and social ads
• Organic search results
• Website and app visits
• Email engagement
• In-store or location-based interactions
• Customer service contacts

The goal is to understand what drives conversion, where users drop off, and how different segments experience the brand differently. But all of this depends on the quality and accuracy of the underlying data.

Where Data Gaps Occur — and Why

Several common scenarios corrupt or incomplete customer journey data:

1. Geo-IP Misattribution

Standard analytics platforms use IP geolocation to assign location to users. But mobile carrier IP ranges are often misclassified — a user in Lagos may be assigned to a London IP pool if they’re roaming or using a multinational carrier. This creates false geographic segments in your analytics.

2. Bot and Proxy Traffic Contamination

Up to 40% of web traffic can be non-human. Bots scraping your site, competitors checking prices, and internal QA teams all show up in analytics and distort conversion funnels, bounce rates, and session data.

3. Carrier-Level Content Blocking

Some mobile carriers filter or transform content before it reaches users. Ad pixels may fail to fire. Tracking scripts may be blocked. Users on certain networks may never trigger the analytics events your platform depends on — creating silent gaps in journey data.

4. Device-Dependent Rendering

Desktop-based analytics collection tools can’t capture how mobile users actually experience pages. Features that break on specific devices — slow loading images, broken CTAs, unresponsive forms — show up as conversion drop-offs without explanation.

How Mobile Proxies Close the Data Gap

Accurate Geo-Segmented Data Collection

By routing analytics collection through real mobile IPs in specific cities and countries, teams can gather journey data that reflects what real users in those locations actually see and do. This allows:

• Verification that geo-targeted content is triggering correct analytics events.
• Calibration of geo-segmentation models with clean, location-confirmed data.
• Identification of regions where tracking is failing silently.

Carrier-Level Tracking Verification

Mobile proxies let analytics teams test whether tracking pixels, conversion tags, and event scripts fire correctly across different carrier networks. If a carrier is stripping scripts or blocking pixels, you’ll see it in your test data — before it silently corrupts months of production analytics.

Real Mobile Session Simulation

Connecting through real mobile IPs on genuine carrier networks replicates the actual conditions under which your users browse. This enables:

• Accurate session replay and funnel analysis from real mobile environments.
• Detection of mobile-specific friction points in the customer journey.
• Validation that mobile-specific tracking tags (app install events, SMS clicks) fire correctly.

Competitive Journey Benchmarking

Beyond your own analytics, mobile proxies allow marketing teams to map competitor customer journeys — seeing how rivals present their funnels, CTAs, pricing, and onboarding flows in specific markets. This context helps teams benchmark conversion performance against real market conditions.

Practical Use Cases by Team

Team	Use Case	Benefit
Analytics	Validate geo-segment accuracy	Eliminate misattributed location data
Paid Media	Verify tracking pixel fires by carrier	Accurate ROAS reporting
Product	Test funnel on real mobile networks	Catch mobile-only drop-off points
CRO	Confirm A/B test bucketing by region	Clean experiment data
Compliance	Audit consent and cookie tracking	GDPR/CCPA data accuracy

Integrating Mobile Proxies into Your Analytics Stack

Mobile proxies work alongside — not instead of — your existing analytics tools. Here’s how to integrate them effectively:

1. Map your key journey events — identify the 10–15 critical events that define your funnel (page view, add-to-cart, checkout start, purchase, etc.).
2. Set up geo-specific test sessions — use mobile proxies to simulate user sessions from your top 5–10 markets and verify all events fire correctly.
3. Automate carrier-coverage checks — schedule weekly automated tests across major carriers per region to catch tracking failures as they occur.
4. Cross-reference with production data — compare test session behavior against production analytics to identify discrepancies that indicate tracking issues.
5. Iterate on findings — use insights from proxy-based testing to patch tracking gaps, improve geo-segmentation, and clean historical data where needed.

FAQs: Mobile Proxies and Customer Journey Data

Q1: Will using mobile proxies for testing skew my production analytics?
Not if configured correctly. Test traffic can be filtered out using IP lists, custom UTM parameters, or separate analytics properties for QA sessions.

Q2: Can mobile proxies help fix attribution model inaccuracies?
Yes. By verifying that events fire correctly across all major markets and carriers, you improve the raw data quality that attribution models depend on.

Q3: How often should analytics validation runs be scheduled?
At minimum, after every major site deployment. For high-traffic e-commerce or media businesses, weekly automated checks are recommended.

Q4: Do mobile proxies work with Google Analytics, Adobe Analytics, and similar platforms?
Yes — mobile proxy sessions generate standard HTTP traffic that is captured by all major analytics platforms in the same way as real user sessions.

Conclusion: Better Data Starts with Real-World Testing

Customer journey analytics is only as good as the data that feeds it. When tracking fails silently across certain carriers, when geo-segments are misclassified, or when mobile-specific conversion events are missed, marketing decisions are built on a broken foundation.

Mobile proxies provide the one thing standard analytics tooling can’t: the ability to experience your customer journey as your customers actually experience it — from their carrier, in their city, on their device.

• They close geo-attribution gaps that distort segmentation.
• They expose carrier-level tracking failures before they corrupt production data.
• They enable competitive journey benchmarking that no other tool can provide.

👉 Bottom line: If your marketing organization is serious about data quality, validating your analytics stack with mobile proxies isn’t optional — it’s a competitive necessity.

Building scraping pipelines with Dagster in 2026

Building scraping pipelines with Dagster is the right answer when scraping is one part of a larger data engineering pipeline. Dagster reframes orchestration around data assets rather than tasks: instead of writing “fetch URLs, parse, store” as separate steps, you declare “products dataset” as a software-defined asset that depends on “raw HTML pages” which depends on “URL list,” and Dagster figures out the execution graph. For pure scraping with no downstream analytics, this is overkill. For scraping that feeds into ML models, dashboards, or warehouses, Dagster’s asset-aware approach pays off in lineage, partitioning, and freshness tracking.

This guide covers Dagster 1.7+ in 2026 for scraping pipelines: software-defined assets, jobs, sensors, partitioning by time and host, IO managers, and a complete end-to-end example. Code is Python 3.12. By the end you will know whether Dagster fits your scraping use case and how to deploy a working pipeline.

Why Dagster for scraping

Dagster’s strengths for scraper-heavy pipelines:

• Software-defined assets: declare data products instead of orchestration steps
• Lineage tracking: see how data flows from raw HTML to final dashboard
• Partition support: per-day, per-host, per-customer partitions with backfill
• IO managers: separate “what to compute” from “where to store it”
• Sensors: trigger pipelines based on external events (file arrival, API webhook)
• Strong type system: Pydantic-style configs and asset metadata
• Asset checks: data quality assertions per asset
• Mature: 6+ years in production, used by Robinhood, Duolingo, etc.

For Dagster’s official documentation, see docs.dagster.io.

Where Dagster does not lead

• For simple scheduled scraping with no downstream pipeline, Prefect or cron are simpler
• For very high-frequency tasks (1000s/sec), the asset model adds overhead
• For teams without data engineering experience, the asset abstraction takes adjustment

For Prefect comparison, see building scraping pipelines with Prefect 3.

Installing Dagster

pip install dagster dagster-webserver
dagster --version  # 1.7+

# Initialize a project
dagster project scaffold --name my-scraper
cd my-scraper

Start the dev server:

dagster dev
# UI at http://localhost:3000

A first asset: scraping a URL

# my_scraper/assets.py
from dagster import asset, AssetExecutionContext
import httpx

@asset
def homepage_html(context: AssetExecutionContext) -> str:
    resp = httpx.get("https://example.com", headers={
        "User-Agent": "Mozilla/5.0 ...",
    })
    resp.raise_for_status()
    context.log.info(f"Fetched {len(resp.text)} bytes")
    return resp.text

@asset
def homepage_titles(homepage_html: str) -> list[str]:
    from selectolax.parser import HTMLParser
    tree = HTMLParser(homepage_html)
    return [n.text(strip=True) for n in tree.css("h2")]

Two assets: homepage_html (the raw HTML) and homepage_titles (parsed titles, depends on homepage_html). Dagster figures out the dependency from the function signature.

In the Dagster UI, you see a graph with two nodes connected by a line. Click “Materialize all” and Dagster fetches the page and parses it.

Partitioned assets: scrape per day, per host

For real scraping, you usually want partitions: one materialization per day, per host, or per customer. Dagster partitions track which slices have been materialized and which need backfilling.

from dagster import asset, DailyPartitionsDefinition, AssetExecutionContext
from datetime import datetime
import httpx

daily_partitions = DailyPartitionsDefinition(start_date="2026-01-01")

@asset(partitions_def=daily_partitions)
def daily_news_html(context: AssetExecutionContext) -> str:
    date_str = context.partition_key  # YYYY-MM-DD
    url = f"https://news.example.com/archive/{date_str}"
    resp = httpx.get(url)
    resp.raise_for_status()
    return resp.text

Now daily_news_html has one materialization per day. Run today’s: Dagster fetches today’s URL. Backfill last 30 days: Dagster fetches each day’s URL.

For multi-dimensional partitions (per-day AND per-host):

from dagster import (
    asset, MultiPartitionsDefinition, StaticPartitionsDefinition,
    DailyPartitionsDefinition,
)

hosts = StaticPartitionsDefinition(["example.com", "news.example.com", "shop.example.com"])
date_partitions = DailyPartitionsDefinition(start_date="2026-01-01")

multi_partitions = MultiPartitionsDefinition({"date": date_partitions, "host": hosts})

@asset(partitions_def=multi_partitions)
def host_daily_pages(context: AssetExecutionContext) -> str:
    keys = context.partition_key.keys_by_dimension
    date_str = keys["date"]
    host = keys["host"]
    url = f"https://{host}/archive/{date_str}"
    resp = httpx.get(url)
    return resp.text

This creates a 2D grid of partitions. Backfill specific cells (date X host) without re-running everything.

IO managers: separating compute from storage

In simple cases, the asset function returns a Python value and Dagster pickles it to local disk. For production, use IO managers to write to Postgres, S3, or your warehouse.

from dagster import IOManager, io_manager, OutputContext, InputContext
import boto3
import json
import io

class S3IOManager(IOManager):
    def __init__(self, bucket: str):
        self.bucket = bucket
        self.s3 = boto3.client("s3")

    def _key(self, context):
        # asset_key is a list of strings
        path = "/".join(context.asset_key.path)
        if context.has_partition_key:
            path = f"{path}/{context.partition_key}"
        return f"{path}.json"

    def handle_output(self, context: OutputContext, obj):
        key = self._key(context)
        body = json.dumps(obj).encode() if not isinstance(obj, bytes) else obj
        self.s3.put_object(Bucket=self.bucket, Key=key, Body=body)
        context.log.info(f"Wrote to s3://{self.bucket}/{key}")

    def load_input(self, context: InputContext):
        key = self._key(context)
        resp = self.s3.get_object(Bucket=self.bucket, Key=key)
        return json.loads(resp["Body"].read())

@io_manager(config_schema={"bucket": str})
def s3_io_manager(init_context):
    return S3IOManager(bucket=init_context.resource_config["bucket"])

Wire it in:

from dagster import Definitions

defs = Definitions(
    assets=[homepage_html, homepage_titles],
    resources={"io_manager": s3_io_manager.configured({"bucket": "my-scraper-data"})},
)

Now every asset’s output goes to S3 instead of local disk. Same code, different storage.

Browser scraping with Playwright as an asset

from dagster import asset, AssetExecutionContext
from playwright.sync_api import sync_playwright

@asset(partitions_def=daily_partitions)
def dynamic_page_html(context: AssetExecutionContext) -> str:
    date_str = context.partition_key
    url = f"https://example.com/dynamic?date={date_str}"

    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        html = page.content()
        browser.close()
    return html

Same pattern: Playwright runs inside the asset function. Dagster orchestrates.

Jobs: grouping assets

Jobs run sets of assets together. Define a job that materializes a related group:

from dagster import define_asset_job, AssetSelection

scrape_news_job = define_asset_job(
    "scrape_news",
    selection=AssetSelection.assets("daily_news_html", "daily_news_articles"),
)

Schedule it:

from dagster import ScheduleDefinition

daily_news_schedule = ScheduleDefinition(
    job=scrape_news_job,
    cron_schedule="0 6 * * *",  # 6am daily
)

Sensors: event-driven materialization

Sensors trigger asset materialization based on external events:

from dagster import sensor, RunRequest, SkipReason
from datetime import datetime
import httpx

@sensor(job=scrape_news_job)
def new_url_sensor(context):
    # Check an API for new URLs
    resp = httpx.get("https://api.example.com/new-urls")
    new_urls = resp.json()

    if not new_urls:
        return SkipReason("No new URLs")

    return RunRequest(
        run_key=str(datetime.utcnow().timestamp()),
        run_config={"ops": {"daily_news_html": {"config": {"urls": new_urls}}}},
    )

Dagster runs the sensor function periodically (default 30 seconds). When it returns a RunRequest, the job triggers.

Asset checks: data quality

Built-in data quality assertions per asset:

from dagster import asset, AssetCheckResult, asset_check

@asset
def products_data(homepage_html: str) -> list[dict]:
    # Parse and return products
    return [{"title": "...", "price": "..."}]

@asset_check(asset=products_data)
def check_min_products(context, products_data):
    count = len(products_data)
    return AssetCheckResult(
        passed=count >= 10,
        metadata={"count": count, "expected_min": 10},
    )

@asset_check(asset=products_data)
def check_no_missing_prices(context, products_data):
    missing = sum(1 for p in products_data if not p.get("price"))
    return AssetCheckResult(
        passed=missing == 0,
        metadata={"missing_count": missing},
    )

After every materialization, Dagster runs the checks and shows pass/fail in the UI. Fail an SLA, fail a build, alert the team.

Comparison: Dagster vs Prefect for scraping

dimension	Dagster	Prefect 3
paradigm	software-defined assets	flows and tasks
best for	data pipelines with downstream ETL	mid-complexity workflows
async support	sync-first, async via Op syntax	excellent async
partitioning	rich (multi-dimensional, backfill)	basic
asset lineage	first-class	via DAGs
hosted option	Dagster Cloud	Prefect Cloud
learning curve	medium-high	low-medium
dashboard	excellent	excellent

Pick Dagster when scraping feeds analytics, ML, or BI. Pick Prefect when scraping is the end product or when async-heavy.

Complete production example

# my_scraper/definitions.py
from dagster import (
    asset, AssetExecutionContext, Definitions,
    DailyPartitionsDefinition, define_asset_job, ScheduleDefinition,
    AssetSelection, IOManager, io_manager, Resource,
)
from dagster_aws.s3 import S3PickleIOManager, S3Resource
import httpx
from selectolax.parser import HTMLParser
import json
from typing import Optional

daily_partitions = DailyPartitionsDefinition(start_date="2026-01-01")

@asset(partitions_def=daily_partitions, group_name="ingestion")
def raw_pages(context: AssetExecutionContext) -> dict[str, str]:
    """Fetch all configured pages for the partition date."""
    date_str = context.partition_key
    urls = [
        f"https://example.com/products?date={date_str}&page={i}"
        for i in range(1, 11)
    ]
    pages = {}
    for url in urls:
        try:
            resp = httpx.get(url, timeout=30)
            resp.raise_for_status()
            pages[url] = resp.text
            context.log.info(f"Fetched {url}: {len(resp.text)} bytes")
        except Exception as e:
            context.log.error(f"Failed {url}: {e}")
    return pages

@asset(partitions_def=daily_partitions, group_name="parsing")
def parsed_products(context: AssetExecutionContext, raw_pages: dict) -> list[dict]:
    """Parse products from all raw pages."""
    products = []
    for url, html in raw_pages.items():
        tree = HTMLParser(html)
        for el in tree.css("article.product"):
            title_el = el.css_first("h2")
            price_el = el.css_first(".price")
            link_el = el.css_first("a")
            products.append({
                "title": title_el.text(strip=True) if title_el else None,
                "price": price_el.text(strip=True) if price_el else None,
                "url": link_el.attributes.get("href") if link_el else None,
                "source_url": url,
                "scraped_date": context.partition_key,
            })
    context.log.info(f"Parsed {len(products)} products")
    return products

@asset(partitions_def=daily_partitions, group_name="storage")
def products_in_db(context: AssetExecutionContext, parsed_products: list) -> int:
    """Write parsed products to Postgres."""
    import asyncpg
    import asyncio

    async def write():
        conn = await asyncpg.connect("postgresql://user:pass@localhost/scraper")
        try:
            await conn.executemany(
                """INSERT INTO products
                   (title, price, url, source_url, scraped_date)
                   VALUES ($1, $2, $3, $4, $5)
                   ON CONFLICT (url, scraped_date) DO NOTHING""",
                [(p["title"], p["price"], p["url"], p["source_url"], p["scraped_date"])
                 for p in parsed_products],
            )
            return len(parsed_products)
        finally:
            await conn.close()

    count = asyncio.run(write())
    context.log.info(f"Stored {count} products to DB")
    return count

@asset_check(asset=parsed_products)
def check_minimum_products(context, parsed_products):
    return AssetCheckResult(
        passed=len(parsed_products) >= 50,
        metadata={"count": len(parsed_products), "expected_min": 50},
    )

scrape_job = define_asset_job(
    "daily_scrape",
    selection=AssetSelection.assets("raw_pages", "parsed_products", "products_in_db"),
)

scrape_schedule = ScheduleDefinition(
    job=scrape_job,
    cron_schedule="0 6 * * *",
    execution_timezone="UTC",
)

defs = Definitions(
    assets=[raw_pages, parsed_products, products_in_db],
    asset_checks=[check_minimum_products],
    jobs=[scrape_job],
    schedules=[scrape_schedule],
    resources={
        # Add IO managers, DB connections, etc.
    },
)

This pipeline:

1. Fetches raw pages daily, partitioned by date
2. Parses products from raw HTML
3. Stores to Postgres with idempotent insert
4. Asserts minimum product count
5. Runs daily at 6am UTC

In the Dagster UI, you see the asset graph, materialization history, and check pass/fail per partition.

Deployment

For production:

• Dagster Cloud (managed): easiest, paid per user + execution
• Self-hosted on Kubernetes: dagster-helm chart, full control
• Self-hosted on a single VM: docker-compose with PostgreSQL, fine for small teams

For Kubernetes:

# values.yaml for dagster-helm
dagsterDaemon:
  enabled: true
runLauncher:
  type: K8sRunLauncher
postgresql:
  enabled: true

Dagster spawns a Kubernetes pod per run, isolating resources.

Operational checklist

For production Dagster scraping in 2026:

• Dagster 1.7+ with Python 3.11+
• Dagster Cloud or self-hosted on Kubernetes
• Partitions defined for backfill capability
• IO managers for production storage (S3, Postgres)
• Asset checks for data quality
• Sensors for event-driven workflows
• Schedules for cron-style runs
• Type hints on all asset functions
• Resource configs separate from code

For broader pipeline patterns, see building scraping pipelines with Prefect 3 and distributed scraping with Apache Kafka.

Common pitfalls

• Asset name conflicts: two assets with the same key error out. Use key_prefix to namespace.
• Partition explosion: multi-dimensional partitions can produce millions of cells. Plan partitioning carefully.
• Sync code in async contexts: Dagster is sync-first; for async, use asyncio.run inside the asset.
• Long-running assets: Dagster default timeout is short. Configure execution timeout per asset.
• IO manager mismatches: changing IO managers mid-flight can cause input/output type mismatches.

FAQ

Q: should I use Dagster or Prefect for pure scraping?
For pure scraping, Prefect’s simpler model is usually a better fit. Dagster pays off when scraping is part of a larger ETL or ML pipeline.

Q: can I migrate from Airflow to Dagster?
Yes, with effort. The asset model is different from DAGs. Most teams migrate one pipeline at a time over months.

Q: does Dagster work with Scrapy?
Yes. Wrap Scrapy invocations as assets. The orchestration value is at the pipeline level, not within individual spiders.

Q: how does cost compare to Prefect?
Self-hosted: similar (just compute). Cloud: Dagster Cloud is more expensive per user, but pricing for execution is comparable.

Q: is Dagster overkill for a 5-spider project?
Probably yes. For 5 unrelated scrapers, cron + Python scripts is fine. For 5 scrapers feeding a unified analytics warehouse, Dagster shines.

Common pitfalls in production Dagster scraping

The first failure mode is the partition backfill blast radius. Dagster’s backfill UI lets you select multiple partitions and re-materialize them. A typo in the partition selector (selecting “all 2026 partitions” when you meant “this week’s partitions”) triggers 365 simultaneous materialization runs, which queues 365 Kubernetes pods, exhausts your cluster’s pod limit, and starves all other pipelines for hours. The fix is to enable backfill concurrency limits in the Dagster instance config, capping simultaneous backfill runs at a sane number:

# dagster.yaml
run_coordinator:
  module: dagster._core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 25
    tag_concurrency_limits:
      - key: dagster/backfill
        limit: 5

This caps backfills at 5 concurrent runs while leaving 20 slots for scheduled and ad-hoc runs. Without this, a single mis-clicked backfill can take down your entire pipeline cluster.

The second pitfall is asset definition import-time side effects. Dagster imports your asset definitions on every code-server reload to build the asset graph. If your asset definition file contains import-time code that fetches from an external service (loading site configs from a database at module import), every code reload triggers that fetch. With 50 developers reloading code 100 times per day, you generate 5000 daily fetches that the upstream service was never designed for, and you sometimes get throttled by your own configuration store. The fix is to defer all external calls to inside the asset function or use a Dagster Resource that lazy-loads:

from dagster import resource, asset, ConfigurableResource

class SiteConfigResource(ConfigurableResource):
    config_url: str
    _cache: dict | None = None

    def get_configs(self) -> dict:
        if self._cache is None:
            self._cache = httpx.get(self.config_url).json()
        return self._cache

@asset
def raw_pages(context, site_configs: SiteConfigResource):
    configs = site_configs.get_configs()  # only fetched when asset runs
    # ...

The third pitfall is the IO manager memory bloat on large asset values. Dagster’s default IO managers (S3PickleIOManager, FilesystemIOManager) serialize the entire asset value to storage between assets. A raw_pages asset returning 10,000 HTML pages of 50KB each is 500MB of data. The pickle serialize-deserialize cycle runs in a single process, peaking at 1GB+ resident memory. Workers OOM-kill, the run fails, and Dagster retries it, blowing through your retry budget. The fix is custom IO managers that stream rather than buffer, or asset partitioning that keeps individual asset values under 100MB:

class StreamingS3IOManager(IOManager):
    def handle_output(self, context, obj):
        # obj is an iterator, not a list
        s3_key = f"{context.asset_key.path[-1]}/{context.partition_key}.jsonl"
        with self.s3_client.open(s3_key, "wb") as f:
            for item in obj:
                f.write(json.dumps(item).encode() + b"\n")

    def load_input(self, context):
        s3_key = f"{context.asset_key.path[-1]}/{context.partition_key}.jsonl"
        return (json.loads(line) for line in self.s3_client.iter_lines(s3_key))

Then design your assets to yield items rather than return lists. Memory usage drops from O(N) to O(1) per asset, regardless of partition size.

Real-world example: Dagster + dbt for ecommerce intelligence

A scraping team built a Dagster pipeline that scraped 30 ecommerce sites daily and fed a dbt-managed warehouse for downstream analytics. The architecture treated scraped data as Dagster assets that materialized into Snowflake, and dbt models as downstream Dagster assets that ran SQL transformations against those tables:

from dagster import asset, AssetExecutionContext, DailyPartitionsDefinition
from dagster_dbt import DbtCliResource, dbt_assets, get_asset_key_for_model
from pathlib import Path

DBT_PROJECT_DIR = Path(__file__).parent / "ecom_dbt"
daily = DailyPartitionsDefinition(start_date="2026-01-01")

@asset(partitions_def=daily, group_name="raw_scrape")
def raw_amazon_products(context: AssetExecutionContext):
    date_str = context.partition_key
    products = scrape_amazon(date=date_str)
    write_to_snowflake("raw.amazon_products", products)
    context.add_output_metadata({
        "row_count": len(products),
        "preview": products[:5],
    })

@asset(partitions_def=daily, group_name="raw_scrape")
def raw_walmart_products(context: AssetExecutionContext):
    date_str = context.partition_key
    products = scrape_walmart(date=date_str)
    write_to_snowflake("raw.walmart_products", products)
    context.add_output_metadata({"row_count": len(products)})

@dbt_assets(
    manifest=DBT_PROJECT_DIR / "target" / "manifest.json",
    partitions_def=daily,
)
def ecom_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

The dbt project contained models like staging_amazon_products (cleans the raw scrape), mart_competitive_pricing (joins all sites for cross-comparison), and ml_features_pricing (rolling windows for the pricing prediction model).

In the Dagster UI, the asset lineage showed the full chain: scrape job -> raw tables -> staging models -> mart models -> ML feature tables. When the pricing prediction model returned anomalous results, the team traced the lineage backward and discovered the root cause was a 24-hour gap in the Walmart scrape (the site had pushed an HTML structure change that broke the parser). The lineage view turned a 4-hour debugging session into a 15-minute one.

Asset checks at each layer caught quality issues before they propagated:

@asset_check(asset=raw_amazon_products)
def amazon_row_count_check(context, raw_amazon_products):
    rows = query_snowflake(
        f"SELECT COUNT(*) FROM raw.amazon_products WHERE scraped_date = '{context.partition_key}'"
    )[0][0]
    return AssetCheckResult(
        passed=rows >= 5000,
        severity=AssetCheckSeverity.ERROR if rows < 1000 else AssetCheckSeverity.WARN,
        metadata={"row_count": rows, "expected_min": 5000},
    )

A check failure marked the partition as unhealthy in the UI and prevented downstream dbt models from running until the issue was resolved. Over six months in production, asset checks caught 47 distinct data quality regressions before they reached the analytics layer.

Comparison: Dagster patterns by scraping scale

A reference table of Dagster patterns that work well at different scraping volumes:

volume	pattern	partition strategy	runtime	notes
<100 URLs/day	single-asset, no partitions	none	local Dagster	simplest
1K-100K URLs/day	per-site assets, daily partitions	DailyPartitionsDefinition	self-hosted	sweet spot
100K-10M URLs/day	per-site assets, hourly partitions	HourlyPartitionsDefinition + multi-dim	k8s with autoscale	needs IO manager tuning
10M+ URLs/day	partitioned ops within assets	DynamicPartitionsDefinition	Dagster Cloud Hybrid + k8s	requires custom IO managers
Realtime triggers	sensors + asset reconciliation	none, sensor-driven	self-hosted	use Prefect instead if pure realtime

For most scraping workloads, the 1K-100K URLs/day band with DailyPartitionsDefinition is the right starting point. Migrate up the table only when partitioning becomes the bottleneck. Migrate down (to a simpler tool) if Dagster’s overhead exceeds your benefit.

Detection: when Dagster is the wrong choice for scraping

Five signals that your scraping workload should NOT live on Dagster:

1. Pure scraping with no downstream ETL: if all you do is scrape and dump to JSON files, Dagster’s asset model is overhead. Use Prefect or cron + scripts.
2. Heavy async workload (1000+ concurrent requests): Dagster is sync-first. Heavy async work runs better in Prefect or Celery.
3. Sub-minute scheduling: Dagster’s minimum schedule interval is one minute. Sub-minute requires sensors or external triggers.
4. Fluid pipeline shape that changes per run: Dagster’s asset model assumes a stable graph. Per-run dynamic shape is awkward.
5. Real-time event-driven only: pure event-driven workloads work better in event-streaming systems (Kafka, NATS) than in orchestrators.

If three or more of these apply, Prefect or a simpler tool is the right answer.

Performance tuning: parallel asset execution

Dagster’s default behavior runs assets sequentially within a job. For scraping where assets are independent (different sites), enable multi-process or k8s execution:

from dagster import multiprocess_executor, define_asset_job

scrape_job = define_asset_job(
    "all_sites",
    selection="*",
    executor_def=multiprocess_executor.configured({
        "max_concurrent": 8,
    }),
)

This runs up to 8 site-scraping assets in parallel within one job run. For Kubernetes-based execution, use k8s_job_executor which spawns one pod per asset:

from dagster_k8s import k8s_job_executor

scrape_job = define_asset_job(
    "all_sites",
    executor_def=k8s_job_executor.configured({
        "max_concurrent": 30,
        "container_config": {
            "resources": {
                "requests": {"memory": "1Gi", "cpu": "500m"},
                "limits": {"memory": "4Gi", "cpu": "2000m"},
            },
        },
    }),
)

Per-asset isolation prevents one runaway scrape from OOM-killing the entire job. The trade-off is pod startup latency (5-15 seconds per asset), which is fine for hourly or daily jobs but punitive for sub-minute schedules.

Wrapping up

Dagster excels when scraping is one node in a larger data pipeline. The software-defined asset model gives lineage, partitioning, and freshness tracking that pure orchestrators do not. For teams that already think in terms of data products and downstream consumers, it pays off. Pair this with our building scraping pipelines with Prefect 3 and distributed scraping with Apache Kafka writeups for the full pipeline picture, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.