Category: Uncategorized

Twenty two hours quoted. Ninety one hours spent. That was one job, and the parser inside it took an afternoon, roughly what I said it would.

Everything that killed the estimate sat outside the code. A login I had not checked for. Four page templates I did not know existed. Six weeks of quiet breakage after the invoice had cleared and I had moved on to something else.

I run mobile proxy lines and production scrapers out of Singapore, so I have written these quotes and I have been handed them by other people. Bad scraping quotes are all bad in the same way. The person quoting priced the code, and the code was never the risk.

The build is the only predictable part

Ask an experienced person how long it takes to fetch a page, pull eight fields out of it and write them to a table, and you will get a good answer. That work is bounded and repetitive. After thirty or forty jobs you can call it inside a couple of hours.

So the build gets estimated well, and then it turns out to be maybe a fifth of the job.

The other four fifths hide in four places, and every one of them is visible before you quote if you actually go and look.

The site moves while you build. Not usually a redesign, just a variant template appearing or an advert slot dropping in above the element you anchored on. A two week build hands the site two weeks to shift under a price you already committed to.

The anti bot layer nobody checked for. This is the one that converts a bounded job into an unbounded one. The fetch loop runs beautifully across two hundred pages on your laptop, and somewhere past page nine hundred the responses turn into challenge pages. Now the job needs residential or mobile addresses, a real browser it never needed before, and a crawl rate a fifth of what you planned. A run cost you priced at zero suddenly has a monthly invoice on it.

The long tail. Eighty percent of records go down one path and the remainder each want their own branch. An out of stock product with no price element at all. An address that arrives as one line of free text where the others give you five fields. A record in a second language. A page on a completely different template because that catalogue came in through an acquisition and never got merged. Twenty minutes each, forty of them, and none of it visible on the first page you opened.

The maintenance tail. You deliver, you invoice, you leave. Three weeks later the price column is blank and you are back in that repository for free, because the quote said “scrape this site” and never said for how long.

The hour before the number

Here is the inspection I run now before any figure leaves my mouth. It takes about an hour.

Pull ten pages by hand. No code, no crawler. Open them in a browser, read them, save the HTML to disk. Ten pages will tell you whether the data is in the HTML at all, whether the layout is stable across the ten, and whether page seven is the awkward one that breaks the assumption you were about to price.

Open the network tab before writing a single selector. If the page is filling itself from a JSON endpoint, the job just became several times cheaper and considerably more stable, and you want that knowledge before you price a parser rather than after. Give it twenty minutes. The endpoint is either there or it is not, and if it is not you go back to reading HTML and quote on that basis.

Make an account. This is the biggest hidden multiplier I know of, and checking costs two minutes. A login means sessions, session expiry, cookie handling per worker, rate limits counted per account instead of per address, and a real chance the site bans the account and takes the crawl down with it. It also drags terms of service into a conversation you were treating as routine.

Count the page templates. Actually count them, by clicking through twenty or thirty records and tallying what comes back. One template means the job is what you assumed. Six templates means six parsers and six sets of test fixtures, which is a different price rather than a slightly larger one.

Then ask the question people forget: how fresh does this data have to be.

Freshness is the architecture, and the architecture is the price

Same site. Same eight columns in the output file. Four completely different systems depending on the answer to that one question.

Monthly is a script and a line in cron. One person runs it, glances at the file, sends it.

Daily starts wanting retries, a place for failed pages to land, and a human who notices when a run does not finish.

Hourly means a queue, workers, deduplication and monitoring that actually pages someone, because a silent failure at hourly cadence produces twenty four bad files before anybody opens one.

Near real time is a separate discipline. Open connections, latency budgets, infrastructure that has to stay up rather than a job that has to finish.

The spread between the ends of that range is something like ten to one on price, for identical fields off an identical site. Which is why “we want it live” deserves a follow up question about what live means in hours. Half the time the client says live and means daily, and you have handed them most of their budget back before writing a line of code.

Two lines on the quote, always

Most of this is a commercial problem wearing a technical costume.

Quote the build as a build. A fixed number of hours to deliver a working collector against the site as it stands on the date of signing, an agreed field list, and an agreed sample that has to parse correctly before the job counts as done.

Quote the run as its own monthly line. Maintenance hours, proxy spend, the time a template repair takes, and a response time you will genuinely honour at 2am on a Tuesday.

If the client only wants the build, sell them the build, and put in writing that the collector stops working on the day the site changes and that repairs are billable. That single sentence has saved me more money than any code I have written.

A scraper is a subscription to somebody else’s website. You have no contract with the site owner, they do not know you exist, and they will redesign whenever their marketing team wants a new hero section. Every one of those redesigns lands on your desk.

Charge for the looking

Here is the position I get argued with about: quoting a fixed price for a site you have never opened is a way to lose money, and it feels safe only because the loss shows up months after the invoice clears.

So I bill for the inspection. A small fixed fee, an hour or two of work, and a written finding at the end. Where the data actually lives. How many page shapes exist. What the anti bot posture looks like. What each freshness option would cost to build and to run.

Then the build quote is a number I believe. And if the site turns out to be horrible, the client paid a couple of hundred dollars to learn that instead of several thousand to discover it slowly through me.

A client who refuses the inspection fee has told you something useful. Usually that four other quotes are on the table and the decision is price alone, which means the winner will be whoever understood the job least.

The one still costing me money

The login miss is the story I tell most often, but it is not the mistake that has cost me the most.

That one is a collector I delivered in early 2024 for a single fixed number, with no run line and no end date on my obligation. It is still running. I have repaired it four times since, twice for a template change and twice when the site tightened its rate limiting. None of those repairs were billed, because nothing in what I wrote gave me the standing to bill them.

The build was maybe fifteen hours. The unpaid tail is past twenty and still counting, and the only reason it has not gone further is that the client is pleasant and the site is boring.

The fix was one paragraph of scope language, and it took me an embarrassingly long time to write it.

Where an hour of looking runs out

An inspection does not make the estimate accurate. It makes it honest, which is a lower bar and a far more useful one. A site can still add a challenge page in month three, or migrate to a framework that paints everything client side, and nothing you observed in August tells you what it does in November.

Inspection also says nothing about whether the job is one worth taking. Public pages only, the robots file honoured, a crawl rate that does not hurt the site, and an official API or a bulk feed preferred every single time one exists. A decent share of the enquiries I get fail that test long before the estimate matters, and the right answer to those is a polite no.

I quote these jobs and then live inside them, so this is the checklist I actually run rather than something copied off a consultancy site. The rest of what I have written on scraping cost, pipeline design and the proxy infrastructure underneath it is over here.

Get new guides and videos first — join the Telegram channel.

I have paid for legal advice about scraping exactly once. One hour, one jurisdiction, one specific job. What I remember is that she would not answer until I had broken my question into pieces, because what I walked in with was four questions wearing one coat.

I am not a lawyer and this is not advice

I have never studied law. I run proxy farms and production scraping pipelines out of Singapore, and everything below came from reading terms of service, getting things wrong, and that one paid hour.

This varies enormously by country. It varies by what the data is, and by what you do with it after you have it. Two operators running an identical scraper in two jurisdictions can land in completely different places.

So if there is real money on it, or a client, go and pay somebody who knows your jurisdiction and your use case. An hour of that costs less than one letter from the other side.

I am also not going to quote court cases. I have read the same summaries as everyone else, I have never been in a courtroom, and secondhand case law from a scraper on the internet is worth nothing to you.

The four questions

Can you reach the page without an account.

Do the terms permit automated collection.

What may you do with the data once it is on your disk.

Who owns the content itself.

Those are separate questions. Answering one does not answer the others, and almost everybody who asks me about this has answered the first and assumed the rest came with it.

Public does not mean permitted

“Publicly reachable” means something narrow. No login, no paywall, no token, nothing you had to get past. The server handed the page to an anonymous request that turned up and asked.

That matters, because getting around a control somebody installed on purpose is where the serious exposure sits. It is the difference between walking through an open door and picking a lock.

But it settles one of four. It says nothing about the terms, nothing about redistribution, nothing about who wrote the paragraphs on the page. I treat it as a floor. No means stop. Yes means keep reading.

Robots.txt is a request, and I follow it

Nothing enforces that file. No gate, no token, no consequence at the protocol level. Your fetcher honours it only if you wrote the code that honours it, and plenty of people treat that as licence to skip it.

My reason for reading it is not the one you usually hear. That file is the only channel a site owner has to tell me anything before I arrive. They do not know who I am and they cannot email me. The conversation runs one way and it is the only one on offer.

If I read it and go anyway, I have surrendered any standing to complain about what happens next. Blocks, bans, a letter. I was told, in the only way available to them, and I went anyway.

One exception I will defend and some people will not like. I will fetch a disallowed path once, by hand, before deciding whether the pipeline goes near it. I do not point a crawler down it.

An account is a promise you made

The moment you register, you clicked something, and it usually said accept. A vague question about public data became a specific commitment in writing.

The same applies to an api key, which people find much easier to accept. Nobody argues that api terms are optional. A login is that shape with a worse interface.

So one page fetched two ways sits in two categories. Anonymous, you are a stranger the server chose to serve. Signed in, you are a party who agreed to something and then did the opposite. That second one is far easier to lose, because nobody has to prove intent or damage. They show the clause and they show the traffic.

There is also a person attached to an account. An email, often a phone, sometimes a card. Anonymous scraping is anonymous. Account scraping is you, by name, in somebody’s logs.

Collecting and using are two different jobs

You can be completely fine gathering something and completely wrong publishing it. Plenty of terms permit access and forbid redistribution a few lines apart in the same document.

Ask it as three questions. Can I fetch this. Can I keep this. Can I republish, resell, or train on this.

Internal analysis is the safest by a distance. A price history you use to set your own prices is a different animal from a comparison site you charge people to read. Resale is where I would want real advice every time. The moment somebody is paying for the data rather than for something you built on it, the conversation changes.

Facts and writing are not the same material

Nothing about reachability or terms answers the copyright question.

The rough shape, read by a layman. Facts are hard to own. A price is a fact. A stock count is a fact. A street address is a fact.

Expression is owned. A nine hundred word review somebody sat down and wrote is theirs. A product photograph belongs to whoever shot it.

So I store facts and a reference back to the source, and I leave the prose alone. Copyright is the easiest of the four to get right, which is why scraped article farms irritate me more than they should.

Personal data is where the weight actually is

Names, emails, phone numbers, photographs, profile pages, anything that points at a specific living person. That category is regulated in most jurisdictions with real teeth, and how visible it was does not undo that.

Singapore has the Personal Data Protection Act. Europe has gdpr. California has its own. They differ in the detail and agree on the principle: the fact that anybody could see it does not mean you may collect it and build on it.

The underrated half is holding it. Collecting takes an afternoon. Holding is forever. You have to secure it, know where every copy lives, delete it when somebody asks, and answer for it if a backup ever walks out of the building.

The risk is also lopsided in a way nothing else in this work is. Get a product price wrong and a dashboard is wrong for a week. Get personal data wrong and there is a regulator in the room.

So the rule is blunt. I do not collect personal fields unless those fields are the entire point of the job. Otherwise they get dropped at the parser, before anything reaches storage.

Rate is an ethics question with its own answer

This one sits outside the four and day to day it matters more than any of them. A scraper that costs somebody real money in serving costs is doing harm even where it is entirely permitted.

I have an unusual angle on this because I sell bandwidth. Proxy traffic on my own lines is metered by the gigabyte, so I know what a gigabyte costs to move. When a headless browser loads a full page it pulls the html, the css, every javascript bundle, the fonts and the images, and the site on the other end is paying its own version of that number to send it. Run two hundred of those in parallel and you have moved a line item onto somebody’s invoice.

On a large platform that disappears into the noise. On a small site on shared hosting it is the whole box, and I have watched one of my own go down exactly that way.

So I run slower than I need to. One request at a time per host, a delay that turns an hour into a night, scheduled for their quiet hours rather than mine.

The five rules I operate by

Prefer an official interface. If there is an api I use it, and if it is metered I pay. A paid channel is the site stating in plain numbers how much automated traffic it wants.

Identify yourself. I put a real name and a contact address in the user agent for anything I am not embarrassed by. Most scraping advice says the opposite, blend in and look like a browser, and that was written for a different kind of job. If you would not sign your name to the traffic, you have your answer.

Go slowly, per above.

Take only what you need. Every extra field is something to justify later and something that can be quietly wrong for months.

Be willing to stop. If a site asks me to stop, I stop that day, and I do not argue about whether they were entitled to ask.

The thing I collected and should not have

I was tracking listing prices across sellers on a marketplace. Three fields mattered: price, currency, timestamp.

But I stored the raw json response whole, because that is exactly what I tell everybody to do. Keep the payload, parse from it later, never lose what you fetched.

That response carried the seller profile with it. Display names, rough locations, contact handles, join dates, review histories. I never used one of them. I never really looked. I just had them, on a box in my house and inside every encrypted backup taken since, for months.

The fix took an afternoon. An allowlist at the fetch step so the raw store only keeps declared fields, then a pass over the old objects to strip and rewrite.

What bothers me is that there was never a decision. I did not weigh it up and choose badly. My default was keep everything and sort it out later, and for personal data that default is the mistake.

The one I still cannot answer

When a page fetches its own data from a json endpoint and no account is required to call it, I treat that endpoint as publicly reachable, same as the page. That is my working position.

The argument against it is that the endpoint was never published for anyone but their own front end, so calling it directly is a different act from loading the page. I do not think that argument is stupid. I have just never seen anything that settles it.

So I behave identically either side of that line. Slow, identified, terms read, no personal fields kept. The full set of guides and the tooling I actually run is at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

My first run scored Firecrawl at 53.3%. That number was wrong by nearly fifty points, and every one of the reasons was mine, not theirs.

I want to open with that because it is the useful part of this write-up. The success rate at the bottom is one provider, on one date, from one connection. The five ways a test harness can quietly libel a tool are general.

Disclosure first

Firecrawl gave me API credits to run this. That buys sample size and nothing else. No review of the draft, no say in the verdict, no agreed placement. I asked for the credits because the free tier is 1,000 credits a month and this run does not fit inside that, for reasons that turn out to be worth a section of their own.

The account is on the free plan otherwise. Version tested: the v2 API (/v2/scrape), 5 and 6 September 2026, from a business line in Singapore.

The method

Same six categories the earlier comparison on this site used: e-commerce, SERP, social, travel, real estate, business listings. Five URLs per category, thirty URLs, four repeats each. N=120.

Every request is the same call:

{"url": "...", "formats": ["markdown"], "onlyMainContent": true,
 "proxy": "auto", "timeout": 60000, "maxAge": 0}

maxAge: 0 matters. Leave it out and you may be scored on Firecrawl’s cache rather than on a live fetch, which measures their storage layer and not their ability to get through a door.

A response counts as a success only if the markdown contains a marker that could not appear on a block page or an error page: “add to cart” for a product page, “guest reviews” for a hotel, a follower count for a profile. Length alone is not enough. A Cloudflare interstitial is a perfectly well-formed 200 with several thousand characters in it.

The full target list and the raw per-request JSONL are available on request. Ask and I will send them, then re-run it and argue with me.

The five bugs, because they are the transferable part

Rate limiting read as failure. The free plan caps at 10 requests a minute. My first pass fired faster than that, and 8 of 30 requests came back as rate-limit errors. Naively bucketed, that scored X.com at 0% — a category “failure” that was entirely me holding the throttle down. Any benchmark that does not say what its request rate was, and what it did with 429s, is not telling you about the provider.

A block-detector that matched the content. I flagged blocks by looking for “captcha” and similar strings. One of my SERP targets was a Google search for “captcha solving service”. The query echoes into the page, the page is 39,000 characters of perfectly good results, and my detector called it a block.

The same bug wearing a different hat. Zillow’s sign-in copy contains “Takes just a moment.” My detector was looking for “just a moment”, which is the Cloudflare interstitial title. Another 50,000-character success scored as a block.

Dead targets. Three Amazon ASINs in my list had been delisted and returned Amazon’s real 404 page. One Booking.com slug was gone. Those are 404s from the target, not failures from the provider, and counting them against Firecrawl is just wrong. They now get their own verdict and sit outside the denominator.

Running out of money. The one I did not see coming. Halfway through the final run my account hit zero and the API started returning 402. Those 31 requests went into the results file as errors, and the summary line dutifully reported 73.3%. My own balance, scored as Firecrawl failing to fetch a page. It now stops the run when it sees a 402 and keeps those rows out of the denominator, which is the difference between 73.3% and the 98.9% below.

Fixing the first three, replacing the dead URLs, and taking the 402s out moved the same provider on the same targets from 53.3% to 98.9%. Nothing about Firecrawl changed in between.

The general lesson is not subtle: when a benchmark and a vendor disagree, the benchmark is the thing that has not been tested.

What it actually scored

category	scored	ok	success	median	p90
e-commerce (Amazon)	20	20	100%	7.49s	9.21s
SERP (Google)	20	19	95%	4.50s	5.21s
social (X.com)	20	20	100%	7.77s	11.29s
travel (Booking)	20	20	100%	5.27s	6.63s
real estate (Zillow)	9	9	100%	5.99s	9.59s
business listings (Yellow Pages)	0	–	no data	–	–
all	89	88	98.9%	5.85s	9.21s

Read the last two rows before the headline. I ran out of credits at request 90 of 120, so Zillow got 9 of its 20 attempts and Yellow Pages got none at all. Yellow Pages is a gap rather than a result. The honest thing is to leave the cell empty rather than quietly drop the category and report a rounder N.

One failure in 89. A Google SERP that came back as 377 characters when the same URL returned 61,814 characters on its other three attempts. Not a block page, not an error, just a stub with a 200 on it.

That is the failure mode worth caring about, and I want to be honest that I cannot tell you what was in those 377 characters, because my harness recorded the length and threw the body away. It saves failure bodies now. That is the kind of thing you only discover by having a failure.

The pricing thing nobody mentions

Credits are not requests. proxy: "auto" escalates to a stealth proxy when a target pushes back, and a stealth fetch bills at several credits instead of one. Two runs, two very different rates. A mixed run of about 102 requests across all six categories took my balance from 1,092 to 677, near enough four credits a request. The final run was weighted towards the hard end (twenty Amazon, twenty X.com, twenty Booking.com) and burned the remaining 677 credits in 89 requests. That is 7.6 each.

So on defended targets the 1,000-credit free tier is worth somewhere between 130 and 250 scrapes, depending entirely on what you point it at. The docs do say stealth costs more. The gap is that the figure you actually burn lands at several times the figure you budgeted, and you cannot work out which multiple in advance because the API does not tell you per request what it charged you. Every comparison table I have seen, including the one on this site, quotes list price per request as if the multiplier were not there.

If you want predictable billing, pin proxy: "basic" and accept the failures. If you want the success rate, budget 4x.

Limits of this run

N=120 is small. It is one provider with no control arm running beside it, so “98.9%” is a statement about these thirty URLs on these two days and not a ranking. Free-tier concurrency is 2, so this says nothing about behaviour at fifty parallel requests, which is where managed scrapers usually start to hurt. Everything ran from one Singapore connection; a US or EU egress may see different treatment on the same targets.

I have not tested the crawl or extract endpoints at all. Only /v2/scrape.

The number I would actually watch

Not the success rate. The spread.

The same Google SERP URL, four consecutive fetches, minutes apart:

URL	chars across 4 repeats
?q=best+web+scraping+api	19,939 to 55,983
?q=residential+proxy+pricing	25,698 to 66,443
?q=datacenter+vs+mobile+proxy	377 to 61,814

Every one of those is scored a success except the 377. A page can be a third of its own size on the next call and still tick every box a success check looks for. Latency moved too, though far less dramatically: median spread within a URL was 1.42s, worst was 7.60s.

A provider that returns a full page nine times and a stub on the tenth will pass a single-shot benchmark and then quietly poison a week of your data, because the stub is a 200 and your pipeline has no reason to think anything went wrong. Which is the argument for schema validation on the way in, and the argument against trusting any success rate (mine included) that was measured once per URL.

A scraper logs into a site, adds an item to a cart, gets two pages into checkout, and then gets logged out. Nothing in the code changed. What changed was the address underneath it. The proxy rotated in the middle of the flow, the site saw the same logged in cookie arrive from a new city, and it dropped the session. That single behavior, whether the proxy holds an address or swaps it, is what rotating and sticky sessions are about, and picking the wrong one is one of the most common ways a working scraper starts failing for no obvious reason.

I run proxy infrastructure and production scrapers, so this is a setting I have gotten wrong and had to fix, not a diagram off a vendor page. Rotating and sticky are not two products you buy. They are two ways of using the same pool. Here is what each one actually does at the proxy layer, how long sticky windows really last, and how I decide between them per job.

What a session actually is

Both settings revolve around one idea, so start there. To a site, a session is a sequence of requests it treats as one visitor. Two things carry that identity. One is the cookie the site handed you at login. The other is the address that cookie keeps arriving from. A careful site cross checks the two. So the identity your scraper presents is not just the cookie, it is the cookie and the ip together, and the moment those two stop agreeing, the site has a contradiction it can act on.

That framing is the whole article in one sentence. Rotation and stickiness are just choices about whether the address stays consistent with the cookie or not.

How rotation works at the proxy layer

Rotation means the gateway hands you a fresh address on some schedule, or a brand new one on every request. You point your client at a single endpoint, and behind that endpoint the provider swaps which upstream address your traffic actually leaves from. The goal is to spread load so no single ip gets hammered, and to keep any one address from building a reputation under your job.

For wide, stateless collection that is exactly what you want. Each page stands alone, there is no session to break, and spreading requests across many addresses keeps a single ip from tripping a rate flag. The cost is that rotation, by design, keeps changing where you appear to come from.

How a sticky session works

A sticky session is the opposite setting on the same pool. Instead of a fresh address each time, the gateway pins one address to you and holds it for a window. You usually request it by putting a session id into the proxy username, or by connecting to a dedicated sticky port. Every request you send through that token leaves from the same upstream ip until the window ends.

Under the hood it is a lease. When you send a session token, the gateway looks it up, reuses the address mapped to it, or picks one and remembers the mapping if there is none yet. Hold the token steady and you hold the address. Let it change or expire, and the gateway is free to hand you someone new. Like any lease, it has a length, and it runs out.

How long sticky windows really last

Sticky windows are shorter than most people assume. A lot of residential pools default to something in the one to ten minute range, and many let you request up to 30 minutes or an hour if you ask for it. Datacenter addresses can be held far longer, because you effectively own that address for the rental.

Residential and mobile stickiness is best effort. You are borrowing a real home line or a real phone, and that connection can drop at any time and take your window with it early. Mobile is looser still, because carrier networks reassign addresses on their own schedule and share each one behind nat. So the same sticky setting can give you a rock steady address on datacenter and a window that quietly resets on residential or mobile. Design for the address changing, not for it to last.

The two failure modes

Both failure modes come from mismatching the setting to the job.

Rotate in the middle of a stateful flow and you get the logout from the opening. You log in from one address, and two requests later the same session cookie shows up from a different ip in a different city. No real person teleports between clicks, so the site ends the session or throws a challenge. The flow breaks, and it looks random until you realize the address moved under you.

Hold one sticky address across a huge stateless crawl and you make the opposite mistake. You funnel thousands of requests through a single ip that a rotating pool was built to spread. That one address climbs the rate counters fast, earns a reputation by itself, and gets throttled or blocked while the rest of your pool sits idle. Stickiness concentrates trust on one door, which is wrong when the job never needed a door held open.

Keep one session’s identity together

A clean mental model prevents most of this. Treat one logical session as one bundle: a single session token, a single cookie jar, one consistent set of headers, all pinned to one address for the window. The bug I see most is a shared cookie jar bleeding across rotating addresses, so the same login state leaks out of three different ips inside a minute. That is a contradiction you built yourself.

Geography is part of that identity too. A sticky window should stay inside one city, and ideally one network, so a flow does not jump from a Singapore address to a US address halfway through checkout. Prices and availability shift by location, so a wandering session can also quietly corrupt the data you collect, not just get you flagged.

Concurrency does not require rotating inside a session

People worry that stickiness kills throughput. It does not, you just parallelize differently. Each sticky session is one worker holding one address for one flow. To run 50 logged in flows at once, you open 50 sticky sessions, each with its own token, address, and cookie jar. You get concurrency from many held addresses side by side, not from rotating inside a single session. Stickiness and scale are not in conflict.

How I decide per job

The decision comes down to one question: does this flow carry state the site ties to the address? If yes, a login, a cart, anything sequential, I go sticky and size the window to the length of the flow with a margin. If no, a wide stateless pull, I rotate, often on every request, to spread the load. Then I test on the real target, because a site can tie a session to the ip harder or looser than you would expect.

Two jobs I ran back to back make it concrete. One was a login gated account area, a few pages deep per visit. Rotating per request logged me out every second or third page, until a 10 minute sticky window let each visit finish clean. The other was a broad price crawl across tens of thousands of public pages. A sticky address there hammered one ip into a rate block within the hour, while per request rotation spread the same work quietly across the pool. Same tool, opposite settings.

The honest limits

Matching rotation to the job keeps your sessions intact and your load spread. It does not make anything undetectable, and no setting on a proxy does. It does nothing about a loud client, a bad tls handshake, or a request rate no human would produce. And it changes nothing about what you are allowed to collect. Public data, the robots file respected, an official api preferred where one exists, and whatever a site’s terms actually say. Rotation and stickiness are plumbing, not permission.

I run this infrastructure in production, so the way I pick between rotating and sticky here is the way I actually run the jobs. If you want the full written guides and the pools and settings I genuinely use, with real block rates and no undetectable promises, they are at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

A captcha appears, and almost everyone reads it as a puzzle. Prove you’re human, click the crosswalks, check the box, wait for the spinner. So the instinct is that the captcha is the test, and if you can just answer it, you’re through. That instinct is backwards. By the time a challenge renders on your screen, a decision about your session has already been made, and it wasn’t leaning your way. The captcha is not the test. It’s the symptom of a test you already failed somewhere upstream.

I run proxy infrastructure and production scrapers, so I’ve spent a lot of time watching how these challenge systems decide who gets waved through and who gets stopped. This is a defensive explainer, written from the site’s point of view: how modern captchas actually work, and why the whole market of paying farms to solve them is a dead end. It isn’t a guide to beating a challenge, because beating the challenge was never the real problem, and once you see how the pieces fit, you’ll understand why.

What a captcha is actually for

Start with the purpose, because it isn’t what the name suggests. A captcha isn’t there to check whether you can read distorted text or spot a bus. It’s there to raise the cost of being an automated client at the exact moment a site is unsure about you. It’s a decision point. The site has some doubt, and rather than block you outright or let you through, it inserts a small tax that’s cheap for a real person and expensive for a machine.

The classic captcha, the wall of warped letters, worked while computers were bad at reading messy text and people were good at it. That gap closed. Modern machine vision reads distorted text more reliably than humans do, so the warped letters stopped separating anyone from anything. That’s the first clue about this whole space: any challenge that’s purely a puzzle eventually gets solved by software. So the providers moved the real work somewhere else.

The real shift: scoring, not puzzles

Here’s the change that matters most. A modern captcha system doesn’t challenge everyone. It scores everyone, quietly, and only shows a visible challenge to the small slice it’s unsure about. Most visitors never see a puzzle at all, because the system already decided they were fine. The visible challenge is the last resort, reserved for traffic that already looks doubtful. So seeing one at all means you’ve landed in the doubtful bucket.

The clearest version of this is the invisible scoring model, the reCAPTCHA v3 style approach. It never shows a puzzle. It sits on the page, watches the session, and hands the site a number, a risk score, roughly how confident it is that you’re a real person. The site owner decides what to do with that number. A good score, let them through. A bad score, ask for a password again, hold the action, or escalate to a visible challenge. The scoring happens in the background, before any decision is shown to you.

hCaptcha, which a lot of sites moved to, works on the same principle. It leans on a privacy story and its own scoring, but underneath it’s still risk based. Cloudflare Turnstile pushed the idea further by trying to drop the image puzzle almost entirely. It runs a series of small checks inside your browser, quiet probes a real browser passes without the user doing anything. You might see a checkbox and a brief spinner, or nothing at all. What it’s really doing in that pause is measuring whether the thing loading the page behaves like a genuine browser or only claims to.

That’s the part people miss about the little “I’m not a robot” checkbox. Clicking a box is trivial. What the widget measures is everything around the click: how the pointer arrived, the timing, the state of the browser, and the score it already computed for the session. A real person with a clean session gets a single click and a green tick; a suspicious session clicks the same box and gets a hard image grid, because the click told the system nothing good enough to skip it.

What actually feeds the score

So what goes into that number? The same stack that drives bot detection generally. The reputation of the ip and the network it sits on. Whether the tls handshake matches the browser it claims to be. Whether the request headers are consistent with each other. What the browser reveals when the provider runs code inside it: the canvas, the fonts, the timing, the small properties a real device has and a cloned one struggles to fake. And how the session behaves over time, the rhythm of the clicks and the moves.

None of those is the captcha. They’re the inputs that decide whether a captcha ever appears. Which leads to the reframe: a captcha is a readout that you already tripped something upstream. By the time the challenge renders, the network you came in on, your handshake, your fingerprint, or your pace has already pushed your score into the range where the site stopped trusting you. The puzzle is downstream of the score. And that single fact is what makes the solver industry a losing proposition.

Why solver farms are a dead end

Solver farms come in two flavors. Human farms, rooms of low paid workers who type the answer for a fraction of a cent. And machine solvers, models trained to answer the common challenge types automatically. Both do the same narrow thing: they take the puzzle that got served and hand back the expected answer. The puzzle in, the answer out.

Here’s why that doesn’t work in any durable way. The puzzle was never the problem. The low score that summoned the puzzle is the problem, and the solver does nothing about it. You can answer the grid perfectly and still hand the site a session that arrived from a flagged network with a cloned fingerprint moving at machine pace. The site scores the token the challenge produces in the context of that whole session, and a right answer from an obviously automated context still looks like exactly what it is.

The economics run against the solver too. Issuing a harder challenge is nearly free for the provider. Solving one costs the farm real money and real time on every request. When a solver service gets good at one challenge type, the provider changes it. The provider also sees this pattern across millions of sites at once, so a trick that starts working on one gets neutralized everywhere within a short window. You’re renting a solution to a problem the other side can reprice against you whenever it wants.

There’s a cleaner reason to stay away from that market. Paying a farm to defeat a site’s challenge is, on most sites, a direct violation of the terms you agreed to. Beyond being brittle, it’s usually against the rules, which makes it indefensible the moment it matters. And it’s fragile: one update from the challenge vendor and a pipeline built on a solver stops overnight. That isn’t infrastructure. It’s a bet you keep having to make again.

The honest fix is to not trip it

The durable fix for captchas is the opposite of what the solver market sells. It’s not defeating them, and it’s not solving them. It’s not tripping them in the first place. A challenge is a signal that your session already looked wrong upstream, so the work that lasts is upstream, making the session not look wrong. That’s the only move that doesn’t decay the week after you make it.

In practice that’s the boring path I keep coming back to. Arrive on a clean network with an honest reputation instead of the cheapest datacenter pool. Identify your client honestly where a site expects it, rather than dressing a server up as a browser it isn’t. Collect public data only, nothing behind a login, a paywall, or anything personal. Move at a pace the site can absorb, space the requests, back off on errors, and cache so you never ask twice. Prefer an official api or data feed when one exists, because that front door doesn’t score you and doesn’t care about your fingerprint at all.

I want to be straight about the limits. None of this makes anything undetectable, and I wouldn’t trust anyone who tells you it does. Detection and scoring keep improving, and what passes quietly today can be flagged tomorrow. A clean network and honest behavior aren’t a trick that beats the wall. They’re the absence of the contradictions that get you scored down in the first place.

I run this infrastructure in production, so the frameworks, proxy setups, scraping apis, and honest tool reviews I write about are the ones I actually deal with. If you want the full written guides and the compliant patterns, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the way this actually works.

Get new guides and videos first — join the Telegram channel.

A scrape that runs once is a script. A scrape that has to run every day for a year, without anyone watching, is infrastructure, and the layer that turns one into the other is scheduling. It decides when your scraper runs, what happens when it fails, and whether you find out the day it breaks or three weeks later from stale data.

I run production scrapers, so I have used every rung of this ladder, from a single cron line on one box to a full orchestrator managing dozens of jobs. Here is how I think about the three real options, cron, Airflow, and Dagster, and how to pick one without overbuilding.

What scheduling actually has to do

It sounds like one job, run this at 6am, but the real work is underneath. A good scheduler remembers whether the last run finished. It retries a failed step without redoing the ones that worked. It stops two runs from colliding when one runs long. It tells you when a job broke, and, harder, when a job succeeded but returned nothing. And it lets you go back and reprocess history when you fix a bug.

Cron does the first sentence. Everything after it is why the heavier tools exist.

Start with cron

Cron is the scheduler already living on every unix box you own. You give it a line, a time and a command, and it runs that command forever. No service to install, no database, no dependencies. For a single scrape that pulls one source once a day into one table, cron is not the lazy choice, it is the correct one.

The trouble is everything cron does not do. It has no memory, so a failed run vanishes and the next starts fresh. It has no retries, so one network blip loses you a day. It will start a new run while the last is still going, and now two copies fight over the same table. And its output goes nowhere you will look, so a broken job surfaces days later as stale data rather than an alert.

Make cron last a little longer

If you only have a job or two, you do not have to abandon cron, you shore it up:

• Wrap the command in a script that takes a lock so runs cannot overlap.
• Make that script exit non-zero and notify you when the work fails.
• Send output to a real log file, not the void.
• Add a heartbeat, an external service you ping only on a clean finish, so a run that never happens alerts you by its silence.

That last one matters most. The scariest failure is not the job that crashes loudly, it is the job that silently never fires.

When you have outgrown cron

Move up a rung when any of these is true: you have steps that depend on each other and want to retry just the one that broke, you need to reprocess last month because you fixed a parser, you are running more than a couple of jobs and cannot remember what runs when, or a second person needs to see what is happening without reading your crontab.

The tools above cron are built around a dag, a directed acyclic graph. That just means steps with an order, where each points to the ones that must finish first: fetch before parse, parse before store. Once the scheduler understands that shape, it can run steps in order, parallelize the independent ones, retry a single failed step, and skip what already succeeded.

Airflow: dags and backfills

Airflow is what most teams reach for first. You write your dag in python, Airflow runs it on a schedule, and you get a ui where every run is a row of green and red boxes. Each task has its own retries and delays, and when one fails you click in, read the log, and rerun just that piece. It has been doing this for years, so almost every database, api, and cloud service has a ready made connector.

Its real superpower is the backfill. Airflow treats each run as owning a slice of time, so when you fix a parser and need to rebuild the last 30 days, you tell it to run that date range and it reprocesses each day in order. That only works if your steps are idempotent, but when they are, repairing history is one command instead of a manual slog.

None of that is free. Airflow is a scheduler, a metadata database, workers, and a web server, several moving parts to install, secure, upgrade, and babysit. Standing all of that up to run one daily scrape is like buying a forklift to carry your groceries. It earns its keep when you have many jobs and several people depending on them.

Dagster and Prefect: data assets and observability

Dagster and Prefect came later with a different framing. Instead of thinking only about tasks that run, Dagster thinks about the data assets those tasks produce, the actual table or dataset you care about. You declare the asset and the code that builds it, and Dagster tracks what it depends on and how fresh it is.

That framing fits scraping well. You do not really care that a task ran, you care whether your products dataset is current and correct. Dagster lets you ask exactly that, shows the lineage behind each asset, and leans on typing so a run that returns the wrong shape gets caught instead of silently writing nulls. Prefect takes a lighter path, wrapping your existing python functions as flows with little ceremony, which suits irregular, dynamic pipelines. Both give you nicer local development and clearer visibility than the older model.

Tool	Best for	Gives you	Costs you
cron	one or two simple jobs	zero setup, runs anywhere	no memory, retries, or visibility
Airflow	scheduled dags and backfills	per-task retries, a run ui, huge connector ecosystem	a scheduler, database, workers, and web server to run
Dagster / Prefect	typed data assets, freshness, lineage	asset observability, typing, strong local dev	newer, another system to learn and host

Retries, idempotency, and the silent empty run

Three things matter no matter which tool you pick.

Put retries in the scheduler, not just the scrape. Your fetch function can back off on a single flaky request, but the scheduler retrying a whole failed task with a growing delay is what saves a run when a site has a bad five minutes.

Make every job idempotent. Running the same job twice must do no harm, or retries and backfills become a gamble. Key records on a stable id and write with an upsert that overwrites cleanly instead of appending.

And catch the silent empty run. The worst failure is not a crash, it is a job that finishes green with zero rows because the site changed a selector. Cron and even a naive Airflow setup call that a success. So count what you collected and compare it to normal, and if you usually pull thousands and today you got 12, fail the run on purpose. Alert on a whole run failing, a success rate falling off a cliff, zero rows where you expected thousands, and a run that never started at all.

How to choose

Match the tool to the scale and the team, and do not overbuild before you need to.

• Solo, one or two scrapes, no complex dependencies: cron with a lock and a heartbeat. Done.
• A handful of interdependent pipelines, backfills, or a small team that needs shared run history: Airflow, because it is proven and everyone can learn it.
• A data heavy setup where freshness, lineage, and typed outputs matter more than raw scheduling: Dagster, or Prefect when the work is irregular and pythonic.

The most common mistake is not too little tooling, it is standing up a whole orchestrator to run a scrape a single cron line handled fine. You can always climb the ladder when the pain is real.

The honest limits

Scheduling makes your collection reliable, not permitted. Running a scraper on time and recovering it cleanly does not change what you are allowed to take. Public data, a robots file respected, a gentle pace, and an official api or bulk feed preferred when one exists, those rules do not move because your orchestrator is nicer.

I run all three of these in production, cron for the small stuff and a real orchestrator for the pipelines that feed everything else. If you want the worked versions, the Airflow and Dagster setups, the idempotency patterns, and the alerting checks that catch an empty run, along with the tools I actually use, they are all at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

Most scrapers reach for a headless browser by default, and it is the most expensive habit in this work. They spin up a full Chromium for a page that would have handed the same data to a plain http request, then pay for that browser on every page, forever. The decision that matters isn’t which browser tool to use. It’s whether you need a browser at all, and most of the time you don’t.

I run production scrapers, so I’ve paid the browser bill when I didn’t have to and watched it show up in the monthly invoice. This is how I decide, where the real cost hides, and the hybrid I run so a job stays cheap where it can and only pays for a browser where a site forces one.

The one test that decides it

Open the page in your browser, view the raw source, and search that source for the thing you want to collect. If the price, the title, the number is already sitting in the html the server sent, you almost never need a browser. If the source is a near empty shell and the content only fills in after the scripts run, then you need something that executes those scripts.

That one look at the raw source settles the large majority of these calls, and it takes about ten seconds. Everything below is really just the consequences of getting that answer right or wrong.

What an http client does, and what it costs

An http client is the simple side. It opens a connection, sends a request, and hands you back exactly the bytes the server returned. It doesn’t run javascript, it doesn’t build a page, it doesn’t know what a button is. It’s just the raw response.

That’s why it’s so light. There’s no browser eating memory, no rendering, no waiting for scripts, so one modest machine can run hundreds of these requests at once. When the data lives in the server html, an http client is the whole job, and it’s fast and cheap. Libraries like requests and httpx do this, and a framework like Scrapy is built on the same idea at scale.

What a headless browser does, and what it costs

A headless browser is the heavy side. It launches a real Chromium engine with no window drawn on screen, loads the page the way a person’s browser would, runs all the javascript, builds the dom, and lets you read the page after it has fully rendered. Playwright, Puppeteer, and Selenium all drive a browser like this.

That full render is the entire reason it exists, and it’s the entire reason it costs so much. A raw http request costs a few milliseconds and a sliver of memory. A full browser page costs hundreds of megabytes of memory, real cpu, and seconds instead of milliseconds.

The gap in real numbers

The difference isn’t small. Per page, a browser routinely costs ten to fifty times the resources of a raw request, and that gap compounds at scale.

	http client	headless browser
Runs javascript	no	yes
Memory per page	a few mb	hundreds of mb
Time per page	milliseconds	seconds
Pages at once, one box	hundreds to thousands	a handful
Breaks when	html layout changes	browser updates, timing, memory

Put a job on it. A million pages with a raw http client on a modest server finishes in hours for a few dollars. The same million pages with a full browser each can need a fleet of machines and a bill many times larger, plus the constant work of keeping all those browsers healthy.

Check for a background api first

Before you assume a page needs a browser, open the developer tools and watch the network tab while the page loads. Very often the content you want arrives as a clean json response from an api the page quietly calls in the background.

When that’s happening, you don’t need to render anything. You call that json endpoint directly with a plain http request and skip the browser entirely. This one habit has saved me more compute than any tool choice, because the cheapest page to collect is the one you never render.

The hybrid: http first, browser only where forced

In practice it’s rarely all browser or all client. I run a fast http client across the whole site for the pages that hand over their data plainly, and I fall back to a browser only for the specific pages that come up empty. So the heavy cost lands on the handful of pages that genuinely need a render, and everything else stays cheap.

A browser really is the only door in a few cases: when the content is rendered on the client and never appears in the raw source, when the data hides behind an interaction like a click or an infinite scroll, or when the page is a single page app that builds itself entirely from javascript. In those cases an http client sees the empty shell and there’s nothing to parse. The rule I follow is simple: crawl wide and cheap with raw requests, render narrow and expensive only where the html arrives empty.

If you do land on a browser, the tool matters less than people think. Playwright and Puppeteer both drive Chromium cleanly, Selenium works but feels older, and I default to Playwright. The expensive part is never which one you pick, it’s the browser itself.

The maintenance burden of browsers at scale

The cost that doesn’t show up in a benchmark shows up at three in the morning. An http client is boring in the best way: it breaks when a site changes its html, and the fix is a quick selector update.

A browser breaks in far more ways. Chromium updates and a flag you relied on changes. A page gets slower and a timing assumption that held last month starts flaking. Memory leaks creep in over a long run until the process bloats and dies. Running browsers at scale becomes its own operations job: managing a pool of them, restarting them before they leak, killing the ones that hang, catching zombie processes, pinning a browser version, and wrapping every page in a timeout so it can’t hang forever. None of that exists on the http side.

There’s a subtler tax too. An http response either arrives or it doesn’t, so you know when you’re done. A browser page is never obviously finished, because scripts keep firing and content keeps arriving, so you write waits, and every wait is time you spend on every single page.

The detection angle, honestly

People ask which side is harder to detect, and it’s mostly the wrong question. Sites decide on the network you arrive from, the handshake your client sends, and the rhythm of your behavior, and both a client and a browser can be loud or quiet depending on how you run them. A real browser engine does produce a more browser like fingerprint, which is one honest reason browser automation exists.

But none of that makes anything undetectable, and this isn’t a guide to evading anything. It cuts the other way too: a headless browser in its default mode leaves its own tells, from the automation flags it sets to the too clean, repeatable shape of its fingerprint. A browser isn’t automatically safer than a client, it just fails in different places. The durable move on either side is to collect public data at a polite rate, respect the robots file, prefer an official api, and stay off the adversarial path entirely.

How I decide, every time

The ladder is short. View the raw source and check whether the data is already in the server html. If it is, use a plain http client and stop there. If it isn’t, open the network tab and look for a background json api you can call directly, and if it exists, use that. Only if both of those fail, when the data truly only exists after the browser renders it, reach for a headless browser, and even then render only the pages that need it.

Whichever side you land on, validate what comes back. The failure that hurts most is the one that doesn’t throw an error, where a client gets a soft block page or a browser reads the dom too early and hands you rows full of nulls. Confirm the fields you expect are present and in a sane range, and fail loudly when they aren’t.

I run this exact setup in production, so the frameworks and tradeoffs here are the ones I actually deal with, not theory. If you want the full written guides, the hybrid configs, and the picks I use and test, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the cheapest tool that gets the data.

Get new guides and videos first — join the Telegram channel.

Three services, one pitch: send a URL, get the page back, and let someone else own the proxies, the browsers, and the retries. ScraperAPI, Zyte, and Bright Data Web Unlocker all sell that promise, and on their landing pages they read almost the same. They aren’t the same. I paid for all three and ran them against the same targets on real jobs, and the differences show up the moment a site fights back.

I run my own proxy infrastructure and production scrapers, so this is a tested comparison, not a vendor roundup. None of these makes scraping undetectable, and none of them makes anything legal that wasn’t already legal. What they sell is convenience behind one endpoint. The only question worth answering is whether that convenience earns its price on your particular job.

The verdict, up front

Tool	Best for	Pricing shape	Hard-target success	Latency	Watch out for
ScraperAPI	Ordinary and medium sites, least setup	Credits per request (more for render or premium)	Lowest of the three on my hardest target	Fastest on simple fetches	Retries on hard sites eat credits
Zyte	Scraping as a standing operation	Closer to per successful request	Strong, a step behind Web Unlocker	Middle	Steeper learning curve
Bright Data Web Unlocker	The sites that beat everything else	Successful request, bandwidth driven	Highest on my hardest target	Slowest per page	Bandwidth pricing climbs fast at scale

Read that table as one test on one set of targets over a few days. Success rates move as sites change their defenses. What holds is the shape: the more specialized and expensive the tool, the better it did on the one site that fought hardest.

What these APIs actually do

Start with what you’re renting, because the branding hides it. A managed scraping API is the infrastructure layer you’d otherwise build, wrapped behind a single URL. You send a request, and it picks a proxy from a pool, spins up a headless browser if the page needs JavaScript, retries when a fetch fails, and hands back finished HTML or parsed fields. You’re not buying magic. You’re paying someone else to run the proxy farm and the browser fleet. Everything below is about how differently these three do it.

The pricing shapes are the real difference

The three don’t even bill the same unit, and that’s where people get surprised.

ScraperAPI mostly sells credits per request, and a request costs more credits when it needs rendering or premium proxies. Zyte bills closer to per successful request, so a failed attempt doesn’t always cost you. Web Unlocker charges for successful requests too, but the network underneath is priced around bandwidth, so a heavy page costs more than a light one.

Three pricing shapes means the cheapest option flips depending on your target. You can’t rank them on sticker price alone, which is exactly why I measure cost per successful record on my own target instead of trusting the pricing page.

ScraperAPI

ScraperAPI is the friendliest to start with. The API is simple, the credit model is easy to reason about, and for static and moderately defended sites it just works. Flip a render parameter when a page needs JavaScript, a country parameter when you need a region, and you get your HTML back. On my easy and medium targets it cleared them at a high rate and stayed cheap.

The honest part: on my hardest target, the kind of site that throws a real challenge and watches behavior, its success rate dropped and the retries started burning credits. It didn’t fail outright, but the cost per successful page climbed enough that it stopped being the cheap option there. That’s positioning, not a flaw. It’s tuned to be affordable across the broad middle of the web, not to win a brawl with the two or three most defended sites you care about.

Zyte

Zyte comes at this from the data-company angle. Its API is paired with a proxy layer that adjusts how hard it works a request, and it leans toward charging for results rather than attempts. That shape matters on a hard target, because a site that needs several tries per page won’t silently multiply your bill the way a pure per-request model can. There’s more of a data platform around it too, for teams that treat scraping as an ongoing operation rather than a weekend script.

Where Zyte asks more of you is the learning curve and reasoning about cost under load. It’s less flip-a-switch than ScraperAPI. On my tests it held a strong success rate on the medium and hard sites and landed in the middle on cost, which is roughly where its whole design points. It’s what I reach for when scraping is a standing part of the operation.

Bright Data Web Unlocker

Web Unlocker is the specialist. It isn’t a general framework, it’s an unblocking endpoint built on Bright Data’s large residential and mobile network, aimed at the sites that beat everything cheaper. You hand it a URL, it works the problem behind the scenes, and it returns the page. On the single hardest target in my set, this cleared it most reliably, and by a clear margin.

That capability isn’t free. Because the network underneath is priced around bandwidth, a heavy page rendered through it costs real money, and a large crawl adds up fast. It’s also not the quickest. The mistake I see people make is reaching for the strongest unblocker for their whole job when only a handful of pages actually need it. Use the specialist where it’s required and its price stays contained.

How I tested

Same set of targets for all three: an easy static site, a medium JavaScript site, and one genuinely hard target. A few thousand requests through each service against each. Then I measured the three numbers that matter: the real success rate on the hard target, the latency per returned page, and the true cost per successful record rather than the advertised cost per request.

On the easy and medium targets, all three cleared them at high rates and the differences barely mattered. The hard target separated them, in the order the table shows. Latency tracked difficulty: the simple fetches came back quickest, and the heavy unblocking calls were noticeably slower per page. If you’re pulling large volume, that per-page latency multiplies into hours, and that’s a real cost, not a footnote.

When to buy, and when to build

Buy when the target is genuinely hard and would otherwise eat weeks, when the volume is modest, or when time is the scarce thing. A service that works this afternoon beats a build that works in three weeks, and if scraping isn’t your core business, buying the boring part is usually right.

Build when the volume is large, when you need full control over how each request is made, or when the targets are simple. At real scale on ordinary targets, your own stack wins on cost by a wide margin, and the proxy layer you put in front of it decides whether it works. In practice I run a hybrid: my own mobile proxies on real carrier SIM cards for the predictable bulk, and a specialist API in reserve for the few sites that fight back hardest.

The honest limits

None of these three makes anything undetectable, whatever the landing page implies, and none of them makes it legal to collect data that was never yours. The same rules apply: public data, the robots file respected, an official API or bulk feed preferred where one exists, a polite rate held. A service absorbs the infrastructure work, but not the responsibility for what you scrape.

I run both sides of this in production, my own stack and these APIs on real jobs. If you want the full reviews with the measured numbers and the picks I actually use, they’re at dataresearchtools.com, tested on real targets with no undetectable promises.

Get new guides and videos first — join the Telegram channel.

The question I get more than almost any other is some version of “how many proxies do I need for this scrape?” People want a round number back, a clean fifty or a thousand they can drop into a budget. The honest answer is that it’s arithmetic, and the number falls out of two things: how fast you want to pull, and how fast the target will quietly tolerate each address. Get that math wrong one way and every address you rent gets burned in an afternoon. Get it wrong the other way and you’re paying for a huge pool you never needed, sometimes making yourself easier to spot in the process.

I run proxy infrastructure and production scrapers, so I size pools for real jobs with real budgets, not off a vendor’s marketing page. Here’s the actual math I use, the formula that turns a target rate into an address count, and the parts most people skip.

Sizing is a rate problem, not a count problem

Throw out the idea that this is a count problem. You’re not really asking how many addresses you need, you’re asking how much traffic you want to push and how thinly you have to spread it so no single address gets noticed. The pool exists to divide your total load across enough addresses that each one stays under the rate a site will accept from a single visitor. Once you see it as spreading a rate, the number almost calculates itself.

The two numbers you actually need

You need two numbers before you can size anything.

The first is your target request rate, how many requests per second you want the whole job to make. That comes from how much data you need and how quickly you need it. You choose it.

The second is the safe rate per ip, how many requests a single address can make against this particular site before that site starts treating it as suspect. You don’t choose this one. The site decides it, and you have to go and measure it. It’s a property of the target, not of your proxies, which is why you can’t read it off anyone’s spec sheet. You find it by running a small test against the real site: one address, slowly raising the rate, watching for the point where the block rate climbs or challenges start appearing. Then you set your safe rate comfortably under that ceiling.

The formula

The formula is simple enough to hold in your head:

ip count is roughly your target request rate divided by the safe rate per ip.

That’s the whole thing. If you want the job to make some number of requests per second, and each address can safely make some smaller number per second, then the first divided by the second tells you how many addresses it takes to carry that load without any one of them going too fast. Everything else is just adjusting that result.

A worked example

Say you want the whole scrape to run at 20 requests per second. And say you’ve measured that this site tolerates about one request every five seconds from a single address before it gets unhappy. That’s a safe rate of roughly 0.2 requests per second per ip.

Divide 20 by 0.2 and you get 100. So you need on the order of 100 addresses to push 20 requests per second while keeping each one at a pace the site accepts. That’s your starting number, not your final one.

Add headroom

100 is the floor, because reality is messier than the math. Some addresses in any pool are already burned before you touch them. Some go stale partway through the job. Some fail for reasons that have nothing to do with you. So pad the clean number, usually by 30 to 50 percent, so the job still hits its rate when a chunk of the pool is unusable. 100 becomes closer to 150 in practice. Round up, never down.

Concurrency is a different dial, and people tangle the two. Concurrency is how many requests are in flight at once. Rate is how many you complete per second over time. The formula is about rate. Don’t size your pool off parallel connections, size it off the sustained rate you push through them.

More proxies is not always safer

Here’s the part that surprises people. A bigger pool is not automatically safer, and past the number the math calls for, more addresses can hurt you.

Every address you add is one more with a reputation you didn’t build and can’t see. A huge pool spread thin is a crowd of strangers, and if a slice of it is already dirty, you’ve inherited blocks you never earned. You’re also paying for capacity you aren’t using, which on metered residential pools is real money leaking for nothing.

There’s a subtler cost too. Edge defenders don’t just watch single addresses, they watch across a whole site and across many sites at once. A sudden swarm of fresh addresses all crawling the same paths at the same speed is itself a pattern, even when no single address is going too fast. Throwing a thousand brand new addresses at a job that only needed a hundred can raise your footprint instead of lowering it. The right size is the smallest pool that carries your rate politely, not the biggest one you can afford.

Pool size and rotation are the same constraint

Rotation and sizing aren’t separate topics. Rotation just cycles you through the pool, handing out a fresh address on a schedule or per request. But rotation can only spread your load if the pool is big enough. If the pool is too small, the rotator comes back around to the same address too soon, and that address ends up carrying more than its safe rate anyway. A pool below the size the formula asks for doesn’t lower your per address rate, no matter how you rotate.

Think of each address as needing a rest between uses. If the safe rate is one hit every five seconds, then after an address is used it should sit idle for about that long before it comes up again. The pool has to be large enough to fill that gap for every address in the rotation. In effect, pool size is your rate multiplied by how long each address needs to rest.

Sticky sessions size differently

Stateful work flips the calculation. If you’re walking a login, a cart, or a multi step flow that has to stay on one visitor, you can’t rotate mid session without contradicting yourself. There you size by concurrent sessions instead of by raw rate: roughly one address per session you want running at once, plus a batch of spares for the ones that fail. It’s a different formula for a different shape of job, and mixing the two up is how people end up wildly over or under provisioned.

Proxy type changes the number, not the math

The type of proxy doesn’t change the formula, it changes the safe rate that goes into it. On a strict site a datacenter address might tolerate almost nothing before it’s flagged, so the safe rate is tiny and the math demands a lot of them. A residential address reads as an ordinary person and tolerates more, so you need fewer. Mobile is the odd one: a single mobile address is already shared behind carrier grade nat by many real subscribers, so you rarely size it by raw address count at all. You size it by concurrent sessions and rotation instead. The tier you pick moves the denominator, which moves the count, but the arithmetic underneath is the same.

The honest limits

Sizing a pool correctly keeps you polite and keeps the job alive. It does not make anything undetectable, and no address count changes what you’re allowed to collect. Public data only, the robots file respected, an official api or bulk feed preferred wherever one exists. The formula is about carrying your load without leaning on any single address too hard. It’s not a way around the rules.

And ignore the headline pool size a vendor prints on the box. Millions of addresses tells you nothing about how many you need or how many will actually work on your target. Your number comes from your own measured safe rate against your own site, run through the formula, plus headroom. I’ve watched a tiny well maintained pool beat a giant dirty one on the same job. Measure first, size from the measurement, and treat the marketing number as noise.

I run this infrastructure in production, so the sizing and the pools I write about are the ones I actually use and test. If you want the full written guides and the proxy pools I genuinely rely on, read them here. No undetectable promises, no guaranteed results, just the way this actually works.

Get new guides and videos first — join the Telegram channel.

Most people store scraped data in whatever was easiest on day one, usually a single Postgres table, and then spend the next year fighting it. The table grows past what it was meant to hold, the analytics queries crawl, and every field change on the source site turns into a migration. The problem is not Postgres. The problem is asking one store to do three different jobs.

Scraped data does not have one right home. It has three, and the skill is knowing which slice of your data belongs in which. I run production scrapers, and pulling the data is only half the work. The other half is storing it so it stays queryable, deduped, and cheap as the volume climbs into the tens of millions of rows.

The three places scraped data lives

Think in three tiers, each solving a different problem.

Object storage holding Parquet files is the cheap, bottomless bucket for raw high volume capture, the stuff you want to keep but rarely query directly. Postgres is the operational layer, the deduped, indexed, current state you serve queries against every day. A warehouse like BigQuery or Snowflake is where analytics at scale lives, the place you run heavy aggregate questions over the whole history.

Almost all the storage pain I see comes from asking one of these to do another’s job.

Land the raw capture first

The first move is to separate raw capture from clean state. When the scraper pulls a page, do not parse it and throw the original away. Write the raw response somewhere cheap and untouched first, then parse from that copy.

I land raw html or raw json responses straight into object storage, one file per fetch or batched, before a single field gets extracted. The parse step reads from there, not from the live site.

The reason this matters is failure. Sites change their layout without warning, and your parser will break. If you kept only the clean parsed rows, a parser bug means you have to rescrape the site to recover, hammering it again and hoping the old data is still there. If you kept the raw capture, you just fix the parser and reparse the files you already have, offline, for free. Raw storage is cheap insurance against your own future mistakes.

Why Parquet for the raw and semi clean layer

You can land raw capture as plain json or csv, and for small jobs that is fine. At volume it hurts.

Parquet is a columnar file format. Instead of storing row by row, it stores all the values of each column together, which compresses far better because similar data sits next to itself. It is typed, so a number stays a number, and it carries its own schema. For large volumes you scan a few columns at a time, it is much smaller on disk and faster to read than csv or json, and every warehouse and analytics tool reads it natively.

The tradeoff: json preserves exactly what the site sent, which is what you want for the truly raw capture. So I keep json for the untouched landing, then convert to Parquet once the shape is known.

Postgres is your operational truth

Postgres is where the clean, current, queryable state lives. Once you have parsed a record, this is the row you actually use, look up, join against, and serve to whatever consumes the data. You get real indexes, real constraints, transactions, and sql everyone already knows.

For operational work, looking up one entity, filtering a few thousand rows, enforcing that a record is unique, nothing beats a boring relational database. Using Postgres is not the mistake. The mistake is trying to keep every raw byte you ever scraped inside it.

Dedup keys and idempotent writes

The most important thing you do in the operational layer is dedup, and it starts with a key. Every record you scrape needs a stable natural id, something the source itself defines: a product code, a listing id, a canonical url. That key tells you whether a row you just parsed is new or something you already have. Put a unique constraint on it.

Then make your writes idempotent, so running the same insert twice changes nothing the second time. Postgres gives you this with an upsert:

insert into products (id, title, price, raw)
values ($1, $2, $3, $4)
on conflict (id) do update
set title = excluded.title,
    price = excluded.price,
    raw   = excluded.raw;

On conflict with the key, it updates the existing row instead of creating a duplicate. This is what lets a pipeline retry safely: a run dies halfway, you start it again, and the records that already landed overwrite themselves cleanly instead of doubling. A stable key plus idempotent writes make a scraper safe to rerun, which you will do constantly.

Handling schema drift

Sites change their fields, and a rigid table will fight you over it. A source adds a column, renames one, or returns a value in a new shape, and suddenly your clean schema does not fit. This is schema drift, and you plan for it rather than getting surprised.

I keep a set of stable core columns I am confident about, the id and the fields I always need, and a jsonb column that holds the rest of the record as is. The structured columns stay clean and indexed for the queries that matter, and the messy, changing parts live in jsonb where a new field does not break anything.

When you have outgrown Postgres

Postgres is excellent right up until it is not. When a single table climbs past tens of millions of rows and your queries turn into big aggregates scanning most of it, a row store starts to struggle.

Counting across a hundred million rows, grouping by month over all of history, joining several large tables for a report: these are analytics questions. A database tuned for operational lookups answers them slowly, and at the cost of the operational work it should be doing. That slowdown is the signal you have outgrown one tier and need the next.

The warehouse is for analytics, not serving

That next tier is a warehouse, something like BigQuery or Snowflake. They are built for exactly the queries that make Postgres sweat, scanning and aggregating enormous columnar datasets. They separate storage from compute, so your data sits cheap and you only pay for the horsepower when a query runs. They read Parquet natively, so loading the raw layer in is straightforward.

The mistake in the other direction is serving live traffic out of a warehouse. Warehouses are built for big scans, not fast single row lookups, and they usually charge by how much data each query scans. Point an app at one for constant small reads and you get slow responses and a surprising bill. Keep operational serving in Postgres and heavy analytics in the warehouse, and let each do what it is good at.

The layered pattern I run

Here is how the three fit together in practice:

• The scraper lands raw responses as files in object storage, cheap and permanent.
• A parse step reads those, extracts clean records, and upserts them into Postgres keyed on a stable id. That becomes the deduped operational truth.
• On a schedule, the clean data loads into the warehouse as Parquet for analytics over the full history.

Raw for recovery, Postgres for serving, warehouse for analysis. Each layer feeds the next, and none is asked to do a job it is bad at.

Choosing by volume and query pattern

If you remember one rule, make it this: choose your store by volume and query pattern, not by habit.

Small data you look up and update row by row belongs in Postgres. Huge volumes of raw capture you rarely query directly belong in flat Parquet on object storage. Heavy aggregate analytics over everything belong in a warehouse. Cost often makes the call: object storage is cents per gigabyte a month, Postgres is a server you pay for whether or not it is busy, and a warehouse charges by how much each query scans. Most real pipelines want all three, and the healthy ones put each slice of data where its size and access pattern fit.

The honest limits

Storage does not change what you were allowed to collect. Public data, a robots file respected, an official api or bulk feed preferred, personal and paywalled data left alone. A clean layered warehouse full of data you should not have collected is still data you should not have collected. Good architecture makes your collection durable and cheap to keep. It does not grant permission.

I run this exact layering in production, raw in object storage, deduped state in Postgres, analytics in a warehouse. If you want the full guides, with real upsert patterns, Parquet partition layouts, and the tools I actually use and test, they are here.

Get new guides and videos first — join the Telegram channel.