Category: Uncategorized

Four percent of the records disagreed.

I had an api and a scraper pointed at the same site, writing into two tables, and after a week I compared them row by row. Ninety six percent matched exactly. Every mismatched record had been edited in the previous twenty four hours: the api was reading a replica with a cache in front of it. That appears nowhere in its documentation, and it would have appeared nowhere in my head if I had migrated the way everyone migrates.

Read the announcement, feel relieved, spend a fortnight on the client, delete the scraper.

I run mobile proxy lines and production scrapers out of Singapore. I have done this migration several times and got it wrong at least once, in a way that cost three days I had already told myself I owned.

Which kind of api are we talking about

Three things get filed under one heading and only one of them is this.

There is the undocumented json endpoint you find in the network tab, the one the site’s front end calls to fill the page. I use them constantly and have written about finding those. That endpoint has one consumer, the site’s own javascript, so nobody there carries an obligation to keep its shape still. I have watched one change twice in a month while the visible layout stayed identical.

There is a third party scraping vendor, where you pay somebody else to do the fetching and the target has no idea that supplier exists. That is a buy versus build question about your own infrastructure, and it leaves your standing with the target where it was.

Then there is this case. The site chose to publish. A documentation page, a version in the path, a changelog, and somewhere a person who gets a ticket when it breaks. That is a different animal, because it is the only one where the target learns your name.

What genuinely gets better

The upside is real and larger than api cynics admit.

Types. A number arrives as a number, a date as a date, an absent value as null instead of an empty element you have to interpret. Your parser stops being the most brittle component you own, and the parser is the thing that fails on somebody else’s design calendar, usually in silence.

A contract. Versioning in the path, a changelog, a deprecation window, occasionally an email before a field moves. Compare that against finding out because a column went blank and a customer got there first.

And the traffic stops being a fight. No mobile addresses on that target, no headless browser, no guessing at what counts as a polite rate. On one job the monthly run cost fell to almost nothing, because the whole of it had been proxy bandwidth for a two megabyte page.

The api is a subset of the page

Here is what almost nobody checks before committing.

The page and the api are two different field selections made for two different audiences. The page was built for a shopper. The api was scoped in a meeting for a partner integration, probably a while ago.

Derived fields fall out first. Anything the front end computes from several other values. Anything a merchandising team added as a badge.

Availability is the standard example. The page says in stock at four locations and names them. The api returns a boolean.

Review counts, position within a category listing, the recently sold counter. Those live in the presentation layer, and the presentation layer is precisely what you have agreed to stop reading.

The sting is that derived fields are frequently the reason your product exists, because nobody else could get them cheaply. The identifiers everybody already has are what the api hands you first.

The quota is a policy

A scraper’s ceiling is physical. Concurrency, address pool, and whatever load you are willing to put on somebody’s server.

An api’s ceiling is a number in a document, and documents get edited. I have watched a free tier drop from sixty calls a minute to twenty, announced in a changelog entry and nowhere else.

Two pieces of arithmetic before you migrate. Divide your record count by the published quota and read the answer in hours, because that is your run regardless of hardware. Then find out what counts as one call, because a list endpoint returning fifty records is a different budget from a detail endpoint you hit once per record.

What the key costs you

A key means an account. An email address, usually a company name, plus a box you tick with a legal agreement sitting behind it.

Before, you were traffic. Anonymous, hard to separate from a browser, and a site that wanted you gone had to find you first. After, you are a row in their database with a name on it and a switch beside it. Revoking a key takes one click from somebody who never has to justify it. Blocking a scraper is a project they can lose.

The agreement contains things nobody reads. Retention limits. Whether you may display it to your own customers. Whether you may use it in anything that competes with them. I have read one that capped storage at twenty four hours, which rules out every historical product you might have planned, and that cap lived in the terms rather than the docs.

Read the termination clause specifically. Whether it requires cause, and what notice you get. That paragraph describes the relationship more accurately than the entire feature list above it.

Pricing arrives after the scraper is gone

The sequencing is deliberate. A new api is usually free while the publisher wants adoption, because they need integrations to point at. Metering shows up later, once there is a population of users who have already deleted their alternative.

That is ordinary product behaviour rather than malice. It is still the risk, because the cheapest moment to own a working scraper is before you learn what the api costs.

An api is a relationship and a scraper is not

Here is the claim, and I do get argued with about it.

The relationship gives you notice, a changelog, a version, somebody to email when a field moves. It also gives them a switch, an agreement you signed, and a price they get to set.

The scraper gives you no notice, no support, and maintenance with no end date. It also gives them nobody to switch off and no invoice to raise.

Neither wins in the abstract. The only question is whether you can live with someone else holding a switch over the thing you sell. If the data is an input to a product, usually yes. If the data is the product, that is a genuine dependency and it should be priced as one rather than treated as good news.

Map your columns before you read the docs

The audit takes about half an hour.

Write down your actual field list, from the output table rather than from memory. For every column, note where it comes from today: a raw value off the page, or something you compute from two or three scraped values.

Then open the field reference and map them. Each column gets a named api field or it gets a question mark.

The question marks are the whole exercise. One is a conversation with your product owner. Three means you are keeping a scraper whatever happens, and the migration has become a hybrid rather than a replacement.

Do this before the meeting where somebody proposes just using the api. It is the only artefact in that room with facts in it.

Run both, then diff the values

A mapping exercise compares names. A diff compares values, and values are where the surprises live.

Run both collectors against the same records on the same schedule, each writing into its own table, for a few weeks. Yes, you are paying twice. The bill is a few weeks of proxy traffic you were already spending, and it finds what documentation cannot describe.

What turns up: an api price that is the base price while the page price includes a promotion; a numeric field present and silently rounded; the replica lag I opened with.

Set your threshold before you look. Mine is that anything above one percent disagreement halts the migration until I understand the cause.

The join you are not going to get

The api’s identifier for a record is often not the identifier your scraper keys on, and it carries no obligation to expose yours. So your history does not join. Two years of stored rows, and the new ones will not line up. Check that field on day one, because a missing shared key means a matching pass, and matching is its own project.

Keep the old collector running

After you migrate, keep the scraper and keep it running rather than parked. A weekly job over a hundred records is enough. It stays green, and on the day the key gets pulled or the quota moves you have something that executes instead of a repository you have to remember the shape of.

Where I got this wrong

That last paragraph is advice I gave myself and then ignored. I migrated, kept the code, and never ran it. Eight months later I needed it back and the site had been redesigned twice. What I actually had was a folder. Rebuilding took three days, which is survivable, and which I had already counted as bought.

The honest limit: I have never had a key revoked on me. The failures I have been hit by are the subset problem and the quota problem. When I tell you the switch matters, that is reasoning rather than a scar.

There are cases where an api is straightforwardly right and none of this applies. A regulator publishing a bulk api is publishing it so you stop crawling them. No commercial lever, no competitive clause, and arguing about relationship risk there is just being difficult.

More on scraping infrastructure, pipeline design and the proxy layer underneath it is at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The fastest scraper is almost never the one that wins. The one that wins is the one still running next month, and the difference between the two is almost entirely about how it handles rate. A scraper that pulls as hard as it can gets noticed, throttled, and blocked, usually within hours. A scraper that moves at a pace the site can absorb keeps going and going. This is the playbook for that second kind: how to handle rate limits, retries, and backoff so your collection stays alive.

I run production scrapers for a living, the kind that have to keep working for months without becoming somebody else’s incident, so this is the part I care about most. It isn’t glamorous. Nobody makes a flashy demo about waiting politely between requests. But it’s the actual skill that separates a toy that runs once from infrastructure that runs for a year. And almost all of it comes down to one idea: take only what the site can comfortably give, and back off the moment it signals otherwise.

What a rate limit really is

Start from the site’s side, because that’s who sets the rate. A rate limit is simply the site saying you may make this many requests in this much time, and no more. It exists to protect the site from being overwhelmed, whether by an attack, a bug, or a scraper pulling too hard. A rate limit isn’t an insult or a challenge to beat. It’s the site telling you exactly how much load it’s willing to carry from you. The polite move, and the durable one, is to listen to that number and stay under it.

The status code that says slow down

Sites have a standard way of telling you that you’ve gone too fast. When you cross a limit, a well behaved server responds with a status code, most commonly 429, which means too many requests. Often it comes with a header that literally tells you how long to wait before trying again. This is the site handing you the answer. The worst thing a scraper can do is ignore that signal and keep hammering, because that’s exactly the behavior that turns a temporary slowdown into a permanent block. Read the code, honor the wait.

Backoff is the core move

The central technique is backoff. When a request fails or gets throttled, you don’t retry instantly. You wait, and you wait longer each time it keeps failing. The first retry might wait a second, the next a few seconds, the next longer still. This gives the site room to recover and signals that you’re a well behaved client, not a battering ram. A naive retry loop that fires again the instant it fails is the single most common way people turn a recoverable hiccup into a hard ban. Backoff is what makes retries safe instead of dangerous.

Why the waits grow

The reason the wait grows each time is worth understanding. A single failure might be random noise, so a short wait and a retry usually clears it. But repeated failures mean something is actually wrong: the site is struggling or actively throttling you, and the right response is to pull back harder, not keep pushing at the same rate. Growing the delay each time means a brief blip costs you almost nothing, while a real problem makes you gracefully retreat instead of piling on. The pattern matches your pressure to the site’s actual state.

Add jitter or you stampede

Here’s a subtle one people miss. If you run many workers and they all fail at the same moment and all back off by the same amount, they’ll all retry at the same instant, producing a synchronized wave that hits the site like a hammer. The fix is jitter: adding a small random amount to each wait so the retries spread out instead of bunching up. It’s a tiny change with a big effect. Without jitter, your polite backoff can accidentally become a coordinated stampede, and the site feels a spike exactly when you meant to ease off.

A backoff in a few lines

The shape of it is small. You catch the failure, compute a growing delay with a little randomness, and try again up to a limit. That’s exponential backoff with jitter, the whole idea in a handful of lines, and it will carry you a long way.

The token bucket idea

Beyond reacting to failures, you want to control your rate before you ever trip a limit. A clean way to think about this is a token bucket. You get a steady supply of tokens, one per allowed request, and each request spends one. When the bucket is empty, you wait for it to refill. This smooths your traffic into an even, predictable stream instead of bursts, and it lets you set your own ceiling comfortably below whatever the site allows. Shaping your rate proactively is far better than sprinting until the site slams the door and then reacting.

Concurrency is a dial, not a maximum

People treat concurrency as a number to maximize, and that’s the mistake. Running more requests at once feels like progress, but past a point it just increases the load the site feels from you and the chance you get flagged. Treat concurrency as a dial you set deliberately, low enough that the site never strains, not as high as your hardware could technically push. I routinely run well below what my machines could handle, on purpose, because the bottleneck I care about is the site’s tolerance, not my throughput. Slower and alive beats fast and banned, every time.

Respect the crawl delay

Many sites publish their preferences in a robots file, and that file often includes a crawl delay, a request to wait a certain amount of time between hits. Honor it. Treating that number as a real instruction rather than a suggestion is both the courteous thing and the smart thing, because a site that publishes a crawl delay is telling you the exact pace at which it will tolerate you. Staying at or under that pace keeps you invisible in the best way: a small, steady stream of requests the site has explicitly said it can carry. Ignoring it is asking for trouble you were warned about.

Cache so you never ask twice

The cheapest request is the one you never make. A huge amount of scraper load is wasteful, fetching the same page again because the pipeline wasn’t tracking what it already had. So cache aggressively. Store what you pull, and before you request anything, check whether you already have a fresh copy. This cuts your load on the site dramatically, speeds up your own job, and shrinks the surface where anything can go wrong. Every page you serve from your own cache is a page you didn’t have to ask the site for, and politeness and efficiency point the same direction here.

Spread the load across time

If you have a big job, don’t try to finish it in one aggressive burst. Spread it across time. A hundred thousand pages pulled gently over a day is nearly invisible, while the same hundred thousand pulled as fast as possible in an hour is a spike that any monitoring will catch. The total work is identical, but the shape the site feels is completely different. Patience is a feature here. A job that isn’t urgent should be spread wide and thin, because the flatter your traffic, the less reason the site ever has to look at you twice.

The circuit breaker

There’s one more pattern worth having, borrowed from resilient systems, called a circuit breaker. The idea is that when failures pile up past a threshold, you stop entirely for a while instead of continuing to retry. If a site is returning errors on nearly everything, that’s a clear sign to pause the whole job, not to keep probing at it. The breaker trips, you wait a good while, and then you test cautiously before resuming. This stops a struggling site from turning into a hard block, and it keeps you from wasting effort against a wall that isn’t going to move right now.

Read the response, not just pass or fail

Politeness also means paying attention to what the site is telling you beyond the raw status. Rising latency, subtle warnings, a challenge appearing where there wasn’t one before, these are all signals that you’re pushing harder than the site wants. A good scraper watches those and eases off before it gets to an outright block. By the time you’re seeing hard failures, you’ve already been rude for a while. The skill is reading the soft signals early and slowing down on your own, so the site never has to make you.

Retry only what is safe to repeat

One caution about retries: they must be safe to repeat. If a request has a side effect, submitting a form or changing something, retrying it can do that twice, and you end up with duplicate or corrupt results. Keep retries to operations that only read, and design your storage so writing the same record twice does no harm. Safe retries are what let backoff work without fear, because you can try again knowing a repeat costs you nothing.

Why polite scrapers last

Put it all together and the philosophy is simple. The durable scraper is the considerate one. It stays under the limit, backs off when told, spreads its load, caches what it can, and reads the site’s signals instead of bulldozing through them. Done this way, most of the blocking machinery never even has a reason to fire, because you aren’t producing the load that triggers it. You’re just a small, steady, well behaved stream of requests, and a site has very little reason to fight a client that isn’t causing it any trouble in the first place.

The honest limits

Let me be straight about the boundaries. Handling rate well keeps a scraper healthy and welcome, but it doesn’t change what you’re allowed to collect. Public data, a robots file respected, an official API or bulk feed preferred where one exists. Politeness isn’t a trick to sneak past anything, it’s the absence of the load that gets you noticed, and it works precisely because it’s genuine restraint rather than a disguise. And no amount of careful pacing makes it right to take data that was never yours, or to ignore what a site has clearly asked you not to do.

To recap: a rate limit is the site telling you how much load it will carry, so listen to it. Honor the slow down signals, retry with backoff that grows each time, add jitter so your workers don’t stampede, and shape your rate proactively with something like a token bucket. Treat concurrency as a deliberate dial, respect the crawl delay, cache so you never ask twice, spread big jobs across time, and trip a circuit breaker when a site is clearly struggling. Be the considerate client, and you get to keep running.

For more breakdowns on scraping infrastructure, proxy setups, and pipeline design straight from production use, visit the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.

Your browser tells a story about itself to every site you visit, and most of that story gets told without your permission and without your knowledge. It isn’t your login, and it isn’t a cookie. It’s the shape of your device itself, assembled from dozens of small signals into something that turns out to be surprisingly unique. Sites use that shape to tell a returning visitor from a stranger, and to tell a real person from an automated one.

I run proxy infrastructure and production scrapers, so I spend a lot of my time on the defending side of this, understanding how detection systems are built and how they operate. I want to be clear about the frame up front: this is an explanation of how the mechanism works, not a guide to defeating it. Faking a device isn’t the durable way to collect data, and it isn’t a game worth playing. Understanding the wall is useful. Pretending it isn’t there is not.

What a fingerprint even is

A browser fingerprint is a profile a site builds by asking your browser a series of harmless-looking questions and combining the answers: what’s your screen size, what fonts do you have, how does your graphics chip draw a shape, what audio hardware do you report. No single answer identifies you. But stack forty of them together and the combination is often unique enough to pick you out of millions, and to recognize you again later with no cookie at all. That’s both the power and the problem of it.

The user agent is just a claim

The most familiar signal is the user agent, the string that names your browser and operating system. On its own it means little, because it’s just text your browser volunteers, and anything can type any string into it. What makes it useful to a site isn’t the claim itself, it’s whether the rest of the browser backs the claim up. A client that says it’s one thing while every other signal says something else has produced a contradiction, and contradictions are exactly what detection systems are built to notice.

What the screen and system reveal

Your browser freely reports a pile of system details: screen resolution and color depth, number of processor cores, amount of memory, operating system and version. Each one is coarse on its own, plenty of people share a common screen size. But combined with everything else, these details narrow the field fast, and they carry consistency requirements. A device claiming to be a phone but reporting a giant desktop screen and dozens of cores is telling two stories at once, and that mismatch is more revealing than any single value.

The canvas signal

Here’s one of the cleverer techniques. A site can ask your browser to draw text and shapes onto a hidden canvas, then read the exact pixels back. The result looks identical to a human eye, but at the pixel level it varies slightly from device to device, because the drawing depends on your graphics hardware, drivers, and system fonts. That tiny variation is stable for your machine and different across machines, so it works like a signature. The site isn’t showing you anything. It’s quietly measuring how your specific hardware renders a picture nobody ever sees.

WebGL and the graphics chip

Closely related is WebGL, which lets a site render three-dimensional graphics and, in doing so, learn about your GPU. It can read the make and model of the chip and observe exactly how it draws a complex scene. Because graphics hardware varies so much across devices, this is a strong contributor to the overall fingerprint. Like the canvas signal, it’s passive from your side. You see a normal page, while underneath it your GPU has quietly described itself in enough detail to help separate your device from the next one.

The fonts you have installed

The set of fonts on your system is another surprisingly telling signal. The exact list depends on your operating system, the software you’ve installed, and the languages you use, so it varies more between people than you’d guess. A site can probe which fonts are present and build a picture from the answer. On its own it’s just one more coarse signal, but it stacks with the others, and an automated environment that ships a bare, identical font list on every instance stands out precisely because real people’s font sets are messy and individual.

Audio, timezone, and language

The signals keep going. The way your device processes a silent audio sample varies with your hardware and software, giving another faint but stable marker. Your timezone and language settings add more, and they carry consistency checks of their own. A visitor whose address places them in one part of the world while their timezone and language claim another has produced yet another small contradiction. None of these is decisive alone. The method is always the same: gather many weak signals and let the combination do the identifying.

The network layer speaks too

Fingerprinting isn’t only about the browser. The network layer talks before the page even loads. The way your client negotiates its encrypted connection, the specific options it offers and the order it lists them, forms a pattern that real browsers produce in well-known ways. A lot of automated tools produce a pattern here that no real browser would send, because the underlying library was never trying to imitate one. So a client can claim to be a current browser in its headers while its connection setup quietly says otherwise, which is one more consistency check a site can run for free.

Entropy, or how the signals combine

The word for what makes this work is entropy, which just means how much a signal narrows the field. A signal everyone shares carries little. A signal that varies a lot carries more. Detection combines many signals so that even though each is weak, the total is strong. You can’t think about any one value in isolation. The system isn’t asking whether your screen size is suspicious. It’s asking whether the whole bundle, taken together, looks like a real person’s device or like something assembled to look like one.

The tells of an automated environment

So what gives an automated setup away? Real devices vary in messy, natural ways across all of these signals, while a fleet of identical automated environments tends to produce the same fingerprint again and again, or one carrying small tells a normal machine would never have. A browser driven by automation can expose properties that a human’s browser doesn’t. The detection system isn’t hunting for one magic flag. It’s looking at the whole shape and asking whether it resembles the natural variety of real people or the suspicious sameness of a cloned machine.

The long tail of signals

And the list keeps growing at the edges. Some environments expose battery status, motion sensors on a phone, the exact way a page scrolls or a pointer moves, even the subtle timing of small operations. Each one is another faint signal added to the pile. You don’t need to memorize the full catalog. The takeaway is that the surface is wide and always expanding, so the number of places a device can quietly describe itself is far larger than most people ever imagine.

Consistency is the real test

Step back and the pattern is clear. Almost every check here is really a consistency check. Does the user agent agree with the rendered fingerprint? Does the timezone agree with the network location? Does the claimed device agree with the reported hardware? Detection is less about catching one forbidden value and more about catching two signals that disagree. A real device is naturally consistent because all its signals come from one real place. An assembled identity has to keep every one of those signals in agreement, and that’s genuinely hard to do.

This tracks real people too

It’s worth naming the other side of this, because fingerprinting isn’t only aimed at bots. The same techniques identify and follow real human visitors across the web with no cookie and no consent, which is a genuine privacy concern. A person clearing their cookies can still be recognized by their fingerprint. This technology sits in an uncomfortable place: useful for telling automation from people, and at the same time a quiet tool for tracking those very people. Understanding how it works is part of understanding the privacy tradeoffs of the modern web, not just the scraping ones.

Why it’s an arms race

This whole area never sits still. Detection methods get sharper, browsers add protections that blur some signals, and the measurements shift as hardware and software change. A signal that’s strong today can weaken tomorrow, and a new one can appear. That’s why I’m wary of anyone who claims a permanent answer to any of it. The ground moves constantly under both the sites doing the measuring and the tools being measured. It’s less a solved problem than an ongoing back and forth with no final state.

What this means for honest collection

So where does a compliant operator stand in all this? Not in the business of faking a device to slip through, because that’s a race you don’t win and it’s the wrong side of the line anyway. The honest position is to understand how the wall works, and then to not need to beat it. Collect public data at a polite rate. Identify yourself honestly where a site expects it. Prefer an official API or data feed, because that’s the front door the site actually built, and it doesn’t care about your fingerprint at all. Understanding detection makes you a better engineer. Trying to defeat it makes you someone else’s incident.

The honest limits

Let me be straight about the boundaries. Nothing here makes anything undetectable, and I wouldn’t trust anyone who says otherwise. Detection keeps improving, and the point of understanding it isn’t to evade it but to know why the compliant path is the durable one. The moment collection depends on faking an identity, or reaches for private data, or ignores what a site clearly asked, it’s left the honest lane entirely, whatever the tooling looks like. The version of this work that lasts is the boring one: public data, honest identification, a gentle pace, and the rules respected.

To recap: a browser fingerprint is dozens of weak signals, the user agent, the screen, the canvas, the graphics chip, the fonts, the audio, the timezone, and the network handshake, combined into something unique enough to identify a device. Detection mostly hunts for contradictions between those signals, since real devices are naturally consistent and assembled ones struggle to be. The same technology quietly tracks real people, and it never stops evolving. The honest response is to understand it and to collect in a way that never depends on defeating it.

For more breakdowns like this on scraping infrastructure, proxy setups, and how detection systems actually work, visit the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.

I have inherited four scrapers written by somebody else. The code was readable inside an afternoon every time. Rebuilding everything around the code took between three days and a fortnight.

One of them had a fixed 2.3 second delay in the fetch loop with no comment next to it. I deleted it, because it looked like debugging somebody forgot to remove. Two days later I understood exactly why it was there.

That is the subject in one story. The repository is the easy half of a handover, and it is the only half most people transfer.

I run mobile proxy lines and production scrapers out of Singapore, so I have been on both ends of this. The jobs that turned sour were almost never badly built. They were badly handed over.

Working is a state it passes through

A scraper is a dependency on a website somebody else owns. No contract exists between you and them, they do not know your project exists, and they can change a template on a Wednesday afternoon for reasons entirely unrelated to you.

So “working” is a condition your collector moves through, between changes at the far end. It does not settle there.

Your client thinks otherwise, and I would not blame them for it. They are used to buying software the way they bought their invoicing tool. You install it, it runs, and a thing that stops running is a defect somebody owes them a fix for.

Leave that assumption in place at the moment of transfer and every future site change lands on you by default. Not through anybody’s bad faith. Because nobody said otherwise while the atmosphere was still friendly.

The paragraph that goes at the top

Before the install steps, before anything technical, one paragraph:

This collector works against the target site as it exists on the date of transfer. The site will change. When it changes, the collector will stop returning correct data. That is expected behaviour and not a fault in the delivery.

I bold it, and I read it out on the handover call, because clients skim documents and remember awkward sentences said to their face.

It reads badly when you are trying to look competent. It reads a great deal better than the email four months later that starts “we paid you to build this”.

Four things that never make it into git

The code transfers itself. It is text, it sits in version control, and any competent engineer can read it. Everything that makes the code actually run is somewhere else.

Credentials, and the whole apparatus behind them. Not just the password for the target account. The email address it was opened with, the phone number that receives the second factor, and whose name the account legally stands in. If it stands in yours, you are permanently in the loop on a system you no longer support. I now open target accounts on the client’s email address from the first day of a build, even when it costs me an hour of back and forth.

The proxy arrangement, including the billing. Which provider, which plan, what a normal month consumes in bandwidth, and whose card renews it. Get that last one wrong in either direction and it hurts. Either your card keeps paying for a client you stopped working for, or it lapses and the collector dies quietly on a Tuesday. I watched a team spend two weeks reading Python because a proxy plan had expired on a card belonging to somebody who left the company. On my own infrastructure a real SIM line runs about ten dollars a month in airtime, so a client inheriting three lines has taken on a thirty dollar monthly commitment, and they should hear that number from me rather than from their bank statement.

The quirks. This is the highest value page in the whole document and it is the one that gets skipped, because it is the only part you cannot generate from the code. Every target does something strange that you found the hard way and then worked around silently. The listing endpoint returns 24 items and the last two are always adverts. The rate limit counts per account rather than per address, so buying more proxies achieves nothing. A background call returns clean JSON, so the parser never touches the rendered HTML at all. Response quality degrades above roughly four pages a second, but only in the evening. None of that lives in the code. Some of it is the reason a line of the code looks idiotic, like a 2.3 second delay nobody explained.

I keep a plain text file per target while I build, one line per surprise. By delivery it is usually twenty or thirty lines long, and it is the page the receiving engineer reads twice.

Three failures, three pages

You cannot document everything that might go wrong. You can document the three things that will.

A selector breaks. The markup changed, a field arrives empty or wrong, and the row count looks perfectly healthy. Say which file holds the selectors, how to confirm the site moved rather than the code, and roughly how long a repair takes in hours.

The block rate climbs. Responses stop being data and start being challenge pages, usually carrying a cheerful 200 status. Say what a block page looks like on this specific site, how to measure the rate, and what to do first, which is nearly always slow the crawl down before anyone starts swapping proxy pools.

The volume moves. Forty thousand rows yesterday, two hundred today. Say how to distinguish a block from a layout change from the client’s own filter, in under ten minutes.

I have never needed a fourth entry. I have needed all three more times than I can count.

One signal they can read without you

Documentation only helps somebody who already suspects a problem exists. What decides whether you get blamed is whether the client finds out from a dashboard or from a stale report five weeks later.

So the final deliverable is a signal they can read themselves. Not the monitoring layer I would build for my own jobs, with rolling baselines per source, block page fingerprints and a run record for every execution. I have written that up separately and it is worth the engineering.

For a handover the requirement is smaller and much stricter. One page or one daily email, readable by somebody who does not write code, in under a minute on a Monday morning. Last successful run and when. Rows collected against what a normal day looks like. Field fill rate for the two or three columns that actually matter. A sentence at the top that says either this looks normal or this does not look normal.

Here is the position I will argue for: handing over a scraper with no monitoring is handing over a lawsuit with a delay on it. The data goes wrong silently, somebody prices a product off it or puts it in front of a board, and when they eventually work backwards to the cause, the last name in the commit history is yours.

Support request or new build

The commercial half is where the goodwill actually drains away. If the document does not draw this line, every site change becomes a free rebuild, and the client is not being unreasonable when they ask for one. From where they sit, they bought a working scraper and it is not working.

A support request is the collector doing something it was not doing at handover, on the same site, for the same fields. That carries a response time and either sits inside a retainer or bills at an agreed hourly rate.

A new build is everything else. A new field, a new page template, a new site, a new output format, a login wall that appeared since delivery, or ten times the volume anybody agreed to. That gets quoted like any other job.

If you already split the build from the run when you quoted the work, this section is copy and paste. If you did not, the handover is your last opportunity to draw the line before it defaults to unlimited.

The one I got wrong

A collector I delivered in March ran for eleven days after I stopped watching it. On day twelve the site renamed a CSS class, the price field started arriving empty, and the client found out five weeks later when their finance team asked why a competitor pricing sheet had not moved since Easter.

The class name was not my mistake. The build was solid: tests, retries, a validation gate that rejected malformed rows, structured logs.

My mistake was that I shipped the repository and a README and called that a handover. No runbook. No quirks file, and that target had three good ones. No monitoring the client could see, because my monitoring was a cron job on my own server that emailed me, and I switched it off the week the invoice cleared.

So the validation gate worked exactly as designed. It rejected the blank rows. The pipeline logged the rejects and carried on. The output file kept landing on schedule with fewer rows in it each week, and nobody was watching the count.

I had built the detection and then handed over the system without it, which is worse than never building it at all. It produced a machine that knew it was broken and had no way to tell anybody. I rebuilt it for free, about eleven hours. Writing the handover template I still use took one afternoon.

What a handover cannot fix

None of this makes a scraper last longer. The site changes when it wants to.

The document also does not survive staff turnover on the client side. The engineer you briefed leaves, the file sits in a drive folder nobody opens, and eighteen months later somebody inherits a collector nobody understands. I have been that somebody.

The honest claim is narrower than it sounds. A good handover moves the surprise from the client to the calendar. They still get broken data eventually. They just find out on the day it breaks, from something they can read, with a page that tells them what to do next.

And it changes nothing about what you were allowed to collect in the first place. Public pages, the robots file honoured, a crawl rate that does not hurt the target, an official API or a bulk feed preferred every single time one exists. A beautifully documented handover of a job you should not have taken is still a job you should not have taken.

Everything else I have written on scraping operations and the infrastructure underneath them is here.

Get new guides and videos first — join the Telegram channel.

There’s a whole category of products that promise to make scraping somebody else’s problem. You send a URL, they send back the HTML or the parsed data, and they handle the proxies, the browsers, and the retries behind the scenes. They’re called scraping APIs, and they’re either the smartest money you’ll spend or a slow leak in your budget, depending entirely on the job. I want to walk through what these services actually do, when buying one is the right call, when you should build your own instead, and how to run the cost math before you commit to either.

I run my own scraping infrastructure, and I’ve also paid for these services on real jobs, so this is a tested view, not a vendor pitch. I’ll be honest about both sides, because the right answer genuinely depends on your volume, your target, and what your time is worth. No service makes scraping undetectable, and none makes legal something that wasn’t already legal. What these tools sell is convenience, and convenience is worth a lot right up until it isn’t.

What a scraping API actually does

Start with what you’re really buying, because the marketing hides it. A scraping API is the infrastructure layer you’d otherwise build, rented by the request. Under the hood it maintains a pool of proxies, spins up headless browsers when a page needs rendering, handles the retries when a fetch fails, and hands you back a clean result. You’re not buying magic. You’re buying someone else running the proxy farm and the browser fleet so you don’t have to. That’s the entire value, and whether it’s worth it comes down to what running that yourself would cost you.

The three flavors you’ll meet

These services aren’t all the same, and they roughly split into three kinds. The simplest just fetch raw HTML through a rotating proxy and return it, which is cheap and fast for static sites. The middle tier renders the page in a real browser and returns the fully loaded HTML, for sites that need JavaScript. The richest tier returns structured data for specific popular targets, so you ask for a product and get clean fields instead of HTML to parse yourself. Price climbs with each step, because each one is doing more of the work you’d otherwise do.

The case for buying

Here’s when I reach for a service without hesitation. When the target is genuinely hard, the kind that has beaten datacenter and residential addresses and needs constant care, letting a specialist absorb that fight is often cheaper than staffing it yourself. When the volume is modest, paying per request costs less than standing up and maintaining your own infrastructure. And when your time is the scarce resource, a service that works today beats a build that works in three weeks. If scraping isn’t your core business, buying the boring part is usually the right trade.

The case for building

Now the other side. When your volume is large, per-request pricing turns brutal. A service that costs a fraction of a cent per page sounds cheap until you multiply it by ten million pages a month, and suddenly you’re paying more every month than a couple of servers and a proxy pool would cost outright. When you need full control over exactly how requests are made, a black box service fights you. And when your targets are simple, you’re paying a premium for infrastructure you didn’t need. At scale, on easy targets, building wins on cost by a wide margin.

Run the crossover math

So run the actual numbers before you decide, because there’s a crossover point and it isn’t subtle. Take your monthly page volume and multiply it by the service’s per-request price. Then estimate what your own stack would cost: the servers, the proxy pool, and an honest slice of your time to maintain it. Below the crossover, the service is cheaper and you should buy. Above it, your own infrastructure is cheaper and you should build. Most people never do this arithmetic and just guess, and the guess is usually wrong in whichever direction flatters the choice they already wanted.

The hidden cost of building

But be honest about the build side, because people lowball it. Running your own scraping infrastructure isn’t just server rent. It’s proxy costs, it’s the engineering time to build the retries and the rotation and the browser fleet, and it’s the ongoing maintenance when a target changes and your stack has to adapt. That last part is the one people forget. The service absorbs that maintenance for you, quietly, every day. When you build, that work becomes yours forever, and it doesn’t show up in the tidy cost estimate you made on day one.

The hidden cost of buying

And be equally honest about the buy side. A service is a dependency you don’t control. Its price can rise, its quality can drift, and if it goes down, your data goes down with it and there’s nothing you can do but wait. You’re also trusting a third party with your targets and your traffic. And there’s lock-in, because the more your pipeline is built around one service’s quirks, the harder it is to leave. Convenience today can become a cage tomorrow, so weigh the cost of depending on someone whose priorities aren’t your priorities.

Test before you trust the marketing

Whatever a service claims, test it on your actual target before you commit budget. The headline success rate on the marketing page is measured on easy sites, not on the specific hard target you care about. So run a real batch, a few thousand requests against the site you actually need, and measure the true success rate, the latency, and the cost per successful page. A service that boasts a very high success rate can quietly fail on your one difficult target, and you only find that out by testing. The vendor’s number is a promise. Your measured number is the truth.

Watch the per-request definition

Read the fine print on what counts as a request, because this is where the bill surprises you. Some services charge you for failed attempts, not just successful ones, so a hard target that needs several tries per page multiplies your cost silently. Some charge extra for rendering, extra for premium proxies, extra for the structured tiers. The sticker price and the price you actually pay can be very different once the target forces the expensive options on. So when you test, measure the real cost per successful record, not the advertised cost per request, because those two numbers are rarely the same.

The hybrid that often wins

It’s not always all or nothing, and the smartest setup is frequently a mix. Build your own stack for the bulk of your volume on the easy targets where you control the cost, and buy a service only for the handful of genuinely hard targets that would otherwise eat your time. That way you pay the premium exactly where it earns its keep and stay cheap everywhere else. I run this split myself: own infrastructure for the predictable heavy lifting, a service in reserve for the few sites that fight back hard enough to be worth outsourcing.

If you build, the proxy layer matters most

If you go the build route, the piece that decides whether it works is the proxy layer, because that’s the trust you arrive with. Cheap datacenter IP addresses will fail on strict targets exactly like they would inside a service, so the addresses you choose are the whole ballgame. This is the layer I run myself, real mobile proxies on real carrier SIM cards, because for the hard targets that’s the most durable trust you can put in front of a scraper. The rest of the build is retries and rendering, but the address is what gets you through the door.

What a service does not solve

Be clear about what a service doesn’t fix, because the marketing blurs it. It doesn’t decide what’s worth collecting, it doesn’t clean or model your data, and it doesn’t understand your target the way you do. It hands you raw results, and the whole job of turning those into something useful is still yours. So a service saves you the infrastructure, not the thinking. I’ve watched people buy an expensive scraping API and still have most of the work in front of them, because the hard part of a data project was never the fetching. It was knowing what to fetch and what to do with it afterward.

Reliability and support are the real product

When you depend on a service, its reliability becomes your reliability, so weigh that before you commit. How often does it go down, how fast does it recover, and is there a real human to reach when a target suddenly stops working. A cheap service with no support is fine until your pipeline breaks on a Monday morning and you’re on your own. I pay attention to the boring signals here: the status history and the response time when I open a ticket, because those tell you more about living with a service for a year than the price or the feature list ever will.

Start small and stay portable

However you lean, don’t marry the decision on day one. Start with the cheapest option that clears your target, prove the job works end to end, and only then scale the spend. Keep your pipeline portable, so the fetch layer is a piece you can swap. If you wrap whichever service you pick behind a thin boundary in your own code, then switching services, or moving from a service to your own stack, is a small change instead of a rewrite. The goal is to keep the choice reversible, because your volume and your targets will change, and the right answer will change with them.

The honest limits

Let me be straight about the boundaries, because no service changes them. Buying a scraping API doesn’t make anything undetectable, whatever the landing page implies, and it doesn’t make it legal to collect data that was never yours to collect. The same rules apply as always: public data, a robots.txt file respected, an official API or bulk feed preferred where one exists, a polite rate held. A service can absorb the infrastructure work for you, but it can’t absorb the responsibility for what you scrape. That stays with you no matter whose proxies the request rides on.

I run both sides of this in production, my own stack and these services on real jobs, so this is a tested comparison, not theory. The whole decision comes down to one question asked honestly: is running this infrastructure yourself cheaper than renting it, at your volume, on your targets. Answer that with real numbers and the choice makes itself.

To recap: a scraping API rents you the proxy and browser infrastructure you’d otherwise build, sold by the request across three tiers of increasing price and convenience. Buy when the target is hard, the volume is modest, or your time is the scarce thing. Build when the volume is large, the targets are simple, or you need full control. Run the crossover math with honest costs, test on your real target, watch what a request actually costs, and mix the two where it pays.

For more breakdowns like this, tested on real infrastructure with no undetectable promises, visit Data Research Tools.

Get new guides and videos first — join the Telegram channel.

Twenty two hours quoted. Ninety one hours spent. That was one job, and the parser inside it took an afternoon, roughly what I said it would.

Everything that killed the estimate sat outside the code. A login I had not checked for. Four page templates I did not know existed. Six weeks of quiet breakage after the invoice had cleared and I had moved on to something else.

I run mobile proxy lines and production scrapers out of Singapore, so I have written these quotes and I have been handed them by other people. Bad scraping quotes are all bad in the same way. The person quoting priced the code, and the code was never the risk.

The build is the only predictable part

Ask an experienced person how long it takes to fetch a page, pull eight fields out of it and write them to a table, and you will get a good answer. That work is bounded and repetitive. After thirty or forty jobs you can call it inside a couple of hours.

So the build gets estimated well, and then it turns out to be maybe a fifth of the job.

The other four fifths hide in four places, and every one of them is visible before you quote if you actually go and look.

The site moves while you build. Not usually a redesign, just a variant template appearing or an advert slot dropping in above the element you anchored on. A two week build hands the site two weeks to shift under a price you already committed to.

The anti bot layer nobody checked for. This is the one that converts a bounded job into an unbounded one. The fetch loop runs beautifully across two hundred pages on your laptop, and somewhere past page nine hundred the responses turn into challenge pages. Now the job needs residential or mobile addresses, a real browser it never needed before, and a crawl rate a fifth of what you planned. A run cost you priced at zero suddenly has a monthly invoice on it.

The long tail. Eighty percent of records go down one path and the remainder each want their own branch. An out of stock product with no price element at all. An address that arrives as one line of free text where the others give you five fields. A record in a second language. A page on a completely different template because that catalogue came in through an acquisition and never got merged. Twenty minutes each, forty of them, and none of it visible on the first page you opened.

The maintenance tail. You deliver, you invoice, you leave. Three weeks later the price column is blank and you are back in that repository for free, because the quote said “scrape this site” and never said for how long.

The hour before the number

Here is the inspection I run now before any figure leaves my mouth. It takes about an hour.

Pull ten pages by hand. No code, no crawler. Open them in a browser, read them, save the HTML to disk. Ten pages will tell you whether the data is in the HTML at all, whether the layout is stable across the ten, and whether page seven is the awkward one that breaks the assumption you were about to price.

Open the network tab before writing a single selector. If the page is filling itself from a JSON endpoint, the job just became several times cheaper and considerably more stable, and you want that knowledge before you price a parser rather than after. Give it twenty minutes. The endpoint is either there or it is not, and if it is not you go back to reading HTML and quote on that basis.

Make an account. This is the biggest hidden multiplier I know of, and checking costs two minutes. A login means sessions, session expiry, cookie handling per worker, rate limits counted per account instead of per address, and a real chance the site bans the account and takes the crawl down with it. It also drags terms of service into a conversation you were treating as routine.

Count the page templates. Actually count them, by clicking through twenty or thirty records and tallying what comes back. One template means the job is what you assumed. Six templates means six parsers and six sets of test fixtures, which is a different price rather than a slightly larger one.

Then ask the question people forget: how fresh does this data have to be.

Freshness is the architecture, and the architecture is the price

Same site. Same eight columns in the output file. Four completely different systems depending on the answer to that one question.

Monthly is a script and a line in cron. One person runs it, glances at the file, sends it.

Daily starts wanting retries, a place for failed pages to land, and a human who notices when a run does not finish.

Hourly means a queue, workers, deduplication and monitoring that actually pages someone, because a silent failure at hourly cadence produces twenty four bad files before anybody opens one.

Near real time is a separate discipline. Open connections, latency budgets, infrastructure that has to stay up rather than a job that has to finish.

The spread between the ends of that range is something like ten to one on price, for identical fields off an identical site. Which is why “we want it live” deserves a follow up question about what live means in hours. Half the time the client says live and means daily, and you have handed them most of their budget back before writing a line of code.

Two lines on the quote, always

Most of this is a commercial problem wearing a technical costume.

Quote the build as a build. A fixed number of hours to deliver a working collector against the site as it stands on the date of signing, an agreed field list, and an agreed sample that has to parse correctly before the job counts as done.

Quote the run as its own monthly line. Maintenance hours, proxy spend, the time a template repair takes, and a response time you will genuinely honour at 2am on a Tuesday.

If the client only wants the build, sell them the build, and put in writing that the collector stops working on the day the site changes and that repairs are billable. That single sentence has saved me more money than any code I have written.

A scraper is a subscription to somebody else’s website. You have no contract with the site owner, they do not know you exist, and they will redesign whenever their marketing team wants a new hero section. Every one of those redesigns lands on your desk.

Charge for the looking

Here is the position I get argued with about: quoting a fixed price for a site you have never opened is a way to lose money, and it feels safe only because the loss shows up months after the invoice clears.

So I bill for the inspection. A small fixed fee, an hour or two of work, and a written finding at the end. Where the data actually lives. How many page shapes exist. What the anti bot posture looks like. What each freshness option would cost to build and to run.

Then the build quote is a number I believe. And if the site turns out to be horrible, the client paid a couple of hundred dollars to learn that instead of several thousand to discover it slowly through me.

A client who refuses the inspection fee has told you something useful. Usually that four other quotes are on the table and the decision is price alone, which means the winner will be whoever understood the job least.

The one still costing me money

The login miss is the story I tell most often, but it is not the mistake that has cost me the most.

That one is a collector I delivered in early 2024 for a single fixed number, with no run line and no end date on my obligation. It is still running. I have repaired it four times since, twice for a template change and twice when the site tightened its rate limiting. None of those repairs were billed, because nothing in what I wrote gave me the standing to bill them.

The build was maybe fifteen hours. The unpaid tail is past twenty and still counting, and the only reason it has not gone further is that the client is pleasant and the site is boring.

The fix was one paragraph of scope language, and it took me an embarrassingly long time to write it.

Where an hour of looking runs out

An inspection does not make the estimate accurate. It makes it honest, which is a lower bar and a far more useful one. A site can still add a challenge page in month three, or migrate to a framework that paints everything client side, and nothing you observed in August tells you what it does in November.

Inspection also says nothing about whether the job is one worth taking. Public pages only, the robots file honoured, a crawl rate that does not hurt the site, and an official API or a bulk feed preferred every single time one exists. A decent share of the enquiries I get fail that test long before the estimate matters, and the right answer to those is a polite no.

I quote these jobs and then live inside them, so this is the checklist I actually run rather than something copied off a consultancy site. The rest of what I have written on scraping cost, pipeline design and the proxy infrastructure underneath it is over here.

Get new guides and videos first — join the Telegram channel.

I have paid for legal advice about scraping exactly once. One hour, one jurisdiction, one specific job. What I remember is that she would not answer until I had broken my question into pieces, because what I walked in with was four questions wearing one coat.

I am not a lawyer and this is not advice

I have never studied law. I run proxy farms and production scraping pipelines out of Singapore, and everything below came from reading terms of service, getting things wrong, and that one paid hour.

This varies enormously by country. It varies by what the data is, and by what you do with it after you have it. Two operators running an identical scraper in two jurisdictions can land in completely different places.

So if there is real money on it, or a client, go and pay somebody who knows your jurisdiction and your use case. An hour of that costs less than one letter from the other side.

I am also not going to quote court cases. I have read the same summaries as everyone else, I have never been in a courtroom, and secondhand case law from a scraper on the internet is worth nothing to you.

The four questions

Can you reach the page without an account.

Do the terms permit automated collection.

What may you do with the data once it is on your disk.

Who owns the content itself.

Those are separate questions. Answering one does not answer the others, and almost everybody who asks me about this has answered the first and assumed the rest came with it.

Public does not mean permitted

“Publicly reachable” means something narrow. No login, no paywall, no token, nothing you had to get past. The server handed the page to an anonymous request that turned up and asked.

That matters, because getting around a control somebody installed on purpose is where the serious exposure sits. It is the difference between walking through an open door and picking a lock.

But it settles one of four. It says nothing about the terms, nothing about redistribution, nothing about who wrote the paragraphs on the page. I treat it as a floor. No means stop. Yes means keep reading.

Robots.txt is a request, and I follow it

Nothing enforces that file. No gate, no token, no consequence at the protocol level. Your fetcher honours it only if you wrote the code that honours it, and plenty of people treat that as licence to skip it.

My reason for reading it is not the one you usually hear. That file is the only channel a site owner has to tell me anything before I arrive. They do not know who I am and they cannot email me. The conversation runs one way and it is the only one on offer.

If I read it and go anyway, I have surrendered any standing to complain about what happens next. Blocks, bans, a letter. I was told, in the only way available to them, and I went anyway.

One exception I will defend and some people will not like. I will fetch a disallowed path once, by hand, before deciding whether the pipeline goes near it. I do not point a crawler down it.

An account is a promise you made

The moment you register, you clicked something, and it usually said accept. A vague question about public data became a specific commitment in writing.

The same applies to an api key, which people find much easier to accept. Nobody argues that api terms are optional. A login is that shape with a worse interface.

So one page fetched two ways sits in two categories. Anonymous, you are a stranger the server chose to serve. Signed in, you are a party who agreed to something and then did the opposite. That second one is far easier to lose, because nobody has to prove intent or damage. They show the clause and they show the traffic.

There is also a person attached to an account. An email, often a phone, sometimes a card. Anonymous scraping is anonymous. Account scraping is you, by name, in somebody’s logs.

Collecting and using are two different jobs

You can be completely fine gathering something and completely wrong publishing it. Plenty of terms permit access and forbid redistribution a few lines apart in the same document.

Ask it as three questions. Can I fetch this. Can I keep this. Can I republish, resell, or train on this.

Internal analysis is the safest by a distance. A price history you use to set your own prices is a different animal from a comparison site you charge people to read. Resale is where I would want real advice every time. The moment somebody is paying for the data rather than for something you built on it, the conversation changes.

Facts and writing are not the same material

Nothing about reachability or terms answers the copyright question.

The rough shape, read by a layman. Facts are hard to own. A price is a fact. A stock count is a fact. A street address is a fact.

Expression is owned. A nine hundred word review somebody sat down and wrote is theirs. A product photograph belongs to whoever shot it.

So I store facts and a reference back to the source, and I leave the prose alone. Copyright is the easiest of the four to get right, which is why scraped article farms irritate me more than they should.

Personal data is where the weight actually is

Names, emails, phone numbers, photographs, profile pages, anything that points at a specific living person. That category is regulated in most jurisdictions with real teeth, and how visible it was does not undo that.

Singapore has the Personal Data Protection Act. Europe has gdpr. California has its own. They differ in the detail and agree on the principle: the fact that anybody could see it does not mean you may collect it and build on it.

The underrated half is holding it. Collecting takes an afternoon. Holding is forever. You have to secure it, know where every copy lives, delete it when somebody asks, and answer for it if a backup ever walks out of the building.

The risk is also lopsided in a way nothing else in this work is. Get a product price wrong and a dashboard is wrong for a week. Get personal data wrong and there is a regulator in the room.

So the rule is blunt. I do not collect personal fields unless those fields are the entire point of the job. Otherwise they get dropped at the parser, before anything reaches storage.

Rate is an ethics question with its own answer

This one sits outside the four and day to day it matters more than any of them. A scraper that costs somebody real money in serving costs is doing harm even where it is entirely permitted.

I have an unusual angle on this because I sell bandwidth. Proxy traffic on my own lines is metered by the gigabyte, so I know what a gigabyte costs to move. When a headless browser loads a full page it pulls the html, the css, every javascript bundle, the fonts and the images, and the site on the other end is paying its own version of that number to send it. Run two hundred of those in parallel and you have moved a line item onto somebody’s invoice.

On a large platform that disappears into the noise. On a small site on shared hosting it is the whole box, and I have watched one of my own go down exactly that way.

So I run slower than I need to. One request at a time per host, a delay that turns an hour into a night, scheduled for their quiet hours rather than mine.

The five rules I operate by

Prefer an official interface. If there is an api I use it, and if it is metered I pay. A paid channel is the site stating in plain numbers how much automated traffic it wants.

Identify yourself. I put a real name and a contact address in the user agent for anything I am not embarrassed by. Most scraping advice says the opposite, blend in and look like a browser, and that was written for a different kind of job. If you would not sign your name to the traffic, you have your answer.

Go slowly, per above.

Take only what you need. Every extra field is something to justify later and something that can be quietly wrong for months.

Be willing to stop. If a site asks me to stop, I stop that day, and I do not argue about whether they were entitled to ask.

The thing I collected and should not have

I was tracking listing prices across sellers on a marketplace. Three fields mattered: price, currency, timestamp.

But I stored the raw json response whole, because that is exactly what I tell everybody to do. Keep the payload, parse from it later, never lose what you fetched.

That response carried the seller profile with it. Display names, rough locations, contact handles, join dates, review histories. I never used one of them. I never really looked. I just had them, on a box in my house and inside every encrypted backup taken since, for months.

The fix took an afternoon. An allowlist at the fetch step so the raw store only keeps declared fields, then a pass over the old objects to strip and rewrite.

What bothers me is that there was never a decision. I did not weigh it up and choose badly. My default was keep everything and sort it out later, and for personal data that default is the mistake.

The one I still cannot answer

When a page fetches its own data from a json endpoint and no account is required to call it, I treat that endpoint as publicly reachable, same as the page. That is my working position.

The argument against it is that the endpoint was never published for anyone but their own front end, so calling it directly is a different act from loading the page. I do not think that argument is stupid. I have just never seen anything that settles it.

So I behave identically either side of that line. Slow, identified, terms read, no personal fields kept. The full set of guides and the tooling I actually run is at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

My first run scored Firecrawl at 53.3%. That number was wrong by nearly fifty points, and every one of the reasons was mine, not theirs.

I want to open with that because it is the useful part of this write-up. The success rate at the bottom is one provider, on one date, from one connection. The five ways a test harness can quietly libel a tool are general.

Disclosure first

Firecrawl gave me API credits to run this. That buys sample size and nothing else. No review of the draft, no say in the verdict, no agreed placement. I asked for the credits because the free tier is 1,000 credits a month and this run does not fit inside that, for reasons that turn out to be worth a section of their own.

The account is on the free plan otherwise. Version tested: the v2 API (/v2/scrape), 5 and 6 September 2026, from a business line in Singapore.

The method

Same six categories the earlier comparison on this site used: e-commerce, SERP, social, travel, real estate, business listings. Five URLs per category, thirty URLs, four repeats each: 120 requests planned. 89 were scored, and the results section below says why.

Every request is the same call:

{"url": "...", "formats": ["markdown"], "onlyMainContent": true,
 "proxy": "auto", "timeout": 60000, "maxAge": 0}

maxAge: 0 matters. Leave it out and you may be scored on Firecrawl’s cache rather than on a live fetch, which measures their storage layer and not their ability to get through a door.

A response counts as a success only if the markdown contains a marker that could not appear on a block page or an error page: “add to cart” for a product page, “guest reviews” for a hotel, a follower count for a profile. Length alone is not enough. A Cloudflare interstitial is a perfectly well-formed 200 with several thousand characters in it.

The full target list and the raw per-request JSONL are available on request. Ask and I will send them, then re-run it and argue with me.

The five bugs, because they are the transferable part

Rate limiting read as failure. The free plan caps at 10 requests a minute. My first pass fired faster than that, and 8 of 30 requests came back as rate-limit errors. Naively bucketed, that scored X.com at 0% — a category “failure” that was entirely me holding the throttle down. Any benchmark that does not say what its request rate was, and what it did with 429s, is not telling you about the provider.

A block-detector that matched the content. I flagged blocks by looking for “captcha” and similar strings. One of my SERP targets was a Google search for “captcha solving service”. The query echoes into the page, the page is 39,000 characters of perfectly good results, and my detector called it a block.

The same bug wearing a different hat. Zillow’s sign-in copy contains “Takes just a moment.” My detector was looking for “just a moment”, which is the Cloudflare interstitial title. Another 50,000-character success scored as a block.

Dead targets. Three Amazon ASINs in my list had been delisted and returned Amazon’s real 404 page. One Booking.com slug was gone. Those are 404s from the target, not failures from the provider, and counting them against Firecrawl is just wrong. They now get their own verdict and sit outside the denominator.

Running out of money. The one I did not see coming. Halfway through the final run my account hit zero and the API started returning 402. Those 31 requests went into the results file as errors, and the summary line dutifully reported 73.3%. My own balance, scored as Firecrawl failing to fetch a page. It now stops the run when it sees a 402 and keeps those rows out of the denominator, which is the difference between 73.3% and the 98.9% below.

Fixing the first three, replacing the dead URLs, and taking the 402s out moved the same provider on the same targets from 53.3% to 98.9%. Nothing about Firecrawl changed in between.

The general lesson is not subtle: when a benchmark and a vendor disagree, the benchmark is the thing that has not been tested.

What it actually scored

category	scored	ok	success	median	p90
e-commerce (Amazon)	20	20	100%	7.49s	9.21s
SERP (Google)	20	19	95%	4.50s	5.21s
social (X.com)	20	20	100%	7.77s	11.29s
travel (Booking)	20	20	100%	5.27s	6.63s
real estate (Zillow)	9	9	100%	5.99s	9.59s
business listings (Yellow Pages)	0	–	no data	–	–
all	89	88	98.9%	5.85s	9.21s

Read the last two rows before the headline. I ran out of credits at request 90 of 120, so Zillow got 9 of its 20 attempts and Yellow Pages got none at all. Yellow Pages is a gap rather than a result. The honest thing is to leave the cell empty rather than quietly drop the category and report a rounder N.

One failure in 89. A Google SERP that came back as 377 characters when the same URL returned 61,814 characters on its other three attempts. Not a block page, not an error, just a stub with a 200 on it.

That is the failure mode worth caring about, and I want to be honest that I cannot tell you what was in those 377 characters, because my harness recorded the length and threw the body away. It saves failure bodies now. That is the kind of thing you only discover by having a failure.

What it costs, per target

The first version of this section quoted two blended rates: about 4 credits a request on a mixed run, and 7.6 on a run weighted towards Amazon, X.com and Booking.com. Both were measured. Both would have misled you, because a blended rate says nothing about the one target you actually care about, and the 7.6 made Booking.com look expensive when it isn’t.

So I priced each target on its own on 15 September 2026, same call as above, two URLs per category. The v2 response carries a creditsUsed field that my harness wasn’t reading the first time around. I also read the account balance before and after every call. The two agreed on all twelve requests.

target	requests	credits each	note
Amazon product page	2	1
Booking.com hotel page	2	1
Zillow search results	2	1
Yellow Pages search results	2	1
X.com profile	2	30	returns structured profile and post data
Google SERP	2	0	both failed; failed requests are not billed

A normal /scrape is 1 credit, with proxy: "auto" on. X.com is the outlier at 30, and that is by design: what comes back is structured profile and post data, not a markdown page. Twenty X.com requests at 30 is 600 credits, out of the 677 the hard-weighted run burned. The X.com calls pushed that average to 7.6, not Booking.com.

Two caveats. Two requests per target is a price check, not a benchmark. And both Google requests on the 15th failed outright (“all scraping engines failed”), where the same category went 19 for 20 on 5 and 6 September. I haven’t re-run Google since, so read that as unexplained, not as a trend.

On the free tier’s 1,000 credits a month, that is roughly 1,000 ordinary pages or 33 X.com profiles. If you are budgeting, count your X.com calls separately. Everything else I pointed it at billed flat.

Limits of this run

N=89 is small. It is one provider with no control arm running beside it, so “98.9%” is a statement about these thirty URLs on these two days and not a ranking. Free-tier concurrency is 2, so this says nothing about behaviour at fifty parallel requests, which is where managed scrapers usually start to hurt. Everything ran from one Singapore connection; a US or EU egress may see different treatment on the same targets.

I have not tested the crawl or extract endpoints at all. Only /v2/scrape.

The number I would actually watch

Not the success rate. The spread.

The same Google SERP URL, four consecutive fetches, minutes apart:

URL	chars across 4 repeats
?q=best+web+scraping+api	19,939 to 55,983
?q=residential+proxy+pricing	25,698 to 66,443
?q=datacenter+vs+mobile+proxy	377 to 61,814

Every one of those is scored a success except the 377. A page can be a third of its own size on the next call and still tick every box a success check looks for. Latency moved too, though far less dramatically: median spread within a URL was 1.42s, worst was 7.60s.

A provider that returns a full page nine times and a stub on the tenth will pass a single-shot benchmark and then quietly poison a week of your data, because the stub is a 200 and your pipeline has no reason to think anything went wrong. Which is the argument for schema validation on the way in, and the argument against trusting any success rate (mine included) that was measured once per URL.

A scraper logs into a site, adds an item to a cart, gets two pages into checkout, and then gets logged out. Nothing in the code changed. What changed was the address underneath it. The proxy rotated in the middle of the flow, the site saw the same logged in cookie arrive from a new city, and it dropped the session. That single behavior, whether the proxy holds an address or swaps it, is what rotating and sticky sessions are about, and picking the wrong one is one of the most common ways a working scraper starts failing for no obvious reason.

I run proxy infrastructure and production scrapers, so this is a setting I have gotten wrong and had to fix, not a diagram off a vendor page. Rotating and sticky are not two products you buy. They are two ways of using the same pool. Here is what each one actually does at the proxy layer, how long sticky windows really last, and how I decide between them per job.

What a session actually is

Both settings revolve around one idea, so start there. To a site, a session is a sequence of requests it treats as one visitor. Two things carry that identity. One is the cookie the site handed you at login. The other is the address that cookie keeps arriving from. A careful site cross checks the two. So the identity your scraper presents is not just the cookie, it is the cookie and the ip together, and the moment those two stop agreeing, the site has a contradiction it can act on.

That framing is the whole article in one sentence. Rotation and stickiness are just choices about whether the address stays consistent with the cookie or not.

How rotation works at the proxy layer

Rotation means the gateway hands you a fresh address on some schedule, or a brand new one on every request. You point your client at a single endpoint, and behind that endpoint the provider swaps which upstream address your traffic actually leaves from. The goal is to spread load so no single ip gets hammered, and to keep any one address from building a reputation under your job.

For wide, stateless collection that is exactly what you want. Each page stands alone, there is no session to break, and spreading requests across many addresses keeps a single ip from tripping a rate flag. The cost is that rotation, by design, keeps changing where you appear to come from.

How a sticky session works

A sticky session is the opposite setting on the same pool. Instead of a fresh address each time, the gateway pins one address to you and holds it for a window. You usually request it by putting a session id into the proxy username, or by connecting to a dedicated sticky port. Every request you send through that token leaves from the same upstream ip until the window ends.

Under the hood it is a lease. When you send a session token, the gateway looks it up, reuses the address mapped to it, or picks one and remembers the mapping if there is none yet. Hold the token steady and you hold the address. Let it change or expire, and the gateway is free to hand you someone new. Like any lease, it has a length, and it runs out.

How long sticky windows really last

Sticky windows are shorter than most people assume. A lot of residential pools default to something in the one to ten minute range, and many let you request up to 30 minutes or an hour if you ask for it. Datacenter addresses can be held far longer, because you effectively own that address for the rental.

Residential and mobile stickiness is best effort. You are borrowing a real home line or a real phone, and that connection can drop at any time and take your window with it early. Mobile is looser still, because carrier networks reassign addresses on their own schedule and share each one behind nat. So the same sticky setting can give you a rock steady address on datacenter and a window that quietly resets on residential or mobile. Design for the address changing, not for it to last.

The two failure modes

Both failure modes come from mismatching the setting to the job.

Rotate in the middle of a stateful flow and you get the logout from the opening. You log in from one address, and two requests later the same session cookie shows up from a different ip in a different city. No real person teleports between clicks, so the site ends the session or throws a challenge. The flow breaks, and it looks random until you realize the address moved under you.

Hold one sticky address across a huge stateless crawl and you make the opposite mistake. You funnel thousands of requests through a single ip that a rotating pool was built to spread. That one address climbs the rate counters fast, earns a reputation by itself, and gets throttled or blocked while the rest of your pool sits idle. Stickiness concentrates trust on one door, which is wrong when the job never needed a door held open.

Keep one session’s identity together

A clean mental model prevents most of this. Treat one logical session as one bundle: a single session token, a single cookie jar, one consistent set of headers, all pinned to one address for the window. The bug I see most is a shared cookie jar bleeding across rotating addresses, so the same login state leaks out of three different ips inside a minute. That is a contradiction you built yourself.

Geography is part of that identity too. A sticky window should stay inside one city, and ideally one network, so a flow does not jump from a Singapore address to a US address halfway through checkout. Prices and availability shift by location, so a wandering session can also quietly corrupt the data you collect, not just get you flagged.

Concurrency does not require rotating inside a session

People worry that stickiness kills throughput. It does not, you just parallelize differently. Each sticky session is one worker holding one address for one flow. To run 50 logged in flows at once, you open 50 sticky sessions, each with its own token, address, and cookie jar. You get concurrency from many held addresses side by side, not from rotating inside a single session. Stickiness and scale are not in conflict.

How I decide per job

The decision comes down to one question: does this flow carry state the site ties to the address? If yes, a login, a cart, anything sequential, I go sticky and size the window to the length of the flow with a margin. If no, a wide stateless pull, I rotate, often on every request, to spread the load. Then I test on the real target, because a site can tie a session to the ip harder or looser than you would expect.

Two jobs I ran back to back make it concrete. One was a login gated account area, a few pages deep per visit. Rotating per request logged me out every second or third page, until a 10 minute sticky window let each visit finish clean. The other was a broad price crawl across tens of thousands of public pages. A sticky address there hammered one ip into a rate block within the hour, while per request rotation spread the same work quietly across the pool. Same tool, opposite settings.

The honest limits

Matching rotation to the job keeps your sessions intact and your load spread. It does not make anything undetectable, and no setting on a proxy does. It does nothing about a loud client, a bad tls handshake, or a request rate no human would produce. And it changes nothing about what you are allowed to collect. Public data, the robots file respected, an official api preferred where one exists, and whatever a site’s terms actually say. Rotation and stickiness are plumbing, not permission.

I run this infrastructure in production, so the way I pick between rotating and sticky here is the way I actually run the jobs. If you want the full written guides and the pools and settings I genuinely use, with real block rates and no undetectable promises, they are at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

A captcha appears, and almost everyone reads it as a puzzle. Prove you’re human, click the crosswalks, check the box, wait for the spinner. So the instinct is that the captcha is the test, and if you can just answer it, you’re through. That instinct is backwards. By the time a challenge renders on your screen, a decision about your session has already been made, and it wasn’t leaning your way. The captcha is not the test. It’s the symptom of a test you already failed somewhere upstream.

I run proxy infrastructure and production scrapers, so I’ve spent a lot of time watching how these challenge systems decide who gets waved through and who gets stopped. This is a defensive explainer, written from the site’s point of view: how modern captchas actually work, and why the whole market of paying farms to solve them is a dead end. It isn’t a guide to beating a challenge, because beating the challenge was never the real problem, and once you see how the pieces fit, you’ll understand why.

What a captcha is actually for

Start with the purpose, because it isn’t what the name suggests. A captcha isn’t there to check whether you can read distorted text or spot a bus. It’s there to raise the cost of being an automated client at the exact moment a site is unsure about you. It’s a decision point. The site has some doubt, and rather than block you outright or let you through, it inserts a small tax that’s cheap for a real person and expensive for a machine.

The classic captcha, the wall of warped letters, worked while computers were bad at reading messy text and people were good at it. That gap closed. Modern machine vision reads distorted text more reliably than humans do, so the warped letters stopped separating anyone from anything. That’s the first clue about this whole space: any challenge that’s purely a puzzle eventually gets solved by software. So the providers moved the real work somewhere else.

The real shift: scoring, not puzzles

Here’s the change that matters most. A modern captcha system doesn’t challenge everyone. It scores everyone, quietly, and only shows a visible challenge to the small slice it’s unsure about. Most visitors never see a puzzle at all, because the system already decided they were fine. The visible challenge is the last resort, reserved for traffic that already looks doubtful. So seeing one at all means you’ve landed in the doubtful bucket.

The clearest version of this is the invisible scoring model, the reCAPTCHA v3 style approach. It never shows a puzzle. It sits on the page, watches the session, and hands the site a number, a risk score, roughly how confident it is that you’re a real person. The site owner decides what to do with that number. A good score, let them through. A bad score, ask for a password again, hold the action, or escalate to a visible challenge. The scoring happens in the background, before any decision is shown to you.

hCaptcha, which a lot of sites moved to, works on the same principle. It leans on a privacy story and its own scoring, but underneath it’s still risk based. Cloudflare Turnstile pushed the idea further by trying to drop the image puzzle almost entirely. It runs a series of small checks inside your browser, quiet probes a real browser passes without the user doing anything. You might see a checkbox and a brief spinner, or nothing at all. What it’s really doing in that pause is measuring whether the thing loading the page behaves like a genuine browser or only claims to.

That’s the part people miss about the little “I’m not a robot” checkbox. Clicking a box is trivial. What the widget measures is everything around the click: how the pointer arrived, the timing, the state of the browser, and the score it already computed for the session. A real person with a clean session gets a single click and a green tick; a suspicious session clicks the same box and gets a hard image grid, because the click told the system nothing good enough to skip it.

What actually feeds the score

So what goes into that number? The same stack that drives bot detection generally. The reputation of the ip and the network it sits on. Whether the tls handshake matches the browser it claims to be. Whether the request headers are consistent with each other. What the browser reveals when the provider runs code inside it: the canvas, the fonts, the timing, the small properties a real device has and a cloned one struggles to fake. And how the session behaves over time, the rhythm of the clicks and the moves.

None of those is the captcha. They’re the inputs that decide whether a captcha ever appears. Which leads to the reframe: a captcha is a readout that you already tripped something upstream. By the time the challenge renders, the network you came in on, your handshake, your fingerprint, or your pace has already pushed your score into the range where the site stopped trusting you. The puzzle is downstream of the score. And that single fact is what makes the solver industry a losing proposition.

Why solver farms are a dead end

Solver farms come in two flavors. Human farms, rooms of low paid workers who type the answer for a fraction of a cent. And machine solvers, models trained to answer the common challenge types automatically. Both do the same narrow thing: they take the puzzle that got served and hand back the expected answer. The puzzle in, the answer out.

Here’s why that doesn’t work in any durable way. The puzzle was never the problem. The low score that summoned the puzzle is the problem, and the solver does nothing about it. You can answer the grid perfectly and still hand the site a session that arrived from a flagged network with a cloned fingerprint moving at machine pace. The site scores the token the challenge produces in the context of that whole session, and a right answer from an obviously automated context still looks like exactly what it is.

The economics run against the solver too. Issuing a harder challenge is nearly free for the provider. Solving one costs the farm real money and real time on every request. When a solver service gets good at one challenge type, the provider changes it. The provider also sees this pattern across millions of sites at once, so a trick that starts working on one gets neutralized everywhere within a short window. You’re renting a solution to a problem the other side can reprice against you whenever it wants.

There’s a cleaner reason to stay away from that market. Paying a farm to defeat a site’s challenge is, on most sites, a direct violation of the terms you agreed to. Beyond being brittle, it’s usually against the rules, which makes it indefensible the moment it matters. And it’s fragile: one update from the challenge vendor and a pipeline built on a solver stops overnight. That isn’t infrastructure. It’s a bet you keep having to make again.

The honest fix is to not trip it

The durable fix for captchas is the opposite of what the solver market sells. It’s not defeating them, and it’s not solving them. It’s not tripping them in the first place. A challenge is a signal that your session already looked wrong upstream, so the work that lasts is upstream, making the session not look wrong. That’s the only move that doesn’t decay the week after you make it.

In practice that’s the boring path I keep coming back to. Arrive on a clean network with an honest reputation instead of the cheapest datacenter pool. Identify your client honestly where a site expects it, rather than dressing a server up as a browser it isn’t. Collect public data only, nothing behind a login, a paywall, or anything personal. Move at a pace the site can absorb, space the requests, back off on errors, and cache so you never ask twice. Prefer an official api or data feed when one exists, because that front door doesn’t score you and doesn’t care about your fingerprint at all.

I want to be straight about the limits. None of this makes anything undetectable, and I wouldn’t trust anyone who tells you it does. Detection and scoring keep improving, and what passes quietly today can be flagged tomorrow. A clean network and honest behavior aren’t a trick that beats the wall. They’re the absence of the contradictions that get you scored down in the first place.

I run this infrastructure in production, so the frameworks, proxy setups, scraping apis, and honest tool reviews I write about are the ones I actually deal with. If you want the full written guides and the compliant patterns, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the way this actually works.

Get new guides and videos first — join the Telegram channel.