Author: Xavier Fok

The first time I wired a language model into a scraper, I thought I had cheated the whole game. I pointed it at a product page, told it in plain words to read the page and give me the name, the price, and whether it was in stock, and it did, perfectly, on the very first try. No selectors, no fiddling with the HTML, no brittle rules to maintain.

Then I ran the same trick across ten thousand pages, and two things happened at once. The bill was frightening, and somewhere in the middle the model handed me a price that was never on the page at all. This piece is about scraping with language models: where they genuinely help, where they quietly hurt you, and how to use one without setting fire to your budget or your data.

A language model is not a scraper

The single idea most of the excitement skips past is this: a language model is not a scraper. It does not fetch a single page. It has no idea what a proxy is, it cannot rotate an address, it cannot obey a rate limit, and it will not keep you welcome on a site.

Every hard part of scraping, getting the page down cleanly and staying gentle while you do it, is completely unchanged. The model only touches the very last step, the reading of a page you have already fetched. It is a clever, expensive reader you bolt onto the end, and remembering that it is only the end is what keeps you sane.

What the old way looked like

To see why people got excited, remember what extraction used to mean. You open the page, find where the price lives, and write a rule that says the price sits inside this element, at this position in the HTML. That rule is a selector, and it works right up until the site changes its layout, at which point your selector points at nothing and your scraper silently starts collecting blanks. Every operator has felt that particular pain.

The promise of the model is that you skip the fragile rule entirely and just ask, in plain language, for the thing you want.

Where the model genuinely wins

The honest answer to “where does that pay off” is messy, varied pages, the ones where no two look alike. If you are pulling product details from a hundred different shops, or job listings from fifty different boards, writing and maintaining a separate selector for every single site is a maintenance job that never ends. A model reads meaning rather than position, so it can pull the price from all hundred layouts without you hand crafting a rule for each. That is the real win, and it is a genuine one.

The second place a model earns its keep is turning prose into fields. Imagine a rental listing where the bedrooms, the floor area, and the year it was built are all buried inside a paragraph of free text, with no tidy element to point a selector at. A classic parser is helpless there, because there is nothing structural to grab. The model reads the sentence the way a person would and lifts the numbers straight out. Any time your data is trapped inside human writing rather than in the shape of the page, that is a job the model is good at.

Where the plain selector still wins

The classic parser is far from dead. If your target is a single site with a stable layout and you need a few million pages from it, a plain selector is basically free. It runs in a fraction of a millisecond, it costs nothing once written, and it gives you the same answer every single time.

Running a language model against every one of those millions of pages is slow, and it costs real money on each and every call. You never pay a model to read what a one line selector already grabs perfectly.

The cost is not a footnote

Every page you hand to a hosted model is charged by the token, both the text going in and the answer coming out. A full HTML page is enormous once you count the tags, the scripts, and the styling, so you are paying to send the model a mountain of markup it does not even need. On ten thousand pages that stings. On ten million it is a budget line you will have to defend to whoever signs the cheques.

Worse, it is not a one time purchase: it lands again on every run, so a nightly job multiplies the bill by the number of nights. The parser costs you the afternoon it took to write and effectively nothing after that.

Strip the page before you send it

The trick that makes the model affordable is the single most important habit here: do not feed raw HTML to the model. Strip the page down first. Throw away the scripts, the styling, the navigation, the footer, all the boilerplate, and hand the model only the slice of text that could plausibly hold your answer.

A smaller input means a smaller bill and, as a bonus, better accuracy, because the model is not hunting through a haystack. The parser and the model are partners: the parser trims the page, and the model reads what is left.

The real danger is silent hallucination

The thing that keeps me up at night is not the cost, it is the lying. A selector that breaks fails loudly. It points at nothing, you get an empty field, and you notice something is wrong. A model that is confused fails quietly. It hands you a clean, confident, perfectly formatted answer that simply is not true: a price it inferred, a date it guessed, a value that appears nowhere on the page.

That plausible, silent wrongness is far more dangerous than an honest error, because it slips into your data looking exactly like the real thing.

How to keep the model honest

You never trust the model blind, and there are a few concrete defences. First, force the output into a strict shape, a fixed set of fields and nothing else, so the model cannot ramble or improvise. Second, validate every field afterward with plain code: is the price actually a number, is the date a real date, does the link sit on the domain it should. Third, and most important, tell the model to copy only what is present and never to reason about what is missing.

The more room you give it to think and infer, the more it invents. The safest use by a wide margin is pure extraction, not clever guessing.

Which model, and where it runs

The big hosted models are the most capable and the quickest to start with, but you send every page off to someone else and pay for the privilege each time. A smaller model you run on your own hardware costs more to set up, yet it is dramatically cheaper at volume and keeps your data in house. The right pick is decided by how many pages you have, not by which model is fashionable this quarter.

The hybrid is the real answer

If you take one practical pattern from all of this, take the hybrid. Write a cheap, boring parser for the ninety percent of your pages that are regular and predictable, and keep the language model in reserve as the fallback for the awkward remainder the selectors cannot handle, plus those free text fields.

You get the speed and near zero cost of parsing across the bulk of the work, and you only pay for the model on the small slice where its flexibility is actually worth the money. That split is how real production systems use this, quietly, without any drama.

There is one more use worth naming: let the model write the parser instead of being the parser. Point it at a new site once, at build time, and ask it to work out the selectors. Then run those cheap, fast selectors forever after. The model does the one off thinking, the parser does the millionfold repeating, and the running cost stays on the floor.

It does not change the front of the pipeline

Here is the part that is easy to forget in all the AI talk: none of this touches the front of your pipeline. The model reads the page after you have fetched it, which means you still have to get the page down cleanly in the first place, and that is the same job it always was. The proxies, the rotation, the gentle pace, the rate limits, every bit of it works exactly as before whether a selector or a model reads the result at the end.

A language model will never get you past a block. It only makes sense of a page you were already able to fetch. That is where a healthy proxy setup still does the heavy lifting, because the smartest reader in the world is useless if you cannot get the pages in front of it. I run my own collection on a managed mobile proxy pool for that reason. The full written walkthrough, the stripping step, the strict shape I force on the output, the validation I run afterward, and the pool that fetches the pages, live at dataresearchtools.com.

The honest limits

A cleverer reader does not change the rules one bit. It grants you no new permission to take anything. Public data is still the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how smart the thing at the end of the pipeline is. The model only helps you understand pages you were already allowed to fetch.

The simple version: a language model is a reader you attach to the end, and it shines on messy, varied pages and on data trapped inside human writing. It is the wrong tool for one stable site with millions of regular pages, where a plain selector is faster, free, and honest. Strip the page, force a strict shape, validate everything, and never let the model guess at what is not there. The fetching, the proxies, and the manners are still entirely your job.

Get new guides and videos first — join the Telegram channel.

The first scraper I ever ran at real scale did fine on a small job and then fell flat on a big one. On ten thousand pages it was quick and quiet, finished before I finished my coffee. Then the job grew to a few million pages, I pointed the same single program at it, and it simply could not keep up.

By the time it reached the end of the list, the pages at the start had already changed. I was collecting a snapshot that was stale before it was even complete. That is the moment a scraper stops being a program and has to become a system. This piece is about how you make that jump: from one program on one machine to a coordinated fleet that finishes the big jobs while they still mean something.

Distributed is coordination, not force

The first thing to understand is that going distributed is not about hitting sites harder. It is about spreading an honest workload across many small workers so it finishes in time, without any single machine, or any single address, carrying the whole thing.

The mental shift is this: stop thinking of your scraper as one clever program, and start thinking of it as a system of small, identical, forgettable workers all pulling from a shared list of work. The unit of work is a single URL to fetch. The heart of the system is a queue that holds those URLs. Everything else, the workers, the storage, the proxies, hangs off that one idea.

Why one machine hits a wall

If you already went async and made a single scraper fast, you might wonder why one box is not enough. The honest answer is that a single machine has ceilings you cannot buy your way past forever. It has only so much memory and so many open connections. It leaves from a narrow slice of addresses. And worst of all it is one point of failure, so when it falls over at three in the morning, the whole job stops.

You can scale up, buy a bigger machine, and that helps for a while. Past a certain size you scale out instead, adding more modest machines, because ten ordinary workers beat one heroic one that you cannot replace when it dies.

The shared queue is the frontier

The piece that makes all of this possible is the shared queue, sometimes called the URL frontier. Instead of a list living inside one program’s memory, you put the URLs to fetch into a real broker that every worker can reach, something like Redis, RabbitMQ, or Kafka.

One side of the system discovers work and pushes URLs in. The other side, the workers, pull URLs out and fetch them. That single move, lifting the queue out of one process and into a shared broker, is what lets you run twenty workers, or two hundred, against the same pile of work without them tripping over each other.

Make your workers stateless

The workers themselves should be as dumb as you can make them, and that is a compliment. An ideal worker does one boring loop: pull a URL from the queue, fetch the page, parse it, write the result, ask for the next one. It holds nothing important in its own memory.

Because each worker is stateless and identical, you can start ten more when the job is big, kill any one of them mid run, and restart it, and you lose nothing but the single page it was holding. That disposability is the entire trick to scaling out.

The workers can afford to be forgetful only because the memory lives somewhere shared. All the state that actually matters, which URLs you have already seen, which ones are done, which ones failed and how many times, sits in a central store that every worker reads and writes: a database or a shared key store off to the side. A worker is just a temporary pair of hands. The real record of the job lives in the middle.

Deduplicate where everyone can see it

The moment you have many workers, a new problem appears that a single program never had: duplication. Two workers can easily discover and fetch the very same URL, wasting requests and polluting your data with copies. You cannot solve this inside one worker, because it cannot see what the others are doing.

You solve it centrally, with a shared record of everything already seen, checked before a URL is ever added to the queue. For huge crawls, people reach for a compact structure like a Bloom filter to hold that seen set cheaply. The point is simple: dedup where every worker can see the answer, not in any single worker’s head.

Coordinate proxies across the whole fleet

Here is the part that ties straight back to running real infrastructure: the proxies. On one machine your address pool was one program’s concern. Across a fleet it becomes a shared resource that every worker draws from, and if you are not careful, each worker politely obeys a per host limit on its own while twenty of them together hammer one site into the ground.

The caps have to be global, not per worker. Your rate budget for any single host, and your rotation across the pool, has to be enforced across the whole fleet, so the site sees one reasonable stream of traffic no matter how many machines are behind it.

Bounded queues and back pressure

You also have to stop your own discovery stage from drowning the system. A crawler that finds links faster than the workers can fetch them will happily shove ten million URLs into the queue and bury the broker.

The fix is a bounded queue and back pressure. You cap how much work is allowed to wait at once, and when the queue is full the discovery side pauses instead of racing ahead. The workers pull at whatever pace they can actually sustain, and the front of the system is forced to match the back. A fast producer must never be allowed to outrun a careful fetcher.

Design for failure, because it is the weather

At this scale failure stops being an event and becomes the weather. With a hundred workers running, one of them is always dying, and you design for that from the start.

The key rule is that a URL a worker took but never finished must find its way back into the queue. Good brokers give you this directly: a message stays reserved but invisible while a worker holds it, and only disappears when the worker confirms success, so a crash quietly returns that URL for someone else to retry. Because a page can therefore be fetched more than once, your writes have to be safe to repeat, so a retry just overwrites the same clean record instead of creating a mess.

Split the pipeline into stages

A scraper that scales well is usually split into stages joined by queues, rather than one worker doing everything from end to end. Fetching is one stage, parsing is another, storing is a third, and a queue sits between each pair.

These stages have very different appetites. Fetching is mostly waiting on the network, while parsing a heavy page can chew real processor time. When they are separate, a slow parser cannot stall your fetchers, and you can scale each stage on its own, running many light fetchers feeding a smaller number of heavier parsers, or the other way round.

Watch the queue depth

You cannot watch a fleet by eye, so it has to report on itself. The numbers worth keeping in front of you are the queue depth, the throughput in good pages per minute, the block rate across the fleet, and the health of the workers themselves.

The most telling one is queue depth over time. If the pile of waiting work is growing, your workers are falling behind and you need more of them or a gentler discovery stage. If it is draining steadily, you are keeping up. That one line is the cockpit for the whole system.

Do not over engineer it

None of this requires an enormous cluster on day one. A single broker and a handful of worker processes spread across two or three ordinary machines will take you an astonishingly long way, into the millions of pages. Add more workers when the queue depth tells you to, and only add more machines when a single one is genuinely full.

And most jobs never need any of this. If a single well written async scraper on one box finishes your job inside the window you have, stop right there. Distribution has a real cost: a broker to run, workers to deploy, state to keep consistent, and failures that are subtler than a crash on one machine. Earn the complexity only when one machine genuinely cannot keep up.

The honest limits

Scale changes your throughput, not your permission. Running a hundred workers does not entitle you to anything a single script could not already take. Public data stays the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how many machines you have behind it.

Spreading your fetching across many workers multiplies your footprint on every site you touch, so a shared pool that rotates cleanly and heals itself between passes is what keeps a fleet quiet and welcome. I run my own heavy jobs on a managed mobile proxy setup for exactly that reason. The full written walkthrough, the queue shapes, the worker layout, and the pool I actually use, live at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

A rolling deploy took about six hours to work through the fleet. For those six hours half my workers ran the new parser and half ran the old one, both writing into the same table, and not one row recorded which of the two had produced it.

I found out because somebody asked why a single day of data had two clearly separated distributions sitting inside it. Not drift. Not a spike. Two modes, produced on the same date, by the same job, under the same name.

I run mobile proxy lines and production scrapers out of Singapore. The first question I ask now when I inherit a pipeline is whether anyone can pick a row and say which code wrote it. Almost nobody can, and the people who cannot are usually the ones telling me their pipeline is fine.

The field that changes meaning without changing shape

Here is the failure this prevents, and it is not a crash.

I had a boolean called in_stock, originally set by checking whether an add to cart button existed on the page. Somebody on my side rewrote that check to look for a small availability badge instead, because the button rendered even on sold out listings and the badge looked more precise.

It was more precise. It was also warehouse specific, so listings available from a second warehouse now came back false.

The column stayed boolean. The fill rate did not move. Nothing went null, no type changed, no request failed. The true rate dropped by about nine points over a week, which is well inside the range that catalogue seasonality produces on its own. Every monitoring check I had passed.

So the table now held one column carrying two different definitions, separated by a deploy that left no trace anywhere in the data. Anyone querying across that boundary was silently comparing two things.

This is a distinct problem from the target site changing on you, which is worth monitoring for on its own and is a separate discipline with separate tooling. It is also distinct from rerun safety. A properly idempotent write path stops a retry from duplicating rows, and it is worth building, but it will happily let two different code versions upsert into the same key without either of them leaving a signature. Idempotency protects the row count. It does nothing for the row’s meaning.

Three columns is most of the fix

Stamp every row with the version of the code that produced it and with when the page was fetched.

In practice that is a short git commit hash, seven characters, read from the repository at build time and baked into the image. Then two timestamps rather than one: fetched_at for when the response came back, parsed_at for when the extraction ran over it.

Those two are identical on the first pass and diverge the moment you reparse anything, which is exactly the situation you want them for. fetched_at is a fact about the world. parsed_at is a fact about your code. Collapsing them into a single “created_at” throws away the distinction at the moment it starts to matter.

If your extractor has fallbacks, record which branch fired. A primary selector and a rescue selector produce rows that look identical and deserve different levels of trust, and once you have that column you can also see the rescue path quietly climbing from two percent of pages to thirty, which is an early warning you would otherwise have to go looking for.

Two things I would insist on. Never let the version be a constant somebody edits by hand, because it will eventually be wrong and a wrong stamp is more dangerous than a missing one. And make the deploy identity granular enough to survive a rolling release, which was my six hour problem: the version was correct on every row, I simply had not been recording it yet.

The usual objection is storage. Three columns on a hundred million rows sounds heavy until you notice the row already contains a url. A repeated seven character hash compresses to near nothing in any column store, and against a row that already carries text fields it is not measurable.

Keep the response

This is the position I will defend: keeping the raw response is the highest value habit in scraping, and it is the first thing that gets cut when someone looks at the storage bill.

The reason is not sentimental. A scraper is not a query you can run again. The source is somebody else’s website, it keeps no history for you, and the page you fetched this morning may not exist by Thursday. Prices move, listings sell, sites redesign, whole catalogues get pulled. Every fetch is the only opportunity you will ever have at that page in that state.

Which means the parsed row is a lossy derivative of something that no longer exists. If your parser had a bug, the raw response is the only thing standing between “we can repair this” and “those numbers are permanently wrong and I have to say so”.

Store the headers with the body, not just the html. Content type, redirect chain, declared encoding. A good share of parsing bugs are encoding bugs, and the evidence for those lives entirely in headers you did not keep.

Where and how you physically store all of that is its own decision, with real tradeoffs around file counts and formats, and a separate question from this one. All that matters for provenance is that the raw capture and the parsed row exist as two artifacts, and that the row carries a pointer to its capture. A content hash or an object key. That pointer is the chain.

The honest version of the retention argument: raw html is bulky and mostly boilerplate, and cold storage with a lifecycle rule at thirty or ninety days is a fair compromise. I run exactly that on most targets.

Just be clear about what the rule buys. You can only reparse the window you kept, so the window has to outlast your detection time. Mine has been badly wrong at least twice. The worst case was a bug a client’s finance team caught during a quarterly review, roughly eleven weeks after it started, against a thirty day retention rule. Nine of those weeks were unrecoverable. I widened the rule that afternoon.

The schema is a contract

The third piece is the one people skip, because it is process rather than code.

Your output schema is a contract with whoever consumes it, and it needs a version separate from the code version. The code version changes on every deploy, including the dozens that change nothing about the output. The schema version changes only when the shape or the meaning of what you emit changes, which is rare and always significant to a downstream consumer.

An integer column, and a plain file in the repository saying what each number means with a date against it. That is the whole implementation.

The discipline that goes with it is the part worth actually holding: never redefine an existing column, add a new one. When the availability logic changed, the correct move was in_stock_badge alongside in_stock, not a quiet redefinition. Old rows keep the old field, new rows populate both for a while, and the boundary is visible to anyone querying instead of buried in a commit message they will never see.

It produces an ugly schema. I would take an ugly schema over a column that means two things depending on the date.

What this actually buys

Three things, and they are the reason any of the above is worth the trouble.

You can reparse history. Fix the parser, run it back over the stored captures, write corrected rows under the new version, keep the old ones. Then diff the two sets, which turns “something was wrong for a while” into “forty one thousand rows were affected, the median error was eleven percent, here they are”. That is a survivable conversation. The other one is not.

You can attribute an anomaly to a deploy instead of arguing about it. Group your metric by code version rather than by date. If the step change sits exactly on a version boundary it is yours. If it sits somewhere else entirely, the site changed or the market did, and you have ruled out the expensive explanation in about ten minutes.

And you can answer the question months later. Which code, which fetch, which page, on what day. That question always arrives from someone senior, about a figure that has already left the building.

The one I got wrong

For the first year I stamped the code version and only kept a single timestamp, updated on write.

Then I reparsed a batch to fix a field, and the reparse overwrote that timestamp on every row it touched. I had turned “this page was fetched on the fourth” into “this row was written on the nineteenth” across several hundred thousand rows, and there was no way back, because I had never stored the original separately. The captures still existed and the fetch times lived in their metadata, so I recovered most of it over a slow weekend. On the targets where the captures had already aged out, I did not.

Provenance has a limit too, and it is worth being blunt about. Knowing which code produced a row tells you nothing about whether that code was right. You can have immaculate provenance on a completely wrong number. What you have bought is the ability to discover it later and repair it, which is not the same as protection.

None of it changes what you should be collecting either. Public pages, the robots file honoured, a crawl rate that does not hurt the target, and an official api or a bulk feed used whenever one exists. Good provenance on data you should not have taken is just a tidier record of the decision.

The pipelines I have seen survive a change of owner are the ones where a new person can take a row at random and walk it back to the response it came from. Everything else is a box producing numbers that people have agreed to trust. The rest of what I have written on pipeline design, storage and the proxy infrastructure underneath it is over here.

Get new guides and videos first — join the Telegram channel.

I once bought a fresh batch of proxies on a Friday, pointed my scrapers at them, and for about a day it was beautiful. The jobs flew, the blocks were near zero, everything came back clean. By the middle of the next week, half of those addresses were quietly useless. Some were timing out, some were handing back CAPTCHA pages, and my overall block rate had crept up so slowly that I almost missed it.

Nothing had crashed. The pool had simply aged, and nobody was watching it age. That is the whole reason this piece exists: a proxy is a living thing you manage, not a static line in a config file.

A proxy pool is a fleet, not a receipt

The mistake I see most often is treating proxies as a one time purchase. You buy a list, paste it in, and assume it keeps working forever. It does not. Addresses get flagged, connections go stale, providers recycle their ranges, and the sites you visit keep learning.

The better mental model is a fleet. Think of the way a taxi company thinks about its cars: some are on the road earning right now, some are in the garage resting, and a few are broken and waiting to be retired. Your job is not to buy the fleet once. It is to always know which addresses are working, which are struggling, and which are done, and to keep the working set topped up as the others fall away.

What healthy means for one proxy

Health is not one thing. For a single proxy there are three separate questions, and they fail independently.

• Can you reach it? Does it connect, or does it refuse and time out?
• Is it fast enough? An address that answers but takes fifteen seconds is dragging your whole run down.
• Is it still trusted by your target? This is the one people forget. An address can be perfectly reachable and perfectly fast while being completely burned in the eyes of the site you care about, handing back a block page on every request.

Health is all three at once. A pool full of fast, reachable, flagged addresses is a pool full of dead weight that looks alive.

Passive and active health checks

There are two ways to learn the health of your pool, and you want both.

Passive checks watch what your real traffic is already telling you. Every request you send through a proxy is a free health check, because it came back with good data, or it came back slow, or it came back as a block. You are collecting that signal anyway, so record it against the proxy that carried it.

Active checks probe on a schedule. You send a cheap test request through each proxy every so often, to a target you control or a neutral endpoint, just to ask whether it is still alive before a real job leans on it. Passive tells you how the pool behaved; active tells you the pool is ready before you spend a run finding out the hard way.

The signals a proxy is going bad

The failure signals are not all the same, and the difference tells you what is actually wrong.

• Connection refused or repeated timeouts mean the proxy itself is down or unreachable. That is a plumbing problem.
• A slow crawl upward in latency means it is overloaded or dying.
• Rising CAPTCHA pages and soft blocks from a proxy that still connects mean that address specifically has been flagged by your target.

That last one is the dangerous signal, because the proxy looks fine at the network level. Only the body of the response gives it away, which is exactly why you inspect responses and do not trust a 200 alone.

Health is per target, not global

Here is the idea that changes how you build the whole thing. Health is not global, it is per target.

An address that is stone cold burned on one big site can be completely clean on another that has never seen it. So a proxy is not simply good or bad, it is good for this site and bad for that one. If you track health as a single yes or no per address, you will throw away perfectly useful proxies just because one aggressive target flagged them. Track the pairing instead, this proxy against that site, and you get far more life out of the pool you already paid for.

Quarantine, do not delete

When a proxy starts failing, the instinct is to delete it, and that is usually too harsh. Addresses recover. A site that flagged an IP an hour ago may forget about it in a day, especially on mobile and residential ranges where the same address gets handed to real people all the time.

So instead of deleting, quarantine. Pull the suspect address out of active rotation, let it rest for a while, then send a quiet probe later to see if it has come back to life. You cool it down rather than throwing it away, and a large share of the pool you would have binned comes back on its own.

Warm new proxies in

The opposite move matters just as much. Do not slam a fresh proxy with your full load on the first request. A brand new source suddenly firing a hundred requests a minute at a site is its own kind of red flag.

Bring new addresses in gently. Give them a smaller share of the traffic at first, watch how the target treats them, and only promote them to full duty once they have behaved. A slow warm up costs you a little speed on day one and saves you a burned address on day two.

Measure the pool, not just the proxy

Zoom out from the single address to the pool as a whole. You need a few numbers that tell you its overall health at a glance:

• How many addresses are usable right now, versus quarantined, versus dead.
• The pool wide success rate and block rate over the last hour.
• The latency spread, not just the average but the slow tail.

Watch those together and you stop reacting to individual proxies and start managing a fleet. You can see the pool getting tired days before it would have taken a job down with it.

Health decides what rotation is allowed to use

Health and rotation are two different jobs that people blur together. Rotation is how you spread requests across addresses so no single one carries a burst. Health is deciding which addresses are even allowed into that rotation in the first place.

The cleanest way to think about it: your health layer maintains a pool of currently trusted addresses, and your rotation layer only ever picks from that trusted set. A burned proxy should never be handed to the rotator. It should already have been pulled out.

Datacenter and mobile fail in different shapes

The kind of proxy you run changes how this plays out. Datacenter addresses tend to fail in blocks: a whole range gets recognized and flagged together, so when one goes, the neighbors often go with it.

Mobile and residential addresses behave very differently, because they are shared with real users and rotate naturally. An address that looks burned this minute can be genuinely clean an hour later through no effort of yours. That natural recovery is a big part of why I lean on mobile pools for the heavy, long running jobs: the pool heals itself between passes instead of just draining away.

Retire and replenish

No matter how well you tend it, a pool leaks. Some fraction of your addresses will burn out for good every week, and if you are not topping the pool up, you are slowly starving your own scrapers.

Build the churn into the plan. Budget for replacement addresses the way you would budget for any consumable, keep an eye on how many usable proxies you have left, and refill before you hit the floor, not after your jobs have already started failing. A pool is a stock you keep replenishing, not a tank you fill once and forget.

Automate the whole loop

A human cannot watch hundreds of addresses by hand, so the whole loop should run on its own:

• Every request feeds its outcome back into a health record for that proxy and target pairing.
• Addresses that cross a failure threshold get quarantined automatically.
• Quarantined ones get retested on a timer and either promoted back or retired for good.
• The pool reports its own headline numbers, so you only get pulled in when the whole fleet is trending down, not for every address that has one bad minute.

The honest limits

A healthy pool is about reliability, not permission. Keeping your addresses clean and rotating makes your collection steadier; it does not entitle you to take anything you could not take before. Public data stays the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how big or how healthy your pool is.

I run this infrastructure in production every day, and a well managed pool is the difference between a job that quietly holds together for months and one that rots in a week while every dashboard stays green. The full written picks I actually use, the exact health checks I log, the quarantine timers, and the mobile pool I run on, live at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

The first scraper I was ever proud of ran all night to do a job that should have taken twenty minutes. It walked a catalog one page at a time: ask for a page, wait for the answer, save it, ask for the next one. Patient, polite, and painfully slow. When I finally looked at where the time actually went, the machine was doing almost nothing. It was sitting idle, waiting for a server on the other side of the world to answer, thousands of times in a row.

That is the whole story of scraper performance, and it is why concurrency is the single biggest lever you have. This is how you fetch many things at once so your scraper stops spending its life waiting.

Why a scraper is slow

A slow scraper is almost never a slow computer. The bottleneck in scraping is hardly ever your processor working too hard. It is your program standing still, waiting on the network.

You send a request, and then there is a long pause while it crosses the network, the server thinks, and the answer travels back. During that pause your code is just holding its breath. If you do one request at a time, you sit through every one of those pauses back to back, and they add up to the whole night. Concurrency is the trick of overlapping those waits, so that while one request is out waiting, twenty others are out waiting at the same time.

Concurrency is not pulling harder

Draw a clean line here, because this is where people get into trouble. Going concurrent is not the same as pulling harder. The goal is not to hit the site with more force, it is to stop wasting the idle time you already had.

That distinction matters. The moment concurrency turns into a flood aimed at one site, you are back in the world of rate limits, getting throttled and blocked. Done right, concurrency makes you faster without making you louder to any single target.

The three models

There are really only three ways to run many requests at once, and each one fits a different bottleneck.

Model	What it is	Best for
Async event loop	One thread juggling thousands of waiting requests	Pure fetching, the lightest option
Threads	A pool of workers, each doing a blocking request	The simple case, moderate concurrency
Processes	Several copies across CPU cores	When parsing, not fetching, is the cost

The async event loop is built for exactly this job. Picture a single thread that never sits still. It fires off a request and, instead of waiting for the answer, immediately turns to the next request and fires that one too, keeping hundreds or thousands of them in the air. When an answer comes back, the loop picks it up and deals with it. Because the work of a scraper is waiting rather than computing, one core running one loop can happily manage thousands of open requests. In Python this is the world of asyncio and the async HTTP libraries.

Threads are the simpler picture, and for a lot of scrapers they are more than enough. You keep a pool of, say, twenty workers, and each one does the plain blocking thing. You may have heard that Python threads cannot really run at the same time because of the global interpreter lock, and for heavy computation that is true. But it does not hurt you here, because your threads are waiting on the network, not computing, and a thread that is waiting steps aside and lets the others run.

Processes are the heavy option, and you reach for them only when your bottleneck moves to parsing: huge pages, heavy pattern matching, tangled documents that take real processor time to pick apart. For most scrapers the parsing is cheap and the waiting is the whole cost, so you rarely need processes just to fetch.

Bounded concurrency is the number that matters

The beginner move, the day you discover you can fetch many at once, is to fire all ten thousand URLs at the same instant. Do not. That is a self inflicted disaster. You open ten thousand sockets, exhaust your own memory and file handles, your machine falls over, and the site sees a wall of traffic arrive in one heartbeat and blocks you on the spot.

What you want is bounded concurrency: a fixed number of requests allowed in flight at once, say ten or twenty, with the rest waiting their turn. A semaphore, or a worker pool of a fixed size, is how you enforce it. This one number is the difference between a fast scraper and a self inflicted outage.

Reuse your connections

Before you even touch concurrency, there is a free speedup most people leave on the table: reusing your connections. Every time you open a brand new request from scratch, you pay for a fresh TCP and TLS handshake, a little round trip of setup before any real work happens.

If you reuse one session and let it keep the connection alive between requests, you skip that setup on every call after the first. It makes you faster, and it is genuinely lighter on the site, fewer connections to accept. One session, reused, is the cheapest win there is.

Cap concurrency per host

Here is the subtlety that separates a scraper that survives from one that gets banned in an hour. Your concurrency should be counted per site, not as one global number. Twenty requests in flight spread across twenty different sites is gentle, two at each. Twenty requests in flight all aimed at one site is a hammer.

So cap how many you run against any single host, keep that per site number low, and let your overall speed come from working many hosts at once. That is how you stay fast in total while staying polite to each individual target.

Where proxies come in

Concurrency and your address are linked. If you run twenty or fifty requests at once all leaving from a single IP, you have concentrated a burst of traffic on one address, and that is exactly the pattern that gets an address flagged and shown the door.

The fix is to spread your concurrent requests across a healthy pool of addresses, so no single one carries the whole burst. Concurrency multiplies your footprint, and a good proxy pool is what distributes it. This is where a clean mobile proxy setup earns its keep on heavy concurrent runs.

Timeouts, queues, and retries

Once you have many requests in flight, three details keep the whole thing from quietly falling apart.

Timeouts stop being optional. If one host goes slow and never answers, that request holds its slot forever, and enough of them will occupy every slot in your fixed pool. Your scraper looks like it is running, but nothing is moving. Give every request a hard timeout so a hung connection gives up and frees its slot.

A bounded queue keeps memory sane. You have one side producing URLs and another side of workers consuming them. If the producer races ahead, it loads a million URLs into memory before a single one is done. A bounded queue applies back pressure, so a fast producer cannot drown a careful fetcher.

Retries have to share the budget. A failed request should try again after a short backoff, but a retry is still a request and still costs an in flight slot. Count retries against the same concurrency budget, or a bad patch where everything is failing will silently double your load at the worst possible moment.

Measure throughput, not requests per second

Raw requests per second is the wrong number to chase. The number that matters is useful throughput: good rows or good pages collected per minute, measured right next to your block rate.

It is easy to turn concurrency up, watch requests per second climb, and feel great, while your block rate climbs right alongside it and half those requests come back as block pages. That is not progress, it is going faster into a wall. So tune it like an experiment: start low, raise the limit while real throughput climbs and blocks stay flat, and settle in just under the point where throughput plateaus or blocks tick up. Every site has its own sweet spot, and the only honest way to find it is to measure your way there.

The honest limits

Faster is not the same as allowed more. Concurrency is about not wasting the idle time you already had, never about overwhelming a site into giving up more than it wants to. Public data stays the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how many hosts you work at once.

I run this in production every day, real scrapers pulling real volume on a schedule, and getting the concurrency model right is the difference between a job that finishes before breakfast and one that runs all night and still gets blocked. The full written picks I actually use, the exact limits, the async fetch loop, and the per host caps, live at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

Four percent of the records disagreed.

I had an api and a scraper pointed at the same site, writing into two tables, and after a week I compared them row by row. Ninety six percent matched exactly. Every mismatched record had been edited in the previous twenty four hours: the api was reading a replica with a cache in front of it. That appears nowhere in its documentation, and it would have appeared nowhere in my head if I had migrated the way everyone migrates.

Read the announcement, feel relieved, spend a fortnight on the client, delete the scraper.

I run mobile proxy lines and production scrapers out of Singapore. I have done this migration several times and got it wrong at least once, in a way that cost three days I had already told myself I owned.

Which kind of api are we talking about

Three things get filed under one heading and only one of them is this.

There is the undocumented json endpoint you find in the network tab, the one the site’s front end calls to fill the page. I use them constantly and have written about finding those. That endpoint has one consumer, the site’s own javascript, so nobody there carries an obligation to keep its shape still. I have watched one change twice in a month while the visible layout stayed identical.

There is a third party scraping vendor, where you pay somebody else to do the fetching and the target has no idea that supplier exists. That is a buy versus build question about your own infrastructure, and it leaves your standing with the target where it was.

Then there is this case. The site chose to publish. A documentation page, a version in the path, a changelog, and somewhere a person who gets a ticket when it breaks. That is a different animal, because it is the only one where the target learns your name.

What genuinely gets better

The upside is real and larger than api cynics admit.

Types. A number arrives as a number, a date as a date, an absent value as null instead of an empty element you have to interpret. Your parser stops being the most brittle component you own, and the parser is the thing that fails on somebody else’s design calendar, usually in silence.

A contract. Versioning in the path, a changelog, a deprecation window, occasionally an email before a field moves. Compare that against finding out because a column went blank and a customer got there first.

And the traffic stops being a fight. No mobile addresses on that target, no headless browser, no guessing at what counts as a polite rate. On one job the monthly run cost fell to almost nothing, because the whole of it had been proxy bandwidth for a two megabyte page.

The api is a subset of the page

Here is what almost nobody checks before committing.

The page and the api are two different field selections made for two different audiences. The page was built for a shopper. The api was scoped in a meeting for a partner integration, probably a while ago.

Derived fields fall out first. Anything the front end computes from several other values. Anything a merchandising team added as a badge.

Availability is the standard example. The page says in stock at four locations and names them. The api returns a boolean.

Review counts, position within a category listing, the recently sold counter. Those live in the presentation layer, and the presentation layer is precisely what you have agreed to stop reading.

The sting is that derived fields are frequently the reason your product exists, because nobody else could get them cheaply. The identifiers everybody already has are what the api hands you first.

The quota is a policy

A scraper’s ceiling is physical. Concurrency, address pool, and whatever load you are willing to put on somebody’s server.

An api’s ceiling is a number in a document, and documents get edited. I have watched a free tier drop from sixty calls a minute to twenty, announced in a changelog entry and nowhere else.

Two pieces of arithmetic before you migrate. Divide your record count by the published quota and read the answer in hours, because that is your run regardless of hardware. Then find out what counts as one call, because a list endpoint returning fifty records is a different budget from a detail endpoint you hit once per record.

What the key costs you

A key means an account. An email address, usually a company name, plus a box you tick with a legal agreement sitting behind it.

Before, you were traffic. Anonymous, hard to separate from a browser, and a site that wanted you gone had to find you first. After, you are a row in their database with a name on it and a switch beside it. Revoking a key takes one click from somebody who never has to justify it. Blocking a scraper is a project they can lose.

The agreement contains things nobody reads. Retention limits. Whether you may display it to your own customers. Whether you may use it in anything that competes with them. I have read one that capped storage at twenty four hours, which rules out every historical product you might have planned, and that cap lived in the terms rather than the docs.

Read the termination clause specifically. Whether it requires cause, and what notice you get. That paragraph describes the relationship more accurately than the entire feature list above it.

Pricing arrives after the scraper is gone

The sequencing is deliberate. A new api is usually free while the publisher wants adoption, because they need integrations to point at. Metering shows up later, once there is a population of users who have already deleted their alternative.

That is ordinary product behaviour rather than malice. It is still the risk, because the cheapest moment to own a working scraper is before you learn what the api costs.

An api is a relationship and a scraper is not

Here is the claim, and I do get argued with about it.

The relationship gives you notice, a changelog, a version, somebody to email when a field moves. It also gives them a switch, an agreement you signed, and a price they get to set.

The scraper gives you no notice, no support, and maintenance with no end date. It also gives them nobody to switch off and no invoice to raise.

Neither wins in the abstract. The only question is whether you can live with someone else holding a switch over the thing you sell. If the data is an input to a product, usually yes. If the data is the product, that is a genuine dependency and it should be priced as one rather than treated as good news.

Map your columns before you read the docs

The audit takes about half an hour.

Write down your actual field list, from the output table rather than from memory. For every column, note where it comes from today: a raw value off the page, or something you compute from two or three scraped values.

Then open the field reference and map them. Each column gets a named api field or it gets a question mark.

The question marks are the whole exercise. One is a conversation with your product owner. Three means you are keeping a scraper whatever happens, and the migration has become a hybrid rather than a replacement.

Do this before the meeting where somebody proposes just using the api. It is the only artefact in that room with facts in it.

Run both, then diff the values

A mapping exercise compares names. A diff compares values, and values are where the surprises live.

Run both collectors against the same records on the same schedule, each writing into its own table, for a few weeks. Yes, you are paying twice. The bill is a few weeks of proxy traffic you were already spending, and it finds what documentation cannot describe.

What turns up: an api price that is the base price while the page price includes a promotion; a numeric field present and silently rounded; the replica lag I opened with.

Set your threshold before you look. Mine is that anything above one percent disagreement halts the migration until I understand the cause.

The join you are not going to get

The api’s identifier for a record is often not the identifier your scraper keys on, and it carries no obligation to expose yours. So your history does not join. Two years of stored rows, and the new ones will not line up. Check that field on day one, because a missing shared key means a matching pass, and matching is its own project.

Keep the old collector running

After you migrate, keep the scraper and keep it running rather than parked. A weekly job over a hundred records is enough. It stays green, and on the day the key gets pulled or the quota moves you have something that executes instead of a repository you have to remember the shape of.

Where I got this wrong

That last paragraph is advice I gave myself and then ignored. I migrated, kept the code, and never ran it. Eight months later I needed it back and the site had been redesigned twice. What I actually had was a folder. Rebuilding took three days, which is survivable, and which I had already counted as bought.

The honest limit: I have never had a key revoked on me. The failures I have been hit by are the subset problem and the quota problem. When I tell you the switch matters, that is reasoning rather than a scar.

There are cases where an api is straightforwardly right and none of this applies. A regulator publishing a bulk api is publishing it so you stop crawling them. No commercial lever, no competitive clause, and arguing about relationship risk there is just being difficult.

More on scraping infrastructure, pipeline design and the proxy layer underneath it is at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The fastest scraper is almost never the one that wins. The one that wins is the one still running next month, and the difference between the two is almost entirely about how it handles rate. A scraper that pulls as hard as it can gets noticed, throttled, and blocked, usually within hours. A scraper that moves at a pace the site can absorb keeps going and going. This is the playbook for that second kind: how to handle rate limits, retries, and backoff so your collection stays alive.

I run production scrapers for a living, the kind that have to keep working for months without becoming somebody else’s incident, so this is the part I care about most. It isn’t glamorous. Nobody makes a flashy demo about waiting politely between requests. But it’s the actual skill that separates a toy that runs once from infrastructure that runs for a year. And almost all of it comes down to one idea: take only what the site can comfortably give, and back off the moment it signals otherwise.

What a rate limit really is

Start from the site’s side, because that’s who sets the rate. A rate limit is simply the site saying you may make this many requests in this much time, and no more. It exists to protect the site from being overwhelmed, whether by an attack, a bug, or a scraper pulling too hard. A rate limit isn’t an insult or a challenge to beat. It’s the site telling you exactly how much load it’s willing to carry from you. The polite move, and the durable one, is to listen to that number and stay under it.

The status code that says slow down

Sites have a standard way of telling you that you’ve gone too fast. When you cross a limit, a well behaved server responds with a status code, most commonly 429, which means too many requests. Often it comes with a header that literally tells you how long to wait before trying again. This is the site handing you the answer. The worst thing a scraper can do is ignore that signal and keep hammering, because that’s exactly the behavior that turns a temporary slowdown into a permanent block. Read the code, honor the wait.

Backoff is the core move

The central technique is backoff. When a request fails or gets throttled, you don’t retry instantly. You wait, and you wait longer each time it keeps failing. The first retry might wait a second, the next a few seconds, the next longer still. This gives the site room to recover and signals that you’re a well behaved client, not a battering ram. A naive retry loop that fires again the instant it fails is the single most common way people turn a recoverable hiccup into a hard ban. Backoff is what makes retries safe instead of dangerous.

Why the waits grow

The reason the wait grows each time is worth understanding. A single failure might be random noise, so a short wait and a retry usually clears it. But repeated failures mean something is actually wrong: the site is struggling or actively throttling you, and the right response is to pull back harder, not keep pushing at the same rate. Growing the delay each time means a brief blip costs you almost nothing, while a real problem makes you gracefully retreat instead of piling on. The pattern matches your pressure to the site’s actual state.

Add jitter or you stampede

Here’s a subtle one people miss. If you run many workers and they all fail at the same moment and all back off by the same amount, they’ll all retry at the same instant, producing a synchronized wave that hits the site like a hammer. The fix is jitter: adding a small random amount to each wait so the retries spread out instead of bunching up. It’s a tiny change with a big effect. Without jitter, your polite backoff can accidentally become a coordinated stampede, and the site feels a spike exactly when you meant to ease off.

A backoff in a few lines

The shape of it is small. You catch the failure, compute a growing delay with a little randomness, and try again up to a limit. That’s exponential backoff with jitter, the whole idea in a handful of lines, and it will carry you a long way.

The token bucket idea

Beyond reacting to failures, you want to control your rate before you ever trip a limit. A clean way to think about this is a token bucket. You get a steady supply of tokens, one per allowed request, and each request spends one. When the bucket is empty, you wait for it to refill. This smooths your traffic into an even, predictable stream instead of bursts, and it lets you set your own ceiling comfortably below whatever the site allows. Shaping your rate proactively is far better than sprinting until the site slams the door and then reacting.

Concurrency is a dial, not a maximum

People treat concurrency as a number to maximize, and that’s the mistake. Running more requests at once feels like progress, but past a point it just increases the load the site feels from you and the chance you get flagged. Treat concurrency as a dial you set deliberately, low enough that the site never strains, not as high as your hardware could technically push. I routinely run well below what my machines could handle, on purpose, because the bottleneck I care about is the site’s tolerance, not my throughput. Slower and alive beats fast and banned, every time.

Respect the crawl delay

Many sites publish their preferences in a robots file, and that file often includes a crawl delay, a request to wait a certain amount of time between hits. Honor it. Treating that number as a real instruction rather than a suggestion is both the courteous thing and the smart thing, because a site that publishes a crawl delay is telling you the exact pace at which it will tolerate you. Staying at or under that pace keeps you invisible in the best way: a small, steady stream of requests the site has explicitly said it can carry. Ignoring it is asking for trouble you were warned about.

Cache so you never ask twice

The cheapest request is the one you never make. A huge amount of scraper load is wasteful, fetching the same page again because the pipeline wasn’t tracking what it already had. So cache aggressively. Store what you pull, and before you request anything, check whether you already have a fresh copy. This cuts your load on the site dramatically, speeds up your own job, and shrinks the surface where anything can go wrong. Every page you serve from your own cache is a page you didn’t have to ask the site for, and politeness and efficiency point the same direction here.

Spread the load across time

If you have a big job, don’t try to finish it in one aggressive burst. Spread it across time. A hundred thousand pages pulled gently over a day is nearly invisible, while the same hundred thousand pulled as fast as possible in an hour is a spike that any monitoring will catch. The total work is identical, but the shape the site feels is completely different. Patience is a feature here. A job that isn’t urgent should be spread wide and thin, because the flatter your traffic, the less reason the site ever has to look at you twice.

The circuit breaker

There’s one more pattern worth having, borrowed from resilient systems, called a circuit breaker. The idea is that when failures pile up past a threshold, you stop entirely for a while instead of continuing to retry. If a site is returning errors on nearly everything, that’s a clear sign to pause the whole job, not to keep probing at it. The breaker trips, you wait a good while, and then you test cautiously before resuming. This stops a struggling site from turning into a hard block, and it keeps you from wasting effort against a wall that isn’t going to move right now.

Read the response, not just pass or fail

Politeness also means paying attention to what the site is telling you beyond the raw status. Rising latency, subtle warnings, a challenge appearing where there wasn’t one before, these are all signals that you’re pushing harder than the site wants. A good scraper watches those and eases off before it gets to an outright block. By the time you’re seeing hard failures, you’ve already been rude for a while. The skill is reading the soft signals early and slowing down on your own, so the site never has to make you.

Retry only what is safe to repeat

One caution about retries: they must be safe to repeat. If a request has a side effect, submitting a form or changing something, retrying it can do that twice, and you end up with duplicate or corrupt results. Keep retries to operations that only read, and design your storage so writing the same record twice does no harm. Safe retries are what let backoff work without fear, because you can try again knowing a repeat costs you nothing.

Why polite scrapers last

Put it all together and the philosophy is simple. The durable scraper is the considerate one. It stays under the limit, backs off when told, spreads its load, caches what it can, and reads the site’s signals instead of bulldozing through them. Done this way, most of the blocking machinery never even has a reason to fire, because you aren’t producing the load that triggers it. You’re just a small, steady, well behaved stream of requests, and a site has very little reason to fight a client that isn’t causing it any trouble in the first place.

The honest limits

Let me be straight about the boundaries. Handling rate well keeps a scraper healthy and welcome, but it doesn’t change what you’re allowed to collect. Public data, a robots file respected, an official API or bulk feed preferred where one exists. Politeness isn’t a trick to sneak past anything, it’s the absence of the load that gets you noticed, and it works precisely because it’s genuine restraint rather than a disguise. And no amount of careful pacing makes it right to take data that was never yours, or to ignore what a site has clearly asked you not to do.

To recap: a rate limit is the site telling you how much load it will carry, so listen to it. Honor the slow down signals, retry with backoff that grows each time, add jitter so your workers don’t stampede, and shape your rate proactively with something like a token bucket. Treat concurrency as a deliberate dial, respect the crawl delay, cache so you never ask twice, spread big jobs across time, and trip a circuit breaker when a site is clearly struggling. Be the considerate client, and you get to keep running.

For more breakdowns on scraping infrastructure, proxy setups, and pipeline design straight from production use, visit the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.

Your browser tells a story about itself to every site you visit, and most of that story gets told without your permission and without your knowledge. It isn’t your login, and it isn’t a cookie. It’s the shape of your device itself, assembled from dozens of small signals into something that turns out to be surprisingly unique. Sites use that shape to tell a returning visitor from a stranger, and to tell a real person from an automated one.

I run proxy infrastructure and production scrapers, so I spend a lot of my time on the defending side of this, understanding how detection systems are built and how they operate. I want to be clear about the frame up front: this is an explanation of how the mechanism works, not a guide to defeating it. Faking a device isn’t the durable way to collect data, and it isn’t a game worth playing. Understanding the wall is useful. Pretending it isn’t there is not.

What a fingerprint even is

A browser fingerprint is a profile a site builds by asking your browser a series of harmless-looking questions and combining the answers: what’s your screen size, what fonts do you have, how does your graphics chip draw a shape, what audio hardware do you report. No single answer identifies you. But stack forty of them together and the combination is often unique enough to pick you out of millions, and to recognize you again later with no cookie at all. That’s both the power and the problem of it.

The user agent is just a claim

The most familiar signal is the user agent, the string that names your browser and operating system. On its own it means little, because it’s just text your browser volunteers, and anything can type any string into it. What makes it useful to a site isn’t the claim itself, it’s whether the rest of the browser backs the claim up. A client that says it’s one thing while every other signal says something else has produced a contradiction, and contradictions are exactly what detection systems are built to notice.

What the screen and system reveal

Your browser freely reports a pile of system details: screen resolution and color depth, number of processor cores, amount of memory, operating system and version. Each one is coarse on its own, plenty of people share a common screen size. But combined with everything else, these details narrow the field fast, and they carry consistency requirements. A device claiming to be a phone but reporting a giant desktop screen and dozens of cores is telling two stories at once, and that mismatch is more revealing than any single value.

The canvas signal

Here’s one of the cleverer techniques. A site can ask your browser to draw text and shapes onto a hidden canvas, then read the exact pixels back. The result looks identical to a human eye, but at the pixel level it varies slightly from device to device, because the drawing depends on your graphics hardware, drivers, and system fonts. That tiny variation is stable for your machine and different across machines, so it works like a signature. The site isn’t showing you anything. It’s quietly measuring how your specific hardware renders a picture nobody ever sees.

WebGL and the graphics chip

Closely related is WebGL, which lets a site render three-dimensional graphics and, in doing so, learn about your GPU. It can read the make and model of the chip and observe exactly how it draws a complex scene. Because graphics hardware varies so much across devices, this is a strong contributor to the overall fingerprint. Like the canvas signal, it’s passive from your side. You see a normal page, while underneath it your GPU has quietly described itself in enough detail to help separate your device from the next one.

The fonts you have installed

The set of fonts on your system is another surprisingly telling signal. The exact list depends on your operating system, the software you’ve installed, and the languages you use, so it varies more between people than you’d guess. A site can probe which fonts are present and build a picture from the answer. On its own it’s just one more coarse signal, but it stacks with the others, and an automated environment that ships a bare, identical font list on every instance stands out precisely because real people’s font sets are messy and individual.

Audio, timezone, and language

The signals keep going. The way your device processes a silent audio sample varies with your hardware and software, giving another faint but stable marker. Your timezone and language settings add more, and they carry consistency checks of their own. A visitor whose address places them in one part of the world while their timezone and language claim another has produced yet another small contradiction. None of these is decisive alone. The method is always the same: gather many weak signals and let the combination do the identifying.

The network layer speaks too

Fingerprinting isn’t only about the browser. The network layer talks before the page even loads. The way your client negotiates its encrypted connection, the specific options it offers and the order it lists them, forms a pattern that real browsers produce in well-known ways. A lot of automated tools produce a pattern here that no real browser would send, because the underlying library was never trying to imitate one. So a client can claim to be a current browser in its headers while its connection setup quietly says otherwise, which is one more consistency check a site can run for free.

Entropy, or how the signals combine

The word for what makes this work is entropy, which just means how much a signal narrows the field. A signal everyone shares carries little. A signal that varies a lot carries more. Detection combines many signals so that even though each is weak, the total is strong. You can’t think about any one value in isolation. The system isn’t asking whether your screen size is suspicious. It’s asking whether the whole bundle, taken together, looks like a real person’s device or like something assembled to look like one.

The tells of an automated environment

So what gives an automated setup away? Real devices vary in messy, natural ways across all of these signals, while a fleet of identical automated environments tends to produce the same fingerprint again and again, or one carrying small tells a normal machine would never have. A browser driven by automation can expose properties that a human’s browser doesn’t. The detection system isn’t hunting for one magic flag. It’s looking at the whole shape and asking whether it resembles the natural variety of real people or the suspicious sameness of a cloned machine.

The long tail of signals

And the list keeps growing at the edges. Some environments expose battery status, motion sensors on a phone, the exact way a page scrolls or a pointer moves, even the subtle timing of small operations. Each one is another faint signal added to the pile. You don’t need to memorize the full catalog. The takeaway is that the surface is wide and always expanding, so the number of places a device can quietly describe itself is far larger than most people ever imagine.

Consistency is the real test

Step back and the pattern is clear. Almost every check here is really a consistency check. Does the user agent agree with the rendered fingerprint? Does the timezone agree with the network location? Does the claimed device agree with the reported hardware? Detection is less about catching one forbidden value and more about catching two signals that disagree. A real device is naturally consistent because all its signals come from one real place. An assembled identity has to keep every one of those signals in agreement, and that’s genuinely hard to do.

This tracks real people too

It’s worth naming the other side of this, because fingerprinting isn’t only aimed at bots. The same techniques identify and follow real human visitors across the web with no cookie and no consent, which is a genuine privacy concern. A person clearing their cookies can still be recognized by their fingerprint. This technology sits in an uncomfortable place: useful for telling automation from people, and at the same time a quiet tool for tracking those very people. Understanding how it works is part of understanding the privacy tradeoffs of the modern web, not just the scraping ones.

Why it’s an arms race

This whole area never sits still. Detection methods get sharper, browsers add protections that blur some signals, and the measurements shift as hardware and software change. A signal that’s strong today can weaken tomorrow, and a new one can appear. That’s why I’m wary of anyone who claims a permanent answer to any of it. The ground moves constantly under both the sites doing the measuring and the tools being measured. It’s less a solved problem than an ongoing back and forth with no final state.

What this means for honest collection

So where does a compliant operator stand in all this? Not in the business of faking a device to slip through, because that’s a race you don’t win and it’s the wrong side of the line anyway. The honest position is to understand how the wall works, and then to not need to beat it. Collect public data at a polite rate. Identify yourself honestly where a site expects it. Prefer an official API or data feed, because that’s the front door the site actually built, and it doesn’t care about your fingerprint at all. Understanding detection makes you a better engineer. Trying to defeat it makes you someone else’s incident.

The honest limits

Let me be straight about the boundaries. Nothing here makes anything undetectable, and I wouldn’t trust anyone who says otherwise. Detection keeps improving, and the point of understanding it isn’t to evade it but to know why the compliant path is the durable one. The moment collection depends on faking an identity, or reaches for private data, or ignores what a site clearly asked, it’s left the honest lane entirely, whatever the tooling looks like. The version of this work that lasts is the boring one: public data, honest identification, a gentle pace, and the rules respected.

To recap: a browser fingerprint is dozens of weak signals, the user agent, the screen, the canvas, the graphics chip, the fonts, the audio, the timezone, and the network handshake, combined into something unique enough to identify a device. Detection mostly hunts for contradictions between those signals, since real devices are naturally consistent and assembled ones struggle to be. The same technology quietly tracks real people, and it never stops evolving. The honest response is to understand it and to collect in a way that never depends on defeating it.

For more breakdowns like this on scraping infrastructure, proxy setups, and how detection systems actually work, visit the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.

I have inherited four scrapers written by somebody else. The code was readable inside an afternoon every time. Rebuilding everything around the code took between three days and a fortnight.

One of them had a fixed 2.3 second delay in the fetch loop with no comment next to it. I deleted it, because it looked like debugging somebody forgot to remove. Two days later I understood exactly why it was there.

That is the subject in one story. The repository is the easy half of a handover, and it is the only half most people transfer.

I run mobile proxy lines and production scrapers out of Singapore, so I have been on both ends of this. The jobs that turned sour were almost never badly built. They were badly handed over.

Working is a state it passes through

A scraper is a dependency on a website somebody else owns. No contract exists between you and them, they do not know your project exists, and they can change a template on a Wednesday afternoon for reasons entirely unrelated to you.

So “working” is a condition your collector moves through, between changes at the far end. It does not settle there.

Your client thinks otherwise, and I would not blame them for it. They are used to buying software the way they bought their invoicing tool. You install it, it runs, and a thing that stops running is a defect somebody owes them a fix for.

Leave that assumption in place at the moment of transfer and every future site change lands on you by default. Not through anybody’s bad faith. Because nobody said otherwise while the atmosphere was still friendly.

The paragraph that goes at the top

Before the install steps, before anything technical, one paragraph:

This collector works against the target site as it exists on the date of transfer. The site will change. When it changes, the collector will stop returning correct data. That is expected behaviour and not a fault in the delivery.

I bold it, and I read it out on the handover call, because clients skim documents and remember awkward sentences said to their face.

It reads badly when you are trying to look competent. It reads a great deal better than the email four months later that starts “we paid you to build this”.

Four things that never make it into git

The code transfers itself. It is text, it sits in version control, and any competent engineer can read it. Everything that makes the code actually run is somewhere else.

Credentials, and the whole apparatus behind them. Not just the password for the target account. The email address it was opened with, the phone number that receives the second factor, and whose name the account legally stands in. If it stands in yours, you are permanently in the loop on a system you no longer support. I now open target accounts on the client’s email address from the first day of a build, even when it costs me an hour of back and forth.

The proxy arrangement, including the billing. Which provider, which plan, what a normal month consumes in bandwidth, and whose card renews it. Get that last one wrong in either direction and it hurts. Either your card keeps paying for a client you stopped working for, or it lapses and the collector dies quietly on a Tuesday. I watched a team spend two weeks reading Python because a proxy plan had expired on a card belonging to somebody who left the company. On my own infrastructure a real SIM line runs about ten dollars a month in airtime, so a client inheriting three lines has taken on a thirty dollar monthly commitment, and they should hear that number from me rather than from their bank statement.

The quirks. This is the highest value page in the whole document and it is the one that gets skipped, because it is the only part you cannot generate from the code. Every target does something strange that you found the hard way and then worked around silently. The listing endpoint returns 24 items and the last two are always adverts. The rate limit counts per account rather than per address, so buying more proxies achieves nothing. A background call returns clean JSON, so the parser never touches the rendered HTML at all. Response quality degrades above roughly four pages a second, but only in the evening. None of that lives in the code. Some of it is the reason a line of the code looks idiotic, like a 2.3 second delay nobody explained.

I keep a plain text file per target while I build, one line per surprise. By delivery it is usually twenty or thirty lines long, and it is the page the receiving engineer reads twice.

Three failures, three pages

You cannot document everything that might go wrong. You can document the three things that will.

A selector breaks. The markup changed, a field arrives empty or wrong, and the row count looks perfectly healthy. Say which file holds the selectors, how to confirm the site moved rather than the code, and roughly how long a repair takes in hours.

The block rate climbs. Responses stop being data and start being challenge pages, usually carrying a cheerful 200 status. Say what a block page looks like on this specific site, how to measure the rate, and what to do first, which is nearly always slow the crawl down before anyone starts swapping proxy pools.

The volume moves. Forty thousand rows yesterday, two hundred today. Say how to distinguish a block from a layout change from the client’s own filter, in under ten minutes.

I have never needed a fourth entry. I have needed all three more times than I can count.

One signal they can read without you

Documentation only helps somebody who already suspects a problem exists. What decides whether you get blamed is whether the client finds out from a dashboard or from a stale report five weeks later.

So the final deliverable is a signal they can read themselves. Not the monitoring layer I would build for my own jobs, with rolling baselines per source, block page fingerprints and a run record for every execution. I have written that up separately and it is worth the engineering.

For a handover the requirement is smaller and much stricter. One page or one daily email, readable by somebody who does not write code, in under a minute on a Monday morning. Last successful run and when. Rows collected against what a normal day looks like. Field fill rate for the two or three columns that actually matter. A sentence at the top that says either this looks normal or this does not look normal.

Here is the position I will argue for: handing over a scraper with no monitoring is handing over a lawsuit with a delay on it. The data goes wrong silently, somebody prices a product off it or puts it in front of a board, and when they eventually work backwards to the cause, the last name in the commit history is yours.

Support request or new build

The commercial half is where the goodwill actually drains away. If the document does not draw this line, every site change becomes a free rebuild, and the client is not being unreasonable when they ask for one. From where they sit, they bought a working scraper and it is not working.

A support request is the collector doing something it was not doing at handover, on the same site, for the same fields. That carries a response time and either sits inside a retainer or bills at an agreed hourly rate.

A new build is everything else. A new field, a new page template, a new site, a new output format, a login wall that appeared since delivery, or ten times the volume anybody agreed to. That gets quoted like any other job.

If you already split the build from the run when you quoted the work, this section is copy and paste. If you did not, the handover is your last opportunity to draw the line before it defaults to unlimited.

The one I got wrong

A collector I delivered in March ran for eleven days after I stopped watching it. On day twelve the site renamed a CSS class, the price field started arriving empty, and the client found out five weeks later when their finance team asked why a competitor pricing sheet had not moved since Easter.

The class name was not my mistake. The build was solid: tests, retries, a validation gate that rejected malformed rows, structured logs.

My mistake was that I shipped the repository and a README and called that a handover. No runbook. No quirks file, and that target had three good ones. No monitoring the client could see, because my monitoring was a cron job on my own server that emailed me, and I switched it off the week the invoice cleared.

So the validation gate worked exactly as designed. It rejected the blank rows. The pipeline logged the rejects and carried on. The output file kept landing on schedule with fewer rows in it each week, and nobody was watching the count.

I had built the detection and then handed over the system without it, which is worse than never building it at all. It produced a machine that knew it was broken and had no way to tell anybody. I rebuilt it for free, about eleven hours. Writing the handover template I still use took one afternoon.

What a handover cannot fix

None of this makes a scraper last longer. The site changes when it wants to.

The document also does not survive staff turnover on the client side. The engineer you briefed leaves, the file sits in a drive folder nobody opens, and eighteen months later somebody inherits a collector nobody understands. I have been that somebody.

The honest claim is narrower than it sounds. A good handover moves the surprise from the client to the calendar. They still get broken data eventually. They just find out on the day it breaks, from something they can read, with a page that tells them what to do next.

And it changes nothing about what you were allowed to collect in the first place. Public pages, the robots file honoured, a crawl rate that does not hurt the target, an official API or a bulk feed preferred every single time one exists. A beautifully documented handover of a job you should not have taken is still a job you should not have taken.

Everything else I have written on scraping operations and the infrastructure underneath them is here.

Get new guides and videos first — join the Telegram channel.

There’s a whole category of products that promise to make scraping somebody else’s problem. You send a URL, they send back the HTML or the parsed data, and they handle the proxies, the browsers, and the retries behind the scenes. They’re called scraping APIs, and they’re either the smartest money you’ll spend or a slow leak in your budget, depending entirely on the job. I want to walk through what these services actually do, when buying one is the right call, when you should build your own instead, and how to run the cost math before you commit to either.

I run my own scraping infrastructure, and I’ve also paid for these services on real jobs, so this is a tested view, not a vendor pitch. I’ll be honest about both sides, because the right answer genuinely depends on your volume, your target, and what your time is worth. No service makes scraping undetectable, and none makes legal something that wasn’t already legal. What these tools sell is convenience, and convenience is worth a lot right up until it isn’t.

What a scraping API actually does

Start with what you’re really buying, because the marketing hides it. A scraping API is the infrastructure layer you’d otherwise build, rented by the request. Under the hood it maintains a pool of proxies, spins up headless browsers when a page needs rendering, handles the retries when a fetch fails, and hands you back a clean result. You’re not buying magic. You’re buying someone else running the proxy farm and the browser fleet so you don’t have to. That’s the entire value, and whether it’s worth it comes down to what running that yourself would cost you.

The three flavors you’ll meet

These services aren’t all the same, and they roughly split into three kinds. The simplest just fetch raw HTML through a rotating proxy and return it, which is cheap and fast for static sites. The middle tier renders the page in a real browser and returns the fully loaded HTML, for sites that need JavaScript. The richest tier returns structured data for specific popular targets, so you ask for a product and get clean fields instead of HTML to parse yourself. Price climbs with each step, because each one is doing more of the work you’d otherwise do.

The case for buying

Here’s when I reach for a service without hesitation. When the target is genuinely hard, the kind that has beaten datacenter and residential addresses and needs constant care, letting a specialist absorb that fight is often cheaper than staffing it yourself. When the volume is modest, paying per request costs less than standing up and maintaining your own infrastructure. And when your time is the scarce resource, a service that works today beats a build that works in three weeks. If scraping isn’t your core business, buying the boring part is usually the right trade.

The case for building

Now the other side. When your volume is large, per-request pricing turns brutal. A service that costs a fraction of a cent per page sounds cheap until you multiply it by ten million pages a month, and suddenly you’re paying more every month than a couple of servers and a proxy pool would cost outright. When you need full control over exactly how requests are made, a black box service fights you. And when your targets are simple, you’re paying a premium for infrastructure you didn’t need. At scale, on easy targets, building wins on cost by a wide margin.

Run the crossover math

So run the actual numbers before you decide, because there’s a crossover point and it isn’t subtle. Take your monthly page volume and multiply it by the service’s per-request price. Then estimate what your own stack would cost: the servers, the proxy pool, and an honest slice of your time to maintain it. Below the crossover, the service is cheaper and you should buy. Above it, your own infrastructure is cheaper and you should build. Most people never do this arithmetic and just guess, and the guess is usually wrong in whichever direction flatters the choice they already wanted.

The hidden cost of building

But be honest about the build side, because people lowball it. Running your own scraping infrastructure isn’t just server rent. It’s proxy costs, it’s the engineering time to build the retries and the rotation and the browser fleet, and it’s the ongoing maintenance when a target changes and your stack has to adapt. That last part is the one people forget. The service absorbs that maintenance for you, quietly, every day. When you build, that work becomes yours forever, and it doesn’t show up in the tidy cost estimate you made on day one.

The hidden cost of buying

And be equally honest about the buy side. A service is a dependency you don’t control. Its price can rise, its quality can drift, and if it goes down, your data goes down with it and there’s nothing you can do but wait. You’re also trusting a third party with your targets and your traffic. And there’s lock-in, because the more your pipeline is built around one service’s quirks, the harder it is to leave. Convenience today can become a cage tomorrow, so weigh the cost of depending on someone whose priorities aren’t your priorities.

Test before you trust the marketing

Whatever a service claims, test it on your actual target before you commit budget. The headline success rate on the marketing page is measured on easy sites, not on the specific hard target you care about. So run a real batch, a few thousand requests against the site you actually need, and measure the true success rate, the latency, and the cost per successful page. A service that boasts a very high success rate can quietly fail on your one difficult target, and you only find that out by testing. The vendor’s number is a promise. Your measured number is the truth.

Watch the per-request definition

Read the fine print on what counts as a request, because this is where the bill surprises you. Some services charge you for failed attempts, not just successful ones, so a hard target that needs several tries per page multiplies your cost silently. Some charge extra for rendering, extra for premium proxies, extra for the structured tiers. The sticker price and the price you actually pay can be very different once the target forces the expensive options on. So when you test, measure the real cost per successful record, not the advertised cost per request, because those two numbers are rarely the same.

The hybrid that often wins

It’s not always all or nothing, and the smartest setup is frequently a mix. Build your own stack for the bulk of your volume on the easy targets where you control the cost, and buy a service only for the handful of genuinely hard targets that would otherwise eat your time. That way you pay the premium exactly where it earns its keep and stay cheap everywhere else. I run this split myself: own infrastructure for the predictable heavy lifting, a service in reserve for the few sites that fight back hard enough to be worth outsourcing.

If you build, the proxy layer matters most

If you go the build route, the piece that decides whether it works is the proxy layer, because that’s the trust you arrive with. Cheap datacenter IP addresses will fail on strict targets exactly like they would inside a service, so the addresses you choose are the whole ballgame. This is the layer I run myself, real mobile proxies on real carrier SIM cards, because for the hard targets that’s the most durable trust you can put in front of a scraper. The rest of the build is retries and rendering, but the address is what gets you through the door.

What a service does not solve

Be clear about what a service doesn’t fix, because the marketing blurs it. It doesn’t decide what’s worth collecting, it doesn’t clean or model your data, and it doesn’t understand your target the way you do. It hands you raw results, and the whole job of turning those into something useful is still yours. So a service saves you the infrastructure, not the thinking. I’ve watched people buy an expensive scraping API and still have most of the work in front of them, because the hard part of a data project was never the fetching. It was knowing what to fetch and what to do with it afterward.

Reliability and support are the real product

When you depend on a service, its reliability becomes your reliability, so weigh that before you commit. How often does it go down, how fast does it recover, and is there a real human to reach when a target suddenly stops working. A cheap service with no support is fine until your pipeline breaks on a Monday morning and you’re on your own. I pay attention to the boring signals here: the status history and the response time when I open a ticket, because those tell you more about living with a service for a year than the price or the feature list ever will.

Start small and stay portable

However you lean, don’t marry the decision on day one. Start with the cheapest option that clears your target, prove the job works end to end, and only then scale the spend. Keep your pipeline portable, so the fetch layer is a piece you can swap. If you wrap whichever service you pick behind a thin boundary in your own code, then switching services, or moving from a service to your own stack, is a small change instead of a rewrite. The goal is to keep the choice reversible, because your volume and your targets will change, and the right answer will change with them.

The honest limits

Let me be straight about the boundaries, because no service changes them. Buying a scraping API doesn’t make anything undetectable, whatever the landing page implies, and it doesn’t make it legal to collect data that was never yours to collect. The same rules apply as always: public data, a robots.txt file respected, an official API or bulk feed preferred where one exists, a polite rate held. A service can absorb the infrastructure work for you, but it can’t absorb the responsibility for what you scrape. That stays with you no matter whose proxies the request rides on.

I run both sides of this in production, my own stack and these services on real jobs, so this is a tested comparison, not theory. The whole decision comes down to one question asked honestly: is running this infrastructure yourself cheaper than renting it, at your volume, on your targets. Answer that with real numbers and the choice makes itself.

To recap: a scraping API rents you the proxy and browser infrastructure you’d otherwise build, sold by the request across three tiers of increasing price and convenience. Buy when the target is hard, the volume is modest, or your time is the scarce thing. Build when the volume is large, the targets are simple, or you need full control. Run the crossover math with honest costs, test on your real target, watch what a request actually costs, and mix the two where it pays.

For more breakdowns like this, tested on real infrastructure with no undetectable promises, visit Data Research Tools.

Get new guides and videos first — join the Telegram channel.