Category: Uncategorized

Most explanations of proxy rotation stop at “use a different ip for each request.” That advice is incomplete enough to be actively harmful. I run proxy infrastructure and production scraping pipelines, and the pools that hold up in production are built around a much narrower idea than “rotate a lot”: match the rotation pattern to the thing a human session actually looks like, and treat the pool itself as infrastructure you monitor, not a bucket of ips you fire and forget.

This is an engineering explainer, not a bypass guide. Nothing here makes a pipeline undetectable, and I would not trust anyone who tells you otherwise. What follows is how rotation actually behaves at the network layer, where it helps, where it hurts, and how to build a pool that keeps working next month instead of just today.

What rotation actually changes

A proxy rotation setup swaps the source ip a request appears to come from. That’s it. It does not change your TLS handshake, your header order, your browser fingerprint, or your request timing. Rotation only touches one signal out of the several a defended site is checking, which is exactly why “just rotate ips” advice underperforms so often. If everything else about the request stays constant while the ip changes every time, you’ve built a fleet of visitors who all type identically, click identically, and only differ by return address. That’s its own pattern, and pattern is the thing detection systems are built to notice.

So the first thing to get straight is what problem rotation is actually solving. It solves rate and reputation at the network layer: no single ip accumulates enough request volume to look like a flood, and no single ip burns its reputation down to zero. It does not solve consistency at the application layer. Those are separate problems and need separate fixes.

Request-level rotation vs session-level rotation

There are two fundamentally different rotation patterns, and picking the wrong one for the job is the most common mistake I see.

Request-level rotation assigns a fresh ip to every single HTTP request. It’s the default in a lot of proxy provider SDKs because it’s the simplest thing to implement. It works reasonably well for stateless collection: hitting the same type of public page repeatedly, where each request stands alone and doesn’t depend on a prior one setting cookies or an authenticated state.

Session-level rotation (often called sticky sessions) holds one ip for a defined window, usually somewhere between a few minutes and the length of a full session, then rotates. This matters anywhere a site expects continuity: a login flow, a multi-step checkout, a search followed by pagination, anything with a session cookie. A real visitor doesn’t change network address between page two and page three of a search result. If your pipeline does, that’s a contradiction a site can check for cheaply, because it just has to compare the ip on the session cookie against the ip on the current request.

The practical rule is boring: match the rotation window to the workflow. Stateless single-page pulls can rotate per request. Anything with a session, a cart, a login, or sequential pagination needs a sticky window long enough to cover the whole flow, then a clean rotation once that flow is done.

Why fast rotation can hurt more than it helps

There’s a temptation to treat rotation speed as a dial you turn up for safety: rotate faster, get flagged less. In practice, over-rotating introduces its own tell. A visitor who never returns on the same ip twice, who pages through a site with a new address every single click, is behaving in a way no real browsing session does. Home and mobile users keep the same address for an entire visit, sometimes for hours. A pool that rotates on every request manufactures a pattern that is, ironically, easier to cluster than a slower, more human-shaped one.

There’s also a cost side that gets ignored. Every rotation is a new TCP and TLS handshake, and on some proxy types a new authentication round trip. Rotating per request against a target that would have tolerated a sticky session for ten minutes just adds latency and connection overhead for no benefit. Rotation frequency should be driven by what the workflow actually needs, not by an assumption that more churn equals more safety.

Pool composition matters more than pool size

A pool of ten thousand cheap datacenter ips and a pool of two hundred well-distributed residential or mobile ips are not comparable tools, and size is the least important number between them. Datacenter ranges are labeled at the network level, cheaply and at scale, by services whose entire business is classifying ip blocks. A datacenter ip starts a request under suspicion regardless of what that specific address has or hasn’t done, because the label is on the neighborhood, not the individual unit.

Residential and mobile ranges start with more default trust because that’s where real traffic overwhelmingly comes from, but they carry their own constraints: mobile ips in particular are shared across many devices behind carrier-grade NAT, so an ip’s recent history isn’t fully yours to control. The point isn’t that one type is universally correct. It’s that pool composition should be chosen against the specific target’s sensitivity, not against a generic “more ips is safer” instinct. A lightly defended target can run fine on a modest datacenter pool. A target with a serious anti-bot stack in front of it will burn through that same pool fast regardless of how large it is.

Rotation triggers that actually correlate with problems

The rotation trigger that matters most in a production pipeline isn’t a timer, it’s the response. Treat every 403, 429, captcha challenge, or unexpected redirect to a verification page as a signal about that specific ip, and retire it from the active pool rather than reusing it on the next request. Reusing a flagged ip immediately just spends more of the pool’s reputation on an address that’s already burned.

Time-based rotation is a reasonable default when you have no better signal, but it should be a floor, not the whole policy. A sensible setup layers both: rotate on a sticky timer to bound normal session length, and rotate immediately, out of band, the moment an ip returns a block signal. Track failure rate per ip over a rolling window too. An ip that’s clean 95% of the time against a given target and one that’s clean 40% of the time are not interchangeable, even if both are technically “up.”

Keep the rest of the request in sync with the ip

Rotation only earns its keep if the parts of the request that aren’t the ip stay consistent with the story the ip is telling. If a session rotates onto a mobile ip, the user agent, TLS fingerprint, and header set should look like something that plausibly came from a mobile browser on that carrier, not a leftover desktop Chrome profile from the previous session. Pair proxy identity with a consistent client fingerprint per session, and rotate both together rather than swapping one and leaving the other stale. This is the same principle any defended site is checking for from its side: do the pieces of this request agree with each other. A rotation policy that ignores it is solving one axis of a two-axis problem.

Monitor the pool like infrastructure

The operational mistake I see most often isn’t a bad rotation policy, it’s no visibility into the pool at all. Ips age out, ranges get relabeled, providers churn their allocations, and a pool that was clean last month can be half-burned today with no code change on your end. Track success rate, latency, and block rate per ip and per subnet, not just in aggregate. Retire consistently poor performers automatically instead of noticing three weeks later that a chunk of the pool has been silently eating your error budget. A rotation strategy is only as good as the feed of pool health it’s reacting to.

The honest limits

None of this is a formula for staying invisible. Detection systems evolve, targets ship new checks, and a pool or a rotation pattern that works cleanly against a given site today can get flagged next month with no warning. What good rotation actually buys you is fewer unforced errors: no single ip absorbing enough volume to become an obvious flood, sessions that don’t contradict themselves by teleporting networks mid-flow, and a pool you can see the health of instead of guessing. That’s an engineering property, not a guarantee, and treating it as anything more than that is how pipelines end up brittle right when you need them not to be.

I run this infrastructure in production, and the proxy setups, scraping frameworks, and honest tool reviews I write about are the ones I actually deal with day to day. For the full guides on building pools, picking proxy types, and the tools I’ve tested, head to dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

Fourteen scheduled jobs on one pipeline. I could name a human or a downstream consumer for nine of them. The other five had been running on a timer for somewhere between eight months and three years, and the honest answer to “who reads this” was that I had no idea.

That ratio is normal. Any pipeline alive longer than a year collects jobs whose reason for existing left the company with the person who asked for them, and the job carries on because a cron entry has no opinion about relevance.

The bill you can see and the bill you cannot

The visible cost is small enough to ignore, which is exactly why it gets ignored. One of those five fetched around 9,000 pages every six hours. On mobile addresses that is a real slice of a line I pay for monthly, plus the parsed table, plus the raw responses behind it, plus a nightly backup of both. Add it up and it is a rounding error against payroll.

The invisible cost is the one that hurts.

An old job breaks occasionally. A proxy goes bad, a template shifts, a run times out. That fires an alert, and somebody opens the alert, recognises the job name, and closes it. Second time they close it faster. By the third month they close it without reading it, and that reflex does not stay politely attached to the one job that trained it. You have taught a person that alerts from this pipeline are noise.

Then there is incident time. When the proxy pool degrades or a queue backs up, the dead job is right there in the graphs near the top of the request count, and somebody has to rule it out before reaching the actual fault. I have spent 40 minutes of a two hour incident eliminating jobs that turned out to have zero readers. That is the price: skilled attention at the worst possible moment, several times a year.

And one cost lands entirely on someone else. Those runs sent roughly three quarters of a million requests to a company that got no order, no signup, and no ad impression out of any of them. Public pages, robots file honoured, a pace slow enough that it would not have registered on their side. All true, and it was still three quarters of a million requests in service of a table with four reads in it.

Why the decision never gets made

Nobody turns these off because nobody can prove they are unused, and the cost of guessing wrong feels lopsided.

Delete it and be wrong, and you have quietly broken something for a colleague who will discover it at month end. Keep it and be wrong, and you pay a few dollars a month that nobody measures. Given that asymmetry, “keep it” wins every review, in every quarter, indefinitely.

The framing is the problem. You are being asked for a judgement call while the evidence is missing, and the correct response is to go and collect the evidence rather than argue about the odds.

A green run is a write side metric

Every scraper I have built reports on itself: runs completed, rows written, error rate, duration, and a chart that sits above 99%. All of that describes what happened on the write path. It confirms the code executed and the rows landed where they were pointed.

It says nothing about whether anybody wanted them.

That is the confusion keeping zombie jobs alive. From the inside, a healthy job and a useful job look identical. A scraper cannot distinguish between feeding a live pricing model and filling a table nobody has opened since last March, because both of those runs are green.

Watching for silent failures in a job that matters is a separate discipline with its own signals, and I have written about it separately. This is the prior question. Before you invest in monitoring a job properly, establish that the job has a reader.

Count the reads

Reads are the only real evidence of use. Everything else is inference with confidence added.

So measure them per table, per file, per endpoint, with an identity attached wherever the platform will give you one.

In Postgres, the counting is already happening. pg_stat_user_tables keeps a per table tally of sequential and index scans that increments on every read. It resets when the server restarts, so snapshot it on a daily cron and store the delta. Two weeks of deltas gives you something to argue from.

On object storage, switch on bucket access logging and retain 90 days. Same signal, cheaper.

Behind an API, log the caller identity on every request. A count tells you the table gets read. An identity tells you who to go and talk to.

Feeding a dashboard, most reporting tools already record a “last viewed” timestamp and viewer per chart. That field is usually sitting unused in a metadata table.

Exclude your own plumbing first

This trap will waste a month if you let it. Your own infrastructure reads the table constantly. The nightly backup reads it. Replication reads it. A health check running SELECT 1 against it every minute reads it, and that alone will report 1,500 reads a day from a table no human has touched.

Give those a dedicated database user or API key and filter that identity out of the count. Twenty minutes of setup, and without it the entire exercise returns noise.

What should remain is a person running a query, or another job in your own pipeline consuming the output. Those two count as use. Everything else is your plumbing talking to itself.

If you cannot instrument reads at all, the crude version still works: switch it off and see who shouts. I have used it. It only stops being reckless once you have announced it, because otherwise you are breaking things and calling it research.

Pause before you delete

Even holding the numbers, people hesitate, and the hesitation is rational. Deletion is the one operation with no undo.

So do not open with deletion. Four steps, and the first is free.

Stop the schedule. Comment out the cron line, pause the DAG, disable the trigger. Code stays. Data stays. Nothing is destroyed and the reversal is a single line.

That step alone captures nearly all the benefit. Requests stop, alerts stop, the job disappears from the incident graphs, and no permanent decision has been made.

Announce it. Wait a defined window. Then delete the schedule and the job definition, and stop there.

Word the announcement as a decision

Put it where your team already argues about deploys, not on a wiki page nobody has opened since onboarding.

And phrase it as a decision with a date rather than a question. Ask “does anyone use this?” and you get silence, silence reads as risk, and the job runs for another year. Say instead: this job is paused, the table stays, and unless something comes up it gets deleted on the 15th. Now silence is an answer, because you stated the default and gave people a cheap way to override it.

That single change of wording has unstuck more of these for me than anything technical.

What survives the deletion

Keep the data. Keep enough of the code to explain how the data was produced.

Somebody will ask, two years from now, what a column in an old report meant. Whether the price included tax. Why the figures jump in one month. The answer to all of that lives inside the parser, and if the parser is gone the answer went with it.

The case for versioning a scraper alongside its output and holding raw responses is made elsewhere on this site, and I am not going to relitigate it here. This is the same argument arriving from the other end. Code is the documentation for data, so the code outlives the schedule that ran it. A repository with a dead job in it costs a few megabytes.

Where a short window lies to you

Here is the one I got wrong, and it is worth more than the rest of this.

I instrumented reads on a job for 30 days, saw nothing outside my own queries, paused it and deleted it a month later. Clean process, exactly the steps above. In January somebody went looking for the year end comparison and found the table stopped in October.

Thirty days of read data proves nothing about anything quarterly, and nothing whatsoever about a report that runs annually. Every business I have worked with has at least one of those.

So the window has to exceed your longest reporting cycle, which for most people means a full year, and nobody will wait a year before turning something off. The way out is the pause. Pause for the year rather than run for the year. Pausing costs nothing, the history stays intact, and if December brings a complaint you restore a cron line and lose an afternoon.

The honest limit: a pause is reversible for the code and not for the calendar. Months spent paused have no rows in them and never will. On a time series that is already a partial deletion, and you owe people that detail in the announcement.

The position

A scraper you cannot prove is used costs more in attention than it costs in servers, and attention is the resource you are actually short of.

I would rather delete a job somebody quietly needed and rebuild it than carry twenty I cannot account for. Rebuilding a collector against a site you have already solved is two days. Twenty unexplained jobs is a permanent tax on every incident and every new person you hand the pipeline to. The rebuild is only cheap because you kept the data, which was always the expensive half, and which the procedure above is designed to protect. More on pipeline hygiene and the proxy infrastructure underneath it is at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

The first time a client asked me what it would cost to scrape a site, I gave them a number off the top of my head, and it was wrong by a mile. I had priced the one thing everyone prices, the proxies, and forgotten almost everything else. By the time the job was running, the real bill was close to three times my guess, not because anything went wrong, but because a scraper costs money in five different places and I had counted one of them.

I run proxy infrastructure and production scrapers for a living, so I have paid every one of these bills myself, more than once. This is not a pricing page I copied. It is where the money actually goes when you run this at scale, and which parts of it people forget until the invoice lands. Here is the whole picture in one table before the detail.

Cost bucket	What you pay for	Usually billed	Easy to forget?
Proxies and bandwidth	The traffic that carries your requests	Per gigabyte or per request	No, everyone prices this
Compute	The machines that run your code	Per hour of CPU and memory	Sometimes
Unblocking	Solvers and scraping APIs when a site pushes back	Per solve or per request	Sometimes
Storage and pipeline	Where the data lands, lives, and gets queried	Per gigabyte, per month	Often
Maintenance	The human hours that keep it all alive	Engineer time	Almost always, and it is the biggest

Most people quote the first row and pretend the other four are free. That is exactly the mistake I made.

Bucket one: proxies and bandwidth

Proxies are the bucket everyone fixates on because it is the easiest to see a price for. Residential and mobile addresses are usually sold by the gigabyte, so every page you pull, and every image and script it drags along, adds to a running meter. Datacenter addresses are cheaper and often sold by the request, but they get blocked far more often, which pushes the cost somewhere less obvious. This bucket is real, but it is almost never the whole story, and treating it as the whole story is how you underquote by a factor of three.

Cost per successful page, not cost per gigabyte

The trap inside the proxy bucket is thinking in cost per gigabyte when you should think in cost per successful page. A cheap datacenter address looks like a bargain until half your requests come back blocked and you fetch each one again, paying twice or three times over for a single page you keep. A pricier mobile address that lands on the first try can be cheaper per good page than the bargain that fails constantly. Do not shop on the sticker rate. Shop on what it costs to end up with one clean page you can use, because that is the number your budget actually feels.

Bucket two: compute

Compute is where the browser question quietly decides your bill. A plain HTTP client that fetches raw HTML is almost free per page. It barely touches memory or processor time, so you can run many of them on a small cheap machine. A real browser is the opposite. It eats memory and processor time for every page it opens, so a collector built on browsers can cost you many times what the same job costs with a plain client. On a cloud bill that difference is not a rounding error, it is often the largest line you have.

The lever is simple, and it is the same one I preach everywhere. Reach for a real browser only on the pages that genuinely need one, where the data is painted on by JavaScript and is not in the raw HTML. For everything else, usually most things, a plain client that grabs the fully formed HTML is faster and dramatically cheaper. Reaching for a browser by default, on pages that never needed it, is the single most common way I watch a compute bill balloon for no reason.

Bucket three: unblocking, and an honest word on detection

Unblocking is the bucket people either overspend on or refuse to think about. CAPTCHA solving services charge you per solve. Scraping APIs charge per request, but they bundle the proxies, the browser, and the retries into one price so you manage none of it yourself. That convenience is genuinely worth paying for on some jobs. The point here is the cost shape: you are paying a premium for someone else to absorb the hard part, and that premium is easy to lose track of when it is buried inside a per request rate.

I have to be honest about this bucket, because it is where people get the wrong idea. None of these services make you undetectable, and none of them buy you the right to take something you were not allowed to take. They help you fetch public pages more reliably, that is all. You still move at a gentle pace, respect the robots file and the site’s terms, and take public data only. The deeper you go into fighting a detection team that ships updates faster than you do, the more this bucket costs and the less durable your setup becomes. The cheapest long run path is almost always the one that is not adversarial in the first place.

Bucket four: storage and the pipeline

Storage is everything that happens to the data after you pull it. You keep the raw pages, the cleaned and parsed results, probably a database or a warehouse to query them, and the machinery that moves data from one stage to the next. None of it is huge on day one, but it grows every day the job runs, and it is a recurring cost, not a one time purchase. A scrape you run nightly quietly multiplies its storage and processing by the number of nights, and that slow accumulation is a part people forget entirely when they budget.

There is a small nasty surprise hiding in this bucket too. Moving data around costs money in ways that are easy to miss. Many cloud providers let you bring data in for free but charge you to take it back out, so the day you export your results or hand them to a client, a bill you never saw coming can appear. It is rarely the biggest number, but it is exactly the kind that ruins a tidy estimate, so leave a little room for it.

Bucket five: maintenance, the one that dwarfs the rest

Maintenance is the bucket that dwarfs the other four on any job that runs for more than a few weeks. Websites change their layout, and when they do your selectors point at nothing and start collecting blanks. Addresses get blocked and pools need topping up. A site adds a new defence and your collector needs adjusting. Every one of those is a human sitting down and fixing something, and human hours are the most expensive thing in this whole picture by a wide margin. A scraper is not a thing you build once, it is a subscription of attention, and the attention is what you are really paying for.

This is also why the cheap tool is so often the expensive one. A fragile collector that saves a little on proxies but breaks every other week, and pulls an engineer off other work each time, costs far more in the end than a slightly pricier setup that just runs quietly. When you count the cost of a scraper, count the hours, not only the dollars on the cloud invoice, because on a long running job the hours are the invoice.

Build cost versus run cost

The single framing that fixes most bad estimates is separating the build cost from the run cost. Building a scraper is a one time job. You write it, test it, get it working, and that is a fixed number of hours you pay once. Running it is a completely different thing, because the proxies, compute, storage, and maintenance all land again on every execution, night after night. People quote the build and forget the run, and the run is where nearly all the money lives over the life of the job. Always price them as two separate things.

Which bucket dominates depends on scale

Which bucket dominates depends entirely on the size of your job, and knowing your regime tells you what to optimise. On a small job, a few thousand pages here and there, the infrastructure is basically free and your own time is the whole cost, so build it fast and do not over engineer it. On a huge job, millions of pages a day, the proxies and compute become the giants and shaving a little off cost per page saves real money. Optimising the wrong bucket for your scale is wasted effort, so work out which regime you are in before you tune anything.

How I estimate a job

When someone asks me to estimate a real job, I walk the five buckets one at a time. I guess how many pages a day, and whether they need a browser or a plain client, which sets the compute. I estimate a cost per successful page for the proxies, allowing for retries, not the sticker rate. I add whatever solving or API help the site will force on me, and storage that grows over time. And then, most important, I add a standing allowance for maintenance, because I know the site will change and something will break. It is rough, but it is honest, and it is never off by a factor of three.

The cheapest scrape is the one you never run

The best cost lever of all is the one people skip: simply not fetching pages you do not need. Cache what you already have so you never pay for it twice. Fetch only what has changed since last time instead of pulling the whole site again every night. Hit the quiet background request that hands you clean data directly, so you skip the heavy browser entirely. Every page you avoid is pure saving across all five buckets at once, no proxy, no compute, no storage, no maintenance. The cheapest scrape in the world is the one you were smart enough not to run.

I run this infrastructure in production myself, so none of this is theory, it is the bill I actually pay. If you want the full written breakdown of all five buckets, the way I estimate a job before I quote it, and the proxy pool I route my own scrapers through, it is all here, with no magic numbers and no promise that your job will cost exactly what mine does, because that is not a thing anyone can honestly tell you without seeing your pages.

Get new guides and videos first — join the Telegram channel.

I reach for a browser automation tool only when a page forces me to, and when I do, the choice is almost always between these three. Here is the short version before the detail.

Tool	Best for	Languages	Browser engines	Main trade-off
Playwright	New scraping work that genuinely needs a browser	Python, JavaScript, Java, .NET	Chromium, Firefox, WebKit	Younger, smaller community than Selenium
Puppeteer	A Node and Chrome shop that wants a lean, fast tool	JavaScript (Node)	Chromium (limited Firefox)	Chrome-centric, effectively one language
Selenium	Existing codebases, unusual languages, real cross-browser QA	Python, Java, JS, C#, Ruby, more	All major browsers	Heaviest to run, manual waits

The verdict, if you want it in one line: for a new scraping job that needs a real browser, pick Playwright. Pick Puppeteer if you live entirely in Node and only care about Chrome. Keep Selenium for large existing codebases, exotic languages, or genuine cross-browser testing. And before any of them, check whether you need a browser at all, because most of the time you do not.

What these three tools actually are

All three are browser drivers. They exist to open a real browser, run the page’s JavaScript, build the live document, and let you click, scroll, wait, and then read the result. You only need one of them when the data you want is painted onto the page by JavaScript and simply is not present in the raw HTML a plain request returns. If a normal HTTP client already gives you the data, none of these tools belong in your pipeline, and reaching for one anyway is the most common way I see scraping costs balloon.

Once you accept that framing, the tools have more in common than the arguments online suggest. Each drives a real engine, executes scripts, and hands you a way to wait for elements and extract them. The selectors and parsing you write at the end look nearly identical whichever you choose. The real differences live in four places: which languages each supports, how it talks to the browser, how it handles waiting, and how many browser engines it can drive.

Selenium: the standard everyone built on

Selenium is the elder, and it earned its place. It made browser automation a normal thing to do, it speaks the WebDriver standard that the whole industry settled on, and it supports more languages than anything else here: Python, Java, JavaScript, C#, Ruby, and more. Its community is enormous, so almost any error you hit has already been hit and documented by someone else. If you have a large existing codebase or your team works in a language the newer tools barely serve, that gravity is real.

The cost is that Selenium carries its history and you feel it. Classically it talks to the browser through an extra driver process sitting between your code and the browser, which adds overhead and moving parts. More painful for a scraper, its waiting was always manual: you tell it to wait for a specific condition, and a slightly wrong condition gives you exactly the intermittent failures that plague a fragile collector. Recent versions closed much of this gap and are genuinely better, but Selenium is still the heaviest of the three to run and the fussiest to keep steady at scale.

Puppeteer: a fast, direct line to Chrome

Puppeteer came from the Chrome team and took a different road. Instead of the WebDriver hop, it talks straight to the browser over the Chrome DevTools Protocol, the same channel the browser’s own developer tools use. That direct line makes it fast and gives you deep control: easy network interception, fine-grained request handling, and clean access to the browser internals. If your stack is Node and you mostly care about Chrome, Puppeteer is a joy to work with, light and quick and close to the metal.

The price of that closeness is reach. Puppeteer grew up as a Chrome and Chromium tool in JavaScript. It has stretched a little beyond that, but it is not the answer if you need real coverage of other browser engines or you work in Python. It is a sharp, focused instrument for the Chrome world rather than a universal one, and that is a perfectly good thing to be as long as the Chrome world is where you actually live.

Playwright: the one I reach for first

Playwright is the newest, built by some of the same people who built Puppeteer after they moved to Microsoft, so think of it as Puppeteer’s lessons applied a second time with wider ambitions. It drives Chromium, Firefox, and WebKit, the engine behind Safari, all through a single API. It supports several languages properly: Python, JavaScript, Java, and .NET. And it bakes in automatic waiting, so before it clicks or reads something, it waits for that thing to actually be ready.

That auto-waiting and the one API across three engines are the real story for a scraper, not a bullet point. The flakiness that eats your evenings usually comes from timing, from your code touching an element a fraction of a second before the page finished drawing it. Playwright waiting by default, for the normal case, removes a whole category of random failures and the scattered sleeps people use to paper over them. On a new job that needs a browser, it is the first tool I reach for, and it is the one I recommend most often.

Speed is not the number that matters

People fight about raw speed far too much. For real scraping, the thing that dominates your time is the page loading over the network and the browser rendering it, not the tiny difference in how a driver sends a command. Do not choose one of these tools on a micro-benchmark someone posted. Choose on how often it fails for no reason and how much of your life you spend maintaining it, because that is where the true cost sits.

Debugging is where you actually pay

One difference never shows up in a feature table and yet it changes your whole week: how each tool helps you when a scrape breaks, and it will break. Playwright ships with a trace viewer that records every step, every network call, and a filmstrip of what the page looked like at each moment, so when a selector suddenly finds nothing you can rewind and watch exactly what went wrong. Puppeteer gives you clean screenshots and the full DevTools Protocol to inspect things yourself. Selenium leaves more of that to you and to add-on tools you wire up. When you keep a scraper alive for months, the tool that shows you why it failed saves more hours than any raw speed ever could.

A word on detection and stealth plugins

All three of these launch a real, automated browser, and a serious site can tell an automated browser from a human one. None of them is undetectable, and I would not trust anyone who tells you otherwise. The point of using a browser is not to sneak past a wall, it is to render JavaScript so you can read a page you are allowed to read. You still run at a gentle pace, you still take public data only, and you still honor the robots file and the site’s terms.

You will also meet a whole ecosystem of stealth plugins that promise to hide these tools from detection. I understand the temptation, but I do not build a business on staying one step ahead of a detection team that ships updates faster than I can. Those patches go stale, they break on the next browser release, and they pull you straight into the adversarial framing that gets addresses and accounts burned. The durable version of this work is the boring one: render only the pages you are allowed to touch, move gently, and take public data, so that most of the detection stack never has a reason to look hard at you.

The cost of a browser, and the proxy angle

A real browser is heavy. It eats memory and processor time, and it does that for every page you open, so a browser-based collector can cost you many times what a plain HTTP scraper costs for the same pages. The discipline is simple: use the expensive tool only on the pages that truly need it, and let a cheap plain client handle the bulk that arrives fully formed in the HTML.

There is a proxy angle here too, and it is larger with browsers than most people expect. A real browser does not fetch one thing per page. It fetches the document and then every image, script, and font the page pulls in, so your traffic, your bandwidth, and your footprint on the site all multiply. Whichever driver you pick, you are still routing that through proxies, and rotation and pace matter more, not less. I run my browser jobs through a managed mobile proxy pool for exactly that reason, so the browser can render in peace while the fetching stays quiet and welcome.

How to choose

In practice the choice is clean. Pick Playwright for new browser work, for the auto-waiting, the several languages, and the one script that runs across three engines. Pick Puppeteer when you live entirely in Node and only need Chrome and want a lean tool with a direct line to the browser. Reach for Selenium when you already have a large codebase on it, when your language is one the others do not serve, or when you genuinely need broad cross-browser coverage for testing as well as collection.

When you need none of them

The option that beats all three is needing none of them. Before you spin up a single browser, check whether the data is sitting in a quiet background request the page makes, or whether the HTML is already fully formed when it arrives. If it is, skip the browser entirely, because a plain client that hits that request directly is faster, cheaper, and steadier than any automated browser will ever be. The browser is the tool of last resort, not the default.

I run this infrastructure in production myself, so none of this is theory. If you want the full written setups for each tool, the waiting patterns that killed my flaky failures, and the proxy pool I actually run browser jobs through, it is all here, with no undetectable promises and no guaranteed results, because that is not a thing anyone can honestly sell you.

Get new guides and videos first — join the Telegram channel.

A client once asked me why one supplier in a price file looked about 40% cheaper than everyone else in the same category. The supplier was not cheaper. That listing was priced per 100 grams while the rest of the file was priced per pack, and my parser had written both into a column called price.

Nothing in that run was broken. The fetch worked, the selector matched, the string cast to a float, the row inserted. The dataset was wrong anyway, and stayed wrong until somebody found it odd enough to ask.

That is the category worth writing about. Not scrapers that fail, which announce themselves. Values that parse successfully and mean something other than what you assumed.

The failure mode with no exception

Most scraping writing is about getting the page. After that the assumption is that extraction is mechanical: find the node, take the text, cast it to a type.

The cast is where the damage happens. float("1.234") works. Parsing 03/04/2026 as a date works. Neither call knows anything about the site it came from, so neither can tell you it has just produced a confident, well typed, wrong answer.

A validation layer will not catch this. I have written separately about validating records before they hit your database and those rules earn their place, but they catch impossible values: a negative price, a date in the year 2900, a required field that came back empty. Everything in this article is entirely possible. It passes every bound you would think to write.

Deduplication does not catch it either. That work, which I have covered on its own, decides whether two rows describe the same thing. Two rows can be identical and both wrong in the same direction, off the same misread convention.

Three ways a date goes quietly wrong

The first is ambiguity. 03/04/2026 is the third of April and it is also the fourth of March. Whichever your date library picked, it picked from a default, usually inherited from the locale of the machine running the job. That machine has never seen the site.

You cannot resolve this from one string. You can resolve it from a column. Pull a few thousand dates from that source and look for a single row where the leading number exceeds 12. That row tells you the convention, and now you have evidence instead of a default. If you are lucky the markup carries a machine readable date beside the human one, with a four digit year, and you can skip the whole exercise.

The second is relative time. “Posted 2 days ago” is not a date, it is an offset from a moment the page never states. Store the string and you have stored nothing. Resolve it against the moment you fetched, store the resolved instant, and store the fetch time next to it. Skip that last part and someone reparsing your stored HTML eight months later gets a different answer with no errors and no way to tell which run was right.

The third is time zones. A date with no zone is a guess you have already made without recording it. A listing posted at 23:40 local sits on either side of a day boundary depending on where you stand, and any report that groups by day will move it between buckets based on nothing more interesting than which server ran the query. I keep the instant in UTC with the offset recorded, and I keep the site’s own calendar date in its own column whenever the day boundary is what someone is going to count.

Money is at least two fields

A currency symbol is not a currency. The dollar sign is shared by Singapore, Hong Kong, Australia, Canada, New Zealand and the United States, several of them on English language sites that look much alike. A scraper that reads the glyph and writes USD is inventing information.

The currency has to come from somewhere with real signal: an explicit three letter code in the markup, structured data, the domain, or the locale you requested. That last one is a separate subject, requesting a locale and keeping the request consistent so the site serves the version you asked for. I have covered it on its own. This article is what happens after the page answers.

The two ends have to agree. Requesting the Singapore storefront and then hard coding USD in the parser produces a dataset that is confidently wrong at scale, and no single row in it looks suspicious.

Then the separators, where I have lost the most money. In one convention 1,234.56 is a thousand and a bit; in another the same value is written 1.234,56. The two characters swap jobs. When both appear you can work it out, because the one closer to the end is the decimal point. The trap is a single separator with exactly three digits after it. 1.234 is either one and a bit or one thousand two hundred and thirty four, and nothing in the value picks between them.

Column level checks catch this where row level checks cannot. Sample a thousand prices from one source. If a third of them terminate in exactly three digits after a dot, that dot is a thousands separator, because real prices do not cluster that way.

Two more money details that bite:

• Minor units are not always hundredths. Japanese yen has no decimal subdivision at all, so a schema that stores everything as cents quietly multiplies by 100.
• Tax handling differs between sites in the same market. One quotes with consumption tax included, the next quotes without, and both call the field “price”.

Converting at scrape time destroys information

The most expensive habit I see in other people’s pipelines is converting currency during extraction and storing only the result.

Once that number lands you can no longer answer what the page said, which rate was applied, or whether that rate came from the scrape date or the run date. If the rate feed was wrong, or two jobs applied the conversion twice, there is no route back. A conversion is not reversible when you did not keep the input.

Store the amount as the page gave it, the currency code beside it, and the rate with its own date as separate fields. The converted figure is then a view you can rebuild whenever your understanding improves.

Quantities need their denominator

A price without its unit is meaningless, and the unit is frequently nowhere near the price field. It lives in the title, on a separate line, in a variant selector, or it is implied by the category and stated nowhere on the page.

Sizes behave the same way. A size 10 garment is different clothing depending on whose sizing the site uses, and a bandwidth figure where one vendor means megabits and the next megabytes is off by a factor of eight.

The rule sitting under every example so far: every value on a page is a rendering of something, produced under a convention the page rarely states. What you scraped is the rendering. What you wanted was the thing.

Keep the string, in the row

So the discipline, which is short.

Store what the page said next to what you decided it meant. The raw text of that field, untouched, in the same row as the parsed value. There is a broader argument about retaining whole raw responses for provenance, and that is a separate topic. This is the narrow version, at field level, and it is the one that pays off during an incident. Nobody joins to cold storage while a customer is waiting.

Never overwrite that string afterwards. Not to trim whitespace, not to repair an encoding, not when a parsing rule improves. It is the only record of what you were handed.

Put the assumption in the column name

A column called price is a lie. A column called price_sgd_cents is a claim somebody can check against a page. Parsing code is where assumptions go to die, because in six months a colleague will query your table and never open your parser. The schema is the document people actually read.

Here is the position I will argue about: any normalisation that discards the original is a decision you cannot revisit, and you will want to revisit it. Every rule you write is your best understanding of a source on the day you wrote it. Sites change and your understanding gets better, and the whole value of a stored dataset is that you can apply what you learned later to what you already collected.

The four months I did not notice

The bug that taught me all of this was a European decimal separator read as a decimal point. That column was wrong by a factor of 1000 for four months.

I had validation. The rule said greater than zero and under a ceiling, and 1.23 sailed through it. I had monitoring on parse failures, and there were none, because nothing failed. What I did not have was the original string in the row, so when the question arrived I could not answer it from the table. I went back to stored pages and some had rotated out.

The repair took an afternoon: keep the raw price text beside the parsed value, and run a per source distribution check that flags when digits after a separator cluster in a way real prices never do. The four months was the noticing, and nothing in the pipeline did the noticing. A person did.

Some pages genuinely never state their convention, and no column trick recovers it. Store those as unresolved and let them be visibly missing. A gap gets a question; a wrong number gets used. Carrying the original text next to every parsed field also costs real storage, and at a few hundred million rows that is a bill I still choose to pay, because a smaller table of numbers I cannot verify is worth less than a larger one I can.

None of this widens what you are allowed to collect: public pages, the robots file honoured, a crawl rate that does not hurt the site, and an official feed preferred every time one exists. The rest of what I have written on pipeline design, storage and the proxy infrastructure underneath it is over here.

Get new guides and videos first — join the Telegram channel.

The first time I wired a language model into a scraper, I thought I had cheated the whole game. I pointed it at a product page, told it in plain words to read the page and give me the name, the price, and whether it was in stock, and it did, perfectly, on the very first try. No selectors, no fiddling with the HTML, no brittle rules to maintain.

Then I ran the same trick across ten thousand pages, and two things happened at once. The bill was frightening, and somewhere in the middle the model handed me a price that was never on the page at all. This piece is about scraping with language models: where they genuinely help, where they quietly hurt you, and how to use one without setting fire to your budget or your data.

A language model is not a scraper

The single idea most of the excitement skips past is this: a language model is not a scraper. It does not fetch a single page. It has no idea what a proxy is, it cannot rotate an address, it cannot obey a rate limit, and it will not keep you welcome on a site.

Every hard part of scraping, getting the page down cleanly and staying gentle while you do it, is completely unchanged. The model only touches the very last step, the reading of a page you have already fetched. It is a clever, expensive reader you bolt onto the end, and remembering that it is only the end is what keeps you sane.

What the old way looked like

To see why people got excited, remember what extraction used to mean. You open the page, find where the price lives, and write a rule that says the price sits inside this element, at this position in the HTML. That rule is a selector, and it works right up until the site changes its layout, at which point your selector points at nothing and your scraper silently starts collecting blanks. Every operator has felt that particular pain.

The promise of the model is that you skip the fragile rule entirely and just ask, in plain language, for the thing you want.

Where the model genuinely wins

The honest answer to “where does that pay off” is messy, varied pages, the ones where no two look alike. If you are pulling product details from a hundred different shops, or job listings from fifty different boards, writing and maintaining a separate selector for every single site is a maintenance job that never ends. A model reads meaning rather than position, so it can pull the price from all hundred layouts without you hand crafting a rule for each. That is the real win, and it is a genuine one.

The second place a model earns its keep is turning prose into fields. Imagine a rental listing where the bedrooms, the floor area, and the year it was built are all buried inside a paragraph of free text, with no tidy element to point a selector at. A classic parser is helpless there, because there is nothing structural to grab. The model reads the sentence the way a person would and lifts the numbers straight out. Any time your data is trapped inside human writing rather than in the shape of the page, that is a job the model is good at.

Where the plain selector still wins

The classic parser is far from dead. If your target is a single site with a stable layout and you need a few million pages from it, a plain selector is basically free. It runs in a fraction of a millisecond, it costs nothing once written, and it gives you the same answer every single time.

Running a language model against every one of those millions of pages is slow, and it costs real money on each and every call. You never pay a model to read what a one line selector already grabs perfectly.

The cost is not a footnote

Every page you hand to a hosted model is charged by the token, both the text going in and the answer coming out. A full HTML page is enormous once you count the tags, the scripts, and the styling, so you are paying to send the model a mountain of markup it does not even need. On ten thousand pages that stings. On ten million it is a budget line you will have to defend to whoever signs the cheques.

Worse, it is not a one time purchase: it lands again on every run, so a nightly job multiplies the bill by the number of nights. The parser costs you the afternoon it took to write and effectively nothing after that.

Strip the page before you send it

The trick that makes the model affordable is the single most important habit here: do not feed raw HTML to the model. Strip the page down first. Throw away the scripts, the styling, the navigation, the footer, all the boilerplate, and hand the model only the slice of text that could plausibly hold your answer.

A smaller input means a smaller bill and, as a bonus, better accuracy, because the model is not hunting through a haystack. The parser and the model are partners: the parser trims the page, and the model reads what is left.

The real danger is silent hallucination

The thing that keeps me up at night is not the cost, it is the lying. A selector that breaks fails loudly. It points at nothing, you get an empty field, and you notice something is wrong. A model that is confused fails quietly. It hands you a clean, confident, perfectly formatted answer that simply is not true: a price it inferred, a date it guessed, a value that appears nowhere on the page.

That plausible, silent wrongness is far more dangerous than an honest error, because it slips into your data looking exactly like the real thing.

How to keep the model honest

You never trust the model blind, and there are a few concrete defences. First, force the output into a strict shape, a fixed set of fields and nothing else, so the model cannot ramble or improvise. Second, validate every field afterward with plain code: is the price actually a number, is the date a real date, does the link sit on the domain it should. Third, and most important, tell the model to copy only what is present and never to reason about what is missing.

The more room you give it to think and infer, the more it invents. The safest use by a wide margin is pure extraction, not clever guessing.

Which model, and where it runs

The big hosted models are the most capable and the quickest to start with, but you send every page off to someone else and pay for the privilege each time. A smaller model you run on your own hardware costs more to set up, yet it is dramatically cheaper at volume and keeps your data in house. The right pick is decided by how many pages you have, not by which model is fashionable this quarter.

The hybrid is the real answer

If you take one practical pattern from all of this, take the hybrid. Write a cheap, boring parser for the ninety percent of your pages that are regular and predictable, and keep the language model in reserve as the fallback for the awkward remainder the selectors cannot handle, plus those free text fields.

You get the speed and near zero cost of parsing across the bulk of the work, and you only pay for the model on the small slice where its flexibility is actually worth the money. That split is how real production systems use this, quietly, without any drama.

There is one more use worth naming: let the model write the parser instead of being the parser. Point it at a new site once, at build time, and ask it to work out the selectors. Then run those cheap, fast selectors forever after. The model does the one off thinking, the parser does the millionfold repeating, and the running cost stays on the floor.

It does not change the front of the pipeline

Here is the part that is easy to forget in all the AI talk: none of this touches the front of your pipeline. The model reads the page after you have fetched it, which means you still have to get the page down cleanly in the first place, and that is the same job it always was. The proxies, the rotation, the gentle pace, the rate limits, every bit of it works exactly as before whether a selector or a model reads the result at the end.

A language model will never get you past a block. It only makes sense of a page you were already able to fetch. That is where a healthy proxy setup still does the heavy lifting, because the smartest reader in the world is useless if you cannot get the pages in front of it. I run my own collection on a managed mobile proxy pool for that reason. The full written walkthrough, the stripping step, the strict shape I force on the output, the validation I run afterward, and the pool that fetches the pages, live at dataresearchtools.com.

The honest limits

A cleverer reader does not change the rules one bit. It grants you no new permission to take anything. Public data is still the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how smart the thing at the end of the pipeline is. The model only helps you understand pages you were already allowed to fetch.

The simple version: a language model is a reader you attach to the end, and it shines on messy, varied pages and on data trapped inside human writing. It is the wrong tool for one stable site with millions of regular pages, where a plain selector is faster, free, and honest. Strip the page, force a strict shape, validate everything, and never let the model guess at what is not there. The fetching, the proxies, and the manners are still entirely your job.

Get new guides and videos first — join the Telegram channel.

The first scraper I ever ran at real scale did fine on a small job and then fell flat on a big one. On ten thousand pages it was quick and quiet, finished before I finished my coffee. Then the job grew to a few million pages, I pointed the same single program at it, and it simply could not keep up.

By the time it reached the end of the list, the pages at the start had already changed. I was collecting a snapshot that was stale before it was even complete. That is the moment a scraper stops being a program and has to become a system. This piece is about how you make that jump: from one program on one machine to a coordinated fleet that finishes the big jobs while they still mean something.

Distributed is coordination, not force

The first thing to understand is that going distributed is not about hitting sites harder. It is about spreading an honest workload across many small workers so it finishes in time, without any single machine, or any single address, carrying the whole thing.

The mental shift is this: stop thinking of your scraper as one clever program, and start thinking of it as a system of small, identical, forgettable workers all pulling from a shared list of work. The unit of work is a single URL to fetch. The heart of the system is a queue that holds those URLs. Everything else, the workers, the storage, the proxies, hangs off that one idea.

Why one machine hits a wall

If you already went async and made a single scraper fast, you might wonder why one box is not enough. The honest answer is that a single machine has ceilings you cannot buy your way past forever. It has only so much memory and so many open connections. It leaves from a narrow slice of addresses. And worst of all it is one point of failure, so when it falls over at three in the morning, the whole job stops.

You can scale up, buy a bigger machine, and that helps for a while. Past a certain size you scale out instead, adding more modest machines, because ten ordinary workers beat one heroic one that you cannot replace when it dies.

The shared queue is the frontier

The piece that makes all of this possible is the shared queue, sometimes called the URL frontier. Instead of a list living inside one program’s memory, you put the URLs to fetch into a real broker that every worker can reach, something like Redis, RabbitMQ, or Kafka.

One side of the system discovers work and pushes URLs in. The other side, the workers, pull URLs out and fetch them. That single move, lifting the queue out of one process and into a shared broker, is what lets you run twenty workers, or two hundred, against the same pile of work without them tripping over each other.

Make your workers stateless

The workers themselves should be as dumb as you can make them, and that is a compliment. An ideal worker does one boring loop: pull a URL from the queue, fetch the page, parse it, write the result, ask for the next one. It holds nothing important in its own memory.

Because each worker is stateless and identical, you can start ten more when the job is big, kill any one of them mid run, and restart it, and you lose nothing but the single page it was holding. That disposability is the entire trick to scaling out.

The workers can afford to be forgetful only because the memory lives somewhere shared. All the state that actually matters, which URLs you have already seen, which ones are done, which ones failed and how many times, sits in a central store that every worker reads and writes: a database or a shared key store off to the side. A worker is just a temporary pair of hands. The real record of the job lives in the middle.

Deduplicate where everyone can see it

The moment you have many workers, a new problem appears that a single program never had: duplication. Two workers can easily discover and fetch the very same URL, wasting requests and polluting your data with copies. You cannot solve this inside one worker, because it cannot see what the others are doing.

You solve it centrally, with a shared record of everything already seen, checked before a URL is ever added to the queue. For huge crawls, people reach for a compact structure like a Bloom filter to hold that seen set cheaply. The point is simple: dedup where every worker can see the answer, not in any single worker’s head.

Coordinate proxies across the whole fleet

Here is the part that ties straight back to running real infrastructure: the proxies. On one machine your address pool was one program’s concern. Across a fleet it becomes a shared resource that every worker draws from, and if you are not careful, each worker politely obeys a per host limit on its own while twenty of them together hammer one site into the ground.

The caps have to be global, not per worker. Your rate budget for any single host, and your rotation across the pool, has to be enforced across the whole fleet, so the site sees one reasonable stream of traffic no matter how many machines are behind it.

Bounded queues and back pressure

You also have to stop your own discovery stage from drowning the system. A crawler that finds links faster than the workers can fetch them will happily shove ten million URLs into the queue and bury the broker.

The fix is a bounded queue and back pressure. You cap how much work is allowed to wait at once, and when the queue is full the discovery side pauses instead of racing ahead. The workers pull at whatever pace they can actually sustain, and the front of the system is forced to match the back. A fast producer must never be allowed to outrun a careful fetcher.

Design for failure, because it is the weather

At this scale failure stops being an event and becomes the weather. With a hundred workers running, one of them is always dying, and you design for that from the start.

The key rule is that a URL a worker took but never finished must find its way back into the queue. Good brokers give you this directly: a message stays reserved but invisible while a worker holds it, and only disappears when the worker confirms success, so a crash quietly returns that URL for someone else to retry. Because a page can therefore be fetched more than once, your writes have to be safe to repeat, so a retry just overwrites the same clean record instead of creating a mess.

Split the pipeline into stages

A scraper that scales well is usually split into stages joined by queues, rather than one worker doing everything from end to end. Fetching is one stage, parsing is another, storing is a third, and a queue sits between each pair.

These stages have very different appetites. Fetching is mostly waiting on the network, while parsing a heavy page can chew real processor time. When they are separate, a slow parser cannot stall your fetchers, and you can scale each stage on its own, running many light fetchers feeding a smaller number of heavier parsers, or the other way round.

Watch the queue depth

You cannot watch a fleet by eye, so it has to report on itself. The numbers worth keeping in front of you are the queue depth, the throughput in good pages per minute, the block rate across the fleet, and the health of the workers themselves.

The most telling one is queue depth over time. If the pile of waiting work is growing, your workers are falling behind and you need more of them or a gentler discovery stage. If it is draining steadily, you are keeping up. That one line is the cockpit for the whole system.

Do not over engineer it

None of this requires an enormous cluster on day one. A single broker and a handful of worker processes spread across two or three ordinary machines will take you an astonishingly long way, into the millions of pages. Add more workers when the queue depth tells you to, and only add more machines when a single one is genuinely full.

And most jobs never need any of this. If a single well written async scraper on one box finishes your job inside the window you have, stop right there. Distribution has a real cost: a broker to run, workers to deploy, state to keep consistent, and failures that are subtler than a crash on one machine. Earn the complexity only when one machine genuinely cannot keep up.

The honest limits

Scale changes your throughput, not your permission. Running a hundred workers does not entitle you to anything a single script could not already take. Public data stays the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how many machines you have behind it.

Spreading your fetching across many workers multiplies your footprint on every site you touch, so a shared pool that rotates cleanly and heals itself between passes is what keeps a fleet quiet and welcome. I run my own heavy jobs on a managed mobile proxy setup for exactly that reason. The full written walkthrough, the queue shapes, the worker layout, and the pool I actually use, live at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

A rolling deploy took about six hours to work through the fleet. For those six hours half my workers ran the new parser and half ran the old one, both writing into the same table, and not one row recorded which of the two had produced it.

I found out because somebody asked why a single day of data had two clearly separated distributions sitting inside it. Not drift. Not a spike. Two modes, produced on the same date, by the same job, under the same name.

I run mobile proxy lines and production scrapers out of Singapore. The first question I ask now when I inherit a pipeline is whether anyone can pick a row and say which code wrote it. Almost nobody can, and the people who cannot are usually the ones telling me their pipeline is fine.

The field that changes meaning without changing shape

Here is the failure this prevents, and it is not a crash.

I had a boolean called in_stock, originally set by checking whether an add to cart button existed on the page. Somebody on my side rewrote that check to look for a small availability badge instead, because the button rendered even on sold out listings and the badge looked more precise.

It was more precise. It was also warehouse specific, so listings available from a second warehouse now came back false.

The column stayed boolean. The fill rate did not move. Nothing went null, no type changed, no request failed. The true rate dropped by about nine points over a week, which is well inside the range that catalogue seasonality produces on its own. Every monitoring check I had passed.

So the table now held one column carrying two different definitions, separated by a deploy that left no trace anywhere in the data. Anyone querying across that boundary was silently comparing two things.

This is a distinct problem from the target site changing on you, which is worth monitoring for on its own and is a separate discipline with separate tooling. It is also distinct from rerun safety. A properly idempotent write path stops a retry from duplicating rows, and it is worth building, but it will happily let two different code versions upsert into the same key without either of them leaving a signature. Idempotency protects the row count. It does nothing for the row’s meaning.

Three columns is most of the fix

Stamp every row with the version of the code that produced it and with when the page was fetched.

In practice that is a short git commit hash, seven characters, read from the repository at build time and baked into the image. Then two timestamps rather than one: fetched_at for when the response came back, parsed_at for when the extraction ran over it.

Those two are identical on the first pass and diverge the moment you reparse anything, which is exactly the situation you want them for. fetched_at is a fact about the world. parsed_at is a fact about your code. Collapsing them into a single “created_at” throws away the distinction at the moment it starts to matter.

If your extractor has fallbacks, record which branch fired. A primary selector and a rescue selector produce rows that look identical and deserve different levels of trust, and once you have that column you can also see the rescue path quietly climbing from two percent of pages to thirty, which is an early warning you would otherwise have to go looking for.

Two things I would insist on. Never let the version be a constant somebody edits by hand, because it will eventually be wrong and a wrong stamp is more dangerous than a missing one. And make the deploy identity granular enough to survive a rolling release, which was my six hour problem: the version was correct on every row, I simply had not been recording it yet.

The usual objection is storage. Three columns on a hundred million rows sounds heavy until you notice the row already contains a url. A repeated seven character hash compresses to near nothing in any column store, and against a row that already carries text fields it is not measurable.

Keep the response

This is the position I will defend: keeping the raw response is the highest value habit in scraping, and it is the first thing that gets cut when someone looks at the storage bill.

The reason is not sentimental. A scraper is not a query you can run again. The source is somebody else’s website, it keeps no history for you, and the page you fetched this morning may not exist by Thursday. Prices move, listings sell, sites redesign, whole catalogues get pulled. Every fetch is the only opportunity you will ever have at that page in that state.

Which means the parsed row is a lossy derivative of something that no longer exists. If your parser had a bug, the raw response is the only thing standing between “we can repair this” and “those numbers are permanently wrong and I have to say so”.

Store the headers with the body, not just the html. Content type, redirect chain, declared encoding. A good share of parsing bugs are encoding bugs, and the evidence for those lives entirely in headers you did not keep.

Where and how you physically store all of that is its own decision, with real tradeoffs around file counts and formats, and a separate question from this one. All that matters for provenance is that the raw capture and the parsed row exist as two artifacts, and that the row carries a pointer to its capture. A content hash or an object key. That pointer is the chain.

The honest version of the retention argument: raw html is bulky and mostly boilerplate, and cold storage with a lifecycle rule at thirty or ninety days is a fair compromise. I run exactly that on most targets.

Just be clear about what the rule buys. You can only reparse the window you kept, so the window has to outlast your detection time. Mine has been badly wrong at least twice. The worst case was a bug a client’s finance team caught during a quarterly review, roughly eleven weeks after it started, against a thirty day retention rule. Nine of those weeks were unrecoverable. I widened the rule that afternoon.

The schema is a contract

The third piece is the one people skip, because it is process rather than code.

Your output schema is a contract with whoever consumes it, and it needs a version separate from the code version. The code version changes on every deploy, including the dozens that change nothing about the output. The schema version changes only when the shape or the meaning of what you emit changes, which is rare and always significant to a downstream consumer.

An integer column, and a plain file in the repository saying what each number means with a date against it. That is the whole implementation.

The discipline that goes with it is the part worth actually holding: never redefine an existing column, add a new one. When the availability logic changed, the correct move was in_stock_badge alongside in_stock, not a quiet redefinition. Old rows keep the old field, new rows populate both for a while, and the boundary is visible to anyone querying instead of buried in a commit message they will never see.

It produces an ugly schema. I would take an ugly schema over a column that means two things depending on the date.

What this actually buys

Three things, and they are the reason any of the above is worth the trouble.

You can reparse history. Fix the parser, run it back over the stored captures, write corrected rows under the new version, keep the old ones. Then diff the two sets, which turns “something was wrong for a while” into “forty one thousand rows were affected, the median error was eleven percent, here they are”. That is a survivable conversation. The other one is not.

You can attribute an anomaly to a deploy instead of arguing about it. Group your metric by code version rather than by date. If the step change sits exactly on a version boundary it is yours. If it sits somewhere else entirely, the site changed or the market did, and you have ruled out the expensive explanation in about ten minutes.

And you can answer the question months later. Which code, which fetch, which page, on what day. That question always arrives from someone senior, about a figure that has already left the building.

The one I got wrong

For the first year I stamped the code version and only kept a single timestamp, updated on write.

Then I reparsed a batch to fix a field, and the reparse overwrote that timestamp on every row it touched. I had turned “this page was fetched on the fourth” into “this row was written on the nineteenth” across several hundred thousand rows, and there was no way back, because I had never stored the original separately. The captures still existed and the fetch times lived in their metadata, so I recovered most of it over a slow weekend. On the targets where the captures had already aged out, I did not.

Provenance has a limit too, and it is worth being blunt about. Knowing which code produced a row tells you nothing about whether that code was right. You can have immaculate provenance on a completely wrong number. What you have bought is the ability to discover it later and repair it, which is not the same as protection.

None of it changes what you should be collecting either. Public pages, the robots file honoured, a crawl rate that does not hurt the target, and an official api or a bulk feed used whenever one exists. Good provenance on data you should not have taken is just a tidier record of the decision.

The pipelines I have seen survive a change of owner are the ones where a new person can take a row at random and walk it back to the response it came from. Everything else is a box producing numbers that people have agreed to trust. The rest of what I have written on pipeline design, storage and the proxy infrastructure underneath it is over here.

Get new guides and videos first — join the Telegram channel.

I once bought a fresh batch of proxies on a Friday, pointed my scrapers at them, and for about a day it was beautiful. The jobs flew, the blocks were near zero, everything came back clean. By the middle of the next week, half of those addresses were quietly useless. Some were timing out, some were handing back CAPTCHA pages, and my overall block rate had crept up so slowly that I almost missed it.

Nothing had crashed. The pool had simply aged, and nobody was watching it age. That is the whole reason this piece exists: a proxy is a living thing you manage, not a static line in a config file.

A proxy pool is a fleet, not a receipt

The mistake I see most often is treating proxies as a one time purchase. You buy a list, paste it in, and assume it keeps working forever. It does not. Addresses get flagged, connections go stale, providers recycle their ranges, and the sites you visit keep learning.

The better mental model is a fleet. Think of the way a taxi company thinks about its cars: some are on the road earning right now, some are in the garage resting, and a few are broken and waiting to be retired. Your job is not to buy the fleet once. It is to always know which addresses are working, which are struggling, and which are done, and to keep the working set topped up as the others fall away.

What healthy means for one proxy

Health is not one thing. For a single proxy there are three separate questions, and they fail independently.

• Can you reach it? Does it connect, or does it refuse and time out?
• Is it fast enough? An address that answers but takes fifteen seconds is dragging your whole run down.
• Is it still trusted by your target? This is the one people forget. An address can be perfectly reachable and perfectly fast while being completely burned in the eyes of the site you care about, handing back a block page on every request.

Health is all three at once. A pool full of fast, reachable, flagged addresses is a pool full of dead weight that looks alive.

Passive and active health checks

There are two ways to learn the health of your pool, and you want both.

Passive checks watch what your real traffic is already telling you. Every request you send through a proxy is a free health check, because it came back with good data, or it came back slow, or it came back as a block. You are collecting that signal anyway, so record it against the proxy that carried it.

Active checks probe on a schedule. You send a cheap test request through each proxy every so often, to a target you control or a neutral endpoint, just to ask whether it is still alive before a real job leans on it. Passive tells you how the pool behaved; active tells you the pool is ready before you spend a run finding out the hard way.

The signals a proxy is going bad

The failure signals are not all the same, and the difference tells you what is actually wrong.

• Connection refused or repeated timeouts mean the proxy itself is down or unreachable. That is a plumbing problem.
• A slow crawl upward in latency means it is overloaded or dying.
• Rising CAPTCHA pages and soft blocks from a proxy that still connects mean that address specifically has been flagged by your target.

That last one is the dangerous signal, because the proxy looks fine at the network level. Only the body of the response gives it away, which is exactly why you inspect responses and do not trust a 200 alone.

Health is per target, not global

Here is the idea that changes how you build the whole thing. Health is not global, it is per target.

An address that is stone cold burned on one big site can be completely clean on another that has never seen it. So a proxy is not simply good or bad, it is good for this site and bad for that one. If you track health as a single yes or no per address, you will throw away perfectly useful proxies just because one aggressive target flagged them. Track the pairing instead, this proxy against that site, and you get far more life out of the pool you already paid for.

Quarantine, do not delete

When a proxy starts failing, the instinct is to delete it, and that is usually too harsh. Addresses recover. A site that flagged an IP an hour ago may forget about it in a day, especially on mobile and residential ranges where the same address gets handed to real people all the time.

So instead of deleting, quarantine. Pull the suspect address out of active rotation, let it rest for a while, then send a quiet probe later to see if it has come back to life. You cool it down rather than throwing it away, and a large share of the pool you would have binned comes back on its own.

Warm new proxies in

The opposite move matters just as much. Do not slam a fresh proxy with your full load on the first request. A brand new source suddenly firing a hundred requests a minute at a site is its own kind of red flag.

Bring new addresses in gently. Give them a smaller share of the traffic at first, watch how the target treats them, and only promote them to full duty once they have behaved. A slow warm up costs you a little speed on day one and saves you a burned address on day two.

Measure the pool, not just the proxy

Zoom out from the single address to the pool as a whole. You need a few numbers that tell you its overall health at a glance:

• How many addresses are usable right now, versus quarantined, versus dead.
• The pool wide success rate and block rate over the last hour.
• The latency spread, not just the average but the slow tail.

Watch those together and you stop reacting to individual proxies and start managing a fleet. You can see the pool getting tired days before it would have taken a job down with it.

Health decides what rotation is allowed to use

Health and rotation are two different jobs that people blur together. Rotation is how you spread requests across addresses so no single one carries a burst. Health is deciding which addresses are even allowed into that rotation in the first place.

The cleanest way to think about it: your health layer maintains a pool of currently trusted addresses, and your rotation layer only ever picks from that trusted set. A burned proxy should never be handed to the rotator. It should already have been pulled out.

Datacenter and mobile fail in different shapes

The kind of proxy you run changes how this plays out. Datacenter addresses tend to fail in blocks: a whole range gets recognized and flagged together, so when one goes, the neighbors often go with it.

Mobile and residential addresses behave very differently, because they are shared with real users and rotate naturally. An address that looks burned this minute can be genuinely clean an hour later through no effort of yours. That natural recovery is a big part of why I lean on mobile pools for the heavy, long running jobs: the pool heals itself between passes instead of just draining away.

Retire and replenish

No matter how well you tend it, a pool leaks. Some fraction of your addresses will burn out for good every week, and if you are not topping the pool up, you are slowly starving your own scrapers.

Build the churn into the plan. Budget for replacement addresses the way you would budget for any consumable, keep an eye on how many usable proxies you have left, and refill before you hit the floor, not after your jobs have already started failing. A pool is a stock you keep replenishing, not a tank you fill once and forget.

Automate the whole loop

A human cannot watch hundreds of addresses by hand, so the whole loop should run on its own:

• Every request feeds its outcome back into a health record for that proxy and target pairing.
• Addresses that cross a failure threshold get quarantined automatically.
• Quarantined ones get retested on a timer and either promoted back or retired for good.
• The pool reports its own headline numbers, so you only get pulled in when the whole fleet is trending down, not for every address that has one bad minute.

The honest limits

A healthy pool is about reliability, not permission. Keeping your addresses clean and rotating makes your collection steadier; it does not entitle you to take anything you could not take before. Public data stays the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how big or how healthy your pool is.

I run this infrastructure in production every day, and a well managed pool is the difference between a job that quietly holds together for months and one that rots in a week while every dashboard stays green. The full written picks I actually use, the exact health checks I log, the quarantine timers, and the mobile pool I run on, live at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

The first scraper I was ever proud of ran all night to do a job that should have taken twenty minutes. It walked a catalog one page at a time: ask for a page, wait for the answer, save it, ask for the next one. Patient, polite, and painfully slow. When I finally looked at where the time actually went, the machine was doing almost nothing. It was sitting idle, waiting for a server on the other side of the world to answer, thousands of times in a row.

That is the whole story of scraper performance, and it is why concurrency is the single biggest lever you have. This is how you fetch many things at once so your scraper stops spending its life waiting.

Why a scraper is slow

A slow scraper is almost never a slow computer. The bottleneck in scraping is hardly ever your processor working too hard. It is your program standing still, waiting on the network.

You send a request, and then there is a long pause while it crosses the network, the server thinks, and the answer travels back. During that pause your code is just holding its breath. If you do one request at a time, you sit through every one of those pauses back to back, and they add up to the whole night. Concurrency is the trick of overlapping those waits, so that while one request is out waiting, twenty others are out waiting at the same time.

Concurrency is not pulling harder

Draw a clean line here, because this is where people get into trouble. Going concurrent is not the same as pulling harder. The goal is not to hit the site with more force, it is to stop wasting the idle time you already had.

That distinction matters. The moment concurrency turns into a flood aimed at one site, you are back in the world of rate limits, getting throttled and blocked. Done right, concurrency makes you faster without making you louder to any single target.

The three models

There are really only three ways to run many requests at once, and each one fits a different bottleneck.

Model	What it is	Best for
Async event loop	One thread juggling thousands of waiting requests	Pure fetching, the lightest option
Threads	A pool of workers, each doing a blocking request	The simple case, moderate concurrency
Processes	Several copies across CPU cores	When parsing, not fetching, is the cost

The async event loop is built for exactly this job. Picture a single thread that never sits still. It fires off a request and, instead of waiting for the answer, immediately turns to the next request and fires that one too, keeping hundreds or thousands of them in the air. When an answer comes back, the loop picks it up and deals with it. Because the work of a scraper is waiting rather than computing, one core running one loop can happily manage thousands of open requests. In Python this is the world of asyncio and the async HTTP libraries.

Threads are the simpler picture, and for a lot of scrapers they are more than enough. You keep a pool of, say, twenty workers, and each one does the plain blocking thing. You may have heard that Python threads cannot really run at the same time because of the global interpreter lock, and for heavy computation that is true. But it does not hurt you here, because your threads are waiting on the network, not computing, and a thread that is waiting steps aside and lets the others run.

Processes are the heavy option, and you reach for them only when your bottleneck moves to parsing: huge pages, heavy pattern matching, tangled documents that take real processor time to pick apart. For most scrapers the parsing is cheap and the waiting is the whole cost, so you rarely need processes just to fetch.

Bounded concurrency is the number that matters

The beginner move, the day you discover you can fetch many at once, is to fire all ten thousand URLs at the same instant. Do not. That is a self inflicted disaster. You open ten thousand sockets, exhaust your own memory and file handles, your machine falls over, and the site sees a wall of traffic arrive in one heartbeat and blocks you on the spot.

What you want is bounded concurrency: a fixed number of requests allowed in flight at once, say ten or twenty, with the rest waiting their turn. A semaphore, or a worker pool of a fixed size, is how you enforce it. This one number is the difference between a fast scraper and a self inflicted outage.

Reuse your connections

Before you even touch concurrency, there is a free speedup most people leave on the table: reusing your connections. Every time you open a brand new request from scratch, you pay for a fresh TCP and TLS handshake, a little round trip of setup before any real work happens.

If you reuse one session and let it keep the connection alive between requests, you skip that setup on every call after the first. It makes you faster, and it is genuinely lighter on the site, fewer connections to accept. One session, reused, is the cheapest win there is.

Cap concurrency per host

Here is the subtlety that separates a scraper that survives from one that gets banned in an hour. Your concurrency should be counted per site, not as one global number. Twenty requests in flight spread across twenty different sites is gentle, two at each. Twenty requests in flight all aimed at one site is a hammer.

So cap how many you run against any single host, keep that per site number low, and let your overall speed come from working many hosts at once. That is how you stay fast in total while staying polite to each individual target.

Where proxies come in

Concurrency and your address are linked. If you run twenty or fifty requests at once all leaving from a single IP, you have concentrated a burst of traffic on one address, and that is exactly the pattern that gets an address flagged and shown the door.

The fix is to spread your concurrent requests across a healthy pool of addresses, so no single one carries the whole burst. Concurrency multiplies your footprint, and a good proxy pool is what distributes it. This is where a clean mobile proxy setup earns its keep on heavy concurrent runs.

Timeouts, queues, and retries

Once you have many requests in flight, three details keep the whole thing from quietly falling apart.

Timeouts stop being optional. If one host goes slow and never answers, that request holds its slot forever, and enough of them will occupy every slot in your fixed pool. Your scraper looks like it is running, but nothing is moving. Give every request a hard timeout so a hung connection gives up and frees its slot.

A bounded queue keeps memory sane. You have one side producing URLs and another side of workers consuming them. If the producer races ahead, it loads a million URLs into memory before a single one is done. A bounded queue applies back pressure, so a fast producer cannot drown a careful fetcher.

Retries have to share the budget. A failed request should try again after a short backoff, but a retry is still a request and still costs an in flight slot. Count retries against the same concurrency budget, or a bad patch where everything is failing will silently double your load at the worst possible moment.

Measure throughput, not requests per second

Raw requests per second is the wrong number to chase. The number that matters is useful throughput: good rows or good pages collected per minute, measured right next to your block rate.

It is easy to turn concurrency up, watch requests per second climb, and feel great, while your block rate climbs right alongside it and half those requests come back as block pages. That is not progress, it is going faster into a wall. So tune it like an experiment: start low, raise the limit while real throughput climbs and blocks stay flat, and settle in just under the point where throughput plateaus or blocks tick up. Every site has its own sweet spot, and the only honest way to find it is to measure your way there.

The honest limits

Faster is not the same as allowed more. Concurrency is about not wasting the idle time you already had, never about overwhelming a site into giving up more than it wants to. Public data stays the target, the robots file still gets respected, the site’s terms still mean what they say, and a gentle pace to each host still holds no matter how many hosts you work at once.

I run this in production every day, real scrapers pulling real volume on a schedule, and getting the concurrency model right is the difference between a job that finishes before breakfast and one that runs all night and still gets blocked. The full written picks I actually use, the exact limits, the async fetch loop, and the per host caps, live at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.