Category: Uncategorized

The two costs of a scrape

Every scrape has two separate costs baked into it: the cost of getting the bytes off the target site, and the cost of turning those bytes into structured data. Most people who are new to scraping treat these as one step, because in a quick script they usually are: request the page, run a parser on the response, write the result, move on. That works fine for a hundred pages. It falls apart at scale, because the fetch is the expensive, risky, rate-limited part, and the parse is the cheap, deterministic, endlessly repeatable part. Once you notice that asymmetry, you stop treating them as one operation.

Fetching costs proxy bandwidth, which on residential or mobile proxy pools is often billed per gigabyte and can be the largest recurring line item in a scraping operation. It costs request budget against a target that’s watching request volume per IP, per session, or per account. It costs time, since a crawl over millions of URLs with polite delays and rotation can run for days. And it carries risk: every request is a chance to trip a rate limit, get served a challenge page, or burn a proxy that was working fine a minute ago. Parsing, by contrast, is just CPU. You can run a parser against the same stored response a hundred times while iterating on a selector, and it costs nothing but compute time.

Splitting fetch from parse

The fix is architectural, not clever: store the raw response the moment you fetch it, before any extraction happens. The fetch stage’s only job is to get the page and write it somewhere durable, unmodified. The parse stage reads from that store, not from the network, and produces the structured output. They can run on completely different schedules. A crawl might run once a day; the parser that turns that day’s HTML into rows might run a dozen times in an afternoon while someone is debugging a broken selector.

This sounds obvious once you say it out loud, but a surprising number of pipelines skip it because storing raw HTML feels wasteful compared to storing the clean JSON you actually wanted. The tradeoff only looks that way until the first time a parser bug ships to production and you realize the only way to fix the corrupted output is to re-crawl a million pages you already had.

What actually goes in the archive

What you store matters as much as the fact that you store it. At minimum, keep the response body, the final URL after redirects, the HTTP status code, the response headers, and a timestamp for when the fetch happened. If the fetch went through a proxy, record which one, or at least which proxy pool and IP class, since that’s often the fastest way to explain why one batch of pages looks different from another.

For JavaScript-heavy pages fetched with a headless browser, store the rendered DOM after execution, not just the initial response. The headless render is the expensive part of that fetch, often several times slower and heavier than a plain HTTP request, so it’s the render output you want to be able to reparse for free, not just the raw markup that arrived before the page’s scripts ran.

It’s also worth storing pages that come back wrong. A response that’s actually a CAPTCHA page, a block page, or a soft-404 dressed up as a 200 is still worth keeping, tagged as such. Knowing that a batch of “successful” fetches were actually challenge pages is useful for tuning a crawl, and you can’t tell that after the fact if you threw the body away because the status code looked fine.

Why parser bugs stop being crawl problems

Once the raw HTML is sitting in storage, a parser bug becomes a code fix and a rerun, not a fresh crawl. Say a site restructures a price block and your extraction starts pulling the wrong number for a subset of pages. Without an archive, fixing that means re-fetching every affected page, which means burning proxy bandwidth and request budget a second time for content you already had. With an archive, you patch the parser, point it at the stored batch, and regenerate the output in minutes.

This matters more than it sounds like on paper, because parser bugs are common in a way network failures aren’t. Sites change markup, add or drop optional fields, switch from server-rendered to client-rendered blocks, or vary their layout by region or A/B test. A selector that worked against your sample set will eventually miss something in the full corpus. Being able to iterate on extraction logic against a fixed, already-paid-for dataset instead of a live target is the difference between a ten-minute fix and a multi-day re-crawl.

Backfilling new fields without a new crawl

The same mechanism pays off when requirements change rather than break. If a project decides it now needs a field nobody thought to extract three months ago, and you kept the raw pages from that period, you can backfill the new field by reparsing the archive instead of explaining to whoever’s paying the proxy bill why you need to re-crawl three months of history. This only works if the field was actually present in the page at fetch time, obviously. Reparsing can’t recover data that was never in the response, and it can’t reflect anything that’s changed on the live site since the page was captured. It’s a way to get more out of a fetch you already paid for, not a substitute for fetching again when you need current data.

The tradeoff: storage isn’t free, and stored pages go stale

None of this is free. Raw HTML is heavier than the structured output you eventually derive from it, sometimes by an order of magnitude, so storing it at scale means real object storage cost and lifecycle decisions. A common pattern is to compress bodies with gzip on write, which HTML compresses well, and to set a retention window: keep raw pages for a defined period, say enough to cover a couple of parser iteration cycles, then prune down to the parsed output once you’re confident the extraction is stable. What you keep forever is the structured data; what you keep temporarily is the raw archive that let you get there safely.

The other real limit is staleness. A stored page is a snapshot of the target site at the moment it was fetched. Reparsing it a week later gives you a better extraction of that same snapshot, not fresh data. If a price, a stock count, or a headline has changed on the live site since you fetched, no amount of reparsing recovers that, only a new fetch does. It’s worth being explicit about this distinction internally, because it’s easy for “we can reparse instead of refetching” to quietly turn into “we don’t need to refetch,” when the two are only interchangeable for fixing how you read old data, not for keeping current with a moving target.

Debugging with the archive

There’s a secondary use for a raw archive that has nothing to do with backfilling: it’s the fastest way to figure out what actually changed when a scraper that worked yesterday breaks today. Instead of guessing at what a target site did differently, you can diff a newly fetched page against the last stored version and see the structural change directly, whether that’s a renamed class, a moved block, or a switch to a different rendering path entirely. That diff is usually a five-minute investigation. Without an archived baseline, the same investigation means re-fetching a known-good page from history, which you may not be able to do at all if the site has moved on since then.

Where this fits in a pipeline

In practice this means a scraping pipeline is really two systems joined by a store: a fetcher that’s optimized for reliability, proxy rotation, and staying under a target’s rate tolerance, and a parser that’s optimized for iteration speed and correctness. The fetcher writes once and moves on. The parser reads as many times as it needs to, against data that’s already paid for. That separation is what makes it possible to fix extraction bugs, add fields, and debug breakage without treating every mistake as a reason to go back and hit the target site again.

If you’re building or auditing a scraping pipeline and want more on how the fetch and parse layers should be structured, proxy infrastructure that supports it, or how bot detection actually works from the defending side, take a look at what we’ve written on it at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

If you’ve ever inherited a scraping operation, you’ve probably seen the pattern: a folder full of scripts, each one named after a site, each one copy-pasted from the last one and then edited until it worked. Forty targets means forty scripts, forty sets of retry logic, forty places where a proxy pool got hardcoded, and forty places where nobody remembers why a particular sleep() call is there. It runs, until it doesn’t, and then nobody wants to touch it.

This is not a scale problem you hit at some huge number. It shows up around scraper five or six, when you notice you’re fixing the same bug in three different files because you fixed it in the first one and forgot to port the fix. By scraper twenty it’s not a nuisance, it’s the reason your team spends more time on maintenance than on new coverage.

The fix isn’t a framework you buy. It’s a decision about what belongs in code and what belongs in configuration, made early and enforced consistently.

What actually varies between targets

Look closely at forty scrapers pulling data from forty different sites and most of what’s “different” is really the same handful of concerns, expressed differently:

• Where the target data lives (a URL pattern, a sitemap, a paginated listing)
• How the page is structured (which selectors or JSON paths hold the fields you want)
• What counts as a rate limit or a block (a status code, a redirect to a challenge page, a specific response body)
• How often the site should be hit and from where

None of that is really “logic.” It’s data about a target. The actual logic, fetch a URL, retry on failure, parse a response, normalize a record, write it to storage, is identical across all forty. The mistake most teams make is letting the parts that should be data (a CSS selector, a rate limit, a base URL) get buried inside the parts that should be code (a Python file with control flow).

A shared core, not a shared script

The way out is a single core library that does the boring, repeatable 80%: HTTP fetching with retry and backoff, proxy assignment, response caching, structured logging, and a normalized output schema. Every scraper imports this core. None of them reimplement it.

What’s left per target is a small definition, not a program. In practice this is often a single config file or a thin adapter class with three or four methods: how to build the request, how to extract fields from a response, and how to detect that a response wasn’t actually valid data (a soft block, an empty result set, a login wall). That adapter might be forty or fifty lines. It is not a scraper in the sense of “a standalone thing that runs.” It’s a description that the core engine consumes.

This is the difference between forty codebases and one codebase with forty configs. When you fix a retry bug in the core, you fix it for all forty targets at once. When a proxy provider changes its authentication format, you change it in one place. The blast radius of any given change becomes predictable, which is the entire point.

Where config-driven scraping breaks down

Config-driven design works well right up until a target does something that genuinely needs procedural logic: a multi-step login flow, a page that renders fields conditionally based on earlier responses, a site where pagination depends on values returned in a previous page rather than a predictable URL pattern. Trying to express that as a static config file usually produces something uglier than just writing code, a config format that’s secretly a badly-designed programming language.

The honest answer here is to let the adapter be code when the target needs code, but keep it isolated. A target that needs a five-step interaction still shouldn’t touch retry logic, proxy selection, or storage. It writes its own extraction and flow logic and still calls into the shared core for everything else. You end up with a few “heavy” adapters among a majority of simple ones, which is a much smaller maintenance surface than forty independent scripts of unknown quality.

Proxy and fingerprint handling belongs in infrastructure, not in scripts

One of the fastest ways to end up with forty divergent codebases is letting proxy logic live inside each scraper. One script rotates proxies every request, another rotates every ten minutes, a third has a hardcoded IP that someone set for a one-off debug session and never removed. None of this is a scraping problem, it’s a networking problem, and it should be solved once, below the scraper layer.

In practice that means the core library asks a proxy manager for an egress point and gets back a working connection, without the scraper code knowing or caring whether that’s a residential IP, a datacenter IP, or a mobile carrier IP. Session stickiness (keeping the same IP for a sequence of related requests), rotation intervals, and per-target IP pools are configuration the proxy layer enforces, not something each scraper decides for itself.

The same goes for browser fingerprinting when a target requires a real browser context rather than a plain HTTP client. Fingerprint consistency (matching a real device profile across headers, TLS handshake, and JS-visible browser properties) is something anti-detect browser tooling and proxy providers build entire products around, and it’s worth understanding briefly why it matters: modern bot detection doesn’t just look at the IP, it correlates network-level signals with browser-level signals and flags mismatches between them. That’s infrastructure work, done by dedicated tooling and reviewed periodically, not something to reinvent per scraper. No proxy or browser fingerprinting setup makes a scraper undetectable, and treating it that way is how operations end up surprised when a target changes its detection logic.

Monitoring forty things instead of one

A second, quieter cost of the copy-paste approach is that nobody actually knows the state of forty independent scripts. Each one logs differently, if it logs at all. Failures show up as “the CSV is empty this morning” rather than an alert.

With a shared core, monitoring becomes a byproduct rather than a separate project. Every run reports through the same structure: rows extracted, request count, error rate, and whatever the target-specific adapter flagged as a soft failure. A dashboard that shows forty rows, one per target, with a last-successful-run timestamp and an error rate, tells you in ten seconds which targets are actually healthy. That’s not possible when every scraper is a bespoke script with bespoke logging, or none at all.

Respecting the target, at scale

Running many scrapers against many sites is also where it becomes easy to lose track of what’s reasonable. A rate limit that made sense for one script, hand-tuned by someone watching logs, doesn’t automatically transfer when it’s one of forty jobs sharing a proxy pool and a scheduler. Centralizing request pacing per target, rather than per script, is also how you avoid accidentally hammering a site because three different scrapers independently decided to hit it. It’s worth treating a target’s published terms of service and robots.txt as an input to that per-target configuration, not an afterthought, and steering clear of anything gated behind a login or paywall that wasn’t meant to be machine-read.

When a new codebase is actually the right call

None of this means one codebase forever. If a new class of target is structurally different enough, JavaScript-heavy single page apps versus static HTML, or a GraphQL API versus scraped markup, it can be worth a second core built for that shape of problem, with its own adapters. The failure mode to avoid isn’t “more than one codebase.” It’s a codebase per target, which is what happens by default when nobody makes the shared-core decision on purpose.

The real test of whether your scraping operation is a system or a pile of scripts is simple: when a target’s HTML structure changes, how many files do you have to touch, and how long does it take you to know it broke in the first place. If the answer is “one file” and “a dashboard told me this morning,” you’ve built infrastructure. If it’s “grep for the site name across the repo” and “a customer complained,” you’re still in script territory, no matter how many scrapers you’re running.

If you’re building out proxy infrastructure, orchestration, or fingerprinting defenses to support an operation like this, you can find more breakdowns like this one on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.

Most explanations of bot detection focus on fingerprinting: canvas hashes, WebGL renderers, TLS handshakes. Those matter, but they only tell a site what your browser is. Behavioural detection tells a site what you do once you’re inside the page, and it’s a lot harder to fake convincingly because it’s not a static value you can spoof once. It’s a pattern that has to hold up over the length of a session.

We run proxy infrastructure and scraping pipelines for a living, so we spend a fair amount of time on the defending side of this problem too, reading how detection vendors describe their own systems and watching how our own automated traffic gets scored. This is a rundown of the behavioural signals that actually move the needle, written from the infrastructure side, not a guide to defeating them.

Fingerprinting versus behaviour

Fingerprinting asks “does this client look like a real browser.” Behavioural detection asks “does this session act like a real person used it.” A headless browser with a patched user agent and a clean TLS fingerprint can pass every static check and still get flagged the moment it clicks a button in 8 milliseconds after page load, because no human reaction time is that fast.

The two layers are complementary. Static fingerprinting filters out the obvious automation. Behavioural analysis catches everything that got past the first filter, including automation running on real browsers with real fingerprints, which is most serious scraping traffic today.

Mouse and pointer movement

Human mouse movement is noisy in a specific way. It accelerates and decelerates non-linearly, overshoots targets slightly and corrects, and rarely travels in a perfectly straight line between two points. Detection systems that sample pointer events (mousemove, pointermove) build a trajectory and check it against that expected noise profile.

Scripted interaction tends to fail this in one of two ways. Either there’s no mouse movement at all before a click (a click() call fired directly on an element, which a real user physically cannot do without first moving a cursor there), or the movement is present but too clean: constant velocity, straight lines, or movement generated from a fixed set of waypoints that repeats across sessions. Some automation frameworks now inject randomised mouse paths specifically to defeat this check, and detection systems have in turn started looking at the distribution of randomness across many sessions from the same source, since a random-number generator has its own statistical fingerprint if you see enough samples from it.

Timing and reaction latency

Every interaction has a time cost attached to it in a human session: time to read a page before scrolling, time to locate a form field before typing, time between a page load and the first click. These intervals cluster around human reaction time, roughly a few hundred milliseconds at the fast end, and they vary session to session because people are inconsistent.

Automated traffic tends to either be too fast (form fields populated instantly, a click fired the same tick the DOM reports “interactive”) or suspiciously uniform (every page on a crawl spending almost exactly the same number of seconds before the next action, because the delay is a fixed sleep() call in the script). Both are visible from server-side timestamps alone, no client instrumentation required, which is why request cadence is one of the cheapest signals for a site to check and one of the first things worth respecting if you’re running any kind of automated pipeline against a site that permits it.

Scroll behaviour

Scrolling has physical properties that are annoying to fake well. Real scroll events come in variable-sized chunks tied to a trackpad, mouse wheel, or touch gesture, with momentum and deceleration. A script that jumps scrollTop straight to the bottom of the page, or scrolls in perfectly even increments at a fixed interval, produces an event pattern that doesn’t correspond to any real input device.

Some detection stacks also check whether scroll events correlate with what’s actually rendered: does the session appear to “read” content proportional to how long it dwelled at a given scroll position, or does it blow through a 3,000-word article in under two seconds. That second case is a common tell for content-scraping bots that render the page just enough to extract text and never intended to simulate reading it.

Form and input behaviour

Keystroke dynamics are one of the older behavioural signals and still one of the most reliable. Real typing has variable inter-key intervals, occasional corrections (backspace usage), and timing that differs from field to field based on familiarity, like an email address typed faster than a one-off comment box. A form filled by setting .value directly in the DOM, or by dispatching synthetic keydown/keyup events at identical intervals, doesn’t reproduce that variance.

Honeypot fields are the low-tech companion to this. A hidden input, invisible via CSS but present in the DOM, that a real user will never focus or fill, but that a script iterating over all form fields sometimes does. It’s not a behavioural signal in the strict sense, but it’s usually scored alongside the input-timing data because it catches the same category of naive automation.

Focus, blur, and tab behaviour

Real browser sessions generate a messy stream of focus and visibility events: tabs get backgrounded when a person checks something else, windows lose focus, visibilitychange fires when a laptop lid closes. A headless session driven end to end without ever losing focus, running in a single unbroken sequence of actions with no idle gaps, stands out precisely because it’s too tidy. People get distracted. Scripts don’t, unless someone has deliberately built idle time into them.

Session-level consistency

Individual signals are noisy and easy to get wrong in isolation, which is why most detection systems don’t rely on any single one. They score a session across many signals and look for internal consistency. A session with human-like mouse movement but robotic timing is still suspicious. A session that behaves perfectly on page one and then reverts to instant, uniform interactions on page five (common when a script only bothers simulating behaviour on the pages it thinks are checked) is a stronger signal than either page alone.

This is also where volume matters more than any single session’s quality. A detection system watching one browsing session has limited signal. A detection system watching ten thousand sessions from adjacent IP ranges, all with subtly correlated timing distributions or repeated mouse-path shapes, has a statistical case that no individual session can hide from. This is one of the reasons proxy and IP diversity gets talked about so much in scraping infrastructure: it’s not because a single IP triggers detection, it’s because correlated behaviour across a narrow IP range is itself a behavioural signal at the aggregate level.

What this actually means operationally

None of this means behavioural mimicry is pointless, and it doesn’t mean any particular tool or technique makes traffic undetectable, because nothing does. What it means practically for anyone running scraping infrastructure is that the reliable long-term approach is closer to good citizenship than to evasion: request at a pace a site can sustain, respect robots.txt and published rate limits, avoid scraping data a site’s terms explicitly prohibit, and treat detection as a signal that a target doesn’t want automated traffic rather than an obstacle to route around by force. Sites that publish an API generally want you to use it instead of scripting their frontend, and that’s usually the actual fix when behavioural detection is blocking a legitimate use case.

For teams building and defending these systems, the pattern worth remembering is that no single behavioural tell is decisive. It’s the combination, scored over a whole session and often over many sessions at once, that separates a real visitor from a script. That’s also why claims of “undetectable” automation don’t hold up to scrutiny: the detection layer is explicitly designed to look at aggregate consistency, not a single checkbox you can tick.

If you want more breakdowns like this on how scraping infrastructure, proxy systems, and detection actually work under the hood, you can find the rest of our explainers and tool reviews on the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.

What personalised pricing actually is

You run the same scraper against the same product page twice, ten minutes apart, and get two different prices. Nothing crashed. No error in the logs. Just two numbers that don’t match. If you’ve spent any time monitoring competitor pricing at scale, you’ve hit this. The instinct is to assume a bug in your own code. Often the real answer is that the site never had one price to begin with.

Personalised pricing means the price (or the discount, the shipping estimate, the “only 2 left” badge) is computed per visitor rather than per product. The product record in the backend has one canonical price, but what gets rendered depends on signals attached to the request: who’s asking, from where, on what device, with what history. Airlines and hotel booking sites have done this for years. Retail has caught up, partly because it’s cheap to do once you already have an experimentation platform in place for UI testing.

For anyone running scrapers for price intelligence, MAP monitoring, or competitive research, this matters because it breaks the basic assumption that “the price” is a stable fact you can pull once and trust.

The mechanics behind price personalisation

Personalisation engines don’t usually hardcode “charge this user more.” They work off a bucketing function: some identifier (a cookie ID, a device fingerprint hash, sometimes IP-derived geo) gets fed into a hash, and the hash output determines which variant of the page you see. This is the same infrastructure used for ordinary A/B testing on layout or copy, just pointed at price or discount logic instead.

Signals that commonly feed the bucketing decision:

• Session and cookie history (returning visitor vs new, cart abandonment)
• Geolocation resolved from IP address
• Device and browser fingerprint (screen size, user agent, sometimes canvas/WebGL signals)
• Account state if logged in (loyalty tier, past purchase volume)
• Referrer (arrived from a coupon site vs direct navigation)
• Currently running experiments the traffic gets randomly assigned into

None of these are exotic. They’re the standard inputs any web analytics stack already collects. The personalisation layer just reuses them to decide which price variant to serve.

Why your two scrapers see different numbers

This is the part that trips people up: two scraper runs against the “same” page are rarely identical requests. Even with the same target URL, they can differ on almost every signal the bucketing function reads.

If your scraping infrastructure rotates proxies between runs, you’re changing the resolved geolocation and often the ASN, which can move you into a different regional pricing zone or a different experiment cohort entirely. If cookies aren’t persisted between runs, every scrape looks like a brand new anonymous visitor, which itself is a bucketing signal (new visitors sometimes get a welcome discount that returning visitors don’t). If your headless browser configuration varies run to run (different viewport, different user agent string picked from a rotation list), the fingerprint hash changes, and so does the bucket.

None of this requires the site to be doing anything adversarial toward scrapers specifically. It’s the same bucketing logic that would show two real human shoppers on two real devices two different prices for the same reason. Your scraper just happens to be triggering the same mechanism from run to run because its own identity isn’t held constant.

A/B testing buckets vs genuine personalisation

Worth separating two things that get conflated. Genuine personalisation ties the price to something durable about you: your account, your loyalty tier, your purchase history. A/B testing ties it to a random assignment that’s supposed to be stable for a given identifier but arbitrary in what it means. If you’re bucket A, you get the control price; if you’re bucket B, you get the 10% off variant. Neither bucket is “the real price.” Both are live in production at the same time, on different slices of traffic, so the business can measure which one converts better.

From a scraper’s point of view these look identical: two runs, two prices, no visible reason. But the practical implication differs. Genuine personalisation might mean a real customer profile would consistently see one price, so a competitor intelligence report built on scraped data should say “we observed a personalised range of X to Y” rather than pretending there’s one ground truth number. An A/B test in progress means the price is temporary and will resolve to one variant once the test ends, and sampling it once tells you almost nothing about where it’ll land.

The role of caching and edge nodes

A layer that gets ignored: CDN and edge caching. Sites using Fastly, Cloudflare, or similar often cache rendered pages at edge locations close to the requester. If pricing is baked into the cached HTML rather than fetched client-side via an API call, then which edge node answered your request matters. A proxy exit in Frankfurt and one in Singapore can hit different cache tiers with different TTLs and different regional price configurations, independent of any personalisation logic at all.

This is a separate failure mode from bucketing, and it’s worth ruling out first because it’s the easiest to diagnose: check response headers for cache status (age, x-cache, cf-cache-status and similar), and check whether the price appears in the initial HTML payload or gets fetched by a subsequent XHR/fetch call to a pricing API. If it’s the latter, that API call is a more reliable point to instrument than the rendered page, since it typically carries fewer caching layers.

Building a scraper that gives you a stable baseline

If the goal is to track a competitor’s price over time, the fix isn’t to fight personalisation, it’s to hold your own request identity constant so you’re comparing like with like. In practice that means:

• Pinning a proxy pool by region deliberately, not rotating randomly between requests when consistency matters more than volume
• Persisting cookies and session state across scheduled runs instead of starting fresh each time
• Keeping browser fingerprint parameters (user agent, viewport, timezone) fixed for a given monitoring identity, rather than randomising per request
• Logging the response headers and request signals alongside the scraped price, so a later price change can be traced back to “new proxy IP” or “cache expired” rather than treated as a mystery

If the goal is closer to market research, understanding the distribution of prices a real customer base actually sees, then the opposite approach applies: deliberately vary geo and session state across multiple identities and treat each scrape as one sample from a distribution, not a single fact. That only works with a large enough sample to say anything statistically meaningful. A handful of runs from a handful of IPs isn’t a distribution, it’s noise with a story attached.

What you can conclude and what you can’t

A single scrape showing a different price than yesterday’s scrape is not evidence of an active A/B test, and it’s not evidence of nothing either. It’s a prompt to check what changed on your side of the request first: proxy exit, session state, fingerprint, cache headers. Only once those are controlled for does a repeated, reproducible price difference across otherwise-identical requests start to look like genuine personalisation worth reporting on.

It’s also worth being honest in any report or dashboard about what’s a measured fact and what’s an inference. “This price varies by region, confirmed across N runs from M distinct pinned locations” is a claim you can stand behind. “This site personalises price by loyalty status” is a much stronger claim that needs an actual logged-in test to support it, not a guess from anonymous traffic.

The ethical and legal edges worth respecting

Price pages that are publicly visible without login are a different category from anything gated behind an account, and personalisation research should stay on the public side of that line. Scraping a site’s published, non-personal pricing to understand a market is a common and legitimate use of scraping infrastructure. Pulling data tied to a specific logged-in account, or a real individual’s personalised offers, isn’t the same thing, and it’s the kind of line worth being deliberate about before a project starts, not after. Whatever a site’s terms of service say about automated access is also worth reading before building a monitoring pipeline against it, since the answer varies a lot by site and isn’t something a scraping tool can decide for you.

If you’re building out price or A/B test monitoring and want to compare notes on proxy infrastructure, session handling, or pipeline design, that’s exactly the kind of thing we write about here.

Get new guides and videos first — join the Telegram channel.

You load the page in a browser and it looks normal. You run the same URL through your scraper and the response is thinner, or it’s a different layout entirely, or the data you need just isn’t there. Nothing crashed. No 403, no CAPTCHA, no error at all. The request “succeeded” and it still gave you the wrong thing.

This is one of the more frustrating failure modes in scraping because it doesn’t look like a failure. Status code 200, valid HTML, page renders fine. It’s just not the page a real visitor sees. Here’s what’s actually happening on the other end, and how to figure out which version you’re getting instead of guessing.

The request isn’t just a URL

A browser hitting a page and a scraper hitting the same URL are sending very different requests even when the URL string is identical. The server sees headers, TLS handshake characteristics, connection behavior, and often a JavaScript execution environment that a bare HTTP client never provides. Any of those can be used to decide what to serve.

The mechanism that matters most here is server-side branching: the backend looks at signals in the request and picks a response template before it ever sends bytes back. This is different from a block, where the server refuses the request outright. A differential response is the server deciding “this looks like X kind of visitor, give them the X version.” Your scraper isn’t rejected, it’s just categorized.

Common signals that feed that decision:

• User-Agent string. The oldest and weakest signal, but still checked. A default python-requests/2.31.0 UA is an instant tell that no browser sent this request.
• Header set and order. Real browsers send a specific, consistent set of headers in a specific order (Accept, Accept-Language, Accept-Encoding, Sec-Fetch-* headers, etc). A scraper using a minimal HTTP client often sends a much shorter list, or sends them in an order no browser produces. Some anti-bot vendors fingerprint header ordering specifically because it’s cheap to check and hard for naive clients to get right.
• TLS and HTTP/2 fingerprinting. The TLS ClientHello (cipher suite order, extensions, supported curves) and the HTTP/2 SETTINGS frame differ between Chrome, Firefox, curl, and most scraping libraries. This is checked before a single byte of your HTTP request is parsed, so it can steer the response before content negotiation even happens.
• Cookie and session state. No-cookie requests, or requests missing a cookie that a real page visit would have set on a prior load, often get routed to a stripped-down or cached fallback version.
• JavaScript execution. If a page’s real content is rendered client-side and your scraper only fetches the raw HTML, you’re not seeing a deliberately different page, you’re seeing the pre-render skeleton. This looks identical to a differential response from the outside but the cause and the fix are different.

None of this requires the site to know it’s talking to your specific scraper. Most of it is bucket sorting: does this request look enough like a browser to get the full experience, or does it get the cheap path.

Why sites actually do this

It’s worth being precise about motive here, because “anti-scraping” is only one reason.

Bot mitigation. Vendors like Cloudflare, Akamai, and DataDome offer tiered responses as an alternative to outright blocking. A suspicious-but-not-confirmed request might get a JS challenge page, a reduced dataset, or a cached generic version instead of the personalized or full page. This buys the detection system more signal without tipping off the requester that they’ve been flagged, and it’s cheaper on their infrastructure than serving full dynamic content to every automated hit.

Legitimate performance and personalization, not detection. A lot of “different content” has nothing to do with bots at all. CDN edge caches serve a generic cached page to first-time or cookie-less visitors and a personalized one to sessions with state. A/B testing frameworks split traffic by cohort. Geo-routing serves a different page by inferred location, and datacenter proxy IPs often geo-resolve to the wrong place or to a hosting provider’s address block rather than a residential one. If your scraper’s IP resolves to “Ashburn, Virginia, hosting provider” instead of the city an actual user would show up in, you may get the generic template for reasons that have nothing to do with fingerprinting.

Rendering path differences. A single-page app that hydrates content via a client-side fetch call will show a scraper the loading skeleton and show a browser the populated page, purely because the scraper never ran the JavaScript that makes the second request. This is the most commonly misdiagnosed case: it looks exactly like deliberate discrimination but the site isn’t doing anything targeted at all.

Separating these three causes matters because the fix is completely different depending on which one you’re hitting.

How to tell which one you’re dealing with

Don’t guess. Diagnose it the same way you’d debug any other network issue: compare requests side by side.

1. Capture the exact request your scraper sends. Log every header, in order, plus the TLS client if you’re setting one explicitly. Most scraping libraries expose this if you turn on debug logging.
2. Capture the exact request a real browser sends to the same URL, using browser devtools’ network tab with “copy as cURL” or similar. Diff the two header sets.
3. Check whether the response references client-side data fetches. View the raw HTML source (not the rendered DOM) for the scraper’s response. If the content you want lives inside a <script> tag as JSON, or the DOM is mostly empty divs with data-attributes, that’s a rendering-path issue, not a detection issue.
4. Check the IP’s reverse DNS and ASN. If it resolves to a known hosting or proxy provider, geo-based and reputation-based serving may explain content differences that have nothing to do with headers or fingerprints.
5. Look at response headers, not just body content. Vary, Cache-Control, Set-Cookie, and any X- custom headers from a CDN can tell you whether you got a cache hit versus a dynamically generated response. A Vary: User-Agent header is a direct admission that content is branching on that field.

This kind of side-by-side diff usually narrows the cause to one of the three buckets above within a few requests, rather than leaving you swapping settings at random.

What responsible operators actually do about it

None of this is a checklist for evading detection, and there isn’t a version of that checklist we’d publish here even if you asked, because “make requests indistinguishable from a browser” runs straight into a site’s terms of service and, depending on what’s being accessed, real legal exposure. What’s worth saying plainly instead:

Operators running production pipelines treat consistent, complete headers and realistic TLS behavior as baseline request hygiene, the same way they’d treat setting a correct Content-Type on a POST. If your target renders content client-side, a headless browser is the correct tool for the job, not a workaround, because you actually need the JavaScript to run to get the data that exists. Residential or mobile-carrier proxy infrastructure exists because IP reputation and geo-resolution are real signals that affect content and rate limits, not because it makes you undetectable, and no proxy vendor that’s honest with you will claim otherwise.

Beyond the technical layer, the boundary that actually matters is what the site’s terms of service and robots.txt say, and whether what you’re collecting is public, non-personal, and not paywalled. A differential response is sometimes the site telling you, in the only language HTTP has, that it doesn’t want this kind of traffic. Diagnosing why it’s happening is good engineering. Working around it just because you can isn’t the same decision as being allowed to.

If you’re building a scraping pipeline and want the header, TLS, and rendering-path diagnostics above wired into your monitoring instead of done by hand each time, that’s the kind of infrastructure problem we write about and build tooling around.

Check out more breakdowns like this one on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.

A folder with 118 dated files in it, one per day, all of them clean. The question was how many of the urls I saw in March were gone by June.

I could not answer it without writing a program. So I wrote one, and it opened all 118 files, held everything in memory, and produced a number about forty minutes later. What I had written was a database. A slow one with no indexes, and it could answer exactly that one question and nothing adjacent to it.

That is the moment this piece is about. The storage decision made by default on day one is usually correct on day one, and it has an expiry date almost nobody sees coming.

The folder is the right first move

One file per run, named with the date, appended to and then left alone. No server, no schema, no migration, no credentials. You can read it with anything, copy it by dragging it, and when a value looks wrong you open the file and look at the line that produced it.

For a source pulling a few thousand records a day this is the quickest correct answer available, and I still start every new source this way. Skipping it because it feels unserious costs people a week of setup they did not need yet.

The moment it expires

It is the same moment every time: the first question you want to ask that spans more than one file.

Everything inside a single day stays trivial. Row count for Tuesday. What this listing cost on the 14th. One file, one pass, done.

Then you want a comparison. What changed. What was there in March and gone by June. How often a field actually moves. Now you are reading 118 files at once and the thing you are writing is a query engine.

The trigger is the shape of the question, not the size of the disk. I have seen 400GB folders that never needed a database because every question was “what did we get yesterday”, and I have seen 2GB folders that needed one within a month.

Two layers, and the split is the whole design

There are two things to store and they have opposite requirements.

The raw capture is exactly what came back, stored as it arrived, never edited. Append only, literally: you do not go in and correct a bad row.

The derived table is parsed, typed, deduped, indexed. It is what you query. The rule that makes this work is that it can be deleted entirely and rebuilt from the raw layer whenever you want.

Keeping them apart is what buys you the right to be wrong later, which you will be.

The rebuild test

One question tells you whether you actually have two layers or one layer wearing two hats.

Delete the derived table right now. Can you have it back by tomorrow morning without touching the source site again?

If the honest answer is that you would have to rescrape, your parsed table is the only copy you own. Every parser bug you ship is then permanent damage to data you already paid for, and the site does not keep history on your behalf.

I had a selector go wrong on a price field and write the shipping cost into it for nine days before anyone noticed. Because the raw responses were on disk, the fix was a reparse that ran over lunch. Without them, nine days of history gone for good.

Raw means raw

The temptation, once you have a raw layer, is to fix things in it. You spot a broken row, you know what it should have said, you edit it.

Don’t. The moment raw becomes editable it stops being evidence, and the entire reason you kept it disappears.

Corrections are new rows. The old one stays wrong forever with its timestamp attached, and the derived table takes the newest.

The boring database is right for longer than you think

For the derived layer, use a relational database. Postgres, MySQL, SQLite if it is one machine with one writer.

Tens of millions of rows on a $200 box with an index on the columns you filter by answers in milliseconds. You get constraints, so bad data bounces at the door instead of landing quietly. You get transactions, so a half finished write leaves no debris to find on Monday. And everything speaks SQL.

People skip it because they buy for the volume they imagine rather than the volume they have. Somebody reads that relational databases fall over at scale, decides they will hit that scale eventually, and stands up a cluster before they have 100,000 rows. Now every question costs more to ask and there is an operational surface to keep alive.

Most scraping projects never leave the range where one boring database is the fastest thing in the building. You will know when you leave it, because a query you run daily will get slow and stay slow.

Columnar files, no server

Between a folder of JSON and a database server sits a third option that gets skipped: column oriented files on disk, which in practice means Parquet.

A query engine reads those files directly. You point it at a folder, you write SQL, and there is no server anywhere: no port, no user accounts, nothing to restart after a reboot. For analytics over a few hundred million rows on one machine this is genuinely good, and it costs nothing while nobody is asking anything.

What it will not do is serve. No unique constraints, no cheap single row lookup, and updating one value means rewriting a file. It answers questions about the pile. It does not hold your current state.

When a search engine is actually required

People stand up a search cluster years before they need one. My test is narrow on purpose.

Are you matching words you cannot list in advance, over free text, with results ranked by how well they matched? If yes, that is a search engine and nothing else does the job. Scoring text against a corpus is a different problem from filtering rows.

If your queries are equality and ranges, “this field equals that”, “this date between those two”, what you want is an index on a column. Adding a cluster for that buys you a second copy of your data to keep in sync and one more thing that can be down at 3am.

Either way the search index is derived, rebuilt from the real store on demand. The day you cannot rebuild it, you have made a search engine your source of truth, and search engines drop documents quietly and have no constraints to stop bad data landing.

What raw pages cost, in gigabytes

Store raw responses compressed. HTML squeezes to roughly a fifth of what came over the wire because it is repetitive text.

Now run the arithmetic once, because almost nobody does. Pages averaging 120KB compress to about 25KB. At 100,000 pages a day that is 2.5GB a day, 75GB a month, 900GB a year. One source, one crawler, unremarkable volume, and you have filled a terabyte in fourteen months without a single alert firing.

Retention gets decided on day one or by the disk

So decide how long you keep raw pages on the day you start keeping them.

The alternative is that the disk hits full at 3am, writes start failing, and you delete under pressure with no idea what mattered. Same decision, made worse, by someone tired.

Ninety days of raw covers essentially every parser bug you catch in time to act on. I have never needed raw HTML from eleven months ago. I have needed last Tuesday’s more times than I can count.

The two columns you will never regret

Put a fetch timestamp and a source identifier on every row. Every table, no exceptions, including the ones you are certain will never need them.

The timestamp is when you fetched it, not when the page claims it was published. Those are different values, the second one lies, and only the first lets you ask when something became wrong.

The source identifier is which site, which endpoint, which scraper wrote this row. It costs a few bytes.

The day you need them the reason is always identical: something in the derived table is wrong and you need to know which run wrote it and when. Without those two columns you cannot answer that at all.

Add a run id while you are in there. Then when you ship a bad parser you delete exactly the damage by run, instead of by date range and prayer.

The storage choice I regret

I kept raw HTML inside the database. Same table as the parsed row, in a text column, because it was one less thing to manage. It worked fine for months.

Then the nightly dump went from four minutes to about fifty. The table had become mostly bytes nobody ever queried, dragged through every backup, every restore, every vacuum, and sitting in cache pushing out the rows I read constantly.

Moving it to compressed files on disk took a weekend and the dump went back under five minutes. What I took from it: anything you never query does not belong inside the thing you query.

A backup you have not restored is not a backup

Nightly dumps to encrypted offsite storage is the easy half, and everyone does it. The half almost nobody does is pulling the backup back down, restoring it into an empty database, and counting the rows.

I run that check on a schedule now, because the first time I tried it the dump turned out to be months of a table that had been renamed underneath it. The job reported success the whole time. Exit code zero, file present, file useless.

The line storage does not move

None of this changes what you were allowed to collect. Public data, a robots file honored, a published crawl delay treated as an instruction, an official API or bulk feed used wherever one exists, and personal or paywalled material left alone.

A tidy two layer store full of data you should not have collected is still data you should not have collected.

The full written guides, table layouts, and the tooling I actually run in production are here.

Get new guides and videos first — join the Telegram channel.

Most scraping advice assumes you control the clock. You pick the interval, you decide when to back off, you choose the hour of the day with the least traffic on the target site. Client scheduled scraping flips that. Someone else, usually a client with their own reporting cadence or downstream system, tells you when the job has to run. Your job is to make that work without breaking the parts of the pipeline that used to be flexible.

This is a common setup for agencies, data vendors, and anyone running scraping as a service. The client wants a price feed updated at 9am their time, or a lead list refreshed every Monday before their sales team logs in. They don’t care about your proxy pool health or the target site’s rate limiting behavior. They just want the data in their system on their clock.

What changes when the trigger isn’t yours

When you control the schedule, you can smooth load over the day. Stagger jobs, spread requests across hours, retry failures whenever capacity opens up. When a client controls the schedule, you lose that flexibility in one specific dimension: the start time is fixed, and it’s fixed for a reason that has nothing to do with your infrastructure.

The practical effect is that you now have a hard deadline attached to every run, not just a target. If the client’s dashboard refreshes at 9am, a scrape that finishes at 9:15 might be functionally useless even if it succeeds. That changes how you think about retries, timeouts, and failure handling. A job that would normally get three attempts spread over an hour now has to get all three attempts inside a much smaller window, or you need a fallback that serves the client stale-but-labeled data rather than nothing.

The thundering herd problem

If you run scraping for multiple clients and several of them ask for the same trigger time, “every day at 9am” or “top of the hour,” you end up with a thundering herd: a burst of jobs that all want proxy IPs, worker capacity, and outbound connections at the same instant. This is an infrastructure problem before it’s a scraping problem. A proxy pool that comfortably handles steady traffic across a day can get saturated by ten clients’ worth of jobs all firing in the same sixty-second window.

The fix isn’t clever, it’s capacity planning. Know how many concurrent client-triggered jobs your worker fleet and proxy pool can actually absorb at once, and queue the overflow rather than letting everything fire simultaneously and degrade for everyone. A simple job queue with a concurrency cap does more for reliability here than any amount of scraper-level optimization. If a client’s schedule is truly rigid down to the minute, that’s a conversation about capacity, not a problem you solve by writing a smarter scraper.

Why perfectly regular intervals are a detection signal

Here’s the part that’s specific to scraping rather than generic job scheduling. Bot detection systems look at request timing as one signal among many, alongside things like TLS fingerprints, header ordering, and behavioral patterns. A request that arrives at exactly the same second every single day, indefinitely, is a pattern a human browsing session doesn’t produce. It’s not proof of automation on its own, but it’s a data point that feeds into a larger fingerprint, and detection systems are built to correlate many small signals rather than rely on one.

This creates real tension in client scheduled scraping. The client wants the run at a fixed time. The target site’s defenses are more likely to flag traffic that never varies. There’s no trick that resolves this cleanly, because the constraint is genuinely in conflict: rigid timing is easier to detect, and the client isn’t asking you to be undetectable, they’re asking for data by a deadline. What you can do honestly is build in the amount of jitter the client’s actual requirement allows. If “9am” really means “sometime in the 9am hour, before the team logs in,” that’s room to randomize the trigger within the window rather than firing on the second. If the client genuinely needs the exact minute, you don’t have that room, and no amount of engineering makes that timing constraint invisible. Be upfront with clients about this tradeoff instead of promising a fixed schedule is risk-free.

Retry logic when you don’t own the deadline

Normal retry design assumes you can push a failed job later. Client scheduled scraping usually can’t. If the run has to land before a specific downstream event, your retry budget has to fit inside that window, which means shorter backoff intervals and a lower ceiling on attempts than you’d use for a job with no deadline.

This pushes more weight onto the health of the run before it starts rather than recovery after it fails. Warm proxy pools, pre-validated sessions, and pre-flight checks on the target site’s response shape all matter more here, because there’s less time to recover from a bad first attempt. If your standard pipeline retries five times over two hours, you need a separate, tighter policy for time-boxed client runs, not the same policy with a shorter timer bolted on.

You also need an honest answer for what happens when the deadline passes and the job hasn’t succeeded. Serving nothing is one option. Serving the last good dataset with a clear staleness timestamp is usually better, but only if the client’s system can actually handle receiving data marked as old rather than treating it as fresh. That’s a conversation to have before the first missed run, not during it.

Monitoring a schedule you didn’t set

When you set the schedule, you build alerting around your own expectations. When the client sets it, your alerting has to be built around theirs, and that’s easy to get wrong if you just reuse existing dashboards. A job that finishes in four minutes might be totally fine on a schedule you control and completely unacceptable on a client’s five-minute window.

Track time-to-completion against the client’s actual deadline, not against some internal SLA that doesn’t match what they asked for. Alert on jobs that are trending toward a miss while they’re still running, not just on jobs that already failed. By the time a client-scheduled job has failed outright, it’s often too late to do anything useful before their deadline passes, so the alert that matters is the one that fires while there’s still time to intervene.

What to ask before agreeing to a client’s schedule

Before committing to a fixed trigger time, it’s worth getting specific answers to a few things: how much slack actually exists around the stated time, what happens on the client’s end if a run is late or missing, whether stale data with a timestamp is acceptable as a fallback, and whether the client is asking for one schedule or effectively the same schedule as several other clients you already serve. None of this is exotic, it’s the same capacity and SLA conversation you’d have for any time-sensitive data pipeline. The scraping-specific wrinkle is just that rigid timing narrows your options for retries and makes the traffic pattern easier to pick out, so it pays to know exactly how rigid the requirement really is before you build around it.

Running scrapers on someone else’s clock is manageable, it just moves the hard problems earlier in the pipeline: capacity planning before the burst, session and proxy readiness before the first request, and an honest fallback plan before the deadline you can’t move.

If you want more breakdowns like this on scraping infrastructure, proxy behavior, and pipeline design, check out the rest of the site here.

Get new guides and videos first — join the Telegram channel.

I once wrote a postmortem blaming a supplier for flaky proxy lines. I had to send a second one a week later saying the lines were fine and the problem was me.

The job config said three retries. The http client underneath it was also set to three. Nine attempts per failing page, forty workers wide, all triggered by one five minute wobble on the target. About three hundred and sixty requests hit that site inside a few seconds, and it did what any sane infrastructure does.

The scraper worked perfectly. The layer above the scraper is what broke, and that layer is what this piece is about.

The number in your config is not the number

Retry counts multiply across layers, and almost nobody writes down the product.

Scrapy retries twice by default through RETRY_TIMES, so a fresh project is already at three attempts before you touch anything. Most cloud sdks retry internally before your code ever sees a failure. Wrap any of that in an orchestrator with its own task retry setting and you have built a multiplier without noticing.

The fix is a decision, not a library. Exactly one layer owns retries, everything below it fails fast and reports upward, and the effective number lives in a comment where a human will read it. For me the owner is the job runner, because it is the only thing that can see how many workers are already in flight. The http client sits at zero and says zero out loud.

What an immediate retry actually does

A retry is a purchase. You are spending a second request against a target that just told you it could not serve the first one, and the naive version spends it at the worst possible moment.

Think about what the target sees. Forty workers fail at the same instant because the cause was shared. All forty come straight back. Same paths, same second, same address ranges. Any site with rate defences reads that as a signature and acts on it.

This is the most common way people get themselves blocked while filing a ticket about the target.

Backoff, and where to cap it

Exponential backoff is one line of arithmetic: wait base * 2 ** attempt. One second, two, four, eight. It works because the delay grows faster than most transient problems last, so a blip costs you a second and a genuine outage parks you somewhere sensible.

Two things people get wrong with it.

Cap the delay. Unbounded doubling will eventually sleep a worker for half an hour, and a worker asleep for half an hour is capacity you are paying for and not using. Sixty seconds is a fine ceiling for most scrapes.

And honor an explicit instruction over your own formula. A 429 usually arrives with a Retry-After header carrying the exact number of seconds the site wants. Your curve is a guess. That header is not. Coming back sooner than a site asked in writing is the rudest thing in the whole stack, and it is the fastest route from a temporary throttle to a durable block.

Jitter is what stops a fleet from reconverging

Backoff alone spreads nothing out.

Forty workers fail in the same second, all compute the same two second wait, and all return in the same second. You moved the spike two seconds to the right. Its shape is identical.

Jitter means randomising the wait: random() * base * 2 ** attempt instead of the bare formula. Now those forty land smeared across a window.

It matters more the more machines you own. I have scrape jobs on four boxes. If all four use the same formula against the same clock, they do not collide once and recover. They reconverge. They fail together, back off by the same amount, and find each other again on every round after that, which is how a fleet of well behaved schedulers produces a synchronised spike that none of them individually asked for.

One question, instead of a table of status codes

Before retrying anything, ask whether anything will be different next time.

If nothing about the request or the world has changed, the retry is asking the same question louder. A permission error is the clean example: credentials that were wrong a second ago are still wrong, and five more attempts produce five identical refusals plus a pattern in someone’s logs.

That question sorts nearly everything, and it keeps working on errors you have never seen before.

Worth retrying, because the cause sits outside your request and is temporary: connection timeouts, resets, 5xx from the origin, and a 429 where you actually wait as instructed.

Not worth retrying: 401 and 403, parse failures, validation failures. A 403 is a decision the site made about the identity behind the request, so repeating the identical request from that identical identity mostly confirms the decision. A parse error retried five times gives you five copies of one stack trace.

404 sits between the two. A URL that worked last week and 404s today is genuinely gone, so log it once and mark it resolved rather than retrying it. A URL you invented by incrementing an id was never there, and that is not a failure at all.

Attempt ceilings and the pile behind them

Every job needs a hard cap. Without one a retry loop is an infinite loop with better manners.

When an item hits the cap it goes somewhere durable with the error attached and the input that produced it. A dead letter table with the url, the attempt count, the last error and a timestamp is enough. So is a folder of json files.

The ceiling on its own is half a solution. I have opened dead letter tables holding forty thousand rows, months of quiet accumulation, with no alert ever fired because technically nothing failed. The items were caught, which is exactly what made them invisible.

So alert on the pile as a rate, not a count. Total size only tells you the job is old. New rows per day compared against a normal week tells you something started breaking on Tuesday.

Idempotency is the permission slip

All of this rests on the job being safe to run twice.

Idempotent means a second run leaves the database exactly where the first one did: same rows, same counts, nothing doubled. With that property a retry is a shrug and a backfill is a command. Without it, every retry is a gamble.

You feel the difference at two in the morning, in the moment you hesitate before clicking rerun. That hesitation is the real cost of a job you cannot safely repeat.

A checkpoint turns a dead job into a resumed one

A long job that dies at eighty percent and restarts from zero has charged you that eighty percent twice, and charged the target for it too.

Checkpoint the cursor. Write down the last page or id you finished, durably, after the rows commit and never before. Do it in the same transaction as the write if your storage allows: a checkpoint that lands before the data produces silent gaps rather than duplicates, and gaps are far harder to notice.

This is the cheapest item on the list and the one skipped most often, because for the first month the job never dies.

Nobody needs to start at zero minutes past

Open your crontab and count how many entries start at minute zero.

So does everyone else’s. Every hourly scrape, plus the backups, the reports and the health checks, all firing on the hour because nobody chose otherwise. Your target meets the worst minute of its day sixty times a day at the same offset, and you are standing in that crowd for no benefit.

Move off it. Pick 17 past, or hash the job name into a minute so the offset is automatic and stable across deploys: minute = crc32(job_name) % 60. The request is identical, the data is identical, you have just stopped arriving with the mob.

It helps your own side too. I had four jobs on the same box all set to minute zero, fighting over one proxy pool and one cpu for four minutes an hour and idling for the other fifty six. They sit at 7, 19, 31 and 43 now, and the box is boring.

Alert on the shape of the output

An exit code of zero means your code reached the end without throwing. That is the entire claim it makes.

A scraper that fetched a block page, parsed nothing out of it, wrote zero rows and returned cleanly is a success by that measure. Every dashboard is green and the data is gone.

So make the run’s own output the alert. Count what you produced, compare against this source’s recent normal, and fail the run deliberately when the number is absurd. A source that gives you eleven thousand rows most days and six today should be red even though nothing errored. Zero rows is the loud version. One field going null while the rows still validate is the quiet one. A run that never started hides longest, because there is no run to inspect and no error to catch, so the absence of a successful run needs its own alert.

Left to exit codes, these get found when somebody downstream asks why a number looks stale. That is a detection time measured in weeks.

What a retry policy does not buy you

None of this changes what you are allowed to collect. Backing off is restraint, and restraint does not widen the target by an inch. Public data, a robots file honored, a published crawl delay treated as an instruction, and an official api or bulk feed used whenever one exists.

I run these scrapers over my own carrier lines every day, so the numbers here are ones I have paid for. The worked versions, the backoff and jitter code, the checkpoint pattern and the alert thresholds I actually use are at dataresearchtools.com.

Get new guides and videos first — join the Telegram channel.

A scraper I run pulls listings off a supplier catalogue every night. It ran for eight months without a complaint. Then somebody wrapped the price block in one more container to fix the spacing on mobile, and the price column went blank for nineteen days before anyone noticed.

Nothing failed. The job exited zero every night, the row count held steady, and one column had quietly turned into a wall of empty strings.

That is the shape of every parser incident worth talking about. The fetch worked and the pipeline was green. The data was wrong.

What copy selector actually gives you

Almost everyone starts here. Open devtools, right click the element, copy selector, paste. It works immediately, which is what makes it dangerous.

Read what you pasted. It is a path through containers: a wrapper, a grid, the second child, a class name that is a hash of some styling rules. Every step in that chain is a promise the site never made you.

Add a wrapper and it breaks. Insert an advert slot above the target and the child index shifts, so the selector still matches something and hands back the wrong value. That is the worse outcome, because nothing looks broken.

If the site is built with a css-in-js library, those class names are generated from the style rules. Change a padding value, ship it, and the hash changes. No redesign required.

The test I use now: read the selector aloud. If you cannot say which field it grabs without opening the page in front of you, it describes position. Position is the shortest-lived thing on a web page.

A ranking of anchors by how long they last

You pick an anchor once per field and then live with it for years, so the order is worth memorising.

Structured data first. Most listing and commerce sites publish a JSON-LD block, microdata attributes or Open Graph tags for search engines. That markup is generated from the same database that renders the page, and the site’s rich results depend on it staying correct, so it gets maintained by people who are not thinking about you at all.

Watch it for staleness. I have seen a JSON-LD price hold the old value for several hours after the visible page updated, so check it against the rendered page when you wire the field up.

Second, identifiers and data attributes the site relies on for its own logic. An id its JavaScript queries to attach a handler. A data-testid its test suite depends on. Renaming one of those costs the site something. You can usually tell by reading them: data-product-price is a deliberate hook, class="flex gap-4" is decoration.

Third, the text label sitting next to the value. On a spec table, find the row whose header cell reads Warranty and take the cell beside it. This survives a column reorder and a complete restyle, because you are following the word a human reads.

Labels have their own failure mode. They are copy, and copy gets rewritten. Warranty becomes Cover. A locale switch returns half of them in German. Match against a small accepted set instead of one exact string.

Last, position in the tree. Second cell, third paragraph, nth-child. Use it when the page gives you nothing else, and leave a comment saying so, because that is the field that will break first.

The empty string that cost me nineteen days

I should own the specific mistake, because everything above came out of it.

I had a helper wrapping every extraction. Selector matched, return the text. Selector missed, return "". I wrote it that way so the calling code stayed tidy with no null checks scattered through it. I was pleased with it at the time.

Then the container changed and every price came back as "".

My required-field validation checked for None. An empty string is not None, so validation passed. The column was declared TEXT, so Postgres took it without comment. Row counts stayed normal, the job stayed green, and a dashboard downstream showed a full table with one column of blanks that nobody had reason to look at.

I found it nineteen days later, and only because a reconciliation against a second source came out wrong.

Worse, I was not storing raw pages then, so there was nothing to reparse. Every one of those listings had to be fetched again, through metered mobile lines, at a pace slow enough to avoid getting blocked. That took most of a week. A handful had already rotated off the site, and those rows are still empty.

The bug was one line. My helper collapsed the difference between a field that is empty and a field that is missing, and those are two different facts about the world. It raises now.

Extract wide and keep the bytes

Two habits, both cheap at the time you write them.

Pull more fields than the current requirement asks for. You already paid for the fetch, and pulling another eight values out of a DOM you are already holding costs nothing measurable.

Then store the raw capture. Compressed bytes, sitting next to the URL, the fetch timestamp and the parser version that produced the row.

That single decision changes the price of every future break:

• With the capture, a redesign is a reparse. Fix the extraction, run it back over stored pages, and history repairs itself at the speed of a batch job. No proxies, no rate limits.
• Without it, the same redesign is a recrawl. Two weeks of pages fetched again through metered bandwidth at a polite pace, and any listing that has rotated off the site since is gone permanently.

You do not need infinite retention. A rolling window of a few weeks covers realistic detection lag, and one saved sample per page type, kept indefinitely, gives you fixtures to test against.

Assertions belong at the field level

Most parsing code fails soft by default. No match, return None or "", record continues down the pipeline looking perfectly normal. That default is the entire problem.

Give the parser two outcomes. Either it produced a record whose shape it recognises, or it refused and named the field that failed.

The assertions themselves are boring, which is fine:

• price parses as a number and is greater than zero
• title is present and under 300 characters
• currency is one of the four you actually deal in
• a date parses and does not land in the future

Each is a couple of minutes of work. Together they turn a redesign into a first-record failure instead of a slow drip of blanks nobody sees.

Do not kill the whole run over one bad record, though. Quarantine it: write the record to a rejects table with the raw HTML attached, increment a counter, carry on. A normal night sits well under 1% rejects. The morning after a redesign it is 90-something, and the HTML you need to write the new selector against is already sitting there.

Fill rate, and the alert that actually fires

Fill rate is the proportion of rows where a given field came back empty. One query per field, and the most useful number in the pipeline.

The mistake is comparing it against a fixed threshold. Real data has legitimately sparse fields: a discount that only exists during a promotion, a second image that only some listings carry. Set the alarm at 90% full and those fields page you nightly until someone mutes the channel. Set it at 10% and a field sliding from 99% to 60% never trips.

Compare a field against its own history instead. Trailing median over the last 14 days, per field and per source, with an alert on a deviation of more than a few points. The absolute level carries almost no information. The delta carries all of it.

Splitting by source matters more than it sounds. A global average across twenty sites hides one broken site completely, because the other nineteen hold the number up for it.

A strategy list per field, and recording the winner

Instead of one selector, give each important field an ordered list of extraction strategies. JSON-LD, then the data attribute, then the label-anchored lookup, then the positional guess. First one to return a plausible value wins.

The half of this that people skip is recording which strategy won. Store the strategy name on the record.

That column is the early warning. If strategy one carried 99% of rows for a month and this morning strategy three is carrying them, the page changed. The data is still correct, so nothing else in the stack has any idea. You get a week to fix it properly instead of an afternoon to fix it badly.

Skip the column and your fallbacks absorb the damage silently until the last one dies too, and then you get the outage anyway with no idea when the decay started.

It does cost more code per field, so scope it. Fields somebody would ring you about get the full ladder. Everything else gets one strategy and a loud assertion.

What a resilient parser still does not fix

None of this makes a parser permanent. A site can restructure hard enough to take out every rung at once, and sooner or later one will. The goal is a break that is cheap to repair and visible the day it happens.

It also has nothing to say about what you are allowed to collect. Public pages, robots.txt respected, a request rate that does not hurt the site, and an official API or a bulk feed used whenever one exists. A parser you never had to write is the only one that cannot break.

We write up the rest of this work, including the fill rate queries, over at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

why your scraper should care about a status page

When a scrape job returns a wall of 500s or timeouts, the first question in the queue is always the same: did we get blocked, or is the site actually down? Get that wrong and you burn cycles on the wrong fix. You rotate proxies chasing a block that was never there, or you keep hammering a broken endpoint assuming it’s your fingerprint. A target’s own status page, the same page they built for their paying customers, answers that question for free.

This isn’t a workaround for anything. Status pages are public by design. Companies stand them up so customers stop opening support tickets during outages. Reading one as part of your pipeline orchestration is no different from reading their published API docs.

what a status page actually shows

Most status pages break the target’s stack into components: API, website, checkout, auth, webhooks, sometimes region by region. Each component carries a live state (operational, degraded, partial outage, major outage) and a history of past incidents with timestamps, root cause notes, and resolution times. Scroll down and you’ll usually find a rolling uptime percentage per component going back 90 days.

That component list is a rough map of how the target has split their own infrastructure. If checkout is its own component separate from the general API, checkout likely runs on different infrastructure or scaling rules than everything else. If “API” and “API v2” appear as separate rows, that tells you they’re running two API generations in parallel, which matters if you’re planning to move a scraper off a deprecated endpoint.

the public API most status pages don’t advertise

A large share of status pages run on a handful of platforms: Atlassian Statuspage, Better Stack, Instatus, or Cachet for self-hosted setups. Atlassian Statuspage in particular exposes a documented JSON API at predictable paths under the status domain, typically something like /api/v2/status.json for the current overall state, /api/v2/summary.json for the full component and incident summary, and /api/v2/incidents.json for incident history. These are meant to be consumed by dashboards and Slack bots, not scraped from rendered HTML.

If a target’s status page runs on one of these platforms, you can pull structured incident data directly instead of parsing a page. That means your pipeline can check target health with a single lightweight request instead of a full page load, and you get machine-readable timestamps instead of parsing “about 2 hours ago” out of relative text.

Not every status page is built this way. Plenty of companies run a bespoke internal page with no public API, just server-rendered HTML. In that case you’re back to parsing markup, which is normal scraping work and carries the same maintenance burden as any other page you track.

reading incident history to separate their outage from your block

The real payoff is correlation. Keep a log of your own scrape failures: timestamp, endpoint, status code, error type. Cross-reference that against the target’s incident history. If your 500 spike at 14:02 UTC lines up with an incident opened on their status page at 14:00 UTC for “API, partial outage,” that’s not a detection problem, that’s a real outage, and the right response is to back off and retry later, not to change your request fingerprint or rotate through your proxy pool looking for a fix that doesn’t exist.

The reverse is just as useful. If your failure rate spikes and the status page shows all green, that’s a real signal too, it tells you the problem is specific to your traffic and worth investigating on your end: a changed selector, a new rate limit, a fingerprint check that started flagging your setup. Either way, the status page turns a guess into a data point instead of a hunch.

subscribe, don’t poll

Most status pages offer RSS or Atom feeds, and many support email or webhook subscriptions for incident updates. If you’re running a production pipeline against a handful of important targets, subscribing to their incident feed and piping it into your monitoring is cheaper than polling the status page on a timer. It also means you find out about a new incident within minutes instead of on your next scheduled check.

For Statuspage-hosted pages, the RSS feed usually lives at a path like /history.rss or /history.atom off the status domain. For a bespoke page, look for a “subscribe to updates” link. This is the same mechanism their own customers use, and it puts the target in the position of proactively telling you when to expect trouble, rather than you inferring it from failed requests after the fact.

what a status page won’t tell you

Don’t over-read it. A status page reflects what the target chooses to disclose, and most companies are conservative about what counts as an incident. Degraded performance that doesn’t cross their internal threshold, a slow database replica, a partial regional issue affecting a small share of traffic, these often never make it onto the page at all. A green status page means the target isn’t reporting a problem. It doesn’t mean nothing is wrong, and it definitely doesn’t mean your traffic specifically is unaffected by whatever defensive measures they run separately from infrastructure health, like rate limiting or bot detection, which live in a completely different system and are never reflected on an uptime page.

Treat the status page as one input, not a verdict. It’s most useful for ruling out “is this a real, target-wide outage” before you spend time debugging your own pipeline. It’s not a substitute for your own error logs, and it has nothing to say about how a target’s bot detection behaves, since that’s a separate, usually undocumented, layer of the stack.

building it into your pipeline

Concretely, this means adding one more health check ahead of your existing retry logic: before a job retries a failed target, check whether that target has an open incident. If the status API, or a subscribed feed, shows an active incident matching the failing component, hold the retry and back off on a longer interval instead of hammering a target that’s already struggling, which is bad practice regardless of whether you’re scraping or not. If the status page is clean, let your normal retry and alerting logic run as usual, because the failure is yours to solve.

It’s a small addition, usually a single scheduled job that polls or subscribes to a handful of status feeds and writes the current state into a table your retry logic can read. But it removes one of the more common false alarms in scraping operations: chasing a phantom block that was actually just the target having a bad day.

If you’re building out monitoring like this across a real pipeline, there’s more on infrastructure, orchestration, and the defensive side of bot detection over on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.