Author: Xavier Fok

The pitch that stopped working

For a long time, the easiest way to sell a dataset was to talk about size. Ten million rows. Every product listing on a marketplace. Full historical coverage back to 2019. Volume was the headline because it was the easiest thing to measure and the easiest thing to put in a sales deck.

Buyers who actually put that data into a pipeline learned the hard way that row count tells you almost nothing about whether the data is usable. A file with ten million rows and no record of how each row was collected, when, from what source, and under what method, is a file you can’t trust for anything that matters. You can’t debug it when a downstream model behaves strangely. You can’t defend it if a client asks where a number came from. You can’t tell the difference between a real price and a price that was served to a blocked or flagged client as a decoy.

That’s the shift. Buyers stopped asking “how much do you have” and started asking “how do you know this is right, and can you show me.”

What provenance means in a scraped dataset

Provenance is the record of how a piece of data came to exist in your dataset. For scraped or collected data specifically, that means being able to answer, per row or at minimum per batch:

• What URL or endpoint was hit, and when
• What method retrieved it: browser automation, a direct HTTP request, an official API, a third-party feed
• What network path it came through: residential proxy, datacenter IP, mobile carrier, or the collector’s own origin IP
• Whether the response was validated against a schema or a checksum before it was accepted
• Whether the same data point was ever re-checked or reconciled against a second collection pass

None of this is glamorous. It’s logging, timestamps, and metadata columns that most people strip out before they ship a “clean” export. But it’s the difference between a dataset a buyer can audit and one they have to take on faith.

Why volume alone became a liability

A pipeline that optimizes purely for throughput tends to degrade quality in ways that aren’t visible until someone downstream hits them. Rotating through a large pool of proxies to avoid rate limits is a normal, legitimate part of running a scraper at scale. But if the pipeline has no way to tell a real page from a soft-blocked page, it will happily ingest thousands of decoy or degraded responses and count them as successful rows. Sites that detect scraping traffic don’t always return a hard 403. Many serve a plausible-looking but subtly wrong page: stale prices, empty inventory, or a cached snapshot, specifically because a silent degrade is harder for the scraper to notice than an outright block. A pipeline built for volume, without validation against a second source or a checksum on structural changes, will ship that bad data at the same confidence as good data.

This is why a buyer who has been burned once starts asking different questions. Not “how many rows,” but “how do you know these rows are real.”

How buyers actually verify provenance

In practice, verification is rarely a full audit of someone else’s infrastructure. It’s a handful of checks that are cheap for the buyer to run and hard for a low-effort collector to fake:

Spot re-collection. The buyer independently re-scrapes or re-requests a small random sample of the dataset and compares it against what was delivered. If the numbers drift wildly on rows that shouldn’t have changed, that’s a signal the original collection had blind spots.

Timestamp coherence. If every row in a “real-time” dataset has the exact same collection timestamp down to the second, that’s a tell that the data was collected in one pass and backfilled, not continuously monitored the way the pitch implied.

Source diversity. A dataset that claims broad market coverage but shows collection patterns consistent with a single IP range or a single automation fingerprint suggests the collector never actually diversified their infrastructure, which raises the odds of the kind of soft-block degradation described above.

Schema and outlier sanity checks. Buyers who work with structured data run basic statistical checks: null rates, outlier distributions, duplicate rates. A collection pipeline with weak validation tends to show up as unusually clean in ways that are actually suspicious, like a dataset with zero missing fields on a site that’s known to have inconsistent listings.

None of these checks require the buyer to see your infrastructure. They require the data itself to hold up under scrutiny, which is really a proxy for whether the collection method was sound in the first place.

What a defensible collection pipeline looks like

From the operator’s side, building for provenance instead of raw volume changes a handful of concrete things about how a pipeline is put together.

Validation happens at ingestion, not after. Every response gets checked against expected structure before it’s written to storage: field presence, plausible value ranges, and a diff against the last known-good version of that record. Responses that fail validation get flagged and either retried through a different path or excluded, not silently accepted.

Collection metadata is a first-class column, not an afterthought. Timestamp, source method, and network path travel with the record, not just in a separate log file that gets thrown away at the end of a run. If a client asks about a specific row six months later, the answer needs to exist.

Proxy infrastructure is chosen for consistency, not just evasion. A rotating pool of proxies helps a pipeline avoid rate limits and IP-based blocking, but the goal from a provenance standpoint isn’t just “don’t get blocked.” It’s knowing, for any given request, what network characteristics it went out with, so that a batch of results can be explained later if something looks off. A pipeline that can’t tell you which proxy pool served a given batch can’t tell you why that batch looks different from the rest.

Detection responses get treated as data, not noise. When a target site’s bot-detection system serves a CAPTCHA, a soft block, or a degraded response, a well-built pipeline records that event rather than silently retrying until it gets something that parses. Knowing the rate of detection events on a given source is itself a signal of how reliable that source’s data is likely to be, and it’s a number worth being able to show a buyer, not hide from one.

Detection systems aren’t the enemy of good data, they’re part of the audit trail

It’s worth saying plainly: bot-detection and fingerprinting systems exist because sites have a legitimate interest in controlling automated access to their infrastructure and protecting their own users’ data. A collection pipeline that respects a site’s terms of service and rate limits, and that treats detection responses as information rather than an obstacle to brute-force around, ends up producing a more auditable dataset almost as a side effect. The discipline required to build a pipeline that behaves well also happens to be the discipline that produces data a buyer can trust.

None of this means any particular proxy pool or scraping setup is undetectable or risk-free. Every collection method carries some rate of failure, block, or drift, and any pipeline claiming otherwise is a pipeline that hasn’t been tested by someone paying close attention.

What this means if you’re the one building the pipeline

If you’re selling data or building a pipeline that others will eventually rely on, the practical takeaway is to instrument before you optimize for scale. Log the network path. Timestamp at the point of collection, not at export. Validate against structure and against a second pass, not just against “did the request return a 200.” Keep the metadata that lets you answer a hard question about a specific row a year from now.

Buyers care about provenance more than volume because volume is easy to fake and provenance is not. A dataset that can explain itself is worth more than one that’s simply bigger, and that gap only gets wider as more buyers learn to ask the second question instead of stopping at the first.

Data Research Tools covers the infrastructure side of this, from proxy selection to pipeline validation to how bot-detection systems actually behave in production. If you’re building or evaluating a scraping pipeline, take a look at the rest of the site here.

Get new guides and videos first — join the Telegram channel.

The wall went up faster than most people noticed

If you’ve run scraping infrastructure for more than a couple of years, you’ve watched the baseline shift. Sites that used to return clean HTML to any request with a browser-like user agent now challenge you with a JavaScript puzzle, a CAPTCHA, or a silent 403 that never explains itself. This isn’t paranoia on the part of site owners. It’s a rational response to a decade of automated traffic that has gotten cheaper to generate and harder to distinguish from a real visitor. Understanding why the gates went up tells you a lot more than any bypass trick ever will, because the reasoning behind a defense usually predicts how it’ll evolve.

Bots cost money before they ever do anything malicious

The first and most boring reason sites block bots is compute cost. Every request that hits an origin server, even a well-behaved one, consumes CPU cycles, database connections, and bandwidth. A site that gets 2% of its traffic from scrapers might not notice. A site that gets 40% from scrapers, which is common for e-commerce catalogs and pricing pages, is effectively running a second, uninvited business on its infrastructure bill. CDN and WAF vendors like Cloudflare, Akamai, and Fastly built entire product lines around this problem because their customers were asking for it directly: stop the traffic that isn’t converting, isn’t reading, and isn’t paying, before it reaches the origin.

This is also why rate limiting is usually the first line of defense rather than the last. It’s cheap to implement, it doesn’t require behavioral modeling, and it catches the least sophisticated scrapers immediately. If you’ve ever run a pipeline that started returning 429s after a burst of requests from a single IP, you hit this layer, not a more exotic one.

Content has value that scraping erodes

Beyond raw compute, there’s a business logic reason. Product listings, pricing data, real estate inventories, and job postings are often the actual product a site sells, directly or through ad impressions and lead generation. A competitor or aggregator that scrapes that data and republishes it captures value without paying for it. This is a different threat model than a research bot pulling a few pages a day, and it’s the one that drives the most aggressive defenses: dynamic pricing sites, ticketing platforms, and marketplaces treat structured data extraction as a direct threat to their business model, not just a traffic nuisance.

License and terms-of-service enforcement sits next to this. Many sites explicitly prohibit automated collection in their terms, and the detection stack exists partly to make that prohibition technically enforceable, not just legally stated. Scraping public pages isn’t automatically illegal, but ignoring a site’s stated terms, working around a paywall, or collecting personal data at scale moves into territory that carries real legal exposure, separate from whatever the bot detection system does to your IPs.

Fraud and abuse are the sharper edge

A meaningful share of bot traffic isn’t research or price comparison at all. It’s credential stuffing against login forms, inventory hoarding on sneaker and ticket drops, fake account creation for promo abuse, and scalping bots that buy up limited stock faster than any human could click. These are the cases that justify the most aggressive detection tooling, because the cost of a false negative is direct financial loss, not just a slightly inflated server bill. Sites that sell anything scarce (event tickets, limited product drops, GPU restocks) invest heavily here because the ROI on stopping a scalping bot is immediate and measurable.

This matters for anyone running legitimate infrastructure because it means detection systems are often tuned defensively, with a strong bias toward blocking anything ambiguous. A research scraper that behaves oddly gets caught in the same net as a checkout bot, because from the defender’s side, false positives are cheaper than false negatives when fraud is on the table.

What the detection stack actually looks at

Modern bot management doesn’t rely on a single signal. It layers several, and understanding the layers explains why simple fixes like rotating a user agent stopped working years ago.

Network-level signals come first: IP reputation, ASN classification, and geolocation. Datacenter IP ranges (AWS, Hetzner, DigitalOcean, and most proxy providers) are flagged differently from residential and mobile carrier ranges, because legitimate consumer traffic almost never originates from a datacenter ASN. This is the single biggest reason the proxy industry shifted so heavily toward residential and mobile IPs over the last several years, and it’s also why those pools carry a real cost: someone has to actually be running traffic through a residential connection for the IP to look residential.

TLS and HTTP fingerprinting sit above that. Every TLS client negotiates a handshake with a specific set of cipher suites, extensions, and ordering, commonly summarized as a JA3 or JA4 fingerprint. Standard HTTP libraries like Python’s requests or Node’s default fetch produce fingerprints that don’t match any real browser, because real browsers negotiate through their own TLS stack (BoringSSL for Chrome, NSS for Firefox) with a distinct signature. A site doesn’t need to inspect your headers to know you’re not using an actual browser; the handshake gives it away before a single byte of HTTP is exchanged.

Above that sits behavioral and environmental fingerprinting: canvas rendering output, WebGL parameters, installed fonts, screen and viewport consistency, mouse movement entropy, and timing between actions. Services like Google’s reCAPTCHA, Cloudflare Turnstile, DataDome, and Human (formerly PerimeterX) score sessions continuously rather than gating once at the door. A session that passes an initial challenge can still get flagged mid-session if its behavior drifts from what a human produces, things like uniformly spaced request timing or a complete absence of scroll and mouse events on a page that a real user would interact with.

Finally, there’s the account and session layer for logged-in surfaces: device fingerprint consistency across sessions, velocity of actions per account, and correlation across seemingly unrelated accounts sharing infrastructure. This is where most of the aggressive fraud-prevention logic lives, and it’s largely invisible until it triggers.

The AI crawler wave changed the calculus again

The last two years added a new category of automated reader that most sites weren’t originally built to handle: large-scale crawlers feeding LLM training and retrieval pipelines. Unlike a search engine crawler that sends traffic back to a site through search results, an AI training crawler often extracts value with no return traffic at all. That asymmetry is why Cloudflare rolled out AI-specific bot blocking and pay-per-crawl mechanisms, and why robots.txt directives targeting specific AI user agents (GPTBot, ClaudeBot, CCBot, and others) became common almost overnight. Site owners who were previously ambivalent about scraping got a concrete, visible reason to tighten defenses across the board, and the tooling built for AI crawlers doubles as tooling against every other kind of bot traffic too.

What this means if you operate scraping infrastructure

None of this is a reason to give up on data collection as a discipline; it’s a reason to be honest about what you’re doing and why. Respecting robots.txt and published API terms, identifying your traffic where a site provides a mechanism to do so, keeping request rates well below what would degrade a site’s performance, and avoiding personal or paywalled data are the baseline for anything defensible, legally or ethically. Beyond that, the technical reality is that detection systems are layered, continuously updated, and tuned to treat ambiguity as risk. There’s no configuration that makes traffic undetectable or a proxy pool that makes a pipeline unbannable, and any vendor claiming otherwise is describing marketing copy, not the architecture underneath it. The sites that matter to you have teams whose entire job is closing the gap you’re relying on, and the honest way to run infrastructure long-term is to build for that reality rather than against it.

If you’re building or maintaining scraping pipelines and want to understand the infrastructure and detection landscape in more depth, we cover proxy architecture, fingerprinting, and pipeline design in detail on Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The job website change monitoring is actually doing

“Website change monitoring” sounds like one feature, but it’s really three separate jobs stitched together: fetch a page on a schedule, compare it to the last version you saved, and tell a human when something meaningful changed. Every product in this space, paid or homemade, is solving those same three problems. The differences show up in how well each step handles the messy parts: pages that render with JavaScript, content that changes every load for reasons that have nothing to do with the thing you actually care about, and targets that don’t want to be polled every five minutes by an automated client.

If you run scraping pipelines for a living, this is familiar territory. A change monitor is just a scraper with a memory. That framing is useful because it means the same infrastructure questions apply: how you fetch, how you store state, how you avoid getting yourself blocked, and how you keep noise out of your alerts.

What a monitoring service does behind the dashboard

Commercial page monitoring tools are built around a few components that are worth understanding before you decide whether to pay for them or write them yourself.

Fetching. Most of these services run a fleet of workers that request your target URL on an interval you set, from minutes to days. Simpler tools do a plain HTTP GET and read the HTML. Anything that markets “works on JavaScript sites” is running a headless browser (Chromium via Playwright or Puppeteer, most commonly) to let the page render before it captures the DOM. Headless rendering is slower and more expensive per check, which is why cheaper tiers on most services either skip it or throttle how often they’ll do it.

Snapshotting. Each fetch gets saved, usually as raw HTML, sometimes as a rendered screenshot, occasionally as an extracted text version that strips markup. The snapshot is the thing every future comparison gets measured against.

Diffing. This is where the real engineering lives. A naive full-text diff on raw HTML is nearly useless because it flags every ad refresh, every rotating testimonial, every CSRF token embedded in a form, and every timestamp printed in the footer. A page can look identical to a human and still produce a 40-line diff. Serious monitoring tools deal with this by scoping the diff to a CSS selector or XPath you define, by running visual diffing on a rendered screenshot instead of markup, or by extracting the readable content (think Readability-style article extraction) before comparing. Some let you exclude specific regions of the page from the comparison entirely.

Alerting and thresholds. Once a diff is computed, the service decides whether it’s worth telling you about. That usually means a percentage-changed threshold, a minimum diff size, or keyword triggers (“only alert me if the word ‘out of stock’ appears”).

Understanding this pipeline is the whole point of this article, because your own version needs to do the same four things whether you’re paying $10 a month for a hosted tool or running it on a box in your closet.

The parts that are harder than they look

Three things trip up nearly everyone who builds this themselves, and they’re the same things paid tools spend most of their engineering effort on.

Noise filtering. The single biggest source of false alerts is diffing content that changes on every load regardless of what you care about: ad slots, “recently viewed” widgets, randomized product carousels, session-specific nonces. If your diff isn’t scoped tightly, you’ll get an alert every single check and stop trusting the tool within a week. Scoping to a specific selector, or stripping known-volatile elements before comparing, is not optional past a handful of pages.

Rendering cost. A plain GET request is cheap: milliseconds, a few KB of bandwidth. Spinning up a headless browser to render a JavaScript-heavy page is orders of magnitude more expensive in CPU and memory, and it doesn’t parallelize for free. If you’re watching a few hundred pages on a tight interval, this is the line item that decides whether your own pipeline actually costs less than a subscription once you count server time.

Getting rate-limited or blocked by the target. Any client that hits the same URL repeatedly on a fixed schedule looks different from normal browser traffic, and sites that care about scraping traffic can and do notice. This isn’t unique to homemade monitors; it’s the same detection surface any scraper deals with; consistent request timing, a single source IP making all the requests, a user agent or TLS fingerprint that doesn’t match a real browser, missing the resource requests (fonts, images, XHR calls) a real page load generates. Commercial monitoring services deal with this by spreading checks across many IPs and rotating user agents at the fleet level, the same way a distributed scraping operation would. If you build your own, you inherit that same problem and the same responsibility: check the target’s robots.txt and terms of service before you start polling it, keep your interval reasonable for what you’re actually trying to catch, and don’t monitor anything gated behind a login or paywall that you don’t have the right to access. None of that is a workaround, it’s just not creating a problem you’ll have to explain later.

Building it yourself: the minimum viable pipeline

If you decide to roll your own, the shape of a working system is small. A fetcher (requests or httpx for static pages, Playwright if you need rendering) on a cron schedule or a simple scheduler like APScheduler. A snapshot store, which can be as basic as timestamped files in a directory or S3 bucket, or rows in Postgres if you want queryable history. A diff step, ideally scoped with a CSS selector rather than diffing the whole document, using something like Python’s difflib for text or a perceptual hash / pixel diff library if you’re comparing screenshots. And an alert step that only fires past a threshold, pushed to wherever you actually look: Telegram, Slack, email, a webhook into whatever dashboard you already run.

This is a weekend project for one or two pages. It stops being trivial once you’re watching dozens of pages across sites with different rendering requirements, different block-evasion needs, and different noise profiles, because at that point you’re maintaining per-site selector configs and a proxy rotation the same way you would for any production scraping job.

Where a paid service earns its subscription

The honest case for paying someone else: they’ve already built the noise-filtering heuristics across thousands of site layouts, they carry the headless rendering cost on infrastructure sized for it, and they’ve already solved the alert-routing and dashboard problem you’d otherwise build yourself. For a handful of pages, especially ones you’re not watching for a living, that’s a reasonable trade of a small monthly fee for not maintaining a scraper.

Where rolling your own wins

The case flips once monitoring is core to what you do rather than a side utility. If you’re already running scraping infrastructure, proxy rotation, and scheduling for other jobs, adding a diff step is marginal cost, not a new system. Owning the pipeline also means you control exactly what counts as a change (down to a single price element or stock-status string), you own the history instead of renting access to it, and you’re not bound by a vendor’s check-interval limits or page-count tiers. The tradeoff is that you own the maintenance too: selector drift when a target redesigns its page, rendering cost as your watch list grows, and the same anti-blocking discipline any scraping operation has to practice.

A rough decision rule

Watching under a dozen pages, on layouts you don’t control and don’t want to maintain selectors for: pay for a service and scope your alerts tightly so you’re not drowning in noise. Watching pages as part of a pipeline that already fetches, renders, and stores data at scale, especially where you need field-level precision like “alert only when this exact price string changes”: build it, because you already have the hard parts (fetching infrastructure, scheduling, storage) and the diff logic is the cheapest piece left to add.

Either way, the quality of the result comes down to the same thing: how tightly you scope the comparison, and how honestly you handle the target site’s traffic expectations.

For more breakdowns like this on scraping infrastructure, proxy setups, and the tools built around them, head back to the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.

A no-code web scraper is a point-and-click interface sitting on top of a browser automation engine. You click the price on a product page, the tool records that as a selector, you click “next page,” it records the pagination pattern, and you hit run. Tools like Octoparse, ParseHub, and Browse AI all follow roughly this model: a visual recorder, a hosted execution environment, and an export button. For a huge slice of what people actually need to pull off the web, that’s genuinely enough. The problem is that the demos are built on the easy 20% of sites, and the marketing rarely tells you where the other 80% lives.

I run scraping infrastructure and proxy pools for a living, so this isn’t a takedown of no-code tools. It’s a map of where they hold up and where they start costing you more time than they save.

What’s actually happening when you click “extract”

Under the hood, most no-code scrapers are running a headless or headed browser (Chromium, usually) with a recorded script layered on top. When you point and click a selector, the tool is generating a CSS selector or XPath expression and storing it. When you run the job, it’s replaying that browser session against the live page, waiting for the DOM to settle, then reading out the text or attribute at that selector.

That’s a completely reasonable way to scrape a static or lightly dynamic page. It falls apart the moment the site’s structure isn’t stable, because a recorded selector is a snapshot of one page state at one point in time. Sites that A/B test layouts, lazy-load content on scroll, or restructure their DOM based on device type will quietly break a recorded selector, and the job keeps running, it just starts pulling nulls or the wrong field. In a code pipeline you’d catch that with a schema check or a row-count alert. In most no-code tools, you find out when you open the export and half the rows are empty.

Where no-code tools are the right call

Small, one-off pulls from mostly-static pages are the sweet spot. Pulling a product catalog from a site that doesn’t hide its data behind login, building a one-time list of business listings, monitoring a handful of competitor prices on pages that don’t change layout often, these are jobs where a visual recorder saves real time and you don’t need to justify writing and maintaining a script for a task you’ll run twice.

They’re also the right call for people who genuinely don’t code and have no path to hiring someone for a single afternoon of data collection. The hosted execution environment (the tool runs the job on their servers, not your laptop) means you don’t need to manage a runtime, a browser binary, or a scheduler. That’s real value, and it’s worth paying for when the job fits the shape these tools are built for.

Where they run out of road

Session state and login flows. Most no-code recorders assume each page load is stateless. Sites that require a login, carry a session cookie through a multi-step flow, or gate content behind an account get difficult fast, because the recorder has to replay an auth sequence reliably every run, and any change to that flow (a new consent screen, a token that rotates, a redirect chain that adds a step) breaks the whole job silently.

Volume and concurrency. No-code platforms sell you a number of “pages” or “credits” per month, and that number maps to a concurrency and rate ceiling on their end, not just a billing line. Once you need thousands of pages a day pulled on a schedule, you’re either paying a steep premium tier or hitting throttling that stretches a job meant to finish in an hour into most of a day.

Anything that needs custom logic. Deduplicating against a previous run, normalizing addresses, joining data across two different site structures, retrying only the rows that failed instead of the whole job, these are ordinary requirements in a real pipeline and awkward-to-impossible in a visual recorder. The tools are built around a linear “visit, extract, export” model, not conditional branching or stateful comparison against prior runs.

Anti-bot systems that go beyond the DOM. This is the one that actually ends the conversation for a lot of use cases, so it’s worth being precise about what’s happening.

What sites are actually checking for

Modern bot defense doesn’t just look at your IP address. It layers several independent signals, and any one of them can flag a session even if the others look clean.

• TLS and HTTP fingerprinting. The order of your TLS cipher suites, HTTP/2 frame settings, and header ordering form a fingerprint (commonly called JA3 or JA4) that’s checked against what a real Chrome or Firefox instance produces. A headless browser that isn’t carefully configured leaves a different signature than the browser it’s pretending to be, and defenders match against known-bad signatures at the network layer before your request ever reaches application code.
• Behavioral signals. Mouse movement curves, scroll timing, and the gap between page load and first interaction all get scored. A script that fires a click the instant the DOM is ready looks nothing like a human who took two seconds to read the page first.
• Header and request pattern consistency. A real browser sends a consistent set of headers in a consistent order, and that pattern changes very little between requests from the same session. Automation frameworks that don’t replicate this closely, or that vary in ways a real browser wouldn’t, get flagged on pattern mismatch alone.
• IP and ASN reputation. Datacenter IP ranges are catalogued and scored by most commercial bot-detection vendors. A request coming from a known hosting provider’s IP block starts with a strike against it regardless of what else is present in the request.

Site operators layer these checks specifically because any single one is beatable in isolation. That’s the defensive logic: no one signal has to be perfect if enough of them are combined, and the combination is what makes casual automation, no-code or otherwise, hit a wall on sites that actually invest in this.

Most no-code tools bundle a proxy pool to help with the IP reputation piece, usually labeled “residential proxies” in the pricing page. What that label doesn’t tell you is the pool’s composition, how sticky a session is (whether you keep the same exit IP across a multi-step flow, which matters for anything stateful), or how the pool is rotated under load. Running my own residential and mobile pools, the difference between a pool built for sticky, low-concurrency sessions and one built for high-throughput rotation is the difference between a job that completes cleanly and one that gets flagged halfway through. A no-code tool’s bundled proxy layer is a black box you can’t tune, and on sites with real defense in place, that’s often the actual bottleneck, not the extraction logic.

None of this means a given proxy or tool is undetectable, unbannable, or safe to point at anything. It means the checks are layered on purpose, and a tool built for point-and-click extraction generally isn’t built to manage that layering.

The actual decision point

The honest way to decide is by volume, site complexity, and how much the job needs to survive change without you babysitting it. A no-code tool is the right tool for a low-volume, mostly-static, run-a-few-times job. Once you need scheduled runs against a site with meaningful anti-bot investment, stateful logins, custom retry and validation logic, or a proxy layer you can actually configure, you’re looking at a real pipeline: your own browser orchestration, a proxy provider you can tune for the session type, and a parsing and storage layer you control. That’s a bigger lift, and it’s only worth it when the no-code path has actually run out of road for your specific job, not by default.

Always check a site’s terms of service and only collect data you have a legitimate right to access. Scraping personal or paywalled data without authorization carries legal risk regardless of which tool you use.

If you’re trying to figure out which side of that line your project falls on, or you want a closer look at how proxy infrastructure and scraping pipelines actually hold up under real load, that’s what we write about at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

Check for a dump before you build a scraper

Every scraping project starts with the same question nobody asks early enough: does this data already exist somewhere as a file? If you’re about to spin up a proxy pool, write parsing logic for a target’s HTML, and build retry and backoff handling for a site that doesn’t want to be crawled, it’s worth ten minutes to check whether someone has already done that work and published the result as a download.

This matters for the same reason proxy infrastructure costs money. Every request you send through residential or mobile IPs has a per-GB or per-request cost, plus the engineering time to keep the pipeline healthy when a target changes its markup or tightens its bot detection. A bulk download has none of that. You fetch a file once, from a server that wants you to have it, and you’re done with the network layer entirely. The rest of the job becomes parsing and storage, which is usually the easier half anyway.

The catch is that bulk dumps are never a full substitute for a live scraper. They’re a snapshot, they’re licensed for specific uses, and they cover what the publisher decided to include, not what you actually need. Knowing which is which up front saves a lot of wasted infrastructure.

Common Crawl: the web crawl you don’t have to run yourself

Common Crawl publishes petabyte-scale monthly crawls of the open web as WARC, WAT, and WET files hosted on AWS S3, free to pull with no auth wall. If your project is something like “find every page that mentions X domain” or “build a training corpus from a broad web sample,” this is usually the first thing to check, because it means you’re not the one operating the crawler, rotating IPs, or absorbing rate limits from millions of individual sites.

The tradeoffs are real. Common Crawl’s coverage is broad but not exhaustive and not guaranteed to include any specific site you care about, since it crawls based on its own seed list and politeness rules. The data is also a month or more stale by the time it’s published, so it’s a poor fit for anything that needs current state. And WARC files are large and require real parsing infrastructure (something like a Spark or DuckDB job against the index) to be useful at scale. You’re trading crawler infrastructure for a different kind of infrastructure, not eliminating infrastructure entirely.

Wikipedia and Wikidata dumps

Wikimedia publishes full database dumps of every language edition of Wikipedia, plus Wikidata’s structured knowledge base, on a regular schedule at dumps.wikimedia.org. These are XML and JSON exports containing full revision history, current article text, and in Wikidata’s case, structured entity relationships. For any project that would otherwise involve scraping Wikipedia article by article, this is a straightforward win: you get the entire encyclopedia in one download, licensed under Creative Commons terms, with no rate limiting to manage because you’re not hitting the live site at all.

The practical hurdle is size and format. The full English dump is tens of gigabytes compressed, and parsing wikitext markup into clean text is its own non-trivial job, since Wikipedia’s markup includes templates, infoboxes, and citations that don’t map cleanly to plain text. Wikidata’s JSON dump solves a different problem (structured facts, not prose) and is worth pulling separately if that’s what you actually need.

Government and public records data

Most national statistics agencies publish their own data as structured downloads rather than expecting anyone to scrape a web frontend. The US has data.gov as an index across federal agencies, the Census Bureau publishes API and bulk file access to its surveys, and Eurostat does the same for EU member states. State and local governments vary a lot in quality, but many publish open data portals (usually running Socrata or CKAN) with CSV or API exports for things like business licenses, property records, and permit filings.

This is the category where bulk data most clearly beats scraping, because government sites are frequently the worst possible scraping target: inconsistent HTML, session-based pagination, and legal terms that are stricter than commercial sites in some jurisdictions since the underlying records can carry their own use restrictions. If the same data is sitting in a downloadable CSV from the same agency, there’s no reason to touch the web frontend at all.

Company filings and financial data

The SEC’s EDGAR system provides full-text search and bulk access to every filing companies submit, including quarterly and annual reports, insider trading disclosures, and prospectuses, going back decades. It’s a public dataset by design, since disclosure is the point of the filing requirement, and the SEC publishes both an API and periodic bulk archives rather than relying on people scraping the filing viewer.

The IRS also publishes bulk data on nonprofit Form 990 filings through the AWS Registry of Open Data, which covers financial details for hundreds of thousands of tax-exempt organizations. Between EDGAR and the 990 archive, a lot of what people build scrapers to extract from investor-relations pages or nonprofit directories is already sitting in a structured, official download.

Developer and code activity: GH Archive

GH Archive records every public GitHub event (pushes, issues, pull requests, stars) and publishes it as hourly JSON archives going back to 2011, mirrored into BigQuery for querying at scale. If a project involves tracking open source activity, dependency trends, or contributor patterns, this replaces what would otherwise be a very fast way to get your IP range blocked by GitHub’s API rate limits. It doesn’t cover private repos or content that isn’t a public event, so it’s an activity log, not a full code archive.

Geographic data: OpenStreetMap

OpenStreetMap publishes its entire global dataset as a “planet file” through planet.openstreetmap.org, plus regional extracts through third parties like Geofabrik, covering roads, buildings, points of interest, and administrative boundaries contributed under an open license. Anyone building something that needs address data, POI locations, or map geometry should check this before scraping a mapping service, both because OSM’s terms are built for bulk reuse and because most commercial map providers explicitly prohibit scraping their tiles or place data in their terms of service.

What bulk data doesn’t solve

None of this replaces a scraper for data that’s inherently live: current pricing on a specific retailer’s site, real-time inventory, a competitor’s changing product catalog, or anything behind a login wall that no one has agreed to publish. Bulk dumps exist because a publisher decided to make that data available on their terms. When the data you need isn’t covered by any of those terms, that’s a different problem, and it’s the one actual scraping infrastructure is built to solve, within whatever the target’s terms of service and applicable law allow.

It’s also worth being honest about staleness and completeness. A monthly Common Crawl snapshot won’t have yesterday’s page changes. A government CSV export is only as current as the agency’s publishing schedule. Treat bulk downloads as a specific tool for a specific job (broad, static, or slowly-changing datasets) and keep scraping infrastructure for what’s actually live and not otherwise published.

Where this fits in a real pipeline

In practice, the projects that run cheapest are the ones that check for a bulk source first, use it to cover as much ground as possible, and reserve live scraping infrastructure, proxies, request pacing, parsing logic tuned to a specific site, for the narrow slice of data that genuinely isn’t available any other way. That’s not a purity argument, it’s a cost argument: every request through a proxy pool has a price, and every page you don’t have to fetch live is infrastructure you don’t have to build or maintain.

If you’re weighing when a project needs real scraping infrastructure versus when a bulk download will do, that’s exactly the kind of tradeoff we cover on the channel and in the rest of the site.

Explore more breakdowns like this on the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.

The moment it happens

It rarely shows up as a sentence. Most of the time it’s a status code, a blank page where data used to be, or a CAPTCHA wall that wasn’t there last week. Sometimes it’s louder than that: an actual email from someone’s legal team, or a note from a hosting provider saying a target has complained about traffic from your IP range. All of these are the same message delivered through different channels, and any team running scraping infrastructure at scale needs to be able to read every one of them, not just the loud ones.

I run production pipelines against a lot of targets, on proxy infrastructure I maintain myself, and I’ve watched this signal arrive in every form below. None of them are ambiguous once you know what you’re looking at.

The quiet way: robots.txt and terms of service

The first and quietest signal is robots.txt. A Disallow directive on a path, or a Crawl-delay value, is a site operator writing down, in a machine-readable file, exactly what they’re willing to have automated tools touch and how fast. It isn’t legally enforceable on its own, but it’s documented intent, and treating it as noise because it’s “just a text file” misreads what it actually is: the operator telling you their preference before anything else happens.

Terms of service go further, usually with an explicit clause on automated access, scraping, or data reuse. ToS language varies wildly in enforceability by jurisdiction and by what you actually do with the data, and that’s a legal question, not an engineering one. What’s consistent is that a ToS clause prohibiting scraping is a stop signal stated in plain language, before you’ve sent a single request that trips a rate limiter.

The technical way: rate limits and soft blocks

Next comes the layer built into the infrastructure itself. A 429 Too Many Requests with a Retry-After header is a server telling a specific client, in the HTTP spec’s own vocabulary, to slow down or stop for a defined window. On the backend, this is usually implemented as a token bucket or sliding window counter keyed on IP, session, or API key, with thresholds that adapt based on traffic patterns for that route. When you see a 429 pattern, it isn’t random. Someone configured a limit, and your traffic crossed it.

Softer versions of the same signal exist below the 429. Some sites serve degraded or stale content to traffic they’ve flagged as automated rather than blocking it outright, so the response looks successful but the data quietly stops being useful. Others throttle response times for suspect clients. These are deliberate design choices by the site’s engineering team to discourage high-volume automated access without an outright ban, and they’re worth treating with the same seriousness as a hard block, because they’re often the first stage before one.

The hard way: fingerprinting and full blocks

Above rate limiting sits bot management infrastructure, and this is where a lot of the actual detection engineering lives. Commercial WAF and bot-management stacks (Cloudflare, Akamai Bot Manager, DataDome, Human/PerimeterX, and similar vendors) combine several signal sources: TLS handshake fingerprints like JA3 or JA4, which expose differences between how a real browser’s TLS stack negotiates a connection versus an HTTP client library; HTTP header order and casing, which differ between browser engines and most scraping frameworks; canvas and WebGL rendering fingerprints collected via JavaScript challenges; timing between requests; and IP and ASN reputation, where ranges known for hosting or proxy traffic get scored differently than residential or mobile ranges from the start.

None of these signals are decisive alone. What actually flags a session is usually a scored combination, run against a model trained on real traffic from that specific site, which is why the same request pattern gets waved through on one target and challenged on another. It’s worth understanding this as a detection system, not a single trick to route around, because that’s genuinely how it’s built and how the vendors who sell it describe it in their own documentation. A challenge page or a hard 403 at this layer is the site’s infrastructure making an automated determination that your traffic doesn’t look like a browser being driven by a person, and pushing past that determination with a workaround doesn’t change the underlying signal, it just delays the next detection.

The direct way: a human tells you

The clearest version of “stop” needs no interpretation. A cease and desist letter, an account suspension email, a message from a site’s legal or trust and safety team naming your traffic or your company specifically. There’s no ambiguity to parse here, and no technical workaround is a reasonable response to it. If this arrives, the scraping activity it names stops, full stop, while you get a lawyer to tell you what happens next.

Why the arms race is a bad bet, not just a bad look

Set the ethics aside for a second and look at it as an operator managing infrastructure and budget. A residential or mobile proxy pool costs real money per IP, and IP reputation on shared infrastructure is a depreciating asset: once a range gets flagged by a major bot-management vendor’s shared reputation database, that damage often follows the range across every site using that vendor, not just the one you were scraping. Burning a proxy pool trying to outlast one target’s detection layer is money spent degrading infrastructure you’ll want for legitimate work later.

Then there’s the fact that the other side usually has more resources allocated to this exact problem than you do. A company running Akamai or DataDome in front of a high-value dataset is paying a vendor whose entire product is detection, updated continuously against traffic from thousands of scrapers, not just you. Treating a stop signal as a puzzle to solve is a fight against a team built to win it, on their own turf, with a budget scoped to the problem.

What a production pipeline should actually do

The operational answer is boring, and that’s the point. Respect robots.txt by default in your crawler configuration, and require a deliberate override, not a silent one, if a target’s ToS explicitly permits otherwise. Implement backoff on 429s that honors the Retry-After header rather than retrying immediately from a different IP. Set conservative default request rates per target and per session, and treat a rising block or challenge rate on a target as a signal to reduce load automatically, not a trigger to rotate proxies harder. Where a site publishes an official API, use it instead of scraping the same data from rendered pages; it’s usually more stable and it’s the access method the site actually wants automated traffic to take. And where a target’s terms are explicit about prohibiting automated access at any scale, that’s the ceiling, not a starting negotiating position.

If a cease and desist actually lands

Stop the named activity immediately. Preserve your logs rather than deleting anything, since you may need them. Talk to a lawyer before responding, since the specifics of what was scraped, how, and under what jurisdiction all change the actual legal exposure, and none of that is something a blog post can tell you in the general case. What it isn’t is a prompt to switch identities and keep the pipeline running unchanged.

Building the habit before you’re asked

The operators who handle this well aren’t the ones with the cleverest workaround ready for the day a target pushes back. They’re the ones who built rate limits, robots.txt compliance, and ToS review into the pipeline before the first request went out, so “the site asked us to stop” is a normal, expected branch in the system rather than an emergency. Treat every one of these signals, quiet or loud, as data about what a target is willing to allow, and design the infrastructure to respect that from the start.

If you’re building or auditing a scraping pipeline and want to see how we think about proxy infrastructure, rate limiting, and detection from the defensive side, take a look at what we’ve written and reviewed here.

Get new guides and videos first — join the Telegram channel.

If you run scraping infrastructure long enough, you eventually sit across from someone whose job is to say no. A prospective enterprise customer’s security team, a data-buyer’s legal counsel, an investor’s diligence checklist, sometimes your own company’s counsel before a new data source goes into production. The questions are rarely about your scraping code. They’re about the chain of custody around the data and whether you can answer, in writing, where it came from and what happens to it after.

This is a walkthrough of the questions that actually come up, based on running production scraping pipelines and proxy infrastructure at scale. None of it is legal advice. Compliance requirements vary by jurisdiction, by data type, and by what the target site’s terms say, and a real review should involve a lawyer who knows your specific situation. What follows is the operator’s view of what gets asked and why the weak answers get flagged.

Where does the data come from

The first question is always provenance. Which sites, which endpoints, and is that a public page, an authenticated page, or an API you’re calling with someone else’s credentials. Reviewers want a source list, not a description of “the web.” If your pipeline touches ten thousand domains, they want to know how new domains get added and who approves them, because “engineering decides” is not an answer a legal team can sign off on.

A related question: is any of this data submitted by users of the target site rather than published by the site itself. User-generated content, reviews, profile fields, and forum posts carry different expectations than a company’s own product catalog, and a reviewer will ask you to draw that line explicitly.

What do the target’s robots.txt and terms of service say

This gets checked, not skipped. A reviewer will ask whether you evaluate robots.txt and terms of service per domain, and what happens when they conflict with your business need. There isn’t a universal legal answer for what robots.txt directives mean in every jurisdiction, and that ambiguity is exactly why reviewers ask the question instead of assuming it’s settled. What they’re actually testing is whether you have a documented policy at all, versus deciding case by case with no record.

The follow-up is almost always: what do you do when a site’s terms explicitly prohibit automated access. “We proceed anyway” is a real answer some companies give, but a compliance review exists to surface that risk to the people who’ll wear it if it becomes a dispute, not to make the risk disappear.

How is personal data handled

If any target page might contain names, emails, phone numbers, or other data tied to an identifiable person, this becomes the longest section of the review. Expect questions on: do you have a lawful basis for collecting it under the regimes that apply to your users and the data subjects (GDPR, CCPA, and others depending on geography), do you have a defined retention period, can a data subject request deletion and does your pipeline have a mechanism to honor that, and do you separate PII from the rest of the dataset so it can be purged without breaking everything downstream.

A pipeline that scrapes personal data with no deletion path and no retention limit is the single fastest way to fail this part of a review. It’s also one of the most common gaps, because scraping systems are usually built for throughput and completeness first, and access control gets bolted on later.

What load does the scraping put on the target

Reviewers ask about request rate, concurrency, and whether you respect any rate limits the target publishes or implies through 429 responses. This isn’t a courtesy question, it’s a liability one: a target site that experiences degraded service from your traffic has a much stronger case against you than one that simply doesn’t want to be scraped. Being able to show a rate-limiting policy, retry-with-backoff behavior, and per-domain concurrency caps is the answer they’re looking for. Being unable to say what your peak request rate against any single domain looks like is a gap they’ll note.

Where does the proxy and IP infrastructure come from

This question surprises people who haven’t been through a serious review, but it’s one of the first things a security team asks once they know proxies are in the stack. They want to know: are these datacenter IPs, residential IPs, or mobile IPs, and if residential or mobile, how was consent obtained from the device or connection owner. Proxy networks built on undisclosed consent, or on SDKs bundled into free apps without clear disclosure to end users, are a recurring source of legal exposure for the proxy provider and for anyone downstream who uses that network without knowing its origin.

A reviewer isn’t asking this to learn how to evade detection, and you shouldn’t be answering it that way either. They’re asking because the provenance of the IP a request came from is now part of the provenance of the data, the same way the source domain is. If you can’t name your proxy vendor and describe how that vendor sources its IP pool, that’s a real gap, not a technicality.

No proxy setup should be described to a reviewer as undetectable or risk-free. Rotation, geographic targeting, and residential versus datacenter choices affect how a target site’s bot-detection systems classify traffic, but none of it changes whether the underlying access was authorized. A reviewer who hears “our proxies can’t be blocked” should hear a red flag, because it answers a question they didn’t ask and dodges the one they did.

How is the collected data stored and who can access it

Standard infosec ground here: encryption at rest and in transit, role-based access control, audit logging on who queried what, and a documented retention schedule. What’s specific to scraping pipelines is the question of derived data. If you scrape a page and then enrich it, join it with another dataset, or run it through an inference model, does the retention and deletion policy for the original source propagate to the derived record. Most pipelines don’t handle this well, because deletion in the source table doesn’t automatically ripple through a warehouse full of joins and cached features.

Who else touches this data

Subprocessors get their own line of questions. If you use a third-party proxy network, a captcha-solving service, a cloud host, or a data enrichment vendor, a reviewer wants that list, along with each vendor’s own data handling terms. This is where a lot of scraping operations discover they’ve never actually read the terms of service for the tools they depend on.

What a weak answer looks like

The reviews that go badly aren’t the ones with a hard finding, they’re the ones with no documentation at all. “We don’t really track that,” “it depends on the engineer,” and “we’ve never had a complaint” are all answers that read as absence of process rather than presence of a good one. A review can survive a genuinely difficult answer about a specific data source. It rarely survives the discovery that nobody could answer the question at the time it was asked.

None of this guarantees a deal closes or a review passes. What it does is put you in a position to answer honestly, with a paper trail, when someone asks. That’s the actual bar, and it’s a lower one than most teams expect, right up until they’re asked to meet it cold.

If you’re building or auditing a scraping pipeline and want more on how the infrastructure side of this works, proxy sourcing, rate limiting, and bot-detection fundamentals, that’s what we cover at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The page you asked for is not the page you get

Request a URL and you expect the page behind it. In practice you often get something else first: a full screen overlay asking you to accept cookies, an age gate, a country selector, or a “we use cookies to improve your experience” banner sitting on top of the content you actually wanted. These are interstitials, and if you run scraping pipelines for a living, they are one of the most common reasons a job that worked yesterday returns an empty payload today.

This isn’t a bot detection story in the DataDome or PerimeterX sense. Cookie banners aren’t there to catch automation. They exist because of consent law: GDPR in the EU, the ePrivacy Directive, CCPA/CPRA in California. But from a pipeline’s point of view, a consent wall behaves a lot like a detection layer, because it sits between your request and the DOM you’re trying to parse, and it changes what “the page” even means at the moment you capture it.

What a cookie banner actually is, technically

Most consent banners aren’t part of the page’s original markup. They’re injected by a consent management platform (CMP), things like OneTrust, Cookiebot, Quantcast Choice, or Didomi. The site loads a CMP script early in the page lifecycle, that script checks for a consent cookie, and if none exists it renders an overlay, usually a fixed-position div or an iframe stacked above everything else with a high z-index.

That overlay does two things that matter to a scraper. First, it physically covers the DOM you want, which matters if you’re taking screenshots or relying on visible-element checks in a headless browser. Second, and more disruptive, many sites gate their own scripts behind consent. Analytics, personalization calls, sometimes even the API calls that populate the content itself, are wrapped in a conditional that waits for a consent signal before firing. No consent, no request, no content, even though the HTML skeleton returned a 200.

That’s the part that trips people up. A curl or a raw HTTP client gets a document that looks complete but is missing everything that was meant to load after the gate. A headless browser gets past the initial load fine but then sits there, because the tool’s “wait until network idle” condition never resolves the way it expects, since the actual content fetch is paused behind a consent event listener that never fires.

The other interstitials riding along with it

Cookie banners rarely travel alone. Once a site has the infrastructure to intercept a page load and show a modal, it tends to get reused for things that have nothing to do with cookies:

• Age verification gates on alcohol, tobacco, or gambling sites
• Country or language splash screens that redirect based on IP geolocation
• “Continue in app” interstitials on mobile-optimized pages
• Newsletter or paywall soft-gates that appear after a scroll or timer event
• Anti-bot middle pages that show a spinner while a JS challenge runs in the background

Each of these adds its own DOM state, its own timing, and its own failure mode. A pipeline built to expect a single consent overlay will often break silently when it hits a geo-splash instead, because the selector it’s watching for never appears and the timeout logic wasn’t written with that case in mind. In production this shows up as partial data, not an obvious crash, which is the worse failure to have.

How the banner becomes a signal, not just an obstacle

Consent platforms increasingly sit next to, or are bundled with, bot detection vendors. That pairing matters. Some CMPs log interaction telemetry: time between page load and the accept click, whether a mouse actually moved across the screen, whether the click landed with the kind of imprecision a real pointer has versus the pixel-perfect coordinates a script tends to produce. That data doesn’t necessarily block a request on its own, but it can feed into a broader trust score that a separate detection layer, Cloudflare, DataDome, PerimeterX/Human, or a custom in-house system, uses to decide whether to challenge later requests from that session or IP.

This is worth naming plainly because it’s counterintuitive: the thing that looks like a UI annoyance can be part of the same signal chain as a CAPTCHA. A session that never dismisses the consent banner, or dismisses it in zero milliseconds with no cursor movement, looks different from a human session, and on some stacks that difference gets remembered.

What this looks like from the operator’s side

Running scraping infrastructure at any real volume means treating consent interstitials as a reliability problem, the same category as rate limits or rotating proxy exits, not as a one-off bug to patch and forget. A few things production pipelines actually build around this, described at the level of what the system does rather than a walkthrough of any one site:

State detection before parsing. Instead of assuming a fetched page is the final page, pipelines check for known CMP markers, specific iframe origins, class names, or the IAB’s Transparency and Consent Framework (TCF) API that many CMPs expose on window, before trusting that the DOM is complete. If those markers are present, the job treats the page as “not yet resolved” rather than parsing it as-is.

Waiting on the right signal, not a fixed timer. A flat sleep(3) is fragile because banner render time varies by network conditions and by which CMP vendor loaded. More resilient pipelines wait on an actual DOM or network event tied to the content they need, rather than a guess at how long a modal takes to appear.

Separating consent-gated requests from consent-free ones. Some of what a scraper needs, product data, article text, listing prices, often lives in an API response that fires independently of the consent flow if you know which endpoint to watch. Understanding a site’s request architecture, through the browser’s own network panel during a manual pass, is generally more stable long term than fighting the CMP’s DOM every run, and it’s the same technique any frontend engineer uses to debug their own site.

Accepting that some pages will fail closed. No pipeline in production has 100% success against every interstitial variant a site might roll out. Teams that run this at scale build monitoring for silent failures, empty fields, unexpected page structure, rather than assuming a 200 status means a good scrape. That monitoring is doing as much work as the fetch logic itself.

None of this is a claim that any particular method gets past every consent wall reliably or without the site noticing. It doesn’t, and anyone telling you otherwise is selling something.

The legal edge is real, not theoretical

Consent banners exist because processing a visitor’s data without permission carries legal exposure for the site operator under GDPR and similar frameworks. Scraping content that sits behind a consent gate touches that same territory from the other direction: you’re interacting with infrastructure whose stated purpose is controlling data flow, and doing so against a site’s terms of service can carry its own exposure depending on jurisdiction and what you do with the data afterward. This is worth treating as a real constraint on scope, not a footnote. Scraping public, non-personal content that happens to sit near a cookie banner is a different risk profile than scraping personal data or paywalled content the banner was put there to protect, and the two shouldn’t be conflated.

Where this fits in a bigger picture

Interstitials are one layer in a stack that also includes IP reputation, TLS fingerprinting, and behavioral scoring. None of them are insurmountable on their own, but they compound, and a pipeline that only accounts for one layer tends to fail at whichever one it ignored. If you’re building scraping infrastructure and want the rest of that stack explained the same way, proxy rotation, fingerprint consistency, request pacing, that’s the kind of thing we break down regularly.

Read more engineering-first breakdowns of scraping infrastructure at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

What a rate limit budget actually is

Every official API you integrate with gives you a budget, whether it’s stated in the docs as “100 requests per minute” or buried in a support ticket after you got throttled once. That budget is not a soft suggestion. It’s enforced the same way memory limits or disk quotas are enforced, by a piece of infrastructure sitting between your client and the application logic, counting your calls and cutting you off when you go over.

Treating a rate limit as a UI inconvenience instead of an infrastructure constraint is the single most common reason data pipelines fall over in production. A scraper that respects robots.txt and an integration that calls a documented API are different animals, but they share one problem: something on the other end has decided how much traffic it will accept from you, and your job is to build a system that never asks for more than that, on purpose, all the time, not just on a good day.

Why pipelines blow through their quota

In practice, teams don’t exceed their quota because they’re being greedy. They exceed it because of shape, not volume. A pipeline that averages 40 requests a minute against a 60/minute limit looks safe on paper. Then a backfill job kicks off, or a retry loop fires after a transient 500, or five parallel workers each think they have the full budget to themselves, and for ninety seconds you’re sending 300 requests a minute into a bucket sized for 60.

The fix isn’t “call less often” in the abstract. It’s knowing exactly which part of your system is capable of bursting, and putting a hard ceiling on it that the code can’t get around, not even during a backfill, not even when someone is debugging locally against production credentials.

Token buckets, sliding windows, and why the algorithm matters

Most APIs enforce limits with one of two algorithms, and knowing which one you’re up against changes how you should shape traffic.

A token bucket refills at a steady rate and lets you spend tokens in bursts up to the bucket size. If the API gives you 60 requests a minute as a token bucket, you can plausibly fire 60 requests in the first second of the minute and then wait, because the bucket doesn’t care about the distribution within the window, only the total and the burst cap.

A sliding window counter is stricter. It looks at a rolling period, often the trailing 60 seconds, and counts every request inside that window regardless of when the window started. There’s no clever burst-then-wait trick here, because the moment you cross the ceiling inside any rolling minute, you’re throttled, and the window doesn’t reset on a clock you can predict from the outside.

The practical difference matters because a client built for token-bucket behavior, one that happily bursts at the start of every minute, will get hammered by a sliding-window API. If the docs don’t say which model is in play, the safest assumption is sliding window, and the safest client behavior is to spread requests evenly across time rather than in bursts, even if bursting would technically be allowed.

Reading the headers the API is already giving you

A well-built API tells you exactly where you stand on every response, and most pipelines ignore that data entirely, which is a waste. Common headers include a remaining-calls counter, a reset timestamp, and on a 429, a Retry-After value telling you exactly how long to back off.

Any client worth running in production should log these headers on every call, not just when something breaks. That gives you a real-time view of how much headroom you actually have, instead of inferring it from a config file that was accurate the day someone wrote it and has drifted since. If a vendor lowers a limit or changes the window size, the headers tell you before your error logs do.

Ignoring Retry-After and rolling your own backoff timer instead is a common mistake. The header is the server telling you precisely when it will accept traffic again. Guessing a shorter interval just to “try sooner” produces another 429 and, on some platforms, extends the penalty window.

Backoff and jitter, done properly

Exponential backoff is well understood: after a failure, wait longer before the next attempt, doubling or near-doubling each time, up to a ceiling. What gets skipped is jitter, and it’s the part that matters most once you’re running more than one worker.

If ten workers all hit a 429 at the same moment and all back off with the exact same exponential schedule, they retry in lockstep, which recreates the exact burst that got them throttled in the first place, just delayed. Adding randomized jitter to each backoff interval spreads those retries out so they don’t collide again. This isn’t a nice-to-have. A backoff strategy without jitter under real concurrent load behaves like no backoff strategy at all, just a slower one.

A circuit breaker sits on top of this. After a run of consecutive failures against an endpoint, the client should stop calling it entirely for a cooldown period instead of continuing to retry into a wall. That protects your own budget from being wasted on calls that are near-certain to fail, and it stops you from looking, from the API provider’s side, like an abusive client that doesn’t respond to being told no.

Prioritizing calls when the budget runs out

A budget-aware pipeline needs a concept of call priority before it runs out of quota, not after. Not every API call in a pipeline is equally important. A call that refreshes a customer-facing dashboard is not the same as a call that’s part of an overnight historical backfill, and when the budget gets tight, the backfill should yield first.

This means building a simple queue with priority tiers rather than one flat list of pending requests, and giving the scheduler the ability to defer low-priority work indefinitely without dropping it. When the budget is healthy, everything runs close to real time. When it’s constrained, the backfill visibly slows down while the interactive path keeps working. Systems that don’t make this distinction tend to fail uniformly, which is worse: everything gets slow or breaks at the same time, including the parts users actually notice.

Caching and incremental fetch instead of asking twice

The cheapest request is the one you never make. A surprising amount of quota gets burned re-fetching data that hasn’t changed, because the pipeline polls on a fixed schedule instead of asking the API what’s new. If an endpoint supports conditional requests, a changed-since parameter, or a webhook for updates, using it is not an optimization, it’s the difference between a pipeline that fits comfortably inside its budget and one that’s permanently at the edge of it.

Caching responses locally, even for a few minutes, absorbs duplicate calls from different parts of a system that don’t know about each other, which happens more often than teams expect once a pipeline has more than one consumer of the same data.

Multiple keys, multiple tenants, and the line you shouldn’t cross

It’s worth being direct about this: spinning up multiple API keys or accounts specifically to multiply a quota beyond what your license or terms of service actually grant is a different thing from good engineering, and it’s the kind of pattern that gets accounts suspended once a provider’s anti-abuse detection notices coordinated usage across credentials. Rate limits are part of a commercial agreement, not just a technical ceiling, and the fix for a budget that’s too small for your workload is to talk to the provider about a higher tier, not to route around the limit with extra keys.

If you’re managing legitimate multi-tenant access, where each customer genuinely has their own key and their own quota, the engineering problem is isolation: making sure one tenant’s burst doesn’t starve another’s budget inside your own scheduler. That’s a real and useful thing to build. It’s a different problem from trying to make five keys look like the quota of one to a provider that never agreed to that.

Monitoring the budget like it’s infrastructure, because it is

The teams that stop getting paged for rate limit failures are the ones that put quota usage on the same dashboard as CPU and memory. Track requests used against requests allowed, per minute and per day, with alerting well before you hit the ceiling, not after the first 429 shows up in the error logs. A budget that’s consistently running at 90% utilization during business hours isn’t a success story, it’s a pipeline one traffic spike away from an outage.

An API rate limit budget is not a wall you hit occasionally. It’s a resource you’re spending continuously, the same way you spend bandwidth or compute, and it deserves the same planning, monitoring, and graceful degradation as any other finite resource in a production system.

If you’re building or evaluating the infrastructure behind a data pipeline, from proxy layers to scraping frameworks to the APIs sitting in front of them, you can find more breakdowns like this one on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.

Most scrapers start with an UPSERT. You pull a page, parse a row, and write it to a table keyed on some natural identifier: product ID, listing URL, ticker symbol. Run it again tomorrow and the same key gets overwritten with tomorrow’s values. It works, it’s simple, and it quietly throws away the one thing that made the data worth collecting in the first place: what changed, and when.

I’ve run this mistake into production more than once. A price tracker that only stores “current price” can tell you what something costs right now, which you could have gotten from the page itself. What it can’t tell you is that the price dropped 40% for six hours last Tuesday, or that a listing went out of stock and came back with a different SKU. That’s the actual signal. Overwrite-in-place deletes it on every write.

What breaks when you don’t have history

Three things go wrong in practice, and none of them show up until you need the data you didn’t keep.

First, you can’t debug your own pipeline. If a field starts returning null or a number stops matching the source page, an overwrite table shows you the broken state and nothing else. You can’t tell if the site changed its markup, if a proxy pool started getting served a stripped-down page, or if your parser regression happened three deploys ago. Without a timeline, every anomaly looks like it just started existing.

Second, you lose the ability to detect site-side changes as events instead of noise. Sites change their bot-detection posture over time: a new challenge page rolled out to a subset of IP ranges, a field silently removed from the response to break scrapers without breaking the human-facing site, a rate limit tightened. If you’re only storing current state, this shows up as one confusing row. If you’re storing history, it shows up as a change point you can correlate against your own request logs, proxy pool, and response codes from that exact window.

Third, you can’t answer “as of” questions later. Anyone doing price history, availability tracking, ranking changes, or review count trends needs the actual sequence of values, not just the latest one. There’s no way to reconstruct a time series from a table that only ever held one row per key.

The pattern: append-only with valid ranges

The fix is a pattern data warehousing has used for decades, usually called slowly changing dimensions type 2. You stop updating rows in place and start inserting new rows, each one tagged with when it became true and when it stopped being true.

A minimal version needs three extra columns beyond your scraped fields: valid_from, valid_to, and is_current. When you scrape a row and it matches the last known state for that key, you do nothing, or you bump a last_seen timestamp on the current row. When it differs, you close out the old row by setting its valid_to to now, and insert a new row with valid_from set to now and valid_to left null (or far in the future, if your query engine handles nulls awkwardly in range comparisons).

That gives you a table where a single SELECT * WHERE is_current = true reproduces the old overwrite behavior exactly, and a full table scan on a key gives you the entire history. You haven’t lost the simple case, you’ve added the one you were missing.

Deciding what counts as a change

The part that actually takes engineering judgment is deciding what triggers a new history row. Compare every column and you’ll generate a new row on every single scrape, because scraped data is noisy: whitespace differences, a review count that ticks up by one, a timestamp field on the source page itself. That turns your history table into a write amplification problem and makes it useless for finding real changes, because real changes drown in noise.

The practical approach is to hash a defined subset of columns, the ones that represent the actual state you care about, and compare that hash against the current row’s hash. Price, stock status, title, category. Not “last updated” strings from the page, not fields you know are flaky from parsing, not anything with sub-second precision you don’t need. Decide the subset deliberately, write it down somewhere, and revisit it when you add new fields, because it’s the single biggest lever on whether your history table stays useful or turns into landfill.

I also keep a separate raw_hash of the entire parsed payload, computed but not used for triggering new rows. That’s useful later for debugging parser changes, because you can tell “the fields I care about didn’t change, but something in the raw response did” apart from “nothing changed at all.”

Where scraping infrastructure makes this harder

A history table built for scraped data has a failure mode a warehouse table pulling from a stable internal API doesn’t: the scrape itself can be wrong, and a wrong scrape looks exactly like a real change if you’re not careful.

A proxy that got flagged and started returning a challenge page instead of the real page will produce parsed fields that are null, empty, or garbage, depending on how defensively your parser handles unexpected HTML. If your change-detection logic naively treats “field is now null” as a real state transition, you’ll write a history row claiming a product went out of stock when what actually happened is your request got blocked. Do that across a rotating proxy pool and you’ll get spurious history entries correlated with nothing except which exit node served the request.

The defense is to validate a response before it’s allowed to trigger a history write. Check that the page you got back has the shape you expect: right template markers, right response size range, absence of known challenge-page fingerprints. If a scrape fails that check, log it as a failed fetch, retry through the normal pipeline, but don’t let it touch the history table. The history table should record changes in the source data, not changes in your ability to reach it. Those are two different signals and conflating them is the single most common way I’ve seen these tables get polluted.

Storage and pruning

History tables grow, and they grow faster than people expect once you’re tracking a few hundred thousand keys with a real, non-noisy change frequency. Two practical mitigations. Partition the table by month or quarter on valid_from, so old partitions can be moved to cheaper storage or dropped by retention policy without touching a live index. And keep the “current state” query path separate from the “full history” query path, either as a materialized view of is_current = true rows or a small mirror table, so your everyday reads aren’t scanning a table that has years of closed-out rows in it.

If you genuinely don’t need indefinite history, decide a retention window up front, ninety days or a year, and prune past it on a schedule. Keeping everything forever by default because deleting felt risky is how these tables end up ten times larger than the actual current dataset for no analytical benefit.

When overwrite is still fine

None of this means every table needs history. If you’re scraping a value you only ever query as “what is it right now,” and you have no use case for the sequence, an overwrite table is simpler, cheaper, and correct for the job. Reference data that changes rarely and where you don’t care about the change itself, category taxonomies, static metadata, doesn’t need type 2 treatment either.

The decision point is whether the change itself is information. If the answer is yes, if a price drop, a status flip, or a field going missing tells you something you’d act on, build the history table from day one. Retrofitting it later means you’re starting your time series from whenever you noticed the problem, not from whenever the pipeline actually started running, and that gap is usually the exact period someone ends up asking about.

If you’re building out scraping infrastructure and want more on pipeline design, proxy management, and how detection systems actually work from an operator’s side, you can find the rest of what we’ve written here.

Get new guides and videos first — join the Telegram channel.