Category: Uncategorized

Every scraping pipeline eventually hits the same fork in the road: what happens to a record the moment it’s extracted. Does it sit in a buffer until a batch job picks it up, or does it get pushed downstream immediately? This decision shapes your storage layer, your proxy usage pattern, your error handling, and your infra bill. It’s also a decision people make by accident more often than they make it on purpose, usually because whatever framework they started with had a default.

This is about picking that model deliberately, based on what your data actually needs.

What batch ingestion actually looks like

In a batch setup, a crawl job runs to completion (or to some checkpoint), writes its output to a file or staging table, and a separate process loads that output into your data warehouse or database on a schedule. The scraper and the loader are decoupled in time. You might run a crawl every six hours, dump results to Parquet files or a JSON lines file, and have a loader job pick those files up and merge them into Postgres or BigQuery.

The defining trait is that data has a built-in delay between extraction and availability. That delay might be minutes if your batches are small and frequent, or it might be a full day if you’re running once nightly.

Batch is the default for most scraping frameworks because it maps naturally onto how a crawl works: you spin up workers, they hit a target list, they finish, and you have a dataset. Scrapy’s item pipelines, for example, are built around this exporter pattern.

What streaming ingestion actually looks like

Streaming ingestion means each record moves toward its destination as soon as it’s produced, without waiting for a batch boundary. A worker extracts a record, serializes it, and pushes it onto a message broker (Kafka, Kinesis, RabbitMQ, or even a simple queue table) immediately. A separate consumer process reads off that broker continuously and writes into storage, often within seconds of the original scrape.

The scraper doesn’t know or care what happens after the message leaves its hands. That’s the point: producer and consumer are decoupled by a queue, not by a schedule.

Streaming infra is heavier to run because something has to be listening all the time. A batch job can start, run, and shut down. A streaming consumer is a long-lived process that needs its own monitoring, its own restart logic, and its own handling for what happens when the broker backs up.

The real question: how stale can this data be

Strip away the tooling and the choice comes down to one thing: what’s the acceptable gap between “this exists on the target site” and “this exists in my database.”

If you’re tracking SEO rankings, competitor pricing for a weekly report, or building a historical dataset of product listings, a few hours of staleness changes nothing. Nobody making a decision off that data cares if the number is from 9am or 3pm. Batch is the right call here, and it’s simpler to build, cheaper to run, and easier to debug.

If you’re watching for price drops to trigger an alert, tracking odds that move by the minute, or feeding a dashboard that someone is actively watching for a live event, an hour-old record is a wrong record. That’s a streaming problem.

A lot of teams reach for streaming because it sounds more sophisticated, then spend months maintaining broker infrastructure for data that gets queried once a day anyway. Match the ingestion model to the actual consumption pattern, not to what feels more advanced.

Failure handling is where they really diverge

This is the part that doesn’t show up in architecture diagrams but eats the most engineering time in practice.

In batch, a failure is usually all-or-nothing at the job level. If your loader chokes on a malformed record halfway through a file, you can often just fix the parser and re-run the whole batch, because the source file is still sitting there intact. Idempotent loads (upsert on a unique key instead of blind insert) make re-runs safe. Validation can happen once, before anything touches production storage, because you have the whole dataset in hand before you commit to loading it.

In streaming, failures happen per-record, continuously, live. A malformed message can’t just wait for you to notice, because more messages are arriving behind it every second. You need a dead-letter queue for records that fail to parse, backpressure handling for when your consumer falls behind the producer rate, and monitoring that tells you the moment lag starts climbing rather than the moment someone notices the dashboard looks wrong. None of this is exotic, it’s standard message-queue architecture, but it’s real work that a batch pipeline doesn’t require.

Schema drift is the other failure mode worth naming. Target sites change their markup, and extracted fields shift shape without warning. Batch pipelines catch this at the validation step before load, so a bad crawl produces a bad file you can inspect and rerun. Streaming pipelines that skip schema validation at the point of ingestion end up with malformed records already sitting in the sink, requiring backfill and cleanup after the fact.

Cost and infrastructure shape

Batch infrastructure scales with how often you run it. A crawl-and-load job that runs four times a day pays for compute four times a day. Between runs, nothing is provisioned, nothing is listening, nothing needs a health check.

Streaming infrastructure runs continuously by design. A Kafka cluster, or even a managed queue service, is a standing cost whether or not records are flowing through it at full volume. Consumers need to be always-on or need their own scaling logic to spin up under load. This isn’t a reason to avoid streaming, but it’s a real operating cost that needs to be weighed against the value of low-latency data, not assumed away because “it’s more scalable.”

There’s also a proxy and request-pattern angle worth thinking through. A batch crawl tends to run in a concentrated burst, hit a target list hard for a defined window, then go quiet until the next scheduled run. A streaming setup that continuously polls or continuously crawls produces a steady, ongoing request pattern instead. Anti-bot systems commonly use request cadence and timing regularity as one signal among many when profiling traffic, alongside header fingerprints, TLS characteristics, and behavioral patterns. A perfectly even, unbroken polling interval is a distinguishing shape that detection systems are built to notice. This is a design constraint to be aware of, not something to route around: it means continuous ingestion pipelines need honest monitoring for block rates and response anomalies over time, since a steady pattern that starts degrading is a signal worth catching early, not something to paper over.

The hybrid most teams actually land on

Pure streaming for scraped data is less common in practice than the marketing around “real-time pipelines” suggests. What most production setups actually run is micro-batching: workers push records to a queue as they’re extracted, but the consumer reads in small batches every few seconds or minutes rather than processing one record at a time. This gets you most of streaming’s latency benefit (low, bounded delay) with a chunk of batch’s operational simplicity (bulk writes, easier validation, fewer per-record failure paths to manage).

Tools like Kafka Connect, or simpler cron-triggered consumers reading off a queue table, sit comfortably in this middle ground. If you’re deciding where to start, this is usually the more defensible default: build the pipeline as batch first, prove out your schema and validation logic, then move to micro-batch or true streaming only once you have a concrete latency requirement that batch can’t meet.

Choosing without guessing

Ask three questions before picking a model. How stale can the data be before it’s useless to whoever consumes it. What does a malformed record cost you if it slips through, and can you afford to inspect the whole batch before committing it, or does data need to land regardless. And what’s the actual request pattern this creates against the target site, and are you set up to monitor it honestly over time.

Batch is the right default for most scraping work: periodic pricing snapshots, ranking trackers, dataset builds, anything where a report or a table gets read later rather than watched live. Streaming earns its complexity when a human or a downstream system is genuinely waiting on the data the moment it changes. Most pipelines that think they need streaming actually need better batch scheduling.

If you’re building out scraping infrastructure and want more breakdowns like this on pipeline design, proxy management, and how detection systems actually work, check out the rest of what we cover here.

Get new guides and videos first — join the Telegram channel.

Most scraping estimates only price the script. Someone quotes a client or a boss “a few days to write the scraper” and treats that as the project cost. It isn’t. The script is the cheapest part of a scraping project over its lifetime. The expensive parts are the ones nobody prices up front: proxy bandwidth, retry overhead from anti-bot systems, and the ongoing labor of keeping parsers working after the target site changes its markup for the fifth time this year.

If you run proxy infrastructure and production pipelines for a living, you learn to cost a scrape the way you’d cost any recurring infrastructure spend, not a one-off build. Here’s the framework I actually use before committing to a scraping project.

Start with request volume, not code

Before anything else, work out how many requests the job actually needs. This is arithmetic, not guesswork:

• Number of target pages or endpoints
• How often you need fresh data (once, daily, hourly)
• Time horizon (a one-time pull vs. an ongoing feed)
• Pagination and detail-page fan-out (a listing page that links to 40 product pages multiplies your count by 41)

A daily scrape of 10,000 product pages isn’t 10,000 requests a day. It’s 10,000 plus whatever category or search pages you crawl to discover those products, plus retries. That total, not the page count you started with, is what every downstream cost scales off.

Where proxy spend actually goes

Proxy cost is usually billed one of two ways: by bandwidth (per gigabyte) or by IP access (per port, per IP, or per concurrent connection). Which one dominates your bill depends on what you’re scraping.

Bandwidth-billed plans punish you for page weight. A product page with large images and heavy JavaScript bundles can be ten times the size of the JSON response you actually need. If you’re paying per gigabyte, rendering full pages in a headless browser when you only need three fields is an expensive way to get there. This is why serious pipelines separate “what does the page cost to fetch” from “what data do I need out of it,” and look hard at whether an API response, a mobile endpoint, or a lighter-weight fetch gets the same data for less transferred.

IP-billed plans punish you differently: you’re paying for access and rotation capacity, not for the megabytes moved. Here the cost driver is how many distinct IPs you need in rotation to keep request patterns from looking like a single source hammering a target, and how quickly those IPs need to be replaced when they stop working.

The tier of proxy you need (datacenter, residential, or mobile) is set by how the target site defends itself, not by preference. Datacenter IPs are cheap and fast but are also the easiest for a target to identify and block, because they come from ranges that don’t belong to consumer ISPs. Residential and mobile IPs look like real user traffic and survive longer against detection systems that block by IP reputation, but they cost more per IP or per gigabyte because that’s what it costs to source and maintain a pool of real consumer-network exit points. None of these tiers make you undetectable. They change your odds and your cost, not your risk to zero.

The retry tax

This is the line item most estimates miss entirely: not every request you send succeeds on the first try, and the gap between “requests sent” and “requests that returned usable data” is where a lot of budgets quietly blow up.

Sites that care about scraping traffic run detection systems that look at request rate, header and TLS fingerprint consistency, IP reputation, and behavioral signals like mouse movement or timing on JS-rendered pages. When a request trips one of those signals, the response you get back might be a rate-limit error, a CAPTCHA challenge, a soft-block page that looks like content but isn’t, or a silent redirect to a decoy. Every one of those is a request you paid bandwidth and IP-rotation cost for and got zero usable data back.

Cost that the way you’d cost defect rate on a production line. If your pipeline’s effective success rate on a given target is, say, seven successful pulls out of every ten attempts, your real request budget is your target volume divided by that success rate, not the raw target volume. A target that’s aggressive about detection can push that ratio hard enough that the “cheap” proxy tier ends up costing more per successful record than the expensive one, because you’re paying for three failed attempts for every one that lands. This is why comparing proxy providers on price-per-gigabyte alone is misleading. What matters is price per successful, usable response against the specific target you’re pulling from, and that number only exists after you’ve run traffic against that target, not before.

Parsing maintenance is a subscription, not a one-time cost

The scraper you ship on day one breaks on a schedule set by the target site, not by you. Layout changes, A/B tests, new JS bundlers, and field renames all quietly break selectors. A site that redesigns its product pages twice a year will cost you two rounds of parser maintenance a year, indefinitely, for as long as the pipeline runs. Budget this as ongoing engineering time, not a bug you fix once and move past. If nobody is assigned to watch for silent parsing failures (a scraper that runs “successfully” but returns empty or malformed fields), you’ll ship bad data for weeks before anyone notices, which is its own cost.

Storage, dedup, and compute

These are smaller than proxy and maintenance cost for most projects, but they’re not zero. Raw HTML or JSON snapshots add up fast if you’re keeping history rather than just the latest scrape. Deduplication logic (deciding whether a record you just pulled is actually new or just an unchanged re-crawl) needs to run somewhere, and if you skip it you pay storage cost for redundant data on every cycle. Orchestration, whether that’s a scheduler, a queue, or a fleet of worker processes, needs compute that scales with your request volume, and that compute cost tracks your retry rate too: every retried request is CPU and memory spent twice.

A simple model to run before you build anything

Put a number, even a rough one, on each of these before you write a line of scraper code:

1. Target requests per cycle. , pages plus discovery overhead
2. Success rate against this specific target. , informed by a small test crawl, not a guess
3. Effective requests needed. = target requests ÷ success rate
4. Proxy cost. = effective requests × cost per request (bandwidth or IP-rotation basis, whichever applies)
5. Build time. , one-time engineering cost
6. Maintenance time per cycle. , recurring, tied to how often the target changes
7. Storage and compute. , recurring, tied to volume and retention

Run a small test crawl against the actual target before finalizing this. A few hundred requests will tell you your real success rate far better than any vendor’s marketing page will, and that single number changes every downstream calculation.

When the math says don’t build it

Sometimes this exercise ends with “buy the data instead.” If a target has an official API, even a paid one, compare its cost per record against your effective scraping cost once you’ve priced in retries and maintenance. If a data vendor already licenses the dataset you need, compare that subscription against your build-plus-maintain total over the same time horizon. Scraping personal data, paywalled content, or anything outside a site’s terms isn’t part of this comparison at all, it’s simply off the table. For everything else, building your own pipeline usually wins when you need custom fields, high frequency, or long-term control over the data. It loses more often than people expect once retry overhead and ongoing maintenance are priced in honestly.

Cost the project like infrastructure, not like a script, and you’ll know before you write any code whether it’s actually worth building.

If you want more breakdowns like this on scraping infrastructure, proxy economics, and the anti-bot systems that shape them, you can find the rest of our writing and videos on the home page.

Get new guides and videos first — join the Telegram channel.

Discovery is the expensive part

Most people budget their scraping costs around the fetch: proxy bandwidth, request volume, maybe headless browser time if the site renders client side. That’s real cost, but it’s not usually the biggest one. The biggest one is discovery, figuring out which URLs exist on a site in the first place.

If you’re crawling a site the naive way, you start at the homepage, extract links, queue them, fetch each page, extract more links, dedupe, repeat. Every one of those discovery hops is a request that burns a proxy IP and rotation budget, and it’s a request that produces zero business value on its own. You’re not extracting data, you’re just finding out where the data might be. On a site with deep pagination, faceted category pages, or infinite scroll, discovery can dwarf the actual extraction traffic.

Sitemaps and feeds skip that phase almost entirely. A site that publishes a sitemap has already done your discovery work and handed you the URL list in one flat file. That’s the whole pitch, and it’s why sitemap scraping is worth building into any pipeline before you write a single crawler rule.

What a sitemap actually contains

A standard XML sitemap is a urlset with a loc entry per URL, plus optional lastmod, changefreq, and priority fields. lastmod is the one that matters operationally, it’s an ISO 8601 timestamp the site claims the page was last modified.

Large sites don’t ship one flat sitemap, they ship a sitemap index: a file whose entries point to other sitemap files instead of pages, often split by content type or date range (sitemap-products-2026-01.xml, sitemap-products-2026-02.xml, and so on). Your parser needs to handle both shapes: fetch the index, recurse into each child sitemap, then flatten to a URL list. Files are frequently gzipped (.xml.gz), so decompression is part of the pipeline, not an edge case.

None of this is exotic. It’s the same protocol Google, Bing, and every other search crawler consumes, which is exactly why it exists: sites publish sitemaps so crawlers don’t have to guess their structure.

Where to find them

Check robots.txt first. It’s supposed to contain a Sitemap: directive pointing to the canonical location, and well-behaved crawlers read it from there rather than guessing. If it’s not listed, /sitemap.xml and /sitemap_index.xml are the common conventions, but they’re conventions, not guarantees. Some CMS platforms nest sitemaps under /sitemap/ or generate per-section files with non-obvious names, and some sites don’t publish one at all. Absence of a sitemap isn’t a signal of anything, it just means you’re back to link-based discovery for that target.

Feeds cover the other half

RSS and Atom feeds do a related but different job. A sitemap is a snapshot of everything that exists. A feed is a stream of what’s new, usually capped at the last 10 to 50 items. For news sites, blogs, changelogs, and job boards, that’s the more useful primitive: instead of re-crawling the whole site to check for new posts, you poll the feed and only see what’s actually changed since your last read.

JSON Feed is a smaller but growing alternative to RSS/Atom, same idea, easier to parse since it’s just JSON instead of XML namespaces. If a site offers both a sitemap and a feed, they’re not redundant, use the sitemap for full-catalog backfill and the feed for ongoing monitoring.

The real saving: incremental fetching

Here’s where sitemaps stop being just a convenience and start being a cost control. Once you’ve done a full sitemap parse and stored the URL list alongside each lastmod value, every subsequent run is a diff, not a full crawl. You compare the new sitemap’s lastmod values against what you stored last time, and only enqueue URLs that actually changed.

That’s the difference between refetching ten thousand pages every night and refetching the two hundred that were actually updated. Every page you don’t have to fetch is bandwidth you don’t spend, proxy rotations you don’t burn, and requests that can’t get you rate limited on the destination site. In a pipeline where proxy cost scales with request volume, this is usually the single biggest lever available, bigger than switching proxy providers or tuning concurrency.

The caveat: lastmod is self-reported by whoever generates the sitemap. Some CMSs set it accurately from the database’s updated_at column. Others just stamp the current time on every generation run regardless of whether anything changed, which makes the field useless for diffing on that particular site. You find this out by observing the field over a few runs, not by trusting it on day one. Treat it as a hint that needs validation, not a contract.

What sitemaps don’t solve

A sitemap gives you a URL and maybe a timestamp. It doesn’t give you the data on the page. You still have to fetch each URL and parse the response, with all the normal considerations that come with that: rendering if the content is client side, proxy rotation if the target rate limits or blocks based on IP reputation, and respecting whatever the site’s terms of service and robots.txt disallow rules say about that specific path.

It’s also worth being clear about what a sitemap is not: it’s not a bypass for content the site has deliberately not made public. Paywalled articles, gated account pages, and personal data behind authentication generally don’t show up in public sitemaps precisely because the site doesn’t want them indexed or crawled. If a page isn’t in the sitemap and isn’t linked from anywhere public, that’s a signal about the publisher’s intent, not a gap to route around.

Where bot detection still applies

Fetching the sitemap file itself is close to risk free from a detection standpoint. It’s a static file that the site published specifically for automated consumption, and server logs are already full of Googlebot, Bingbot, and a dozen SEO tools hitting it constantly. A WAF or bot management layer has little reason to flag a GET request to /sitemap.xml.

That protection ends the moment you start fetching the URLs listed inside it. If a sitemap index contains fifty thousand product pages and you fetch all of them back to back from one IP, you look exactly like a scraper doing bulk extraction, because that’s what’s happening. The site’s rate limiting, WAF rules, and fingerprinting systems evaluate that traffic on its own terms: request rate, header consistency, TLS fingerprint, and behavioral patterns like sequential ID access. Sitemaps make discovery cheap, they don’t make fetching invisible. Any pipeline that treats a sitemap URL list as an excuse to hammer a site at full concurrency will run into the same defenses as any other undisciplined crawler.

Building it into a pipeline

The concrete flow looks like this: fetch robots.txt, extract the Sitemap: line, fetch that file, check if it’s an index or a urlset, recurse if it’s an index, decompress if gzipped, parse loc and lastmod out of every entry, diff against your last stored state, and hand the changed URLs to your normal fetch layer with whatever proxy rotation, rate limiting, and rendering logic you already run.

Most scraping frameworks have this half built already. Scrapy ships a SitemapSpider that handles index recursion and gzip out of the box, and it’s a reasonable starting point to read even if you end up writing your own parser for a specific pipeline’s storage format.

Sitemap and feed parsing won’t replace the rest of your scraping stack. It replaces the part of it that was never adding value in the first place, the part where you were paying proxy and compute cost just to figure out where the pages were.

If you’re building out a pipeline and want more on structuring the fetch layer, proxy rotation, or the frameworks that handle this well, check out the rest of Data Research Tools.

Get new guides and videos first — join the Telegram channel.

What backpressure actually means

Backpressure is what happens when a system tells its upstream producer “slow down, I can’t keep up.” In a scraper pipeline, the producer is usually something enqueueing URLs to fetch, and the consumers are worker processes pulling from that queue to make requests, parse responses, and write results somewhere. If the producer keeps adding work faster than the workers can drain it, something downstream has to absorb the difference. Without backpressure, that something is usually your RAM, and RAM always loses.

This isn’t an abstract concern. Any pipeline that separates “decide what to scrape” from “go scrape it” has this problem built in, because those two steps rarely run at the same speed. A URL discovery step (crawling sitemaps, paginating a search API, expanding a seed list) can produce tens of thousands of URLs in seconds. Actually fetching each one involves a network round trip, a proxy handoff, and often a deliberate delay to avoid tripping rate limits. The mismatch is structural, not a bug you can code away.

Why scraping pipelines hit this harder than most systems

A typical web service has a fairly predictable consumer: a database or an internal API with known latency. A scraping pipeline’s consumer is the open internet, through a proxy pool, hitting targets that actively rate-limit, throttle, or block you. That means your effective consumption rate isn’t constant. It can drop suddenly when a target site tightens its rate limiting, when a proxy pool’s healthy IP count shrinks, or when a target starts returning more challenge pages that need to be retried.

If your producer doesn’t know any of that is happening, it keeps enqueueing at the same rate it always has. The queue grows. If that queue lives in process memory (a Python list, an in-memory asyncio.Queue with no maxsize, a Go channel with no buffer limit), you get an out-of-memory kill with no warning beyond a slow memory graph nobody was watching. If the queue lives in Redis or RabbitMQ, you don’t crash, but you do build a backlog that takes hours to work through even after the underlying slowdown resolves, because now you’re paying for both the current work and the debt.

Bounded queues are the first fix, not an optimization

The simplest form of backpressure is a bounded queue: give it a max size, and make the producer block (or explicitly handle rejection) when it’s full. A Python asyncio.Queue(maxsize=5000) will suspend the producer coroutine on await queue.put(item) once it hits capacity, and resume it only as workers pull items off. A buffered Go channel does the same thing with make(chan Job, 5000). Celery, RQ, and most Redis-backed task queues expose similar concepts through queue length checks and worker prefetch limits.

The reason this matters more than it sounds is that an unbounded queue isn’t neutral, it’s a decision to let memory become your buffer of last resort. Bounding the queue forces the slowdown to happen where you can see it and reason about it, instead of silently in the OS memory allocator. When the queue is full and the producer is blocked, that’s your pipeline telling you, correctly, that fetching is currently the bottleneck.

Worker pool sizing has a ceiling set by the proxy layer, not your CPU

It’s tempting to size a worker pool by what your machine can handle: number of cores, memory available, that kind of thing. For scraping, the real ceiling is usually somewhere else entirely: how many concurrent connections your proxy pool can sustain per target domain without pushing your error rate up, and how much concurrent load a target’s rate limiting will tolerate before it starts serving 429s or CAPTCHAs instead of real pages.

Running 200 workers against a target that only tolerates 20 concurrent connections per IP, spread across a proxy pool with a limited number of healthy exits, doesn’t get you more throughput. It gets you more of your requests coming back as errors, more retries, and a queue that looks busy but isn’t actually making progress. This is where a lot of “why is my scraper slow” debugging ends up: the bottleneck isn’t the worker count, it’s that the workers are being throttled and the pipeline doesn’t have a mechanism to notice and adjust.

A worker pool that’s coupled to real backpressure signals (queue depth, per-domain error rate, proxy pool health) can scale itself down when the target or the proxy layer is struggling, and back up when things recover. A fixed worker count can’t do either.

Signals worth listening to

A few concrete signals tend to be useful for deciding when to slow a scraper pipeline down:

• Queue depth relative to a high watermark. If the queue is consistently near its cap, you’re producing faster than you consume, and either need more consumer capacity or a slower producer.
• Per-domain or per-proxy error rate. A rising rate of timeouts, connection resets, or non-200 responses from a specific target or through a specific proxy is usually a much earlier signal than a full queue.
• Retry queue growth. If items are landing back in a retry queue faster than they’re being drained, that’s the pipeline quietly losing ground even while the main queue looks fine.
• Proxy pool health. A shrinking count of proxies passing health checks means your effective concurrency ceiling just dropped, whether or not the worker pool config changed.

None of these signals are useful in isolation, but a producer that checks queue depth and per-domain error rate before deciding whether to enqueue more work is doing real backpressure, not just hoping the queue never fills.

Retries, backoff, and dead letters need to cooperate with backpressure

Retry logic and backpressure interact in a way that’s easy to get wrong. If a request fails and gets immediately re-enqueued, and that happens across thousands of items during a target-side slowdown, the retry logic itself becomes a second producer piling more load onto an already-struggling consumer. Exponential backoff on retries (waiting longer between each successive retry of the same item) helps, but it needs to be paired with a hard cap on retry attempts and a dead-letter queue for items that keep failing, so they stop cycling back through the main pipeline indefinitely.

Without a dead-letter path, a batch of URLs that started returning consistent errors (a changed page structure, a blocked proxy range, a target that started geo-blocking a region) will keep consuming worker time on every retry cycle, which is capacity taken away from work that could actually succeed.

A pipeline shape that tends to hold up

In practice, a scraper pipeline that survives real-world target and proxy variability usually has these pieces, in this relationship:

1. A producer that checks queue depth (and ideally per-domain error rate) before enqueueing new URLs, rather than enqueueing on a fixed schedule.
2. A bounded queue, sized to hold a few minutes of work, not the entire day’s crawl.
3. A worker pool whose concurrency is tuned to the proxy pool’s sustainable per-domain rate, not to available CPU.
4. A separate retry queue with capped, backed-off retries and a dead-letter destination for items that exhaust their attempts.
5. Monitoring on queue depth, error rate, and proxy pool health, because any one of them can be the actual constraint at a given moment.

None of this is exotic. It’s the same producer-consumer discipline that shows up in any queue-backed system. What’s specific to scraping is that the consumer’s real capacity moves around constantly, driven by targets and proxy infrastructure you don’t control, which is exactly why the backpressure has to be measured and reactive rather than assumed.

If you’re building out pipeline infrastructure and want more breakdowns like this on queues, proxies, and the operational side of scraping, check out the rest of what we cover at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The first time a pipeline that’s been running clean for weeks suddenly returns nothing but 403s, the instinct is to reach for a bigger proxy pool. That’s usually the wrong first move. A block is a symptom, not a diagnosis, and different causes need completely different responses. Rotating IPs against a fingerprint-based block does nothing except burn IPs. Slowing your request rate against a hard IP ban does nothing except waste time. You need to know which system actually fired before you decide what to change.

This isn’t a guide to getting past anything. It’s a guide to reading what happened, because most operators skip that step and end up guessing.

A block is rarely one thing

Modern anti-bot stacks (things like Cloudflare, Akamai, PerimeterX/HUMAN, DataDome, or an in-house WAF) layer several independent checks. A request can fail at any of them, and each layer produces a different symptom if you know where to look:

• Network-level reputation. The IP itself is flagged, either because it’s a known datacenter range, a proxy ASN, or has recent abuse history against that target or others behind the same protection vendor.
• TLS and HTTP fingerprinting. The TLS handshake (cipher order, extensions, JA3/JA4 hash) and the HTTP header order/casing reveal the client library. requests, curl, and headless Chrome all leave distinct signatures well before any content is fetched.
• Browser and device fingerprinting. For JS-rendering targets, canvas rendering, WebGL parameters, font enumeration, and navigator properties get hashed into a fingerprint that’s checked for consistency and for known-automation markers (webdriver flags, missing plugins, unusual screen/timezone combos).
• Behavioral scoring. Mouse movement, scroll patterns, timing between actions, and request cadence feed a session-level trust score that decays with mechanical, too-regular behavior.
• Rate and pattern limits. Even a fully human-looking session gets throttled or blocked if it requests URLs in a pattern no real user would, like walking a sitemap sequentially at high frequency.

A single block event can be any one of these, or a combination. Treating them as interchangeable is the single most common mistake in production scraping.

Read the response before you change anything

Before touching your scraper’s config, pull the actual response, not just the status code.

A flat 403 with no body, returned instantly, usually means the request never reached the origin. That’s a network or TLS-layer rejection, often based on IP or fingerprint reputation, and it happened at the edge.

A 200 that returns a CAPTCHA page, an interstitial “checking your browser” page, or a JS challenge means you passed the network layer but tripped a behavioral or fingerprint check that wants further proof.

A 429 with a Retry-After header is the most honest response you can get. It’s a rate limiter telling you exactly what it wants: fewer requests, spaced further apart.

Intermittent failures, where some requests succeed and others from the same IP fail, point at session or behavioral scoring rather than a hard IP ban. Full failure of every request from a given exit node, with successes from other nodes, points at IP reputation.

Check response headers and timing too. A block that arrives in single-digit milliseconds is almost always an edge/CDN decision, not something the origin application server computed.

Why “just add more proxies” often doesn’t help

This is where running actual proxy infrastructure changes how you think about the problem. A clean, high-reputation residential or mobile IP absolutely helps against network-layer blocks, that’s the whole value proposition of that kind of infrastructure. But it does nothing against a TLS fingerprint mismatch, a JS-challenge that your HTTP client can’t execute, or a behavioral score built on a session that clicked through five pages in 400 milliseconds. Swapping the IP on a broken client just gets you a slightly longer runway before the next layer catches the same session.

The proxy is infrastructure for identity and geography. It isn’t a substitute for a client that presents a coherent, self-consistent fingerprint, and it isn’t a substitute for request patterns that look like they came from a real workflow instead of a for-loop.

If you’re seeing blocks and your first move is always “rotate the pool,” it’s worth checking whether you’re solving the layer that’s actually failing, or just the layer that’s easiest to change.

When the honest answer is to stop and reassess

Not every block is a technical puzzle to work around. Some are a signal about whether the approach is right at all.

If a target’s terms of service explicitly prohibit automated access and you’re scraping it anyway, a block is the site doing exactly what it said it would do. Pushing harder against that is a decision with legal exposure, not just an engineering one, and it’s worth having someone who understands the relevant terms and jurisdiction look at it before continuing.

If the data sits behind a login or paywall, a block there isn’t a technical obstacle, it’s an access-control boundary. Scraping around it, rather than through a licensed data feed or a paid account with actual usage rights, changes the nature of what you’re doing.

If the site publishes an official API, even a rate-limited or paid one, that’s usually the better path long-term. It’s slower to set up and sometimes costs money, but it doesn’t degrade every time the target ships a new detection update, and it doesn’t put you in an arms race you didn’t choose to be in.

None of this means every block is a moral question. Plenty of scraping is entirely legitimate, price comparison on public product pages, aggregating public listings, monitoring your own competitors’ public marketing copy. The point is that “we got blocked” is a good moment to check which category the job falls into, not just to reach for a workaround.

Building pipelines that fail informatively

Whatever the target, the thing worth investing in is a pipeline that tells you which layer failed instead of just returning empty results. Log the status code, the response body length, the timing, and whether a challenge page was served, per request, per proxy exit, per target. Over a few thousand requests, that data tells a clear story: is failure concentrated on specific IPs, specific times of day, specific request patterns, or is it flat across everything (which usually means a fingerprint or account-level signal, not a network one).

That diagnostic layer is worth more than any single technique for getting past a block, because blocks change. A detection vendor ships an update, a target adds a new WAF rule, and whatever worked last month stops working with no warning. A pipeline that can tell you “this failure mode changed” is durable. A pipeline tuned to beat one specific check is not.

If you’re building or operating scraping infrastructure and want the engineering side explained without the vendor hype, the rest of the site covers proxy infrastructure, orchestration, and how detection systems actually work under the hood.

Get new guides and videos first — join the Telegram channel.

The default nobody questions

Almost every pipeline starts the same way. You need fresh data from some source, so you write a loop that checks it every 30 or 60 seconds, and you ship it. It works on day one. It’s also the single most common source of wasted infrastructure spend we see when we look at how scraping and data pipelines are actually built.

Polling isn’t wrong. It’s just a default that a lot of teams never revisit, even after the source they’re checking makes it clear it doesn’t change that often, or offers a better way to be told.

What polling actually costs

Polling means you ask “did anything change?” on a fixed schedule, whether or not the answer is yes. Run that against an API or a page every minute and you make 1,440 requests a day per source. If the underlying data changes twice a day, 1,438 of those requests were wasted round trips.

That waste shows up in a few concrete places:

• Rate limits. Most APIs cap requests per minute or per day. A polling loop that’s too aggressive burns through that budget on empty checks, leaving no headroom when you actually need to pull something on demand.
• IP and session exposure. If you’re pulling data through proxies because the source rate-limits or blocks by IP, every poll is a request that has to look legitimate. A tight polling interval run across many targets from the same pool of IPs increases the request volume those IPs generate, which is exactly the kind of pattern that fingerprinting and rate-limiting systems are built to notice. This is a defensive concern for the site being polled, not a trick to route around, but it’s a real cost on your side too: more requests means more exposure surface for infrastructure you’re paying to keep quiet.
• Latency you didn’t need. If a change happens right after your last poll, you don’t find out until the next interval. Tighten the interval to reduce that lag and you multiply the wasted-request problem. You’re always trading freshness against load.
• Compute and storage for no-op checks. Every poll response still has to be parsed, diffed against the last known state, and logged somewhere, even when nothing changed. That’s real CPU and real storage spent confirming the absence of news.

None of this means polling is a bad architecture. It means polling has a cost curve that gets steep fast once you’re watching more than a handful of sources, or once freshness requirements tighten.

What a webhook actually buys you

A webhook flips the direction of the request. Instead of you asking the source “anything new?” on a timer, the source calls a URL you control the moment something happens. You register an endpoint, the source’s system fires an HTTP POST to it when an event occurs, and your endpoint does whatever it needs to do with that payload.

The win is straightforward: you make zero requests when nothing is happening, and you find out about a change within seconds of it occurring instead of within one polling interval. For a pipeline watching order events, price changes, or content updates across many sources, that’s often the difference between a system that scales linearly with the number of sources you watch and one that scales with the number of events that actually happen, which is almost always a much smaller number.

Webhooks come with their own engineering requirements, and skipping them is where a lot of “we added webhooks” projects quietly fail:

• Signature verification. A webhook endpoint is a public URL. Anyone who finds it can POST to it. Legitimate providers sign their payloads (typically an HMAC in a header) so your endpoint can confirm the request actually came from the source and wasn’t forged or replayed by someone else.
• At-least-once delivery. Most webhook systems retry on failure, which means your endpoint can receive the same event twice. If your handler isn’t idempotent, a retried webhook can double-process an order or duplicate a record. The fix is to key every incoming event on its unique ID and check whether you’ve already handled it before doing anything with it.
• Endpoint uptime. If your receiving endpoint is down when the event fires, you’re relying entirely on the source’s retry policy to eventually redeliver it. Some sources retry for hours, some give up after a few attempts and never tell you.
• No backfill. A webhook tells you about the future. If your endpoint was down, or you’re standing up a new pipeline and need last month’s events, webhooks don’t help. You still need a polling or bulk-export path for catch-up.

Streaming: the step past webhooks

Webhooks are still request-response, one event, one HTTP call, one endpoint. Streaming protocols (server-sent events, WebSocket feeds, or a message broker like Kafka or a managed queue sitting between the source and your consumers) go further: they hold a persistent connection or a durable log, and they can push a continuous sequence of events without the overhead of a new HTTP handshake per event.

The practical difference shows up at volume. A source pushing a handful of events a day is fine as individual webhook calls. A source pushing thousands of events a second (order books, sensor telemetry, high-frequency price feeds) needs a channel built for sustained throughput, backpressure, and ordering guarantees, which is what streaming infrastructure is designed for and plain webhook delivery isn’t.

If you’re building the ingestion side of a scraping or data pipeline and the source only offers a page to scrape, none of this is available to you directly, but the same principle still applies internally: once your scrapers pull data, feed it into a queue rather than writing straight to a database from every scraper process. That gives you the backpressure handling and retry semantics of a streaming architecture even when the original source is stuck on HTTP responses.

When polling is still the right call

Polling remains the correct choice more often than webhook advocates like to admit:

• The source doesn’t offer webhooks or a stream at all. A lot of scraping targets never will, because they weren’t built as an API for a client, they’re a website for a browser.
• The data changes slowly and predictably (daily pricing, weekly inventory snapshots). A daily poll costs almost nothing and adds no architectural complexity.
• You need a point-in-time snapshot for a report or an audit, not an ongoing feed.

The engineering move that actually helps here isn’t polling harder, it’s polling smarter. Conditional requests (sending an If-Modified-Since or If-None-Match header and getting a cheap 304 back when nothing changed) cut the cost of a no-op poll to almost nothing on sources that support HTTP caching semantics. Exponential backoff after repeated empty responses stretches your interval automatically when a source has gone quiet, then tightens back up when activity resumes. Both are a fraction of the engineering effort of standing up webhook infrastructure, and both meaningfully reduce request volume without touching your architecture.

Building a pipeline that doesn’t care which one it gets

The sources you’re pulling from won’t standardize for your convenience. Some will offer webhooks, some will offer a stream, most scraping targets will offer neither. A pipeline built to depend on one delivery mechanism breaks the day you add a source that only supports another.

The fix is to normalize at the ingestion boundary: every source, whether it arrives via webhook, stream, or polled scrape, gets converted into the same internal event format and pushed into the same queue before anything downstream touches it. Your processing logic, your deduplication, your storage layer, none of it should know or care whether the event that just arrived was pushed to it or pulled by it. That separation is what lets you add a webhook-based source next to a polled one without rewriting the consumer side, and it’s the difference between a pipeline that scales by adding sources and one that needs a rewrite every time a new source shows up with a different delivery model.

Polling isn’t a mistake. Polling forever, on every source, without ever checking whether a better option exists, is the actual cost center. Worth an afternoon auditing which of your sources could drop their polling interval to near zero if you just asked them for a webhook.

For more breakdowns like this on pipeline architecture, proxy infrastructure, and how scraping systems actually get built and defended in production, visit the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.

Why connection setup matters more than people think

Most scraping post-mortems focus on the request layer: headers, user agents, rate limits. Fewer people look one layer down, at what happens before a single byte of HTTP is exchanged. Every HTTPS request starts with a TCP handshake and a TLS handshake, and at scale, that setup cost is often bigger than the cost of the actual HTTP exchange. If you’re running a pipeline that fires thousands of requests an hour through a proxy pool, the difference between negotiating a fresh TLS session on every request and reusing one is a real, measurable chunk of your latency and CPU budget.

This isn’t a bypass technique. It’s plumbing. But plumbing has consequences for both performance and for how a target site’s defenses read your traffic, so it’s worth understanding the mechanics rather than treating requests.get() as a black box.

What a TLS handshake actually costs

A full TLS 1.2 handshake takes two round trips before any application data moves: client hello, server hello plus certificate plus key exchange, then the client’s key exchange and finished message. TLS 1.3 collapsed that to one round trip in the common case, and added a 0-RTT mode for resumed connections where the client can send application data in its very first flight, though 0-RTT has its own replay-risk caveats that most server operators disable or restrict.

On top of the round trips there’s real CPU work: asymmetric key exchange (typically ECDHE), certificate parsing and chain validation, and key derivation. None of this is expensive for a single request. It becomes expensive when you’re doing it tens of thousands of times a minute across a fleet of proxy exit nodes, especially if each exit node is a resource-constrained box like a mobile modem gateway rather than a beefy cloud VM.

Session resumption: session IDs, session tickets, and PSK

TLS has had a resumption mechanism since TLS 1.0 in the form of session IDs: the server caches session state keyed by an ID it hands the client, and the client can present that ID on a later connection to skip the full handshake. Session tickets (RFC 5077) moved that state out of server memory and into an encrypted blob the client stores and replays, which scales better across a load-balanced server fleet since no shared session cache is needed.

TLS 1.3 folded both mechanisms into a single PSK (pre-shared key) resumption model. After a full handshake, the server can issue one or more session tickets. On a later connection, the client offers a ticket, and if the server accepts it, the handshake becomes an abbreviated one-round-trip (or 0-RTT) exchange instead of the full negotiation.

For a scraper, the practical effect is: if your HTTP client reuses a TLS session across requests to the same host, you pay the full handshake cost once and then get cheap resumed handshakes for a while, until the ticket lifetime expires. If your client (or your proxy layer) tears down and rebuilds the TLS connection on every request, you’re paying full handshake cost every single time, which shows up as higher latency and higher CPU load on both ends of the connection.

HTTP/2 multiplexing and why connection reuse compounds the win

HTTP/1.1 scraping code often opens one connection per request, or relies on a small keep-alive pool. HTTP/2, negotiated via ALPN during the TLS handshake, changes the shape of the problem: a single TCP+TLS connection carries multiple concurrent streams, each an independent request/response pair, multiplexed over one wire with HPACK header compression to cut repeated header overhead.

That means the payoff from TLS session reuse and HTTP/2 connection reuse stack. Instead of N connections each paying handshake cost, you get one connection amortizing its handshake cost across N logical requests. For a pipeline scraping paginated listings or repeated API calls against the same host, this is often the single biggest lever for reducing wall-clock time and reducing the load you place on the target’s server, which matters both for your own throughput and for not being the reason a site’s ops team starts rate-limiting a subnet.

The tradeoff is operational complexity. A connection pool keyed by host, with sane idle-timeout and max-requests-per-connection settings, is more code than “open a socket per request,” and it requires your scraping framework and your proxy layer to actually agree on connection lifetime. If your proxy silently rotates the upstream IP mid-session, or your HTTP client and your proxy pool have different ideas about when a connection is stale, you can end up with confusing failures that look like server-side blocking but are actually your own connection-reuse logic falling over.

The fingerprinting side, explained defensively

Connection-level behavior is also one of the harder things to control, which is exactly why it’s used defensively by anti-bot systems. A TLS ClientHello carries the cipher suite list, extension list and order, supported groups, and signature algorithms in a specific sequence. That sequence is fingerprintable (JA3 is the best-known hash of this), and most HTTP client libraries (a stock requests/urllib3 stack, a given version of Go’s net/http, curl-impersonate, browser TLS stacks) produce a distinctive, stable fingerprint. Bot-detection vendors compare an incoming ClientHello’s fingerprint against a known-bad list of automation libraries, and separately watch for a mismatch between what the TLS fingerprint claims (a browser) and what the HTTP layer reveals (missing browser-only headers, a User-Agent string that doesn’t match the negotiated TLS profile).

HTTP/2 adds its own fingerprint surface on top of TLS: the order and values of the SETTINGS frame, the WINDOW_UPDATE behavior, and the order of HTTP/2 pseudo-headers (:method, :authority, :scheme, :path) in a request. Real browsers have consistent, well-documented orderings here; many HTTP client libraries either don’t support HTTP/2 pseudo-header ordering configuration at all or use an order that doesn’t match any real browser, which is itself a usable signal for detection systems, independent of anything at the TLS layer.

It’s worth being precise about what this means and doesn’t mean. Matching a browser’s TLS and HTTP/2 fingerprint is not a way to make traffic “undetectable.” Detection systems that care about this layer generally combine it with other signals: request timing, IP reputation, behavioral patterns, and JavaScript-side checks that a plain HTTP client can’t answer at all. A consistent, resumed TLS session that behaves like a real browser at the connection layer removes one signal from the pile; it doesn’t remove the pile. Sites that invest in this kind of defense are explicitly trying to make automated traffic expensive to disguise, and no proxy or client configuration changes that fact.

What this means for pipeline design

The practical takeaway for anyone running a production scraping pipeline is to treat connection reuse as a performance and reliability problem first. Configure your HTTP client to negotiate ALPN for HTTP/2 where the target supports it, keep TLS sessions and HTTP/2 connections alive for a sensible pool of requests per host, and make sure your proxy layer’s session affinity (how long a given proxy IP stays assigned to a given session) is consistent with your HTTP client’s connection lifetime. Tearing down a TLS session on every request while paying for premium residential or mobile proxy bandwidth is a straightforward way to waste both money and time.

Do this with realistic expectations. Session reuse and protocol-correct behavior reduce operational cost and remove one class of easy tells. They are not a guarantee against blocking, and no vendor or configuration can promise that in good faith. Any site that has invested in bot detection is looking at more than your connection fingerprint, and treating this as a checklist item to “beat” detection rather than an infrastructure efficiency question is the wrong frame.

If you’re building or auditing a scraping stack and want to see how these pieces fit together with proxy infrastructure and orchestration, head back to the Data Research Tools home page for the rest of our tooling breakdowns and infrastructure guides.

Get new guides and videos first — join the Telegram channel.

Most scraping projects start the same way. Someone needs a dataset, writes a script, points it at a page, and gets a CSV. That part is easy and most tutorials stop right there. The problem is that a one-off scrape and a monitored feed are two different engineering problems wearing the same request library, and the gap between them is where most home-grown pipelines quietly die.

If you’ve ever had a scraper “work” for two weeks and then silently stop producing useful data, this is usually why. Nobody designed for the second run.

What actually changes between a script and a feed

A script answers one question: what does this page look like right now. A feed answers a harder question: what changed since the last time I looked, and can I trust that the absence of new data means nothing happened, rather than my collector failing quietly.

That second question forces a handful of decisions you can skip entirely when you’re just pulling data once:

• How do you know a run succeeded versus returned an empty page that looks structurally fine
• How do you store results so you can tell new records from ones you already have
• What do you do when the page’s HTML structure shifts under you
• Who gets told when the pipeline goes quiet, and how fast

None of these are exotic problems. They’re the same operational questions you’d ask of any recurring data pipeline. Scraping just adds a layer of fragility because you don’t control the source.

Idempotent runs before anything else

The first fix, and the cheapest one, is making each run idempotent. A run that appends blindly to a file or table will duplicate data the moment you rerun it after a partial failure, and partial failures are the normal case for a scheduled job, not the exception.

The practical pattern is to hash each record on a stable set of fields (not the whole row, since timestamps or scrape metadata shouldn’t be part of the hash) and use that hash as a natural key. Upsert on the hash instead of inserting blindly. This does two things at once: reruns after a crash don’t create duplicates, and you get a free signal for “this record didn’t change since last time,” which is the entire point of a monitored feed. You’re not scraping to collect rows anymore, you’re scraping to detect deltas.

Diffing is the actual product

Once you’re hashing records, the feed part of “monitored feed” becomes a diff operation, not a scrape operation. Each run produces a set of hashes. Compare that set against the last known set and you get three buckets: unchanged, new, and disappeared. Most people build alerting only on “new” and ignore “disappeared,” which is a mistake, because a page that used to list 40 items and now lists 3 is very often your collector breaking, not the underlying data actually shrinking.

This is the single highest-value check in the whole system: track row count per run over a rolling window and alert on a drop past some threshold, say more than half the trailing average. It catches selector breakage, layout changes, and soft blocks (pages that render but serve you a stripped-down or cached version) all with one cheap metric, before you ever have to look at the HTML.

Selectors will drift, plan the failure not the prevention

You cannot prevent a target site from changing its markup. What you can control is how loudly your pipeline fails when it does. The naive version wraps a selector in a try/except and moves on, which produces exactly the silent-failure pattern that makes people lose trust in their own data months later.

The better version separates “the request succeeded” from “the parse produced the shape I expect.” Validate the parsed output against a schema, even a loose one (these fields must be non-empty, this field must parse as a number, this list must have at least N items). When validation fails, don’t discard the run, quarantine it. Store the raw HTML or JSON response alongside a failure flag so you can diagnose what changed without having to reproduce the failure live. Re-fetching a live page to debug a break that happened three days ago rarely shows you the same thing.

Scheduling is an orchestration problem, not a cron problem

A single cron job calling a script works fine until you have more than a handful of targets, at which point you run into the real constraints: staggering requests so you’re not hammering every source at the same second, retrying failed runs with backoff instead of just trying again next cycle, and making sure a slow target doesn’t block the whole batch.

This is where actual orchestration (something like a task queue with retry and backoff semantics, or a scheduler that tracks job state rather than firing blind) earns its keep over a crontab full of shell scripts. The specific tool matters less than the properties: each job has a state you can query, failed jobs retry with increasing delay instead of retrying immediately into the same block, and one stuck job doesn’t starve the others.

The proxy question is about consistency, not evasion

A one-off scrape usually runs from a single IP because it runs once. A feed runs on a schedule, indefinitely, and that changes the shape of the problem entirely. Repeated automated requests from one address build a request pattern over time, and sites that care about scraping traffic are generally looking at pattern and volume more than any single request.

The operational reality of running proxy infrastructure for recurring collection is mostly about spreading load and matching request cadence to what a target would consider normal traffic for the page in question, not about hiding the fact that a request is automated. Rotating source IPs, respecting reasonable concurrency limits per target, and backing off when you see elevated error rates are all things you’d do for basic politeness and pipeline stability even if detection weren’t a consideration at all. A feed that hits the same page every five minutes forever is going to look different from a browsing session no matter what IP it comes from, and infrastructure choices don’t change that.

What proxy rotation actually buys a monitored feed is resilience: if one exit IP starts returning degraded responses (rate-limited pages, CAPTCHAs, or thin content) you have a signal to route around it rather than a hard outage. That’s an operational health question, and it’s worth treating IP-level error rates as their own metric alongside row-count anomalies, because the two failure modes look different in your logs. A row-count drop with a clean HTTP 200 usually means the page changed. A spike in non-200 responses or CAPTCHA pages usually means something upstream is responding differently to your traffic, and the right response is to slow down and investigate, not to escalate.

Storage shape follows query shape

Decide early whether downstream consumers need the full history or just current state, because it changes your schema. An append-only table with a hash and a first_seen/last_seen pair supports both: current state is a query filtered to the latest hash per key, and history is just not filtering. Trying to retrofit history onto a table you built as upsert-only is a much worse day than deciding up front.

What “monitored” costs you in practice

None of this is free. A monitored feed needs somewhere to store run metadata, a place to send alerts, and someone who looks at those alerts. The honest tradeoff is that a one-off scrape takes an afternoon and a monitored feed takes ongoing attention, the same as any other production data pipeline. The payoff is that you stop finding out three weeks later that a feed has been silently returning empty results, which is the actual failure mode this whole design exists to catch.

If you’re building out this kind of pipeline and want to see how the orchestration, proxy, and monitoring pieces fit together in practice, there’s more on the home page.

Get new guides and videos first — join the Telegram channel.

Why this matters if you run scraping infrastructure

Anyone who runs a scraping pipeline at any real scale eventually stops thinking about “getting blocked” as a single event and starts thinking about it as a vendor. A 403 on one site, a CAPTCHA wall on another, a silent shadow-ban on a third, these usually trace back to one of a small number of companies whose entire business is telling websites which requests are human and which aren’t. Understanding what those companies actually sell, and how their systems work, changes how you read your own error logs. It also matters if you’re on the defending side, deciding what to put in front of your own site or API.

This is not a guide to getting past any of these systems. It’s an explanation of what they are, what signals they collect, and why a request that looks fine to you can still get flagged. That’s the part worth understanding whether you’re running proxies, building a scraper, or trying to keep bots off your own checkout page.

What an anti bot vendor actually sells

An anti bot vendor doesn’t sell a firewall rule. It sells a risk score. The product sits between the origin server and the internet, usually as a reverse proxy or a CDN-layer module, and for every request it computes a probability that the request came from a script rather than a person. That score then drives a decision: pass, challenge, throttle, or block.

The reason this became a standalone industry instead of something every site builds in-house is that the signal set needed to score traffic accurately keeps expanding and keeps going stale. A fingerprinting technique that worked in 2023 gets adapted around within months, so vendors run this as a continuously updated service, not a static ruleset. Cloudflare, Akamai, DataDome, HUMAN Security (formerly PerimeterX), Kasada, and Imperva are the names an operator runs into most often, each with a different mix of the techniques below and a different appetite for how aggressively to challenge borderline traffic.

Network and transport layer signals

Before a single byte of your request payload matters, the vendor’s edge has already looked at how the connection itself was made. TLS handshakes have a fingerprint, commonly referred to by the shorthand JA3 or the newer JA4, built from the cipher suites, extensions, and elliptic curves a client offers in a specific order. Real browsers produce a fairly narrow, predictable set of these fingerprints because there are only a handful of browser engines in wide use. HTTP libraries and headless automation frameworks often produce a fingerprint that doesn’t match any real browser release, or matches a browser version that’s inconsistent with the User-Agent header sent later in the request. That mismatch alone is a strong signal.

IP reputation sits alongside this. Vendors maintain databases scored on the history of an IP or subnet: how much traffic it’s sent, whether it’s a known data center range, whether it’s previously triggered CAPTCHAs or been associated with credential stuffing. This is exactly why residential and mobile IP space behaves differently from data center IP space in front of these systems, a data center /24 has no cover traffic, while a carrier-grade NAT range on a mobile network carries thousands of real subscribers behind it, which changes the baseline probability the system assigns to any single request from that range.

Browser and device layer signals

For traffic that reaches a JavaScript challenge, the vendor’s script runs in the client and collects a much deeper fingerprint: canvas rendering output, WebGL renderer strings, installed fonts, screen and viewport dimensions, audio context fingerprints, the exact set and order of navigator properties, and timing characteristics of how the browser executes certain operations. None of these individually proves automation. Together, compared against the enormous baseline of real browser sessions the vendor has observed, they form a distribution, and headless or automated browser environments tend to sit at the tails of that distribution rather than in the middle of it.

This is also where a lot of automation tooling gets caught not because it’s doing anything malicious, but because it’s internally inconsistent. A browser automation stack that reports itself as Chrome 124 on Windows but exposes WebGL renderer strings associated with a Linux software renderer is flagging a mismatch that a real user’s machine would never produce. Vendors build detection around these inconsistencies precisely because they’re cheap to check and expensive for automation frameworks to fully eliminate.

Behavioral signals

The layer above device fingerprinting is behavioral: how the mouse moves before a click, the timing and rhythm of keystrokes, scroll velocity, and the sequence of pages visited relative to what a human shopping or browsing pattern usually looks like. This is the piece that’s hardest to fake convincingly at scale, because it’s not a static value to spoof once, it’s a continuous stream that has to look statistically like a human across an entire session, not just at the login page. DataDome, HUMAN Security, and Kasada in particular lean heavily on this layer, running models that score the entire session rather than a single request.

Honeypots are a low-tech but effective companion to behavioral scoring: hidden form fields or links that are invisible to a real browser rendering the page normally but visible to something parsing raw HTML. Filling in a field a human would never see, or following a link that’s positioned off-screen, is close to a guaranteed automation signal because there’s essentially no legitimate way for a human to trigger it.

Risk scoring and challenge escalation

None of these signals work in isolation. Vendors combine them into a composite score, typically through machine learning models trained on enormous volumes of labeled traffic across their entire customer base, which is part of why the large vendors have an advantage smaller in-house solutions don’t: they see patterns across thousands of sites, not just one. A session that scores low risk passes silently. A session in the middle gets an interactive challenge, a CAPTCHA, a proof-of-work puzzle, or a JavaScript computation that has to complete correctly and within a plausible time window. A session that scores high risk gets blocked outright, sometimes with a generic error page that gives no indication a bot system was even involved, which is itself a deliberate design choice, giving less feedback makes it harder to iterate against the system.

A brief note on the major vendors

Cloudflare’s Bot Management is deployed at the edge across a huge share of the web simply because Cloudflare is a CDN first, so its bot product benefits from visibility into an enormous volume of baseline traffic. Akamai’s Bot Manager has a similarly broad footprint through its CDN business and has historically focused heavily on the network and TLS layer. DataDome and HUMAN Security market themselves specifically as bot and fraud protection specialists rather than CDNs, and lean hard into behavioral and session scoring. Kasada is known for aggressive JavaScript-based challenges that are deliberately expensive to solve programmatically. Imperva rounds out the group with a product that sits alongside its WAF offering. None of these should be described as unbeatable or as a guaranteed wall, and none of them should be described as weak, either. What each one actually stops depends on the traffic pattern, the target site’s configuration, and how much of the vendor’s default tuning the site operator has left in place versus customized.

What this means in practice

If you operate scraping infrastructure, the practical takeaway is that a block is rarely explained by one bad thing you did. It’s usually a combination: an IP with poor reputation, a TLS fingerprint that doesn’t match the declared browser, a session with no mouse movement before a form submission, or a request rate inconsistent with how a human browses. If you operate a site and you’re evaluating one of these vendors, the questions worth asking are about false positive rates on real users, latency added to legitimate traffic, and how much visibility you get into why a given session was challenged, not just whether the vendor’s marketing claims a high block rate.

We write about this side of the infrastructure regularly at Data Research Tools because we run proxy fleets and scraping pipelines ourselves and deal with these systems as a daily operational reality, not a theoretical one.

Read more breakdowns of scraping infrastructure, proxy setups, and bot detection systems on our site here.

Get new guides and videos first — join the Telegram channel.

Why catalogue scraping is a different problem

A single product page is easy. Fetch it, parse the DOM or the embedded JSON, done. Ecommerce scraping at catalogue scale is a different job entirely, because the target isn’t one page, it’s tens of thousands of pages that change price, stock, and structure on their own schedule, sitting behind infrastructure that’s specifically built to notice repeated automated traffic.

Most of the engineering effort in a real catalogue scrape doesn’t go into parsing. It goes into keeping a crawl running for days without falling over, without getting a growing share of its requests blocked, and without producing a dataset that’s already stale by the time it lands in a database. That’s the part that doesn’t show up in a “how to scrape with BeautifulSoup” tutorial.

Pagination and listing state

Product listings paginate in a few different ways, and each one changes how you have to crawl it.

Offset or page-number pagination (?page=4) is the simplest to walk but the least stable. If items are inserted or removed while you’re crawling, page boundaries shift and you either skip products or reprocess them. Cursor-based pagination (an opaque token pointing to “the next batch”) is more common on larger catalogues because it’s resilient to that kind of drift, but it means your crawl state has to persist the cursor, not just a page count, or a resume after a failure starts back at page one.

Infinite scroll is the hardest case, because the listing is built by client-side JavaScript firing XHR or fetch calls as the user scrolls, and the HTML you get from a plain HTTP request often just won’t contain the product grid at all. Handling that means either driving a real browser (Playwright or Puppeteer) to trigger those scroll events, or reverse engineering the underlying API calls the frontend makes and hitting those directly. The second option is far cheaper at scale, since a headless browser instance costs orders of magnitude more CPU and memory per page than a direct HTTP request, but it only works when the API responses are stable and not obfuscated behind session-bound tokens that expire quickly.

Variants, options, and the normalization problem

A “product” in an ecommerce catalogue is rarely one row of data. A t-shirt listing might expand into a dozen SKUs across size and color, each with its own price, stock count, and sometimes its own images. Some sites render every variant on the same URL and swap data via JavaScript when you pick an option; others give each variant a distinct URL or query parameter.

The practical effect: you can’t design a catalogue scraper around “one row per page.” You need a data model that separates the parent listing (title, description, category, brand) from the variant-level attributes (SKU, option values, price, availability), and you need to decide up front whether out-of-stock variants get scraped at all, since some storefronts stop rendering them in the DOM entirely once inventory hits zero. Get this wrong on day one and you end up re-crawling the whole catalogue later just to backfill a field you didn’t know you needed.

Proxy infrastructure for catalogue-scale crawls

Once a crawl runs past a few thousand requests to the same domain, IP-based rate limiting becomes the binding constraint, not your own bandwidth or CPU. This is where proxy choice actually matters, and it’s worth being specific instead of hand-wavy about it.

Datacenter IPs are cheap and fast, but they’re also the easiest class of IP for a site to flag, because datacenter ASNs are well known and easy to blocklist or throttle wholesale. Residential and mobile IPs route through real consumer ISPs, so they blend into normal traffic patterns in a way datacenter ranges don’t, which is why they’re the default choice for sustained catalogue crawls against sites with active anti-bot tooling. That doesn’t make them invisible. It changes the cost and detection profile, not the fact that a detection system exists on the other end.

Session behavior matters as much as IP class. Some catalogue crawls need a sticky session, same exit IP across a sequence of requests, when a site ties pagination cursors or A/B-tested category structures to a session cookie. Others benefit from rotating IPs per request specifically to avoid a single IP accumulating enough request volume to trip a velocity threshold. Geo-targeted proxy pools matter for catalogues with region-specific pricing or currency, since fetching from the wrong country’s exit IP silently gives you the wrong price, not an error you’d notice.

Concurrency has to be tuned per target, not set once globally. A crawl that’s polite by one site’s standards can still look like an attack to a smaller storefront with less request headroom. Backoff on 429 and 503 responses, honoring Retry-After headers when they’re present, and capping concurrent connections per domain are baseline engineering discipline here, not optional extras.

How the other side actually defends against this

It’s worth understanding catalogue defense from the site operator’s perspective, because it explains why brittle scrapers break the way they do.

Modern anti-bot systems don’t rely on a single signal. They combine IP reputation (is this ASN associated with datacenter or proxy traffic, has this IP hit us before) with TLS and HTTP fingerprinting (does the client’s negotiation order and header set match a real browser, or a library’s default), and with behavioral signals (mouse movement, scroll timing, request cadence that’s too regular to be human). A request that nails the User-Agent string but sends headers in the wrong order, or opens a raw TCP connection with a TLS fingerprint that doesn’t match any real browser, gets flagged regardless of what the User-Agent claims.

Rate anomaly detection looks at request velocity and pattern rather than any single request. A crawler hitting every product page in strict URL order, at a perfectly even interval, is a much easier pattern to flag than organic browsing, even if each individual request looks legitimate. This is defensive architecture worth understanding on its own terms: it’s how production sites protect infrastructure cost, pricing data, and inventory signals from being harvested by competitors, and it’s not a puzzle to be defeated so much as a cost function to respect. Nothing in this pipeline, proxy or otherwise, makes a scraper undetectable or a target’s terms of service irrelevant. Every anti-bot vendor’s own marketing will tell you their detection rate isn’t 100%, and no proxy vendor’s will tell you their evasion rate is either. Treat both claims with equal skepticism.

Storage, dedup, and change detection

Catalogue data is only useful if you can tell what changed. Re-scraping ten thousand products and dumping ten thousand new rows every run turns your database into an append-only log nobody can query. A workable pipeline hashes each product’s relevant fields (price, stock, title) and only writes a new row when the hash changes, with a last_seen timestamp updated on every pass regardless. That gives you both a change history and a live snapshot from the same table, and it keeps write volume proportional to actual catalogue churn instead of crawl frequency.

Scheduling should match how fast the data actually moves. Price and stock are volatile and might warrant daily or hourly checks on a subset of high-priority SKUs. Descriptions, categories, and images change rarely and re-crawling them daily is wasted request budget that would be better spent on politeness margin elsewhere in the crawl.

Staying inside the lines

Product catalogues, prices, and stock counts on public storefronts sit in a different category from anything behind a login wall or a paywall. Scraping account data, customer reviews tied to identifiable people, or any content that requires authentication to view crosses into territory this kind of infrastructure shouldn’t be pointed at. Check a site’s terms of service and robots.txt before building anything durable against it, and treat a site’s explicit request not to be crawled as the end of the conversation, not an obstacle to route around.

Where this fits

We build and write about the infrastructure layer of this problem at Data Research Tools: proxy pools, orchestration frameworks, and the pipeline patterns that hold up past a few thousand requests. If you’re evaluating tools for a catalogue project, start with what your target actually enforces before picking infrastructure to match it.

Read more scraping infrastructure breakdowns on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.