Category: Uncategorized

Most scraping pipelines treat every non-200 response the same way: log it, retry it, move on. That works until volume goes up and the retry logic starts making things worse. A 429 and a 500 are not the same problem, and treating them identically burns proxy budget on one and gives up too early on the other. Running proxy infrastructure at scale means building an actual taxonomy, not just a catch block.

This is the breakdown we use operationally. It’s not a list of codes with dictionary definitions. It’s what each one tells you about where the failure is happening, and what response actually makes sense.

Why the taxonomy matters more than the retry count

A generic scraper retries on failure and gives up after N attempts. That’s fine for a hobby script pulling a few hundred pages a day. At production volume, undifferentiated retries do two bad things: they hammer a target that’s already rate limiting you, which pushes you further into a block, and they waste proxy sessions retrying something that will never succeed no matter how many times you ask, like a 404 on a page that was actually removed.

The fix is classifying every response into one of a small number of buckets before deciding what to do with it: succeeded, needs a different identity, needs to slow down, target-side failure, or gone. Each bucket has a different correct action. Get the classification wrong and your pipeline either burns through proxies for nothing or gives up on pages that would have worked on the next try.

2xx doesn’t always mean success

A 200 status code means the server sent a response body. It does not mean the response body is the page you wanted. Sites serving a soft block, a CAPTCHA challenge page, or an interstitial “verify you’re human” screen frequently return 200. The HTTP layer is happy. The content layer is empty or wrong.

This is the single most common blind spot in scraper monitoring dashboards that only track status codes. If your success metric is “percentage of 200s,” you can be blocked on 40% of requests and your dashboard will show green. The only real fix is validating the payload, not just the status line: check for expected selectors, a minimum content length, or a known marker string before counting a request as successful. Status code and content validity are two separate checks and both need to pass.

3xx: the site telling you it moved, or telling you something else

Redirects are usually benign. A 301 on a product page that got a new URL slug, a 302 during a login flow, these are normal site behavior and following them is correct.

The pattern worth watching is a redirect chain that always lands on the same destination regardless of the requested URL, often a login wall, a “please enable JavaScript” page, or a generic error page. That’s a redirect being used as a soft block. The distinction matters for your taxonomy: a 301 to a plausible new URL is a success path, a redirect that funnels everything to one fixed landing page is a block and should be logged as one, not silently followed and counted as a completed fetch.

401 and 403: two different problems that look similar

401 means the request needed credentials and didn’t have valid ones. In scraping contexts this shows up on APIs and authenticated endpoints, and it’s usually a configuration problem on your end: expired token, wrong header, session cookie that didn’t get carried over.

403 is broader and more common on scraping targets. It can mean an IP-based block, a fingerprint-based block, a WAF rule matching the request pattern, or geographic restriction. Sites make these decisions using signals like request rate from a single IP, TLS handshake fingerprint, header ordering and completeness, and behavioral patterns across a session. This is defensive infrastructure working as intended from the site’s side, and it’s worth understanding it that way rather than as a random obstacle: the 403 is a decision, not a glitch, and it usually means something about the request pattern, not just the destination, tripped a rule.

Operationally, a 403 should route differently than a 401. A 401 means fix the request. A 403 means the identity making the request, the IP and its associated fingerprint, has been flagged, and repeating the identical request from the same identity is very unlikely to change the outcome.

404: gone, moved, or never existed, and each needs a different response

A 404 on a URL you scraped successfully last week usually means the page was actually removed or the site restructured its URL scheme. A 404 on a URL you constructed yourself, say by incrementing an ID, might mean you’ve walked past the end of a valid range, not that anything is wrong.

The mistake we see most often is treating 404 as a transient error and retrying it. It isn’t transient. Retrying a 404 ten times wastes ten requests confirming what the first one already told you. The correct action is to log it once, mark that URL as resolved-not-found, and move on. If 404 rate on previously-valid URLs suddenly spikes across a whole domain, that’s a signal the site restructured, and it’s worth flagging for a human to check the URL pattern rather than continuing to auto-retry.

429: the most honest status code in the list

429 (Too Many Requests) is the site explicitly telling you the rate is the problem, not the identity, not the content of the request. It’s the clearest signal in the whole taxonomy because it names the mechanism.

The correct response to a 429 is to slow down, and specifically to respect a Retry-After header if the site sends one. Rotating to a different proxy immediately after a 429 without changing pace just moves the same rate-limiting pressure to a new IP, which is treating a pacing problem as an identity problem. Some pipelines do both, easing the per-identity rate and distributing load across a pool, but the rate change is the part that actually addresses what the 429 said. A pipeline that only ever rotates IPs on 429 and never adjusts request pacing is optimizing the wrong variable.

5xx: figuring out if it’s them or you

500, 502, 503, and 504 all indicate the failure is server-side, but they’re not equally informative. A 500 means the origin server hit an unhandled error, often unrelated to you specifically. A 502 or 504 usually means something in front of the origin, a load balancer or reverse proxy, couldn’t get a valid response from the backend in time. A 503 is frequently deliberate: “service unavailable” is also how some infrastructure signals load shedding or maintenance, and on sites with bot defenses it can appear as a temporary block state rather than a genuine outage.

The practical test is whether the 5xx rate correlates with your own request volume or shows up independent of it. If a small, well-paced scraper gets 503s at the same rate as a request storm would, the site’s front end is treating your traffic as excessive load, whether or not that’s the actual cause. If 5xx rates track a known site outage or deploy window, it’s genuinely their problem and a longer backoff with a handful of retries is the right move.

The failures with no status code at all

A meaningful share of scraping failures never get a status code because the connection didn’t complete: DNS resolution failures, TCP connection refused, TLS handshake failures, and read timeouts where the connection opened but no response came back in time.

These deserve their own bucket because they point at infrastructure, not content. A connection refused from every request through one proxy but not others points at that proxy or that egress IP, not the target site. A TLS handshake failure that only happens on certain proxy exit nodes usually means an outdated or misconfigured client stack on that node. Lumping these in with 5xx server errors hides infrastructure problems on your own side of the connection.

Putting it together

A working taxonomy sorts every outcome into a small number of buckets: validated success (200 plus content check passed), soft block (200 with failed content check, or a redirect loop to a fixed destination), rate limited (429, respond by slowing down), identity blocked (403, respond by rotating identity, not just retrying), resolved-not-found (404, log once, don’t retry), target-side failure (5xx, backoff and limited retry), and connection-level failure (no status code, investigate the proxy or network path). Each bucket maps to one clear action. The value isn’t in the categories themselves, it’s in refusing to let a single generic retry loop handle all seven of these differently-shaped problems the same way.

If you’re building or debugging a scraping pipeline and want more on the infrastructure side of this, proxy behavior, fingerprinting, and how detection systems actually work, we write about it regularly at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The bill scales with requests, not with data

Proxy providers charge by bandwidth or by request count, and either way the number goes up every time your scraper hits the network. Most pipelines don’t need to hit the network nearly as often as they do. A price monitor that checks the same product page every ten minutes is paying for ten fresh fetches an hour when the price probably changed once, if at all. A lead-gen crawler that re-walks a directory site’s category pages every run is buying bandwidth for HTML it already has sitting on disk from yesterday.

Caching doesn’t reduce how much data you need. It reduces how many times you pay a proxy to go get data you already have. That’s the whole value proposition, and it’s worth being precise about it, because caching gets oversold as a performance trick when its real job here is a cost-control lever sitting between your scraper and your IP pool.

What’s actually worth caching

Not everything you scrape is equally volatile, and that’s the entire basis for a caching strategy. Split your targets into three rough buckets before you write any caching code:

Near-static. Category taxonomies, “about” pages, terms of service, historical archive pages, anything with a URL that implies it won’t change again. These can sit in cache for days or weeks with almost no risk of serving stale data that matters.

Slow-moving. Product listings, job postings, review counts, search result pages for stable queries. These change on the order of hours, sometimes once a day. A cache with a TTL matched to your actual monitoring cadence saves the bulk of your proxy spend here, because this bucket is usually the largest share of request volume in a monitoring pipeline.

Volatile. Live pricing during a sale, inventory counts, anything time-sensitive enough that a stale read is a wrong read. This bucket should mostly bypass cache, or use a TTL measured in seconds rather than minutes.

The mistake I see most often is treating every URL as volatile by default, which means every URL pays full proxy cost on every run. Sort your target list into these buckets first. The caching layer only pays for itself on the URLs where staleness is tolerable.

A cache key that doesn’t lie to you

The cache key has to represent everything that changes the response, or the cache will confidently hand back the wrong page. URL path and query string are the obvious components, but they’re not sufficient. If your scraper sends a different Accept-Language header per region, or your proxy pool rotates through residential exits in different countries that get geo-targeted content, the same URL can return materially different HTML depending on which parameters and which context produced the fetch.

Build the key from URL plus the specific request parameters that affect the response body: query params, the country or region you’re routing through if the target geo-targets, and any pagination or sort parameters. Leave out things that don’t affect content, like a cache-busting timestamp param the target site ignores, or your own request ID header. A key that’s too narrow causes cache misses you didn’t need. A key that’s too broad causes cache hits that return the wrong content for the request that’s asking, which is worse, because it fails silently.

Matching TTL to how fast the target actually moves

Once URLs are bucketed, TTL is a direct translation of “how often do I actually need fresh data” into seconds. If your monitoring job runs every ten minutes but a category page only reshuffles once a day, a TTL of a few hours doesn’t cost you any real freshness and it collapses dozens of would-be fetches into one.

Set TTL per bucket, not globally. A single blanket TTL either wastes proxy spend on the near-static bucket, because you’re re-fetching pages that never change, or serves stale data from the volatile bucket, because a two-hour TTL is fine for a category listing and useless for a flash-sale price. If you can observe how often a given page actually changes, by diffing cached copies over a week, use that to tune the TTL instead of guessing. Pages that turned out static in practice can move to the near-static bucket even if you assumed otherwise going in.

Where the cache sits in the pipeline

The cache belongs between the request scheduler and the proxy client, as a layer the scheduler checks before it ever assigns a proxy to a job. Concretely: the scheduler looks up the cache key, and only if it’s a miss or expired does the job go into the queue that consumes a proxy. This is a different role from an HTTP cache sitting behind a CDN, because you control both ends and you’re optimizing for proxy cost, not for latency to an end user.

For storage, a key-value store like Redis works well when you need shared state across multiple scraper workers, so two workers don’t independently fetch the same URL in the same window. A flat store keyed by a hash of the cache key, with the response body and a TTL, is enough for a single-machine pipeline. Whatever you use, store enough metadata alongside the body to make debugging possible later: the fetch timestamp, the proxy or exit node that served it, and the response status code. When something looks wrong three days from now, that metadata is what tells you whether the problem is the cache or the fetch.

The failure mode nobody mentions: caching a block page

This is the one that actually costs money instead of saving it. If your fetch gets a CAPTCHA challenge, a rate-limit response, or a soft-block page that still returns HTTP 200, and your caching logic only checks status code before writing to cache, you’ve just cached the block page. Every subsequent read for that TTL window returns a page with no data in it, and your pipeline reports the target as unchanged or empty instead of blocked. You won’t notice until someone asks why a page that’s been live for months suddenly shows no products.

Guard the write path, not just the read path. Before writing a response to cache, check for the markers that indicate a challenge or block: known CAPTCHA page signatures, unexpected redirect targets, a response body far shorter than the page’s normal size, or the absence of the DOM elements your parser expects to find. Treat those as fetch failures that don’t get cached, not as valid responses to store. A cache is only trustworthy if what goes into it is verified to be real content, and that check has to run before the write, not after something downstream complains.

Conditional requests, where the target supports them

Some sites still honor If-Modified-Since or If-None-Match with a 304 Not Modified response, which costs far less bandwidth than a full page fetch and, depending on your provider’s pricing model, may cost less against your proxy quota too. If a target returns ETag or Last-Modified headers on its responses, store them alongside the cached body and send them back on the next check. A 304 confirms your cached copy is still good without paying for the full transfer. This isn’t something you can rely on across a target list, plenty of sites don’t set these headers meaningfully, but where it works it’s close to free freshness verification.

Deduplicating requests that never needed a proxy at all

Separate from time-based caching, watch for the case where your own pipeline issues the same request twice in the same run, two different scraper jobs both need the same category page, or a retry queue re-adds a job that’s already in flight. A short-lived in-flight lock, keyed the same way as the cache, lets the second caller wait for the first fetch to complete and read its result instead of starting a duplicate fetch through a second proxy. This has nothing to do with target volatility. It’s just not paying twice for a fetch you were already about to make once.

What caching doesn’t fix

A cache reduces how often you go back to a target for data that hasn’t changed. It does nothing for a pipeline that’s scraping more targets than it needs, storing full page bodies when it only needs three fields, or running on a schedule tighter than the business case actually requires. Those are separate cost problems, and they’re usually the bigger ones. Caching is the layer that stops you from paying for the same answer twice. It’s not a substitute for asking whether you needed to ask the question that often in the first place.

If you’re building out scraping infrastructure and want more on proxy architecture, pipeline design, and how bot detection actually works from the defensive side, browse the rest of the site here.

Get new guides and videos first — join the Telegram channel.

Most scrapers die the same way. Not from a ban, not from a lawsuit, but from the person who wrote it leaving the company, and nobody else being able to figure out why it stopped working three weeks later. We run scraping pipelines across multiple targets and proxy pools day to day, and the single biggest cost we’ve seen isn’t infrastructure. It’s the hours spent reverse-engineering a script that only made sense to the person who wrote it, at the time they wrote it.

A scraper that runs once on your laptop is a script. A scraper that keeps running after you’ve moved on to something else is infrastructure. The gap between the two is not clever code. It’s a handful of boring decisions made early, about structure, logging, and failure handling, that determine whether the next person can open the repo and actually do something with it.

Separate what changes from what doesn’t

Every scraper has two kinds of logic mixed together: the part that’s specific to one site (which selector holds the price, which endpoint returns the listing) and the part that’s the same no matter what you’re scraping (retry behavior, proxy assignment, output format, logging).

When these live in the same function, every site change forces someone to read code they don’t own. The fix is a config layer. Site-specific values, selectors, URL patterns, pagination rules, field mappings, should live in a config file or database row, not inline in the scraping logic. The scraper itself becomes a generic engine that takes a config and executes it.

This matters more than it sounds like it should, because the person maintaining this six months from now is almost never the person who understands the target site’s HTML. They understand Python, or the pipeline, or the proxy layer. If fixing a broken selector means editing a YAML file instead of tracing through a class hierarchy, that person can fix it in five minutes instead of opening a ticket.

Assume selectors will break, because they will

Sites change their markup. Class names get regenerated by build tools, layouts get A/B tested, entire page structures get rewritten for a redesign. A scraper that treats a CSS selector as a permanent contract is a scraper that fails silently the day someone on the target site ships a frontend update.

The practical response is to make failure visible and specific. If a selector returns nothing, that should raise a distinct, loggable error, not get treated as “field is empty” and passed downstream as null. We’ve seen pipelines that quietly filled a database with empty price fields for two weeks because the scraper treated a missing selector match the same as a legitimately empty value. Nobody noticed until a downstream report looked wrong.

Where you can, prefer selectors that target structural or semantic attributes over ones that target generated class names, since those tend to survive redesigns longer. But don’t over-engineer this. No selector strategy survives a full site rewrite. What you’re really building is fast detection when it breaks, not a selector that never breaks.

Build the retry and backoff logic once, correctly

Every scraper needs retry logic for network failures, rate limit responses, and transient errors. The mistake we see constantly is retry logic copy-pasted into every scraper, slightly differently each time, with no shared understanding of what counts as retryable versus what counts as a real failure that should stop the run.

Write this once as a shared component: exponential backoff with jitter, a cap on retry count, and a clear distinction between “the network hiccuped, try again” and “the target is returning something that means stop, like a persistent block page or a change in response structure that suggests the whole approach needs revisiting.” A scraper that keeps retrying against a page that’s telling it to stop isn’t resilient, it’s just hammering a server that has already told it no. That’s bad practice regardless of whether it eventually gets detected, and it’s the kind of behavior that gets an IP range or account flagged for everyone using it, not just you.

The retry component should also record why it retried. A pile of retry logs with no reason attached is close to useless when you’re trying to figure out whether a target changed something or your proxy pool degraded.

Logging is for the person who isn’t you

The most common failure mode in inherited scrapers isn’t bad logic, it’s logic with no visibility into what it’s doing. A scraper that fails and prints nothing but a stack trace forces the next person to reproduce the failure locally before they can even start debugging it.

Log at the level of decisions, not just errors: which config was loaded, which proxy or session was used for a given request, what the response status was, how many items were parsed successfully versus skipped, and why anything was skipped. This doesn’t need to be elaborate. A structured log line per request, written to something searchable, turns a two-hour debugging session into a five-minute grep.

Separate operational logs from data output. The scraper’s job is to produce clean records; its logs are a different stream entirely, meant for humans, not for the pipeline downstream. Mixing the two makes both harder to use.

Proxies and sessions are infrastructure, not an afterthought

If a scraper is going to run repeatedly against the same target, how it manages IPs and sessions needs to be a first-class part of the design, not a requests.get() call with a random proxy bolted on. That means deciding upfront how sessions are assigned to identities, how rotation is triggered, and what happens when a proxy in the pool is dead or rate limited by the target.

None of this makes a scraper undetectable, and no configuration of proxies or headers guarantees a site won’t recognize automated traffic. What good proxy management actually buys you is stability: fewer failed requests from exhausted or flagged IPs, and a pipeline that degrades gracefully (skip and log, don’t crash the whole run) when one part of the pool stops working. If you’re evaluating proxy providers for this kind of work, judge them on measurable things, connection stability, documented pool size, response latency under load, not marketing claims about being undetectable, because that claim isn’t something any vendor can actually back up.

Respect published rate limits and robots.txt where they exist. This isn’t just a compliance checkbox, it’s also what keeps a shared proxy pool usable, since aggressive request rates from one script degrade the pool for every other job running on it.

Documentation that isn’t a wall of text

A maintainable scraper needs a short README that answers four questions: what does this scrape, how do you run it, where does the config live, and what does a healthy run look like versus a broken one. That last one is the part people skip, and it’s the one that saves the most time. If the next person doesn’t know that “200 items in 40 minutes” is normal and “12 items in 40 minutes” means something’s wrong, they’ll waste hours figuring that out from scratch, or worse, ship broken data because it looked plausible.

Skip long docstrings describing what each function does structurally. Code with reasonable names already communicates that. Spend the documentation budget on the things that aren’t visible from reading the code: why a particular wait time was chosen, why one field is parsed from a different endpoint than the rest, what happened the last time this broke.

Testing without depending on the live site

You can’t unit test against a live target reliably, since the site changes and you don’t want tests hitting it constantly anyway. Save representative HTML or JSON responses as fixtures and test your parsing logic against those. This catches most regressions, parsing errors from a code change, a field that got dropped, without needing network access, and it gives you a fast way to confirm a fix actually works before deploying it against the real target.

The actual measure of maintainability

None of this is exciting. Config separation, structured logging, shared retry logic, and a short README aren’t the parts of a scraping project anyone shows off. But they’re the difference between a pipeline that survives turnover and one that gets rewritten from scratch every time someone new inherits it. If you’re building scraping infrastructure that’s meant to outlast the person who wrote it, that’s the actual bar, not clever selectors, but whether someone else can open the repo, understand what broke, and fix it without you.

If you’re building out scraping infrastructure and want more on framework choices, proxy setup, and pipeline design from people running it in production, check out the rest of what we’ve written at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The gap between “it scraped” and “it’s correct”

A scraper that returns 200 OK and a parsed object is not the same thing as a scraper that returned a correct record. Anyone who has run a pipeline past a few million rows has seen the failure mode: the job finishes green, the row count looks right, and three weeks later someone finds that 8% of the “price” field is actually a shipping estimate because a site changed a CSS class and the parser quietly grabbed the wrong <span>.

The scraper didn’t crash. It didn’t even throw a warning. It just picked up the wrong node and wrote a plausible-looking number into a plausible-looking column. That’s the core problem with scraped data: most failures aren’t exceptions, they’re silent substitutions. A validation layer exists to catch exactly that class of error, before it becomes a row in production.

Why this has to live between extraction and storage

Teams often bolt validation onto the end of the pipeline, as a query that runs against the database after the fact. That’s too late. Once a bad batch lands in the same table as good data, you’ve mixed the two, and cleaning it up means writing a reconciliation script that has to guess which rows came from the bad run. If validation instead sits between the parser and the insert, a bad batch never reaches the table. It gets logged, held, or routed to a review queue, and the production data stays trustworthy.

This means validation is a pipeline stage, not a database constraint you add later. Constraints (NOT NULL, foreign keys, unique indexes) are useful as a last line of defense, but they only catch structural violations. They will not catch a price field that’s technically a valid decimal but is off by two orders of magnitude because the parser read the wrong element.

Schema validation: catching shape before catching meaning

The first layer is structural. Every record coming out of a parser should be checked against a schema before it’s queued for insert: field presence, type, and basic bounds. If you’re on Python, Pydantic or jsonschema do this cheaply. The point isn’t the library, it’s the discipline of defining what a valid record looks like as code, not as a mental model you re-derive every time something breaks.

Concretely, this means:

• Required fields are actually present, not just present as empty strings or null masquerading as valid data
• Numeric fields parse as numbers, not as strings that happen to look numeric
• Enums (currency codes, status fields, category labels) are checked against a known set, not accepted as free text
• String fields have sane length bounds, so a scraper that accidentally captured an entire page body into a title field gets flagged instead of silently truncated by a database column limit

Schema validation is cheap and catches maybe 60% of real-world scraping failures: missing fields when a page returns a partial render, type mismatches when a site A/B tests a different markup structure, truncated strings when a request gets cut off mid-response. It will not catch the harder failures, where the record is well-formed but wrong.

Semantic validation: catching wrong when it looks right

This is the layer that actually matters for data quality, and it’s the one most pipelines skip because it’s harder to write generically. Semantic checks encode what you know about the domain, not what a schema library can infer from types.

A few patterns that hold across most scraping targets:

Range and plausibility checks. If you’re scraping product prices in SGD, a value of 4 or 400,000 for a consumer product is a red flag worth a second look, not an automatic reject, but a flag. Set soft bounds per category if you have them, and log outliers to a review table instead of silently accepting or silently dropping them.

Cross-field consistency. If a record has a “discount price” higher than its “list price,” something upstream broke, most likely the parser grabbed two different DOM nodes that used to be in a fixed order and no longer are. If a record has a “posted date” that’s in the future relative to your scrape time, your date parser is misreading a format (this happens constantly with sites that mix DD/MM and MM/DD without a clear locale signal).

Referential plausibility. If you’re scraping listings that should map to a fixed set of categories, sellers, or locations, check new values against your known set before accepting them as new entities. A sudden new “category” that’s actually a JavaScript error string rendered into the DOM is a common artifact of scraping a page mid-render.

Duplicate and near-duplicate detection. Re-scraping the same listing across runs is normal. What you want to catch is the same underlying entity showing up with a different ID because the site rotates listing IDs, or the same content getting inserted twice because a retry succeeded on a request that had already succeeded once. A content hash (normalize whitespace, hash key fields) checked against recent inserts before writing catches this at negligible cost.

Handling drift, not just errors

Sites change their markup. This is not an edge case, it’s the normal operating condition of any scraper that runs for more than a few weeks. The validation layer’s job here is not just to catch bad records, it’s to catch the moment drift starts, so you find out from a monitoring alert rather than from a customer complaint.

The practical version of this: track the pass rate of your validation layer per source, per run, as a metric over time. If a source normally validates at 98% and a run comes in at 61%, that’s not 37% of records being individually wrong, it’s almost always one selector breaking across the whole batch. A sudden drop in validation pass rate is a far more useful signal than any single failed record, because it tells you where to look in the parser instead of making you audit rows one at a time.

Set a threshold (we’ve found somewhere around a 10-15 point drop in pass rate is a reasonable alert trigger, though this depends heavily on how noisy your source normally is) and treat crossing it as “stop and check the parser,” not “let the run finish and clean up later.”

Where bad records should go, not just how to catch them

A validation failure is information, and throwing it away loses that information. Records that fail validation should land in a dead-letter table or queue with the raw payload, the specific check that failed, and a timestamp, not just get dropped from the pipeline silently. This gives you three things: a way to reprocess records once you fix the parser, a way to measure how often each failure mode actually occurs, and an audit trail if someone downstream asks why a number looks off.

This is also where you build the feedback loop that improves the parser over time. If “cross-field price check” is triggering on 5% of records from one source, that’s not necessarily 5% bad scrapes, it might be a legitimate promotional pricing pattern your rule doesn’t account for yet. Reviewing the dead-letter queue periodically is how validation rules stay accurate instead of becoming a source of false rejects that quietly shrink your dataset.

Keeping this proportional to the pipeline

None of this requires a heavyweight data quality platform for most pipelines. A schema check, a handful of domain-specific semantic rules, a content hash for dedup, and a dead-letter table cover the overwhelming majority of real failures we’ve seen in production scraping. The mistake isn’t skipping some exotic validation technique, it’s skipping validation entirely and finding out about a broken selector from a downstream report instead of from your own pipeline.

If you’re building or auditing a scraping pipeline and want to see how the rest of the infrastructure, orchestration, proxy layers, and framework choices fit around this, we cover it in more depth on the site.

Explore more scraping infrastructure guides and tool breakdowns on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.

Why mobile app APIs look different from web scraping

Most scraping conversations start with a browser and a webpage. Mobile app API scraping starts somewhere else entirely: a phone or emulator running a native app, and a private REST or GraphQL endpoint that app talks to behind the scenes. There’s no HTML to parse. There’s no DOM. There’s a request, a JSON response, and a set of headers that the app’s developers never expected an outsider to read closely.

That’s the appeal for data teams. A mobile backend API is often cleaner and more stable than a public web page, because it’s built for a machine (the app) to consume, not for a human to read. Fields are structured, pagination is explicit, and the response shape rarely changes without a version bump. It’s also why these endpoints get more defensive engineering than a typical webpage, not less. The people who built the app assumed the only client talking to their servers would be their own app, running on real hardware, and a lot of what gets called “anti-bot” in this space is really just enforcement of that assumption.

We run proxy infrastructure and scraping pipelines for a living, and the mobile surface is where we see the widest gap between what people expect scraping to look like and what it actually involves. It’s not a URL and a requests.get(). It’s a stack of defenses layered specifically because the traffic is expected to originate from one app binary on one class of device.

What actually sits between the app and the data

Certificate pinning. Native apps can hardcode which TLS certificates (or public keys) they’ll trust, instead of relying on the device’s normal certificate store. This is a defense against traffic interception in general, not scraping specifically, but it has the side effect of blocking the kind of TLS-terminating proxy setups that make web scraping practical. When a pinned app refuses to talk over a modified certificate chain, that’s the pinning doing exactly what it was built to do. Circumventing it on a device you don’t control, or on someone else’s production app, moves well past “researching how mobile clients work” into unauthorized tampering, and we don’t cover how to do that here.

Request signing. A lot of mobile APIs sign each request with an HMAC or similar scheme, built from a device-held secret, a timestamp, and the request body. The server recomputes the signature and rejects anything that doesn’t match. This means the “endpoint and parameters” you’d get from watching network traffic aren’t enough on their own; the signing key and algorithm live inside the compiled app, often obfuscated, and are treated by the app owner as the actual access-control boundary. Signing exists precisely to make traffic replay and unauthorized clients expensive to build.

Device attestation. Google’s Play Integrity API and Apple’s DeviceCheck/App Attest let a backend ask “is this request coming from an unmodified copy of my app, running on a real, unrooted device, installed through the official store” and get a cryptographically backed answer. This is a meaningfully different signal from anything available in browser-based bot detection, because it reaches down into the OS and hardware attestation layer rather than just inspecting request headers. A backend that enforces attestation is explicitly designing to exclude emulators, rooted devices, and repackaged app builds.

TLS and transport fingerprinting. Independent of the app layer, the TLS handshake itself carries a fingerprint (commonly summarized as JA3 or JA4) built from cipher suite order, extensions, and other client-hello details. HTTP/2 frame ordering and settings add another layer on top. Real mobile OS network stacks produce fingerprints that are consistent and well-documented; many HTTP client libraries used for scraping produce a visibly different one by default. Detection systems compare what they see against known-good mobile stack fingerprints, which is part of why “just hit the same endpoint with a Python HTTP client” tends to get flagged even when the headers look identical to a real app’s.

Behavioral and rate signals. Above all the cryptographic layers, most mobile backends also watch account- and IP-level behavior: request cadence, sequence (does this client fetch a profile before it fetches that profile’s posts, the way the real app’s navigation would), and volume relative to what a single human using a single phone would plausibly generate. This is the same class of signal used in web bot detection, just applied to app-shaped traffic instead of page-shaped traffic.

None of these layers are marketed as unbeatable by the vendors who build them, and we’re not going to claim otherwise either. They raise cost and reduce reliability for unauthorized automated access. Whether that’s enough for a given app depends entirely on how many of these layers the team actually implemented and how well.

Where proxy infrastructure fits, and where it doesn’t

Because we operate proxy infrastructure ourselves, we get asked a lot whether routing traffic through mobile or residential IPs “solves” mobile API access. It solves exactly one layer: IP reputation. Datacenter IP ranges are heavily represented on abuse block lists and get flagged faster by systems that check ASN and IP reputation as one input among several. Mobile carrier IPs, because of carrier-grade NAT, look like they’re shared by thousands of real subscribers, which changes how a reputation system scores them.

But IP reputation is one input, not the whole system. It does nothing about certificate pinning, nothing about request signing, and nothing about device attestation. A request from a pristine residential IP that fails signature verification gets rejected exactly the same as one from a flagged datacenter IP. Anyone evaluating a “mobile proxy” product as a way to access a pinned, signed, attestation-checked API is solving the wrong layer of the stack, and we’d rather say that plainly than sell a proxy pool as a fix for a problem it can’t touch.

The line we actually hold to

There’s a real, legitimate reason to look at how a mobile app talks to its own backend: understanding your own app’s traffic, building interoperability tools with a provider’s consent, security research disclosed responsibly, or academic study of how these defenses are built. There’s also a line past that, where accessing a private API without authorization, working around technical protections on a system you don’t have permission to test, or pulling personal or paywalled data crosses into territory we won’t walk through step by step, regardless of how the target defends itself.

We also won’t tell you a given proxy pool, device farm, or signing workaround is undetectable or risk-free, because that claim isn’t ours to make about someone else’s detection stack, and every layer described above exists precisely to make that kind of guarantee false over time. Detection systems get updated. Attestation requirements tighten. What worked against a backend last quarter is not a permanent state.

What this means if you’re evaluating a data source

If a team is deciding whether a mobile app’s backend is a viable, sanctioned data source, the honest checklist looks like this: is there a public API or partner program instead of a private one, does the provider’s terms of service permit this kind of access, and if you’re building your own client, have you gotten explicit permission rather than inferring it from the absence of a “no” in a public document. Structured JSON is a nice property of mobile backends. It doesn’t change what permission and terms of service require.

We write about the defensive side of this space (pinning, attestation, fingerprinting, rate systems) and about the proxy and pipeline infrastructure that legitimate, permissioned data collection actually runs on, because that’s the work we do daily. If you want the rest of our breakdowns on scraping infrastructure, bot detection internals, and tool reviews grounded in what we actually run, start at the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.

A scraper that throws an exception is a good day. Someone sees the stack trace, someone fixes it, the pipeline moves on. The bad day is the one where the job finishes clean, logs say success, HTTP status is 200, and the output file has three rows in it instead of three thousand. Nothing crashed. Nothing paged anyone. The data just quietly wasn’t there.

This is the failure mode that does the most damage in production scraping, because it doesn’t announce itself. A crash is loud. An empty result set dressed up as a successful run is silent, and silent failures compound: the next run overwrites yesterday’s good data with today’s empty set, a dashboard downstream shows a gap that gets read as “no news today” instead of “the pipeline broke,” and by the time a human notices, a week of data is missing and nobody can say exactly when it started.

Why “it ran” isn’t the same as “it worked”

Most scraping frameworks treat success as “the request completed and the code didn’t throw.” That’s a network-layer definition of success. It says nothing about whether the page you got back is the page you wanted.

There are several ways a request can complete cleanly and still hand you nothing useful:

A challenge page instead of content. Many anti-bot systems don’t return a 403 or a 429 when they’re suspicious of a request. They return 200, with a page that looks like HTML but contains a JavaScript challenge, a CAPTCHA widget, or an “checking your browser” interstitial. If your parser is looking for a specific CSS selector and that selector isn’t on the challenge page, you get an empty result, not an error. The status code lied by omission.

Selector drift. A site redesigns a product listing page. The container div that used to hold .item-card now holds .product-tile. The request succeeds, the HTML is real content, and your parser still returns zero rows because it’s looking for a class name that no longer exists. This is the most common cause of silent-empty in practice, and it has nothing to do with blocking at all.

JavaScript that never rendered. If a page builds its content client-side and your fetcher is a plain HTTP client rather than a headless browser, you get the empty shell HTML and none of the injected content. Status 200, body present, target data absent.

Soft rate limiting. Some sites respond to excessive request volume not with a block but with a stub response, a cached placeholder, or a page that omits the dynamic sections entirely while keeping the static shell intact. It’s designed to look normal to a script that isn’t checking closely.

All four of these produce the same observable symptom from a pipeline’s point of view: a request that “succeeded” and a parse that returned nothing. The fix isn’t a smarter scraper. It’s a pipeline that stops trusting HTTP status as a proxy for data quality.

Check the shape before you check the content

The cheapest and highest-value guard is a shape check that runs immediately after parsing and before anything touches storage. It doesn’t need to understand the data, it just needs to know what “normal” looks like in aggregate.

Row count against a floor. If a job has returned somewhere between 800 and 1,200 records on every run for the past month, a run that returns 4 records should never reach the database unflagged, even if every one of those 4 records is individually valid.

Byte size of the raw response, before parsing. A challenge page and a real listing page are often wildly different in size. Logging response size per request and comparing it against a rolling baseline catches soft blocks even when the parser itself doesn’t error out.

Presence of expected markers. Rather than trusting that a selector match means the page is real, check for a handful of strings or elements that should exist on a genuine page but wouldn’t exist on a challenge page, and vice versa: strings that only show up on interstitials (captcha, verify you are human, checking your browser) and should never appear in a healthy scrape.

Schema completeness. If your target record has a price, a title, and a URL, a “successful” parse that returns records with all three fields null isn’t successful. Validate required fields exist and are non-empty before the record is written, not after.

None of these checks need to be clever. They need to run every time, on every batch, before data lands anywhere downstream systems will read from.

Baselines matter more than absolutes

A hard rule like “reject if zero rows” catches the extreme case but misses the more common one: a job that used to return 1,000 rows and now returns 40. Forty is not zero. It will pass a naive check and still represent a pipeline that’s mostly broken.

The more durable approach is comparing each run against a rolling baseline built from recent history, not a fixed number set once and forgotten. If a job’s typical output sits in a known range, a run that falls outside a few standard deviations from that range is worth flagging even if it’s technically nonzero. Sites change layout gradually sometimes, and a slow bleed from 1,000 rows to 600 over two weeks is exactly the kind of thing a static threshold will never catch but a trend comparison will.

This also has to account for legitimate variance. A site that only lists 200 items on a Sunday isn’t broken, it’s just Sunday. Baselines that ignore day-of-week or seasonal patterns generate false alarms, and false alarms are how teams end up ignoring the alert channel entirely.

Where the check lives in the pipeline

Putting the validation step after storage means bad data has already landed before anyone finds out. The check needs to sit between parse and write: parse the response, run the shape and baseline checks, and only then commit to the database or the file store. A run that fails validation should be quarantined, not silently dropped and not silently accepted. Quarantine means the raw response and the parse output both get saved somewhere inspectable, so a human or a secondary process can look at exactly what came back and decide whether it’s a selector problem, a blocking problem, or a real change in the underlying data.

Retrying immediately on failure is tempting but often wastes the run. If a request got a challenge page because of how it looked (missing headers, a proxy IP with a poor reputation, a fingerprint that doesn’t match a real browser), retrying the identical request from the identical source usually produces the identical result. A better pattern is to flag the failure, stop that job from overwriting good historical data, and let a separate diagnostic pass investigate root cause rather than burning request volume in a retry loop that can’t succeed.

Alerting without training people to ignore it

An alert that fires on every run with slightly lower volume than yesterday will get muted within a week. The signal worth paging someone for is a sustained pattern: three consecutive runs below baseline, not one. A single anomalous run is worth logging. A trend is worth waking someone up for.

It also helps to separate “this looks like blocking” from “this looks like a broken selector” in the alert itself, because the fix is completely different. A blocking signature usually shows response size and structure changing sharply while request success rate (in the HTTP sense) stays high. A selector break usually shows response size staying normal while the specific fields being extracted go to zero or null. Logging both signals separately, rather than collapsing everything into one “job failed” bucket, saves the person on call from having to re-diagnose from scratch every time.

The underlying point

A scraping pipeline that only checks “did the request return 200” is checking the wrong layer. The network succeeding and the data being real are two different claims, and production systems that treat them as the same thing find out the hard way, usually when someone downstream asks why a report has been flat for two weeks. Detection has to happen at the data layer: shape, volume against baseline, and required-field completeness, checked before anything is written and quarantined loudly when it doesn’t hold up.

If you’re building out pipeline monitoring, proxy infrastructure, or want a closer look at how scraping frameworks handle failure detection in practice, more breakdowns like this live on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.

A text file that everyone treats as law and almost no one enforces

Every scraping team runs into robots.txt in the first week. Someone checks it, sees a Disallow: /api/ line, and asks whether that means scraping the API is illegal. The honest answer is that robots.txt is not law. It is a voluntary signaling protocol, and what it signals is intent, not permission or prohibition in a legal sense. Understanding what the file actually does, mechanically, is the only way to reason clearly about the legal questions that come after it.

What robots.txt actually is

The Robots Exclusion Protocol dates back to 1994. A site places a plain text file at /robots.txt with rules like:

User-agent: *
Disallow: /admin/
Crawl-delay: 2
Sitemap: https://example.com/sitemap.xml

Nothing on the server enforces this file. There’s no gate, no token check, no authentication layer tied to it. A crawler reads the file voluntarily, parses the User-agent block that matches its own identifier, and then chooses whether to honor the Disallow and Crawl-delay directives. Googlebot, Bingbot, and most reputable crawlers do honor it, because ignoring it can get a crawler’s IP ranges blocked and because search engines have a business reason to stay on good terms with site owners. A scraper you write yourself has no such obligation baked in. Whether it respects robots.txt depends entirely on whether you tell it to.

That’s the whole mechanism. It is a courtesy notice, not a firewall. Any operator running a real pipeline should already know this, because we build our own fetchers and none of them get robots.txt enforcement for free from the OS or the HTTP stack. We have to read the file and implement the check ourselves, the same way a browser doesn’t check it either.

Where the actual legal exposure comes from

Because robots.txt isn’t enforced technically, it also isn’t the thing courts have leaned on most heavily when scraping disputes go to litigation. Two bodies of law matter more:

The Computer Fraud and Abuse Act (CFAA) is the US federal statute most often cited in scraping cases. It criminalizes accessing a computer “without authorization” or “exceeding authorized access.” For years, defendants argued that violating a site’s terms of service, or scraping past a robots.txt disallow, counted as “without authorization.” Courts have narrowed that reading substantially. In Van Buren v. United States (2021), the Supreme Court held that “exceeds authorized access” applies to accessing off-limits areas of a system you’re otherwise permitted to use, not to misusing access you already have for a purpose the owner dislikes. In hiQ Labs v. LinkedIn (9th Circuit, most recently affirmed in 2022 after remand), the court held that scraping data that is publicly accessible without logging in generally does not violate the CFAA, because there’s no access barrier to circumvent in the first place. Robots.txt wasn’t the deciding factor in either case; the presence or absence of an actual technical access barrier was.

Contract and terms-of-service law is the other track, and it runs independently of the CFAA. A site’s terms of service can prohibit automated access even for public pages. If you create an account, or even just use the site, under terms that ban scraping, a court can treat that as a breach of contract claim rather than a computer-crime claim. This is a lower bar for a plaintiff to clear than CFAA liability, and it doesn’t require proving anything about robots.txt at all. Some site owners have also succeeded with trespass-to-chattels theories, arguing that scraping traffic imposed a measurable burden on their servers.

So the practical picture is: robots.txt is evidence of the site owner’s stated wishes, which can be relevant context in a dispute, but the actual liability usually turns on whether there was an access barrier to bypass (CFAA), what the terms of service said (contract law), and what kind of data was collected (privacy statutes like GDPR or CCPA apply regardless of what any robots.txt file says, because they’re about the nature of the data, not the method of collection).

What this means for how we build

None of this is a green light to disregard robots.txt because it “isn’t technically law.” In production infrastructure, respecting it is good engineering practice independent of the legal question:

• Crawl-delay and rate limits protect your own infrastructure too. A crawler that ignores a site’s stated crawl-delay is more likely to get flagged by the target’s bot-detection layer, burn through proxy pool reputation faster, and trigger IP-range blocks that cost you working exit nodes. Respecting the stated delay is cheaper than fighting the fallout.
• Disallow directives often mark the exact paths a site’s detection systems watch most closely. A /Disallow: /search-results/ line is frequently a hint about where rate-based or fingerprint-based defenses are tuned tightest. Site operators use crawl-delay and disallow rules as one signal among several (alongside request headers, TLS fingerprints, and behavioral patterns) to separate expected crawler traffic from unexpected load.
• The file tells you what the site owner considers acceptable, which matters for the contract-law exposure even when it doesn’t resolve the CFAA question. If a dispute ever gets to a demand letter or a court filing, a scraper that ignored an explicit, published Disallow directive on a site whose terms also prohibited scraping is in a materially worse position than one that stayed within stated bounds.

The parts robots.txt says nothing about

Robots.txt has no concept of personal data, and it is not a privacy control. A page can be fully allowed by robots.txt and still contain personal data whose collection, storage, or resale is regulated separately under GDPR, CCPA, or similar statutes depending on jurisdiction and the nature of the data subjects. Scraping something robots.txt permits doesn’t clear that separate legal question, and scraping something it disallows doesn’t automatically create liability under privacy law either. These are different axes entirely, and treating “robots.txt allowed it” as a compliance checkbox for personal data is a mistake we see teams make regularly.

Robots.txt also says nothing about paywalled or authenticated content. Whatever a site allows crawlers to see through robots.txt applies to publicly reachable pages. Content sitting behind a login or a paywall is a different legal and technical situation, generally involving actual access controls, which is exactly the kind of barrier the CFAA and hiQ line of cases treat as significant.

The bottom line for anyone running this infrastructure

Robots.txt is a directive, not a lock. It tells you what a site owner has published as their preference, and courts have generally treated it as one data point among several rather than a dispositive legal boundary. The stronger, more consistent legal lines run through whether there was an actual access barrier, what a site’s terms of service say, and what kind of data is being collected. None of this is a substitute for legal advice about a specific use case and specific jurisdiction; it’s the operating context worth understanding before you decide how a crawler behaves.

If you’re building scraping infrastructure and want the engineering side, from proxy pool design to how bot detection actually evaluates a request, that’s what we cover here at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

Every scraping team hits the same wall eventually. A target site stops returning useful HTML in the initial response, the data only shows up after the page runs its JavaScript, and the obvious fix is to bolt on a headless browser. That fix works. It also quietly multiplies your infrastructure bill, because a browser doing full page rendering is a completely different cost profile than an HTTP client pulling text.

I run proxy infrastructure and scraping pipelines for a living, and the single most expensive mistake I see is reaching for a browser by default instead of by necessity. Here’s what actually drives the cost, and where the real savings are.

Why rendering costs more than fetching

A plain HTTP request to a static page is cheap: open a socket, send a request, read a response, parse text. The resource cost is close to the size of the payload.

A headless browser render is a different machine entirely. It has to launch or reuse a browser process, load the HTML, fetch every linked resource the page references (stylesheets, fonts, images, ad scripts, analytics beacons), build a DOM, run the page’s JavaScript, and often wait for network activity to settle before the data you want is actually present. Each of those steps consumes CPU, memory, and bandwidth that a plain HTTP fetch never touches. A single browser tab holding a modern, script-heavy page will hold noticeably more resident memory than an HTTP client ever needs, and it holds that memory for the entire time the page is open, not just for the length of a request.

Multiply that by concurrency. Ten HTTP requests in flight are trivial. Ten browser tabs in flight are ten browser-sized processes competing for CPU and RAM at the same time. This is the first place budgets get blown: teams size their rendering fleet the same way they’d size an HTTP-based crawler, then wonder why the servers fall over or the cloud bill spikes.

Check for the API before you reach for a browser

Before touching a browser at all, it’s worth confirming the page actually needs one. Most “JavaScript heavy” sites render client-side by calling a JSON API in the background and building the DOM from the response. If you open the network panel in a browser’s developer tools while the page loads, you can usually see that underlying request: a fetch or XHR call returning structured JSON with the exact data the page displays.

If that endpoint exists and returns usable data, you don’t need a browser for that page at all. A plain HTTP client hitting the API directly gets you the same data at a fraction of the compute and bandwidth cost, with no DOM to build and no JavaScript to execute. This single check, done before writing any scraper code, is the biggest cost lever available. In my own pipelines, the pages that end up actually needing a full browser are a minority; most “JS-heavy” sites turn out to have a clean data API underneath the presentation layer.

The cases where you genuinely need a browser are narrower: pages where the data is computed client-side from multiple sources with no single clean endpoint, or where a request requires a token or parameter that’s generated by executing the page’s own script logic. In those cases, a browser is doing real work you can’t skip.

Trim what the browser actually loads

If a browser is genuinely required, the next lever is cutting what it fetches. By default, a full render pulls every asset a page references: images, web fonts, third-party ad and tracking scripts, analytics beacons. None of that is needed if you’re only after the data in the DOM.

Browser automation tools (Playwright, Puppeteer, Selenium) all support request interception, letting you block requests by resource type before they hit the network. Blocking images, fonts, stylesheets, and known ad/tracker domains cuts both the bandwidth pulled per page and the time it takes the page to settle, since the browser isn’t waiting on dozens of unrelated network calls it doesn’t need to finish.

This matters twice over if you’re routing traffic through paid proxy bandwidth, which is standard for scraping sites that block bare datacenter IPs. Residential and mobile proxy bandwidth is typically billed per gigabyte and costs substantially more per GB than datacenter bandwidth. A render that pulls a full page’s worth of images and third-party scripts through that proxy is paying a premium price for bytes you’re going to throw away. Trimming the render doesn’t just save render time, it directly cuts the proxy bill.

Pool browsers, don’t spawn them

Cold-launching a new browser process for every page is expensive and slow: process startup, extension/profile initialization, and warmup all add latency and CPU overhead that has nothing to do with the page you’re actually scraping. The fix is a fixed pool of long-lived browser instances (or browser contexts within a smaller number of instances) that requests get queued through, rather than an unbounded number of browsers spun up per job.

A pool also gives you a hard ceiling on memory and CPU usage, which is what actually protects your budget. Instead of discovering your cost when the bill arrives, you cap concurrency at a number your hardware or cloud instance can sustain, and jobs queue behind that cap instead of spawning more processes than you can afford to run.

Self-hosted rendering vs. paying per page

There are two broad ways to get headless rendering capacity: run your own browser farm on your own compute, or pay a hosted rendering API per successful page load.

Hosted rendering services bundle browser infrastructure, and often proxy rotation, into a per-page price. That’s predictable and removes the operational burden of managing browser crashes, memory leaks, and scaling, but the cost scales linearly with volume. At high, sustained page counts, that per-page fee usually adds up to more than the equivalent compute cost of running your own pool, because you’re paying someone else’s margin on top of the underlying infrastructure they’re also renting.

Self-hosting flips the tradeoff: fixed compute cost regardless of volume, but you own the operational work of keeping browsers stable, restarting crashed instances, rotating proxies yourself, and monitoring memory leaks that browser automation tools are prone to over long-running sessions. For spiky or low-volume scraping, paying per render avoids paying for infrastructure that sits idle most of the time. For steady, high-volume pipelines, self-hosting is usually the better unit economics, but only if someone is actually maintaining it, since an unmaintained browser farm degrades quietly until it’s burning compute on failed renders.

Neither approach is inherently the “right” one. It’s a volume and maintenance-capacity question, not a universal answer.

How sites detect automated browsers

It’s worth understanding, from a defensive perspective, why this space is adversarial at all. Sites that invest in bot management look for signals that distinguish an automated browser from a person clicking around. Headless browsers historically exposed detectable markers: a navigator.webdriver flag set to true, missing browser plugins or MIME type lists that a real install has, differences in how canvas and WebGL rendering report back given the same commands, and inconsistencies between the HTTP-level TLS handshake and the browser identity claimed in the user-agent header. Behavioral signals matter too: real users produce irregular mouse movement, scroll, and timing patterns, while scripted interactions tend to be too mechanically consistent.

None of this is a checklist to defeat. It’s the reason no proxy, browser configuration, or automation tool can be marketed as undetectable or risk-free, and any claim that one is should be treated skeptically. Detection and evasion are an ongoing arms race between site operators protecting their infrastructure and the automation trying to blend in, and the balance shifts constantly in both directions. Any scraping pipeline that assumes permanent access to a given site is planning around a false assumption. Build for graceful failure: detect blocks, log them, back off, and don’t treat a scraper’s continued access as guaranteed.

Cache before you re-render

A lot of rendering spend is redundant: re-rendering pages whose underlying data hasn’t changed since the last crawl. If you’re pulling the same listing or profile pages on a schedule, cache the extracted result with a time-to-live that matches how often the source data actually changes, and skip the render entirely on a cache hit. For the parts of a site that aren’t behind JavaScript, conditional requests using ETag or Last-Modified headers let the server tell you nothing changed without you paying for a full fetch, let alone a full render.

This is the least glamorous lever and also one of the most effective, because it doesn’t require touching your rendering architecture at all. It just stops you from paying twice for the same answer.

The tool matters less than the architecture around it

Playwright, Puppeteer, and Selenium all do the same core job: drive a real browser engine programmatically. The differences between them (API ergonomics, protocol overhead, language support) are real but small next to the architectural choices above. A well-pooled, resource-trimmed, cache-aware pipeline built on any of the three will outperform a naive implementation on the “best” one. Pick the tool that fits your language and team, then spend the engineering time on the architecture, not the tool choice.

The bottom line

The money in JavaScript scraping doesn’t leak out through the browser tool you pick. It leaks out through rendering pages that never needed a browser in the first place, pulling bytes through paid proxy bandwidth that the scrape never uses, and spinning up more browser processes than the job actually requires. Check for the underlying API first. Trim the render to what you need. Pool your browsers instead of spawning them per request. Cache aggressively. Those four changes, in that order, are what separate a scraping pipeline that scales from one that quietly outgrows its budget.

If you want more breakdowns like this, straight from people who actually run the infrastructure, check out the rest of the site.

Get new guides and videos first — join the Telegram channel.

Every scraping project eventually hits this question: do you need a browser at all, and if you do, does it need to be headless or “real”? People ask it like there’s a universal answer. There isn’t. The right choice depends entirely on what the target site is checking for, and most teams never actually look at that before picking a tool.

I run proxy infrastructure and scraping pipelines for a living, which means I care about this decision for a boring reason: cost. A headless Chrome instance and a full browser session with a visible rendering pipeline do not cost the same in CPU, memory, or IP reputation. Picking the heavier option by default because it “seems safer” burns money on every job that didn’t need it. Picking the lighter option blindly gets you blocked on every job that did.

Start one level below “headless vs real”

Before deciding between headless and headful, ask whether you need a browser at all. A browser exists to run JavaScript and render a DOM. If the data you want is present in the raw HTML response, or comes back from an API endpoint the page itself calls, a plain HTTP client is faster, cheaper, and has none of the runtime signals a browser exposes. Open the network tab, watch what the page actually fetches, and check if that endpoint returns clean JSON with a normal request. A surprising number of “we need Playwright” projects are really “we didn’t check the XHR calls” projects.

Once JavaScript-rendered content, client-side routing, or interaction (scrolling, clicking, form state) is genuinely required, you’re into browser automation territory. That’s where headless vs real actually matters.

What “headless” means at the engine level

Headless Chrome and Chromium run the exact same rendering and JavaScript engine as the desktop browser. The DOM it builds, the CSS it computes, the JS it executes, all of that is identical. What’s different is the absence of a windowing system and a handful of runtime properties that only exist because there’s no actual display.

That gap is small but structurally hard to close, because it’s not cosmetic. Headless mode historically reported a different navigator.webdriver value, exposed a smaller or missing window.chrome object, and had gaps in things like plugin lists, permission query results, and certain rendering paths (WebGL and canvas output can differ subtly depending on the software vs hardware rendering path in use). Newer Chrome headless modes (“headless=new”) close some of these gaps but not all of them, and the exact set of differences changes with every browser release.

How detection systems actually look for this

I’ll describe the mechanics here, not a recipe for beating them, because the mechanics are what determine whether headless is viable for a given target.

Fingerprinting systems that care about automation generally check three layers:

Runtime object inspection. The page’s own JavaScript reads properties off navigator, window, and document and looks for values that don’t match a normal browser: missing plugins, an absent or stubbed chrome object, permission APIs that return inconsistent results, or automation-specific properties left behind by the driver.

Rendering fingerprints. Canvas, WebGL, and audio-context fingerprinting render a scene and hash the output. Differences in GPU rendering paths, font availability, or software vs hardware acceleration between a real display session and a headless one can shift that hash in ways that are consistent across a whole fleet of bots using the same headless config, which makes them clusterable even without any single “obvious” tell.

Protocol and behavioral signals. Sites and their bot-management vendors watch for the Chrome DevTools Protocol connection itself, timing patterns that look scripted (uniform delays, no mouse jitter, instant form fills), and session-level patterns like a fleet of “different” visitors that share the exact same viewport, timezone, and rendering fingerprint.

None of this means headless is universally detectable, and none of it means a real browser window is universally invisible. It means the target’s specific bot-management stack determines how much of the gap actually matters for that site.

Why a “real” browser isn’t a free pass

This is the part people get wrong most often. Running a full, visible Chrome window instead of headless mode removes one category of signal, but if that window is still being driven by Selenium, Playwright, or Puppeteer through the DevTools Protocol, most of the automation surface is still there. The DOM read/write patterns, the CDP connection, and the timing signature of a script clicking things don’t go away just because a window is rendering.

A real browser session helps when the target is specifically checking headless-only signals (rendering fingerprints, navigator.webdriver, plugin enumeration). It does nothing for a target that’s watching the CDP protocol itself or behavioral timing, because those exist regardless of whether a window is visible. Vendors who market “undetectable browser” products are selling against a subset of checks, not the whole category, and no honest description of these tools can promise otherwise. Anyone claiming zero detection risk is describing marketing copy, not a tested outcome.

The cost side nobody skips past for long

Full browser sessions are expensive to run at scale in a way that compounds fast. Each real Chrome instance, headless or not, holds memory in the hundreds of megabytes and eats CPU on every render, script execution, and paint cycle. Run a few hundred of these concurrently and you’re provisioning real compute, not just API calls. Headless without a GUI shaves off the windowing and compositing overhead, which is the main reason it exists as a mode at all: it was built for CI and server automation, not evasion.

If your pipeline needs thousands of page loads a day and the target doesn’t fingerprint aggressively, the honest move is HTTP requests where you can and headless browser scraping where you can’t avoid a render. Reserve full browser sessions for the specific targets that actually require them, and treat that as a per-target decision, not a blanket policy.

Where proxies fit into this decision

Browser choice and IP infrastructure are not separable, and treating them as two independent decisions is a common mistake. A residential or mobile IP paired with an obviously scripted headless session with zero mouse movement and instant page loads is still a mismatched fingerprint, just a differently mismatched one. Detection systems that correlate network-layer signals (IP reputation, ASN, connection reuse patterns) with browser-layer signals will catch a clean IP behind a sloppy automation profile just as easily as a dirty IP behind a clean one. The browser and the network identity need to be consistent with each other, and consistent with the behavior pattern of the session, not just individually “good.”

A decision framework that actually holds up

Check the network tab first. If the data is available without executing JavaScript, skip the browser entirely.

If you need rendering but the target has no meaningful bot-management layer (most low-traffic or internal sites), headless is almost always sufficient and dramatically cheaper to run at volume.

If the target uses a known bot-management vendor or you’re seeing blocks that correlate with browser fingerprint checks specifically, testing with a real, visibly-rendered session (still through your automation stack) tells you whether the gap is rendering-related or protocol/behavior-related. If blocks persist in a real window too, the problem isn’t headless mode, it’s your timing and interaction pattern, and no browser choice fixes that on its own.

Budget for the fact that heavier browser sessions cost more per request in compute and, often, in the class of proxy required to match them credibly. Don’t default to the expensive option for jobs that don’t need it, and don’t assume the cheap option is safe just because it’s cheap.

There’s no single right answer to headless vs real browser. There’s a right answer for a specific target, checked against what that target is actually measuring, re-evaluated whenever the target’s bot management changes, because it will.

If you want more breakdowns like this on how scraping infrastructure actually behaves under load, along with the tool reviews and pipeline notes that inform them, check out the rest of what we’re building at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

What “login wall” actually means for a pipeline

Most scraping guides assume the page you want is public: no account, no cookie, just an HTTP GET and a parser. Authenticated scraping is a different problem. The data sits behind a session, and a session is not a URL, it’s a state machine. You log in once, the server hands you something that proves who you are on every later request, and your pipeline has to carry that proof around correctly or every request after the first one fails.

This matters because a lot of production scraping work is legitimately authenticated: pulling your own account data out of a SaaS dashboard that has no API, checking your own listings on a marketplace, monitoring a service you’re a paying customer of. The infrastructure question is the same regardless of what’s behind the wall: how do you keep a session alive, detect when it dies, and not trip the fraud and abuse systems that every login-gated service runs. This piece covers that mechanics side. It does not cover getting into an account you don’t have rights to, or pulling other users’ private data. That’s a legal and ethical line, not just a technical one, and the rest of this article assumes you’re on the right side of it.

Cookies, tokens, and what a session actually is

When you authenticate against a web app, the server typically responds with one of two things: a session cookie (an opaque ID that maps to server-side state) or a token (a JWT or similar, often stored client-side and sent as an Authorization header). Some apps use both, a short-lived access token plus a longer-lived refresh token.

A scraper that only handles the login request and throws away what comes back after is not doing authenticated scraping, it’s doing authenticated scraping once. The actual work is in the client that follows: persisting the cookie jar or token across requests, attaching it consistently, and refreshing it before it expires rather than after. requests.Session() in Python, Scrapy’s cookie middleware, and Playwright or Puppeteer’s persistent browser context all exist specifically to solve this: they hold onto whatever the server issued and replay it correctly on every subsequent call. If you’re writing this by hand instead of using one of those, you’ll eventually get the header ordering or cookie domain scoping wrong and the session will silently stop working.

CSRF tokens complicate this further. Many login and form-submission flows embed a token in the page HTML that has to be read and included in the next POST, separate from the cookie itself. Scrapers that hardcode a captured token instead of scraping it fresh from the current page break the first time the site rotates it, which is often every session.

Binding a session to a proxy, and why that’s not optional

Here’s the part that trips up people who move from open scraping to authenticated scraping: the IP address is now part of the identity, not just a routing detail. A session that logs in from one IP and then makes subsequent requests from a different one looks, from the server’s side, exactly like a stolen session cookie being replayed from a new location. Fraud and account-security systems are built to catch exactly that pattern, because it’s also what actual account takeover looks like.

Practically, this means each authenticated session needs to be pinned to a single sticky IP for its lifetime, not pulled from a rotating pool on every request the way you might for anonymous scraping. If you’re running this at any scale, that means your proxy layer needs sticky sessions with a defined TTL, and your session-management layer needs to track which proxy each logged-in session is bound to and reuse it consistently. Mixing rotating-proxy scraping logic with authenticated flows is a common cause of sessions dying within minutes of login.

Detecting a dead session before it wastes a scrape run

Sessions expire, get revoked, or hit concurrent-session limits (a lot of services cap how many active sessions one account can hold, and will kick the oldest one when a new login happens). None of this shows up as an HTTP error most of the time. It shows up as a 200 response with a login page in the body, because most apps redirect an unauthenticated request to /login rather than returning 401.

This is the single most common silent failure mode in authenticated pipelines: the scraper keeps running, keeps getting 200s, and keeps parsing a login form instead of the data it wanted, for hours, before anyone notices the output is empty or garbage. The fix is structural, not clever: check the response body or a specific DOM element that only exists post-login, on every request, not just the first one. Treat a detected logout as a distinct pipeline event, not a parse error, and re-authenticate through the actual login flow rather than retrying the same dead cookie.

Rate limits don’t relax because you’re logged in

There’s an intuition that authenticated access is “trusted” and therefore less restricted. Often it’s the opposite. An authenticated endpoint can attribute every request to a specific account, which means the server can apply per-account rate limiting that’s both stricter and harder to route around than IP-based limiting on public pages, since switching IPs doesn’t help when the throttle key is the account, not the address. Pacing an authenticated crawl usually needs to be tuned separately from an anonymous one, and tighter.

Multi-factor authentication is a security control, not an obstacle to script past

Some login flows require a second factor: an SMS code, an authenticator app, a push approval. That control exists specifically to prevent automated or unauthorized login. Automating around it (scripted OTP interception, SIM-based code farms, or push-approval automation) is the account-security equivalent of picking a lock, and it’s also usually a direct terms-of-service violation on top of whatever legal exposure applies to the account itself. If a workflow requires MFA on every login, the sane engineering answer is to log in rarely and keep the resulting session alive as long as the service allows, not to try to defeat the second factor.

What server-side systems actually watch for

From the defending side, services that care about authenticated abuse don’t just check whether a cookie is valid. They look at behavioral consistency across a session: does the request timing match human navigation, does the device and TLS fingerprint stay the same for the life of the session, does the account suddenly access pages in an order or at a volume no real user would. This is the same fingerprinting and bot-detection layer covered elsewhere on this site, applied specifically to logged-in traffic rather than anonymous traffic. A scraper that handles cookies perfectly but hammers an authenticated endpoint at a constant interval with no variance, or one whose TLS fingerprint doesn’t match the browser its user agent claims to be, is still visible to that layer. Getting the session mechanics right is necessary but it isn’t the whole picture.

Building it so it doesn’t fall over in a week

The pattern that holds up in production separates concerns cleanly: a session-management layer that owns login, token refresh, and proxy pinning per account; a scraping layer that consumes valid sessions and never touches the login flow directly; and monitoring that alerts on logout detection, not just on request failures. Treat the login itself as a rare, expensive operation you protect and log, and treat the session as a resource with a lifetime you manage explicitly rather than something you re-derive from scratch every run.

None of this makes an authenticated pipeline immune to being blocked, rate-limited, or having a session revoked. It just means the failure shows up as a clear signal your monitoring catches, instead of a pipeline that quietly runs for hours against a login page.

If you’re building out the proxy and session infrastructure this kind of pipeline depends on, see what we run.

Get new guides and videos first — join the Telegram channel.