Category: Uncategorized

Why this comes up at all

Most scraping pipelines start the same way: point a headless browser at a URL, wait for the page to settle, pull the DOM, parse it. That works, and for a lot of sites it’s the only option. But it’s also the most expensive way to get data. A full browser context burns CPU and RAM rendering CSS, executing every script on the page, loading fonts and images you don’t need, and waiting on network idle timers that exist for a human’s benefit, not yours.

Somewhere in the middle of debugging a slow pipeline, you open the network tab to see what the page is actually loading, and you notice a single POST request to /graphql carrying the entire payload you were trying to scrape out of the rendered HTML. At that point the question changes from “how do I render this page reliably” to “why am I rendering this page at all.”

That’s the shift this article is about. Not a trick, just recognizing the shape of the underlying request and deciding whether talking to it directly is the better engineering choice.

What’s actually happening under the page

A lot of modern web apps, especially ones built with React, Vue, or Svelte on top of a GraphQL backend, don’t fetch data as a series of REST calls. They send one POST request to a single endpoint, usually /graphql or /api/graphql, with a JSON body that contains three things: an operationName, a query string written in GraphQL’s query language, and a variables object. The server reads the query, resolves exactly the fields the client asked for, and returns a JSON response shaped to match.

That’s the whole appeal of GraphQL from the frontend team’s side: one round trip, no over-fetching, no under-fetching. From a scraping perspective it means the data you want is very often sitting in a single, already-structured JSON response, with none of the HTML wrapper, none of the CSS class soup, none of the guesswork about whether a value is in a data-* attribute or buried three divs deep.

How to spot it

You don’t need anything exotic to find this. Open the browser’s dev tools, go to the network tab, filter by Fetch/XHR, and reload the page. If the app is built on GraphQL, you’ll see requests to the same endpoint over and over, each with a different operationName and a different query body. Click into one, look at the response, and compare it against what’s rendered on the page. If the numbers match, you’ve found the source of truth.

The request headers matter here too. You’ll usually see a content-type: application/json, sometimes a apollo-require-preflight or similar client-library header, and whatever auth the app uses, a bearer token, a session cookie, or both. Those headers exist because the server is checking them, not because they’re decorative, so replaying the request means replaying the full set, not just the query body.

Introspection, and why it’s often turned off

GraphQL has a built-in feature called introspection: you can send a special query asking the schema to describe itself, every type, every field, every argument. In development this is how tools like GraphiQL build their autocomplete. In production, a lot of teams disable it deliberately, precisely because it hands an outside party a complete map of the API surface, including fields that were never meant to be public-facing.

If introspection is off, you don’t get the map for free. You’re limited to the operations you can actually observe being sent by the real client, which is usually enough for a defined scraping target but won’t give you the full schema. This is a defensive choice on the server’s part, and it’s common enough that assuming introspection will be available is a bad default.

Persisted queries close the door further

Some of the more sophisticated setups, GitHub and several large consumer apps among them, use a technique called automatic persisted queries (APQ). Instead of sending the full query text on every request, the client sends a hash of the query. The server maintains a lookup from hash to query text, either pre-registered at build time or cached the first time it sees the full query. Repeat requests only need to send the hash plus the variables.

From the outside, this means the request body you see in the network tab might just be a SHA-256 hash and a variables object, with no readable query text anywhere in the traffic. You can still replay that exact request since the hash-to-query mapping already exists server-side, but you can’t construct new queries against fields you haven’t observed being requested by the real client, because you have no query text to hash. It’s a meaningful narrowing of what’s reachable, and it’s worth recognizing as intentional rather than incidental.

The parts that don’t go away

None of this removes the things that make scraping hard in general. The endpoint still checks authentication the same way the REST equivalent would. Session cookies expire. Bearer tokens tied to a login still require a login. Rate limiting still applies to API traffic, and in a lot of shops it applies more aggressively to the GraphQL endpoint than to the general page routes, because the team knows it’s the direct line to their database and it’s the first thing they’d rate-limit if they were worried about load.

Query complexity is its own defensive layer specific to GraphQL. Because a client can technically ask for deeply nested relationships in one request, a naive schema is an open invitation to a single query that fans out into millions of database rows. Most production GraphQL servers run cost analysis or depth limiting on incoming queries and reject or throttle anything that crosses a threshold. That’s not a bot-detection measure in the CAPTCHA sense, it’s closer to a resource-protection measure, but it behaves the same way from your side: some queries you’d like to send just get rejected with a complexity error.

And underneath all of it, the transport-level fingerprinting that applies to any HTTP traffic still applies here. TLS handshake characteristics, header ordering, request timing, all of it gets evaluated by the same bot-detection systems whether the request is a page load or a GraphQL POST. Swapping from rendered HTML to a direct API call changes what you’re asking for, not who’s watching the wire.

Where this fits in a pipeline

The practical upside is real: a direct GraphQL call skips browser startup time, skips rendering, and returns exactly the fields you specified rather than a full page you then have to parse. On infrastructure that’s paying for compute per job, that’s a meaningful cost difference at volume. It’s also less brittle in one specific way, since you’re no longer dependent on a CSS selector or DOM structure that a frontend deploy can silently change. The tradeoff is that you’re now coupled to the query shape and the schema itself, and a backend change to a field name or a resolver will break you just as fast, just in a different place.

This only applies to endpoints and data you’re authorized to access under the site’s terms. A GraphQL endpoint enforcing the same auth as the rendered page means the same account, permission, and usage boundaries apply, an API call isn’t a loophole around the rules that govern the account making it. And introspection being off, or persisted queries being in place, isn’t a bug to route around, it’s the operator’s stated choice about what surface they’re willing to expose. Treat it as one.

If you want more of this kind of infrastructure-level breakdown, from how scraping pipelines are actually built to how bot detection systems work on the defensive side, that’s what we cover at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

What “a real collection schedule” actually looks like

Most orchestration writeups compare Dagster and Prefect on toy examples: a three-step ETL that pulls a CSV, transforms it, and loads it into a warehouse. That’s not what a collection schedule looks like in production. A real one is more like this: forty to a few hundred sources, each with its own cadence (some hourly, some daily, some weekly), each capable of failing independently, and each generating a dataset that downstream jobs depend on. Some runs need to be backfilled for a specific date range after a source changes its markup. Some need rate limiting so you don’t hammer a target. Some need to be paused without touching the other ninety-nine.

That’s the frame this comparison uses. Both Dagster and Prefect can run this kind of schedule. They get there with different assumptions baked into their core model, and those assumptions matter more once you’re past a handful of jobs.

Two different mental models

Dagster is built around software-defined assets. You declare what a piece of data is (a table of scraped listings, a normalized dataset, a report) and how it’s produced, and Dagster tracks the dependency graph between those assets, including their materialization history. The orchestration layer is secondary to the data model. This shows up directly in how you’d represent a collection job: each source’s output is an asset, and Dagster can tell you which ones are stale, which ones failed to materialize on their last scheduled run, and what depends on them downstream.

Prefect is built around flows and tasks: you write Python functions, decorate them, and Prefect manages execution, retries, and state around that code. There’s no separate asset abstraction unless you opt into Prefect’s newer asset features. The mental model is closer to “here’s a function that does the thing, run it on this schedule, handle failure this way.” For teams whose scraping logic already lives in Python scripts with clear entry points, this is a smaller lift: you decorate the existing function rather than restructuring around a dependency graph.

Neither model is wrong for scraping. Dagster’s asset framing pays off once you care about freshness and lineage across many sources feeding a shared downstream dataset. Prefect’s flow framing pays off when the logic itself is dynamic and imperative, like a flow that decides at runtime how many pages to crawl based on what the first page returns.

Retries, rate limits, and the things that actually break

In a collection schedule, the failures aren’t dramatic. They’re mostly a source returning a 429, a proxy exiting a pool mid-run, a target changing a CSS selector, or a request timing out because a destination site is slow that day. What matters is how gracefully the orchestrator absorbs that.

Both tools support retry policies at the unit-of-work level: Dagster on ops and assets, Prefect on tasks. Both support configurable backoff and retry counts. The practical difference is granularity. Prefect’s task-level retries are easy to reason about in a single flow, since a task maps closely to “one HTTP call” or “one page fetch,” and you can wrap individual calls in their own retry logic without touching the rest of the flow. Dagster’s retries are usually set at the asset or op level, which tends to be a coarser unit, so a transient failure on one HTTP call inside a larger asset can trigger a rerun of more work than strictly failed, unless you break that asset down further.

Neither orchestrator manages proxy rotation, IP health, or backoff against a specific target’s rate limits for you. That logic lives in your scraper code or in the proxy layer in front of it. What the orchestrator does is decide what happens to the job when that code raises an exception: retry it, mark it failed, alert someone, or quarantine that one source while letting the rest of the schedule proceed. That’s the layer where these two tools actually differ, not in whether they can technically retry a task.

Backfills and partitioning

This is where the asset model earns its complexity. Dagster has native partitions: you can define a daily or per-source partition on an asset, and if you need to reprocess the last fourteen days of collection for one source because a parser bug corrupted the output, you select that date range and one source, and Dagster runs exactly those materializations. The partition history is visible in the UI as a grid, which is a genuinely useful view when you’re auditing “which of our 90 sources actually collected successfully yesterday.”

Prefect doesn’t have a first-class partition concept in the same sense. You get there by parameterizing flow runs, for example a flow that takes a source_id and date and triggers however many runs you need for a backfill, often scripted or scheduled through deployments. It works, and for teams that already think in terms of “just run this function with different arguments,” it’s a more familiar shape. But you’re building the bookkeeping of what’s been backfilled and what hasn’t yourself, whereas Dagster keeps that state for you as part of the asset’s materialization record.

If backfills and per-source freshness tracking are a regular part of the job, this is a real point in Dagster’s favor. If backfills are rare and mostly “rerun this flow with different dates,” Prefect’s simpler model is less overhead for the same outcome.

Watching the pipeline when it’s 3am and a source changed

Every collection schedule eventually has a night where a target site redesigns its page and a scraper starts silently returning empty results instead of erroring. Neither orchestrator will catch that automatically. What they give you is a place to put the check.

Dagster has asset checks and freshness policies: you can attach a check that asserts a materialized asset has more than N rows, or that it was updated within the expected window, and have that surface as a failed check in the same UI where you see the asset graph. It ties the “did this run” question and the “was this run any good” question together in one view.

Prefect gives you automations, which are event-driven rules you can attach to state changes (a flow run failing, a task retrying past a threshold) and route to notifications. You can build an equivalent quality check by adding a validation step inside the flow that raises if the output looks wrong, which then triggers the failure state and any automation attached to it. It’s the same outcome, assembled from slightly more general-purpose parts.

Deployment and where the workers actually run

Both are open source and self-hostable, and both also offer a managed cloud option if you don’t want to run the control plane yourself. Dagster runs a daemon process plus code locations that hold your definitions; Prefect runs work pools and workers that pick up scheduled flow runs. Operationally they land in a similar place: a persistent process or two, a database backing run history, and workers that need network access to wherever you’re actually running the scrape (which matters if your collection jobs need to originate from specific egress points or proxy infrastructure rather than wherever the orchestrator’s control plane happens to sit).

Which one we’d reach for

For a collection schedule with a stable set of sources where lineage, per-source freshness, and clean backfills matter, Dagster’s asset model does real work for you that you’d otherwise build by hand. For a schedule where the logic is more ad hoc, changes often, or is still closer to a pile of scripts than a stable dependency graph, Prefect’s flow model gets you running with less upfront structure. Neither one replaces the retry logic, rate limit awareness, or proxy management that has to live in your scraping code either way. The orchestrator decides what happens around a failure. It doesn’t decide whether the failure happens in the first place.

If you’re building out collection infrastructure and want more breakdowns like this on orchestration, proxy setups, and scraping tooling, check out the rest of the site here.

Get new guides and videos first — join the Telegram channel.

The problem with just rotating an IP

Anyone who has run a proxy farm long enough learns this the hard way: swapping the IP address in front of a request doesn’t make the request anonymous. The site on the other end isn’t just looking at where the traffic came from. It’s looking at what rendered the page, what generated the audio buffer, and which fonts got measured during layout. Those three signals, canvas, audio, and fonts, are some of the oldest and still most widely deployed pieces of browser fingerprinting. This is a look at how each one actually works under the hood, not a guide to getting around them.

What a canvas fingerprint actually measures

Canvas fingerprinting uses the HTML5 <canvas> element, which was built for drawing graphics in the browser, as a side channel for hardware and software identification. A script draws some combination of text, shapes, and gradients onto an off-screen canvas, then calls toDataURL() or getImageData() to read back the raw pixel data.

The trick is that the same drawing instructions don’t produce identical pixels across machines. Anti-aliasing, sub-pixel rendering, font hinting, and GPU-level rasterization all vary by operating system, graphics driver, and even the specific GPU model. Two machines running the same browser version can still produce pixel-level differences because one is rendering through an Nvidia driver and the other through Intel integrated graphics. The script hashes the resulting pixel buffer, and that hash becomes a stable identifier that persists across sessions, cookie clears, and IP changes.

This is why canvas fingerprinting is popular with detection vendors: it’s cheap to compute, it requires no user interaction, and it exposes real hardware and driver differences that are hard to fake convincingly because faking them means either running on genuinely different hardware or intercepting and rewriting the canvas API output in a way that itself can look inconsistent with everything else the browser reports.

Audio context fingerprinting

The Web Audio API works on a similar principle. A script creates an OfflineAudioContext, generates a signal (often a sine wave or triangle wave through an oscillator node), routes it through a dynamics compressor, and reads back the resulting audio buffer.

Audio processing on a real device runs through the actual audio stack: the sample rate the hardware supports, the specific implementation of oscillator and compressor nodes in that browser build, and floating-point rounding behavior that differs slightly between CPU architectures and operating systems. Even though nothing is played out loud, the numeric buffer that comes back carries small, consistent variations tied to the underlying hardware and software stack.

Like canvas, this produces a hash that’s stable for a given device and browser combination, and it costs the site almost nothing to run since it happens in milliseconds without any visible UI change.

Font enumeration and rendering signals

Font fingerprinting works differently depending on the method. The older approach is enumeration: a script tries to render text in a long list of font names and measures the resulting text width and height. If a font isn’t installed, the browser silently falls back to a default, and the measured dimensions match that fallback. If it is installed, the dimensions differ. Repeating this across a few hundred font names produces a list of which fonts are present on the machine, which is itself a fairly identifying signal because font libraries vary by OS, OS version, installed creative software (design tools bundle their own fonts), and region-specific language packs.

A newer and quieter approach reads font metrics directly through canvas text measurement (measureText()) without needing a visible list of font names at all, since the rendering differences alone leak information about the underlying font-rendering engine and installed font set.

Fonts matter to fingerprinting because they correlate with real-world identity signals that are hard to fake as a set. A machine reporting Adobe Creative Cloud fonts, a specific set of CJK language packs, and a Windows font-hinting profile paints a coherent picture of a real desktop. A synthetic environment that reports a font list mismatched with its claimed OS, or an oddly short and generic font list, stands out precisely because real machines rarely look that clean.

How these signals combine into a fingerprint

None of these three signals is decisive on its own. What makes device fingerprinting effective is combining dozens of signals like this, canvas hash, audio hash, font list, WebGL renderer string, screen resolution, timezone, installed plugins, hardware concurrency, into a composite fingerprint. Detection systems then look for two things: internal consistency and population rarity.

Consistency means checking whether the signals agree with each other. A user agent claiming macOS should come with a font list that matches macOS’s default fonts, a WebGL renderer string from Apple’s graphics stack, and canvas rendering behavior consistent with macOS’s font hinting. When automation frameworks spoof a user agent without correspondingly adjusting canvas and audio output, that mismatch is often more useful to a detection system than any single signal in isolation.

Rarity means checking how many other visitors share the exact same fingerprint. A completely unique combination of signals is memorable and easy to re-identify across visits, which cuts against anyone trying to look anonymous. But a fingerprint that’s suspiciously identical across thousands of sessions, which happens when a fleet of automated browsers all run the same environment configuration, is just as much of a signal in the other direction.

Why this matters for anyone running scraping infrastructure

Running proxy infrastructure at any real scale means confronting the fact that IP rotation and fingerprint management are separate problems. A clean residential IP behind a browser that reports a canvas hash shared by ten thousand other sessions, or an audio fingerprint that doesn’t match its claimed OS, gets flagged regardless of how clean the network path looks. This is one of the reasons headless browser detection has moved well past checking navigator.webdriver and now leans heavily on these rendering-level signals, because they’re expensive to fake convincingly and cheap for the site to check.

It’s also why browser automation tools differ so much in how they’re received by anti-bot systems. A framework that runs a real, unmodified browser engine on real hardware produces canvas and audio output that’s genuinely tied to that hardware, for better or worse. A framework that patches the canvas or audio API to return spoofed or randomized values can end up creating a different kind of tell: values that are internally inconsistent, or that change on every session in a way real hardware never would, since real GPU and audio stack behavior is stable for a given machine, not random per page load.

None of this makes any single tool undetectable or a detection system infallible. Fingerprinting is probabilistic. Sites tune their thresholds based on false positive tolerance, and a fingerprint that would get flagged on a payments page might pass unnoticed on a low-value content page. What’s consistent across the systems we’ve looked at is that they’re evaluating the whole signal set together, not any one piece of it.

How defenders actually use this

On the defensive side, canvas, audio, and font signals get folded into a device reputation score alongside behavioral signals like mouse movement, typing cadence, and navigation timing. A mismatched or overly generic fingerprint doesn’t usually trigger an outright block by itself. More often it raises a risk score that combines with other factors, IP reputation, request rate, session history, before a decision gets made to serve a CAPTCHA, throttle the request, or silently degrade the response. Understanding this layered approach is more useful for anyone building or evaluating scraping infrastructure than trying to defeat any single check, because the check that matters is rarely the one you can see.

If you’re building or auditing scraping and proxy infrastructure and want more breakdowns like this on how detection systems actually work, Data Research Tools covers the engineering side of fingerprinting, bot detection, and the pipeline tooling that has to account for it.

Get new guides and videos first — join the Telegram channel.

The question nobody schedules time for

Every scraping pipeline we’ve run ends the same way. The crawl finishes, the parsed data lands in a database or a CSV, someone ships the report, and the raw HTML sits in a bucket somewhere because deleting it felt premature and nobody wanted to be the one who pulled the trigger too early. Six months later that bucket is still there, still costing money, and nobody remembers why.

Raw HTML retention isn’t a technical problem so much as a decision-making problem. The technical part (where to store it, how to compress it) is easy. The hard part is deciding when it stops being useful and starts being liability. This post walks through how we think about that decision on our own infrastructure.

Why raw HTML piles up in the first place

Most scraping pipelines separate collection from parsing. You fetch the page, save the raw response, then run a parser against it to pull structured fields. That separation exists for a good reason: if your parser has a bug, or the site changes its markup mid-crawl, you can reprocess the raw HTML instead of re-scraping the target. Re-scraping costs proxy bandwidth, time, and risk of getting rate-limited or blocked again. Reprocessing a file on disk costs nothing but CPU.

That’s a real, defensible reason to keep raw HTML around, at least for a while. The problem is that “at least for a while” quietly turns into “indefinitely” because nobody sets an expiry date at collection time. A crawl that pulls 200,000 pages a day, saved uncompressed, adds up to real storage spend within weeks. We’ve seen internal projects where raw HTML storage cost more per month than the proxy bill that generated it, simply because retention was never revisited after the project’s active phase ended.

What raw HTML is actually worth after the project ends

Once a project is “done” (the data has been extracted, validated, and delivered), the raw HTML has three possible futures:

It gets reprocessed. Someone finds a parsing bug, a new field gets requested, or a downstream consumer wants a value that wasn’t captured the first time. This is the strongest argument for retention, but it has a shelf life. If the site’s markup has since changed, or the underlying page no longer exists, the raw HTML is a snapshot of a moment that can’t be reproduced by scraping again. That snapshot value is highest in the days and weeks right after collection and drops off as the target site evolves out from under it.

It gets audited. Some projects need to prove what a page said at a point in time, for a compliance record, a dispute, or a research citation. This is a narrower use case than most teams assume. If nobody has asked for that kind of audit trail in the project’s lifetime, it’s unlikely they’ll ask for it later, and if there’s a real regulatory or contractual reason to keep evidence, that reason should be explicit and documented, not assumed.

It never gets touched again. This is the most common outcome by far. The parsed data was correct, the project shipped, and the raw HTML sits untouched until someone notices the storage bill or a retention audit forces the question.

The mistake is treating all raw HTML as if it falls into the first or second bucket by default. It doesn’t. Most of it falls into the third, and the only way to know which bucket a given batch belongs to is to look at actual access patterns, not assumptions made when the crawl was designed.

A framework that actually holds up

We don’t keep raw HTML on a single fixed retention period across every project, because projects differ too much in size and purpose. Instead we think in three tiers.

Short-term hot storage (days to a couple of weeks). Immediately after collection, raw HTML stays fully accessible for reprocessing while the parser is still being validated. This is the window where bugs get caught and fields get added. Keep it uncompressed or lightly compressed, keep it fast to query, and treat this period as part of the active project, not archival.

Compressed cold storage (weeks to a few months, project dependent). Once the parsed output has been validated and delivered, raw HTML moves to cheaper, slower storage, compressed. Gzip or zstd on raw HTML routinely gets 80-90% size reduction because markup is extremely repetitive text. At this point access should be rare and deliberate: someone requests it because a specific reprocessing need came up, not because it’s routinely queried. The retention length here should map to how often that kind of request has historically happened on similar projects, not to a gut feeling.

Deletion. After the cold storage window closes without a reprocessing need materializing, delete it. This is the step teams skip, because deletion feels like it forecloses an option, while keeping data feels free. It isn’t free. Storage costs money every month it sits there, and stored personal or scraped data that nobody has a current use for is exactly the kind of thing that turns into a liability if a target site, a customer, or a regulator asks what you’re holding and why.

The lengths of each tier depend on the project. A one-off market research pull that fed a single report might go from hot to deleted in under a month. A pipeline feeding an ongoing dataset that gets reprocessed regularly might keep a rolling cold-storage window measured in months, refreshed as new crawls replace old ones. The point isn’t to copy a specific number, it’s to set a number in advance and write it down, so retention is a decision made once at project design time instead of a default that nobody revisits.

The legal and scope angle

Raw HTML retention isn’t just a storage cost question. If the pages you scraped included anything that falls under personal data rules, the raw markup is a second copy of that data sitting outside your structured store, often with less access control and less awareness that it exists at all. A parsed database might have field-level permissions and a defined schema. A folder of raw HTML dumps usually doesn’t.

This matters most for projects touching regulated categories: anything with names, contact details, or other personal information embedded in the page. Treat raw HTML holding personal data with the same retention discipline you’d apply to the parsed output, not a looser one. If your parsed data has a defined retention and deletion policy because of GDPR, CCPA, or a contractual term with a client, the raw HTML copy needs to be covered by that same policy, or deleted sooner, since it typically has less structure and less oversight.

For projects that never touched personal data, purely product listings, public pricing, aggregate metrics, the legal pressure is lower but the storage argument still holds. Keeping terabytes of raw markup for pages that were fully parsed and validated months ago is a cost with no offsetting benefit.

What we actually do

On our own pipelines, retention gets set as a config value at project creation, not decided ad hoc later. Hot storage runs for the length of the active QA phase (usually the first one to two weeks after a crawl completes, since that’s when we’ve historically found parsing issues). Cold storage runs for a window tied to the project’s expected reprocessing cadence, logged in the project’s own documentation so the reasoning is visible later, not just the number. Deletion is a scheduled job, not a manual cleanup someone has to remember to run.

The failure mode we’ve seen most often isn’t keeping HTML too short, it’s never deciding at all, and ending up with years of accumulated raw pages that nobody can justify keeping and nobody wants to be responsible for deleting. Set the retention window when you design the pipeline, write down why you picked it, and let deletion happen automatically. That single habit prevents most of the storage and compliance headaches that raw HTML tends to create.

If you’re building out scraping infrastructure and want more on pipeline design, proxy management, and the operational side of running these systems at scale, check out the rest of what we cover at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The problem Docker is actually solving

A scraping stack has more moving parts than most people account for when it “worked on my laptop.” There’s the browser binary, the driver that talks to it, a rendering engine with its own font and codec dependencies, a language runtime, a pile of Python or Node packages, and often a headless display server like Xvfb sitting underneath the browser to give it something to render into. Any one of those can drift between two machines and change how a page renders, how a target site fingerprints the client, or whether the scrape runs at all.

Docker doesn’t remove that complexity. It freezes it. An image is a snapshot of exact package versions, exact library paths, exact browser builds, built once and run identically wherever the container starts. That’s the whole value proposition for a scraping environment: not speed, not isolation for its own sake, but the elimination of “which version of Chromium is actually on this box” as a debugging question.

What actually breaks a headless stack across machines

Most environment drift in scraping infrastructure comes from a small set of repeat offenders.

Browser auto-updates are the biggest one. If Chrome or Chromium updates itself between deployments, the User-Agent string, the JS engine’s behavior, and sometimes the DOM APIs available all shift slightly. A scraper tuned against one build can start failing silently against the next.

System font packages are another. Headless Chromium renders using whatever fonts are installed at the OS level. A base image with a different font set than production produces different canvas and text-measurement fingerprints, which matters both for rendering fidelity and for any fingerprinting-adjacent detection running on the target side.

GPU and rendering flags matter too. Whether SwiftShader, ANGLE, or a real GPU backend is in play changes WebGL and Canvas output at the byte level. Locale and timezone packages affect Intl API output and date formatting. And the driver-to-browser version pairing (chromedriver to Chrome, for instance) is notoriously strict; a one-point version mismatch can throw a hard connection error before a single page loads.

None of these are exotic. They’re the ordinary cost of running software on more than one machine, and they show up constantly in farm operations where the same job gets scheduled across a heterogeneous pool of hosts.

What “pin everything” means in practice

A reproducible scraping image pins at every layer, not just the top one.

Base OS image: a specific tag, not latest. debian:bookworm-slim pinned to a digest, not debian:stable, which moves underneath you.

Browser binary: installed from a specific version, not the distro’s rolling package. Teams running headless Chrome or Chromium at scale typically pull a specific build number and disable the browser’s self-update mechanism entirely, since an image that updates itself at runtime isn’t reproducible by definition.

Driver and automation library: version-locked to match the browser build. Playwright and Puppeteer both bundle a browser download tied to the library version for this exact reason, which is part of why teams that need version discipline lean on the framework’s bundled browser instead of pointing it at a system install.

Language dependencies: a lockfile, not a loose requirements list. package-lock.json, poetry.lock, pip-compile output. The Dockerfile installs from the lockfile, so docker build produces the same dependency tree today and six months from now, as long as the base image and lockfile don’t change.

System libraries: the ones headless Chromium needs to actually render (libnss3, libatk, libgbm, font packages) are enumerated explicitly rather than pulled in as transitive dependencies of something else, because “something else” changes over time.

The result is an image where two containers built from the same Dockerfile, run six months apart, produce the same browser fingerprint surface. That’s the actual deliverable: not “it runs,” but “it renders identically.”

Xvfb, display servers, and why headless isn’t always headless

Plenty of scraping stacks don’t run pure headless mode. Some target sites behave differently against headless Chromium than against a “headed” browser, since certain automation flags and missing browser features are themselves detectable signals. The workaround infrastructure teams use is a virtual framebuffer, Xvfb, which gives the browser a real X display to render into inside a container that has no physical screen.

That adds another dependency to pin: the Xvfb package version, the resolution and color depth it’s started with, and the startup script that launches Xvfb before the browser process. It’s a small piece but it’s exactly the kind of thing that gets hand-configured once on a dev machine, never written down, and then silently missing when the job moves to a fresh host. Baking the Xvfb start command into the image’s entrypoint, rather than into a README, is what makes it survive the move.

Multi-stage builds keep the runtime image honest

Headless browser dependencies are heavy. A naive Dockerfile that runs apt-get install for build tools, compiles anything from source, and leaves all of that in the final image ends up bloated and slower to pull, which matters when a farm is starting containers across many hosts on a schedule.

Multi-stage builds separate the build environment from the runtime image. Compilation, dependency resolution, and any wheel-building happen in an intermediate stage; only the final artifacts, the installed packages and the browser binary, get copied into the slim runtime image that actually ships. This isn’t just about image size. A smaller, more explicit runtime image also means fewer packages that could silently update or conflict, which is a reproducibility win as much as a storage one.

Where proxy and network config fits

The browser and driver aren’t the only things that need to be identical across runs. Proxy configuration, DNS resolution behavior, and any custom CA certificates for TLS interception (used in some debugging setups, not target-side evasion) all belong inside the image or its startup config, version-controlled alongside the Dockerfile. If proxy credentials or endpoint lists live in an .env file injected at runtime rather than baked into the image, that’s the correct pattern: it keeps the image itself portable and free of secrets, while still making the configuration surface explicit and auditable rather than something set by hand on one box and forgotten.

Teams running distributed scraping infrastructure across many exit points tend to treat proxy assignment as orchestration-layer concern, not image-layer concern, for exactly this reason. The image defines how the browser behaves. The orchestrator decides which proxy that container talks through on a given run.

What Docker doesn’t fix

It’s worth being direct about the limits here. A container gives you a reproducible software environment. It does not give you a consistent network fingerprint, IP reputation, or TLS handshake signature; those come from whatever’s routing the container’s traffic, not the image. It doesn’t make a headless browser behave like a residential user’s browser on its own, since sites doing serious bot detection look at signals well beyond the browser version: mouse and scroll telemetry, timing patterns, TLS fingerprints (JA3/JA4), and IP reputation data that no Dockerfile touches.

And reproducibility isn’t a detection-evasion technique. Freezing a browser version doesn’t hide automation flags; it just means the automation flags stay consistent instead of drifting unpredictably. Detection systems that fingerprint navigator.webdriver, headless-specific rendering quirks, or missing browser plugins will still see the same signals in a pinned container that they’d see anywhere else running the same configuration. Docker’s job here is honesty about state, not concealment of it.

What actually goes wrong when teams skip this

The realistic failure mode isn’t dramatic. It’s a scraper that worked last month, gets redeployed to a new host or after a routine apt-get upgrade in the Dockerfile, and now silently returns malformed pages or empty results because Chromium auto-updated to a build with a different rendering path, or a font package changed how a canvas fingerprint gets computed downstream. Nobody notices until the data quality checks catch it, if there are any. Pinning versions and building in stages doesn’t make debugging unnecessary. It removes one whole category of debugging: the kind where the code didn’t change but the environment did.

Want more breakdowns of the infrastructure behind production scraping and proxy pipelines? Check out the rest of the site here.

Get new guides and videos first — join the Telegram channel.

A proxied scraper can look correctly routed on every metric you check and still leak. The TCP connection goes out through the exit node, the response comes back through the exit node, and the target sees the proxy’s IP the whole time. But before any of that happens, something on the collection host had to turn a hostname into an IP address, and that step doesn’t automatically travel through the proxy just because the rest of the request does. When it doesn’t, the operator’s real network path gets exposed to whatever DNS resolver handled the lookup, even though the actual HTTP traffic never touched it.

This is what people mean by a DNS leak in a proxy context, and it’s a separate failure mode from a proxy going down or getting blocked. The proxy can be working perfectly and the leak still happens, because resolution and connection are two different steps in the stack, handled by different pieces of code, and they don’t inherit the same routing by default.

What’s actually leaking

DNS resolution is a client asking a resolver “what’s the IP for this hostname,” and the resolver answering. If that question gets asked from the collection host directly, over the host’s normal internet connection, then whoever operates that resolver, whether it’s the host’s ISP, a public resolver like a DoH provider, or a resolver baked into a container’s default network, sees the real source IP and ASN of the machine doing the crawling, along with the hostname it’s about to visit.

None of that is visible to the target site. The target still only sees the proxy’s exit IP. The leak is upstream of the target, at the resolver. But it’s still a leak, because it defeats the point of routing the collection through a proxy pool in the first place: the goal was to make the crawling traffic look like it originated somewhere else, and DNS resolution quietly reveals where it actually originated.

SOCKS5 has two resolution modes, and they look almost identical

This is the most common place the leak shows up, because the difference is one letter and most people don’t notice it. SOCKS5 supports sending either a raw IP address or a domain name as the destination address type. If a client sends the domain name, the proxy server resolves it. If the client resolves it first and sends an IP, the client’s own resolver does the work.

curl and most SOCKS5 libraries expose this as two separate proxy schemes. socks5:// resolves the hostname locally, on the machine running curl, before ever talking to the proxy. socks5h:// passes the hostname through and lets the proxy server do the resolution. Same proxy, same exit IP, completely different DNS path. Writing socks5://proxyhost:1080 instead of socks5h://proxyhost:1080 is a one-character typo that produces a fully working scraper with a fully leaking resolver.

Python’s requests library, when backed by PySocks, has the same distinction. So does proxychains, which calls this the “remote DNS” setting. It’s worth actually checking which one a given tool defaults to rather than assuming, because defaults vary and some libraries silently fall back to local resolution if the remote lookup fails, which means a leak can appear intermittently under specific error conditions rather than consistently.

HTTP and HTTPS proxies are usually fine, with some exceptions

Forward HTTP proxies work differently and are generally safer on this specific point. For plain HTTP, the client sends an absolute-form request line, GET http://example.com/path HTTP/1.1, straight to the proxy, and the proxy is the one that resolves the hostname. For HTTPS, the client sends CONNECT example.com:443 to the proxy and the proxy resolves and opens the tunnel. In both cases the client never needs to know the target’s IP address at all.

Where this breaks down is in code that resolves ahead of the actual request for some other reason. Health check logic that pings a hostname before deciding to route through it. Certificate pinning implementations that resolve to fetch or validate against a known IP. Connection pooling libraries that pre-resolve to key their pool by IP instead of hostname. Retry logic that catches a connection failure and falls back to a “direct” resolve-and-connect path without going through the configured proxy at all. None of these are exotic; they show up in production HTTP clients regularly, and they resolve outside the proxy without the operator ever setting a socks5:// scheme wrong, because the leak isn’t in the proxy configuration, it’s in a secondary code path that doesn’t check whether a proxy is configured before it runs.

Containers add another resolver in the chain

Running collection workers in containers adds a hop that’s easy to overlook. Docker’s default bridge network ships an embedded DNS server at 127.0.0.11 inside each container, which forwards queries it can’t answer to whatever resolver the host is configured to use, usually from /etc/resolv.conf on the host or a systemd-resolved stub. If the application inside the container honors HTTP_PROXY and HTTPS_PROXY environment variables for its HTTP traffic but has no equivalent DNS configuration, the actual request goes through the proxy while the resolution goes through the container’s embedded resolver, out to the host, out to whatever upstream DNS the host uses, over the host’s real network interface.

This gets worse if the host, or the resolver library inside the container, is configured to use DNS over HTTPS. A DoH query to a public resolver is itself an HTTPS connection, and unless something explicitly routes that connection through the proxy too, it goes out directly, over the real interface, to the DoH provider, who now has a direct log entry connecting the operator’s real IP to the hostname about to be scraped.

What this costs, beyond getting blocked

A DNS leak doesn’t usually get a scraper detected by the target site directly, since the target never sees the leaked query. The cost is different. It hands metadata about the operation, which hostnames are being crawled and how often, to a third party the operator doesn’t control: an ISP, a public DoH provider, or whatever resolver happens to be configured on the box. For anyone running a proxy pool specifically to keep collection infrastructure separate from their own identity, that’s the exact thing the proxy pool was supposed to prevent.

It also creates a stable fingerprint across an otherwise rotating setup. If exit IPs rotate through a proxy pool but every session resolves DNS from the same origin ASN, that origin is a constant sitting underneath a variable, and it’s the kind of constant that correlates sessions together even when the visible IP changes on every request.

How to actually check for it

The reliable way to check is a packet capture on the collection host while the scraper runs, filtering for UDP port 53, TCP port 853 for DNS over TLS, and TCP 443 traffic to known DoH resolvers. Anything showing up on those filters, other than traffic addressed to the proxy itself, is a leak. This is more trustworthy than reading library documentation, because the actual behavior can drift between versions and between code paths in ways the docs don’t cover.

curl -v against a real target through the proxy is a fast manual check. With a working socks5h:// proxy, curl’s verbose output shows it connecting to the proxy host and never resolving the target itself. With socks5://, the log line shows curl resolving the target’s hostname to an IP before it ever talks to the proxy, which is visible right there in the trace.

Third-party DNS leak test services, the kind that hand back a unique subdomain and log which resolvers query it, work well for this too, as long as the request that triggers the lookup goes through the exact same code path and network configuration as the real scraper, not a browser tab opened separately on the same machine. Testing the browser instead of the actual worker process is a common way to get a clean result that doesn’t reflect what the pipeline actually does.

A short defensive checklist

Use socks5h:// rather than socks5:// for any SOCKS proxy, or the equivalent “remote DNS” flag for whatever tool is in use, and confirm it with a packet capture rather than trusting the config. For HTTP and HTTPS proxies, verify with a capture that the client is genuinely sending CONNECT or absolute-form requests rather than falling back to a direct resolve-and-connect path under retry or pooling logic. In containerized setups, either point the container’s resolver explicitly at something routed through the same proxy path, or run DNS resolution through a tool that tunnels it over the proxy alongside the HTTP traffic, rather than leaving it on the container’s default embedded resolver. For browser automation, disable DNS prefetching in the automation profile, since prefetch can trigger a lookup ahead of the navigation logic that’s supposed to be proxy-aware, and keep in mind that WebRTC is a separate leak vector in real browser sessions worth checking independently, since most proxies only carry TCP and WebRTC’s ICE gathering can use UDP that never touches the proxy at all. Re-verify after any change to the proxy library version, the container base image, or the automation framework, since this is exactly the kind of behavior that changes silently between releases.

None of this is a one-time fix. It’s a check that belongs in the same rotation as proxy health checks, because a resolver leak and a dead proxy fail in opposite directions: one makes the scraper stop working where you’d notice immediately, and the other keeps it working while quietly exposing the thing the proxy was there to hide.

Data Research Tools writes about scraping infrastructure, proxy pools, and the operational details that don’t show up in vendor marketing. Read more on the home page.

Get new guides and videos first — join the Telegram channel.

The bug that doesn’t crash anything

Most scraper failures are loud. A selector breaks, a 403 shows up, a proxy pool dies, and you know within minutes. Encoding bugs are the opposite. They don’t throw exceptions. They write a slightly wrong string into a column and move on to the next row, week after week, until someone runs a query that depends on the string being right and gets nothing back.

We’ve had this happen on pipelines that had been running clean for months. Nobody touched the code. Nothing alerted. The rot was already in every row, just waiting for the right query to expose it.

Where a page’s real encoding actually lives

A page can declare its character encoding in three places, and they don’t always agree.

The HTTP response can send a Content-Type header with a charset parameter, like text/html; charset=utf-8. This is supposed to be authoritative, but plenty of servers omit it, or set it once at the CDN edge and never update it when the origin app changes.

The HTML document itself can declare encoding in a <meta charset="..."> tag, or the older <meta http-equiv="Content-Type" content="text/html; charset=..."> form. This is what the browser actually trusts when rendering, and it’s often more accurate than the HTTP header on older or poorly maintained sites, because whoever built the template set it once and it’s been correct ever since, even as the server config drifted.

A byte order mark (BOM) at the very start of the file can also signal encoding, mostly on UTF-16 or UTF-8-with-BOM files, which show up more in scraped CSV or XML feeds than HTML.

When these three disagree, a scraper that only checks one of them decodes the bytes wrong, silently.

What your scraper does when the signals disagree

Most HTTP libraries pick the header first. If the header is missing or wrong, behavior differs by library and version. Some fall back to a default (historically ISO-8859-1 for text/* content without an explicit charset, per the old HTTP spec), some run a statistical guesser like chardet or cchardet against the byte stream, and some just hand you raw bytes and expect you to decide.

ISO-8859-1 is the dangerous default, because it maps every byte value to something. It never fails to decode. Feed it UTF-8 bytes for an accented character, a curly quote, or an em dash, and it happily produces a valid but wrong string. No exception, no warning. A curly right single quote in UTF-8 is three bytes; decoded as ISO-8859-1, it becomes three separate garbage characters that get treated as perfectly normal text and stored as-is.

That’s mojibake, and it’s the most common form of scraped-text corruption. It doesn’t look like an error. It looks like a string with a few weird characters in it, and a human skimming a sample of ten rows might not even notice.

The double-encoding trap

The second-most-common failure is double-encoding, and it’s nastier because it can happen downstream of a correct decode. If you take a string that’s already correctly decoded to Unicode and encode it to UTF-8 bytes, then something later in the pipeline decodes those bytes as ISO-8859-1 or Windows-1252 and re-saves the result as a Python string, you get a specific, recognizable pattern: an accented character like “é” turns into “é”. If that string then gets encoded to UTF-8 again by a different stage that assumes it’s still clean text, you get a second layer of garbling on top of the first.

This tends to happen at pipeline boundaries: a scraper writes clean UTF-8 to a file, a second script reads that file with the wrong default encoding, transforms the text, and writes it back out. Each hop is individually reasonable-looking code. The corruption only exists because two hops disagree about what encoding the bytes in between represent.

Why it takes months to show up

None of this breaks anything at write time. The row gets inserted, the job finishes, the count matches expectations. The corruption sits there as data, not as a failure.

It surfaces later, and usually through one of a few specific triggers. A filter or LIKE query on a name or title that contains an accented character stops matching, because the stored bytes don’t match the bytes in the query anymore, even though both look “the same” on screen. A deduplication or join step that keys on a text field starts producing duplicates it shouldn’t, because two representations of what a human considers the same string decode to different byte sequences. NFC versus NFD normalization is a common source of this: some systems (notably certain macOS-originated content and some CMS export tools) store accented characters as a base letter plus a combining mark instead of a single precomposed character, and those two forms are visually identical but byte-for-byte different.

Or the trigger is external: a target site adds a section in a non-Latin script, or a new source gets added to the pipeline, and the heuristics that happened to work fine on mostly-Latin text fail hard on Cyrillic or CJK text, because a statistical charset guesser trained on the wrong sample size makes different mistakes at scale.

By the time any of this is noticed, the raw source bytes are usually long gone. If the pipeline only ever stored the decoded (and now wrongly decoded) string, there’s no way to recover the original text. Re-scraping the source might not even be possible if the page has changed or the source is gone.

The database layer adds its own trap

Encoding bugs aren’t only introduced during decoding. They can be introduced at the storage layer even when the decode was correct. MySQL’s utf8 character set, despite the name, is a 3-byte-max encoding and can’t store the full Unicode range, including a lot of CJK extension characters and all emoji, which are 4 bytes in UTF-8. A column defined as utf8 rather than utf8mb4 will silently truncate or reject those characters depending on the SQL mode, and in permissive modes it can insert a partial or replacement character instead of erroring.

This is a pure configuration bug, invisible until a specific character shows up in scraped content. It’s common on tables that were created years ago under an old default and never migrated, sitting quietly correct for every row that happens to only contain characters within the 3-byte range.

Building a pipeline that actually catches this

A few habits make this class of bug findable instead of invisible.

Keep raw response bytes, or at least a hash of them, alongside the decoded text for some retention window. If corruption is found later, having the original bytes is the only way to re-decode correctly instead of guessing.

Resolve encoding with an explicit priority chain rather than trusting a single signal: BOM first if present, then the HTTP header if it declares a charset, then the HTML meta tag, then a statistical guess as a last resort, and log which method was used per response. That log is what lets you go back later and find exactly which pages or which time range used the fallback guesser, instead of auditing the whole dataset.

Normalize text to a single Unicode normalization form (NFC is the common choice) at ingestion time, consistently, so that visually identical strings from different sources are also byte-identical.

Run a small canary check as part of routine pipeline monitoring: a handful of known strings containing accented Latin characters, curly punctuation, a CJK sample, and an emoji, decoded and compared against expected output. It’s cheap to run on a schedule and it catches a broken default encoding or a bad library upgrade before it’s written into a few hundred thousand rows.

None of this prevents every encoding bug. But it turns “we found garbage in the data and have no idea when it started or what the correct text was” into “we found garbage in the data, here’s the exact run it started in, and here are the raw bytes to fix it.”

If you’re building or auditing a scraping pipeline and want more from us on the infrastructure side of this work, take a look at what we cover on Data Research Tools.

Get new guides and videos first — join the Telegram channel.

Every operator who’s run proxies at any real scale has had this argument with a vendor at some point. Your scraper logs 40GB for the month. The invoice says 61GB. Support tells you their meter is correct, points at a dashboard you can’t independently verify, and the conversation stalls. Before you assume anyone’s lying, it’s worth understanding that these two numbers are rarely measuring the same thing in the first place.

The two meters sit in different places

Your application counts bytes at the point where your code touches them: the response body your HTTP client hands back after it’s been decompressed, decoded, and parsed. The vendor’s meter sits upstream of that, usually right at the proxy server or gateway, counting whatever crossed the wire between their infrastructure and the target site, or between their infrastructure and you.

Those are two different vantage points on the same connection, and a lot happens between them that never shows up in your application-level log: TLS handshakes, failed connection attempts, redirect hops, retransmissions, and in some cases the vendor’s own per-request billing floor. None of that is fraud. It’s just that “bandwidth used” isn’t a single well-defined number until you specify where you’re measuring it and what you’re counting.

TLS handshakes aren’t free, and rotation means paying for them repeatedly

Every new TLS connection involves a ClientHello, a ServerHello, a certificate chain from the target site, and a key exchange, all before a single byte of the actual page comes back. That overhead is small per connection, but it’s not zero, and it’s billed as transferred bytes because it genuinely was transferred.

The part that catches people off guard is what rotation does to this. If you’re running a rotating residential or mobile pool and requesting a fresh exit IP on every request, you’re also usually forcing a fresh TLS connection on every request, because the old connection was tied to the old IP. No keep-alive, no session resumption. If your workflow does that thousands of times a day, the accumulated handshake overhead is a real line item, even though your app only ever sees “page 1, page 2, page 3” and never sees the handshakes at all.

A failed request still crossed the wire

This is the one people miss most often. If a request times out, gets refused by the target, or your library aborts it after a partial response, your application often doesn’t log any bytes for it, because as far as your code is concerned, nothing usable came back. But the DNS lookup happened. The TCP handshake happened. In many cases the TLS handshake happened too, and sometimes response headers or a partial body already crossed the proxy before the connection died.

The vendor’s meter, sitting closer to the wire, counted all of that. A pipeline with a high retry rate, whether from flaky mobile carrier links, aggressive target-side blocking, or an unstable modem, generates a meaningful amount of billed traffic that never produces a single row in your scraped-data table. If your retry rate is high, this is often the single biggest source of the gap.

Redirect chains inflate the real total

A lot of client libraries follow redirects transparently and only report the byte count of the final response, or they report the total but attribute it to one logical “request” in your logs. If a target site bounces you through two or three redirects before landing on the real page, each hop is a separate connection with its own headers and handshake, and the vendor’s meter counts every one of them individually. Your dashboard might show “1 request, 8KB,” when the wire actually carried three connections and several kilobytes more.

Minimum billable units turn small responses into fixed costs

Some proxy vendors, particularly on mobile and residential pools, bill with a minimum unit per request rather than metering the exact byte count. If a target returns a 400-byte JSON error and the vendor’s floor is a few kilobytes per request, you’ll be billed the floor. This exists partly because the carrier-side or gateway-side accounting the vendor is reselling often works the same way, and partly because handling a request at all has fixed cost regardless of payload size. It’s worth checking your vendor’s documentation for a stated minimum, because if one exists, a workload with a lot of small responses (status checks, availability pings, lightweight API calls) will look disproportionately expensive relative to the bytes your app actually sees.

Compression direction can go either way

This one can cut against the vendor too, so it’s worth being precise about it. If a vendor meters wire bytes, they’re counting the compressed payload as it actually traveled, gzip or brotli-encoded. Your HTTP client, on the other hand, typically reports the decompressed size after it inflates the response for you to parse. For text-heavy pages, decompressed HTML is commonly several times larger than what actually crossed the wire, which means your app-level number can legitimately be higher than the vendor’s, not lower. If your dashboard number and the invoice disagree, check which direction the gap runs before assuming either side is wrong. Compression is one of the few places where the app-level count can plausibly exceed the metered count.

Buffering proxies fetch before they let you cancel

Some proxy setups, particularly ones that do content inspection or caching at the edge, pull the entire response from the target before streaming it back to you. If your client sets a byte cap or cancels a download early because it only needs the first part of a page, the proxy may have already fetched the full body upstream before your cancel signal reached it. You get billed for what the proxy pulled, not for what your client accepted. This is architecture-dependent and not universal, but it’s worth checking with a vendor whose product does any kind of inline processing, rather than a plain pass-through tunnel.

What actually closes the gap

Chasing this down by comparing two aggregate numbers rarely resolves anything, because aggregates hide exactly the detail you need. The more useful approach is request-level correlation:

Run a small sample of traffic through a local proxy you control, like mitmproxy, sitting between your application and the vendor’s proxy, so you can see wire-level bytes on your own side of the connection, including handshakes and failed connections. Compare that against your vendor’s per-request logs if they expose them, matched by timestamp or request ID rather than by day-level totals. Separately track your retry and timeout rate, since that’s usually the largest hidden contributor and the easiest one to fix by reducing unnecessary reconnects.

If you do that and the gap holds steady in proportion to your traffic, and lines up with your handshake and retry overhead, that’s a metering difference, not a billing error. If the gap is erratic, or shows up on days with no significant traffic at all, that’s worth raising with the vendor directly, with the request-level log in hand rather than a monthly total. A vendor that can’t produce per-request logs to reconcile against is giving you a weaker guarantee than one that can, and that’s a reasonable thing to weigh when you’re evaluating who to route traffic through.

None of this is about proving a vendor is cheating you. Most of the gap between what your code sees and what you’re billed for is explainable by where the two counters sit relative to the wire. Understanding that split is what turns “the invoice looks wrong” into a specific, checkable claim.

If you’re building out scraping infrastructure and want more of this kind of ground-level detail on proxies, pipelines, and the systems you’re actually running, take a look at what else we’ve written on Data Research Tools.

Get new guides and videos first — join the Telegram channel.

Anyone who has spent a few months buying data-only SIMs, flashing modem firmware, and untangling USB hubs has asked the same question at 2 a.m.: is this actually cheaper than just paying a mobile proxy vendor per gigabyte? The honest answer is “it depends on what you’re optimizing for,” but that’s not a useful sentence on its own, so let’s break down what each path actually costs, in time and money, and where each one falls apart.

This isn’t a pitch for self hosted mobile proxies as the universally correct choice. It’s a rundown of what you’re actually signing up for on each side, based on running physical modem infrastructure and also having paid for pool access when it made more sense.

What “owning modems” actually means

Owning your own mobile proxy infrastructure means physical 4G or 5G modems (or Android phones running as tethered hotspots), each with its own SIM, plugged into USB hubs on a host machine, running some kind of rotation and health-check layer on top. The IP you hand to a scraping job is the actual carrier-assigned IP of a real device sitting in a real cell tower’s coverage area.

The stack has layers most people underestimate until they’re three months in:

• Hardware: modems or phones, USB hubs (often the actual bottleneck, since consumer hubs cap out well before you expect), a host server with enough USB controllers to avoid port starvation.
• SIMs and carrier relationships: data-only SIMs from the carrier whose IP space you want, provisioned in volume, monitored for balance and throttling.
• Rotation logic: something has to cycle the modem’s IP on a schedule or on demand, usually by toggling the interface or requesting a new bearer from the carrier’s network.
• Health monitoring: dead ports, modems stuck on a stale IP, SIMs that silently got capped or blacklisted by the carrier. All of this fails quietly if nobody’s watching it.
• Physical location and power: real estate, electricity, and a stable internet uplink for the host, plus someone within driving distance to swap hardware when it dies.

None of this is exotic. It’s closer to running a small ISP’s access layer than it is to renting a cloud VM. That’s the tradeoff in one sentence: you get full control of the IP’s provenance, but you inherit everything that comes with owning physical infrastructure.

What renting a pool actually buys

Renting from a mobile proxy provider means you’re buying access to someone else’s version of the stack above. The provider deals with modems, SIMs, carrier throttling, and rotation; you get an API endpoint or a gateway credential and you pull IPs from a shared pool.

This shifts the cost structure from capital and operational effort to a straightforward per-GB or per-IP rate. It also shifts risk. When a provider’s pool includes IPs that other customers are hammering with obviously automated traffic, that reputation is now shared across everyone on the pool, including you. You have no visibility into what those other tenants are doing, and you can’t audit it. Vendors vary a lot in how they segment traffic and how aggressively they cycle out flagged IPs, and that variance is exactly the kind of thing worth testing before committing volume, not something to take on faith from a sales page.

The other thing you’re buying is elasticity. Scaling from ten concurrent sessions to a thousand is a config change on a rented pool. On owned hardware it’s a hardware order, a SIM provisioning request, and physical install time.

The real cost comparison

The number people fixate on is per-GB price, and rented pools usually look cheaper at low volume. But that comparison only holds if you ignore what owning removes from the equation over time.

At meaningful, sustained volume, the economics shift. A modem plus SIM plan has a fixed monthly cost regardless of how much of that capacity you actually use, and if you’re running near-constant utilization, the effective per-GB cost drops well below what a rented pool charges, because you’re not paying someone else’s margin on top of the same underlying carrier data. The breakeven point depends entirely on your utilization rate, hardware costs in your region, and the SIM plans you can get, so there’s no universal number, but the shape of the curve is consistent: rented pools win at low and variable volume, owned infrastructure wins at high and steady volume, once you factor in your own time as a real cost.

That last part is where people lie to themselves. If a farm requires ten hours a week of your attention to keep alive, that’s not free. Price it at whatever your time is actually worth and redo the math before deciding owning is cheaper.

Where owning modems actually wins

The strongest case for self hosted mobile proxies isn’t price. It’s provenance and control.

When you own the modem, you know exactly what that IP has done, historically, because you’re the only one using it. You control the rotation cadence, so you’re not fighting a provider’s default rotation interval against your own session requirements. You can pin a specific IP to a specific long-running session and keep it stable for as long as you need, which matters for workflows that require session continuity rather than a fresh IP on every request. And you’re not sharing carrier-level reputation with strangers whose traffic patterns you can’t see or influence.

For teams running production scraping pipelines against targets with mature bot detection, that provenance matters more than the sticker price. A detection system that fingerprints traffic at the ASN and behavioral level doesn’t care how cheap your proxy was; it cares whether the IP and the traffic pattern behind it look like a real device doing normal things over time. An IP with a clean, consistent, single-tenant history behaves very differently, from the detector’s side, than an IP that’s been through five unrelated customers in the last hour.

Where renting wins

Renting wins on anything that isn’t steady-state. Short-term projects, spiky workloads, geographic coverage you don’t want to build out physically (good luck racking modems in every country you need residential-quality mobile IPs from), and any situation where you need scale before you need optimization.

It also wins on operational simplicity for teams that don’t want a hardware function inside an engineering org. Not every scraping team should also be in the business of managing USB hub firmware and SIM balances. That’s a legitimate specialization decision, not a compromise.

The hybrid pattern that’s more common than either extreme

In practice, most serious operations end up running a mix: owned modems for the core, steady-state traffic where provenance and cost predictability matter most, and a rented pool as overflow capacity for spikes or for geographies where building physical presence doesn’t make sense yet. The rotation and health-check layer usually needs to treat both as interchangeable pools behind the same orchestration logic, which is its own engineering problem, but it means you’re not locked into an all-or-nothing decision on day one.

What this means for bot detection, defensively

None of this is a bypass technique, and it shouldn’t be read as one. Detection systems on the other end are built to evaluate exactly the signals discussed here: IP reputation history, ASN behavior, rotation patterns, and session consistency, alongside browser and device fingerprinting. Nothing about owning a modem farm or renting a pool makes traffic undetectable, and no proxy source, owned or rented, guarantees a scraping job avoids being flagged, throttled, or blocked. The point of understanding the infrastructure is the same reason a defender studies attacker tooling: knowing how the pieces work is what lets you build systems, on either side, that behave predictably instead of guessing.

If you’re deciding between building or renting, run the numbers on your own utilization rate before trusting anyone’s per-GB pitch, including ours.

For more breakdowns like this on proxy infrastructure, scraping pipelines, and how detection systems actually work, check out the rest of the site here.

Get new guides and videos first — join the Telegram channel.

The credential surface is bigger than people expect

When people think about running scrapers, they think about parsing HTML and rotating IPs. The part that actually breaks production pipelines is more mundane: keeping track of every secret a worker needs to do its job. A single scraper node might hold proxy gateway auth, a captcha-solving API key, a storage bucket key for writing output, a database connection string, and sometimes a login session for a site that requires authentication. Multiply that by a fleet of workers running across several servers, and the number of secrets in play is easily in the hundreds, even before you count staging and dev copies.

Every one of those secrets has a different lifecycle. Proxy credentials might rotate monthly when a provider reissues sub-user passwords. API keys for a scraping API often don’t expire unless you rotate them manually. Session cookies for authenticated scraping expire in hours or days. Treating all of these as “config” and shoving them into a single file is how fleets end up with stale, leaked, or over-privileged credentials that nobody remembers issuing.

Where secrets actually leak

The failure mode isn’t usually a dramatic breach. It’s a .env file that gets copied into a Docker image and pushed to a registry, or a config file committed to a private repo that later goes public, or a proxy password pasted into a chat thread and never rotated. Credentials sit in shell history on a worker box for years because nobody thought about it after initial setup.

The other common leak path is logs. A scraper that logs its full request, including headers, will happily write proxy auth or an API key to a log file that then gets shipped to a log aggregator with looser access controls than the secrets store itself. If your logging pipeline has broader read access than your secrets manager, your logs are the weaker link, not the vault.

None of this requires an attacker. It just requires a fleet growing faster than the process for managing what it depends on.

Centralize the store, not the access pattern

The fix isn’t a single dramatic tool purchase. It’s treating secrets as a category of data with its own storage and access rules, separate from application config. That can be a dedicated secrets manager (HashiCorp Vault, AWS Secrets Manager, or similar), or for smaller fleets, an encrypted key-value store with an access log. What matters is two things: secrets are injected at runtime, not baked into images or checked into git, and every read is attributable to a specific worker or process.

Baking a credential into a container image means that credential lives forever in every layer of that image, in every registry it’s pushed to, and in every copy anyone pulls. Injecting it as an environment variable or a mounted file at container start means the credential’s lifetime is tied to the running process, and rotating it doesn’t require rebuilding and redeploying images across the fleet.

A pre-commit hook that scans for credential-shaped strings (API key patterns, Authorization: headers, connection strings) before a commit lands catches the accidental-git-commit failure mode before it becomes a public leak. It’s a small thing, but it’s often the single highest-value control for a team that hasn’t had one before.

Rotation is a process, not a button

A secrets manager gives you the ability to rotate a credential. It doesn’t rotate it for you unless you build a process around it. For a scraper fleet, rotation means issuing the new credential, updating the secrets store, having workers pick up the new value without a coordinated restart causing a gap in coverage, and confirming the old credential is actually revoked on the provider side, not just replaced in your own config.

The part that trips people up is the gap between “new credential is live” and “old credential is dead.” If workers pull secrets on start rather than on a live-reload cycle, a rotation means either restarting the fleet in waves or accepting that some workers run on the old credential until their next restart. For proxy gateway auth, that’s usually fine, the old and new credentials can coexist briefly. For something like a database password, you want that overlap window as short as you can make it.

Scope credentials to the smallest useful unit

The biggest structural mistake is one master credential shared across an entire fleet. It’s the simplest thing to set up, and it’s also why a single compromised worker, a single leaked log line, or a single misconfigured job can affect every other worker running the same job. If your proxy provider or scraping API supports sub-accounts or per-worker keys, use them. It costs more to set up and more to manage, but it means revoking one worker’s access doesn’t mean re-issuing credentials fleet-wide.

This matters more once you’re running scrapers on behalf of multiple customers or projects. A credential scoped to one customer’s job shouldn’t be reachable by a worker running another customer’s job. Mixing them in a shared secrets namespace because it’s convenient is exactly how one customer’s leaked key becomes everyone’s incident.

Auth failures are a signal, not noise

Fleets that don’t monitor credential health tend to find out a key expired when the whole pipeline goes quiet and someone has to dig through logs to figure out why. A spike in 401 or 403 responses across workers is a specific, useful signal, and it’s worth treating differently from the general noise of a scraping job hitting rate limits or bot-detection blocks. The two look similar from a distance, both show up as failed requests, but they mean different things: a 403 pattern that correlates with a specific credential and started at a specific timestamp is almost always an auth problem, not a detection problem. Alerting on that distinction separately from general error-rate monitoring saves a lot of debugging time, because “the proxy vendor rotated our sub-user password” and “the target site tightened its detection” require completely different fixes.

Proxy credentials are their own category

Proxy authentication deserves separate treatment because it usually works differently from a typical API key. Most providers offer gateway-level auth, a single username and password that routes through a shared endpoint, or per-port and sub-user structures where each worker, or each customer, gets a distinct set of credentials tied to a specific allocation of IPs or bandwidth. Gateway auth is simpler to manage but means every worker shares fate: if that one credential gets flagged or rate-limited by the provider, the whole fleet feels it. Sub-user credentials cost more setup effort but isolate that risk to the worker or job that caused it.

IP allowlisting is the other common pattern, especially for datacenter proxies, and it comes with its own management burden: every server that runs a worker needs its outbound IP registered with the provider, which means your secrets management process now includes network topology, not just credential storage. When servers get rebuilt or moved, allowlists go stale the same way passwords do, and a worker failing silently because its IP fell off an allowlist looks a lot like a proxy outage until someone checks the provider dashboard.

What this looks like in practice

None of this requires exotic tooling. It requires deciding, before the fleet grows past a handful of workers, that secrets get one storage location, get injected at runtime, get scoped per worker or per customer wherever the provider allows it, and get rotated on a schedule someone actually owns. The fleets that get burned aren’t usually the ones without a secrets manager. They’re the ones where nobody can say, on request, exactly which credential a given worker is using right now and when it was last rotated.

For more on the infrastructure side of running production scraping pipelines, proxy management, and the tooling that supports it, check out the rest of Data Research Tools.

Get new guides and videos first — join the Telegram channel.