Category: Uncategorized

Every scraper I have thrown away and rebuilt, I threw away because of the source I picked. Never once because the parsing code was bad. Parsing is a day of work. The source decides your infrastructure bill, your failure rate, and how many mornings you spend repairing something that worked fine yesterday.

Most people make that decision in about four seconds. They open the page, see data on it, and start writing selectors.

Spend ten minutes on it instead.

The order to check

Four surfaces on a typical site, ranked. Stop at the first one carrying the fields you actually need.

1. A documented api.
2. An undocumented json endpoint the page calls for itself.
3. A bulk export, a feed, or a sitemap.
4. The rendered html.

That ranking is by expected lifespan, and it ranks them by cost per record too. The surfaces with a contract behind them ship the least junk down the wire.

Five minutes on the api question

People skip this because they assume an api will be paid or gated. Check anyway. Look for a developer link in the footer, try docs. and developer. on the domain, search the company name with the word api. Plenty of businesses publish one and never surface it anywhere obvious.

When you find one, resist the urge to celebrate. Open the field reference and compare it against the record you need, field by field, before you write a line. An api missing the two fields your product depends on burns a week before you notice, usually after the ingestion layer is already built around the wrong shape.

Read the quota with the same suspicion. A limit of one request per second caps you at about 86,000 records a day, whatever hardware you throw at it.

The endpoint the page calls for itself

No public api, so look at what the page is already doing on your behalf. Open devtools, network tab, filter to fetch and xhr, reload. On anything built in the last several years you will watch the page fetch its own data as json and assemble the markup from those responses in the browser.

That json is the better surface for one plain reason: it has a schema. Somebody on the other side treats it as the interface between their front end and their back end, so it moves on their release cycle. Layout moves on their design cycle, and design ships far more often than schemas do.

Things to check while you are in there. Whether the response needs a header the page sets or a session cookie. How it paginates, normally an offset or a cursor. Whether the payload carries more fields than the page renders, which it very often does.

This technique changes none of the rules, though. Same server, same terms of service, same rate limits, same defenses in front of it. Finding the endpoint does nothing to make your traffic invisible. It lets you ask for the same data more cheaply, which the operator generally prefers, since one json response costs them a fraction of a full page render. Read the terms and respect the published limits.

Files beat pages

Before you fetch anything page by page, check whether the dataset already exists as a file. Government portals, regulators and a lot of research bodies publish flat dumps: csv, json lines, sometimes an entire database. Failing that, an xml sitemap hands you the url list without crawling for it, and an rss feed hands you new items without polling. Neither of those gives you the data on the page, but both delete the discovery half of the job, which is frequently the larger half of the bill.

The mobile template nobody checks

This is the one I think is most underrated. Try the mobile version.

Plenty of sites still keep a separate lighter template, sometimes on an m. subdomain, sometimes served on the same url to a phone user agent. It exists because somebody wanted the page usable on a bad connection, so it was built small. Small usually means server rendered, with a payload that can run a tenth the weight of the desktop page.

If desktop needs a headless browser and mobile does not, that single difference decides the size of the machine you rent.

Two caveats. Mobile templates sometimes carry fewer fields, so check parity first. And separate mobile sites are being retired in favour of responsive layouts, so the trick is fading.

The ten minute test

The question that sets your entire tooling budget is whether the page is assembled on the server or in the browser. You can answer it without writing code.

Pick a string you can see on the page, something unique. A price, a product name, a review count. Fetch the raw page with curl and search the response for that string.

If it is there, the page is server rendered and an http client is the whole answer.

If it is missing, look for a script tag holding a large json blob. Frameworks routinely embed the full page state that way, and if your string is in there, parse the blob rather than the markup.

If it is in neither place, the page is assembled client side, so go back to the network tab and find the call that fetched it. That call is your real source. The browser was only ever the thing making the call for you.

Rendering the page yourself is the last resort. It should feel like a defeat.

What that answer costs

An http client pulling json holds a few megabytes of memory per worker, so hundreds fit on one modest box. A headless browser instance wants between 300 MB and a gigabyte of ram, plus real cpu for layout and script execution. The machine running four hundred http workers runs maybe eight browsers.

Bandwidth is where it hurts if you buy proxy traffic by the gigabyte. A json request pulls a few kilobytes. Loading the same page in a browser pulls html, css, javascript bundles, fonts, images, and every tracking pixel anybody ever added. Two orders of magnitude more traffic for the same fields is normal.

Html first is a subscription you never cancel

Go straight to selectors and you have signed up for maintenance with no end date, arriving on somebody else’s release schedule.

Layout changes break you. That one is obvious. Less obvious: many modern css class names are generated at build time, so they rotate on every deploy whether or not anything visible moved. A selector written against one of those is a selector written against a build hash.

Split tests break you in the nastiest way, because they only break part of your traffic. Half your requests parse, half return empty, and the symptom looks like a proxy problem. I watched a team chase that for three days.

The tax is per target, permanently. Ten sites parsed from html is ten things failing independently on ten calendars you do not control.

Terms and limits are part of the choice

Two things to read before you build, both of which take longer to fix than to check.

The terms of service, specifically what they say about automated access and reuse of the data. Some sites permit crawling and forbid redistribution. Some forbid both. Some run a paid api precisely because they want automated users on a metered channel, and the honest answer there is to pay for it. If the terms say no, the answer is no, and technical ease has no bearing on it.

Then the published limits. robots.txt tells you which paths the site asks crawlers to stay out of, and it sometimes carries a crawl delay. Ignoring a stated preference is how a source that worked for a year suddenly stops working for everybody.

Check what kind of data it is, too. Personal data sits in a different legal category from product prices in most jurisdictions, and public visibility does not change the category.

The source I picked badly

I needed pricing across about a dozen vendors, and there was an aggregator carrying all of them. One site, one layout, one pagination scheme. One scraper instead of twelve. I took it without thinking very hard.

It worked, which was the problem.

Staleness showed up first, months in. The aggregator refreshed some vendors daily and others whenever it felt like it, and nothing on the page said which. Records three weeks old sat beside records from that morning wearing identical timestamps, because the timestamp was mine rather than theirs.

Then the normalisation. They rounded prices, mapped vendor plan names onto their own categories, and quietly dropped anything that did not fit their schema. What I had was one company’s opinion of the market with the edges trimmed off, and I had been calling it the market.

The thing that killed it was an upstream change on their side. A whole segment of my numbers shifted overnight with no change in my code and nothing in my logs. A customer facing dashboard had been wrong for a stretch I could not pin down afterwards.

The rebuild was eleven separate scrapers against the vendors themselves, four of which had public apis I never bothered to look for. Six weeks of stored data went in the bin because I could not tell which rows were real.

One convenient aggregator puts a company you have no relationship with between you and the truth, and you find out when their editorial decisions leak into your numbers.

Where this rule breaks

The honest caveat. I said json endpoints are more stable than html, and that is true on average and nowhere near a guarantee. I have had an internal endpoint change its response shape twice in a month on a site whose visible layout had not moved in a year. Internal apis have no external users and therefore no compatibility obligation, so nobody deprecates gracefully when the only client is their own front end.

That makes it a bet, and you cover a bet with monitoring. Validate the shape of every response against what you expect, alert when a field disappears or a type changes, and you hear about it within the hour rather than from a customer next month.

More on proxy infrastructure, pipeline design and the tools I actually run is at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The 2am page that taught me this

A few years back I had a scraping job that pulled product listings every six hours. The orchestrator’s health check timed out at 30 seconds. The page, on a bad network night, took 34 seconds to fully render and get scraped. The orchestrator declared the job dead and kicked off a retry while the original process was still mid-write. Two processes, same target, same time window, both inserting rows into the same table.

By morning we had roughly double the expected row count for that run. Nothing crashed, nothing errored, no alert fired. The pipeline looked healthy. The only sign something was wrong was a downstream dashboard where “unique products” jumped 40% overnight with no matching jump in the actual catalog.

That’s the failure mode idempotent scraper design exists to prevent. Not the scraper crashing, but the scraper succeeding twice.

What “idempotent” actually means for a scraper

In plumbing terms, an idempotent operation produces the same end state no matter how many times you run it. Run it once, run it five times, the database looks identical. For a scraper this means: if the job reruns because of a retry, a cron overlap, a manual backfill, or an operator rerunning a failed batch, the table doesn’t grow just because the code executed again.

This is different from the job being correct. A scraper can extract data perfectly and still not be idempotent, because idempotency is a property of the write path, not the extraction logic. The parsing can be flawless and you can still double your rows if the insert logic doesn’t know how to recognize “I’ve already written this.”

Natural keys beat auto-increment IDs

The single biggest source of duplicate rows I’ve seen in production scrapers is relying on an auto-increment primary key with no unique constraint on anything else. Every insert gets a new ID, so nothing in the database structure stops the same listing from being written twice.

The fix is to identify what actually makes a row unique in the source system and enforce that at the schema level. For a product scrape that might be (marketplace, product_id). For a job listing scrape it might be (source_url, posted_date) if the site reuses URLs for reposted listings. For anything without a stable ID from the source, you fall back to a composite of fields that together are stable: seller name, listing title, price, and category, hashed into one column.

Put a unique constraint on that key in the database, not just a check in application code. Application-level dedup checks race under concurrency exactly the way my 2am incident did. A database-level unique constraint doesn’t.

Upsert, don’t insert

Once you have a natural key, switch the write path from INSERT to an upsert: INSERT ... ON CONFLICT (natural_key) DO UPDATE in Postgres, MERGE in most other engines, or a manual read-then-write in systems that don’t support atomic upserts natively.

This does two things. It makes reruns safe (the second run just updates the existing row instead of adding a new one), and it gives you free change tracking, since the DO UPDATE clause runs on every rerun even when nothing changed. If you want to know when a price actually changed versus when the scraper simply ran again, you need one more piece.

Content hashing catches the sneaky duplicates

Natural keys solve exact duplicates. They don’t solve the case where the source system doesn’t give you a stable identifier at all, which happens more than you’d expect with scraped content: forum posts, comment threads, classified ads without persistent IDs, PDF-derived text.

For those, hash the normalized content (strip whitespace variance, lowercase, drop volatile fields like “viewed 12 times” counters) and store the hash alongside the row. On the next run, compute the hash again before writing. If it matches an existing hash for that source URL, skip the write entirely rather than upserting an identical row under a new synthetic key.

This also solves a second, quieter problem: sites that serve slightly different HTML on every request (randomized ad slots, A/B test markup, session-specific tracking pixels) which would otherwise make every scrape look like new content even when nothing meaningful changed. Normalize before you hash, or you’ll dedup nothing.

Checkpoints and watermarks, not “just rerun from scratch”

For paginated or incremental scrapes, idempotency also applies to where the job starts, not just how it writes. A scraper that always starts from page 1 and walks forward is safe to rerun by design, if slow. A scraper that resumes from “the last page we saw” needs that watermark stored somewhere durable, and it needs to be updated only after a page’s rows are confirmed written, not before.

I’ve seen the watermark get updated at the start of processing a page, as a kind of progress marker, with the actual database write happening after. If the job dies between those two steps, the watermark says “done” but the row never landed. On rerun, that page gets silently skipped rather than duplicated, which is arguably worse: it’s data loss that produces no error and no duplicate to notice. Update the watermark after the write commits, in the same transaction if your storage layer allows it.

The orchestration layer will retry you whether you like it or not

Airflow, cron with a wrapper, Kubernetes CronJobs, whatever you’re running on, the orchestration layer has its own opinions about retries that are independent of your scraper’s logic. Timeouts, missed heartbeats, and pod evictions all trigger reruns you didn’t ask for. Treating “the scraper only runs once per schedule” as an assumption is how the 2am incident happened.

The practical fix is a job lock: before a run starts, it checks for an active lock keyed to that job and time window (a row in a job_runs table with a status column works fine, you don’t need anything fancier). If a lock already exists and isn’t stale, the new invocation exits immediately instead of scraping in parallel with itself. Set a lock timeout longer than your slowest expected run, or you’ll block legitimate reruns after real failures.

Idempotency keys for the write path

For scrapers that fan out into a queue (scrape workers push extracted records onto Kafka, SQS, or similar, and a separate consumer writes to the database), the natural-key upsert has to happen at the consumer, not just be assumed from the producer side. Queues generally guarantee at-least-once delivery, not exactly-once, which means the same message can arrive at the consumer twice on its own, with no scraper rerun involved at all.

Attach an idempotency key to each message (the natural key plus a run ID works well) and have the consumer check it against a short-lived dedup store, or rely on the same database-level unique constraint from the upsert step. The queue’s delivery guarantee and your database’s write guarantee are two separate problems. Solving one doesn’t solve the other.

Test it by actually rerunning it

None of this is verified by reading the code. The only real test is running the job twice against the same source data and diffing row counts before and after. Run it once, note the count. Run it again immediately, with no changes on the source side. If the count moved, something in the write path isn’t idempotent yet, and it’s worth finding which layer failed: no unique constraint, no content hash, a race in the lock, or a watermark updated too early.

Do this in a staging environment against a snapshot of real target pages, not synthetic test fixtures. Fixtures tend to be too clean to expose the messy cases, like a source page that renders with a different ad slot ID on every load, that cause false “new content” detections in production.

A short checklist

• Unique constraint on a natural key or content hash, enforced by the database, not just application code
• Writes go through upsert, not raw insert
• Watermarks and checkpoints update only after the write commits
• A job lock keyed to the schedule window, with a timeout longer than the slowest normal run
• Consumer-side dedup if the pipeline uses a queue between scraping and storage
• A rerun test, done for real against live or snapshotted source pages, not assumed from reading the code

Idempotent scraper design is not exciting work. It doesn’t show up in a demo and nobody notices it when it’s working, which is the point. The scrapers that end up needing a 2am incident review are the ones where this got skipped because the pipeline looked fine on every individual run.

We write more on pipeline reliability, proxy infrastructure, and the operational side of scraping at scale over at Data Research Tools .

Get new guides and videos first — join the Telegram channel.

The parser that worked fine until it didn’t

A parser is a bet on the shape of a page staying still. It never does. A div gets renamed, a class gets a hash appended for cache-busting, a price field moves from a data attribute into rendered text, and the parser keeps running without a single error. It just starts returning None, or an empty list, or the wrong field mapped to the wrong key. Nobody notices until a downstream report looks wrong, and by then you’re debugging three weeks of bad data instead of one bad deploy.

That gap between “the code ran” and “the code was correct” is the whole problem with scraping at any real scale. Regression testing against last year’s HTML closes it, or at least gives you a fighting chance of catching drift before it reaches production.

Why “it worked yesterday” tells you nothing

Running a parser against a live site and getting a clean result proves the parser works against today’s markup. It says nothing about tomorrow’s. Sites change on their own schedule: A/B tests roll out to a slice of traffic, frameworks get upgraded, a marketing team redesigns a product page, a CDN starts serving slightly different HTML to different regions. None of that shows up as a deploy you can watch for. It shows up as a diff you never see, because nobody’s diffing anything.

If your only test is “run it against the live site and see if it throws,” you’ve built a smoke test, not a regression test. It confirms the parser survives today’s specific HTML. It can’t tell you whether the parser still agrees with what it extracted six months ago, or whether a field quietly started returning empty strings for a subset of pages.

Build a corpus, not a memory

The fix is mechanical: keep the actual HTML. Every time a pipeline pulls a page for real, save a copy of the raw response alongside the parsed output, on a rotation that captures meaningful time gaps rather than just the last few runs. A snapshot from a year ago is worth more than ten snapshots from last week, because the year-old one is the one likely to expose a redesign, a markup rename, or a field that got restructured somewhere in between.

Store snapshots keyed by source and date, not overwritten in place. The moment you overwrite “latest.html” with today’s pull, you’ve destroyed the one thing that makes regression testing possible: a fixed, known-good input you can replay against a new parser version. Treat old HTML like you’d treat a database migration test fixture. It’s not clutter, it’s the only ground truth you have for “did this parser change behavior.”

This is also where a lot of teams cut corners because storage feels like overhead for something that “isn’t the real pipeline.” It is the real pipeline. A scraping operation without a snapshot archive has no way to answer the question “did last week’s deploy break anything on pages we haven’t re-crawled yet,” and that question comes up constantly once you’re running more than a handful of sources.

Test the shape, not just the run

A parser regression test that only checks “no exception was thrown” will pass right through the most common failure mode: a selector that used to match now matches nothing, and the code path that handles a missing field just returns null instead of failing loud. That’s worse than a crash. A crash gets noticed. A null gets written to a database and shows up as a gap in a report three weeks later.

Write assertions against the actual extracted values, not just the absence of errors. Pin known snapshots to known-correct outputs: this specific archived page, pulled on this date, should produce this exact price, this exact title, this exact count of listed items. When you run the current parser against that same snapshot and the output no longer matches, that’s a real regression, and it’s caught at test time instead of in production.

For fields where the source value legitimately changes over time (stock counts, prices, timestamps), test the structure instead of the exact value: field is present, field is the expected type, field falls within a sane range, list has a nonzero length when the source page clearly has items. The goal isn’t cosmetic diffing against last year’s numbers. It’s confirming the parser still knows where to look.

Diffing parser versions against the same input

The second half of the setup is running old and new parser code against the same frozen snapshot and comparing outputs directly. This catches a different class of bug than the shape assertions above: it catches regressions introduced by your own changes, not just drift introduced by the site. If a refactor to the parsing logic changes how a field gets normalized, or a fix for one page type accidentally breaks extraction on another, replaying the full snapshot corpus surfaces it immediately, field by field, without waiting for a live crawl to expose it.

This is the part that actually earns the name “regression testing.” Anyone can write a script that checks a scraper doesn’t crash. Fewer people wire up a suite that reruns every historical snapshot on every parser change and flags exactly which extracted fields changed value, which snapshots newly fail a structural assertion, and which ones that used to fail now pass. That last category matters too. A fix that quietly breaks something else is a much more common failure than a fix that does nothing.

Incremental drift versus a full redesign

Not all breakage looks the same, and the test suite should tell you which kind you’re dealing with. Incremental drift shows up as a handful of snapshots failing while most still pass: a single template variant changed, or one region’s markup diverged. That’s usually a targeted fix to a selector or a fallback path.

A full redesign shows up as most or all snapshots failing at once, often on the same field or the same section of the page. That’s not a bug to patch, it’s a rewrite of that part of the parser, and it’s worth treating it that way rather than layering conditional logic onto a selector that no longer describes the page. Parsers that accumulate years of “if this class exists, else check that other class, else fall back to a third selector” become unreadable and slow, and they hide exactly the kind of failure this whole exercise is meant to catch.

Put it in CI, not in your head

None of this holds up if it depends on someone remembering to run it manually before a deploy. The snapshot replay and diff belong in the same pipeline as any other test suite: run on every change to parsing logic, block the merge if a structural assertion fails on a snapshot that used to pass, and surface the diff output somewhere a human actually reads it. A regression suite that exists but isn’t wired into the deploy gate is a suite that gets skipped the first time someone’s in a hurry, which is exactly when it’s most needed.

Production monitoring is the backstop, not the plan

Even a solid snapshot corpus won’t catch a live site change the instant it happens, because your corpus only reflects markup you’ve already seen. That’s what live field-level monitoring is for: tracking null rates, empty-list rates, and out-of-range values on production output over time, and alerting when they move. Regression testing catches known drift and prevents you from shipping a parser that already fails against history. Production monitoring catches new drift the moment it starts happening on sites you’re actively crawling. You need both. Treating live monitoring as the whole strategy means you only find out about breakage after it’s already cost you data; treating snapshot testing as the whole strategy means you’re blind to changes that haven’t happened yet.

What this actually buys you

The payoff isn’t a parser that never breaks. Sites will keep changing markup, and any parser aimed at real-world pages will eventually hit one it doesn’t handle. What a snapshot-based regression suite buys you is knowing about it fast, knowing which field or which page type is affected, and knowing whether the cause was the site or your own last commit. That’s the difference between a five-minute fix caught in CI and a data quality incident someone finds a month later.

If you’re building or maintaining scraping infrastructure and want more on parser design, pipeline testing, and the operational side of running scrapers at scale, there’s more on the site.

Data Research Tools

Get new guides and videos first — join the Telegram channel.

Why search engines are a different animal

Most scraping targets are e-commerce catalogs, review sites, or job boards. Search engines are a different category of problem. You’re not pulling structured data from a predictable template. You’re querying a system whose entire business model depends on knowing whether the thing on the other end of the request is a person clicking a mouse or a script issuing HTTP requests in a loop.

Search engines see more automated traffic than almost any other class of website, because rank tracking, SEO tools, ad verification, and market research all depend on the same data: what shows up on a results page for a given query. That means the defenses are mature, layered, and tuned specifically against scraping, not bolted on as an afterthought.

When we run pipelines that touch search result pages for legitimate purposes like rank tracking or SERP feature monitoring, the failure modes are consistent enough that they’re worth documenting. This is a look at what breaks and why, not a guide to getting around it.

The request looks fine and still gets blocked

A request to a search engine carries far more signal than the URL and headers suggest. The server sees TLS handshake details (cipher order, extensions, JA3-style fingerprints), HTTP/2 frame ordering, header casing and order, and whether the client actually renders JavaScript versus just fetching HTML. A plain requests or curl call in Python matches none of a real browser’s fingerprint, even with a convincing User-Agent string spoofed on top.

This is why headless browser tooling exists for this kind of work at all. Playwright or Puppeteer against a real Chromium build gets you a browser-shaped TLS and JS fingerprint for free. But default headless configurations still expose tells: missing navigator.webdriver masking, unusual Accept-Language ordering, viewport and font lists that don’t match a stock consumer machine, WebGL renderer strings that scream “cloud VM.” Search engines correlate dozens of these signals rather than checking any single one, so patching one tell rarely fixes a block on its own.

Rate and pattern, not just volume

A common assumption is that staying under some request-per-minute threshold is enough. In practice, search engines watch pattern shape more than raw rate. A single IP issuing queries at machine-perfect intervals, always for the same query structure, always from the same fingerprint, is a stronger signal than raw volume. Real human search behavior is bursty and irregular: people pause to read results, follow links, come back, refine queries with typos and corrections.

This is also where proxy infrastructure choices matter for reliability, not evasion. A pool of datacenter IPs that all belong to the same /24 block from a hosting provider is trivially identifiable as non-residential traffic, because search engines maintain ASN-level reputation data going back years. Residential and mobile IP pools carry different reputation because they’re shared with real consumer traffic, but they come with their own operational costs: shared IPs mean shared history, so a proxy that was flagged by another user’s traffic before it reached you carries that baggage in.

None of this makes any pool “clean” or “safe.” It changes the odds and the failure mode, not whether failure is possible.

CAPTCHAs are a checkpoint, not a wall

When a search engine’s risk scoring crosses a threshold for a given session, the typical response isn’t an outright block. It’s an interactive challenge: a CAPTCHA, a “verify you’re human” interstitial, or a soft rate-limit page. This is a deliberate design choice. It costs the search engine almost nothing to serve, it filters out unsophisticated bots immediately, and it gives the system a clean signal about what kind of traffic is hitting a given IP or session.

From an engineering standpoint, this is where a lot of naive scraping pipelines quietly die. The script gets a 200 response, parses the page, and finds a CAPTCHA form instead of search results. If the pipeline doesn’t specifically check for challenge markup, it happily stores empty or garbage data and nobody notices until a report downstream looks wrong. Any pipeline touching search results needs an explicit check for challenge pages as a distinct failure state, not just HTTP status codes.

Third-party CAPTCHA-solving services exist and are marketed heavily to the scraping community. We don’t cover how to use them here, because that shifts a technical problem into a terms-of-service and legal one, and because their reliability numbers are usually vendor-reported rather than independently verified.

Structural changes break parsers constantly

Even setting detection aside, search result pages are some of the most actively A/B-tested UI on the internet. Layout experiments roll out to small traffic percentages, meaning your scraper can pass for weeks and then start failing for a subset of runs because a fraction of sessions are seeing a variant markup structure. Featured snippets, knowledge panels, and ad blocks shift the DOM position of organic results depending on query type, so a parser that assumes “organic results are the fourth div down” breaks the moment a query triggers a shopping carousel it didn’t expect.

This is a maintenance cost that’s easy to underestimate when scoping a scraping project. Selector-based parsing needs to be paired with schema validation on the output, not just try/except blocks around the fetch. If a run returns a result count of zero or a snippet field of unexpected length, that should fail loudly rather than write a null into a database that a report reads from next week.

Geography and personalization complicate everything

Search results are localized and personalized by design, which means “scrape a search engine” is never actually one task. Results vary by the apparent location of the request (inferred from IP geolocation), by language settings, by device type reported in the user agent, and in logged-in contexts by account history. A rank tracking pipeline that doesn’t pin geography to a proxy exit node consistent with the market being measured is producing numbers that don’t mean what the report claims they mean.

This is one of the more overlooked failure modes because it doesn’t look like a failure. The scrape succeeds, the page parses cleanly, the data looks plausible. It’s just wrong for the question being asked, because the exit IP was in a different country than the market the client cares about. Getting this right requires proxy infrastructure with genuine geographic diversity and metadata you can trust, not just a vendor’s claimed country list.

What a defensible pipeline actually looks for

Put together, a search-engine-facing pipeline that holds up over time tends to share a few properties: it treats challenge pages as a distinct, logged failure state rather than parsing them as empty results; it validates output structure against an expected schema on every run rather than trusting that yesterday’s selectors still apply; it separates concerns between fingerprint realism (browser engine, TLS stack) and traffic pattern (timing, query variety, session lifetime) instead of assuming one fixes the other; and it treats every proxy pool’s reputation as a decaying, unverified asset rather than a fixed property to configure once.

None of that guarantees uptime. Detection systems adapt, and any concrete technique described in a blog post has a shelf life measured in months at best. The pipelines that survive longest are the ones instrumented to notice when they’ve started failing, rather than the ones that assume the initial setup keeps working.

If you’re building or maintaining scraping infrastructure and want more of this kind of operator-level detail on proxy pools, fingerprinting, and pipeline design, check out the rest of what we cover here.

Get new guides and videos first — join the Telegram channel.

What schema drift actually is

Schema drift is when the shape of the data you’re pulling changes without the site announcing it. A class name gets renamed during a frontend refactor. A price field moves from a data-price attribute to a nested JSON blob in a script tag. An API response adds a wrapper object around what used to be a flat array. Nothing about the site “breaks” from a user’s perspective. Your parser breaks anyway, because it was written against a specific structure, not against the site’s intent.

This is different from a full redesign, which you’d notice immediately because half your fields go empty at once. Drift is quieter. It’s a single field silently returning null, or a price parser picking up a strikethrough “was” price instead of the current one, or a pagination selector that still technically matches something, just not the thing you think it matches. The pipeline keeps running. The job status stays green. The data going into your warehouse is just wrong, or partially wrong, and nobody notices until a downstream report looks off three weeks later.

Why it’s worse than an outage

An outage is annoying but self-limiting. Requests fail, retries exhaust, alerts fire, someone looks at it that day. Drift doesn’t fail loudly. It degrades. A scraper hitting a product listing page can keep returning HTTP 200 with a fully rendered page and still be extracting the wrong number, because the selector matched a different element after a markup change. If your only health check is “did the request succeed,” drift sails right past it.

The cost compounds with pipeline size. A two-site scraper can be checked by hand. A pipeline pulling from 400 target sites cannot be eyeballed, and that’s exactly the scale at which drift causes the most damage, because a single bad deploy on one target site can quietly poison a slice of your dataset for days before anyone cross-references it against a source of truth.

The three places drift actually hides

Drift shows up in three layers, and each needs a different check.

DOM structure. Class names, IDs, and element nesting change during frontend framework upgrades or A/B tests. This is the most common form and the easiest to detect, because you can diff the structural tree of a page over time independent of the specific values inside it.

API and JSON contracts. Sites backed by internal APIs (even ones your scraper reverse-engineers rather than calls directly, like the JSON payload embedded in a <script> tag) change field names, nesting, or types when the backend team ships a new version. This kind of drift is sneakier because it’s driven by backend releases that have nothing to do with the visible page and often aren’t announced anywhere public.

Semantic drift. The structure stays the same but the meaning of a field shifts. A “price” field starts including tax in some regions and not others. A “status” enum gains a new value your parser doesn’t map, so it falls through to a default. This is the hardest to catch with structural monitoring alone, because nothing about the shape of the data changed, only its content.

Structural fingerprints beat brittle selectors

The core defensive pattern is to stop treating your scraper’s output as the only signal you monitor, and start monitoring the structure it’s parsed from as a separate, versioned artifact.

Concretely: on every scrape, compute a lightweight fingerprint of the page or response before you parse it. For HTML, that can be a normalized tree of tag names and key attributes, stripped of dynamic content like timestamps or session tokens, then hashed. For JSON, it’s a schema shape: field names, types, and nesting, independent of values. Store that fingerprint alongside the scrape. When the fingerprint for a target changes, you get a signal before you know whether the change actually broke anything.

This matters because it decouples “the site changed” from “my parser broke.” Those are different events with different urgency. A fingerprint change on a low-priority field can wait for the next sprint. A fingerprint change on a field feeding a client-facing dashboard needs someone looking at it same-day.

Field-level validation as a second layer

Structural fingerprints catch shape changes. They don’t catch a selector that still matches something, just the wrong something. For that you need field-level sanity checks that run against every extracted value, not just the structure around it.

Practical checks that catch real drift in production:

• Type and range bounds. A price field that returns a negative number, a string, or a value ten times the historical median for that product category is worth flagging, even if extraction “succeeded.”
• Null rate tracking per field, per target. If a field that’s populated 98% of the time across the last thirty days suddenly drops to 40%, that’s drift, not noise. Track this as a time series, not a single threshold check, because normal null rates vary by category and you need a baseline per field to know what “normal” looks like for that specific target.
• Cross-field consistency. If a “discounted price” is higher than a “list price,” something in the parsing logic pointed at the wrong element. These relationships rarely change even when markup does, which makes them a stable check to run against.
• Cardinality checks on enums. A status or category field that starts returning a value never seen before in the historical distribution is either a genuine new category on the site or a parser reading the wrong attribute. Either way it needs a human to classify it once.

None of this requires exotic tooling. It’s the same data quality discipline you’d apply to any ETL pipeline, just applied specifically to fields sourced from HTML or JSON you don’t control the schema of.

Sampling frequency and alert fatigue

A detection system that fires on every minor DOM tweak trains people to ignore it, and an ignored alert is worse than no alert because it creates false confidence. The fix is tiering targets by how much a break actually costs you, and matching monitoring frequency to that tier.

High-value targets, the ones feeding client deliverables or revenue-adjacent reports, get fingerprint checks on every run and same-day review of any change. Mid-tier targets get a daily rollup of fingerprint diffs and null-rate anomalies rather than a check per scrape. Low-priority targets can be batched into a weekly review. This isn’t about ignoring drift on lower-tier sites, it’s about matching response time to actual cost, because a monitoring system that pages someone for every target at every scrape gets its alerts muted within a week.

What to do when drift fires

When a fingerprint or field-level check flags a target, the first move is always to look at the actual page or response by hand, not to patch the selector blind. A structural change can mean the class name moved, or it can mean the site added a completely new layout variant that only some traffic sees, in which case a single selector fix might work for 80% of requests and silently fail on the other 20%.

Version your parsers per target and keep the old version running in parallel against a small sample while the new one rolls out. This catches cases where a “fix” solves the sample page you looked at but misses a variant. It also gives you a rollback path if the new parser turns out to be wrong in a different way than the old one was.

Log the change itself, not just the fix. A running record of what changed on each target, when, and how you adapted is the single most useful artifact for the next time that site redesigns something, because frontend teams tend to iterate on the same components repeatedly rather than rebuilding from scratch each time.

The maintenance budget you’re actually paying

Every production scraping operation pays a recurring tax in parser maintenance, whether or not anyone puts a number on it. Sites are not static targets. They ship frontend changes on their own release cadence, unrelated to your pipeline, and treating your scraper as a one-time build rather than a maintained piece of software is how teams end up with weeks-old silent data corruption instead of a same-day fix. Budgeting real engineering time for schema monitoring, not just for writing new scrapers, is the difference between catching drift in hours and finding it in a client’s angry email.

If you’re building or maintaining scraping infrastructure and want more on pipeline monitoring, proxy management, and the defensive side of bot detection, there’s more on the home page.

Get new guides and videos first — join the Telegram channel.

Nobody reads default_settings.py before their first crawl. You run scrapy startproject, write a spider, hit go, and it works. That’s the whole point of a framework: it gets you moving before you understand what you just turned on. The problem shows up later, usually in production, usually as a support ticket that says “the crawler stalled” or “we’re getting banned everywhere” with no obvious cause.

Every one of those tickets traces back to a setting that was never touched. Scrapy’s defaults aren’t wrong, they’re tuned for getting a first crawl running without configuration. Once you’re running a real pipeline against real sites, every one of those defaults has a cost attached, and most of them are invisible until something breaks.

The user agent that announces itself

The stock USER_AGENT value is Scrapy/VERSION (+https://scrapy.org). It’s built from the installed package version at import time, which means it’s not just identifying you as an automated client, it’s telling the target exactly which library and roughly which release you’re running, with a link to the project homepage attached.

This isn’t a stealth failure, it’s not meant to be stealthy at all. It’s a courtesy string so site operators know what’s hitting their logs and can find the framework’s docs if they want to set a crawl policy. But if you ship it to production unmodified, you’ve told every log-reading system on the other end what you are before it even has to look at your request pattern. The cost here isn’t dramatic, it’s just wasted effort: any WAF rule, rate limiter, or analyst doing a five-minute log review gets an easy signal for free. Setting a real, project-specific UA string is one of the first things any production config changes, and it should be paired with the rest of your request fingerprint, not treated as the whole fix.

Concurrency with no pacing

CONCURRENT_REQUESTS defaults to 16 globally, CONCURRENT_REQUESTS_PER_DOMAIN defaults to 8, and DOWNLOAD_DELAY defaults to 0. AUTOTHROTTLE_ENABLED is off. Put together, that’s a spider that will happily open eight simultaneous connections to a single domain with zero delay between requests, the moment it has eight URLs queued for that domain.

For a tutorial site or your own test server, that’s fine. Against a real production target, that request pattern is a burst, and bursts are exactly the shape that rate limiters and basic bot-detection heuristics are built to catch. It doesn’t matter how good your proxy pool is if the timing of the requests looks like a script instead of a browser session. A person clicking through pages doesn’t fire eight requests in the same tenth of a second.

The cost compounds because of what happens next: RETRY_TIMES defaults to 2, and Scrapy retries on a defined set of HTTP status codes including 429 and 503, which are the exact codes a rate limiter returns. So the default behavior when you get throttled is to immediately retry into the same throttle, twice, still with no added delay unless you’ve turned on AUTOTHROTTLE_ENABLED or set your own DOWNLOAD_DELAY. That’s compute and IP reputation spent making the problem worse, not better. In a farm running against multiple targets, an unthrottled spider hitting one flaky domain can burn through retry budget and proxy trust for the whole run.

The robots.txt setting that depends on how you started

This is the one that catches experienced people, not beginners. The actual library default for ROBOTSTXT_OBEY in scrapy/settings/default_settings.py is False. But the settings.py file that scrapy startproject generates for you sets it to True. So depending on whether your spider was born from the CLI template or built by importing Scrapy directly into an existing codebase, orchestration script, or Docker image, you can end up with opposite behavior for the exact same “default” setting.

This matters operationally, not just legally. Robots.txt is a policy signal, and plenty of teams have a compliance requirement to respect it regardless of what’s technically enforceable. If your production spiders were bootstrapped outside the standard template, either by copying an old settings file, building a custom base spider class, or running Scrapy as a library inside another framework, it’s worth explicitly checking which value you actually have rather than assuming the CLI default followed you. This is a five-minute check that avoids a genuinely awkward conversation with legal or with a partner site later.

Cookies you didn’t ask for

COOKIES_ENABLED defaults to True. Scrapy will accept and replay cookies per spider by default via its cookie middleware, which is usually what you want for anything involving login state or session-based pagination. But it also means your crawl is building session continuity you may not have intended, and that continuity is itself a fingerprinting surface. A session that persists across thousands of requests to the same domain, at machine-precision timing, with a consistent (or absent) cookie jar, is a much stronger and more stable identifier than IP alone. If you don’t need session state for a given spider, turning cookies off removes one more axis a detection system can use to stitch your requests together across proxy rotations.

The debug console left listening

TELNETCONSOLE_ENABLED is True by default, giving you a live telnet shell into the running process for inspecting the engine, stats, and queues, bound to 127.0.0.1 by default. That’s a genuinely useful debugging tool. The cost shows up in containerized deployments, where “localhost” doesn’t mean what it means on a bare-metal box. Depending on your network mode, a container’s loopback interface can end up reachable from other places you didn’t intend, and a debug console into a running scraping process is not something you want discoverable. It’s worth confirming explicitly whether it’s needed in your production images, rather than inheriting it because nobody revisited the setting after the container config changed hands.

DNS caching in a rotating-proxy world

DNSCACHE_ENABLED defaults to True with a cache size of 10,000 entries. That’s a sensible default for a single long crawl against a fixed target: don’t re-resolve names you’ve already resolved. But in an infrastructure where proxies rotate and requests may resolve DNS differently depending on which upstream or resolver a proxy is using, a long-lived, unbounded-feeling cache inside the Scrapy process can quietly serve stale resolution data across a run that spans hours. It rarely causes an outright failure, it shows up as odd intermittent behavior that’s hard to reproduce, because the caching layer is invisible unless you go looking for it.

What actually changes for production

None of this means Scrapy’s defaults are badly chosen. They’re chosen correctly for what they’re for: get a new user to a working crawl in ten minutes without reading documentation. The cost only appears when a tutorial config gets promoted to a production pipeline without anyone doing a settings audit in between.

The fix isn’t a single flag, it’s a checklist: a real, maintained User-Agent tied to your actual project, deliberate concurrency and delay values matched to the target rather than the framework ceiling, AUTOTHROTTLE_ENABLED turned on so retry storms don’t compound rate-limit hits, an explicit and documented decision on ROBOTSTXT_OBEY rather than an inherited one, cookies scoped to spiders that actually need session state, and the telnet console reviewed against your actual container network model. Every one of these is a two-line settings change. The cost of skipping them is measured in wasted proxy budget, incomplete crawls, and debugging sessions that start with “it worked in dev.”

If you’re running scraping infrastructure at any real scale, treating the settings file as part of the codebase to review, not a one-time bootstrap step, is the difference between a pipeline that degrades gracefully and one that quietly bleeds retries and reputation until someone notices the data stopped coming in.

Want more of this kind of infrastructure-level breakdown? Head back to the Data Research Tools home page for our pipeline orchestration, proxy infrastructure, and bot-detection explainers.

Get new guides and videos first — join the Telegram channel.

Every scraping tutorial has the same line in it somewhere: rotate your user agent string and you’ll look like a different visitor on every request. It’s cheap advice to give because it’s cheap to implement. Pick a list of a few hundred browser strings, cycle through them, ship it. The problem is that running actual proxy infrastructure and actual scraping pipelines for years teaches you something the tutorials skip: a rotated user agent, on its own, usually makes a request easier to flag, not harder.

This isn’t a claim that user agent rotation is pointless in every configuration. It’s a claim that rotating the string in isolation, which is what almost every scraper does, creates an inconsistency that is more visible to a detection system than just sending one honest, stable user agent would have been.

What a user agent string actually does

The user agent header is just a self-reported label. The client says “I am Chrome 124 on Windows 11” and the server has no way to independently confirm that from the header alone. Detection systems learned this a long time ago, which is why no serious anti-bot system relies on the user agent string as a primary signal anymore. It’s treated as a claim, and the rest of the request is checked against that claim.

That checking is the part rotation ignores. A real Chrome 124 request doesn’t arrive alone. It arrives with a specific TLS handshake fingerprint (which cipher suites, extensions, and curves the client offers, and in what order), a specific set of HTTP/2 frame settings, a specific header order and capitalization pattern, and, on top of all that, a set of sec-ch-ua client hint headers that are supposed to describe the exact same browser and version the user agent string claims. All of those are generated by the actual browser engine. None of them change when a script swaps a string in a Python dict.

Rotating one field while the rest stays fixed

Here’s the mismatch that gives rotation away in practice. A scraping script built on requests or a bare HTTP client has one TLS fingerprint: the one that library produces, which looks nothing like any real browser’s. If that script cycles through fifty different user agent strings claiming to be Chrome, Firefox, and Safari on different operating systems, every one of those fifty requests still shares the same underlying TLS handshake, the same header ordering, and the same absence of client hint headers a real browser would send.

To a detection system watching at the connection layer, this looks like one client wearing fifty different name tags. The user agent is changing every request while everything below it stays frozen. That’s a stronger, more consistent signal than a scraper that just picked one user agent and stuck with it, because a static-but-honest fingerprint at least looks like it belongs to one real, if boring, visitor. A fingerprint where one field jumps around while ten others stay locked looks like automation doing exactly what automation tutorials tell it to do.

Volume makes the pattern worse

The mismatch above is bad enough for a single request. It compounds at scale, which is where most scraping actually lives. A pipeline making thousands of requests an hour from a rotating pool of user agents, but from a small number of TLS fingerprints, is handing a detection system a trivial clustering problem. Group requests by TLS fingerprint or by header order instead of by user agent, and the “different visitors” collapse back into the same small handful of clients almost instantly.

This is worth being explicit about because it’s counter to the pitch: rotation doesn’t add entropy to the traffic. It adds entropy to exactly one field while every other field, the ones that are harder to fake and therefore weighted more heavily, stays put. Real diversity in a population of browsers comes from different people running different browser versions, on different operating systems, with different installed extensions and screen sizes, each of those things correlated with each other in the way real devices actually are. A rotating list of strings applied on top of one script’s fixed TLS stack doesn’t recreate that. It highlights the absence of it.

Why this matters more now than it used to

User agent strings themselves are also becoming less useful as a spoofing target for an unrelated reason: Chrome and other Chromium-based browsers have been freezing and simplifying the user agent string for years as part of the User-Agent Reduction effort, moving the detailed version and platform information into the sec-ch-ua client hints instead. That means a modern real browser sends a comparatively generic user agent string alongside a set of structured client hint headers that carry the specific claims. A scraper that rotates the old-style detailed user agent string without also generating consistent, matching client hint headers is producing a combination that a current browser wouldn’t produce in the first place. The mismatch isn’t subtle to a system checking for it.

What actually reduces detection risk

None of this means fingerprinting is unbeatable or that any particular counter-technique guarantees a clean pass. It isn’t and it doesn’t, and any claim that a tool makes a scraper undetectable should be treated as marketing until you’ve watched it hold up under an actual test. What the infrastructure side of this does support is a narrower, more honest point: consistency across the whole request, not variety in one header, is what a fingerprint check is actually built around.

In practice that means treating a browser identity as a bundle, not a single swappable field. If a pipeline claims to be a specific browser and OS, the TLS handshake, the header order, the client hints, and the user agent string all need to agree with each other, and that bundle should stay stable for the life of a session rather than reshuffling on every request. Running an actual browser engine (something like a real Chromium instance rather than a bare HTTP client) gets you a fingerprint that’s internally consistent by construction, because it’s the browser generating all of those layers itself, not a script trying to fake them independently. That’s a heavier approach than swapping a string in a list, and it costs more in compute and complexity, which is exactly why the cheaper trick got popularized in the first place.

The proxy side matters here too, and it’s worth being precise about what it does and doesn’t fix. A residential or mobile IP changes where the request appears to come from. It says nothing about what the request looks like once it arrives, and a detection system that’s checking TLS and header consistency doesn’t care what IP range the mismatched fingerprint showed up on. Good proxy infrastructure and a consistent client fingerprint solve different problems, and treating either one as a substitute for the other is how a scraping setup ends up looking automated for reasons that have nothing to do with the IP at all.

The takeaway

User agent rotation survives as advice because it’s easy to explain and easy to ship, not because it holds up against how detection systems actually evaluate a request. On its own, it swaps the one field that was never treated as trustworthy in the first place, while leaving the fields that are harder to fake completely static. If anything, that combination is a more legible pattern than doing nothing at all. The parts of a scraping setup worth investing in are the ones that keep a whole request internally consistent, not the ones that randomize a single header and call it done.

If you’re building or auditing scraping infrastructure and want a clearer picture of how proxy pools, fingerprinting, and detection systems actually fit together, we write about this from the operator’s side, not the marketing side, at Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The problem nobody plans for

Most scraping projects start the same way. You write a crawler, it hits a few hundred pages, you save each response as a file, and everything works. Then the run grows to a million pages, or ten million, and the storage layer you never thought about becomes the thing that breaks first, not the scraper itself.

Picking an html storage format is not a cosmetic decision. It determines how fast you can reprocess data when your parser changes, how much disk and I/O you burn, whether backups finish overnight or not at all, and whether your pipeline can survive a crash halfway through a crawl. This post walks through the real options we use in production pipelines and where each one falls apart.

Why one file per page breaks down

Saving each scraped page as its own file on disk is the obvious starting point, and it’s fine at small scale. The trouble shows up on three fronts once you cross into the millions.

First, filesystems have limits on the number of files a single directory can hold efficiently. Ext4 and NTFS will technically let you keep piling files into one folder, but directory listing, file creation, and even simple ls calls slow down noticeably as the count climbs. You end up building a bucketing scheme (hashing the URL into subfolders) just to keep the filesystem usable, which is extra engineering for something that should be an implementation detail.

Second, most filesystems allocate storage in fixed block sizes, commonly 4KB. An HTML page that’s 2KB on the wire still consumes a full block on disk. Multiply that waste by ten million small files and you’re paying for storage you never actually used.

Third, and this is the one that bites people hardest, backing up or moving ten million small files is slow in a way that has nothing to do with total data size. Copying one 50GB file is fast. Copying ten million files that add up to 50GB can take hours, because the overhead is per-file, not per-byte. If you’ve ever run rsync on a directory full of scraped pages and watched the file counter barely move, you know this pain.

The WARC format exists for exactly this reason

Web archiving projects hit this same wall decades ago, which is why the WARC format (Web ARChive) exists. It’s the format Common Crawl and the Internet Archive use to store crawled pages, and it solves the small-files problem directly: instead of one file per page, you write a single large file containing many page records back to back, each with its own header (URL, timestamp, HTTP response headers, content length) followed by the raw response body.

Readers use byte offsets stored in an index to jump straight to a given record without reading the whole file, so you get random access without the filesystem overhead of millions of individual files. Because WARC preserves the full HTTP response including headers, it’s also a genuinely honest archival format. You’re not just saving the HTML, you’re saving what the server actually sent, which matters if you ever need to debug why a parser misbehaved on a specific page (redirect chains, unexpected content types, gzip encoding issues, all of that lives in the headers).

The tradeoff is tooling. WARC readers and writers are a layer you have to add to your pipeline, and if your team is used to just opening files, it’s a mental shift. For archival-grade or very large crawls, it’s worth it. For a scraping job that tops out at a few hundred thousand pages, it may be more format than you need.

Object storage as the middle ground

For most production scraping pipelines we run, the practical answer sits between “one file per page on local disk” and “roll your own WARC pipeline.” S3-compatible object storage (AWS S3, or self-hosted alternatives like MinIO) handles the small-files problem differently: it’s built to store billions of objects, and listing, versioning, and lifecycle rules are handled by the storage layer instead of your filesystem.

The key design decision is still the same one you’d face locally: one object per page, or bundled objects. One object per page (keyed by a hash of the URL, not the URL itself, since URLs contain characters that don’t play nicely as object keys) is simple and gives you direct access to any single page. Bundling groups of pages into compressed archives (effectively mini-WARC files) written to object storage cuts down on the number of objects and the per-request overhead, at the cost of needing an index to find a given page inside a bundle.

Which one makes sense depends on your access pattern. If you mostly reprocess data in bulk (rerun a parser over the whole dataset), bundling wins because you’re streaming through data anyway. If you need to randomly fetch individual pages by URL on demand, one object per page with a hash-based key is simpler to reason about, even if it costs more in object count.

Compression is not optional at this scale

HTML compresses well because it’s repetitive text: tag names, whitespace, boilerplate navigation and footer markup that repeats across every page on a site. Storing raw, uncompressed HTML at scale is close to the easiest optimization you can skip.

Gzip is the safe default. It’s supported everywhere, decompresses fast, and every language has a mature library for it. Zstandard (zstd) is the newer option worth knowing about: it generally compresses faster than gzip at a comparable ratio, and decompression is noticeably quicker, which matters if your reprocessing jobs decompress the same archive repeatedly. Neither is universally “better,” the right pick depends on whether you’re optimizing for write throughput, read throughput, or compatibility with existing tooling in your stack. Don’t take a specific ratio or speed number as gospel from a blog post, including this one. Test it against your own HTML, since the actual compression ratio depends heavily on how repetitive the pages in your dataset are.

One thing worth doing regardless of compression algorithm: dedupe before you store. If your crawler revisits the same page, or if a site serves near-identical content across thousands of URLs (pagination boilerplate, templated product pages), hashing the content and storing a pointer to an existing blob instead of a fresh copy can save more space than compression algorithm choice ever will.

Should you even keep the raw HTML

This is the question that gets skipped most often. Teams default to keeping every raw page forever because storage feels cheap, but raw HTML has a real cost: it’s bulky, most of it is boilerplate you’ll never look at again, and keeping it means keeping decompression and parsing logic working indefinitely.

The honest tradeoff is this. Keep raw HTML if you expect your extraction logic to change, if the source site’s structure is your business (competitive monitoring, price tracking where you need to re-verify what was actually shown), or if you need to prove what a page contained at a point in time. Don’t keep it if you only ever need the parsed fields and you’re confident your parser is stable. In that case, extract once, store the structured output (JSON, a database row), and discard the HTML, or keep it for a short retention window instead of forever.

A common middle path in production pipelines: store raw HTML in a cheap, cold-tier object storage bucket with a lifecycle rule that deletes it after 30 or 90 days, and store parsed output in a proper database indefinitely. That gives you a recovery window if your parser turns out to be broken without paying to store raw pages forever.

A rough decision guide

If you’re under a few hundred thousand pages and mostly experimenting, files on disk with gzip compression is fine, don’t over-engineer it. Once you’re consistently crawling into the millions, move to object storage with hash-based keys or bundled archives, and pick your compression algorithm based on whether you read or write more often. If you’re building something closer to a general-purpose archive that other tools or teams will consume, WARC is worth the tooling investment because it’s a standard format with existing readers, not something you invented that only your pipeline understands. And separately from format, decide deliberately how long you actually need the raw HTML, because that decision affects your storage bill far more than the format choice does.

None of this is exciting engineering, but it’s the layer that determines whether a scraping pipeline survives contact with real scale or quietly grinds to a halt six months in.

For more breakdowns of scraping infrastructure, proxy setups, and pipeline design, check out the rest of Data Research Tools here.

Get new guides and videos first — join the Telegram channel.

Why the queue matters more at 3am than at 3pm

A scraper you babysit for twenty minutes can get away with almost any queue, including no queue at all. A scraper running from midnight to 8am can’t. Nobody is watching it. If a worker crashes, if a proxy pool goes stale, if one target site starts throwing 403s at 2am, the queue is the only thing deciding whether the run recovers on its own or just stops producing data until someone notices in the morning.

That’s the real design question behind “scraper job queue”: not which library has the nicest API, but what happens to a job when something goes wrong while no human is looking.

Why an in-memory list doesn’t survive the night

A lot of scrapers start life as a Python list of URLs and a for loop. That’s fine for a few hundred pages run interactively. It falls apart on an overnight job for a specific reason: the list lives inside one process’s memory. If that process dies, whether from an unhandled exception, an out-of-memory kill, or the host rebooting for a routine patch, every job that hasn’t finished is gone. There’s no record it ever existed.

You can wrap the loop in more try/except blocks, but that doesn’t fix the underlying problem. The job list needs to live somewhere that survives the worker process crashing. That’s the first real requirement for an overnight queue: persistence outside the worker.

Persistence: where the job list actually lives

Once persistence is the requirement, the practical options split into a few tiers.

Redis is the common middle ground. Lists, sorted sets, or Redis Streams can all act as a queue backend, and tools like RQ, Celery (with Redis as broker), or BullMQ build retry logic, scheduling, and worker pools on top of it. Redis is fast and simple to run, but by default it’s in-memory with periodic snapshotting, so you need to actually configure AOF persistence or accept that a Redis restart at the wrong moment can lose recent jobs.

A message broker like RabbitMQ gives you durable queues, acknowledgments, and dead letter exchanges out of the box, at the cost of more moving parts to operate. It’s a reasonable choice once you have multiple worker types (fetchers, parsers, retriers) that need distinct queues and routing rules.

A managed queue like SQS removes the operational burden entirely. You don’t run anything, you get durability and visibility timeouts by default, and it scales without you thinking about it. The tradeoff is cost at high message volume and a bit less flexibility in retry logic than you get building it yourself.

A plain Postgres table is underrated for smaller overnight jobs. A jobs table with a status column, SELECT ... FOR UPDATE SKIP LOCKED to let multiple workers pull rows without colliding, and a cron-style sweep for stuck rows gets you most of what a dedicated queue gives you, without adding a new piece of infrastructure to your stack. If you already run Postgres for the scraped data, this is often the path of least resistance.

None of these is universally correct. The right one depends on how many workers you’re running, how much operational overhead you’re willing to own, and whether you already have the infrastructure in place.

Retries and the lease pattern

The reason a queue beats a list isn’t just “it survives a crash.” It’s that a real queue has an answer to: what happens to a job that a worker picked up but never finished?

Most durable queues use some version of a lease, also called a visibility timeout. When a worker pulls a job, the queue doesn’t delete it immediately, it hides it for a set window, say five minutes. If the worker finishes and acknowledges the job, it’s removed for good. If the worker crashes, times out, or never checks back in, the lease expires and the job becomes visible again for another worker to pick up.

This matters specifically for scraping because scrape jobs fail in ways that are often transient: a proxy that’s temporarily rate limited, a target site that’s slow, a connection reset. A lease-based retry means the job gets tried again automatically without you writing manual retry code, as long as you set the lease window sensibly. Too short, and jobs get double-processed while a slow-but-working request is still in flight. Too long, and a genuinely dead worker leaves the job stuck for a while before it comes back.

Not every retry should look the same

A flat “retry three times” policy treats a temporarily overloaded proxy the same as a page that returns a hard 404 or a site that’s actively blocking the request pattern. Those aren’t the same failure and shouldn’t get the same response.

A queue that supports backoff (waiting longer between each retry) handles transient network issues reasonably well. But a job that keeps failing because a target has started returning CAPTCHA pages or has changed its markup structure isn’t going to succeed on retry four. That’s what a dead letter queue is for: after N failed attempts, move the job to a separate holding queue instead of retrying forever, so it doesn’t burn worker time all night and instead surfaces in the morning as a bucket of “these need a human.” Building this in from the start saves you from a queue full of jobs quietly failing on loop from 1am to 7am.

Rate limiting belongs in the queue’s job model, not just the code

A queue that’s a plain FIFO with no concept of “how many of these can run against this target right now” will happily let fifty workers hit the same domain simultaneously the moment there’s a backlog. Overnight, with nobody watching the response codes, that’s how a scraper goes from “running” to “entirely blocked” between midnight and sunrise.

This is where the queue and the proxy pool have to work together operationally, not as separate concerns. Concurrency limits per domain, and ideally per proxy or per IP, need to be enforced at the point where jobs get dispatched to workers, not hoped for in application code that a tired engineer wrote at 11pm before the run started. Some queue libraries support per-queue or per-tag concurrency caps directly (Celery has rate limiting per task, RQ can be run with separate worker pools per queue). If your queue doesn’t support this natively, routing jobs for different targets into separate named queues, each with its own worker concurrency, is a low-tech way to get the same effect.

Ordering: FIFO isn’t always what you want

Plenty of scrapers default to FIFO because it’s the obvious behavior, and for orderly crawls of a fixed URL list, that’s fine. But if part of your pipeline is discovering new URLs while it runs, for example a crawler that queues newly found product pages as it goes, a strict FIFO queue means fresh discoveries sit behind whatever was already queued, sometimes for hours. Depending on how time sensitive the data is, a LIFO or priority queue for freshly discovered URLs can matter more than raw throughput. This is a case where the answer depends entirely on what the scrape is actually for, not a default you should copy from someone else’s stack.

Watching a queue you can’t watch

The honest failure mode for an overnight scraper isn’t usually a crash, it’s a slow silent stall: queue depth climbing because workers can’t keep up, or a target site quietly returning empty pages that count as “success” but produce no data. A queue you can query for depth, in-flight count, and dead letter count gives you something to alert on. A basic threshold alert, queue depth above some number for more than N minutes, or dead letter count above zero, catches most of what would otherwise be an unpleasant morning discovery. This is a bigger part of “choosing a queue” than it looks like up front: pick something you can actually inspect while it’s running, not just something that moves jobs from A to B.

What this comes down to

There’s no single right queue for an overnight scraper. A solo operator running a few thousand pages a night is well served by Redis plus RQ or a Postgres table with SKIP LOCKED. A team running multiple pipelines against dozens of targets probably wants RabbitMQ or SQS with proper dead letter routing and per-domain concurrency limits. What doesn’t scale to an unattended run, regardless of size, is a plain in-memory list with no lease semantics, no retry policy, and no way to see what’s stuck. The queue is the part of the system that has to keep working when you’re asleep, so it’s worth choosing on those terms rather than on which one was fastest to wire up on a Tuesday afternoon.

If you’re setting up the rest of an overnight pipeline, from proxy rotation to fingerprint handling, we cover the infrastructure side in more depth on Data Research Tools.

Get new guides and videos first — join the Telegram channel.

The backfill problem is a shape problem

Most scraping pipelines are designed around steady state: check for new listings every hour, pull today’s prices, watch for changed pages. That traffic pattern is gentle. A backfill is not. When someone asks for three years of historical pricing, five years of job postings, or every review a product has ever received, you are not adding a small daily load to your pipeline. You are trying to compress months or years of “normal” traffic into a window of days or weeks.

That compression is the whole problem. Proxy bills are driven by volume and bandwidth over time. Detection systems are driven by request velocity and pattern anomalies. A backfill spikes both at once, which is why teams that budget fine for ongoing collection get blindsided the first time they try to go back and fill in history.

Why bursty traffic costs more than steady traffic

Two costs stack on top of each other during a backfill: proxy spend and ban rate, and they feed each other.

Proxy spend is usually metered by bandwidth (GB transferred) or by request/IP allocation, depending on the provider and proxy type. Ten thousand requests spread over a month barely register. Ten thousand requests in six hours from a small pool of IPs looks like a burst regardless of how the bill is structured, because you are consuming a much higher rate of bandwidth or session allocation in a shorter window, which for metered plans is exactly what you are paying for.

Ban rate rises with velocity because that is one of the more reliable signals defenders have. A single IP or fingerprint requesting pages at a pace no human or normal browsing session would produce is a pattern that rate limiters and bot-detection systems are specifically built to catch, independent of whether the IP itself has a bad reputation. Sites that see almost no traffic to five-year-old paginated archives suddenly getting hit in sequence are seeing a pattern that does not occur in organic traffic. When bans climb, retries climb, and retries are the thing that quietly doubles your bandwidth bill without doubling the amount of new data you collected.

None of this means a backfill is impossible or that any given technique makes it invisible. It means the traffic shape itself is the primary cost driver, and the fix is mostly about changing that shape, not about finding a proxy that magically absorbs it.

Check whether you need to scrape at all

Before spinning up a crawl, it is worth a genuine hour of checking whether the historical data already exists somewhere cheaper to pull than the live site.

Common Crawl has archived a large slice of the public web on a recurring schedule for over a decade, and if the pages you need were live and crawlable at the time, there is a real chance they are sitting in a WARC file already. The Internet Archive’s Wayback Machine indexes individual URLs over time and can be queried for available snapshots. Neither will have everything, and both come with their own access patterns and rate limits worth respecting, but pulling from an archive costs a fraction of the bandwidth and creates zero load on the target site.

Many data sources also have licensed historical exports, bulk data dumps, or paid API tiers specifically because backfill demand is common enough that vendors monetize it directly. A paid export that costs money up front is frequently cheaper in total than the proxy bandwidth, retry overhead, and engineering time of scraping the same history page by page. This is worth pricing out before writing a single line of crawler code.

Match the proxy type to the actual defense, not to habit

A lot of teams default to residential proxies for every job because residential IPs are harder to distinguish from organic traffic. That is true, and it is also the most expensive proxy tier per GB by a wide margin on most providers. If the target is a static content site, a government archive, or a source with minimal bot detection, that premium is often wasted spend for a backfill where volume, not stealth, is the dominant cost.

The decision that actually matters is: what is defending this specific target? A site with aggressive fingerprinting, TLS/JA3 checks, and behavioral analysis is a different problem than a site with a basic per-IP rate limit and no other defenses. Datacenter proxies are usually far cheaper per GB and per request, and for the second case they are frequently sufficient. Spending residential-tier budget against a target that only checks request rate is a self-inflicted cost overrun, and it is one of the most common mistakes teams make when a backfill job gets handed to whoever last built the incremental scraper, using the same proxy pool by default rather than re-evaluating it for the new job’s scale.

Headless browsers are the most expensive way to fetch a page

If the data you need is present in the raw HTML or reachable through the site’s own API calls, a headless browser is usually overkill for a backfill, and it is expensive in a way that compounds. A full page render pulls images, stylesheets, fonts, ad scripts, and analytics beacons, easily five to ten times the bytes of the actual content payload depending on the page. Multiply that by proxy bandwidth pricing across a multi-year backfill and the browser overhead can dwarf the cost of the data itself.

Blocking unnecessary resource types in the browser (images, fonts, media) or, better, checking whether the page’s data is available through a lighter HTTP request in the first place, is one of the highest-leverage cost cuts available. Save headless rendering for pages that genuinely require JavaScript execution to expose the data, and profile bandwidth per page type before committing a whole backfill to the heaviest fetch method by default.

Spread the load instead of compressing it

The single biggest lever for both cost and stability is time. A backfill does not have to finish in a weekend. Stretching the same total request volume across a longer window, with concurrency and per-target rate limits set closer to what a busy but organic crawler would produce, reduces both the burst-driven proxy spend and the velocity signal that triggers rate limiting. This is a scheduling decision, not a workaround: it means setting realistic deadlines with whoever requested the backfill and treating “how fast do we actually need this” as a real design input rather than an afterthought.

Checkpoint the crawl so failures don’t cost twice

Backfills fail partway through more often than incremental jobs, because they run longer and touch more surface area. Without checkpointing, a crash or a ban three-quarters of the way through means re-fetching everything, which doubles the bandwidth spend for zero new data. Persisting crawl state (which URLs or date ranges are done, which are pending, which failed and why) means a restart resumes instead of repeats. Deduping against already-collected records before re-queuing anything is a small amount of engineering effort that directly protects the budget.

Watch the run while it’s happening

A backfill should be monitored live, not kicked off and checked the next morning. Tracking error rate, ban rate, and response codes per target and per proxy pool in real time lets you stop a run that has started failing before it burns through the rest of the budget on requests that are mostly getting blocked anyway. A job that is failing 40% of the time six hours in is not going to recover on its own, and letting it run to completion just to see the final number is an expensive way to learn that.

Where the line is

None of this changes what is and is not appropriate to collect. Historical backfill of public, non-personal data is a different exercise from scraping paywalled content, content behind a login, or personal data at scale, and it does not become more acceptable because it is old. Respecting robots.txt and a site’s terms where they apply, and being honest about what a target actually permits, matters just as much for three-year-old pages as it does for today’s.

If you’re building out a backfill and want to think through proxy sizing, bandwidth math, or archive-first sourcing for a specific target, that’s the kind of thing we dig into on Data Research Tools.

Get new guides and videos first — join the Telegram channel.