Choosing a storage format for millions of scraped HTML pages

The problem nobody plans for

Most scraping projects start the same way. You write a crawler, it hits a few hundred pages, you save each response as a file, and everything works. Then the run grows to a million pages, or ten million, and the storage layer you never thought about becomes the thing that breaks first, not the scraper itself.

Picking an html storage format is not a cosmetic decision. It determines how fast you can reprocess data when your parser changes, how much disk and I/O you burn, whether backups finish overnight or not at all, and whether your pipeline can survive a crash halfway through a crawl. This post walks through the real options we use in production pipelines and where each one falls apart.

Why one file per page breaks down

Saving each scraped page as its own file on disk is the obvious starting point, and it’s fine at small scale. The trouble shows up on three fronts once you cross into the millions.

First, filesystems have limits on the number of files a single directory can hold efficiently. Ext4 and NTFS will technically let you keep piling files into one folder, but directory listing, file creation, and even simple ls calls slow down noticeably as the count climbs. You end up building a bucketing scheme (hashing the URL into subfolders) just to keep the filesystem usable, which is extra engineering for something that should be an implementation detail.

Second, most filesystems allocate storage in fixed block sizes, commonly 4KB. An HTML page that’s 2KB on the wire still consumes a full block on disk. Multiply that waste by ten million small files and you’re paying for storage you never actually used.

Third, and this is the one that bites people hardest, backing up or moving ten million small files is slow in a way that has nothing to do with total data size. Copying one 50GB file is fast. Copying ten million files that add up to 50GB can take hours, because the overhead is per-file, not per-byte. If you’ve ever run rsync on a directory full of scraped pages and watched the file counter barely move, you know this pain.

The WARC format exists for exactly this reason

Web archiving projects hit this same wall decades ago, which is why the WARC format (Web ARChive) exists. It’s the format Common Crawl and the Internet Archive use to store crawled pages, and it solves the small-files problem directly: instead of one file per page, you write a single large file containing many page records back to back, each with its own header (URL, timestamp, HTTP response headers, content length) followed by the raw response body.

Readers use byte offsets stored in an index to jump straight to a given record without reading the whole file, so you get random access without the filesystem overhead of millions of individual files. Because WARC preserves the full HTTP response including headers, it’s also a genuinely honest archival format. You’re not just saving the HTML, you’re saving what the server actually sent, which matters if you ever need to debug why a parser misbehaved on a specific page (redirect chains, unexpected content types, gzip encoding issues, all of that lives in the headers).

The tradeoff is tooling. WARC readers and writers are a layer you have to add to your pipeline, and if your team is used to just opening files, it’s a mental shift. For archival-grade or very large crawls, it’s worth it. For a scraping job that tops out at a few hundred thousand pages, it may be more format than you need.

Object storage as the middle ground

For most production scraping pipelines we run, the practical answer sits between “one file per page on local disk” and “roll your own WARC pipeline.” S3-compatible object storage (AWS S3, or self-hosted alternatives like MinIO) handles the small-files problem differently: it’s built to store billions of objects, and listing, versioning, and lifecycle rules are handled by the storage layer instead of your filesystem.

The key design decision is still the same one you’d face locally: one object per page, or bundled objects. One object per page (keyed by a hash of the URL, not the URL itself, since URLs contain characters that don’t play nicely as object keys) is simple and gives you direct access to any single page. Bundling groups of pages into compressed archives (effectively mini-WARC files) written to object storage cuts down on the number of objects and the per-request overhead, at the cost of needing an index to find a given page inside a bundle.

Which one makes sense depends on your access pattern. If you mostly reprocess data in bulk (rerun a parser over the whole dataset), bundling wins because you’re streaming through data anyway. If you need to randomly fetch individual pages by URL on demand, one object per page with a hash-based key is simpler to reason about, even if it costs more in object count.

Compression is not optional at this scale

HTML compresses well because it’s repetitive text: tag names, whitespace, boilerplate navigation and footer markup that repeats across every page on a site. Storing raw, uncompressed HTML at scale is close to the easiest optimization you can skip.

Gzip is the safe default. It’s supported everywhere, decompresses fast, and every language has a mature library for it. Zstandard (zstd) is the newer option worth knowing about: it generally compresses faster than gzip at a comparable ratio, and decompression is noticeably quicker, which matters if your reprocessing jobs decompress the same archive repeatedly. Neither is universally “better,” the right pick depends on whether you’re optimizing for write throughput, read throughput, or compatibility with existing tooling in your stack. Don’t take a specific ratio or speed number as gospel from a blog post, including this one. Test it against your own HTML, since the actual compression ratio depends heavily on how repetitive the pages in your dataset are.

One thing worth doing regardless of compression algorithm: dedupe before you store. If your crawler revisits the same page, or if a site serves near-identical content across thousands of URLs (pagination boilerplate, templated product pages), hashing the content and storing a pointer to an existing blob instead of a fresh copy can save more space than compression algorithm choice ever will.

Should you even keep the raw HTML

This is the question that gets skipped most often. Teams default to keeping every raw page forever because storage feels cheap, but raw HTML has a real cost: it’s bulky, most of it is boilerplate you’ll never look at again, and keeping it means keeping decompression and parsing logic working indefinitely.

The honest tradeoff is this. Keep raw HTML if you expect your extraction logic to change, if the source site’s structure is your business (competitive monitoring, price tracking where you need to re-verify what was actually shown), or if you need to prove what a page contained at a point in time. Don’t keep it if you only ever need the parsed fields and you’re confident your parser is stable. In that case, extract once, store the structured output (JSON, a database row), and discard the HTML, or keep it for a short retention window instead of forever.

A common middle path in production pipelines: store raw HTML in a cheap, cold-tier object storage bucket with a lifecycle rule that deletes it after 30 or 90 days, and store parsed output in a proper database indefinitely. That gives you a recovery window if your parser turns out to be broken without paying to store raw pages forever.

A rough decision guide

If you’re under a few hundred thousand pages and mostly experimenting, files on disk with gzip compression is fine, don’t over-engineer it. Once you’re consistently crawling into the millions, move to object storage with hash-based keys or bundled archives, and pick your compression algorithm based on whether you read or write more often. If you’re building something closer to a general-purpose archive that other tools or teams will consume, WARC is worth the tooling investment because it’s a standard format with existing readers, not something you invented that only your pipeline understands. And separately from format, decide deliberately how long you actually need the raw HTML, because that decision affects your storage bill far more than the format choice does.

None of this is exciting engineering, but it’s the layer that determines whether a scraping pipeline survives contact with real scale or quietly grinds to a halt six months in.

For more breakdowns of scraping infrastructure, proxy setups, and pipeline design, check out the rest of Data Research Tools here.

Get new guides and videos first — join the Telegram channel.