Deciding How Long To Keep Raw HTML After A Project Ends

The question nobody schedules time for

Every scraping pipeline we’ve run ends the same way. The crawl finishes, the parsed data lands in a database or a CSV, someone ships the report, and the raw HTML sits in a bucket somewhere because deleting it felt premature and nobody wanted to be the one who pulled the trigger too early. Six months later that bucket is still there, still costing money, and nobody remembers why.

Raw HTML retention isn’t a technical problem so much as a decision-making problem. The technical part (where to store it, how to compress it) is easy. The hard part is deciding when it stops being useful and starts being liability. This post walks through how we think about that decision on our own infrastructure.

Why raw HTML piles up in the first place

Most scraping pipelines separate collection from parsing. You fetch the page, save the raw response, then run a parser against it to pull structured fields. That separation exists for a good reason: if your parser has a bug, or the site changes its markup mid-crawl, you can reprocess the raw HTML instead of re-scraping the target. Re-scraping costs proxy bandwidth, time, and risk of getting rate-limited or blocked again. Reprocessing a file on disk costs nothing but CPU.

That’s a real, defensible reason to keep raw HTML around, at least for a while. The problem is that “at least for a while” quietly turns into “indefinitely” because nobody sets an expiry date at collection time. A crawl that pulls 200,000 pages a day, saved uncompressed, adds up to real storage spend within weeks. We’ve seen internal projects where raw HTML storage cost more per month than the proxy bill that generated it, simply because retention was never revisited after the project’s active phase ended.

What raw HTML is actually worth after the project ends

Once a project is “done” (the data has been extracted, validated, and delivered), the raw HTML has three possible futures:

It gets reprocessed. Someone finds a parsing bug, a new field gets requested, or a downstream consumer wants a value that wasn’t captured the first time. This is the strongest argument for retention, but it has a shelf life. If the site’s markup has since changed, or the underlying page no longer exists, the raw HTML is a snapshot of a moment that can’t be reproduced by scraping again. That snapshot value is highest in the days and weeks right after collection and drops off as the target site evolves out from under it.

It gets audited. Some projects need to prove what a page said at a point in time, for a compliance record, a dispute, or a research citation. This is a narrower use case than most teams assume. If nobody has asked for that kind of audit trail in the project’s lifetime, it’s unlikely they’ll ask for it later, and if there’s a real regulatory or contractual reason to keep evidence, that reason should be explicit and documented, not assumed.

It never gets touched again. This is the most common outcome by far. The parsed data was correct, the project shipped, and the raw HTML sits untouched until someone notices the storage bill or a retention audit forces the question.

The mistake is treating all raw HTML as if it falls into the first or second bucket by default. It doesn’t. Most of it falls into the third, and the only way to know which bucket a given batch belongs to is to look at actual access patterns, not assumptions made when the crawl was designed.

A framework that actually holds up

We don’t keep raw HTML on a single fixed retention period across every project, because projects differ too much in size and purpose. Instead we think in three tiers.

Short-term hot storage (days to a couple of weeks). Immediately after collection, raw HTML stays fully accessible for reprocessing while the parser is still being validated. This is the window where bugs get caught and fields get added. Keep it uncompressed or lightly compressed, keep it fast to query, and treat this period as part of the active project, not archival.

Compressed cold storage (weeks to a few months, project dependent). Once the parsed output has been validated and delivered, raw HTML moves to cheaper, slower storage, compressed. Gzip or zstd on raw HTML routinely gets 80-90% size reduction because markup is extremely repetitive text. At this point access should be rare and deliberate: someone requests it because a specific reprocessing need came up, not because it’s routinely queried. The retention length here should map to how often that kind of request has historically happened on similar projects, not to a gut feeling.

Deletion. After the cold storage window closes without a reprocessing need materializing, delete it. This is the step teams skip, because deletion feels like it forecloses an option, while keeping data feels free. It isn’t free. Storage costs money every month it sits there, and stored personal or scraped data that nobody has a current use for is exactly the kind of thing that turns into a liability if a target site, a customer, or a regulator asks what you’re holding and why.

The lengths of each tier depend on the project. A one-off market research pull that fed a single report might go from hot to deleted in under a month. A pipeline feeding an ongoing dataset that gets reprocessed regularly might keep a rolling cold-storage window measured in months, refreshed as new crawls replace old ones. The point isn’t to copy a specific number, it’s to set a number in advance and write it down, so retention is a decision made once at project design time instead of a default that nobody revisits.

The legal and scope angle

Raw HTML retention isn’t just a storage cost question. If the pages you scraped included anything that falls under personal data rules, the raw markup is a second copy of that data sitting outside your structured store, often with less access control and less awareness that it exists at all. A parsed database might have field-level permissions and a defined schema. A folder of raw HTML dumps usually doesn’t.

This matters most for projects touching regulated categories: anything with names, contact details, or other personal information embedded in the page. Treat raw HTML holding personal data with the same retention discipline you’d apply to the parsed output, not a looser one. If your parsed data has a defined retention and deletion policy because of GDPR, CCPA, or a contractual term with a client, the raw HTML copy needs to be covered by that same policy, or deleted sooner, since it typically has less structure and less oversight.

For projects that never touched personal data, purely product listings, public pricing, aggregate metrics, the legal pressure is lower but the storage argument still holds. Keeping terabytes of raw markup for pages that were fully parsed and validated months ago is a cost with no offsetting benefit.

What we actually do

On our own pipelines, retention gets set as a config value at project creation, not decided ad hoc later. Hot storage runs for the length of the active QA phase (usually the first one to two weeks after a crawl completes, since that’s when we’ve historically found parsing issues). Cold storage runs for a window tied to the project’s expected reprocessing cadence, logged in the project’s own documentation so the reasoning is visible later, not just the number. Deletion is a scheduled job, not a manual cleanup someone has to remember to run.

The failure mode we’ve seen most often isn’t keeping HTML too short, it’s never deciding at all, and ending up with years of accumulated raw pages that nobody can justify keeping and nobody wants to be responsible for deleting. Set the retention window when you design the pipeline, write down why you picked it, and let deletion happen automatically. That single habit prevents most of the storage and compliance headaches that raw HTML tends to create.

If you’re building out scraping infrastructure and want more on pipeline design, proxy management, and the operational side of running these systems at scale, check out the rest of what we cover at Data Research Tools.

Get new guides and videos first — join the Telegram channel.