Backfilling a Dataset After a Schema Change

What actually breaks when a schema changes

A schema change sounds like a database problem. In a scraping pipeline it’s usually a source problem that shows up as a database problem. A site restructures its product page, an API adds a field and renames two others, a listings endpoint starts nesting price inside a new object instead of a top-level key. Your parser was built against the old shape. It doesn’t crash, most of the time. It just starts returning nulls, or silently drops a field, or maps the wrong value into the wrong column because the position in the JSON shifted but the key name didn’t.

The part that catches people out is the gap the change leaves behind. Fixing the parser is the easy half. Everything scraped before the change is stored under the old schema. Everything after is stored under the new one, assuming you caught it fast enough. Anyone querying that table now has to know which rows came from which era, or the query is wrong without erroring. Backfilling is the work of closing that gap: getting every historical row into a consistent shape so the dataset behaves like it was always collected the same way.

Replay versus re-fetch

There are two ways to backfill, and they cost completely different things.

Replay means you already have the raw response saved from when you first scraped it, and you run that raw HTML or JSON back through the new parser. This is fast, doesn’t touch the target site again, and doesn’t burn any proxy or bandwidth budget. It only works if you kept the raw data.

Re-fetch means going back out and pulling the page or endpoint again, because you never stored the raw response, only the parsed columns. This means the record you get now reflects the site as it is today, not as it was on the date you originally collected it. For anything price-sensitive, inventory-sensitive, or time-sensitive, that’s a different dataset wearing the old dataset’s timestamps.

If your pipeline only ever stored parsed output and threw away the raw page, you’ve already made this decision for future-you, and it wasn’t the flexible one. Teams that get burned by a schema change once usually change exactly one thing afterward: they start keeping raw snapshots, compressed and cheap, specifically so the next schema change is a replay job instead of a re-scrape.

Why raw snapshot storage is the real fix

Storing raw HTML or raw JSON responses looks wasteful until the day a schema changes, at which point it’s the only thing that makes a clean backfill possible. The storage cost is real but it’s usually small relative to the cost of a re-scrape: no proxy rotation, no rate limiting to respect, no risk of the historical page no longer existing because the listing expired or the product got delisted.

Retention policy matters here. Keeping raw responses forever isn’t necessary for most use cases, but keeping them for a rolling window, long enough to cover a typical schema-change discovery lag, is what turns a backfill from a multi-day re-scrape into a batch job you can run overnight against data you already own. If you’re running your own scraping infrastructure rather than pulling from a managed API, this is one of the clearer arguments for owning the pipeline end to end: you control what gets kept, not just what gets parsed.

Version the schema instead of overwriting it

The instinct when a schema changes is to update the parser and let it write into the same columns. That works for new data going forward, but it’s the wrong move for the backfill itself, because it destroys the ability to tell what changed and when.

A more durable pattern is to treat the schema as versioned. Each row carries a schema version alongside the parsed fields, or the raw response is tagged with a parser version at ingest time. When you backfill, you’re not mutating history in place, you’re generating a new set of parsed rows from the old raw data under the new schema version, and either replacing the old parsed rows or keeping both and pointing consumers at the latest version. This costs a bit more storage and a bit more pipeline discipline, but it means a bad backfill is reversible. If the new parser has a bug that only shows up on 2 percent of historical pages, you can roll back to the previous version instead of re-running a re-fetch against a site that’s moved on again.

Test the backfill before you trust it

Running a backfill across the entire history in one pass is how a small parser bug becomes a dataset-wide problem. The standard approach is the same one you’d use for any data migration: run it against a sample first, diff the output against what you’d expect, and only then run it against everything.

A useful sample isn’t random. Pull rows from just before the schema change, just after, and a handful from well before it, since older pages sometimes have structural quirks a newer parser wasn’t written to handle. Check null rates per field before and after the backfill. A field that was 99 percent populated before and is suddenly 40 percent populated after is a parser bug, not a data quality issue with the source. Check for fields that used to hold one type and now hold another, since a schema change that turns a string price into a structured price object is exactly the kind of thing that parses without error but produces garbage if the parser wasn’t updated to match.

This is the same discipline as testing any database migration on a copy before running it in production. Nothing exotic about it. The difference with scraped data is that there’s no source of truth to compare against except the raw response itself, which is another reason keeping it matters.

What re-fetching actually costs

Sometimes replay isn’t an option, either because raw data wasn’t kept or because the backfill needs current values rather than the values as they were. In that case the backfill is a re-scrape, and it competes with your live scraping jobs for the same resources: proxy pool capacity, rate limits on the target, and time.

Treat a backfill re-scrape as a batch job with its own budget, not something that runs at full speed alongside live collection. Throttling it below your normal request rate, running it during off-peak hours for the target site, and spreading it across a longer window all reduce the chance that a large backfill run looks like a burst of unusual traffic to whatever detection is on the other end. Throttling this way is the same operational courtesy you’d extend to any large batch job hitting a third-party service: don’t hammer it just because you can. It’s not a way to hide the backfill.

It’s also worth being honest that a re-fetch backfill can’t fully undo the original mistake. If the source page has since changed content, been deleted, or moved behind a login, the backfilled row will not represent the historical state, and anyone using the dataset for trend analysis needs to know that gap exists rather than have it silently smoothed over.

Catching the next one earlier

The backfill itself is a one-time cost. The recurring cost is the detection lag between when a site changes its structure and when someone notices the parser is producing bad data. Monitoring null rates and field type distributions per source, per day, is the cheapest way to shrink that gap. A sudden jump in nulls for a field that’s normally reliable is usually a schema change before it’s anything else, and catching it in a day instead of three weeks is the difference between a small backfill and a large one.

Pipeline orchestration tools that support scheduled data quality checks, not just scheduled scraping runs, make this easier to enforce as a standing practice rather than something someone remembers to do manually after the fact.

If you’re building out a scraping pipeline and want more on how we handle schema drift, raw storage, and proxy infrastructure on the collection side, check out the rest of what we’ve written and built at Data Research Tools.

Get new guides and videos first — join the Telegram channel.