Versioning your scrapers and their output

A rolling deploy took about six hours to work through the fleet. For those six hours half my workers ran the new parser and half ran the old one, both writing into the same table, and not one row recorded which of the two had produced it.

I found out because somebody asked why a single day of data had two clearly separated distributions sitting inside it. Not drift. Not a spike. Two modes, produced on the same date, by the same job, under the same name.

I run mobile proxy lines and production scrapers out of Singapore. The first question I ask now when I inherit a pipeline is whether anyone can pick a row and say which code wrote it. Almost nobody can, and the people who cannot are usually the ones telling me their pipeline is fine.

The field that changes meaning without changing shape

Here is the failure this prevents, and it is not a crash.

I had a boolean called in_stock, originally set by checking whether an add to cart button existed on the page. Somebody on my side rewrote that check to look for a small availability badge instead, because the button rendered even on sold out listings and the badge looked more precise.

It was more precise. It was also warehouse specific, so listings available from a second warehouse now came back false.

The column stayed boolean. The fill rate did not move. Nothing went null, no type changed, no request failed. The true rate dropped by about nine points over a week, which is well inside the range that catalogue seasonality produces on its own. Every monitoring check I had passed.

So the table now held one column carrying two different definitions, separated by a deploy that left no trace anywhere in the data. Anyone querying across that boundary was silently comparing two things.

This is a distinct problem from the target site changing on you, which is worth monitoring for on its own and is a separate discipline with separate tooling. It is also distinct from rerun safety. A properly idempotent write path stops a retry from duplicating rows, and it is worth building, but it will happily let two different code versions upsert into the same key without either of them leaving a signature. Idempotency protects the row count. It does nothing for the row’s meaning.

Three columns is most of the fix

Stamp every row with the version of the code that produced it and with when the page was fetched.

In practice that is a short git commit hash, seven characters, read from the repository at build time and baked into the image. Then two timestamps rather than one: fetched_at for when the response came back, parsed_at for when the extraction ran over it.

Those two are identical on the first pass and diverge the moment you reparse anything, which is exactly the situation you want them for. fetched_at is a fact about the world. parsed_at is a fact about your code. Collapsing them into a single “created_at” throws away the distinction at the moment it starts to matter.

If your extractor has fallbacks, record which branch fired. A primary selector and a rescue selector produce rows that look identical and deserve different levels of trust, and once you have that column you can also see the rescue path quietly climbing from two percent of pages to thirty, which is an early warning you would otherwise have to go looking for.

Two things I would insist on. Never let the version be a constant somebody edits by hand, because it will eventually be wrong and a wrong stamp is more dangerous than a missing one. And make the deploy identity granular enough to survive a rolling release, which was my six hour problem: the version was correct on every row, I simply had not been recording it yet.

The usual objection is storage. Three columns on a hundred million rows sounds heavy until you notice the row already contains a url. A repeated seven character hash compresses to near nothing in any column store, and against a row that already carries text fields it is not measurable.

Keep the response

This is the position I will defend: keeping the raw response is the highest value habit in scraping, and it is the first thing that gets cut when someone looks at the storage bill.

The reason is not sentimental. A scraper is not a query you can run again. The source is somebody else’s website, it keeps no history for you, and the page you fetched this morning may not exist by Thursday. Prices move, listings sell, sites redesign, whole catalogues get pulled. Every fetch is the only opportunity you will ever have at that page in that state.

Which means the parsed row is a lossy derivative of something that no longer exists. If your parser had a bug, the raw response is the only thing standing between “we can repair this” and “those numbers are permanently wrong and I have to say so”.

Store the headers with the body, not just the html. Content type, redirect chain, declared encoding. A good share of parsing bugs are encoding bugs, and the evidence for those lives entirely in headers you did not keep.

Where and how you physically store all of that is its own decision, with real tradeoffs around file counts and formats, and a separate question from this one. All that matters for provenance is that the raw capture and the parsed row exist as two artifacts, and that the row carries a pointer to its capture. A content hash or an object key. That pointer is the chain.

The honest version of the retention argument: raw html is bulky and mostly boilerplate, and cold storage with a lifecycle rule at thirty or ninety days is a fair compromise. I run exactly that on most targets.

Just be clear about what the rule buys. You can only reparse the window you kept, so the window has to outlast your detection time. Mine has been badly wrong at least twice. The worst case was a bug a client’s finance team caught during a quarterly review, roughly eleven weeks after it started, against a thirty day retention rule. Nine of those weeks were unrecoverable. I widened the rule that afternoon.

The schema is a contract

The third piece is the one people skip, because it is process rather than code.

Your output schema is a contract with whoever consumes it, and it needs a version separate from the code version. The code version changes on every deploy, including the dozens that change nothing about the output. The schema version changes only when the shape or the meaning of what you emit changes, which is rare and always significant to a downstream consumer.

An integer column, and a plain file in the repository saying what each number means with a date against it. That is the whole implementation.

The discipline that goes with it is the part worth actually holding: never redefine an existing column, add a new one. When the availability logic changed, the correct move was in_stock_badge alongside in_stock, not a quiet redefinition. Old rows keep the old field, new rows populate both for a while, and the boundary is visible to anyone querying instead of buried in a commit message they will never see.

It produces an ugly schema. I would take an ugly schema over a column that means two things depending on the date.

What this actually buys

Three things, and they are the reason any of the above is worth the trouble.

You can reparse history. Fix the parser, run it back over the stored captures, write corrected rows under the new version, keep the old ones. Then diff the two sets, which turns “something was wrong for a while” into “forty one thousand rows were affected, the median error was eleven percent, here they are”. That is a survivable conversation. The other one is not.

You can attribute an anomaly to a deploy instead of arguing about it. Group your metric by code version rather than by date. If the step change sits exactly on a version boundary it is yours. If it sits somewhere else entirely, the site changed or the market did, and you have ruled out the expensive explanation in about ten minutes.

And you can answer the question months later. Which code, which fetch, which page, on what day. That question always arrives from someone senior, about a figure that has already left the building.

The one I got wrong

For the first year I stamped the code version and only kept a single timestamp, updated on write.

Then I reparsed a batch to fix a field, and the reparse overwrote that timestamp on every row it touched. I had turned “this page was fetched on the fourth” into “this row was written on the nineteenth” across several hundred thousand rows, and there was no way back, because I had never stored the original separately. The captures still existed and the fetch times lived in their metadata, so I recovered most of it over a slow weekend. On the targets where the captures had already aged out, I did not.

Provenance has a limit too, and it is worth being blunt about. Knowing which code produced a row tells you nothing about whether that code was right. You can have immaculate provenance on a completely wrong number. What you have bought is the ability to discover it later and repair it, which is not the same as protection.

None of it changes what you should be collecting either. Public pages, the robots file honoured, a crawl rate that does not hurt the target, and an official api or a bulk feed used whenever one exists. Good provenance on data you should not have taken is just a tidier record of the decision.

The pipelines I have seen survive a change of owner are the ones where a new person can take a row at random and walk it back to the response it came from. Everything else is a box producing numbers that people have agreed to trust. The rest of what I have written on pipeline design, storage and the proxy infrastructure underneath it is over here.

Get new guides and videos first — join the Telegram channel.