What to check when a target site publishes an api

Four percent of the records disagreed.

I had an api and a scraper pointed at the same site, writing into two tables, and after a week I compared them row by row. Ninety six percent matched exactly. Every mismatched record had been edited in the previous twenty four hours: the api was reading a replica with a cache in front of it. That appears nowhere in its documentation, and it would have appeared nowhere in my head if I had migrated the way everyone migrates.

Read the announcement, feel relieved, spend a fortnight on the client, delete the scraper.

I run mobile proxy lines and production scrapers out of Singapore. I have done this migration several times and got it wrong at least once, in a way that cost three days I had already told myself I owned.

Which kind of api are we talking about

Three things get filed under one heading and only one of them is this.

There is the undocumented json endpoint you find in the network tab, the one the site’s front end calls to fill the page. I use them constantly and have written about finding those. That endpoint has one consumer, the site’s own javascript, so nobody there carries an obligation to keep its shape still. I have watched one change twice in a month while the visible layout stayed identical.

There is a third party scraping vendor, where you pay somebody else to do the fetching and the target has no idea that supplier exists. That is a buy versus build question about your own infrastructure, and it leaves your standing with the target where it was.

Then there is this case. The site chose to publish. A documentation page, a version in the path, a changelog, and somewhere a person who gets a ticket when it breaks. That is a different animal, because it is the only one where the target learns your name.

What genuinely gets better

The upside is real and larger than api cynics admit.

Types. A number arrives as a number, a date as a date, an absent value as null instead of an empty element you have to interpret. Your parser stops being the most brittle component you own, and the parser is the thing that fails on somebody else’s design calendar, usually in silence.

A contract. Versioning in the path, a changelog, a deprecation window, occasionally an email before a field moves. Compare that against finding out because a column went blank and a customer got there first.

And the traffic stops being a fight. No mobile addresses on that target, no headless browser, no guessing at what counts as a polite rate. On one job the monthly run cost fell to almost nothing, because the whole of it had been proxy bandwidth for a two megabyte page.

The api is a subset of the page

Here is what almost nobody checks before committing.

The page and the api are two different field selections made for two different audiences. The page was built for a shopper. The api was scoped in a meeting for a partner integration, probably a while ago.

Derived fields fall out first. Anything the front end computes from several other values. Anything a merchandising team added as a badge.

Availability is the standard example. The page says in stock at four locations and names them. The api returns a boolean.

Review counts, position within a category listing, the recently sold counter. Those live in the presentation layer, and the presentation layer is precisely what you have agreed to stop reading.

The sting is that derived fields are frequently the reason your product exists, because nobody else could get them cheaply. The identifiers everybody already has are what the api hands you first.

The quota is a policy

A scraper’s ceiling is physical. Concurrency, address pool, and whatever load you are willing to put on somebody’s server.

An api’s ceiling is a number in a document, and documents get edited. I have watched a free tier drop from sixty calls a minute to twenty, announced in a changelog entry and nowhere else.

Two pieces of arithmetic before you migrate. Divide your record count by the published quota and read the answer in hours, because that is your run regardless of hardware. Then find out what counts as one call, because a list endpoint returning fifty records is a different budget from a detail endpoint you hit once per record.

What the key costs you

A key means an account. An email address, usually a company name, plus a box you tick with a legal agreement sitting behind it.

Before, you were traffic. Anonymous, hard to separate from a browser, and a site that wanted you gone had to find you first. After, you are a row in their database with a name on it and a switch beside it. Revoking a key takes one click from somebody who never has to justify it. Blocking a scraper is a project they can lose.

The agreement contains things nobody reads. Retention limits. Whether you may display it to your own customers. Whether you may use it in anything that competes with them. I have read one that capped storage at twenty four hours, which rules out every historical product you might have planned, and that cap lived in the terms rather than the docs.

Read the termination clause specifically. Whether it requires cause, and what notice you get. That paragraph describes the relationship more accurately than the entire feature list above it.

Pricing arrives after the scraper is gone

The sequencing is deliberate. A new api is usually free while the publisher wants adoption, because they need integrations to point at. Metering shows up later, once there is a population of users who have already deleted their alternative.

That is ordinary product behaviour rather than malice. It is still the risk, because the cheapest moment to own a working scraper is before you learn what the api costs.

An api is a relationship and a scraper is not

Here is the claim, and I do get argued with about it.

The relationship gives you notice, a changelog, a version, somebody to email when a field moves. It also gives them a switch, an agreement you signed, and a price they get to set.

The scraper gives you no notice, no support, and maintenance with no end date. It also gives them nobody to switch off and no invoice to raise.

Neither wins in the abstract. The only question is whether you can live with someone else holding a switch over the thing you sell. If the data is an input to a product, usually yes. If the data is the product, that is a genuine dependency and it should be priced as one rather than treated as good news.

Map your columns before you read the docs

The audit takes about half an hour.

Write down your actual field list, from the output table rather than from memory. For every column, note where it comes from today: a raw value off the page, or something you compute from two or three scraped values.

Then open the field reference and map them. Each column gets a named api field or it gets a question mark.

The question marks are the whole exercise. One is a conversation with your product owner. Three means you are keeping a scraper whatever happens, and the migration has become a hybrid rather than a replacement.

Do this before the meeting where somebody proposes just using the api. It is the only artefact in that room with facts in it.

Run both, then diff the values

A mapping exercise compares names. A diff compares values, and values are where the surprises live.

Run both collectors against the same records on the same schedule, each writing into its own table, for a few weeks. Yes, you are paying twice. The bill is a few weeks of proxy traffic you were already spending, and it finds what documentation cannot describe.

What turns up: an api price that is the base price while the page price includes a promotion; a numeric field present and silently rounded; the replica lag I opened with.

Set your threshold before you look. Mine is that anything above one percent disagreement halts the migration until I understand the cause.

The join you are not going to get

The api’s identifier for a record is often not the identifier your scraper keys on, and it carries no obligation to expose yours. So your history does not join. Two years of stored rows, and the new ones will not line up. Check that field on day one, because a missing shared key means a matching pass, and matching is its own project.

Keep the old collector running

After you migrate, keep the scraper and keep it running rather than parked. A weekly job over a hundred records is enough. It stays green, and on the day the key gets pulled or the quota moves you have something that executes instead of a repository you have to remember the shape of.

Where I got this wrong

That last paragraph is advice I gave myself and then ignored. I migrated, kept the code, and never ran it. Eight months later I needed it back and the site had been redesigned twice. What I actually had was a folder. Rebuilding took three days, which is survivable, and which I had already counted as bought.

The honest limit: I have never had a key revoked on me. The failures I have been hit by are the subset problem and the quota problem. When I tell you the switch matters, that is reasoning rather than a scar.

There are cases where an api is straightforwardly right and none of this applies. A regulator publishing a bulk api is publishing it so you stop crawling them. No commercial lever, no competitive clause, and arguing about relationship risk there is just being difficult.

More on scraping infrastructure, pipeline design and the proxy layer underneath it is at Data Research Tools.

Get new guides and videos first — join the Telegram channel.