Handing a scraper over to a client

I have inherited four scrapers written by somebody else. The code was readable inside an afternoon every time. Rebuilding everything around the code took between three days and a fortnight.

One of them had a fixed 2.3 second delay in the fetch loop with no comment next to it. I deleted it, because it looked like debugging somebody forgot to remove. Two days later I understood exactly why it was there.

That is the subject in one story. The repository is the easy half of a handover, and it is the only half most people transfer.

I run mobile proxy lines and production scrapers out of Singapore, so I have been on both ends of this. The jobs that turned sour were almost never badly built. They were badly handed over.

Working is a state it passes through

A scraper is a dependency on a website somebody else owns. No contract exists between you and them, they do not know your project exists, and they can change a template on a Wednesday afternoon for reasons entirely unrelated to you.

So “working” is a condition your collector moves through, between changes at the far end. It does not settle there.

Your client thinks otherwise, and I would not blame them for it. They are used to buying software the way they bought their invoicing tool. You install it, it runs, and a thing that stops running is a defect somebody owes them a fix for.

Leave that assumption in place at the moment of transfer and every future site change lands on you by default. Not through anybody’s bad faith. Because nobody said otherwise while the atmosphere was still friendly.

The paragraph that goes at the top

Before the install steps, before anything technical, one paragraph:

This collector works against the target site as it exists on the date of transfer. The site will change. When it changes, the collector will stop returning correct data. That is expected behaviour and not a fault in the delivery.

I bold it, and I read it out on the handover call, because clients skim documents and remember awkward sentences said to their face.

It reads badly when you are trying to look competent. It reads a great deal better than the email four months later that starts “we paid you to build this”.

Four things that never make it into git

The code transfers itself. It is text, it sits in version control, and any competent engineer can read it. Everything that makes the code actually run is somewhere else.

Credentials, and the whole apparatus behind them. Not just the password for the target account. The email address it was opened with, the phone number that receives the second factor, and whose name the account legally stands in. If it stands in yours, you are permanently in the loop on a system you no longer support. I now open target accounts on the client’s email address from the first day of a build, even when it costs me an hour of back and forth.

The proxy arrangement, including the billing. Which provider, which plan, what a normal month consumes in bandwidth, and whose card renews it. Get that last one wrong in either direction and it hurts. Either your card keeps paying for a client you stopped working for, or it lapses and the collector dies quietly on a Tuesday. I watched a team spend two weeks reading Python because a proxy plan had expired on a card belonging to somebody who left the company. On my own infrastructure a real SIM line runs about ten dollars a month in airtime, so a client inheriting three lines has taken on a thirty dollar monthly commitment, and they should hear that number from me rather than from their bank statement.

The quirks. This is the highest value page in the whole document and it is the one that gets skipped, because it is the only part you cannot generate from the code. Every target does something strange that you found the hard way and then worked around silently. The listing endpoint returns 24 items and the last two are always adverts. The rate limit counts per account rather than per address, so buying more proxies achieves nothing. A background call returns clean JSON, so the parser never touches the rendered HTML at all. Response quality degrades above roughly four pages a second, but only in the evening. None of that lives in the code. Some of it is the reason a line of the code looks idiotic, like a 2.3 second delay nobody explained.

I keep a plain text file per target while I build, one line per surprise. By delivery it is usually twenty or thirty lines long, and it is the page the receiving engineer reads twice.

Three failures, three pages

You cannot document everything that might go wrong. You can document the three things that will.

A selector breaks. The markup changed, a field arrives empty or wrong, and the row count looks perfectly healthy. Say which file holds the selectors, how to confirm the site moved rather than the code, and roughly how long a repair takes in hours.

The block rate climbs. Responses stop being data and start being challenge pages, usually carrying a cheerful 200 status. Say what a block page looks like on this specific site, how to measure the rate, and what to do first, which is nearly always slow the crawl down before anyone starts swapping proxy pools.

The volume moves. Forty thousand rows yesterday, two hundred today. Say how to distinguish a block from a layout change from the client’s own filter, in under ten minutes.

I have never needed a fourth entry. I have needed all three more times than I can count.

One signal they can read without you

Documentation only helps somebody who already suspects a problem exists. What decides whether you get blamed is whether the client finds out from a dashboard or from a stale report five weeks later.

So the final deliverable is a signal they can read themselves. Not the monitoring layer I would build for my own jobs, with rolling baselines per source, block page fingerprints and a run record for every execution. I have written that up separately and it is worth the engineering.

For a handover the requirement is smaller and much stricter. One page or one daily email, readable by somebody who does not write code, in under a minute on a Monday morning. Last successful run and when. Rows collected against what a normal day looks like. Field fill rate for the two or three columns that actually matter. A sentence at the top that says either this looks normal or this does not look normal.

Here is the position I will argue for: handing over a scraper with no monitoring is handing over a lawsuit with a delay on it. The data goes wrong silently, somebody prices a product off it or puts it in front of a board, and when they eventually work backwards to the cause, the last name in the commit history is yours.

Support request or new build

The commercial half is where the goodwill actually drains away. If the document does not draw this line, every site change becomes a free rebuild, and the client is not being unreasonable when they ask for one. From where they sit, they bought a working scraper and it is not working.

A support request is the collector doing something it was not doing at handover, on the same site, for the same fields. That carries a response time and either sits inside a retainer or bills at an agreed hourly rate.

A new build is everything else. A new field, a new page template, a new site, a new output format, a login wall that appeared since delivery, or ten times the volume anybody agreed to. That gets quoted like any other job.

If you already split the build from the run when you quoted the work, this section is copy and paste. If you did not, the handover is your last opportunity to draw the line before it defaults to unlimited.

The one I got wrong

A collector I delivered in March ran for eleven days after I stopped watching it. On day twelve the site renamed a CSS class, the price field started arriving empty, and the client found out five weeks later when their finance team asked why a competitor pricing sheet had not moved since Easter.

The class name was not my mistake. The build was solid: tests, retries, a validation gate that rejected malformed rows, structured logs.

My mistake was that I shipped the repository and a README and called that a handover. No runbook. No quirks file, and that target had three good ones. No monitoring the client could see, because my monitoring was a cron job on my own server that emailed me, and I switched it off the week the invoice cleared.

So the validation gate worked exactly as designed. It rejected the blank rows. The pipeline logged the rejects and carried on. The output file kept landing on schedule with fewer rows in it each week, and nobody was watching the count.

I had built the detection and then handed over the system without it, which is worse than never building it at all. It produced a machine that knew it was broken and had no way to tell anybody. I rebuilt it for free, about eleven hours. Writing the handover template I still use took one afternoon.

What a handover cannot fix

None of this makes a scraper last longer. The site changes when it wants to.

The document also does not survive staff turnover on the client side. The engineer you briefed leaves, the file sits in a drive folder nobody opens, and eighteen months later somebody inherits a collector nobody understands. I have been that somebody.

The honest claim is narrower than it sounds. A good handover moves the surprise from the client to the calendar. They still get broken data eventually. They just find out on the day it breaks, from something they can read, with a page that tells them what to do next.

And it changes nothing about what you were allowed to collect in the first place. Public pages, the robots file honoured, a crawl rate that does not hurt the target, an official API or a bulk feed preferred every single time one exists. A beautifully documented handover of a job you should not have taken is still a job you should not have taken.

Everything else I have written on scraping operations and the infrastructure underneath them is here.

Get new guides and videos first — join the Telegram channel.