Estimating a scraping project before you quote it

Twenty two hours quoted. Ninety one hours spent. That was one job, and the parser inside it took an afternoon, roughly what I said it would.

Everything that killed the estimate sat outside the code. A login I had not checked for. Four page templates I did not know existed. Six weeks of quiet breakage after the invoice had cleared and I had moved on to something else.

I run mobile proxy lines and production scrapers out of Singapore, so I have written these quotes and I have been handed them by other people. Bad scraping quotes are all bad in the same way. The person quoting priced the code, and the code was never the risk.

The build is the only predictable part

Ask an experienced person how long it takes to fetch a page, pull eight fields out of it and write them to a table, and you will get a good answer. That work is bounded and repetitive. After thirty or forty jobs you can call it inside a couple of hours.

So the build gets estimated well, and then it turns out to be maybe a fifth of the job.

The other four fifths hide in four places, and every one of them is visible before you quote if you actually go and look.

The site moves while you build. Not usually a redesign, just a variant template appearing or an advert slot dropping in above the element you anchored on. A two week build hands the site two weeks to shift under a price you already committed to.

The anti bot layer nobody checked for. This is the one that converts a bounded job into an unbounded one. The fetch loop runs beautifully across two hundred pages on your laptop, and somewhere past page nine hundred the responses turn into challenge pages. Now the job needs residential or mobile addresses, a real browser it never needed before, and a crawl rate a fifth of what you planned. A run cost you priced at zero suddenly has a monthly invoice on it.

The long tail. Eighty percent of records go down one path and the remainder each want their own branch. An out of stock product with no price element at all. An address that arrives as one line of free text where the others give you five fields. A record in a second language. A page on a completely different template because that catalogue came in through an acquisition and never got merged. Twenty minutes each, forty of them, and none of it visible on the first page you opened.

The maintenance tail. You deliver, you invoice, you leave. Three weeks later the price column is blank and you are back in that repository for free, because the quote said “scrape this site” and never said for how long.

The hour before the number

Here is the inspection I run now before any figure leaves my mouth. It takes about an hour.

Pull ten pages by hand. No code, no crawler. Open them in a browser, read them, save the HTML to disk. Ten pages will tell you whether the data is in the HTML at all, whether the layout is stable across the ten, and whether page seven is the awkward one that breaks the assumption you were about to price.

Open the network tab before writing a single selector. If the page is filling itself from a JSON endpoint, the job just became several times cheaper and considerably more stable, and you want that knowledge before you price a parser rather than after. Give it twenty minutes. The endpoint is either there or it is not, and if it is not you go back to reading HTML and quote on that basis.

Make an account. This is the biggest hidden multiplier I know of, and checking costs two minutes. A login means sessions, session expiry, cookie handling per worker, rate limits counted per account instead of per address, and a real chance the site bans the account and takes the crawl down with it. It also drags terms of service into a conversation you were treating as routine.

Count the page templates. Actually count them, by clicking through twenty or thirty records and tallying what comes back. One template means the job is what you assumed. Six templates means six parsers and six sets of test fixtures, which is a different price rather than a slightly larger one.

Then ask the question people forget: how fresh does this data have to be.

Freshness is the architecture, and the architecture is the price

Same site. Same eight columns in the output file. Four completely different systems depending on the answer to that one question.

Monthly is a script and a line in cron. One person runs it, glances at the file, sends it.

Daily starts wanting retries, a place for failed pages to land, and a human who notices when a run does not finish.

Hourly means a queue, workers, deduplication and monitoring that actually pages someone, because a silent failure at hourly cadence produces twenty four bad files before anybody opens one.

Near real time is a separate discipline. Open connections, latency budgets, infrastructure that has to stay up rather than a job that has to finish.

The spread between the ends of that range is something like ten to one on price, for identical fields off an identical site. Which is why “we want it live” deserves a follow up question about what live means in hours. Half the time the client says live and means daily, and you have handed them most of their budget back before writing a line of code.

Two lines on the quote, always

Most of this is a commercial problem wearing a technical costume.

Quote the build as a build. A fixed number of hours to deliver a working collector against the site as it stands on the date of signing, an agreed field list, and an agreed sample that has to parse correctly before the job counts as done.

Quote the run as its own monthly line. Maintenance hours, proxy spend, the time a template repair takes, and a response time you will genuinely honour at 2am on a Tuesday.

If the client only wants the build, sell them the build, and put in writing that the collector stops working on the day the site changes and that repairs are billable. That single sentence has saved me more money than any code I have written.

A scraper is a subscription to somebody else’s website. You have no contract with the site owner, they do not know you exist, and they will redesign whenever their marketing team wants a new hero section. Every one of those redesigns lands on your desk.

Charge for the looking

Here is the position I get argued with about: quoting a fixed price for a site you have never opened is a way to lose money, and it feels safe only because the loss shows up months after the invoice clears.

So I bill for the inspection. A small fixed fee, an hour or two of work, and a written finding at the end. Where the data actually lives. How many page shapes exist. What the anti bot posture looks like. What each freshness option would cost to build and to run.

Then the build quote is a number I believe. And if the site turns out to be horrible, the client paid a couple of hundred dollars to learn that instead of several thousand to discover it slowly through me.

A client who refuses the inspection fee has told you something useful. Usually that four other quotes are on the table and the decision is price alone, which means the winner will be whoever understood the job least.

The one still costing me money

The login miss is the story I tell most often, but it is not the mistake that has cost me the most.

That one is a collector I delivered in early 2024 for a single fixed number, with no run line and no end date on my obligation. It is still running. I have repaired it four times since, twice for a template change and twice when the site tightened its rate limiting. None of those repairs were billed, because nothing in what I wrote gave me the standing to bill them.

The build was maybe fifteen hours. The unpaid tail is past twenty and still counting, and the only reason it has not gone further is that the client is pleasant and the site is boring.

The fix was one paragraph of scope language, and it took me an embarrassingly long time to write it.

Where an hour of looking runs out

An inspection does not make the estimate accurate. It makes it honest, which is a lower bar and a far more useful one. A site can still add a challenge page in month three, or migrate to a framework that paints everything client side, and nothing you observed in August tells you what it does in November.

Inspection also says nothing about whether the job is one worth taking. Public pages only, the robots file honoured, a crawl rate that does not hurt the site, and an official API or a bulk feed preferred every single time one exists. A decent share of the enquiries I get fail that test long before the estimate matters, and the right answer to those is a polite no.

I quote these jobs and then live inside them, so this is the checklist I actually run rather than something copied off a consultancy site. The rest of what I have written on scraping cost, pipeline design and the proxy infrastructure underneath it is over here.

Get new guides and videos first — join the Telegram channel.