Firecrawl measured: what N=89 against six real targets actually says

My first run scored Firecrawl at 53.3%. That number was wrong by nearly fifty points, and every one of the reasons was mine, not theirs.

I want to open with that because it is the useful part of this write-up. The success rate at the bottom is one provider, on one date, from one connection. The five ways a test harness can quietly libel a tool are general.

Disclosure first

Firecrawl gave me API credits to run this. That buys sample size and nothing else. No review of the draft, no say in the verdict, no agreed placement. I asked for the credits because the free tier is 1,000 credits a month and this run does not fit inside that, for reasons that turn out to be worth a section of their own.

The account is on the free plan otherwise. Version tested: the v2 API (/v2/scrape), 5 and 6 September 2026, from a business line in Singapore.

The method

Same six categories the earlier comparison on this site used: e-commerce, SERP, social, travel, real estate, business listings. Five URLs per category, thirty URLs, four repeats each: 120 requests planned. 89 were scored, and the results section below says why.

Every request is the same call:

{"url": "...", "formats": ["markdown"], "onlyMainContent": true,
 "proxy": "auto", "timeout": 60000, "maxAge": 0}

maxAge: 0 matters. Leave it out and you may be scored on Firecrawl’s cache rather than on a live fetch, which measures their storage layer and not their ability to get through a door.

A response counts as a success only if the markdown contains a marker that could not appear on a block page or an error page: “add to cart” for a product page, “guest reviews” for a hotel, a follower count for a profile. Length alone is not enough. A Cloudflare interstitial is a perfectly well-formed 200 with several thousand characters in it.

The full target list and the raw per-request JSONL are available on request. Ask and I will send them, then re-run it and argue with me.

The five bugs, because they are the transferable part

Rate limiting read as failure. The free plan caps at 10 requests a minute. My first pass fired faster than that, and 8 of 30 requests came back as rate-limit errors. Naively bucketed, that scored X.com at 0% — a category “failure” that was entirely me holding the throttle down. Any benchmark that does not say what its request rate was, and what it did with 429s, is not telling you about the provider.

A block-detector that matched the content. I flagged blocks by looking for “captcha” and similar strings. One of my SERP targets was a Google search for “captcha solving service”. The query echoes into the page, the page is 39,000 characters of perfectly good results, and my detector called it a block.

The same bug wearing a different hat. Zillow’s sign-in copy contains “Takes just a moment.” My detector was looking for “just a moment”, which is the Cloudflare interstitial title. Another 50,000-character success scored as a block.

Dead targets. Three Amazon ASINs in my list had been delisted and returned Amazon’s real 404 page. One Booking.com slug was gone. Those are 404s from the target, not failures from the provider, and counting them against Firecrawl is just wrong. They now get their own verdict and sit outside the denominator.

Running out of money. The one I did not see coming. Halfway through the final run my account hit zero and the API started returning 402. Those 31 requests went into the results file as errors, and the summary line dutifully reported 73.3%. My own balance, scored as Firecrawl failing to fetch a page. It now stops the run when it sees a 402 and keeps those rows out of the denominator, which is the difference between 73.3% and the 98.9% below.

Fixing the first three, replacing the dead URLs, and taking the 402s out moved the same provider on the same targets from 53.3% to 98.9%. Nothing about Firecrawl changed in between.

The general lesson is not subtle: when a benchmark and a vendor disagree, the benchmark is the thing that has not been tested.

What it actually scored

Read the last two rows before the headline. I ran out of credits at request 90 of 120, so Zillow got 9 of its 20 attempts and Yellow Pages got none at all. Yellow Pages is a gap rather than a result. The honest thing is to leave the cell empty rather than quietly drop the category and report a rounder N.

One failure in 89. A Google SERP that came back as 377 characters when the same URL returned 61,814 characters on its other three attempts. Not a block page, not an error, just a stub with a 200 on it.

That is the failure mode worth caring about, and I want to be honest that I cannot tell you what was in those 377 characters, because my harness recorded the length and threw the body away. It saves failure bodies now. That is the kind of thing you only discover by having a failure.

What it costs, per target

The first version of this section quoted two blended rates: about 4 credits a request on a mixed run, and 7.6 on a run weighted towards Amazon, X.com and Booking.com. Both were measured. Both would have misled you, because a blended rate says nothing about the one target you actually care about, and the 7.6 made Booking.com look expensive when it isn’t.

So I priced each target on its own on 15 September 2026, same call as above, two URLs per category. The v2 response carries a creditsUsed field that my harness wasn’t reading the first time around. I also read the account balance before and after every call. The two agreed on all twelve requests.

A normal /scrape is 1 credit, with proxy: "auto" on. X.com is the outlier at 30, and that is by design: what comes back is structured profile and post data, not a markdown page. Twenty X.com requests at 30 is 600 credits, out of the 677 the hard-weighted run burned. The X.com calls pushed that average to 7.6, not Booking.com.

Two caveats. Two requests per target is a price check, not a benchmark. And both Google requests on the 15th failed outright (“all scraping engines failed”), where the same category went 19 for 20 on 5 and 6 September. I haven’t re-run Google since, so read that as unexplained, not as a trend.

On the free tier’s 1,000 credits a month, that is roughly 1,000 ordinary pages or 33 X.com profiles. If you are budgeting, count your X.com calls separately. Everything else I pointed it at billed flat.

Limits of this run

N=89 is small. It is one provider with no control arm running beside it, so “98.9%” is a statement about these thirty URLs on these two days and not a ranking. Free-tier concurrency is 2, so this says nothing about behaviour at fifty parallel requests, which is where managed scrapers usually start to hurt. Everything ran from one Singapore connection; a US or EU egress may see different treatment on the same targets.

I have not tested the crawl or extract endpoints at all. Only /v2/scrape.

The number I would actually watch

Not the success rate. The spread.

The same Google SERP URL, four consecutive fetches, minutes apart:

Every one of those is scored a success except the 377. A page can be a third of its own size on the next call and still tick every box a success check looks for. Latency moved too, though far less dramatically: median spread within a URL was 1.42s, worst was 7.60s.

A provider that returns a full page nine times and a stub on the tenth will pass a single-shot benchmark and then quietly poison a week of your data, because the stub is a 200 and your pipeline has no reason to think anything went wrong. Which is the argument for schema validation on the way in, and the argument against trusting any success rate (mine included) that was measured once per URL.