Firecrawl measured: what N=120 against six real targets actually says

My first run scored Firecrawl at 53.3%. That number was wrong by nearly fifty points, and every one of the reasons was mine, not theirs.

I want to open with that because it is the useful part of this write-up. The success rate at the bottom is one provider, on one date, from one connection. The five ways a test harness can quietly libel a tool are general.

Disclosure first

Firecrawl gave me API credits to run this. That buys sample size and nothing else. No review of the draft, no say in the verdict, no agreed placement. I asked for the credits because the free tier is 1,000 credits a month and this run does not fit inside that, for reasons that turn out to be worth a section of their own.

The account is on the free plan otherwise. Version tested: the v2 API (/v2/scrape), 5 and 6 September 2026, from a business line in Singapore.

The method

Same six categories the earlier comparison on this site used: e-commerce, SERP, social, travel, real estate, business listings. Five URLs per category, thirty URLs, four repeats each. N=120.

Every request is the same call:

{"url": "...", "formats": ["markdown"], "onlyMainContent": true,
 "proxy": "auto", "timeout": 60000, "maxAge": 0}

maxAge: 0 matters. Leave it out and you may be scored on Firecrawl’s cache rather than on a live fetch, which measures their storage layer and not their ability to get through a door.

A response counts as a success only if the markdown contains a marker that could not appear on a block page or an error page: “add to cart” for a product page, “guest reviews” for a hotel, a follower count for a profile. Length alone is not enough. A Cloudflare interstitial is a perfectly well-formed 200 with several thousand characters in it.

The full target list and the raw per-request JSONL are available on request. Ask and I will send them, then re-run it and argue with me.

The five bugs, because they are the transferable part

Rate limiting read as failure. The free plan caps at 10 requests a minute. My first pass fired faster than that, and 8 of 30 requests came back as rate-limit errors. Naively bucketed, that scored X.com at 0% — a category “failure” that was entirely me holding the throttle down. Any benchmark that does not say what its request rate was, and what it did with 429s, is not telling you about the provider.

A block-detector that matched the content. I flagged blocks by looking for “captcha” and similar strings. One of my SERP targets was a Google search for “captcha solving service”. The query echoes into the page, the page is 39,000 characters of perfectly good results, and my detector called it a block.

The same bug wearing a different hat. Zillow’s sign-in copy contains “Takes just a moment.” My detector was looking for “just a moment”, which is the Cloudflare interstitial title. Another 50,000-character success scored as a block.

Dead targets. Three Amazon ASINs in my list had been delisted and returned Amazon’s real 404 page. One Booking.com slug was gone. Those are 404s from the target, not failures from the provider, and counting them against Firecrawl is just wrong. They now get their own verdict and sit outside the denominator.

Running out of money. The one I did not see coming. Halfway through the final run my account hit zero and the API started returning 402. Those 31 requests went into the results file as errors, and the summary line dutifully reported 73.3%. My own balance, scored as Firecrawl failing to fetch a page. It now stops the run when it sees a 402 and keeps those rows out of the denominator, which is the difference between 73.3% and the 98.9% below.

Fixing the first three, replacing the dead URLs, and taking the 402s out moved the same provider on the same targets from 53.3% to 98.9%. Nothing about Firecrawl changed in between.

The general lesson is not subtle: when a benchmark and a vendor disagree, the benchmark is the thing that has not been tested.

What it actually scored

Read the last two rows before the headline. I ran out of credits at request 90 of 120, so Zillow got 9 of its 20 attempts and Yellow Pages got none at all. Yellow Pages is a gap rather than a result. The honest thing is to leave the cell empty rather than quietly drop the category and report a rounder N.

One failure in 89. A Google SERP that came back as 377 characters when the same URL returned 61,814 characters on its other three attempts. Not a block page, not an error, just a stub with a 200 on it.

That is the failure mode worth caring about, and I want to be honest that I cannot tell you what was in those 377 characters, because my harness recorded the length and threw the body away. It saves failure bodies now. That is the kind of thing you only discover by having a failure.

The pricing thing nobody mentions

Credits are not requests. proxy: "auto" escalates to a stealth proxy when a target pushes back, and a stealth fetch bills at several credits instead of one. Two runs, two very different rates. A mixed run of about 102 requests across all six categories took my balance from 1,092 to 677, near enough four credits a request. The final run was weighted towards the hard end (twenty Amazon, twenty X.com, twenty Booking.com) and burned the remaining 677 credits in 89 requests. That is 7.6 each.

So on defended targets the 1,000-credit free tier is worth somewhere between 130 and 250 scrapes, depending entirely on what you point it at. The docs do say stealth costs more. The gap is that the figure you actually burn lands at several times the figure you budgeted, and you cannot work out which multiple in advance because the API does not tell you per request what it charged you. Every comparison table I have seen, including the one on this site, quotes list price per request as if the multiplier were not there.

If you want predictable billing, pin proxy: "basic" and accept the failures. If you want the success rate, budget 4x.

Limits of this run

N=120 is small. It is one provider with no control arm running beside it, so “98.9%” is a statement about these thirty URLs on these two days and not a ranking. Free-tier concurrency is 2, so this says nothing about behaviour at fifty parallel requests, which is where managed scrapers usually start to hurt. Everything ran from one Singapore connection; a US or EU egress may see different treatment on the same targets.

I have not tested the crawl or extract endpoints at all. Only /v2/scrape.

The number I would actually watch

Not the success rate. The spread.

The same Google SERP URL, four consecutive fetches, minutes apart:

Every one of those is scored a success except the 377. A page can be a third of its own size on the next call and still tick every box a success check looks for. Latency moved too, though far less dramatically: median spread within a URL was 1.42s, worst was 7.60s.

A provider that returns a full page nine times and a stub on the tenth will pass a single-shot benchmark and then quietly poison a week of your data, because the stub is a 200 and your pipeline has no reason to think anything went wrong. Which is the argument for schema validation on the way in, and the argument against trusting any success rate (mine included) that was measured once per URL.