firecrawl vs crawl4ai vs jina reader: which llm scraping tool in 2026?

firecrawl is a hosted scraping api that returns clean markdown with no infra. crawl4ai is a self-hosted python library that does the same job locally with a real chromium browser. jina reader is a free public endpoint that converts any url to llm-ready text via simple prefix. firecrawl wins on speed-to-production. crawl4ai wins on cost at scale. jina reader wins on simplicity for prototypes.

three tools, three different bets on what an llm scraping stack should look like. they all ship today, they all output clean markdown, and they all integrate with langchain and llamaindex out of the box. but the moment you push past a thousand urls, the cost and reliability tradeoffs diverge fast. this comparison breaks down where each one fits.

the short version

if you’re building an mvp this week, jina reader. if you’re shipping a product to production this month, firecrawl. if you’re scraping six figures of urls a month, crawl4ai self-hosted.

what each tool actually is

firecrawl, by mendable.ai, is a hosted api. you send a post request with a url, you get markdown back. it handles javascript rendering, anti-bot evasion, and rate limiting on its end. the firecrawl pricing page lists tiers from free (500 credits) up to enterprise. each url with javascript counts as five credits, plain html as one.

crawl4ai is an open-source python library. apache 2.0, hosted at github.com/unclecode/crawl4ai. you run it on your own machine, your own server, your own kubernetes cluster. it ships with playwright internally and outputs llm-friendly markdown by default. for a step-by-step walkthrough, see the crawl4ai tutorial.

jina reader is the simplest of the three. you prepend https://r.jina.ai/ to any url and you get back the rendered content as plain text. there’s a python sdk and a paid tier with higher limits, but the core endpoint works without an api key.

installation and time-to-first-scrape

speed of setup matters when you’re evaluating tools. here’s what each looks like cold.

firecrawl:

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-your-key")
result = app.scrape_url("https://example.com")
print(result["markdown"])

sign up, paste a key, three lines of code. about 90 seconds end-to-end.

crawl4ai:

pip install -U crawl4ai
crawl4ai-setup

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown)

asyncio.run(main())

the setup downloads playwright browsers, which on a typical laptop takes 2-3 minutes the first time. after that, scraping is instant.

jina reader:

curl https://r.jina.ai/https://example.com

that’s the whole api. one curl command, no signup, no install. for higher rate limits you pass an Authorization: Bearer <key> header.

output quality on real pages

the marketing pages all promise clean markdown. real websites don’t always cooperate. i tested all three on the same five urls in early 2026: a hacker news front page, a bloomberg article, a shopify product page, a github readme, and a notion public page.

hacker news. all three returned readable markdown. firecrawl and crawl4ai preserved the rank numbers. jina reader dropped them. minor difference unless you’re parsing the structure.

bloomberg article. the paywall warning. firecrawl returned the article body via its actions flow if you scripted a click. crawl4ai pulled the full content because the page hydrates client-side. jina reader returned the paywall stub only. crawl4ai’s edge here comes from its real browser context.

shopify product. all three handled the dynamic price and variant rendering. firecrawl’s output was the most concise. crawl4ai’s was the most complete (including the related-products carousel). jina reader sat in the middle.

github readme. identical output across all three. these are static markdown anyway.

notion public page. crawl4ai and firecrawl both returned the full content. jina reader timed out twice on a 50-block page during testing.

verdict: firecrawl and crawl4ai are roughly tied on quality. jina reader is good but trips on heavy spas.

pricing breakdown for 2026

at low volume, jina reader wins on cost (free or near-free for prototyping). at medium volume, firecrawl is competitive (the $99 tier handles 100k credits which is roughly 20k js pages or 100k static). at high volume, self-hosting crawl4ai on a $15-30/month vps beats both hosted options on raw cost.

the catch with self-hosting: you’re paying with your time. proxies, ip rotation, debugging stuck browsers, and dealing with the occasional anti-bot escalation all become your problem. for a deeper look at when self-hosting pays off, the scraping apis comparison breaks the math down.

javascript and dynamic content

all three tools render javascript. how well they do it varies.

firecrawl uses a custom browser pool with built-in stealth patches. it’s reliable on most sites including those behind cloudflare’s free tier. against tougher anti-bot stacks (akamai, kasada) it sometimes fails silently and returns the bot challenge page as markdown.

crawl4ai uses playwright with chromium. you can swap to chromium-stealth, firefox, or webkit. you can attach proxies, persistent profiles, and custom user agents. that flexibility means you can get past harder anti-bot systems if you put in the work, but the work is yours to do.

jina reader uses its own renderer. it handles most spas correctly but doesn’t expose any configuration. there’s no proxy option, no header customization, no waiting strategy. what you get is what jina decided is good enough.

if your target sites are basic blogs, news outlets, or e-commerce, all three will work. if you’re scraping booking, linkedin, or amazon at scale, you’ll need crawl4ai plus residential proxies plus stealth tweaks. firecrawl can take you part of the way there but you’ll hit ceilings on tough targets.

structured extraction

raw markdown is fine for rag. structured data is what you actually want most of the time. price, name, sku, author, date.

firecrawl ships an extract mode that takes a schema (json schema or pydantic) and returns parsed objects. it uses an llm internally so it’s pay-per-token plus the credit cost. accuracy on well-defined fields is around 95% in my testing.

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-your-key")
result = app.scrape_url(
    "https://shop.example.com/product/123",
    {"formats": ["json"], "jsonOptions": {"schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "price": {"type": "number"},
            "in_stock": {"type": "boolean"},
        },
    }}},
)
print(result["json"])

crawl4ai gives you two paths: a deterministic css extraction strategy (free, fast, brittle to layout changes) and an llm extraction strategy (you supply the api key and pay your own llm bill). the css route is the production sweet spot.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

schema = {
    "name": "product",
    "baseSelector": "div.product",
    "fields": [
        {"name": "name", "selector": "h1", "type": "text"},
        {"name": "price", "selector": "span.price", "type": "text"},
    ],
}
cfg = CrawlerRunConfig(extraction_strategy=JsonCssExtractionStrategy(schema))

jina reader has no structured extraction. you get markdown only. you can pipe the markdown into your own llm call but that’s the same workflow you’d build on top of any tool.

anti-bot, proxies, and stealth

this is where the gap opens widest.

firecrawl runs from a managed pool. you don’t choose the ip. you don’t choose the geolocation. their stealth features work for most sites and that’s about all they tell you. for sites that block their pool, you’re stuck.

crawl4ai accepts any proxy you give it. residential, mobile, datacenter, your own raspberry pi at home. the residential proxy primer covers the differences. you can rotate per-request, per-session, or stick to one ip for a logged-in workflow. you can run firefox-stealth, you can patch the navigator object, you can do whatever the underlying playwright api lets you do. that’s a lot.

jina reader has no proxy or stealth controls. it works or it doesn’t.

ecosystem and llm framework support

all three are well-supported in the python rag ecosystem. firecrawl and jina also have node.js sdks that are first-class. crawl4ai is python-only.

where each tool wins

firecrawl wins when: you’re shipping a rag product, you have a budget for tooling, you want zero infrastructure, and your urls are mostly mainstream sites. it’s the path of least resistance for getting from idea to production.

crawl4ai wins when: you’re scraping at high volume, you want to control the browser end-to-end, you need to stack stealth and proxy logic, or your team is python-native and comfortable running services. cost-wise it’s unbeatable past about 50k pages a month.

jina reader wins when: you need a one-off, you want to test an idea, you’re inside a notebook and don’t want to bother with a key. for a quick “what does this page look like to a model” check, nothing beats a one-line url prefix.

a hybrid stack that works

most teams i’ve seen ship something like this. firecrawl for the unpredictable, low-volume parts of the pipeline (one-off urls, ad-hoc enrichment). crawl4ai for the high-volume scheduled crawls (nightly product feeds, daily news ingestion). jina reader for ide-level prototyping and quick checks.

you don’t have to pick one. the python pattern is simple:

import os, requests
from crawl4ai import AsyncWebCrawler
from firecrawl import FirecrawlApp

async def fetch_markdown(url, mode="auto"):
    if mode == "fast":
        r = requests.get(f"https://r.jina.ai/{url}")
        return r.text
    elif mode == "managed":
        return FirecrawlApp().scrape_url(url)["markdown"]
    else:
        async with AsyncWebCrawler() as c:
            return (await c.arun(url=url)).markdown

route by url, route by volume, route by reliability requirement. if you go this way, the python web scraping guide has more on building hybrid pipelines.

faq

which is fastest, firecrawl or crawl4ai?
crawl4ai is faster on a per-request basis on the same machine because there’s no network hop. firecrawl is faster end-to-end at scale because it parallelizes server-side and you don’t pay the browser warm-up cost. for sub-second single-url latency, crawl4ai with a hot browser wins.

is jina reader really free?
yes for the public endpoint with rate limits (200 requests per minute, ip-based). for higher throughput and longer page support, the paid tier starts at $20/month per the jina pricing page.

can firecrawl scrape behind a login?
yes. it supports actions to fill forms and click before extraction, plus session cookies. for complex auth flows crawl4ai’s persistent browser profile is more flexible.

which is best for rag?
all three feed clean markdown to a vector store. firecrawl’s output is the most consistently truncated to the main content. crawl4ai with the PruningContentFilter is comparable. jina reader is fine but a bit noisier on long pages.

does crawl4ai bypass cloudflare?
sometimes, with the right proxy and stealth config. firecrawl handles cloudflare’s basic challenges out of the box. for harder anti-bot systems neither tool works without manual help.

which one supports browser-use or agent integration?
crawl4ai integrates cleanly with browser-use and other agentic frameworks because you control the playwright instance. firecrawl exposes an extract agentic api but with less control.

conclusion

firecrawl is the right answer for most teams shipping rag in 2026 because the time saved on infra outweighs the api cost. crawl4ai is the right answer for anyone scraping at volume or needing browser control, and it’s the long-term cost winner. jina reader is the right answer for prototypes and single-url fetches.

pick the one that matches your bottleneck. if your bottleneck is engineering time, pay for firecrawl. if your bottleneck is per-page cost, run crawl4ai. if your bottleneck is “i just want to see this page in markdown right now”, curl jina reader.

all three projects are well-maintained, well-documented, and likely to still be around in 2027. you can switch later. start with the one that gets you scraping today.

how to use crawl4ai for llm-ready web scraping (python tutorial 2026)

crawl4ai is an open-source python library that turns any webpage into clean, llm-ready markdown in one async call. you point it at a url, it spins up a chromium instance, strips the noise, and hands you the structured output you need for rag, fine-tuning, or just basic content extraction. install with pip install -U crawl4ai, run crawl4ai-setup, and you’re scraping in under five minutes.

most python scrapers were built before language models existed. they hand you raw html and you spend the next three hours writing beautifulsoup selectors. crawl4ai flipped that workflow. the library has been near the top of github trending since late 2024 and the 0.5.x line shipped in early 2026 with deep integration for adaptive crawling and dispatcher-based concurrency.

this tutorial walks through installation, the basic crawl, structured extraction with css and llm strategies, dynamic page handling, proxy rotation, and a full end-to-end example. if you’ve used playwright before, this will feel familiar. if you haven’t, that’s fine too. crawl4ai handles the browser plumbing for you.

why crawl4ai instead of beautifulsoup or scrapy

scrapy is still the right pick if you’re crawling millions of pages and you have time to write spiders. beautifulsoup is fine for static html. crawl4ai sits in a different lane. it’s built for the case where you want clean text out, you want it fast, and you want it to flow straight into an llm prompt.

three things make it different from older tools.

first, the default output is markdown, not html. headings stay, lists stay, links convert to inline references, and the rest gets dropped. you don’t write a single selector to get readable text.

second, it ships with a real browser by default. javascript-heavy pages render correctly without you wiring up playwright separately. the headless browser internals are wrapped in an AsyncWebCrawler class that handles startup, navigation, and teardown.

third, it’s async-first. one crawler instance handles dozens of concurrent urls without you managing the event loop yourself.

installing crawl4ai

the install is two commands. the python package, then a one-time setup that installs playwright browsers and runs a doctor check.

pip install -U crawl4ai
crawl4ai-setup

if the setup script throws errors about missing system libraries on linux, run the diagnostic:

crawl4ai-doctor

on macos and windows the playwright install usually just works. on a fresh ubuntu container you may need apt-get install -y libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libpango-1.0-0 libcairo2 libasound2 before the browsers will launch.

verify the install:

python -c "import crawl4ai; print(crawl4ai.__version__)"

you should see 0.5.x or higher. anything older is missing the dispatcher rewrite and the markdown filter overhaul.

your first crawl

the simplest possible script. one url in, markdown out.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown[:2000])

asyncio.run(main())

run that. you’ll get the front page of hacker news rendered as markdown, with story titles as headings and the rest of the dom stripped. no selectors, no parsing, no cleanup pass.

the result object holds more than markdown. it also has result.html (the raw rendered html), result.cleaned_html (post-filter), result.media (a dict of images, audio, video extracted from the page), result.links (internal and external link lists), and result.metadata (title, description, og tags).

controlling the crawl with browser and run configs

the defaults are sensible but you’ll outgrow them fast. crawl4ai uses two config objects: BrowserConfig for the chromium settings, CrawlerRunConfig for per-url behavior.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_cfg = BrowserConfig(
        headless=True,
        viewport_width=1920,
        viewport_height=1080,
        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36",
    )

    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        wait_until="networkidle",
        page_timeout=30000,
        screenshot=True,
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com/dynamic-page",
            config=run_cfg,
        )
        print(result.markdown)
        if result.screenshot:
            with open("page.png", "wb") as f:
                import base64
                f.write(base64.b64decode(result.screenshot))

asyncio.run(main())

wait_until="networkidle" is critical for spas. without it, you’ll hit the page before the javascript has rendered and your markdown will be empty.

cache_mode=CacheMode.BYPASS forces a fresh fetch. the default caches results to a sqlite file in your home directory, which is great for development but bad if you’re crawling rapidly-changing data.

extracting structured data with css selectors

raw markdown is great for rag pipelines. for everything else you usually want fields. product price, author name, publish date. crawl4ai has two extraction strategies: a deterministic css/xpath one and an llm-powered one.

css first. it’s fast, free, and predictable.

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

schema = {
    "name": "hn_stories",
    "baseSelector": "tr.athing",
    "fields": [
        {"name": "title", "selector": "span.titleline > a", "type": "text"},
        {"name": "url", "selector": "span.titleline > a", "type": "attribute", "attribute": "href"},
        {"name": "rank", "selector": "span.rank", "type": "text"},
    ],
}

async def main():
    cfg = CrawlerRunConfig(
        extraction_strategy=JsonCssExtractionStrategy(schema, verbose=False),
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com", config=cfg)
        stories = json.loads(result.extracted_content)
        for s in stories[:5]:
            print(s)

asyncio.run(main())

the schema describes what to extract. baseSelector finds repeated rows. each row gets the listed fields pulled out. you get json back, ready for a database or a csv.

extracting with an llm when selectors won’t hold

some sites change layouts often. some pages have data scattered in prose. that’s where the llm strategy earns its keep.

import asyncio
import os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel

class Article(BaseModel):
    headline: str
    author: str
    published_date: str
    summary: str

async def main():
    llm_cfg = LLMConfig(
        provider="openai/gpt-4o-mini",
        api_token=os.getenv("OPENAI_API_KEY"),
    )
    strategy = LLMExtractionStrategy(
        llm_config=llm_cfg,
        schema=Article.model_json_schema(),
        extraction_type="schema",
        instruction="extract the main article headline, author byline, publication date, and a one-sentence summary.",
    )
    cfg = CrawlerRunConfig(extraction_strategy=strategy)

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.theverge.com/some-article-url",
            config=cfg,
        )
        print(result.extracted_content)

asyncio.run(main())

gpt-4o-mini is cheap, fast, and accurate enough for most extraction work. the openai pricing page lists current per-token costs. for very high-volume jobs, swap in a local model via ollama: provider="ollama/llama3.1:8b".

a word of caution. llm extraction is non-deterministic. for production pipelines on stable sites, use css selectors. reserve the llm path for messy sources and one-off jobs.

handling javascript, infinite scroll, and clicks

modern sites love to hide their content behind scroll triggers and click handlers. crawl4ai exposes js_code and wait_for for this.

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

scroll_js = """
(async () => {
    for (let i = 0; i < 5; i++) {
        window.scrollTo(0, document.body.scrollHeight);
        await new Promise(r => setTimeout(r, 1500));
    }
})();
"""

async def main():
    cfg = CrawlerRunConfig(
        js_code=scroll_js,
        wait_for="css:.product-card:nth-child(50)",
        page_timeout=60000,
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example-shop.com/category",
            config=cfg,
        )
        print(f"loaded {result.markdown.count('product-card')} product cards")

asyncio.run(main())

the js scrolls five times, the wait_for blocks until the 50th product card appears, and only then does the markdown extraction run.

adding proxies for blocked sites

once you scale past a few hundred requests, you’ll start hitting rate limits and ip blocks. crawl4ai accepts a proxy in the browser config.

from crawl4ai import BrowserConfig

browser_cfg = BrowserConfig(
    headless=True,
    proxy_config={
        "server": "http://proxy.example.com:8080",
        "username": "user",
        "password": "pass",
    },
)

for rotating residential proxies you’ll typically point at a single endpoint that rotates the exit ip on every request. mobile proxy networks like singapore mobile proxy work the same way. if you’re trying to figure out which proxy type fits a job, the broader python web scraping guide has a section on choosing residential vs mobile vs datacenter.

crawling many urls in parallel

the arun_many method takes a list and a dispatcher. the memory-adaptive dispatcher is the sane default.

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher

urls = [f"https://example.com/page/{i}" for i in range(1, 101)]

async def main():
    dispatcher = MemoryAdaptiveDispatcher(
        memory_threshold_percent=80.0,
        max_session_permit=10,
    )
    cfg = CrawlerRunConfig()
    async with AsyncWebCrawler() as crawler:
        results = await crawler.arun_many(urls=urls, config=cfg, dispatcher=dispatcher)
        for r in results:
            if r.success:
                print(r.url, len(r.markdown))

asyncio.run(main())

the dispatcher caps concurrent sessions at 10 and pauses if memory crosses 80%. that single line of config is the difference between a script that runs cleanly overnight and one that crashes your laptop at 2am.

cleaning the markdown for llm input

the default markdown is good. it’s not perfect. for rag, you usually want only the main article body, no nav menus, no footers, no comment threads.

crawl4ai exposes a content filter for exactly that.

from crawl4ai import CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

md_gen = DefaultMarkdownGenerator(
    content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed"),
)
cfg = CrawlerRunConfig(markdown_generator=md_gen)

the pruning filter scores each block by text density and link ratio, drops anything below the threshold, and gives you a tighter result.markdown.fit_markdown field optimized for embedding.

a complete production-ready example

putting it together. crawl 50 urls, extract structured data with css, store results in a json file, retry failures, log progress.

import asyncio
import json
from pathlib import Path
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher

SCHEMA = {
    "name": "products",
    "baseSelector": "div.product",
    "fields": [
        {"name": "name", "selector": "h2.product-name", "type": "text"},
        {"name": "price", "selector": "span.price", "type": "text"},
        {"name": "stock", "selector": "span.availability", "type": "text"},
    ],
}

URLS = [f"https://shop.example.com/category/page-{i}" for i in range(1, 51)]

async def main():
    browser_cfg = BrowserConfig(headless=True)
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=JsonCssExtractionStrategy(SCHEMA),
        wait_until="networkidle",
        page_timeout=30000,
    )
    dispatcher = MemoryAdaptiveDispatcher(max_session_permit=8)

    out = Path("products.jsonl")
    failures = []

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        results = await crawler.arun_many(urls=URLS, config=run_cfg, dispatcher=dispatcher)
        with out.open("w") as f:
            for r in results:
                if r.success and r.extracted_content:
                    products = json.loads(r.extracted_content)
                    for p in products:
                        p["source_url"] = r.url
                        f.write(json.dumps(p) + "\n")
                else:
                    failures.append(r.url)

    print(f"saved {sum(1 for _ in out.open())} products. failures: {len(failures)}")

asyncio.run(main())

that script is the skeleton of a real production scraper. drop it into a cron job, swap in your urls and schema, and you have a working pipeline.

faq

is crawl4ai free?
yes. it’s mit-licensed and open-source. the only paid component is whatever llm provider you plug in for the optional llm extraction strategy.

does crawl4ai bypass cloudflare?
not by default. it uses standard chromium. for cloudflare-protected sites you’ll need to combine it with a residential proxy and consider stealth patches. the cloudflare turnstile bypass guide covers the techniques.

can crawl4ai scrape behind a login?
yes. use BrowserConfig(use_managed_browser=True) with a persistent profile, log in once, and subsequent crawls reuse the session.

how is crawl4ai different from firecrawl?
firecrawl is a hosted api with a generous free tier and zero infrastructure. crawl4ai is a python library you run yourself. for a side-by-side, see the firecrawl vs crawl4ai vs jina comparison.

what python versions does crawl4ai support?
3.10 and above. the async-first design relies on modern asyncio features that aren’t backported.

can i use crawl4ai with rag frameworks like langchain?
yes, and it’s actually the killer use case. the markdown output drops straight into a langchain document loader. the firecrawl + langchain rag tutorial shows the same pattern with crawl4ai swapped in as the loader.

conclusion

crawl4ai is the cleanest way to get llm-ready text out of the modern web with python. one async call, one markdown blob, no selector pain unless you want it. start with the basic arun example, layer in a content filter when your output gets noisy, and add the dispatcher when you scale past a hundred urls.

the official repo at github.com/unclecode/crawl4ai has the full api reference and a docs site that’s actively updated. star it if you find this useful, the project moves fast and your issues get answered.