how to collect AI training data at scale: scraping, licensing, APIs

AI training data collection is one of the fastest growing use cases for web scraping in 2026. whether you’re fine-tuning a domain LLM, building a vector database for RAG, or training a custom vision model, you need clean, diverse, legally-defensible data at meaningful scale. this guide covers the four sourcing paths (scraping, licensing, APIs, public datasets), the infrastructure decisions, and the legal lines you can’t cross.

the four sources of training data

every AI dataset combines some mix of these. each has trade-offs.

most teams build a base from public datasets, fill gaps with custom scraping, and license sensitive verticals (legal, medical, financial) where ToS issues bite hardest. for the proxy infrastructure side, see proxies for ML and AI training data collection.

start with public datasets

before you scrape a single URL, check if someone already collected what you need. public datasets cover billions of pages and millions of images, all pre-cleaned and deduplicated.

Common Crawl publishes monthly snapshots of ~3-4 billion web pages in WARC format. it’s the foundation of most foundation models (GPT-3, LLaMA, Mistral all used it). access via S3 (s3://commoncrawl/) for free, but bandwidth costs add up.

HuggingFace Hub hosts thousands of curated datasets including FineWeb (15T tokens of filtered Common Crawl), C4 (Google’s cleaned web text), and domain-specific datasets like PubMed, ArXiv, and StackOverflow dumps.

The Pile, RedPajama, Dolma are research-grade open datasets that have already been deduplicated and quality-filtered. start here if you’re training a foundation LLM.

Wikipedia + Wikidata dumps are free, structured, and updated monthly. excellent for factual grounding.

if your needs match these, save yourself months of infrastructure work and just download.

when custom scraping makes sense

scraping is the right choice when:

• you need data that’s fresh (yesterday, not last year’s snapshot)
• you need a niche vertical (specific industry forums, regional news, product reviews)
• you need structured data the public sets don’t preserve (HTML tables, schema.org markup, image alt text)
• the source is a real-time stream (social media, news feeds)

scraping at AI scale means tens of millions to billions of pages. infrastructure decisions compound at that volume.

scraping infrastructure for AI scale

three components define the budget: proxy bandwidth, compute, and storage.

proxies. residential proxies cost $4-15 per GB. at 50KB average per page, 1 billion pages = 50TB = $200K-$750K in proxy bandwidth alone. for AI-scale, you want high-bandwidth datacenter proxies for the easy 70% of pages and reserve residential for the 30% that need it. our provider comparison ranks options by per-GB cost.

compute. Python with asyncio + httpx handles 200-500 pages/second per core. 1 billion pages on a 32-core box at 50% utilization takes about 40 days. faster with distributed runners (Apache Beam, Ray, Apify Actors) at higher cost.

storage. raw HTML is 50-200KB per page. cleaned text is 5-20KB. 1 billion pages of raw HTML = 50-200TB. S3 standard at $23/TB/month = $1,150-$4,600 monthly. cheaper with S3 Glacier or Backblaze B2 for cold storage.

start with smaller batches (10M pages) to validate the pipeline before scaling.

a minimal AI scraping pipeline

import asyncio
import httpx
import json
from pathlib import Path
from urllib.parse import urlparse

async def fetch_one(client, url, sem):
    async with sem:
        try:
            r = await client.get(url, timeout=20)
            return {
                'url': url,
                'status': r.status_code,
                'html': r.text if r.status_code == 200 else None,
                'content_type': r.headers.get('content-type', ''),
            }
        except Exception as e:
            return {'url': url, 'error': str(e)}

async def crawl(urls, output_path, concurrency=100):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(
        proxies={'all://': 'http://user:pass@proxy.example.com:8000'},
        http2=True,
        follow_redirects=True,
    ) as client:
        tasks = [fetch_one(client, url, sem) for url in urls]
        results = []
        for coro in asyncio.as_completed(tasks):
            results.append(await coro)
            if len(results) % 1000 == 0:
                with open(output_path, 'a') as f:
                    for r in results[-1000:]:
                        f.write(json.dumps(r) + '\n')

# usage
urls = Path('seed_urls.txt').read_text().splitlines()
asyncio.run(crawl(urls, 'pages.jsonl'))

JSON Lines is the right format for streaming AI data. one record per line, easy to filter and dedupe with jq or pandas. for the broader Python toolkit, see our Python web scraping guide.

clean the data

raw HTML is useless to a language model. you need to extract text, strip boilerplate, and remove low-quality content.

from trafilatura import extract

def clean(html):
    return extract(
        html,
        include_comments=False,
        include_tables=False,
        no_fallback=False,
    )

trafilatura is the industry standard for HTML to clean text. it removes navigation, ads, footers, and cookie banners while preserving article structure. used by HuggingFace’s FineWeb pipeline.

then filter by quality:
– minimum length (300+ words for LLM training)
– language detection (langdetect or fasttext)
– duplicate detection (MinHash + LSH for fuzzy dedup)
– profanity and PII filters
– model-based quality classifier (FineWeb-Edu uses one)

for very large datasets, deduplication is more important than collection. ~30-50% of any web crawl is duplicate or near-duplicate content.

sourcing options for vertical data

different domains have different rules.

news: NewsAPI, GDELT, Common Crawl news subset, or licensed feeds from Reuters/AP. NYT and WSJ explicitly prohibit AI training. licensing is the safe path. our news APIs comparison covers options.

social media: X/Twitter API ($200K+/year for full firehose), Reddit API ($0.24 per 1k requests), Mastodon (free, smaller volume). don’t scrape social platforms outside their APIs in 2026; both X and Reddit successfully sued multiple AI labs in 2024.

academic: ArXiv (free, ~2.4M papers), PubMed (free, ~36M abstracts), CORE (free, 280M open-access papers). always available via official bulk download endpoints.

code: GitHub via gharchive.org (BigQuery dumps), Software Heritage, StackOverflow data dump. respect the license of each repo; only permissive licenses (MIT, Apache, BSD) are safe for training redistributable models.

legal: licensed databases (Westlaw, LexisNexis) or court PACER data. court records are public but the bulk-access mechanisms are clunky. CourtListener is the best free open dataset.

ecommerce/product: Amazon Product Advertising API for affiliates only, Shopify product sitemaps for store-by-store. mostly need scraping for breadth.

the legal layer

three legal regimes apply to AI training data in 2026.

copyright. training on copyrighted works without permission is contested. the US has had partial fair-use rulings (Authors Guild v. Google, 2015 Google Books). EU has explicit exceptions for “text and data mining” (Article 4 of the DSM Directive) but rights holders can opt out via robots.txt or specific tags. always check the AI-specific signals: ai.txt, robots.txt directives for GPTBot, ClaudeBot, Google-Extended.

terms of service. most websites prohibit AI training in their ToS. enforcement is inconsistent but lawsuits are increasing. NYT v. OpenAI (2023), Getty v. Stability AI (2023), and Reddit v. Anthropic (2025) are the high-profile cases. ignore ToS at your peril.

privacy law. GDPR (EU), CCPA (California), and similar laws restrict processing personal data without lawful basis. scraping public profiles is not automatically GDPR-compliant. PII filters are mandatory if you train on web data and serve EU users.

the cleanest legal posture: train only on (a) public datasets that have explicit AI-training licenses, (b) data you own, (c) data you’ve licensed, or (d) data covered by a clear fair-use argument. consult a lawyer before scraping any commercial site for AI training.

best practices for AI dataset hygiene

most production AI datasets go through 10-20 cleaning stages. each stage drops 5-30% of data. expect your final dataset to be 10-30% the size of raw crawl.

faq

can I just scrape the entire internet for training data?
no. you can scrape sites that allow it (per robots.txt and ai.txt), use public datasets, or license data. wholesale scraping of sites that prohibit it (in ToS or robots.txt) creates significant legal exposure, especially after the 2024-2025 AI lawsuits.

how much data do I actually need?
depends on the model. a 7B-parameter LLM benefits from 1-2T tokens of training data (~10TB cleaned text). a domain fine-tune needs only 100M-1B tokens (1-10GB). a RAG pipeline can work with megabytes if the retrieval is good. start small and scale up only when quality plateaus.

what’s the difference between scraping for AI vs scraping for analytics?
AI scraping prioritizes diversity, deduplication, and quality filtering over completeness. analytics scraping prioritizes structure (extracting specific fields like price or rating) over volume. AI pipelines clean text aggressively; analytics pipelines preserve structure.

should I use a managed scraping API for AI data?
for under 100M pages, yes. Bright Data, Apify, and ScraperAPI handle the proxy and CAPTCHA layer for you. above 100M pages, the math flips and self-hosted with residential proxies becomes cheaper.

do I need residential proxies for AI scraping?
depends on targets. Common Crawl, HuggingFace, ArXiv, and most news APIs work fine with datacenter IPs. Amazon, LinkedIn, Instagram require residential or mobile. our proxies for ML training page covers the typical mix.

how do I handle copyright in training data?
three options: license everything (expensive, clean), restrict to clearly permissive sources (limits scale), or rely on fair-use arguments (legally murky, requires legal counsel). foundation model labs increasingly take option 1 or 2 after the 2024-2025 lawsuits.

conclusion

AI training data collection in 2026 is a multi-layered problem: pick your sources (public, scraped, licensed, API), build infrastructure that scales (proxies, async runners, JSONL storage), clean aggressively (trafilatura, MinHash, quality classifiers), and respect the legal boundaries (robots.txt, ToS, privacy law).

most teams underinvest in cleaning and overinvest in volume. a 100GB dataset of clean, deduplicated, well-balanced text outperforms a 10TB dataset of raw crawl every time. focus on quality from the start.

if you’re building a domain model or RAG system, start with public datasets, fill gaps with targeted scraping, and license sensitive data. that path stays out of legal trouble while giving you 90% of the data quality the big labs have.

scrapegraphai tutorial: ai-powered scraping without selectors (2026)

scrapegraphai is an open-source python library that scrapes any website by describing what you want in plain english. it sends the rendered html to an llm, which extracts structured json without you writing css or xpath selectors. install with pip install scrapegraphai, plug in an openai or local ollama key, point it at a url, and it returns parsed data. it is useful for one-off scrapes, prototype work, and small sites where selectors break weekly.

this tutorial covers install, the four pipeline types, proxy and headless integration, and when to use it versus a traditional scrapy or playwright stack.

what scrapegraphai is

scrapegraphai (github: scrapegraph-ai/scrapegraph-ai) is a graph-based web scraping framework that uses an llm to extract data instead of selectors.

the workflow:

1. you provide a url and a natural-language prompt (“get all product names and prices”).
2. scrapegraphai fetches the page (with optional headless browser).
3. it cleans and chunks the html.
4. an llm parses the chunks into structured json matching your prompt.

no css, xpath, or regex. selector drift on the target site does not break your scraper unless the page structure changes so much the llm cannot find the data.

installation in 2026

pip install scrapegraphai
playwright install chromium

the playwright install is needed for the smart_scraper graph that uses a real browser. for static-html scraping (no js), the playwright step is optional.

set your llm api key as an environment variable:

export OPENAI_API_KEY="sk-..."

scrapegraphai supports openai, anthropic, groq, and local ollama out of the box. for cost-conscious development, use ollama with a 7b model.

your first scrape: smartscraper graph

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "api_key": "sk-...",
        "model": "openai/gpt-4o-mini",
    },
    "verbose": False,
    "headless": True,
}

scraper = SmartScraperGraph(
    prompt="list all article titles and their authors on this page",
    source="https://hnrss.org/frontpage",
    config=graph_config,
)

result = scraper.run()
print(result)

output (truncated):

{
    "articles": [
        {"title": "show hn: a new approach to ai scraping", "author": "alex"},
        {"title": "rust 1.78 released", "author": "rust-lang team"},
        ...
    ]
}

no selectors. the llm read the rendered page and returned what you asked for in json.

the four main graph types

smartscrapergraph

single-page extraction. you give it a url and a prompt, it returns json. this is the workhorse for 80 percent of use cases.

searchgraph

google-search-driven scraping. you give it a query, it searches google, picks top results, and runs smartscraper on each.

from scrapegraphai.graphs import SearchGraph

graph = SearchGraph(
    prompt="find python scraping libraries with examples",
    config=graph_config,
)
result = graph.run()

useful for research-style scraping where you do not have a fixed url list.

speechgraph

extracts data and converts the result to audio via tts. useful for accessibility apps. less commonly used but it ships in the library.

smartscrapermultigraph

batch version of smartscraper. give it a list of urls, run the same prompt against each in parallel.

from scrapegraphai.graphs import SmartScraperMultiGraph

urls = [
    "https://example.com/product/1",
    "https://example.com/product/2",
    "https://example.com/product/3",
]

graph = SmartScraperMultiGraph(
    prompt="extract product name, price, and stock status",
    source=urls,
    config=graph_config,
)
result = graph.run()

the multi-graph is concurrent under the hood. it is the right pick for scraping a list of similar pages.

adding proxies

production scrapers need proxies. scrapegraphai accepts a proxy in the config:

graph_config = {
    "llm": {
        "api_key": "sk-...",
        "model": "openai/gpt-4o-mini",
    },
    "loader_kwargs": {
        "proxy": {
            "server": "http://proxy.example.com:8080",
            "username": "user",
            "password": "pass",
        },
    },
    "headless": True,
}

this passes the proxy to playwright, which routes both the page fetch and any sub-resources through the proxy.

for proxy rotation across many requests, wrap your scrape calls in a loop and switch the config per call. for the full pattern see our python proxy rotation guide.

using local llms with ollama

api costs add up fast on real workloads. each smartscraper run sends the rendered html to the llm, which can be 5,000 to 50,000 tokens. for high-volume scraping, run a local model.

ollama pull llama3.1:8b
ollama serve

then update config:

graph_config = {
    "llm": {
        "model": "ollama/llama3.1",
        "temperature": 0,
        "format": "json",
        "model_tokens": 8192,
        "base_url": "http://localhost:11434",
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": "http://localhost:11434",
    },
    "verbose": False,
}

a 4060ti or m2 max can run llama3.1 8b at usable speed for scraping. the trade-off is extraction quality. gpt-4o-mini is more reliable on messy pages than 8b local models.

for cost-free development and prototyping, ollama is the right choice. for production, gpt-4o-mini at $0.15 per million input tokens is usually cheaper than running a gpu.

handling js-heavy and login-walled sites

smartscraper uses playwright by default with headless: true. for sites that require login, pass cookies via playwright before scraping:

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {"api_key": "sk-...", "model": "openai/gpt-4o-mini"},
    "loader_kwargs": {
        "extra_http_headers": {
            "cookie": "session=abc123; user_token=xyz",
        },
    },
    "headless": True,
}

for sites with strong anti-bot (cloudflare, datadome, perimeterx), pair scrapegraphai with a residential proxy. the llm can still parse the rendered page, but you need a browser the target lets through. see our best web scraping apis comparison for managed options that bundle this.

when to use scrapegraphai vs traditional scrapy

for high-volume production scraping with a stable schema, a hand-coded scrapy spider is still cheaper and more reliable. for quick scrapes, prototypes, or sites where the html structure shifts, scrapegraphai saves significant time.

cost math for openai api

gpt-4o-mini in 2026 is roughly $0.15 per million input tokens and $0.60 per million output tokens.

a typical product-page scrape sends 10,000 input tokens (cleaned html) and outputs 500 tokens (json). cost per page:

• input: 10,000 / 1,000,000 * $0.15 = $0.0015
• output: 500 / 1,000,000 * $0.60 = $0.0003
• total: roughly $0.0018 per page

for 1000 pages, $1.80. for 100,000 pages, $180. budget llm cost into your scrape estimate.

debugging tips

set verbose=True to see every llm call and intermediate output. this is the fastest way to figure out why a prompt is not extracting what you expect.

start prompts simple. “list all product names” works better than a 5-clause instruction with edge cases. add complexity once the basic prompt works.

inspect the cleaned html scrapegraphai sends to the llm. it strips scripts, styles, and a lot of noise. if your target data is in a script tag or rendered late, you may need to pre-render harder before passing to the graph.

for stable schemas, define a pydantic model and pass it as the schema arg. the llm will fill the model exactly, which improves consistency.

from pydantic import BaseModel
from typing import List

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool

class ProductList(BaseModel):
    products: List[Product]

scraper = SmartScraperGraph(
    prompt="extract all products",
    source="https://example.com/shop",
    config=graph_config,
    schema=ProductList,
)

official docs at the scrapegraphai github.

faq

what is scrapegraphai used for?

ai-powered web scraping. you describe what you want in english and an llm extracts structured json from any url, no css or xpath needed.

is scrapegraphai free?

yes, the library is open source. you pay only for the llm api you use (openai, anthropic, etc.). with ollama and a local model, the entire stack is free.

does scrapegraphai handle javascript-rendered pages?

yes. it uses playwright under the hood with headless: true by default. for sites that need to scroll or click before content loads, you can extend the loader to run custom js.

how does scrapegraphai compare to firecrawl?

firecrawl is a managed scraping api. scrapegraphai is a self-hosted python library. firecrawl handles the infra and proxies. scrapegraphai gives you full control and lower cost at scale, but you wire up your own browser and proxies.

can i use scrapegraphai with proxies?

yes. pass proxy details in loader_kwargs.proxy. it routes through playwright. for rotation across many requests, swap the proxy per call or wrap in a custom session pool.

what is the cost per page using scrapegraphai with gpt-4o-mini?

roughly $0.0018 per page for a typical product page (10k input tokens, 500 output tokens). 1000 pages costs about $1.80. for high-volume production, run ollama locally to drop llm cost to zero.

the bottom line

scrapegraphai is the right tool for prototype scrapes, schema-flexible jobs, and sites where selectors break too often to maintain. with gpt-4o-mini, the cost is around $0.002 per page, which beats most managed scraping apis at small scale.

for high-volume production with a stable target, scrapy with proper selectors is still cheaper and faster. but for the long tail of “i need to scrape this once and i do not want to write selectors,” scrapegraphai is the fastest path from url to json in 2026.

start with smartscrapergraph, add proxies once you scale, and switch to ollama if api costs become the bottleneck. the library is actively developed and the api is stable enough to depend on.

How to Use Proxies with Browser-Use (Agentic AI Web Scraping)

browser-use is the python library that lets a language model drive a real chromium browser. it works great out of the box, but the moment you point it at a site that fingerprints aggressively (linkedin, indeed, amazon, facebook), your agent’s session dies in 2-3 page loads. a proxy fixes that. this tutorial shows the working setup in under 200 lines.

what is browser-use

browser-use wraps playwright and exposes a high-level api the llm can call. you give it a goal in natural language (“find the cheapest flight from singapore to tokyo on march 20”), and it clicks, types, scrolls, and extracts. the project is open-source at github.com/browser-use/browser-use and as of may 2026 it sits at version 0.3.x with weekly releases.

if you’ve never used it, our headless browser automation guide covers the chromium fundamentals first.

why you need a proxy with browser-use

three reasons:

(1) your home or datacenter ip gets flagged within minutes on protected sites. browser-use makes thousands of requests per session if the agent is exploring.

(2) geo-restricted content. asking the agent to “compare amazon prices in the us, uk, japan” requires three different residential exits.

(3) parallel agents. running 10 agents from the same ip is the fastest way to a captcha wall.

we benchmarked which proxies actually survive browser-use sessions in our browser-use and operator proxy comparison. short version: residential mobile beats datacenter for protected sites, datacenter is fine for everything else.

installing browser-use

pip install browser-use
playwright install chromium

you need python 3.11+. browser-use uses async, so all examples below run inside asyncio.run(...).

the simplest possible proxy setup

browser-use exposes a BrowserContextConfig that accepts a chromium proxy block. here’s the minimum:

import asyncio
from browser_use import Agent, Browser, BrowserConfig
from langchain_openai import ChatOpenAI

async def main():
    browser = Browser(
        config=BrowserConfig(
            proxy={
                "server": "http://gate.dataresearchtools.com:8000",
                "username": "user-session-abc123",
                "password": "your_password",
            }
        )
    )

    agent = Agent(
        task="go to httpbin.org/ip and tell me the ip you see",
        llm=ChatOpenAI(model="gpt-4o"),
        browser=browser,
    )

    result = await agent.run()
    print(result)
    await browser.close()

asyncio.run(main())

if you see your proxy’s ip in the output, you’re done. if you see your home ip, the proxy block didn’t apply. usually a typo in the server url.

sticky session vs rotating

most residential providers expose two flavors of credentials. a sticky session keeps the same exit ip for a fixed window (10-30 minutes typical). a rotating session swaps the ip on every request.

for browser-use, you want sticky. the agent navigates, clicks, fills forms across multiple pages within a single task. if the ip rotates mid-task, you’ll fail captchas, lose login cookies, and confuse the target site’s rate limiting in ways that look more bot-like, not less.

proxy={
    "server": "http://gate.provider.com:8000",
    # session-id pinned for 30 minutes
    "username": "user-country-us-session-xyz789",
    "password": "your_password",
}

format varies per provider. bright data uses brd-customer-XXX-zone-residential-session-YYY, oxylabs uses customer-USER-cc-us-sessid-XYZ. check your dashboard.

adding country and city targeting

agent tasks often need a specific geo. drop the country code into the username:

proxy={
    "server": "http://gate.provider.com:8000",
    "username": "user-country-jp-city-tokyo-session-abc",
    "password": "your_password",
}

verify with a quick check before the real task:

agent = Agent(
    task="go to ifconfig.co and report the country and city shown",
    llm=ChatOpenAI(model="gpt-4o"),
    browser=browser,
)

if the agent reports tokyo, japan, you’re geo-targeted correctly.

handling auth challenges

some providers require you to whitelist your client ip instead of using user/pass. that breaks if your agent runs from a serverless function with a changing ip. switch to user/pass auth in the dashboard before debugging anything.

if you see a chromium error like ERR_PROXY_CONNECTION_FAILED, the credentials are wrong or your account has zero balance. log into the provider, check the gateway url is the current one, and try again.

per-tab proxy with multi-context

a single agent can run multiple tabs, each with its own proxy. this is how you compare amazon.com vs amazon.co.jp in one task.

from browser_use import Browser, BrowserConfig

browser = Browser(config=BrowserConfig())

us_context = await browser.new_context(
    proxy={"server": "http://gate.provider.com:8000",
           "username": "user-country-us-session-1",
           "password": "pwd"}
)

jp_context = await browser.new_context(
    proxy={"server": "http://gate.provider.com:8000",
           "username": "user-country-jp-session-2",
           "password": "pwd"}
)

then attach each context to its own agent task and run them concurrently with asyncio.gather.

debugging: confirm the proxy is actually used

when nothing seems to work, run this 5-line check first:

import requests

resp = requests.get(
    "https://api.ipify.org?format=json",
    proxies={
        "http": "http://user:pwd@gate.provider.com:8000",
        "https": "http://user:pwd@gate.provider.com:8000",
    },
    timeout=10,
)
print(resp.json())

if requests can hit the proxy and gets back the right ip, the credentials and gateway are correct. then the bug is in your browser-use config, not your proxy account. saves an hour of staring at chromium logs.

rotating ips between tasks (not within a task)

if you want each new agent task to get a fresh ip but keep the ip stable inside the task, increment the session id between runs:

import uuid

def make_proxy():
    return {
        "server": "http://gate.provider.com:8000",
        "username": f"user-session-{uuid.uuid4().hex[:8]}",
        "password": "pwd",
    }

for task in tasks:
    browser = Browser(config=BrowserConfig(proxy=make_proxy()))
    agent = Agent(task=task, llm=llm, browser=browser)
    await agent.run()
    await browser.close()

clean, simple, and survives the longest scraping sessions.

handling captchas

browser-use’s llm tries to solve captchas itself. it fails on hcaptcha and recaptcha v3 most of the time. for production, hand off captchas to a solver:

from browser_use import Agent

agent = Agent(
    task="...",
    llm=llm,
    browser=browser,
    extend_system_message=(
        "if you see a captcha, do not try to solve it. "
        "call solve_captcha(image_url) and wait."
    ),
)

then wire solve_captcha to capsolver or 2captcha as a custom tool. cheaper than burning gpt-4o tokens on a recaptcha grid.

real-world setup for protected sites

linkedin, amazon, indeed, and similar sites profile fingerprints aggressively. residential alone is not enough. the working stack:

• mobile or residential rotating proxy with sticky 10-min sessions
• chromium launched with --disable-blink-features=AutomationControlled
• a real user-agent string that matches the chromium version
• realistic viewport (1920×1080, not the default 1280×720)
• 2-3 second random delays between actions

browser = Browser(
    config=BrowserConfig(
        proxy={...},
        chrome_args=[
            "--disable-blink-features=AutomationControlled",
            "--window-size=1920,1080",
        ],
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    )
)

this combination passes most fingerprint checks in 2026.

frequently asked questions

does browser-use support socks5 proxies?

yes, but with caveats. chromium accepts socks5:// in the server field but ignores user/pass auth on socks5. use http proxies if your provider requires authentication.

can i use free proxies with browser-use?

technically yes, in practice no. free proxies are slow, blocked everywhere worth scraping, and often middlemen. you’ll waste more in llm tokens retrying failed pages than a paid proxy costs.

how much does a browser-use scraping session cost in proxy bandwidth?

a typical 5-minute browsing task uses 50-150mb. residential at $4/gb means 20-60 cents per task. mobile at $8/gb roughly doubles that.

why do my agents get captchas even with residential proxies?

three usual culprits: ip is on a residential pool but the asn looks datacenter, your browser fingerprint is too clean, or you’re hitting the same domain too fast across multiple agents.

can i rotate proxies inside a single task?

you can but you shouldn’t. mid-task ip rotation breaks session cookies and triggers more captchas, not fewer.

what’s the cheapest proxy that works with browser-use?

isp proxies. roughly $1-2/gb, faster than residential, and pass most fingerprint checks except on the most paranoid sites.

final thoughts

a proxy is the smallest config change that doubles a browser-use agent’s survival rate. start with residential sticky sessions, add country targeting when needed, and pre-flight every credential change with the 5-line requests test before you fight chromium. once it works, it works for thousands of tasks.

build a rag data pipeline with firecrawl and langchain (python 2026)

firecrawl crawls a website and returns clean markdown ready for embedding. langchain handles the chunking, embedding, vector storage, and retrieval-augmented question answering. together they let you build a production rag pipeline in under 100 lines of python: scrape a documentation site, chunk and embed the content, store it in a local vector database, and ask questions that get grounded answers with source citations. the full setup runs on your laptop in under 10 minutes.

retrieval-augmented generation is how most production llm apps work in 2026. you don’t ask a model to know everything, you give it the relevant context from your own data. for context built from web sources (docs sites, blogs, knowledge bases, sec filings), the bottleneck is getting clean, structured content out of the web. firecrawl removes that bottleneck.

this tutorial builds a working rag pipeline end-to-end. by the end you’ll have a chatbot that answers questions about any documentation site you point it at, with citations back to source urls.

what you’ll build

a python script that:
1. crawls a target documentation or content site with firecrawl, getting clean markdown
2. chunks the markdown into 500-1000 token segments with langchain
3. embeds each chunk with openai or a local embedding model
4. stores embeddings in chromadb (local) or pinecone/qdrant (cloud)
5. accepts a user question, retrieves the top-k relevant chunks
6. passes those chunks plus the question to a chat model
7. returns the answer with source url citations

real production rag setups are more complex (re-ranking, query rewriting, hybrid search, evaluation) but this skeleton is the foundation everyone builds on.

prerequisites

you need:
– python 3.10+
– a firecrawl api key from firecrawl.dev. free tier gives 500 credits, enough for the tutorial.
– an openai api key, or you can swap in a local model later
– about 10 minutes

pip install firecrawl-py langchain langchain-openai langchain-community langchain-chroma chromadb tiktoken

set environment variables:

export FIRECRAWL_API_KEY="fc-your-key"
export OPENAI_API_KEY="sk-your-key"

step 1: crawl with firecrawl

firecrawl has two relevant endpoints: scrape_url for a single page, and crawl_url for a whole site. for rag, you almost always want the crawl.

import os
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key=os.environ["FIRECRAWL_API_KEY"])

crawl_result = app.crawl_url(
    "https://docs.python.org/3/library/asyncio.html",
    params={
        "limit": 50,
        "scrapeOptions": {
            "formats": ["markdown"],
            "onlyMainContent": True,
        },
    },
    poll_interval=5,
)

pages = crawl_result["data"]
print(f"crawled {len(pages)} pages")
for p in pages[:3]:
    print(" -", p["metadata"]["sourceURL"])

limit: 50 caps the crawl to 50 pages. for prototype work this is plenty. onlyMainContent: True strips navigation, headers, footers, and ad blocks, leaving clean article markdown.

each page in pages has:
– markdown: the cleaned content
– metadata.sourceURL: original url
– metadata.title, metadata.description: page metadata
– metadata.statusCode: http status

if you only need a single page or url list, swap crawl_url for scrape_url in a loop.

step 2: chunk the markdown

you can’t just throw a 50-page document at an embedding model. you need to split it into chunks. langchain’s RecursiveCharacterTextSplitter is the standard choice.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.documents import Document

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    separators=["\n\n", "\n", ". ", " ", ""],
)

docs = []
for page in pages:
    if not page.get("markdown"):
        continue
    chunks = splitter.split_text(page["markdown"])
    for chunk in chunks:
        docs.append(Document(
            page_content=chunk,
            metadata={
                "source": page["metadata"]["sourceURL"],
                "title": page["metadata"].get("title", ""),
            },
        ))

print(f"created {len(docs)} chunks from {len(pages)} pages")

chunk_size=1000 means each chunk is up to 1000 characters (roughly 200-300 tokens). chunk_overlap=150 means each chunk shares 150 characters with the next, so context isn’t lost at chunk boundaries. these are sensible defaults for documentation content. for narrative text or papers, larger chunks (1500-2000 chars) work better.

the separators list tells the splitter where to break, in order of preference. paragraph breaks first, then line breaks, then sentences, then words.

step 3: embed and store

embed the chunks and store them in chromadb. chroma runs locally with no setup.

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = Chroma.from_documents(
    documents=docs,
    embedding=embeddings,
    persist_directory="./chroma_db",
    collection_name="asyncio_docs",
)
print(f"stored {vectorstore._collection.count()} embeddings in chroma")

text-embedding-3-small is openai’s cheap embedding model. $0.02 per million tokens in 2026. for 50 documentation pages with ~50k tokens total, the embedding cost is about $0.001. negligible.

for local embeddings (no api cost, runs on your machine), swap to:

from langchain_huggingface import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")

bge-small-en-v1.5 is a 384-dim embedding model that runs fast on cpu and rivals openai’s small for english docs.

persist_directory="./chroma_db" saves the database to disk so you don’t re-embed on every run.

step 4: build the retrieval qa chain

the retrieval part. langchain has a few patterns for this. the modern lcel (langchain expression language) approach:

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

prompt = ChatPromptTemplate.from_messages([
    ("system", "you are a helpful assistant. answer the question using only the context below. cite sources by url at the end. if the context doesn't have the answer, say so."),
    ("human", "context:\n{context}\n\nquestion: {question}"),
])

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def format_docs(docs):
    return "\n\n".join(
        f"[source: {d.metadata['source']}]\n{d.page_content}"
        for d in docs
    )

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

answer = chain.invoke("what is asyncio.gather and when should i use it?")
print(answer)

k=5 retrieves the top 5 most similar chunks. for documentation sites this is usually enough. for less structured content, k=10 or higher.

temperature=0 keeps the model from hallucinating. for rag, you almost always want low temperature.

the system prompt instructs the model to cite sources by url. you’ll see urls at the end of each answer, traceable back to the original docs page.

step 5: full working script

putting it all together. this is the entire pipeline in one file.

import os
from firecrawl import FirecrawlApp
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

# configuration
TARGET_URL = "https://docs.python.org/3/library/asyncio.html"
CRAWL_LIMIT = 30
COLLECTION = "asyncio_rag"
PERSIST_DIR = "./chroma_db"

def crawl(url, limit):
    app = FirecrawlApp(api_key=os.environ["FIRECRAWL_API_KEY"])
    result = app.crawl_url(url, params={
        "limit": limit,
        "scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True},
    }, poll_interval=5)
    return result["data"]

def to_documents(pages):
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000, chunk_overlap=150,
        separators=["\n\n", "\n", ". ", " ", ""],
    )
    docs = []
    for page in pages:
        if not page.get("markdown"):
            continue
        for chunk in splitter.split_text(page["markdown"]):
            docs.append(Document(
                page_content=chunk,
                metadata={
                    "source": page["metadata"]["sourceURL"],
                    "title": page["metadata"].get("title", ""),
                },
            ))
    return docs

def build_or_load_vectorstore(docs):
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    if os.path.exists(PERSIST_DIR):
        return Chroma(
            collection_name=COLLECTION,
            embedding_function=embeddings,
            persist_directory=PERSIST_DIR,
        )
    return Chroma.from_documents(
        documents=docs,
        embedding=embeddings,
        persist_directory=PERSIST_DIR,
        collection_name=COLLECTION,
    )

def build_chain(vectorstore):
    retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
    prompt = ChatPromptTemplate.from_messages([
        ("system", "you are a helpful assistant. answer using only the context. cite sources by url. if the context doesn't have the answer, say you don't know."),
        ("human", "context:\n{context}\n\nquestion: {question}"),
    ])
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

    def format_docs(docs):
        return "\n\n".join(f"[source: {d.metadata['source']}]\n{d.page_content}" for d in docs)

    return (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt | llm | StrOutputParser()
    )

def main():
    if not os.path.exists(PERSIST_DIR):
        print("crawling and embedding...")
        pages = crawl(TARGET_URL, CRAWL_LIMIT)
        docs = to_documents(pages)
        vs = build_or_load_vectorstore(docs)
        print(f"indexed {vs._collection.count()} chunks")
    else:
        print("loading existing index...")
        vs = build_or_load_vectorstore(None)

    chain = build_chain(vs)

    print("\nask questions. ctrl-c to quit.\n")
    while True:
        try:
            q = input("you: ").strip()
            if not q:
                continue
            print(f"bot: {chain.invoke(q)}\n")
        except KeyboardInterrupt:
            break

if __name__ == "__main__":
    main()

run it:

python rag.py

first run crawls and embeds. subsequent runs reuse the persisted index. you get a working command-line chatbot grounded in your target site’s content.

production considerations

the script above works. it’s not production-ready. things you’d want to add:

re-ranking. the top-5 by vector similarity isn’t always the most relevant 5. add a reranker like cohere rerank or BAAI/bge-reranker-base to reorder retrieved chunks before passing to the llm. dramatic quality improvement.

hybrid search. vector similarity misses keyword-exact matches. add bm25 search alongside vector search and combine results. langchain has EnsembleRetriever for this.

incremental updates. real docs sites change. add a scheduler that re-crawls weekly and updates only changed pages by comparing content hashes.

rate limiting and retries. firecrawl’s free tier is 500 credits. crawls can take minutes. add backoff and retry logic for production reliability.

monitoring. log every query, every retrieved chunk, every llm response. you’ll learn what’s working and what isn’t only by looking at real interactions.

evaluation. create a set of 50-100 reference q&a pairs for your domain. run them through the pipeline weekly. measure accuracy. iterate on chunk size, k, and prompts based on the data.

for the broader landscape of scraping apis you might use instead of firecrawl, see the scraping apis comparison. for the side-by-side with crawl4ai and jina, see firecrawl vs crawl4ai vs jina.

swap-in alternatives

the architecture is modular. each component has alternatives.

crawler: firecrawl (hosted), crawl4ai (self-hosted), jina reader (free public api), playwright (full diy)

chunker: recursivecharacter (default), tokenize-based splitting (more accurate token counts), semantic chunkers (openai’s chunker, llamaindex’s nodeparser)

embedding model: openai text-embedding-3-small ($0.02/1m tokens), text-embedding-3-large ($0.13/1m, better quality), local (free, slower): bge-small, bge-large, voyage embeddings

vector store: chroma (local file-based), qdrant (open source, scalable), pinecone (hosted, premium), pgvector (postgres extension), weaviate, milvus

chat model: gpt-4o-mini (cheap, fast), gpt-4o (better quality), claude 3.7 sonnet (best for reasoning), claude haiku (cheapest), local llama 3.1 via ollama (free, privacy)

mix and match based on cost, latency, privacy, and quality requirements. the langchain abstractions make swapping any one component a 1-2 line change.

cost estimate at scale

for a small team building a docs chatbot:

• firecrawl: $19/month covers the crawl
• embeddings: $0.50 to embed 25m tokens of crawled text (about 5000 docs pages)
• vector store: free if local chroma, $20/month for a hosted starter (qdrant cloud, pinecone)
• llm queries: gpt-4o-mini at $0.15/$0.60 per 1m input/output tokens. with k=5 chunks, average query costs ~$0.001-0.003.

monthly all-in for a chatbot serving 10k queries on a 50-page site: under $50.

faq

why use firecrawl instead of just requests + beautifulsoup?
firecrawl handles javascript-rendered pages, anti-bot challenges, sitemap traversal, and clean markdown extraction in one api call. doing those four things yourself takes weeks.

how big should chunks be for rag?
500-1000 characters (roughly 100-250 tokens) is the sweet spot for most documentation content. larger for narrative or papers, smaller for q&a-style content.

which is better, openai or local embeddings?
openai text-embedding-3-small wins on quality and is cheap enough that cost is rarely the deciding factor. use local embeddings if privacy or air-gapped deployment matters. bge-large is the strongest local option in 2026.

can i use claude instead of gpt-4o-mini for the rag chain?
yes. swap ChatOpenAI for ChatAnthropic in the chain. claude haiku is comparable in cost to gpt-4o-mini. claude sonnet 3.7 produces stronger answers but costs ~5x more.

should i use chroma in production?
chroma is fine for under 1m vectors and a single-process app. for production scale (multiple replicas, millions of vectors, high-throughput retrieval), use qdrant, pinecone, or pgvector.

how do i handle pdf or docx documents in this pipeline?
firecrawl can ingest pdfs directly via scrape_url with formats: ["markdown"]. for docx, use unstructured or langchain_community.document_loaders.UnstructuredWordDocumentLoader. then feed the resulting text into the same chunking and embedding flow.

conclusion

firecrawl plus langchain is the fastest way to build a working rag pipeline in 2026. firecrawl handles the part that’s expensive to build (clean web content extraction). langchain handles the part that’s tedious to write (chunking, embedding, retrieval, prompting).

the script in this tutorial is the skeleton of every web-grounded rag system you’ll see in production. start there, measure quality on your real questions, then add re-ranking, hybrid search, and evaluation as the data demands. for the broader python web scraping foundation, the complete python guide covers everything underneath this stack.

ship a prototype this weekend. iterate on quality next week. that’s the rag playbook.

firecrawl vs crawl4ai vs jina reader: which llm scraping tool in 2026?

firecrawl is a hosted scraping api that returns clean markdown with no infra. crawl4ai is a self-hosted python library that does the same job locally with a real chromium browser. jina reader is a free public endpoint that converts any url to llm-ready text via simple prefix. firecrawl wins on speed-to-production. crawl4ai wins on cost at scale. jina reader wins on simplicity for prototypes.

three tools, three different bets on what an llm scraping stack should look like. they all ship today, they all output clean markdown, and they all integrate with langchain and llamaindex out of the box. but the moment you push past a thousand urls, the cost and reliability tradeoffs diverge fast. this comparison breaks down where each one fits.

the short version

if you’re building an mvp this week, jina reader. if you’re shipping a product to production this month, firecrawl. if you’re scraping six figures of urls a month, crawl4ai self-hosted.

what each tool actually is

firecrawl, by mendable.ai, is a hosted api. you send a post request with a url, you get markdown back. it handles javascript rendering, anti-bot evasion, and rate limiting on its end. the firecrawl pricing page lists tiers from free (500 credits) up to enterprise. each url with javascript counts as five credits, plain html as one.

crawl4ai is an open-source python library. apache 2.0, hosted at github.com/unclecode/crawl4ai. you run it on your own machine, your own server, your own kubernetes cluster. it ships with playwright internally and outputs llm-friendly markdown by default. for a step-by-step walkthrough, see the crawl4ai tutorial.

jina reader is the simplest of the three. you prepend https://r.jina.ai/ to any url and you get back the rendered content as plain text. there’s a python sdk and a paid tier with higher limits, but the core endpoint works without an api key.

installation and time-to-first-scrape

speed of setup matters when you’re evaluating tools. here’s what each looks like cold.

firecrawl:

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-your-key")
result = app.scrape_url("https://example.com")
print(result["markdown"])

sign up, paste a key, three lines of code. about 90 seconds end-to-end.

crawl4ai:

pip install -U crawl4ai
crawl4ai-setup

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown)

asyncio.run(main())

the setup downloads playwright browsers, which on a typical laptop takes 2-3 minutes the first time. after that, scraping is instant.

jina reader:

curl https://r.jina.ai/https://example.com

that’s the whole api. one curl command, no signup, no install. for higher rate limits you pass an Authorization: Bearer <key> header.

output quality on real pages

the marketing pages all promise clean markdown. real websites don’t always cooperate. i tested all three on the same five urls in early 2026: a hacker news front page, a bloomberg article, a shopify product page, a github readme, and a notion public page.

hacker news. all three returned readable markdown. firecrawl and crawl4ai preserved the rank numbers. jina reader dropped them. minor difference unless you’re parsing the structure.

bloomberg article. the paywall warning. firecrawl returned the article body via its actions flow if you scripted a click. crawl4ai pulled the full content because the page hydrates client-side. jina reader returned the paywall stub only. crawl4ai’s edge here comes from its real browser context.

shopify product. all three handled the dynamic price and variant rendering. firecrawl’s output was the most concise. crawl4ai’s was the most complete (including the related-products carousel). jina reader sat in the middle.

github readme. identical output across all three. these are static markdown anyway.

notion public page. crawl4ai and firecrawl both returned the full content. jina reader timed out twice on a 50-block page during testing.

verdict: firecrawl and crawl4ai are roughly tied on quality. jina reader is good but trips on heavy spas.

pricing breakdown for 2026

at low volume, jina reader wins on cost (free or near-free for prototyping). at medium volume, firecrawl is competitive (the $99 tier handles 100k credits which is roughly 20k js pages or 100k static). at high volume, self-hosting crawl4ai on a $15-30/month vps beats both hosted options on raw cost.

the catch with self-hosting: you’re paying with your time. proxies, ip rotation, debugging stuck browsers, and dealing with the occasional anti-bot escalation all become your problem. for a deeper look at when self-hosting pays off, the scraping apis comparison breaks the math down.

javascript and dynamic content

all three tools render javascript. how well they do it varies.

firecrawl uses a custom browser pool with built-in stealth patches. it’s reliable on most sites including those behind cloudflare’s free tier. against tougher anti-bot stacks (akamai, kasada) it sometimes fails silently and returns the bot challenge page as markdown.

crawl4ai uses playwright with chromium. you can swap to chromium-stealth, firefox, or webkit. you can attach proxies, persistent profiles, and custom user agents. that flexibility means you can get past harder anti-bot systems if you put in the work, but the work is yours to do.

jina reader uses its own renderer. it handles most spas correctly but doesn’t expose any configuration. there’s no proxy option, no header customization, no waiting strategy. what you get is what jina decided is good enough.

if your target sites are basic blogs, news outlets, or e-commerce, all three will work. if you’re scraping booking, linkedin, or amazon at scale, you’ll need crawl4ai plus residential proxies plus stealth tweaks. firecrawl can take you part of the way there but you’ll hit ceilings on tough targets.

structured extraction

raw markdown is fine for rag. structured data is what you actually want most of the time. price, name, sku, author, date.

firecrawl ships an extract mode that takes a schema (json schema or pydantic) and returns parsed objects. it uses an llm internally so it’s pay-per-token plus the credit cost. accuracy on well-defined fields is around 95% in my testing.

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-your-key")
result = app.scrape_url(
    "https://shop.example.com/product/123",
    {"formats": ["json"], "jsonOptions": {"schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "price": {"type": "number"},
            "in_stock": {"type": "boolean"},
        },
    }}},
)
print(result["json"])

crawl4ai gives you two paths: a deterministic css extraction strategy (free, fast, brittle to layout changes) and an llm extraction strategy (you supply the api key and pay your own llm bill). the css route is the production sweet spot.

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

schema = {
    "name": "product",
    "baseSelector": "div.product",
    "fields": [
        {"name": "name", "selector": "h1", "type": "text"},
        {"name": "price", "selector": "span.price", "type": "text"},
    ],
}
cfg = CrawlerRunConfig(extraction_strategy=JsonCssExtractionStrategy(schema))

jina reader has no structured extraction. you get markdown only. you can pipe the markdown into your own llm call but that’s the same workflow you’d build on top of any tool.

anti-bot, proxies, and stealth

this is where the gap opens widest.

firecrawl runs from a managed pool. you don’t choose the ip. you don’t choose the geolocation. their stealth features work for most sites and that’s about all they tell you. for sites that block their pool, you’re stuck.

crawl4ai accepts any proxy you give it. residential, mobile, datacenter, your own raspberry pi at home. the residential proxy primer covers the differences. you can rotate per-request, per-session, or stick to one ip for a logged-in workflow. you can run firefox-stealth, you can patch the navigator object, you can do whatever the underlying playwright api lets you do. that’s a lot.

jina reader has no proxy or stealth controls. it works or it doesn’t.

ecosystem and llm framework support

all three are well-supported in the python rag ecosystem. firecrawl and jina also have node.js sdks that are first-class. crawl4ai is python-only.

where each tool wins

firecrawl wins when: you’re shipping a rag product, you have a budget for tooling, you want zero infrastructure, and your urls are mostly mainstream sites. it’s the path of least resistance for getting from idea to production.

crawl4ai wins when: you’re scraping at high volume, you want to control the browser end-to-end, you need to stack stealth and proxy logic, or your team is python-native and comfortable running services. cost-wise it’s unbeatable past about 50k pages a month.

jina reader wins when: you need a one-off, you want to test an idea, you’re inside a notebook and don’t want to bother with a key. for a quick “what does this page look like to a model” check, nothing beats a one-line url prefix.

a hybrid stack that works

most teams i’ve seen ship something like this. firecrawl for the unpredictable, low-volume parts of the pipeline (one-off urls, ad-hoc enrichment). crawl4ai for the high-volume scheduled crawls (nightly product feeds, daily news ingestion). jina reader for ide-level prototyping and quick checks.

you don’t have to pick one. the python pattern is simple:

import os, requests
from crawl4ai import AsyncWebCrawler
from firecrawl import FirecrawlApp

async def fetch_markdown(url, mode="auto"):
    if mode == "fast":
        r = requests.get(f"https://r.jina.ai/{url}")
        return r.text
    elif mode == "managed":
        return FirecrawlApp().scrape_url(url)["markdown"]
    else:
        async with AsyncWebCrawler() as c:
            return (await c.arun(url=url)).markdown

route by url, route by volume, route by reliability requirement. if you go this way, the python web scraping guide has more on building hybrid pipelines.

faq

which is fastest, firecrawl or crawl4ai?
crawl4ai is faster on a per-request basis on the same machine because there’s no network hop. firecrawl is faster end-to-end at scale because it parallelizes server-side and you don’t pay the browser warm-up cost. for sub-second single-url latency, crawl4ai with a hot browser wins.

is jina reader really free?
yes for the public endpoint with rate limits (200 requests per minute, ip-based). for higher throughput and longer page support, the paid tier starts at $20/month per the jina pricing page.

can firecrawl scrape behind a login?
yes. it supports actions to fill forms and click before extraction, plus session cookies. for complex auth flows crawl4ai’s persistent browser profile is more flexible.

which is best for rag?
all three feed clean markdown to a vector store. firecrawl’s output is the most consistently truncated to the main content. crawl4ai with the PruningContentFilter is comparable. jina reader is fine but a bit noisier on long pages.

does crawl4ai bypass cloudflare?
sometimes, with the right proxy and stealth config. firecrawl handles cloudflare’s basic challenges out of the box. for harder anti-bot systems neither tool works without manual help.

which one supports browser-use or agent integration?
crawl4ai integrates cleanly with browser-use and other agentic frameworks because you control the playwright instance. firecrawl exposes an extract agentic api but with less control.

conclusion

firecrawl is the right answer for most teams shipping rag in 2026 because the time saved on infra outweighs the api cost. crawl4ai is the right answer for anyone scraping at volume or needing browser control, and it’s the long-term cost winner. jina reader is the right answer for prototypes and single-url fetches.

pick the one that matches your bottleneck. if your bottleneck is engineering time, pay for firecrawl. if your bottleneck is per-page cost, run crawl4ai. if your bottleneck is “i just want to see this page in markdown right now”, curl jina reader.

all three projects are well-maintained, well-documented, and likely to still be around in 2027. you can switch later. start with the one that gets you scraping today.

how to use crawl4ai for llm-ready web scraping (python tutorial 2026)

crawl4ai is an open-source python library that turns any webpage into clean, llm-ready markdown in one async call. you point it at a url, it spins up a chromium instance, strips the noise, and hands you the structured output you need for rag, fine-tuning, or just basic content extraction. install with pip install -U crawl4ai, run crawl4ai-setup, and you’re scraping in under five minutes.

most python scrapers were built before language models existed. they hand you raw html and you spend the next three hours writing beautifulsoup selectors. crawl4ai flipped that workflow. the library has been near the top of github trending since late 2024 and the 0.5.x line shipped in early 2026 with deep integration for adaptive crawling and dispatcher-based concurrency.

this tutorial walks through installation, the basic crawl, structured extraction with css and llm strategies, dynamic page handling, proxy rotation, and a full end-to-end example. if you’ve used playwright before, this will feel familiar. if you haven’t, that’s fine too. crawl4ai handles the browser plumbing for you.

why crawl4ai instead of beautifulsoup or scrapy

scrapy is still the right pick if you’re crawling millions of pages and you have time to write spiders. beautifulsoup is fine for static html. crawl4ai sits in a different lane. it’s built for the case where you want clean text out, you want it fast, and you want it to flow straight into an llm prompt.

three things make it different from older tools.

first, the default output is markdown, not html. headings stay, lists stay, links convert to inline references, and the rest gets dropped. you don’t write a single selector to get readable text.

second, it ships with a real browser by default. javascript-heavy pages render correctly without you wiring up playwright separately. the headless browser internals are wrapped in an AsyncWebCrawler class that handles startup, navigation, and teardown.

third, it’s async-first. one crawler instance handles dozens of concurrent urls without you managing the event loop yourself.

installing crawl4ai

the install is two commands. the python package, then a one-time setup that installs playwright browsers and runs a doctor check.

pip install -U crawl4ai
crawl4ai-setup

if the setup script throws errors about missing system libraries on linux, run the diagnostic:

crawl4ai-doctor

on macos and windows the playwright install usually just works. on a fresh ubuntu container you may need apt-get install -y libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 libgbm1 libpango-1.0-0 libcairo2 libasound2 before the browsers will launch.

verify the install:

python -c "import crawl4ai; print(crawl4ai.__version__)"

you should see 0.5.x or higher. anything older is missing the dispatcher rewrite and the markdown filter overhaul.

your first crawl

the simplest possible script. one url in, markdown out.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown[:2000])

asyncio.run(main())

run that. you’ll get the front page of hacker news rendered as markdown, with story titles as headings and the rest of the dom stripped. no selectors, no parsing, no cleanup pass.

the result object holds more than markdown. it also has result.html (the raw rendered html), result.cleaned_html (post-filter), result.media (a dict of images, audio, video extracted from the page), result.links (internal and external link lists), and result.metadata (title, description, og tags).

controlling the crawl with browser and run configs

the defaults are sensible but you’ll outgrow them fast. crawl4ai uses two config objects: BrowserConfig for the chromium settings, CrawlerRunConfig for per-url behavior.

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main():
    browser_cfg = BrowserConfig(
        headless=True,
        viewport_width=1920,
        viewport_height=1080,
        user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36",
    )

    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        wait_until="networkidle",
        page_timeout=30000,
        screenshot=True,
    )

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        result = await crawler.arun(
            url="https://example.com/dynamic-page",
            config=run_cfg,
        )
        print(result.markdown)
        if result.screenshot:
            with open("page.png", "wb") as f:
                import base64
                f.write(base64.b64decode(result.screenshot))

asyncio.run(main())

wait_until="networkidle" is critical for spas. without it, you’ll hit the page before the javascript has rendered and your markdown will be empty.

cache_mode=CacheMode.BYPASS forces a fresh fetch. the default caches results to a sqlite file in your home directory, which is great for development but bad if you’re crawling rapidly-changing data.

extracting structured data with css selectors

raw markdown is great for rag pipelines. for everything else you usually want fields. product price, author name, publish date. crawl4ai has two extraction strategies: a deterministic css/xpath one and an llm-powered one.

css first. it’s fast, free, and predictable.

import asyncio
import json
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

schema = {
    "name": "hn_stories",
    "baseSelector": "tr.athing",
    "fields": [
        {"name": "title", "selector": "span.titleline > a", "type": "text"},
        {"name": "url", "selector": "span.titleline > a", "type": "attribute", "attribute": "href"},
        {"name": "rank", "selector": "span.rank", "type": "text"},
    ],
}

async def main():
    cfg = CrawlerRunConfig(
        extraction_strategy=JsonCssExtractionStrategy(schema, verbose=False),
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com", config=cfg)
        stories = json.loads(result.extracted_content)
        for s in stories[:5]:
            print(s)

asyncio.run(main())

the schema describes what to extract. baseSelector finds repeated rows. each row gets the listed fields pulled out. you get json back, ready for a database or a csv.

extracting with an llm when selectors won’t hold

some sites change layouts often. some pages have data scattered in prose. that’s where the llm strategy earns its keep.

import asyncio
import os
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel

class Article(BaseModel):
    headline: str
    author: str
    published_date: str
    summary: str

async def main():
    llm_cfg = LLMConfig(
        provider="openai/gpt-4o-mini",
        api_token=os.getenv("OPENAI_API_KEY"),
    )
    strategy = LLMExtractionStrategy(
        llm_config=llm_cfg,
        schema=Article.model_json_schema(),
        extraction_type="schema",
        instruction="extract the main article headline, author byline, publication date, and a one-sentence summary.",
    )
    cfg = CrawlerRunConfig(extraction_strategy=strategy)

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.theverge.com/some-article-url",
            config=cfg,
        )
        print(result.extracted_content)

asyncio.run(main())

gpt-4o-mini is cheap, fast, and accurate enough for most extraction work. the openai pricing page lists current per-token costs. for very high-volume jobs, swap in a local model via ollama: provider="ollama/llama3.1:8b".

a word of caution. llm extraction is non-deterministic. for production pipelines on stable sites, use css selectors. reserve the llm path for messy sources and one-off jobs.

handling javascript, infinite scroll, and clicks

modern sites love to hide their content behind scroll triggers and click handlers. crawl4ai exposes js_code and wait_for for this.

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

scroll_js = """
(async () => {
    for (let i = 0; i < 5; i++) {
        window.scrollTo(0, document.body.scrollHeight);
        await new Promise(r => setTimeout(r, 1500));
    }
})();
"""

async def main():
    cfg = CrawlerRunConfig(
        js_code=scroll_js,
        wait_for="css:.product-card:nth-child(50)",
        page_timeout=60000,
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://example-shop.com/category",
            config=cfg,
        )
        print(f"loaded {result.markdown.count('product-card')} product cards")

asyncio.run(main())

the js scrolls five times, the wait_for blocks until the 50th product card appears, and only then does the markdown extraction run.

adding proxies for blocked sites

once you scale past a few hundred requests, you’ll start hitting rate limits and ip blocks. crawl4ai accepts a proxy in the browser config.

from crawl4ai import BrowserConfig

browser_cfg = BrowserConfig(
    headless=True,
    proxy_config={
        "server": "http://proxy.example.com:8080",
        "username": "user",
        "password": "pass",
    },
)

for rotating residential proxies you’ll typically point at a single endpoint that rotates the exit ip on every request. mobile proxy networks like singapore mobile proxy work the same way. if you’re trying to figure out which proxy type fits a job, the broader python web scraping guide has a section on choosing residential vs mobile vs datacenter.

crawling many urls in parallel

the arun_many method takes a list and a dispatcher. the memory-adaptive dispatcher is the sane default.

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher

urls = [f"https://example.com/page/{i}" for i in range(1, 101)]

async def main():
    dispatcher = MemoryAdaptiveDispatcher(
        memory_threshold_percent=80.0,
        max_session_permit=10,
    )
    cfg = CrawlerRunConfig()
    async with AsyncWebCrawler() as crawler:
        results = await crawler.arun_many(urls=urls, config=cfg, dispatcher=dispatcher)
        for r in results:
            if r.success:
                print(r.url, len(r.markdown))

asyncio.run(main())

the dispatcher caps concurrent sessions at 10 and pauses if memory crosses 80%. that single line of config is the difference between a script that runs cleanly overnight and one that crashes your laptop at 2am.

cleaning the markdown for llm input

the default markdown is good. it’s not perfect. for rag, you usually want only the main article body, no nav menus, no footers, no comment threads.

crawl4ai exposes a content filter for exactly that.

from crawl4ai import CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

md_gen = DefaultMarkdownGenerator(
    content_filter=PruningContentFilter(threshold=0.48, threshold_type="fixed"),
)
cfg = CrawlerRunConfig(markdown_generator=md_gen)

the pruning filter scores each block by text density and link ratio, drops anything below the threshold, and gives you a tighter result.markdown.fit_markdown field optimized for embedding.

a complete production-ready example

putting it together. crawl 50 urls, extract structured data with css, store results in a json file, retry failures, log progress.

import asyncio
import json
from pathlib import Path
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher

SCHEMA = {
    "name": "products",
    "baseSelector": "div.product",
    "fields": [
        {"name": "name", "selector": "h2.product-name", "type": "text"},
        {"name": "price", "selector": "span.price", "type": "text"},
        {"name": "stock", "selector": "span.availability", "type": "text"},
    ],
}

URLS = [f"https://shop.example.com/category/page-{i}" for i in range(1, 51)]

async def main():
    browser_cfg = BrowserConfig(headless=True)
    run_cfg = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        extraction_strategy=JsonCssExtractionStrategy(SCHEMA),
        wait_until="networkidle",
        page_timeout=30000,
    )
    dispatcher = MemoryAdaptiveDispatcher(max_session_permit=8)

    out = Path("products.jsonl")
    failures = []

    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        results = await crawler.arun_many(urls=URLS, config=run_cfg, dispatcher=dispatcher)
        with out.open("w") as f:
            for r in results:
                if r.success and r.extracted_content:
                    products = json.loads(r.extracted_content)
                    for p in products:
                        p["source_url"] = r.url
                        f.write(json.dumps(p) + "\n")
                else:
                    failures.append(r.url)

    print(f"saved {sum(1 for _ in out.open())} products. failures: {len(failures)}")

asyncio.run(main())

that script is the skeleton of a real production scraper. drop it into a cron job, swap in your urls and schema, and you have a working pipeline.

faq

is crawl4ai free?
yes. it’s mit-licensed and open-source. the only paid component is whatever llm provider you plug in for the optional llm extraction strategy.

does crawl4ai bypass cloudflare?
not by default. it uses standard chromium. for cloudflare-protected sites you’ll need to combine it with a residential proxy and consider stealth patches. the cloudflare turnstile bypass guide covers the techniques.

can crawl4ai scrape behind a login?
yes. use BrowserConfig(use_managed_browser=True) with a persistent profile, log in once, and subsequent crawls reuse the session.

how is crawl4ai different from firecrawl?
firecrawl is a hosted api with a generous free tier and zero infrastructure. crawl4ai is a python library you run yourself. for a side-by-side, see the firecrawl vs crawl4ai vs jina comparison.

what python versions does crawl4ai support?
3.10 and above. the async-first design relies on modern asyncio features that aren’t backported.

can i use crawl4ai with rag frameworks like langchain?
yes, and it’s actually the killer use case. the markdown output drops straight into a langchain document loader. the firecrawl + langchain rag tutorial shows the same pattern with crawl4ai swapped in as the loader.

conclusion

crawl4ai is the cleanest way to get llm-ready text out of the modern web with python. one async call, one markdown blob, no selector pain unless you want it. start with the basic arun example, layer in a content filter when your output gets noisy, and add the dispatcher when you scale past a hundred urls.

the official repo at github.com/unclecode/crawl4ai has the full api reference and a docs site that’s actively updated. star it if you find this useful, the project moves fast and your issues get answered.