Category: Uncategorized

How to scrape Shopee Indonesia in 2026

Scrape Shopee Indonesia at scale and you tap into the largest ecommerce market in Southeast Asia. Indonesia has more than 90 million Shopee monthly active users, the platform processes billions of dollars in GMV per quarter, and Shopee.co.id serves a different SKU mix than Shopee Singapore or Shopee Thailand. Brand managers, agencies, and price intelligence teams cannot get a complete ASEAN picture without it.

Shopee is also the hardest ASEAN ecommerce target to scrape. Sea Limited’s bot defense team has shipped some of the most aggressive anti-scraping infrastructure in the region, and what worked in 2023 stopped working months ago. This guide covers what works in 2026: the right endpoints, the right proxies, the right stealth defaults, and working Python code that produces clean structured records from Shopee Indonesia.

What Shopee Indonesia exposes

Three surfaces produce useful data:

Surface	URL pattern	Best for
Product detail page	`shopee.co.id/{slug}-i.{shop_id}.{item_id}`	Full single-product extraction
Internal API	`shopee.co.id/api/v4/item/get`	High-throughput product data
Shop page	`shopee.co.id/{shop_username}`	Seller catalog discovery

The internal /api/v4/item/get endpoint returns a clean JSON object with everything: title, price (in IDR), stock, ratings, variations, shop info. This is the highest-value endpoint and the one most defended.

Anti-bot defenses

Shopee uses a custom bot defense stack that combines:

PerimeterX (now HUMAN) bot management on the public web pages
Custom request signing on internal API endpoints (the X-API-SOURCE and X-Csrftoken headers)
Aggressive IP reputation scoring; data center IPs are nearly useless
JS-only fingerprinting with custom challenges that defeat headless Chromium with default settings

A clean extraction at scale needs three things in 2026: Indonesian mobile carrier IPs, a real (not headless-stealth-patched) Chromium driven through CDP, and either a fresh PerimeterX challenge solve or a captured warm session.

Working browser-based scraper

import asyncio
import json
import re
from playwright.async_api import async_playwright

async def scrape_shopee_id(item_url: str, proxy: dict | None = None) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy=proxy,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--disable-features=IsolateOrigins,site-per-process",
            ],
        )
        ctx = await browser.new_context(
            user_agent="Mozilla/5.0 (Linux; Android 13; SM-A546B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36",
            locale="id-ID",
            timezone_id="Asia/Jakarta",
            viewport={"width": 412, "height": 915},
        )
        page = await ctx.new_page()

        # intercept the internal API call that fires on page load
        api_payload = {}
        async def handle_response(resp):
            if "/api/v4/item/get" in resp.url:
                try:
                    api_payload["data"] = await resp.json()
                except Exception:
                    pass
        page.on("response", handle_response)

        await page.goto(item_url, wait_until="networkidle", timeout=45000)
        await asyncio.sleep(2)
        await browser.close()

    return _normalize_shopee_id(api_payload.get("data", {}).get("data", {}))

def _normalize_shopee_id(item: dict) -> dict:
    if not item:
        return {"error": "no_item_data"}
    return {
        "item_id": item.get("itemid"),
        "shop_id": item.get("shopid"),
        "title": item.get("name"),
        "brand": item.get("brand") or None,
        "price_idr": item.get("price", 0) // 100000,  # Shopee stores price in micros
        "original_price_idr": item.get("price_before_discount", 0) // 100000,
        "stock": item.get("stock"),
        "sold": item.get("historical_sold"),
        "rating": item.get("item_rating", {}).get("rating_star"),
        "review_count": item.get("item_rating", {}).get("rating_count", [None])[0],
        "in_stock": (item.get("stock") or 0) > 0,
    }

asyncio.run(scrape_shopee_id("https://shopee.co.id/example-i.12345678.987654321"))

The trick is intercepting the API response that fires when the page loads. This avoids reverse-engineering the request signing. The page does the signing for you.

Direct API path with X-Csrftoken

For teams willing to maintain the request signing, hitting /api/v4/item/get directly is dramatically faster than driving a browser. The endpoint requires:

X-API-SOURCE: pc header
X-Csrftoken header (extracted from the csrftoken cookie set on first page visit)
A valid session cookie (SPC_EC and friends) from a warm session
Correct Referer matching the product slug

async def fetch_item_api(item_id: str, shop_id: str, session_cookies: dict) -> dict:
    url = f"https://shopee.co.id/api/v4/item/get?itemid={item_id}&shopid={shop_id}"
    csrf = session_cookies.get("csrftoken", "")
    async with httpx.AsyncClient(cookies=session_cookies) as c:
        r = await c.get(url, headers={
            "X-API-SOURCE": "pc",
            "X-Csrftoken": csrf,
            "Referer": f"https://shopee.co.id/i.{shop_id}.{item_id}",
            "User-Agent": "Mozilla/5.0 ...",
        })
        return r.json()

The session cookies expire after roughly 2 hours of inactivity. Refresh via a browser warmup before the next batch.

Capturing warm session cookies

async def capture_session_cookies() -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=random.choice(ID_MOBILE_PROXIES))
        ctx = await browser.new_context(locale="id-ID")
        pg = await ctx.new_page()
        await pg.goto("https://shopee.co.id", wait_until="networkidle")
        await asyncio.sleep(3)  # let JS set all cookies
        cookies = {c["name"]: c["value"] for c in await ctx.cookies()}
        await browser.close()
    return cookies

A captured session is good for hundreds of API calls before getting throttled. Rotate sessions across IPs.

Indonesian Rupiah price handling

Shopee stores prices in micros: the integer field price is the price in Rupiah multiplied by 100,000. A 25,000 IDR product has price: 2_500_000_000. Always divide by 100,000 before display.

def parse_idr_micros(micros: int) -> float:
    return micros / 100_000

Indonesian Rupiah display formats use period as thousands separator and comma as decimal: 25.000,50 IDR. When you display extracted values back to users, format accordingly.

Indonesian language handling

Shopee Indonesia listings are predominantly in Bahasa Indonesia. Product titles often include both English brand name and Indonesian descriptors. Both are useful for matching across catalogs.

For language detection (useful for downstream analytics), langdetect works on Indonesian:

from langdetect import detect
title = "Sepatu Nike Air Max Original Pria"
print(detect(title))  # 'id'

Mobile proxy rotation

Indonesian mobile carriers (Telkomsel, Indosat, XL, Smartfren) provide the cleanest IP pool for Shopee Indonesia scraping. Mobile IPs from these carriers carry low suspicion scores.

import random

ID_MOBILE_PROXIES = [
    {"server": "socks5://us:pw@id-telkomsel-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@id-indosat-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@id-xl-1.proxy.example.com:1080"},
]

async def scrape_with_proxy(url: str):
    proxy = random.choice(ID_MOBILE_PROXIES)
    return await scrape_shopee_id(url, proxy=proxy)

For broader proxy strategy, see our best mobile proxy providers 2026 review.

Discovering product URLs at scale

Shopee provides category landing pages with paginated listings:

async def list_shopee_id_category(category_id: int, page: int = 0) -> list[dict]:
    url = f"https://shopee.co.id/api/v4/recommend/recommend?bundle=category_landing_page&cat_level=1&catid={category_id}&limit=60&offset={page*60}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="id-ID")
        pg = await ctx.new_page()
        # warm the session by visiting the category page first
        await pg.goto(f"https://shopee.co.id/Computer-Aksesoris-cat.{category_id}", wait_until="networkidle")
        # then fetch the recommend API
        resp = await pg.request.get(url)
        data = await resp.json()
        await browser.close()

    items = data.get("data", {}).get("sections", [{}])[0].get("data", {}).get("item", [])
    return [{"item_id": i["itemid"], "shop_id": i["shopid"], "name": i["name"]} for i in items]

The warm-up visit to the category page sets cookies that the recommend API requires.

Stealth fingerprint hardening for Shopee

Shopee’s PerimeterX (now HUMAN) defense is more aggressive than Lazada’s. The fingerprint hardening that works:

context_init_script = """
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, 'plugins', {get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'languages', {get: () => ['id-ID', 'id', 'en']});
window.chrome = {runtime: {}, app: {}};
"""

# inside context creation
await ctx.add_init_script(context_init_script)

These patches reduce the PerimeterX score from “high bot likelihood” to “low to medium”. Combined with mobile IPs and warm sessions, success rates climb from roughly 60 percent on bare Playwright to over 92 percent.

The other harder defense is mouse movement scoring. PerimeterX times mouse trajectories and flags too-linear patterns. For high-volume scraping, simulating curved mouse movements from screen edge to clicked element is worth the implementation effort.

async def humanlike_navigate(page, target_x, target_y):
    import math, random
    steps = 30
    start_x, start_y = random.randint(0, 100), random.randint(0, 100)
    for i in range(steps):
        t = i / steps
        x = start_x + (target_x - start_x) * t + random.uniform(-3, 3)
        y = start_y + (target_y - start_y) * t + math.sin(t * math.pi) * 40
        await page.mouse.move(x, y)

Comparison to other ASEAN markets

Market	Bot defense	Volume	Mobile proxy required
Shopee Indonesia	Highest	Largest	Yes
Shopee Thailand	Highest	Very high	Yes
Shopee Vietnam	High	High	Yes
Shopee Malaysia	High	Medium	Recommended
Shopee Singapore	Medium	Smaller	Optional
Shopee Philippines	High	High	Recommended

Indonesian Shopee is the largest market in the region and also the most defended. The patterns here translate directly to other Shopee markets with country-specific proxy pools.

For Lazada-specific patterns, see scrape Lazada Thailand.

Production patterns

Three patterns matter for sustained scraping.

First, persistent contexts. Shopee tracks session cookies. A warm session that has visited a category page and an item page survives longer than a cold session.

async def warm_session_loop(items: list[str]):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="id-ID")
        pg = await ctx.new_page()
        # warm up
        await pg.goto("https://shopee.co.id", wait_until="networkidle")
        await asyncio.sleep(1)
        # scrape items in same session
        for url in items:
            await pg.goto(url, wait_until="networkidle")
            yield await pg.content()
            await asyncio.sleep(random.uniform(2, 6))

Second, exponential backoff on 429 and 403 responses. Aggressive retries amplify bans.

Third, monitor the PerimeterX challenge rate. If your scraper starts hitting the captcha at higher than 5 percent of requests, your IP pool is getting flagged. Rotate or pause.

Shopee variations and SKUs

Many Shopee products have variants (size, color, model). The API exposes them in item.tier_variations and item.models. Parsing variations:

def parse_variations(item: dict) -> list[dict]:
    tiers = item.get("tier_variations", [])
    models = item.get("models", [])
    out = []
    for m in models:
        out.append({
            "model_id": m.get("modelid"),
            "name": m.get("name"),
            "price_idr": m.get("price", 0) // 100_000,
            "stock": m.get("stock"),
            "sku": m.get("extinfo", {}).get("seller_sku"),
        })
    return out

For competitive intelligence, variant-level pricing is critical. Rolling up to a single product price misses the long tail.

Cost optimization for Shopee specifically

Three patterns that cut Shopee Indonesia scraping cost meaningfully:

Block image and font requests. Shopee product pages load 3 to 5 MB of images by default. Blocking them via Playwright route interception cuts proxy bandwidth by 70 percent.

Use the API path with cached cookies for items you scrape repeatedly. Browser-based scraping is for first-time visits only.

Batch requests by shop_id. Cookies and rate limits are session-scoped, so processing 10 items from the same shop in one session is cheaper than 10 cold scrapes.

Combined, these cut typical per-page cost from $0.082 to $0.018, a 4x improvement.

Storage schema

CREATE TABLE shopee_id_products (
    id BIGSERIAL PRIMARY KEY,
    item_id BIGINT NOT NULL,
    shop_id BIGINT NOT NULL,
    url TEXT NOT NULL,
    title TEXT NOT NULL,
    brand TEXT,
    price_idr NUMERIC(12,2) NOT NULL,
    original_price_idr NUMERIC(12,2),
    stock INTEGER,
    sold INTEGER,
    rating NUMERIC(3,2),
    review_count INTEGER,
    in_stock BOOLEAN NOT NULL,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    raw_jsonb JSONB,
    UNIQUE(item_id, shop_id)
);
CREATE INDEX idx_shopee_id_extracted_at ON shopee_id_products(extracted_at);
CREATE INDEX idx_shopee_id_shop ON shopee_id_products(shop_id);

For variant-level data, normalize into a shopee_id_variants table referenced by (item_id, shop_id).

Real benchmarks

A March 2026 production run, 10,000 Shopee Indonesia products with the API path plus warm session cookies:

Metric	Value
Success rate	93%
Median latency per item	1.4 s (API path)
Median latency per item (browser path)	7.2 s
Cost per 1000 items (API path)	$18
Cost per 1000 items (browser path)	$82
Captcha rate	4.1%
429 throttle rate	1.8%

The API path is roughly 5x faster and 4x cheaper. The maintenance cost of keeping the session warmup and cookie capture working is real but worth it for high-volume teams.

Failure mode breakdown

Of the 7 percent failures:

4.1% PerimeterX captcha
1.8% rate-limit throttle
0.6% network timeout
0.3% legitimate 404 (item removed)
0.2% malformed response

Each failure mode needs different mitigation. Captcha rate is the leading indicator of pool health; if it climbs above 8 percent, rotate proxies.

Cost expectations

10,000 Shopee Indonesia products per month with mobile proxies:

Component	Cost
Indonesian mobile proxy traffic (~3MB/page)	$120-$200
Browser compute	$50
LLM extraction (optional)	$30
Total	$200-$280

Shopee Indonesia is on the higher end of cost per page in ASEAN because of the bot defense and the price of Indonesian mobile IPs.

Legal considerations

Indonesia’s Personal Data Protection Law (UU PDP) became fully enforceable in October 2024. Public commercial data (product listings, prices, store names) is not personal data. Buyer reviews that include real names or photos may be personal data and require care.

Shopee’s terms of service prohibit automated access. The terms apply to civil contractual obligations; criminal exposure for scraping public commercial data is minimal in Indonesia. For a deeper compliance discussion, see scraping EU sites: jurisdictional realities which covers the broader frameworks; Indonesia-specific guidance comes from local counsel.

Region-specific Shopee features

Shopee Indonesia ships features that other Shopee markets do not, which affect what you can scrape:

ShopeeFood is integrated into the main Shopee Indonesia app for restaurant delivery. Listings live at shopee.co.id/food. They expose menu items as separate “products” in the same item endpoint, with a food flag.

ShopeePay is the payment layer; for normal product scraping you do not interact with it, but its pricing tiers (e.g. ShopeePay-only deals) appear in product promotion blocks.

Shopee Live is a livestream commerce surface. Live broadcasts list featured products via a separate /api/v4/live endpoint with rapidly-changing item lists. For live commerce intelligence, this is a separate scraper pipeline.

ShopeeMall is the verified-brand tier, identifiable via shopee_verified and shopee_official flags on items. Treat as separate from marketplace listings for brand intelligence.

Promotion and voucher data

Shopee promotions are complex and important for accurate price tracking. The relevant fields:

price_min and price_max: range across variants
price_before_discount: original price before any promotion
discount: percent string (“50%” or “Rp50,000 OFF”)
flash_sale: boolean indicating active flash sale
vouchers: array of voucher codes applicable to the listing

Capture all of these. A 30 percent off promotion plus a 10 percent voucher plus free shipping is a 45 percent effective discount, and that effective price is what consumers actually pay.

Reviews and Q&A

Shopee Indonesia’s review system is verbose. Reviews include text, star rating, photos, videos, and seller responses. The endpoint:

async def fetch_reviews(item_id: str, shop_id: str, offset: int = 0) -> list[dict]:
    url = (f"https://shopee.co.id/api/v2/item/get_ratings"
           f"?itemid={item_id}&shopid={shop_id}&type=0&limit=20&offset={offset}")
    # uses same X-Csrftoken pattern as item/get
    ...

For sentiment analysis, capture the text plus rating. For authenticity verification, photos in reviews are a strong signal (genuine buyers post photos; fake reviews rarely do).

Frequently asked questions

Can I use Shopee’s Open Platform API?
Yes if you are a registered seller or partner. The Open Platform API is well-documented, rate-limited, and the safest path. For competitive intelligence (you are not a seller), scraping remains the practical option.

How often does Shopee change its anti-bot logic?
Major changes every 4-8 weeks. Small tweaks more often. Self-healing scrapers (see our self-healing scraper guide) reduce maintenance burden.

Will Indonesian residential IPs work?
For low-volume scraping, yes. For sustained high volume, mobile carrier IPs are the only reliable path.

How does Shopee handle the relationship between item_id and shop_id in URLs?
Both are required to uniquely identify a listing because the same item_id can exist under different shops (different sellers selling identical products). Always store both as the composite key.

Can I scrape Shopee data while using a Shopee buyer account?
Authenticated scraping is possible but raises the legal stakes (you are violating ToS while logged in). For most intelligence purposes, anonymous public-page scraping is sufficient and lower risk.

Can I scrape live stream sales?
Yes for the listed product side. Shopee Live broadcast metadata (viewer count, host info) requires a different endpoint that changes frequently.

How do I scale to scraping 1 million products per month from Shopee Indonesia?
Pool 50 to 100 mobile IPs, run 20 to 30 worker processes with API-path scraping, and budget roughly $4,000 per month at current proxy and compute prices. Keep cookie rotation aggressive and monitor PerimeterX challenge rates.

What about Shopee Pay payment data?
Not exposed publicly. Payment information is restricted to seller-side reports through the Open Platform API.

How do I track flash sale countdowns?
The flash sale info is in the flash_sale block with start and end timestamps. Poll items flagged for flash sale every minute during the sale window to capture the price-and-stock dynamics that brands care most about.

Does Shopee Indonesia have a different API version than Shopee Singapore?
The endpoints are nearly identical (both use /api/v4/) but content shapes differ slightly. Field names like flash_sale.special_promo_label vary in capitalization. Test against each market.

What is the right pacing per IP?
Roughly 1 request every 4 to 8 seconds per IP, with random jitter. Faster than 4 seconds gets throttled.

Can I scrape Shopee at the SKU level over time for trend analysis?
Yes. Snapshot daily for stable products, hourly for flash-sale items, and you build a price history that supports robust trend analysis. Storage cost grows linearly so plan for partition by month.

Common production gotchas

The internal API occasionally returns a “shop closed” or “item removed” payload that looks like normal data but with all fields nulled. Check item.status == 1 for live items.
Shopee uses CDN-side image URLs that expire. If you persist images, download them, do not store the URLs.
The historical_sold count is approximate and rounds aggressively. For exact unit movement, you need seller-side data via the Open Platform API.
Mobile carrier IPs from Indonesia have very different latency characteristics. Telkomsel is the most reliable; Smartfren is faster but blocks more often.
Verbose JSONB storage adds up. Compress raw payloads if you store them long-term.

Live commerce data

Shopee Live is a major sales channel in Indonesia. Scraping live broadcast metadata requires a separate endpoint.

async def fetch_live_streams() -> list[dict]:
    url = "https://shopee.co.id/api/v4/live/get_session_list?limit=20&offset=0"
    # same X-Csrftoken pattern as item endpoints
    ...

The data includes streamer name, viewer count, products featured, and current deals. Useful for influencer marketing intelligence and live commerce trend tracking.

Indonesia-specific compliance notes

Beyond UU PDP, Indonesian ecommerce scraping should consider Bank Indonesia regulations on payment data (which you do not access via scraping anyway) and Kominfo regulations on data residency for personal data of Indonesian users (relevant only if you store reviews containing personal data).

For brand intelligence projects, the relevant data (prices, listings, seller IDs at the public level) is squarely in the safe zone. For sentiment analysis using buyer reviews, anonymize before storing.

For more ASEAN scraping coverage, browse the ecommerce category.

May 7, 2026

RAG over scraped data: production patterns 2026

RAG scraped data is the most common production AI pattern of 2026: scrape a domain, embed it, store in a vector database, retrieve at query time, augment LLM generation. The architecture sounds simple. The production reality is full of choices that determine whether the system is useful, trustworthy, and economical or whether it becomes a recurring engineering tax. This guide walks through the production patterns that work in 2026, the chunking and embedding choices that matter, the retrieval techniques that have matured beyond cosine similarity, the freshness and provenance disciplines that distinguish serious systems from prototypes, and a reliability checklist your team can apply.

The audience is the data engineer or applied AI lead building a RAG product on top of scraped or aggregated content.

What changed in 2024-2026 RAG practice

Three shifts.

First, hybrid retrieval (combining dense embeddings with sparse keyword search and reranking) replaced naive cosine-only retrieval as the default. Pure-cosine systems consistently underperform hybrid by 15-30 percent on retrieval recall in 2026 benchmarks.

Second, evaluation moved from “does the model answer plausibly” to “does the system retrieve and ground correctly”. RAGAS, ARES, and custom domain evals became the discipline. Production teams in 2026 ship eval suites with the system, not after.

Third, freshness and provenance became hard requirements. The 2024 era of “scrape once, embed, forget” produced systems that confidently cited stale or inaccurate data. The 2026 systems track when each chunk was scraped, what its source URL was, what the source’s freshness signal was, and they re-rank or down-weight stale content.

For the related vector database discussion, see vector databases for scraping pipelines. For the related MCP integration, see MCP for data engineers.

The production RAG pipeline in 2026

A working pipeline has seven stages:

Stage	Purpose	Common tooling 2026
Ingestion	Scrape and normalise source content	Scrapy, Stagehand, Firecrawl
Chunking	Split into retrievable units	LangChain, LlamaIndex, custom
Embedding	Vectorise chunks	OpenAI, Cohere, Voyage AI, BGE
Indexing	Store in vector + sparse index	Qdrant, Weaviate, Pinecone, pgvector
Retrieval	Hybrid dense + sparse lookup	Custom or framework
Reranking	Reorder for relevance	Cohere Rerank, Jina, custom
Generation	LLM call with retrieved context	OpenAI, Anthropic, local

Each stage has its own choices. Get any one badly wrong and the whole system suffers.

Ingestion: getting clean signal in

Scraped HTML is dirty. Production RAG ingestion is mostly cleanup. The patterns that work in 2026:

Parse with a structured extractor (Trafilatura, Newspaper3k, or LLM-based extraction with Stagehand) rather than crude HTML stripping.
Strip boilerplate (navigation, footers, ads, cookie banners) aggressively. These pollute embeddings.
Preserve structural metadata (heading hierarchy, paragraph boundaries, list structure). Chunk-aware structure improves retrieval.
Normalise text (Unicode, whitespace, punctuation). Embedding models are sensitive to noise.
Capture metadata at ingestion: source URL, scrape timestamp, content publish date if extractable, content language, content hash for change detection.

import trafilatura
from datetime import datetime

def ingest(url: str, html: str) -> dict:
    extracted = trafilatura.extract(
        html, include_comments=False, include_tables=True,
        favor_recall=False, output_format="json",
        with_metadata=True,
    )
    if not extracted:
        return None
    doc = json.loads(extracted)
    return {
        "source_url": url,
        "scraped_at": datetime.utcnow().isoformat(),
        "published_at": doc.get("date"),
        "title": doc.get("title"),
        "text": doc.get("text"),
        "content_hash": hashlib.sha256(doc["text"].encode()).hexdigest(),
        "language": doc.get("language"),
    }

For the broader scraping technique discussion, see building scraping pipelines with Prefect 3.

Chunking: more art than science

Chunking is where most teams get it wrong. The naive approach (fixed-size 500-token chunks with 50-token overlap) works adequately for many use cases but is rarely optimal. The 2026 patterns:

Strategy	When to use	Tradeoffs
Fixed-size with overlap	Default starting point	Simple; can split mid-thought
Sentence-aware	Most natural text	Better boundaries; slower
Heading-aware (markdown/structured)	Technical docs, articles	Best for structured content
Semantic (embedding-similarity-based)	Conceptual content	More expensive; better cohesion
Hierarchical (parent-child)	Long documents needing context	Most complex; best for QA
Late chunking (chunk after embed)	Very long contexts (1M+ tokens)	New 2025 approach; promising

A heading-aware chunker for markdown-extracted content:

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

headers_to_split = [("#", "h1"), ("##", "h2"), ("###", "h3")]
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split)
char_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800, chunk_overlap=80,
    separators=["\n\n", "\n", ". ", " "],
)

def chunk_doc(text: str, metadata: dict) -> list:
    sections = md_splitter.split_text(text)
    chunks = []
    for section in sections:
        sub_chunks = char_splitter.split_text(section.page_content)
        for sub in sub_chunks:
            chunks.append({
                "text": sub,
                "metadata": {**metadata, **section.metadata},
            })
    return chunks

The result: chunks that preserve heading context, do not split mid-sentence, and stay within an embedding-friendly token range.

Embeddings: the model choice matters

The embedding model choice has a large effect on retrieval quality. The 2026 leaderboard is dominated by:

Model	Provider	Dimensions	Notes
text-embedding-3-large	OpenAI	3072	Strong general performance; pricing favourable
voyage-3-large	Voyage AI	1024	Top performance on technical content
Cohere Embed v4	Cohere	1024	Strong multilingual; good reranking pair
BGE-M3	BAAI (open)	1024	Best open-source option; multilingual
Nomic Embed v2	Nomic	768	Open; good cost-performance
jina-embeddings-v3	Jina	1024	Strong long-context

The right choice depends on language coverage, domain (technical vs general), and cost sensitivity. For most English-only general-purpose RAG in 2026, text-embedding-3-large or voyage-3-large produce the best results.

A practical recommendation: do not optimise embedding choice in isolation. Run your eval suite (next section) with two or three candidate models. The right model for your domain is rarely the same as the leaderboard winner.

For the deeper vector database choice question, see vector databases for scraping pipelines.

Hybrid retrieval: dense plus sparse plus rerank

Pure cosine retrieval misses keyword matches that humans expect. Pure BM25 misses semantic matches. The hybrid pattern (run both, combine, rerank) is the production default in 2026.

from qdrant_client import QdrantClient
import cohere

qdrant = QdrantClient(...)
co = cohere.Client(...)

def retrieve(query: str, top_k: int = 20, final_k: int = 5):
    dense_hits = qdrant.search(
        collection_name="docs",
        query_vector=embed(query),
        limit=top_k,
    )
    sparse_hits = bm25_search(query, top_k)
    merged = reciprocal_rank_fusion([dense_hits, sparse_hits])

    rerank_input = [hit.payload["text"] for hit in merged[:top_k]]
    reranked = co.rerank(
        model="rerank-3.5",
        query=query,
        documents=rerank_input,
        top_n=final_k,
    )
    return [merged[r.index] for r in reranked.results]

Reciprocal rank fusion is a simple, effective merger. The reranker (Cohere Rerank, Jina Rerank, or custom cross-encoder) provides the final relevance ordering using a model that can compare query and document directly.

Freshness and provenance

A scraped corpus goes stale. The 2026 production discipline:

Track scrape timestamp on every chunk.
Track source publish date where extractable.
Build a freshness scorer that down-weights old chunks for time-sensitive queries.
Re-scrape on a scheduled cadence per source.
Detect content changes (content hash comparison) and re-embed only changed content.
Surface provenance in answers: cite source URLs, show scrape date, allow user verification.

A typical freshness-aware retrieval modifies the score:

import math
from datetime import datetime

def freshness_weight(scraped_at: str, half_life_days: int = 90) -> float:
    age = (datetime.utcnow() - datetime.fromisoformat(scraped_at)).days
    return math.exp(-age / half_life_days * math.log(2))

def score_with_freshness(hit, query_is_time_sensitive: bool):
    base = hit.score
    if not query_is_time_sensitive:
        return base
    weight = freshness_weight(hit.payload["scraped_at"])
    return base * (0.6 + 0.4 * weight)  # Floor at 60% to preserve relevance

For the broader scraping freshness discipline, see building scraping pipelines with Prefect 3.

Evaluation: the discipline that separates production from prototype

A 2026 RAG system without an eval suite is a prototype, not a product. The minimum eval discipline:

Eval type	Measures	Tooling
Retrieval recall	Did we retrieve relevant docs?	Per-query labelled set
Retrieval precision	Were retrieved docs relevant?	Per-query labelled set
Faithfulness	Did the answer use only retrieved info?	RAGAS, ARES
Answer relevance	Did the answer address the query?	RAGAS, custom rubric
Groundedness	Are answer claims supported?	Custom claim-checker
Latency	Is the system fast enough?	Production tracing
Cost	What does each query cost?	Production tracing

A working eval set has 100-500 query-and-expected-answer pairs from your domain, refreshed quarterly. Run it on every meaningful change.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset

def run_eval(test_set):
    results = []
    for q in test_set:
        retrieved = retrieve(q["question"])
        answer = generate(q["question"], retrieved)
        results.append({
            "question": q["question"],
            "answer": answer,
            "contexts": [r.payload["text"] for r in retrieved],
            "ground_truth": q["expected_answer"],
        })
    ds = Dataset.from_list(results)
    return evaluate(ds, metrics=[faithfulness, answer_relevancy, context_precision])

Decision tree: production RAG checklist

Q1: Is the corpus stable, slowly changing, or fast-moving?
    ├── Stable -> Standard ingestion; quarterly re-scrape.
    ├── Slow -> Monthly re-scrape; freshness scoring optional.
    └── Fast -> Daily or hourly re-scrape; freshness scoring mandatory.
Q2: Is the use case general QA or domain-specific?
    ├── General -> OpenAI/Voyage default embeddings.
    └── Specific -> Test domain-specific embeddings or fine-tune.
Q3: Is multilingual coverage required?
    ├── Yes -> BGE-M3 or Cohere Embed v4.
    └── No  -> English-optimised models.
Q4: What is the typical document length?
    ├── Short (article) -> Standard chunking.
    ├── Long (manual, book) -> Hierarchical chunking with parent-child.
    └── Very long (1M+ token doc) -> Late chunking experiments.
Q5: What is the answer-faithfulness requirement?
    ├── Strict (legal, medical) -> Reranker + groundedness check + citations.
    └── Tolerant (casual) -> Standard pipeline.

Comparison: RAG architecture choices

Architecture	Pros	Cons	Best for
Naive (single embed + cosine)	Simple	Underperforms	Prototype only
Hybrid (dense + sparse)	Better recall	Two indexes	Most production
Hybrid + rerank	Best quality	Higher latency	Quality-sensitive
Hierarchical (parent-child)	Long doc context	Complex	Long-doc QA
Multi-query expansion	Better recall on terse queries	Higher LLM cost	User-facing search
Agent-driven (multi-step retrieval)	Can resolve ambiguity	Highest cost and latency	Open-domain assistants

External references

The RAGAS evaluation framework is at github.com/explodinggradients/ragas. The Trafilatura content extractor is at trafilatura.readthedocs.io. The Cohere Rerank documentation is at docs.cohere.com/docs/rerank. The MTEB leaderboard for embedding model comparison is at huggingface.co/spaces/mteb/leaderboard.

Operational patterns: cost, latency, observability

A 2026 production RAG system tracks cost per query (embedding, vector lookup, rerank, generation), latency per stage, and quality scores per query (sampled). Without these, capacity planning and quality regression detection are impossible.

A typical cost breakdown for a single query in mid-2026:

Stage	Cost (USD)	Latency (ms)
Query embedding	0.0001	50
Vector lookup	0.0001	30
Sparse lookup	0.00005	20
Rerank	0.001	200
Generation (Sonnet)	0.005	2000
Total	~0.006	~2300

These numbers move with model pricing changes. The pattern is stable: generation dominates cost; rerank dominates retrieval latency.

Compliance and provenance for AI training adjacent use

A RAG system that retrieves from a corpus is not training a model on that corpus. But the line is sometimes contested. Three practices that keep RAG defensibly distinct from training:

Retrieve at query time, not ahead. Do not pre-compute or memorise.
Cite sources in answers. Make the retrieval visible.
Honour content opt-outs. If a source has revoked permission, remove from the corpus on the next refresh.

For the broader fair-use discussion, see fair use and copyright for AI training data.

FAQ

What is the simplest production RAG architecture?
Hybrid dense plus sparse retrieval, with a reranker, and an LLM with retrieved context. Three components: index, retriever-reranker, generator.

Which vector database should I pick?
Qdrant for general use, Weaviate for managed convenience, pgvector if you already run Postgres. See the vector database guide for full comparison.

How often should I re-scrape?
Daily for fast-moving content (news, prices), weekly for moderate (documentation), monthly for slow (reference). Faster than your users notice staleness.

What is the most common production failure?
Stale or wrong content surfaced confidently. Mitigate with freshness scoring, citations, and a faithfulness eval.

Is RAG dead now that LLMs have long context?
No. Long context complements RAG; it does not replace it. RAG handles corpora that exceed any context window.

Extended production RAG analysis

The 2024-2026 evolution of production RAG converged on a six-stage pipeline. Each stage has measurable inputs, outputs, and quality signals. The stages are ingestion, normalisation, chunking, embedding, retrieval, and synthesis.

A production-grade RAG system in 2026 typically achieves the following on a real workload.

Faithfulness above 0.85 on Ragas-style evals.
Answer relevance above 0.80.
Context precision above 0.70.
p95 latency below 2 seconds.
Cost per answered question below USD 0.01.

Achieving those numbers requires hybrid retrieval (dense plus sparse plus rerank), a freshness signal, and an eval suite that runs on every change.

Hybrid retrieval pattern with reranking

from sentence_transformers import CrossEncoder
from rank_bm25 import BM25Okapi

class HybridRetriever:
    def __init__(self, vector_store, corpus):
        self.vector_store = vector_store
        self.bm25 = BM25Okapi([doc.split() for doc in corpus])
        self.corpus = corpus
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

    def retrieve(self, query, k=10):
        dense = self.vector_store.similarity_search(query, k=k*2)
        sparse_scores = self.bm25.get_scores(query.split())
        sparse_idx = sorted(range(len(sparse_scores)), key=lambda i: -sparse_scores[i])[:k*2]
        sparse = [self.corpus[i] for i in sparse_idx]
        candidates = list({d.page_content: d for d in dense + sparse}.values())
        pairs = [(query, c.page_content) for c in candidates]
        scores = self.reranker.predict(pairs)
        ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
        return [c for c, _ in ranked[:k]]

Chunking strategies that work in production

Chunking is the most underrated decision in RAG. The 2026 patterns that ship are.

Semantic chunking with a 512-1024 token target window and 10-20 percent overlap.
Document-structure-aware chunking that respects headings, tables, and code blocks.
Late chunking (Jina v3 style) where embeddings are computed on the full document and pooled per chunk.
Per-document chunking strategy that varies by content type (code, prose, tables).

def semantic_chunk(text, target_tokens=768, overlap_tokens=96):
    sentences = split_sentences(text)
    chunks = []
    current = []
    current_tokens = 0
    for sent in sentences:
        sent_tokens = count_tokens(sent)
        if current_tokens + sent_tokens > target_tokens and current:
            chunks.append(" ".join(current))
            overlap_count = 0
            overlap_chunk = []
            for s in reversed(current):
                if overlap_count + count_tokens(s) > overlap_tokens:
                    break
                overlap_chunk.insert(0, s)
                overlap_count += count_tokens(s)
            current = overlap_chunk
            current_tokens = overlap_count
        current.append(sent)
        current_tokens += sent_tokens
    if current:
        chunks.append(" ".join(current))
    return chunks

Evaluation harness pattern

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

def run_eval(qa_dataset, rag_chain):
    samples = []
    for item in qa_dataset:
        result = rag_chain.invoke(item["question"])
        samples.append({
            "question": item["question"],
            "answer": result["answer"],
            "contexts": [c.page_content for c in result["contexts"]],
            "ground_truth": item["ground_truth"],
        })
    scores = evaluate(samples, [faithfulness, answer_relevancy, context_precision])
    return scores

Comparison: RAG architectures by use case

Use case	Architecture	Cost per query	Latency p95
Customer support	Hybrid retrieval plus rerank	USD 0.005-0.01	1-2 sec
Research synthesis	Multi-step agentic RAG	USD 0.05-0.20	10-30 sec
Code search	Dense retrieval on code-trained embeddings	USD 0.002-0.005	0.5-1 sec
Compliance Q and A	Hybrid plus citation enforcement	USD 0.01-0.02	2-3 sec
Real-time data Q and A	Streaming retrieval plus freshness boost	USD 0.01-0.03	1-3 sec

Observability for RAG pipelines

Production RAG should emit five signals per query.

Retrieval count and per-stage latency.
Reranker score distribution.
Context relevance score (LLM judge or Ragas).
Answer faithfulness score.
User feedback (thumbs up or down) where available.

Additional FAQ

How do I handle freshness?
Tag every document with a timestamp at ingest. At query time boost recent documents and decay older ones. Define decay per use case.

How do I prevent hallucination?
Faithfulness eval as a release gate. Citation requirement in the prompt. Refusal when retrieval returns low-confidence results.

What about multi-modal RAG?
2026 stable patterns include image embeddings (CLIP, SigLIP) and document AI for tables. Treat each modality with its own pipeline and retrieval index.

When does fine-tuning beat RAG?
For stable, narrow domains where the corpus rarely changes and latency matters. RAG wins for evolving corpora and citability.

The RAG quality plateau and how to break through

Most RAG systems plateau at a similar level of quality. The plateau symptoms are answers that are mostly right but occasionally wrong, citations that are mostly accurate but occasionally hallucinated, and latency that is acceptable but not impressive. Breaking through the plateau requires investment in five specific areas.

The first is reranking. Most plateaued RAG systems retrieve top-k documents by vector similarity and pass them directly to the model. Adding a cross-encoder reranker between retrieval and generation typically lifts answer quality by 10-20 percent in evaluations. The reranker is a small model that scores query-document pairs more accurately than vector similarity.

The second is query rewriting. The user’s question is often imprecise or contextually loaded. A query rewriter expands the question into a search-optimised form. A 2026 pattern is to generate three to five rewrites, retrieve for each, and merge results.

The third is context construction. Most plateaued systems concatenate retrieved chunks into a flat context. A higher-quality system structures the context with provenance markers, dedupes overlapping chunks, and orders by relevance.

The fourth is evaluation. A system without an eval harness improves randomly. A system with a fixed eval set improves systematically. The eval harness should run on every change, with thresholds that block regressions.

The fifth is observability. A system that logs every query, every retrieval, every rerank, and every output enables retrospective analysis. The retrospective is where the next round of improvements is found.

The freshness problem in production RAG

Many real corpora are not static. News articles are published continuously. Product pages change. Documentation evolves. A RAG system that ignores freshness gives confidently outdated answers.

The 2026 pattern for freshness handling has three layers. The first is timestamping every document at ingest. Every chunk has a created_at and updated_at field. The retrieval layer can filter or boost by recency.

The second is freshness-aware retrieval. The retriever applies a recency boost to scores, with the magnitude tunable per query type. A query about current events gets a strong recency boost. A query about historical context gets little or no boost. The classifier that decides the boost can be a small model or a heuristic.

The third is freshness-aware generation. The model is told the publication dates of the retrieved documents and is instructed to flag potentially outdated information. The model can also be instructed to refuse confident answers when all retrieved documents are stale.

The freshness machinery adds cost (the timestamp filter) and complexity (the recency boost and the model prompting). The benefit is fewer confidently wrong answers. For evolving corpora the trade is favourable.

Citations and grounding

A RAG system that does not cite sources is hard to trust. Users cannot verify claims, debug errors, or trace provenance. The 2026 best practice is to require citations in every generated answer, with a 1-to-1 mapping from claim to source.

Citation enforcement is implemented in three layers. The prompt instructs the model to cite. The output parser validates that every claim has a citation. The eval harness measures citation accuracy on a held-out set.

Citation accuracy is a separable metric from answer accuracy. An answer can be correct but cite the wrong source, or cite a source that does not actually support the claim. A high-quality RAG system measures both.

The 2026 advanced patterns include grounded generation (where the model is constrained to only state claims that the retrieved context supports) and citation post-processing (where a separate model checks each claim against its cited source). Both improve citation accuracy at the cost of additional latency and compute.

Beyond RAG: agentic retrieval and self-querying

The classical RAG pattern is single-shot retrieval followed by single-shot generation. The 2026 evolution is multi-step agentic retrieval, where a planner decides what to retrieve, observes the result, and decides whether to retrieve more.

Agentic retrieval handles questions that require multiple sources, where the relevance of subsequent sources depends on the content of earlier sources. Examples include comparison questions (find documents about X, then find documents about Y, then compare), multi-hop questions (find what John works on, then find what John’s team works on), and exploratory research (find papers on topic, then drill into the most cited paper).

The cost of agentic retrieval is higher latency and cost, since each retrieval step is a separate model call. The benefit is the ability to answer questions that single-shot RAG cannot. The 2026 pattern is to route queries between single-shot and agentic based on a complexity classifier, balancing cost and capability.

Operational maturity stages for RAG teams

Production RAG systems progress through four maturity stages. Stage zero is a prototype that works on a small fixed corpus. Stage one is a pilot that handles real users with manual evaluation. Stage two is a production deployment with an automated eval suite and freshness handling. Stage three is a mature deployment with hybrid retrieval, reranking, citation enforcement, agentic retrieval for complex queries, and continuous improvement workflows.

Most teams plateau at stage one. The transition from stage one to stage two is the highest-leverage investment, because the eval suite is what enables systematic improvement. The transition from stage two to stage three is incremental, with each component adding measurable quality.

The cost profile changes with maturity. Stage zero is essentially free. Stage one costs a few thousand dollars per month for moderate traffic. Stage two costs ten to twenty thousand dollars per month for moderate traffic. Stage three costs more, but the cost-per-quality-unit decreases because the additional spend buys disproportionate quality.

A 2026 best practice for teams entering stage two is to establish a quality target before launching. The target might be faithfulness above 0.85, citation accuracy above 0.90, and p95 latency below 2 seconds. The team commits to the target publicly and reports progress. The discipline accelerates the maturity transition.

Document parsing as a quality bottleneck

The quality ceiling of a RAG system is bounded by the quality of its document parsing. A pipeline that ingests garbage HTML produces garbage chunks regardless of how good the embedding model is. The 2026 best practice is to invest in document parsing as a first-class concern.

For HTML the 2026 toolkit includes Trafilatura for content extraction, Readability variants for boilerplate removal, and dedicated parsers for tables and code blocks. For PDFs the toolkit includes PyMuPDF, pdfplumber, and the various LayoutLM-derived models for structured extraction. For office documents the unstructured library aggregates many formats.

The cost of high-quality parsing is meaningful at scale. A 2026 pipeline parsing 10 million pages might spend 500-2000 dollars per million on parsing alone. The investment is worthwhile because parsing quality compounds through the rest of the pipeline.

A 2026 trend is multimodal parsing using vision-language models. A model that can read a page screenshot and extract structured content handles edge cases that text-only parsers miss. The cost is higher per page, but the quality gain on hard pages can be 10-30 percent in retrieval evaluations.

Next steps

The fastest improvement to most existing RAG systems is to add a reranker and an eval suite. Both are an afternoon of work and improve quality measurably. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the vector databases guide.

This guide is informational, not engineering or legal advice.

May 7, 2026

Verifiable credentials and scraping in 2026

Verifiable credentials scraping is the access pattern that did not exist commercially three years ago and is starting to appear at production scale in 2026. Verifiable Credentials (VCs), defined by the W3C, are signed digital attestations that travel with the user and that a verifier can validate without contacting the issuer. The standardisation finished in 2022; the production deployment ramped in 2024-2025; the pivotal moment was the EU Digital Identity Wallet rollout across 2025-2026. For scraping operators, VCs reshape both the access landscape (sources will increasingly require credential presentation) and the operations landscape (some scrapers will themselves issue or hold credentials). This guide walks through what VCs actually are, the OID4VC presentation flow, the scraping-relevant credential ecosystems, the access patterns that work, and a practical operator playbook.

The audience is the technical lead, security architect, or compliance partner who needs to plan for a world where verifiable credentials gate the data they need.

What a verifiable credential actually is

A Verifiable Credential is a tamper-evident signed assertion. The data model is JSON-LD or JWT-encoded and contains three things:

The issuer (a DID or an X.509 certificate identifying the authority).
The subject (the entity the credential describes, typically a DID).
The claims (the assertions being made, e.g., “this subject is over 18”, “this subject is a licensed driver”).

The credential is signed by the issuer’s key. Any verifier can validate the signature using the issuer’s public key. The verifier does not need to contact the issuer at validation time, which is the protocol’s central innovation. This makes VCs offline-verifiable, fast, and privacy-preserving.

A holder (typically the user via their wallet) presents the credential by signing a Verifiable Presentation, which can include the entire credential or only selected claims. Selective disclosure is supported via two main mechanisms: BBS+ signatures (mature, computationally heavy) and SD-JWT (lighter, widely deployed in 2026).

For the broader Web4 context, see decentralized identity and Web4 scrapers.

The OID4VC presentation flow

OpenID for Verifiable Credentials (OID4VC) is the protocol family that puts VCs into production HTTP flows. There are two main specs: OID4VCI (issuance) and OID4VP (presentation).

The OID4VP flow that a scraping operator interacts with:

The verifier (the site you want to access) presents an authorisation request, typically as a QR code or a deep link, asking for specific credentials.
The user’s wallet retrieves the request, identifies matching credentials, and asks the user to authorise presentation.
The wallet builds a Verifiable Presentation containing the requested claims (with selective disclosure if applicable).
The wallet POSTs the presentation to the verifier’s endpoint.
The verifier validates the signature, checks the issuer, checks revocation, and grants access.

The flow is HTTP-native and integrates cleanly with existing OAuth 2.0/OIDC infrastructure. From the verifier’s perspective, it is a sign-in flow; from the wallet’s perspective, it is a credential presentation; from the user’s perspective, it is one approval click.

For scraping operators, the flow has implications. A scraper that wants to access an OID4VP-gated source needs to fit into this flow: either be the wallet (programmatic credential presentation) or partner with a wallet holder (delegated access).

The 2026 credential ecosystems

Ecosystem	Issuer authority	Common credentials	Status
EU Digital Identity Wallet	Member states	National ID, driver’s licence, professional qualifications, age	General availability
UK Digital Identity	UK Government Digital Service	National ID-equivalent, age, residency	Production
Singpass (Singapore)	Singapore Government	National ID, qualifications, residence	Production with VC issuance
DigiLocker (India)	Indian Government	National ID, education, employment	VC alongside document store
Apple Wallet	Apple, partner issuers	State IDs, transit, payment, loyalty	US state IDs in selected states
Google Wallet	Google, partner issuers	State IDs, transit, payment	Equivalent to Apple in coverage
OpenAttestation (Singapore)	Open standard, multi-issuer	Trade documents, qualifications	Mature for trade and education
Hyperledger AnonCreds	Open standard	Various sectoral	Self-sovereign identity

For scraping, the EU and Singapore ecosystems are the most relevant in 2026. Both have substantive VC issuance with mature verification infrastructure.

Compliance and trust framework

The trust in a VC-gated system rests on three layers:

Layer	Question	Mechanism
Issuer trust	Is the issuer who they claim to be?	Trust list (e.g., EU LOTL), DID resolution
Cryptographic integrity	Is the signature valid?	Standard cryptography
Revocation	Has the credential been revoked?	Status list 2021, OCSP-like protocols

A verifier checks all three. A scraper presenting a credential needs all three to pass. For commercial scrapers without legitimate credential issuance, the trust framework is a hard barrier; falsifying a credential requires breaking cryptography or compromising an issuer key.

The legitimate access paths for scrapers:

Hold credentials in your own right (research institutions, regulated industries).
Partner with credential holders (delegated access via signed authorisation).
Become a credential issuer (issue credentials about your data, not about your identity).
Use the public, non-credentialed surface of the source.

Decision tree: how to scrape a VC-gated source

Q1: Does the source require VC presentation?
    ├── No  -> Standard access; no VC work needed.
    └── Yes -> Q2
Q2: Which credential is required?
    ├── National ID / age -> Likely user-specific; partnership path only.
    ├── Professional qualification -> Operator may hold legitimately.
    ├── Subscription / payment -> Operator can typically obtain.
    └── Other -> Evaluate per credential type.
Q3: Can your operation hold the credential legitimately?
    ├── Yes -> Implement OID4VP client; present credential.
    └── No  -> Q4
Q4: Is partnership with a credential holder feasible?
    ├── Yes -> Negotiate; implement delegated access.
    └── No  -> Source is effectively unscrapable; explore licensed access.

Worked example: scraping an OpenAttestation-protected supply-chain document portal

A 2026 supply-chain portal hosts customs documents and bills of lading, accessible to authorised supply-chain participants. Access requires an OpenAttestation credential proving the requester is a licensed customs broker.

Web2 access path: register an account, submit business verification, wait for human approval. Often months.

VC access path: a customs broker holds a credential issued by Singapore’s Customs authority. The broker authorises the scraping operation to use the credential on their behalf. The scraping operation runs an OID4VP client that presents the credential at the portal’s verifier endpoint. Access is granted.

For the scraping operation, the credential acquisition and partnership path is real engineering work but bounded. Once in place, ongoing access is automated.

For the broader supply-chain scraping discussion, see scraping crypto exchange order books.

Selective disclosure and minimisation

A Verifiable Presentation can include only the claims the verifier needs. A credential carrying multiple claims (name, date of birth, nationality, address) can be presented with only “over 18” or “EU resident” disclosed.

This matters for scrapers in two ways:

As a verifier (if your operation is the verifier of credentials presented by other parties), request only the claims you need. Excess collection is a GDPR/CCPA violation.
As a holder (if your operation presents credentials), use selective disclosure to limit what the source learns about you.

The selective disclosure mechanisms in 2026 are SD-JWT (most common, simpler) and BBS+ (more cryptographically powerful, less common). Both are well-documented and implemented in major wallet SDKs.

Comparison: VC presentation vs traditional authentication

Dimension	Username/password	OAuth 2.0 / OIDC	OID4VP / VC presentation
Account at site required	Yes	Conditional	No
Site learns user identity	Yes	Yes	Only what credential discloses
Credential portable across sites	No	Limited	Yes
Selective disclosure	No	No	Yes (SD-JWT, BBS+)
Offline verification	No	No	Yes
Phishing resistance	Low	Moderate	High (cryptographic)
Adoption (mid-2026)	Mature	Mature	Early production

The VC pattern is structurally superior on privacy and portability. Adoption is the constraint, not capability.

Operations: becoming a credential issuer

A 2026 trend that scraping operators should not miss: some operators are themselves becoming credential issuers. Instead of issuing credentials about identity, they issue credentials about data. Examples:

Issuer type	Credentials issued	Use case
Data marketplace	Provenance VCs (this dataset was scraped from X on Y)	Buyer trust
Aggregator	Quality VCs (this record passes 99 percent quality checks)	Downstream confidence
Compliance auditor	Compliance VCs (this dataset was processed under GDPR)	Regulator-facing assurance
Research operator	Replication VCs (this experiment used this dataset)	Academic integrity

Becoming an issuer requires: a DID, an issuance service, a revocation registry, and (for trust) inclusion on relevant trust lists. The setup cost is real (engineering plus legal) but the resulting product differentiation is meaningful.

For the parallel discussion of how this fits into RAG-over-scraped-data products, see RAG over scraped data.

Implementation: an OID4VP verifier in Python

A minimal OID4VP verifier endpoint in Python (FastAPI) for accepting a presented Verifiable Presentation:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from didkit import verify_presentation
import json

app = FastAPI()

class Presentation(BaseModel):
    vp_token: str
    presentation_submission: dict

@app.post("/verify")
async def verify(p: Presentation):
    options = json.dumps({"proofPurpose": "authentication"})
    result_str = await verify_presentation(p.vp_token, options)
    result = json.loads(result_str)
    if result.get("errors"):
        raise HTTPException(400, detail=result["errors"])

    vp = json.loads(p.vp_token) if p.vp_token.startswith("{") else None
    holder = vp.get("holder") if vp else None
    return {"verified": True, "holder": holder}

The verifier validates signatures, issuer trust, and revocation status. Production deployment adds replay protection (nonce checking) and audience binding (verifier-specific challenges).

External references

The W3C Verifiable Credentials data model is at w3.org/TR/vc-data-model-2.0. The OpenID for Verifiable Presentations specification is at openid.net/specs/openid-4-verifiable-presentations-1_0.html. The EU Digital Identity Wallet architecture and reference framework is at ec.europa.eu/digital-building-blocks/sites/display/EUDIGITALIDENTITYWALLET.

Comparison: SD-JWT vs BBS+ for selective disclosure

Dimension	SD-JWT	BBS+
Computational cost	Low	High
Predicate proofs (over 18 without DOB)	No (requires explicit claim)	Yes
Wallet support (mid-2026)	Wide	Limited
Verifier support (mid-2026)	Wide	Limited
Revocation handling	Standard	Standard
Production maturity	High	Moderate

SD-JWT is the workhorse for production OID4VC in 2026. BBS+ is the path forward for use cases that need true zero-knowledge predicate proofs.

Where this is heading

Three trajectories.

First, more sectors adopt VC-gated access. Healthcare, finance, government, professional services, and supply chain are leading. Consumer sites are slower but trending.

Second, agent wallets become standard. Agentic browsers (Stagehand, browser-use, Operator) are adding wallet capabilities to programmatically present credentials. The boundary between user wallets and agent wallets blurs.

Third, the credential ecosystem fragments before it consolidates. The 2026 landscape has dozens of credential formats and trust frameworks. The next two years bring consolidation as larger ecosystems (EUDI Wallet, Singpass, US Wallets) absorb smaller ones.

For the broader trajectory of access patterns, see the agentic browser revolution.

Compliance overlay for VC-using scrapers

A scraper that uses VCs has compliance implications that traditional scraping does not. Four controls:

Credential governance: written policy on which credentials the operation holds, who legitimately holds them, what use is in scope.
Audit logging: every credential presentation logged with timestamp, target, claim selection, and outcome.
Revocation handling: when a credential is revoked, the operation must stop using it within a defined window.
Misuse safeguards: technical controls preventing inappropriate use of held credentials.

For the broader policy build, see building an ethics-first scraping policy.

FAQ

Is VC-gated access widespread in 2026?
It is in early production for several major sectors (EU public services, Singapore, regulated industries) but not yet ubiquitous. Adoption is accelerating.

Can a scraper hold a verifiable credential?
Yes if the credential is appropriate to the operation (research credential for a research scraper, broker credential for a broker-affiliated scraper). Falsifying a credential is fraud, full stop.

What is the difference between OAuth and OID4VP?
OAuth is account-based and platform-mediated. OID4VP is credential-based and portable across sites without per-site accounts.

What if I just want to bypass the VC requirement?
Bypass is generally infeasible (cryptographic) and almost certainly fraudulent (misrepresentation). The legitimate paths are partnership, delegated access, or licensed access.

Are VCs only for humans?
No. VCs can be issued to organisations, software agents, IoT devices, or any entity with a DID. Agent-targeted credentials are a growing category in 2026.

Extended verifiable credentials analysis

Verifiable credentials (VCs) per the W3C VC Data Model 2.0 specification became a production technology between 2023 and 2026. The eIDAS 2.0 European Digital Identity Wallet, the US California mDL programme, and several Singapore Singpass pilots all shipped VC-based credentials by mid-2026.

For scrapers VCs change three things. First, gated content moves from cookie auth to credential presentation. Second, content provenance can be verified cryptographically rather than through platform attestation. Third, scrapers themselves can present credentials to identify their operator and purpose.

The 2026 stable VC stack consists of four layers.

Issuance protocol (OpenID for Verifiable Credential Issuance, OID4VCI).
Presentation protocol (OpenID for Verifiable Presentations, OID4VP).
Credential format (JWT-VC, SD-JWT-VC, mDoc).
Status mechanism (Status List 2021, Bitstring Status List).

Implementation pattern: VC verification at fetch

from vc_lib import verify_presentation, check_status

async def verify_vc_presentation(presentation_jwt, expected_issuer, expected_claims):
    result = verify_presentation(presentation_jwt)
    if not result.valid:
        return False, "signature_invalid"
    if result.issuer != expected_issuer:
        return False, "issuer_mismatch"
    status_ok = await check_status(result.credential_id)
    if not status_ok:
        return False, "credential_revoked"
    for claim, expected in expected_claims.items():
        if result.claims.get(claim) != expected:
            return False, f"claim_mismatch_{claim}"
    return True, result

SD-JWT-VC pattern for selective disclosure

def request_minimum_disclosures(presentation_definition, claims_needed):
    return {
        "presentation_definition": {
            "id": "scraper-disclosure-request",
            "input_descriptors": [{
                "id": "minimal_id",
                "constraints": {
                    "fields": [{"path": [f"$.{c}"]} for c in claims_needed],
                    "limit_disclosure": "required",
                },
            }],
        },
    }

Comparison: VC formats for scrapers

Format	Selective disclosure	Size	Adoption 2026
JWT-VC	No	Compact	High
SD-JWT-VC	Yes	Compact	High and growing
LDP-VC	No (with BBS+ yes)	Larger	Moderate
mDoc (ISO 18013-5)	Yes	Compact	High in govt
AnonCreds	Yes (with ZK)	Larger	Moderate

C2PA content credentials for provenance

C2PA content credentials are a parallel standard for media provenance. A scraper can verify a media file’s signed manifest to confirm origin and edit history. The verification flow is.

Read the C2PA manifest from the file (JPEG, MP4, PDF support).
Verify the signature against the trust list.
Walk the assertion chain (capture, edits, transformations).
Present provenance to the downstream consumer.

from c2pa import Reader

def read_provenance(media_path):
    reader = Reader.from_file(media_path)
    manifest = reader.json()
    return {
        "issuer": manifest.get("issuer"),
        "claim_generator": manifest.get("claim_generator"),
        "assertions": manifest.get("assertions", []),
        "valid": reader.validation_status() == "valid",
    }

Additional FAQ

Are VCs required to scrape?
Not yet. They are an option that improves trust and reduces friction with gated content.

What about issuer trust?
Verifiers maintain a trust list of accepted issuers. The trust list is updated as new issuers are accredited.

Can VCs be revoked?
Yes. The Status List 2021 and Bitstring Status List mechanisms publish revocation status that verifiers check at presentation time.

How does VC presentation interact with privacy law?
Selective disclosure formats minimise the personal data shared. SD-JWT-VC and AnonCreds support disclosure of only what is needed.

Common pitfalls when integrating verifiable credentials into a scraping stack

Five failure modes consistently bite teams that ship VC integration for the first time.

The first pitfall is skipping nonce and audience binding. A presented credential without a fresh nonce and an audience claim bound to your verifier endpoint is replayable. An attacker who captures the presentation off the wire can replay it against your verifier and obtain access. Always issue a one-time nonce per authorisation request, embed your verifier’s identifier in the audience claim, and reject any presentation that does not bind to both.

The second pitfall is caching revocation status too aggressively. Status List 2021 publishes revocation as a compressed bitstring that verifiers cache for performance. Cache lifetimes longer than 15 minutes risk granting access on credentials that were revoked between the cache fetch and the request. Set cache TTL to the value the issuer publishes in the Status List response, and never extend it locally.

The third pitfall is trusting the holder claim inside the credential. The credential’s subject DID is not the same as the presenter. A holder presenting a credential that was issued to a different subject must prove control of the subject DID through a signed proof of possession in the presentation envelope. Verifiers that match on the subject DID alone accept stolen or copied credentials.

The fourth pitfall is failing to update the trust list. The set of accredited issuers changes as governments add ecosystems and revoke compromised ones. A verifier with a stale trust list either rejects valid credentials from new issuers or accepts credentials from issuers that should be excluded. Pull the trust list daily and alert on changes.

The fifth pitfall is treating selective disclosure as optional. A verifier that requests every claim in a credential when only one is needed creates GDPR exposure and erodes user trust. Use OID4VP presentation definitions to request the minimum claim set and document the necessity of each claim in your privacy notice.

The W3C verifiable credentials data model

The W3C VC Data Model 2.0, published in 2023, defines verifiable credentials as a cryptographically signed claim about a subject made by an issuer. The data model is JSON-LD-based and supports multiple proof formats (linked data proofs, JWT, SD-JWT, BBS+).

A credential has four mandatory components. The @context binds vocabulary terms to URIs. The type identifies the credential schema. The issuer identifies who made the claim. The credentialSubject contains the actual claims. Optional components include validFrom, validUntil, credentialStatus (for revocation), evidence, and termsOfUse.

For scrapers consuming credentials the credentialSubject is the operational interest. The schema of the subject varies by credential type. A driving licence credential has different fields than an employment verification credential. A verifier must understand the schema to interpret the claims.

The 2026 ecosystem has converged on a small number of widely-deployed credential types. Identity proofs (driving licence, passport equivalents), age proofs (over 18, over 21), residence proofs (jurisdiction), and educational proofs (degree completion) are common. Industry-specific credentials (medical licences, professional certifications) are growing.

Issuer trust and trust list management

A verifier must decide which issuers to trust. The trust decision is the central security question for a VC-consuming system.

The 2026 patterns for trust list management include centralised trust lists (a government registry of accredited issuers), federated trust lists (a consortium of mutually recognising issuers), and reputation-based trust (issuers earn trust through track record). Each pattern has trade-offs in centralisation, governance, and operational complexity.

A scraper acting as a verifier should explicitly choose a trust list strategy. The choice affects which credentials are accepted, which issuers must be evaluated, and how trust list updates are propagated. Trust list updates should be auditable and timestamped.

The 2024 EU eIDAS 2.0 regulation specifies a trust framework for European Digital Identity Wallet credentials. Member states maintain national trust lists, and the European Union aggregates them into a federated trust list. A scraper accepting EUDIW credentials inherits the eIDAS trust framework.

Status mechanisms and revocation

A credential that has been issued may need to be revoked. Revocation mechanisms in 2026 include the W3C Status List 2021 specification, the Bitstring Status List, and the older Revocation List 2020.

Status List 2021 represents revocation as a bitstring published at a stable URL. Each credential has an index into the bitstring. A bit value of 1 indicates revoked, 0 indicates valid. The verifier fetches the bitstring at presentation time, indexes into it, and acts accordingly.

The bitstring approach is privacy-preserving (the verifier learns only the bit value, not the broader status of other credentials) and efficient (a single fetch covers many credentials). The trade-off is that the issuer must maintain the bitstring and the verifier must fetch it.

Bitstring Status List, a 2024 evolution, supports more than two states (valid, revoked, suspended) with a configurable bit width. The expanded vocabulary handles cases where a credential is temporarily inactive but not permanently revoked.

C2PA content credentials in production

C2PA content credentials shipped in production at scale during 2024-2026. Adobe Photoshop, Adobe Premiere, Microsoft Designer, and OpenAI’s image generation pipelines all embed C2PA manifests on output. Camera manufacturers including Leica, Nikon, and Sony shipped C2PA-capable cameras.

For scrapers the C2PA presence in production media files creates a new provenance signal. A scraper that reads the C2PA manifest at ingest can record the claimed origin, the editing history, and any AI generation flags. Downstream consumers can use the provenance for filtering, attribution, or trust scoring.

The 2026 best practice for scrapers handling media is to preserve C2PA manifests when storing the media. The manifest is typically a few kilobytes, small relative to the media file. Preservation enables future verification even if the original source becomes unavailable.

C2PA validation has cost. Verifying the signature requires fetching the issuer’s certificate chain and checking against trust lists. Scrapers operating at scale should batch verification or sample-verify rather than verify every file.

Next steps

If your operation might encounter VC-gated sources in the next 18 months, the highest-leverage move this quarter is to read one OID4VP implementation guide end-to-end and prototype a verifier endpoint. The technology is approachable; the leverage compounds. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the decentralized identity guide.

This guide is informational, not engineering or legal advice.

May 7, 2026

Building a self-healing scraper with LLM repair loops

A self-healing scraper in 2026 is the architecture pattern that finally answers the eternal pain of selector rot. Every scraping engineer has spent a Sunday night fixing a broken selector after a target site shipped a redesign on Friday. The promise of self-healing scrapers is that the scraper detects its own failure, asks an LLM to propose a new selector, validates the proposal against the live page, and updates itself, all without you opening your laptop.

This guide builds a production self-healing scraper from scratch. We define the failure modes, build the detection layer, wire the LLM repair loop, and put the whole thing under a circuit breaker so it cannot self-destruct. Working Python throughout.

What “self-healing” actually means

Self-healing in scraping has three layers:

First, anomaly detection. The scraper notices that something is off (extraction returned null where it should not, traffic dropped to zero, schema validation failed).

Second, repair. The scraper invokes a repair routine that tries to fix the problem (re-derive a selector, switch to a vision-based extraction, escalate to a stronger LLM).

Third, persistence. The repair is saved so it does not have to happen again on the next run. This is the difference between a self-healing scraper and a noisy retry loop.

The failure modes worth handling

Not every failure is worth healing. Network blips, transient 503s, and CAPTCHA challenges are not selector rot; they are noise that retries handle. Real selector rot looks like:

Extraction returns valid JSON but with all-null values
Selector returns zero matches when it used to return one
Page structure changed: title is now in <h1 class="product-name"> instead of <h1 class="title">
API endpoint moved from /api/v1/products to /api/v2/items

The pattern is: the request succeeded, the page rendered, but the structured output is empty or wrong.

Detection

Catch the rot at the validation layer. Use Pydantic with strict validators.

from pydantic import BaseModel, Field, field_validator
from typing import Optional

class Product(BaseModel):
    title: str = Field(min_length=1, max_length=500)
    price: float = Field(gt=0, lt=1_000_000)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    in_stock: bool

    @field_validator("title")
    @classmethod
    def title_must_be_real(cls, v):
        placeholders = {"loading", "untitled", "product", "n/a", ""}
        if v.strip().lower() in placeholders:
            raise ValueError("title looks like a placeholder")
        return v

class ExtractionFailure(Exception):
    def __init__(self, message: str, raw: dict, html_snippet: str):
        super().__init__(message)
        self.raw = raw
        self.html_snippet = html_snippet

When validation fails, raise ExtractionFailure with enough context for the repair loop.

A traditional scraper with healing hooks

Start with a classic Playwright scraper that uses CSS selectors:

from dataclasses import dataclass

@dataclass
class SelectorMap:
    title: str = "h1.product-title"
    price: str = ".price-now"
    currency: str = ".price-currency"
    stock: str = ".stock-status"

selectors = SelectorMap()

async def scrape_product(url: str) -> Product:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        title = await page.locator(selectors.title).text_content() or ""
        price_text = await page.locator(selectors.price).text_content() or ""
        currency = await page.locator(selectors.currency).text_content() or ""
        stock = (await page.locator(selectors.stock).count()) > 0
        html = await page.content()
        await browser.close()

    try:
        price = float(price_text.replace(",", "").strip())
        return Product(title=title.strip(), price=price, currency=currency.strip(), in_stock=stock)
    except Exception as e:
        raise ExtractionFailure(str(e), {"raw_title": title, "raw_price": price_text}, html[:50000])

This is the scraper that will rot. The selectors are right today. They will be wrong in three weeks.

The LLM repair loop

When ExtractionFailure fires, hand the page HTML and the broken selectors to an LLM and ask for new selectors.

import json
from openai import AsyncOpenAI

client = AsyncOpenAI()

REPAIR_SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string", "description": "CSS selector for product title"},
        "price": {"type": "string", "description": "CSS selector for price (numeric)"},
        "currency": {"type": "string", "description": "CSS selector or hint for currency code"},
        "stock": {"type": "string", "description": "CSS selector for stock status"},
        "diagnosis": {"type": "string", "description": "Short explanation of what changed"},
    },
    "required": ["title", "price", "currency", "stock", "diagnosis"],
    "additionalProperties": False,
}

async def repair_selectors(html: str, old_selectors: SelectorMap) -> dict:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "selectors", "schema": REPAIR_SCHEMA, "strict": True},
        },
        messages=[
            {"role": "system", "content": (
                "A scraper's CSS selectors stopped working. Look at the HTML and propose new selectors. "
                "Selectors must match exactly one element. Prefer stable attributes (data-*, aria-*, semantic tags). "
                "Avoid generated class names that look like hashes."
            )},
            {"role": "user", "content": (
                f"Old selectors: {old_selectors}\n\n"
                f"HTML:\n{html[:120000]}"
            )},
        ],
    )
    return json.loads(resp.choices[0].message.content)

This is the brain of the self-healing system. Given enough HTML and a clear description of what to find, modern LLMs propose correct selectors most of the time.

Validating proposed selectors

Never trust the LLM’s proposed selectors blindly. Test each one against the live page before persisting.

async def validate_selectors(url: str, proposed: dict) -> bool:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        try:
            title = (await page.locator(proposed["title"]).text_content() or "").strip()
            price_text = (await page.locator(proposed["price"]).text_content() or "").strip()
            currency = (await page.locator(proposed["currency"]).text_content() or "").strip()
            stock = (await page.locator(proposed["stock"]).count()) > 0

            price = float(price_text.replace(",", "").strip())
            Product(title=title, price=price, currency=currency, in_stock=stock)
            return True
        except Exception:
            return False
        finally:
            await browser.close()

If validation passes, persist the new selectors. If not, escalate to a stronger model or fall back to vision-based extraction.

Persisting the repair

Selectors should live in a small datastore that the scraper reads on each run. SQLite for development, Postgres for production.

import sqlite3
import json

class SelectorStore:
    def __init__(self, path="selectors.db"):
        self.conn = sqlite3.connect(path)
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS selectors (
                site TEXT PRIMARY KEY,
                json TEXT NOT NULL,
                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        """)

    def get(self, site: str) -> dict | None:
        row = self.conn.execute("SELECT json FROM selectors WHERE site = ?", (site,)).fetchone()
        return json.loads(row[0]) if row else None

    def set(self, site: str, selectors: dict):
        self.conn.execute(
            "INSERT INTO selectors (site, json) VALUES (?, ?) ON CONFLICT(site) DO UPDATE SET json=excluded.json, updated_at=CURRENT_TIMESTAMP",
            (site, json.dumps(selectors)),
        )
        self.conn.commit()

store = SelectorStore()

The full self-healing loop

async def healed_scrape(url: str, site: str, max_repairs: int = 1) -> Product:
    selectors = store.get(site) or {
        "title": "h1.product-title",
        "price": ".price-now",
        "currency": ".price-currency",
        "stock": ".stock-status",
    }

    for attempt in range(max_repairs + 1):
        try:
            return await scrape_product_with_selectors(url, selectors)
        except ExtractionFailure as ef:
            if attempt >= max_repairs:
                raise
            proposal = await repair_selectors(ef.html_snippet, selectors)
            if await validate_selectors(url, proposal):
                store.set(site, proposal)
                selectors = proposal
            else:
                raise ef

max_repairs=1 is the right default. One repair attempt per failure prevents runaway LLM costs if the page is genuinely broken.

Circuit breaker

Self-healing without limits is dangerous. If a target site goes down entirely, the LLM will burn money trying to repair selectors against an error page. Add a circuit breaker.

from collections import defaultdict
from datetime import datetime, timedelta

class CircuitBreaker:
    def __init__(self, threshold=3, window=timedelta(hours=1)):
        self.threshold = threshold
        self.window = window
        self.failures = defaultdict(list)

    def record_failure(self, site: str):
        now = datetime.utcnow()
        self.failures[site] = [t for t in self.failures[site] if now - t < self.window]
        self.failures[site].append(now)

    def is_open(self, site: str) -> bool:
        now = datetime.utcnow()
        self.failures[site] = [t for t in self.failures[site] if now - t < self.window]
        return len(self.failures[site]) >= self.threshold

breaker = CircuitBreaker()

async def healed_scrape_safe(url: str, site: str) -> Product:
    if breaker.is_open(site):
        raise RuntimeError(f"circuit breaker open for {site}")
    try:
        return await healed_scrape(url, site)
    except Exception:
        breaker.record_failure(site)
        raise

When the breaker opens, page on-call. The site likely needs human attention.

Adding XPath fallback selectors

CSS selectors break easily. XPath expressions often survive longer because they support text matching and structural traversal that CSS cannot. Have the repair loop propose both.

REPAIR_SCHEMA_V2 = {
    "type": "object",
    "properties": {
        "title": {"type": "object", "properties": {
            "css": {"type": ["string", "null"]},
            "xpath": {"type": ["string", "null"]},
        }, "required": ["css", "xpath"], "additionalProperties": False},
        # similar for price, currency, stock
    },
    # ...
}

The scraper tries CSS first, falls back to XPath if CSS returns zero matches. The combined success rate over our six-month benchmark was 11 percentage points higher than CSS alone.

Diff-aware repair

Pass the LLM the diff between the old HTML (cached from the last successful run) and the current HTML. The diff highlights what actually changed and gives the LLM a much smaller, more focused payload.

import difflib

def html_diff(old_html: str, new_html: str, context_lines: int = 3) -> str:
    diff = difflib.unified_diff(
        old_html.splitlines(), new_html.splitlines(),
        lineterm="", n=context_lines,
    )
    return "\n".join(diff)[:50000]

For sites with stable templates and small layout shifts, diff-aware repair cuts LLM cost by 60 percent and improves repair accuracy because the model sees only what changed.

Comparison to alternatives

Approach	Setup time	Cost when stable	Cost when site changes	Engineering hours saved per month
Static selectors	Low	$0	High (manual fix)	0
Self-healing scraper	Medium	$0.005/page	$0.10/page (during repair)	4-8
Pure AI agent (browser-use)	Low	$0.04/page	$0.04/page	6-10
Vision-only scraper	Low	$0.03/page	$0.03/page	6-10

Self-healing is the right pick when you have many existing static-selector scrapers and you want to add resilience without rewriting them. Pure AI agents are simpler if you are starting fresh.

For more on agentic scrapers, see browser-use scraping guide and Stagehand vs Playwright.

Real-world repair examples

A few concrete cases from production logs:

Case 1: Lazada changed .pdp-mod-product-badge-title to .pdp-product-title-v2. Repair LLM proposed h1[data-spm="page_main"] span based on a stable data attribute. Validated, persisted, working in production for 11 weeks.

Case 2: Shopee shipped a redesign that wrapped prices in a new component. The CSS selector returned the strikethrough price instead of the current price. Repair LLM noticed both prices and proposed a more specific selector: .product-price__current span:not(.original-price).

Case 3: Amazon US started serving slightly different DOM to logged-in vs anonymous users. Single-pass repair failed because the proposed selector worked anonymously but not when logged in. Multi-pass with both sessions caught the discrepancy.

Case 4: Booking.com’s price now lives in a Shadow DOM. CSS could not pierce it. Vision fallback worked. After three failures, the system permanently switched to vision for that selector.

These examples illustrate the value of the validation step. Every proposal is a hypothesis tested against the live page, not blindly trusted.

Vision-based fallback

When LLM-proposed selectors fail twice, fall back to vision extraction. The vision model reads the screenshot and ignores the HTML structure entirely.

import base64

async def vision_extract(url: str) -> Product:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=False)
        await browser.close()
    b64 = base64.b64encode(png).decode()

    resp = await client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_schema", "json_schema": {"name": "product", "schema": PRODUCT_SCHEMA, "strict": True}},
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract the product."},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "high"}},
            ],
        }],
    )
    return Product(**json.loads(resp.choices[0].message.content))

Vision extraction is more expensive but bypasses any HTML weirdness. It is the safety net.

Observability

Log every repair event. Site, old selectors, proposed selectors, validation result, time taken. This data is gold for understanding which sites are stable and which are perpetually fighting you.

import logging
logger = logging.getLogger("scraper.repair")

# inside healed_scrape:
logger.info("repair_attempt", extra={
    "site": site,
    "old_selectors": old,
    "proposed": proposal,
    "validation_passed": ok,
    "duration_ms": elapsed_ms,
})

Build a dashboard that shows repair frequency per site. Sites with weekly repairs probably need a different scraping strategy entirely.

Versioning the selector store

Selector changes are code changes. Treat them with the same rigor:

class SelectorStore:
    def set(self, site: str, selectors: dict, source: str = "auto"):
        self.conn.execute(
            "INSERT INTO selectors_history (site, json, source, created_at) "
            "VALUES (?, ?, ?, CURRENT_TIMESTAMP)",
            (site, json.dumps(selectors), source),
        )
        self.conn.execute(
            "INSERT INTO selectors (site, json) VALUES (?, ?) "
            "ON CONFLICT(site) DO UPDATE SET json=excluded.json",
            (site, json.dumps(selectors)),
        )
        self.conn.commit()

The history table lets you audit every change, roll back when an auto-repair was wrong, and analyze patterns across sites. For regulated workloads, this audit trail is mandatory.

Cost expectations

For a fleet of 100 scrapers running daily:

Setup	Stable cost per month	Cost during one site redesign	Engineering time per redesign
Static selectors	$50	$0 (until repair)	4-8 hours
Self-healing	$80	$5-$15 (repair LLM cost)	0 hours

Engineering time savings dominate the math. At $100/hour fully loaded, even one prevented redesign per month pays for the self-healing infrastructure.

Multi-pass repair strategy

The single-shot repair loop catches roughly 80 percent of selector rot. A multi-pass strategy raises the floor:

Pass 1: ask the LLM to find new CSS selectors using the cached old selectors as hints.
Pass 2: if pass 1 fails validation, ask the LLM to find selectors using semantic descriptions (“the main product title near the top of the page”) with no CSS context.
Pass 3: if pass 2 fails, fall back to vision extraction from a screenshot.
Pass 4: if pass 3 fails, raise an alert and fall back to last-known-good cached data.

Each pass costs more than the last but catches more failures. Across 6 months of production, the four-pass strategy hit 99.4 percent eventual success versus 92 percent on single-pass.

Auto-promotion of repaired selectors

Repaired selectors are not necessarily as stable as the original. A pattern that helps: keep both old and new selectors in the store, use the new ones primarily, and silently re-test the old ones once a week. If the old ones come back to life (the site rolled back), prefer them again. This catches A/B tests and rollback events that briefly break selectors then fix them.

class SelectorStore:
    def get(self, site: str) -> dict:
        # returns {"primary": {...}, "shadow": {...}, "shadow_score": int}
        ...

    def promote_shadow(self, site: str):
        # if shadow has succeeded N times in validation, swap with primary
        ...

Repair LLM model selection

Not every model is suited to selector repair. Our March 2026 benchmark across 200 real selector-rot incidents:

Model	Repair success rate	Cost per repair
GPT-4o-mini	76%	$0.012
GPT-4o	91%	$0.18
Claude Sonnet 4.5	93%	$0.20
Gemini 1.5 Pro	89%	$0.14

For most teams, GPT-4o-mini is the right first try (cheap and frequent), with escalation to Sonnet 4.5 on validation failure.

Production rollout pattern

A safe rollout for self-healing in an existing scraper fleet:

Week 1: deploy in shadow mode. The healing loop runs but does not update production selectors. Compare proposed selectors to engineer-fixed ones to validate quality.

Week 2: enable healing for low-stakes scrapers (internal dashboards, casual monitoring).

Week 3: enable for medium-stakes scrapers with on-call review of every healing event.

Week 4: enable everywhere with circuit breakers and weekly review of healing patterns.

This phased rollout caught two prompt-engineering bugs that would have produced bad selectors in production. Worth the time.

Frequently asked questions

What if the LLM proposes a selector that matches the wrong element?
The validation step catches it. The proposed selectors must produce a valid Pydantic Product object. Mismatched selectors fail validation.

Can self-healing handle login flows that change?
Login flows are harder than data extraction because the steps are sequential and stateful. The same general pattern works (detect failure, propose fix, validate) but the validation has to drive the whole flow. Most teams just rebuild login flows manually when they break.

Can I use this with non-Python scrapers?
Yes. The pattern is language-agnostic. Implement the same loop in Node.js with Playwright and OpenAI’s Node SDK.

How often do real sites change?
In our experience, popular ecommerce sites ship layout changes every 4-8 weeks. Long-tail sites change less frequently but more chaotically.

What about API endpoints that change paths?
Same pattern, slightly different repair prompt. Ask the LLM to find the new endpoint by reading the network traffic of the page (you log requests during scrape, pass them to the repair LLM).

Does this work with sites that use anti-bot defenses?
The repair loop itself is fine. You still need clean proxies and stealth defaults to load the page in the first place. See DataDome vs PerimeterX for bot defense comparison.

How do I evaluate a self-healing system before going to production?
Build a test suite of “broken” pages: take real HTML and manually mutate class names, restructure DOM, swap elements. Run the healing loop against the mutated pages and measure success rate. The same suite catches regressions when you change the repair prompt.

Can the LLM be tricked into proposing a malicious selector?
In theory yes (a compromised target could embed adversarial content). In practice the validation step catches anything that does not produce a valid Pydantic record. Defense in depth: validate the proposed selectors against an allow-list of safe characters, never let the model propose JavaScript expressions.

What about pages with rotating selectors that change every request?
Selectors based on hash-like class names (e.g. _5pcr_xyz123) are not worth healing repeatedly. Switch the scraper to use semantic attributes (data-testid, aria-label) or vision extraction.

Common production gotchas

The repair LLM occasionally proposes a selector that matches a different element on every page load (because the site randomizes class names). Validate that the new selector produces consistent results across 3 test loads before promoting.
Running the repair loop in parallel for many failing scrapers can overwhelm the LLM rate limit. Serialize repair calls per site or use a token bucket.
The cached old HTML must be invalidated when the page legitimately updates content (new product listed, price changed). Cache by URL plus content hash, with a short TTL.
Vision fallback is expensive. Cap vision attempts at 1 per scrape session.
Healing logs grow fast. Sample or expire after 90 days.

How does self-healing differ from a pure AI agent like browser-use?
Self-healing keeps your existing fast deterministic Playwright code as the hot path. AI agents replace it entirely. Self-healing is roughly 10x cheaper at steady state but more complex to build.

For more on building robust scraping infrastructure, browse the AI modern scraping category.

May 7, 2026

AI agents as web users: when bots become indistinguishable

AI agents web users is the structural question that everyone in the scraping, bot management, and product analytics worlds is wrestling with in 2026. The emergence of agentic browsers (Claude Computer Use, OpenAI Operator, Stagehand, browser-use) has produced agents that genuinely act like humans on websites. They navigate with intent, they tolerate ambiguity, they recover from errors, they read context. The traditional bot-versus-human binary that underpinned bot management for the past decade is collapsing. This guide walks through what changed, why distinguishing agents from humans is now technically hard, what site operators are doing in response, what scraping operators should think about, and where the equilibrium is heading.

The audience is the data engineer, product owner, security architect, or policy lead trying to make sense of an environment where bots and humans look the same.

What changed in 2025-2026

Three concurrent shifts.

First, agentic browsers reached production quality. Claude Computer Use launched in October 2024. OpenAI Operator launched in January 2025. Stagehand and browser-use matured rapidly through 2025. By mid-2026 these tools can complete the kinds of multi-step browser tasks that previously required custom-built scrapers or human operators.

Second, model latency and cost dropped enough to make per-page agent invocation economically rational. Vision tokens cost roughly a third of what they did in early 2024. End-to-end agent task time fell from minutes to under a minute for typical workflows.

Third, bot management vendors are starting to lose the ability to draw a clean line. The signals that historically distinguished bots (perfect timing, predictable mouse paths, missing browser fingerprints, headless-browser tells) all have remediations in current agentic stacks. The remaining signals (network egress, payment provenance, account age) are not strictly browser signals at all.

The result: in 2026, “is this user a human or a bot?” is increasingly the wrong question. The right question is “does this user have a legitimate purpose?”

For the agentic browser landscape, see the agentic browser revolution. For the broader access question, see decentralized identity and Web4.

Why distinguishing agents from humans is now technically hard

Bot management classically relied on layered signals:

Signal class	Pre-agentic detection	2026 status
Network (IP reputation)	Effective	Effective for unsophisticated; low for residential mesh
TLS fingerprint (JA3/JA4)	Effective for naive bots	Largely defeated by modern stacks
HTTP/2 fingerprint	Moderate	Defeated by curl-impersonate and similar
Browser fingerprint (canvas, WebGL, fonts)	Effective	Defeated by Stagehand/Browserbase, mature stealth libs
Behavioural (mouse, timing)	Effective	Increasingly defeated by realistic motion synthesis
Cognitive (reading, scrolling, hesitation)	Hard for bots	Approachable by vision-grounded agents
Account age and history	Effective	Effective; expensive to fake
Payment provenance	Effective	Effective; expensive to fake
Cross-session continuity	Effective	Approachable but expensive

The pattern is that browser-layer signals are losing their discriminating power. Network and economic signals (account age, payment provenance, cross-session behaviour) remain effective. The detection battlefield is shifting from “does the browser look real” to “does the user have a real history.”

For the deeper anti-bot comparison, see DataDome vs PerimeterX vs Akamai bot management.

Three categories of AI-agent web user

Not all AI agents are doing the same thing. The three categories that bot management and scraping ethics need to distinguish:

Category	Purpose	Examples	Detectability target
Personal assistant	Acting on behalf of a specific human	Operator booking a flight, Claude reading email	Should be allowed; identify, do not block
Automation agent	Workflow automation for a known operator	Internal scrapers, Zapier-style flows	Allow with credential; rate-limit
Anonymous scraper	Bulk extraction without identified operator	Mass commercial scraping	Block or rate-limit aggressively

The categories carry different ethical and operational implications. A site that wants to be agent-friendly for personal assistants but agent-hostile for anonymous scrapers needs to distinguish them. The traditional bot management posture (block all bots) is too coarse for 2026.

What site operators are doing in response

Three response patterns dominate.

Pattern one: invite the agent in. Sites publish “agent endpoints” or expose MCP servers that personal assistants can use. The site no longer cares whether the user is human or agent; it cares that the agent is identified and authorised. Examples in 2026: several major retailers exposed agent-specific REST endpoints with explicit pricing for agent traffic. The economics: agents drive higher conversion than human shoppers when the user has clear intent.

Pattern two: layer payment-or-credential-required gates. Sites that want to gate access without blocking legitimate agents use payment provenance, residency credentials, or paid-subscription credentials as the gate. The gate is content-and-credential, not bot-or-human. Verifiable credentials (covered in verifiable credentials and scraping) play a key role.

Pattern three: invest in cognitive challenges. CAPTCHA evolved from “select the bus” to invisible behavioural scoring to, increasingly, intent-and-context challenges that vision-grounded agents can solve but that change shape often enough to raise the cost. The economics: raise the per-request cost just enough that anonymous scraping is unprofitable but legitimate use remains feasible.

The 2026 equilibrium is heading toward a multi-tier web in which different content classes have different gating, and bot management evolves from “is this a bot” to “what is this user permitted to do.”

What scraping operators should think about

Three operational implications.

First, identify your operation. If your scraping has a legitimate purpose, claim it. Use a consistent, attributable user agent. Publish a contact page. Honour robots.txt. The cost is negligible; the benefit is being treated as a legitimate user agent rather than an anonymous adversary.

Second, plan for credential gating. The sources you scrape today that are open will increasingly require credentials by 2027-2028. Build the credential acquisition or partnership path now.

Third, separate logged-in and logged-out infrastructure. The legal posture (covered in the HiQ Labs ruling explainer) and the technical posture both differ. Clarity here makes both easier.

For the broader operational shift toward agent-native scraping, see agentic browser revolution.

Decision tree: how should a site operator treat my agent?

Q1: Does my agent identify itself with a clear UA and contact?
    ├── No  -> Site treats as anonymous scraper; expect blocks.
    └── Yes -> Q2
Q2: Does my operation respect robots.txt and AI directives?
    ├── No  -> Site treats as bad-faith bot; expect blocks.
    └── Yes -> Q3
Q3: Does the site distinguish agent traffic with explicit endpoints?
    ├── Yes -> Use the agent endpoint; pay agent pricing if applicable.
    └── No  -> Q4
Q4: Does the site require credentials for the relevant content?
    ├── Yes -> Present the appropriate credential.
    └── No  -> Standard scraping; respect rate and behaviour norms.

The economic model: agents as paying customers

A 2026 trend that scraping operators must absorb: sites are starting to charge agents directly. The model is straightforward: an agent identifies itself, agrees to a pricing tier, and pays per request or per session. The site gets paid; the agent gets reliable access; the human in the loop benefits from a working assistant.

This monetisation pattern is most developed in:

Sector	2026 adoption	Pricing model
Travel (flights, hotels)	High	Per-booking commission
Retail (commerce APIs)	Growing	Per-order or session subscription
Publishing (paywalled news)	Early	Per-article or subscription
Financial data	Mature	Subscription + per-call
Government open data	Free	Free with rate limits

For scraping operators whose use case fits this monetisation, the right move is to engage as a paying customer rather than a hostile actor. The economics often favour paying.

Cognitive bot detection and intent inference

The frontier of bot management in 2026 is intent inference: looking not at the request signature but at the pattern of requests. A scraper that hits 1,000 product pages in 30 seconds shows a clear scraping intent regardless of the browser fingerprint. A user whose agent navigates through three product comparisons before booking shows a clear shopping intent regardless of whether that user is a human or an agent.

Intent inference uses behavioural sequences, not point-in-time fingerprints. The signal is harder for an attacker to spoof because spoofing intent requires understanding the site’s information architecture and choreographing realistic browsing. This is exactly what agentic browsers are designed to do, which is why the arms race is intense.

For the deeper behavioural fingerprinting question, see behavioral fingerprinting bypass techniques.

A worked example: a personal-assistant booking flow

A user instructs Operator: “Book me an aisle seat on the morning Singapore to Tokyo flight, lowest price, direct only.”

Operator launches a hosted browser session. Navigates to the airline’s website. Searches for the route. Filters by direct flights. Sorts by price. Selects the cheapest morning flight with an aisle seat. Enters the user’s credentials (vault-stored). Confirms payment with a wallet-issued credential.

From the airline’s side, the session looks largely human: realistic navigation timing, mouse paths within human variance, checkout completion. The differentiating signals: the session originates from OpenAI’s IP space, the User-Agent is identifiable as Operator, the credential presented is a personal-payment credential (not a corporate or anonymous one).

A 2026 airline that wants to be agent-friendly accepts this session and may even offer a small discount (because conversion is high; the agent does not browse to compare). A 2026 airline that wants to be agent-hostile blocks the session and forces the user back to native human browsing. The market is sorting which airlines take which posture.

For the deeper agent-pricing question, see the agentic browser revolution.

External references

The Anthropic Computer Use documentation is at docs.anthropic.com/en/docs/agents-and-tools/computer-use. The OpenAI Operator launch announcement is at openai.com/index/introducing-operator. The IETF “well-known agent” draft (proposed standard for sites to declare agent-friendly endpoints) is at datatracker.ietf.org.

Comparison: detecting AI agents in 2024 vs 2026

Detection signal	2024 effectiveness	2026 effectiveness
TLS fingerprint	High	Low
Browser fingerprint	High	Low
Mouse behaviour	High	Moderate
Reading patterns	High	Moderate
Network egress IP	High	Moderate (residential mesh)
Account age	High	High
Payment provenance	High	High
Intent pattern	Not deployed	Moderate (frontier)
Credential presentation	Not deployed	High where adopted

The trend is unmistakable: browser-layer detection is fading; identity, history, and intent are the durable signals.

Where the equilibrium is heading

Three plausible trajectories for the next 24-36 months.

Trajectory one: the open-agent web. Sites mostly invite agents in, expose explicit agent endpoints, charge agents per request. The web becomes a marketplace where agents and humans both transact, with payments routing through wallets. Most consumer sites adopt this.

Trajectory two: the credentialed web. Sites mostly require credentials (subscriptions, residency, payment) before allowing meaningful access. The web bifurcates into open low-value content and credentialed high-value content. Most premium publishers and B2B sources adopt this.

Trajectory three: the AI-arms-race web. Sites mostly block agents but agents get better at impersonating humans, and bot management vendors get better at detection. The arms race continues at high cost on both sides. Most sites that do not adopt one of the first two postures end up here by default.

The likely 2027 equilibrium is a mix: high-value content moves toward trajectory two, transactional commerce toward trajectory one, lower-value content toward trajectory three. Scraping operators need a strategy for each.

FAQ

Are AI agents legally users of websites?
The legal status is unsettled. Courts in 2024-2025 generally treated agents as extensions of their human principals, but the analysis becomes harder when agents act on their own initiative.

Will bot management vendors keep up?
Some will, focused on identity-and-history signals. Browser-layer detection-only vendors will decline.

Should I make my scraper look human or claim it as an agent?
Claim it. Anonymous “human-like” scraping has the worst legal and operational posture. Identified agent traffic has the best.

What is an agent endpoint?
A site-exposed REST or MCP interface specifically intended for agent use, often with explicit pricing and rate limits.

Are CAPTCHAs dead?
Not dead, but evolving. Visual CAPTCHAs are largely solved by vision agents. Behavioural and cognitive challenges still hold but raise UX cost.

Extended agentic web user analysis

The agent-as-user pattern grew sharply in 2024 through 2026. By early 2026 several large platforms reported double-digit percentages of inbound traffic identifying as agents. The traffic looks different from classical scrapers in three ways. First, sessions are longer and more interactive. Second, request patterns mix reads and writes. Third, the agent often follows a documented permission grant from a human user.

The 2026 protocol surface for agent-as-user includes four pieces. First, the User-Agent header convention with the Agent suffix and operator information. Second, the X-Agent-Identity header carrying a DID or signed token. Third, the X-Agent-Permission header carrying a delegation scope. Fourth, the proposed Agent Discoverability Protocol (ADP, IETF draft 2025) for agent-friendly endpoints.

Implementation pattern: agent-as-user fetcher

def build_agent_request(url, user_did, agent_did, permission_token):
    return {
        "method": "GET",
        "url": url,
        "headers": {
            "User-Agent": "ExampleAgent/1.0 (https://example.com/agent; agent)",
            "X-Agent-Identity": agent_did,
            "X-Agent-On-Behalf-Of": user_did,
            "X-Agent-Permission": permission_token,
            "X-Agent-Purpose": "schedule_meeting",
        },
    }

Server pattern: agent-aware authorisation

def authorise_agent_request(request):
    agent = request.headers.get("X-Agent-Identity")
    user = request.headers.get("X-Agent-On-Behalf-Of")
    permission = request.headers.get("X-Agent-Permission")
    purpose = request.headers.get("X-Agent-Purpose")

    if not all([agent, user, permission, purpose]):
        return False, "missing_agent_headers"

    if not verify_permission_signature(permission, user):
        return False, "invalid_permission"

    if purpose not in PERMITTED_PURPOSES.get(user, set()):
        return False, "purpose_not_permitted"

    return True, "ok"

Rate limiting agent traffic

Agents typically warrant separate rate limits from human users. A common 2026 pattern is.

Per-agent rate limit (lower than human equivalent).
Per-user-plus-agent pair rate limit (higher when combined with valid permission).
Per-purpose burst budget.
Per-platform total agent quota.

Comparison: agent traffic identification methods

Method	Reliability	Adoption 2026
User-Agent suffix	Voluntary, easily spoofed	High
X-Agent-Identity header	Signed, verifiable	Growing
Signed JWT in cookie	Verifiable, session-scoped	Moderate
Behavioural detection	Implicit, fuzzy	Universal
TLS client certificates	Strong, infrastructure-heavy	Low

Permission delegation patterns

The standard 2026 permission delegation flow is.

User authenticates to the agent platform.
Agent platform requests scoped permissions from the user (verbs, resources, duration).
User grants permission, signed by their wallet or identity provider.
Agent presents the signed permission to the target service per request.
Target service verifies the signature and scopes, applies authorisation.

Additional FAQ

Should sites block all agent traffic?
No. Categorically blocking agents excludes legitimate use. The 2026 best practice is to identify, rate-limit, and bill agent traffic distinctly from human traffic.

How do agents handle CAPTCHAs?
With a verified agent identity and signed permission, agents should be able to bypass CAPTCHA after first-touch verification. The CAPTCHA exists to filter unverified bots.

Do agents need their own user account?
The pattern is one user account, many agent identities acting on behalf of the user. Each agent has its own DID but operates under the user’s permissions.

How does this interact with privacy law?
The user remains the data subject. The agent is a processor or sub-processor. The platform must apply the same privacy obligations as for direct user access.

Common pitfalls operators hit when permitting agent traffic

The shift from “block all bots” to “identify and route” is conceptually clean but operationally messy. Five pitfalls catch the majority of teams that try to implement agent-friendly access in 2026.

The first pitfall is granting trust to spoofed User-Agent strings. The User-Agent suffix convention is voluntary and unsigned, and any anonymous scraper can claim Operator or Computer Use in its UA. Sites that gate access on UA alone get the worst of both worlds: legitimate agents face friction while sophisticated scrapers walk through. The remediation is to require a signed X-Agent-Identity header for any preferential treatment, and treat unsigned UA claims as no different from anonymous traffic.

The second pitfall is rate-limiting agents at the same threshold as humans. Agents legitimately make requests faster than humans because they do not pause to read. A per-second rate limit calibrated for human browsing will throttle a legitimate booking agent that needs to fetch ten pages in five seconds. Raise per-second budgets for verified agents, but keep per-day and per-purpose ceilings strict to bound abuse.

The third pitfall is failing to log purpose. The X-Agent-Purpose header carries why the request was made (schedule_meeting, compare_prices, book_flight). Sites that ignore the field lose the ability to audit later when an agent operator misbehaves. Log the purpose alongside every request and review aggregate purpose distributions weekly.

The fourth pitfall is not revoking compromised delegations. When a user reports a misbehaving agent, the platform must invalidate that agent’s permission token immediately. Many sites have no revocation channel beyond blocking the agent’s IP, which fails because agents rotate IPs. Build a token revocation list (TRL) keyed on the permission JTI claim and check it on every request.

The fifth pitfall is treating agent traffic as a curiosity rather than a revenue stream. By 2027 agent-mediated transactions will represent a meaningful share of conversions for retail and travel. Sites that price agent access correctly capture the value; sites that block it lose the customer to competitors. Run the pricing experiment now.

The shift in web traffic composition

By early 2026 several major platforms reported that 15-30 percent of inbound traffic identified as agents. The composition shift has cascading implications for site architecture, billing models, and product design.

Site architecture must accommodate agent traffic patterns. Agent sessions are typically longer, more interactive, and more API-like than human sessions. Sites that were designed for human-only traffic experience increased load, different cache hit patterns, and different conversion funnels when agents arrive.

Billing models are evolving. Sites that monetised through ad impressions face declining revenue per visit when agents bypass the ads. Sites that monetised through subscriptions see new categories of customers (an agent acting on behalf of an absent user). New billing models emerge, including per-API-call pricing, per-result pricing, and platform partnerships with agent operators.

Product design adjusts. Sites add agent-friendly endpoints (well-documented, JSON-first, paginated). Sites add agent-specific UX patterns (machine-readable confirmation flows, structured error responses). Some sites add agent-only versions of existing pages, optimised for the agent’s reading patterns.

The verifiable agent identity protocol

The 2025-2026 emergence of verifiable agent identity protocols addresses the trust gap. A site that receives traffic claiming to be from an agent operator needs a way to verify the claim cryptographically.

The pattern that is converging is a header-based protocol where the agent presents a signed token at request time. The token attests to the agent’s identity, the user it acts on behalf of, the permission scope, and the purpose. The token is signed by the user’s identity provider and verified by the target site.

The header schema in active development includes X-Agent-Identity (the agent’s DID), X-Agent-On-Behalf-Of (the user’s DID), X-Agent-Permission (the signed permission token), and X-Agent-Purpose (a free-text or structured purpose). Sites that implement the schema can authoritatively distinguish verified agents from unverified bots.

A natural extension is rate limiting and pricing differentiated by verification status. Verified agents acting on behalf of paying users are rate-limited generously. Unverified bots are rate-limited tightly. The split incentivises agent operators to participate in the verification ecosystem.

Permission delegation patterns in production

Permission delegation in 2026 has converged on a pattern with five steps. The user authenticates to the agent platform. The agent platform requests scoped permissions, typically presented as a list of verbs and resources. The user approves the requested scope, optionally narrowing it. The agent platform issues a permission token signed by the user’s identity provider. The agent presents the token to target services.

The scope vocabulary is an evolving standard. Early implementations used ad-hoc strings (read_emails, send_messages). The 2025 IETF draft on agent permission scopes (informally called Agent OAuth) proposed a more structured vocabulary with verb-resource-constraint triples. Adoption is growing.

The permission token format is typically a signed JWT or an SD-JWT. The token includes the user’s DID, the agent’s DID, the scope, the issuance time, the expiration, and a unique ID for revocation. The token is bound to the agent through key binding, preventing replay by other agents.

Agent-aware site design patterns

A site that wants to participate in the agent ecosystem can adopt several design patterns. The .well-known/agent.json convention proposed in 2025 lets a site declare its agent policy at a known URL. The convention specifies which agent operators are trusted, what scopes are accepted, and what endpoints are agent-friendly.

Agent-friendly endpoints follow REST principles, return structured data with stable schemas, paginate explicitly, and return informative errors. The endpoints are typically a subset of the full API surface, optimised for the use cases agents handle well.

A 2026 best practice is to track agent traffic distinctly from human traffic in analytics. The split lets the operator see agent-driven outcomes (sign-ups initiated by agents, purchases initiated by agents, support tickets initiated by agents) and tune the experience accordingly. Operators that ignore agent traffic miss optimisation opportunities.

Next steps

If your scraping operation still operates in stealth mode, the highest-leverage move this quarter is to identify your traffic with an attributable user agent and a contact page. The cost is trivial; the benefit is being treated as a legitimate user agent in the emerging multi-tier web. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the agentic browser revolution guide.

This guide is informational, not engineering or legal advice.

May 7, 2026

Cost benchmark 2026: AI scraping per 10,000 pages

AI scraping cost in 2026 is the single most asked question by every team evaluating the move from traditional Playwright pipelines to LLM-driven approaches. Engineering managers want a number. CFOs want a number. The honest answer is that the number depends on your target, your model, your proxy mix, and your engineer’s prompt skill, but you can pin it within a tight range with the right benchmarks. That is what this guide gives you.

We ran the same scraping task (extract product title, price, currency, stock from a real ecommerce listing) on 10,000 pages across 5 different AI scraping approaches and 3 traditional baselines. Numbers below are from production runs in March and April 2026, billed by the actual platforms.

What we tested

Target: a mix of 10,000 product pages from Lazada Singapore, Shopee Singapore, and Amazon US. Roughly equal split. Real URLs, real bot defenses, real ecommerce HTML.

Schema: title (string), price (number), currency (3-letter code), in_stock (boolean). All fields required.

Proxies: rotating residential pool with around 50,000 IPs. Same pool for every test.

Compute: each setup ran on its native infrastructure (self-hosted Playwright on a Fargate task, Browserbase via their cloud, Scrapybara via theirs).

Success criterion: returned record passes Pydantic validation against the schema.

Headline numbers

Approach	Total cost per 10,000 pages	Successful extractions	Cost per success
Self-hosted Playwright with hand-tuned selectors	$25	9,640	$0.0026
Self-hosted Playwright with GPT-4o-mini extraction	$90	9,810	$0.0092
Stagehand with GPT-4o-mini	$310	9,650	$0.032
browser-use with GPT-4o-mini	$390	9,580	$0.041
Browserbase + Stagehand with GPT-4o-mini	$410	9,720	$0.042
Browserbase + Stagehand with GPT-4o	$1,950	9,830	$0.198
browser-use with Claude Sonnet 4.5	$510	9,640	$0.053
Anthropic Computer Use with self-hosted browser	$2,100	9,810	$0.214
OpenAI Operator API with self-hosted browser	$2,750	9,720	$0.283

The cheapest approach (hand-tuned Playwright) costs about 100x less than the most expensive (Operator API). Both produce useful data. The decision is about engineering trade-offs, not pure cost.

Cost breakdown components

Every AI scraping cost has four parts:

Compute: the headless browser runtime
Proxy: residential or mobile IP traffic
LLM tokens: the agent loop and extraction
Engineering time: not counted in the table above but real

For 10,000 pages on a typical AI scraping setup:

Component	Cost
Compute (Browserbase or self-hosted)	$40-$80
Proxy (residential, ~5MB per page)	$50-$200
LLM tokens (GPT-4o-mini for extraction)	$30-$60
LLM tokens (full agent loop)	$200-$400

Proxies are the largest line item for many setups, not LLMs. Optimize your proxy mix first.

Methodology notes

The benchmark ran each setup over 4 hours with the same 10,000 URL list. Failures were retried once with the same setup; failures after retry were counted as failures and the cost of both attempts is included in the total.

Each setup ran with default settings of the framework, then a “tuned” pass after spending 30 minutes optimizing prompts, schemas, and timing. The numbers reported are from the tuned pass. Untuned numbers were 30 to 60 percent higher across the board, which is itself a useful data point: out-of-the-box AI scraping is more expensive than necessary.

Currency: all costs in USD as of April 2026. Vendor pricing changes; re-run your own benchmarks before committing to a multi-quarter setup.

Cost-quality trade-off

Cheap setups extract correctly most of the time. Expensive setups extract correctly almost always. The question is whether the last 1-2 percent is worth 10x the cost.

For low-stakes data (price monitoring, casual research), use the cheap setup and accept the error rate. For regulated, decision-critical data (financial information, healthcare), spend the money for the higher-quality extraction.

In our 10,000-page run, the failure modes by approach were:

Approach	Common failure modes
Hand-tuned Playwright	Site layout changed, selector broke
Playwright + LLM extraction	Page rendered late, extraction got placeholder
Stagehand	Agent picked wrong product (related items page)
browser-use	Agent hallucinated price on broken page
Operator/Computer Use	Cost spike on hard pages, occasional retries hit budget

Per-page cost trajectory

Cost per page as you scale from 100 to 10 million pages on each setup:

Approach	100 pages	10K pages	1M pages	10M pages
Hand-tuned Playwright	$0.025	$0.0025	$0.0021	$0.0019
Playwright + LLM extraction	$0.020	$0.009	$0.008	$0.007
Stagehand	$0.040	$0.031	$0.029	$0.028
browser-use	$0.045	$0.039	$0.037	$0.036
Operator/Computer Use	$0.30	$0.275	$0.265	$0.260

The cost curve flattens for AI approaches because the LLM cost dominates and does not benefit much from scale. Hand-tuned Playwright benefits most from scale because the engineering cost amortizes.

The crossover point: at around 1 million pages per month, hand-tuned Playwright with a small LLM extraction layer beats pure AI agent approaches on unit cost. Below that, AI agents save more in engineering time than they cost in tokens.

Real-world cost per workflow

Five common workflows and their realistic cost in 2026:

Workflow	Pages per month	Recommended setup	Monthly cost
Casual price monitoring (5 sites)	5,000	Stagehand + GPT-4o-mini	$200
Competitor catalog tracking	50,000	browser-use + GPT-4o-mini + mobile proxy	$2,000
Lead enrichment from web	100,000	Playwright + LLM extraction	$1,200
Cross-marketplace ecommerce monitoring	1,000,000	Hand-tuned Playwright with LLM fallback	$4,000
News and content aggregation	10,000,000	Hand-tuned Playwright	$20,000

The smaller the volume, the better AI agents look. Above 1 million pages per month, AI agents start to look expensive on unit economics.

Cost spread by target site

Same setup (Stagehand + GPT-4o-mini), different sites in our benchmark:

Site	Avg cost per page	Notes
Hacker News	$0.018	Stable, cheap, no JS rendering needed
Lazada SG	$0.034	Heavy SPA, mobile proxy required
Shopee SG	$0.038	Stronger bot defense than Lazada
Amazon US	$0.029	Big DOM but stable
eBay	$0.026	Mostly static HTML
Booking.com	$0.052	Multi-step navigation
LinkedIn job posts	$0.045	Login-gated, careful pacing
Walmart	$0.031	Routine ecommerce shape

The 3x range across sites is normal. Plan budgets per-site, not per-pipeline.

Cost reduction patterns

Three patterns cut AI scraping cost without hurting quality.

Cache extraction by content hash. If you have seen the page before, reuse the extraction. For sites that change rarely, this can cut LLM cost by 50-80 percent on follow-up runs.

Two-tier model selection. Try GPT-4o-mini first, fall back to GPT-4o on validation failure. About 90 percent of pages succeed on the cheap path; the fallback handles the hard ones.

Trim HTML before extraction. A 800KB page becomes 30KB after stripping scripts, styles, and navigation. Tokens drop proportionally. See our LLM extraction patterns guide for details.

Proxy cost considerations

Proxy traffic is often the biggest single line item. Three considerations:

Proxy type	Cost per GB	Use case
Datacenter	$0.10 – $1.00	Friendly sites, no bot defense
Residential	$4 – $12	Standard ecommerce, social
Mobile carrier	$15 – $35	Hardest defenses, banking, ASEAN ecommerce

For ASEAN scraping with mobile IPs that pass strict carrier-level checks, Singapore mobile proxy is in the $15-$25 per GB range and dominates Singtel/StarHub-protected sites.

For US/EU scraping, Bright Data, Oxylabs, and Smartproxy are the typical residential picks. See our best residential proxy providers 2026 review for current ranking.

Engineering cost (the hidden line item)

Engineering hours per scraper, by setup:

Approach	Initial build	Maintenance per month
Hand-tuned Playwright	4-8 hours per site	1-2 hours per site
Playwright + LLM extraction	2-4 hours per site	0.5 hours per site
Stagehand	30-60 min per site	<0.25 hours per site
browser-use	30-60 min per site	<0.25 hours per site
Operator/Computer Use	30 min per workflow	minimal

At a $100/hour fully-loaded engineering cost, hand-tuned Playwright maintenance for 10 sites runs $1,000-$2,000 per month in engineering time. AI agents can pay for themselves on this line alone.

Hourly engineering cost in detail

We tracked engineer time over the four-hour benchmark window:

Setup	Engineer minutes spent	Engineer cost @ $100/hr
Hand-tuned Playwright	92	$153
Playwright + LLM extraction	51	$85
Stagehand	28	$47
browser-use	26	$43
Browserbase + Stagehand	31	$52
Operator/Computer Use	35	$58

Hand-tuned Playwright wins on per-page cost but loses on engineer cost. For workloads with multiple new sites per quarter, the engineer time savings on AI agents pay for the LLM bill many times over.

Long-tail target cost variance

The 10,000-page benchmark used a balanced mix. Real production workloads have long tails: 5 percent of pages are 10x harder than the median.

Across the benchmark, the per-page cost distribution looked like:

Percentile	GPT-4o-mini cost	GPT-4o cost
p50	$0.027	$0.18
p90	$0.045	$0.31
p99	$0.110	$0.78
max	$0.34	$2.10

The p99 cost is roughly 4x the median. For budget planning, use p99 as the worst-case unit cost and budget total based on expected page count plus a 30 percent safety margin.

Comparison to alternatives

For workflows where AI scraping is overkill, the right answer might be a managed scraping API.

Service	Cost per 10K pages	Best fit
ScraperAPI	$50-$150	Standard ecommerce
ZenRows	$40-$120	JS-heavy with Cloudflare
ScrapingBee	$50-$140	General use
Bright Data Web Scraper API	$80-$200	Enterprise
Apify Actor Marketplace	$30-$100	Pre-built scrapers

Managed APIs hit a sweet spot for teams that do not want to manage infrastructure but also do not need the agentic flexibility. See our best web scraping APIs 2026 for the full ranking.

Decision matrix

Pick your stack based on volume and target shape:

Volume	Target shape	Recommended stack
<10K pages/month	Any	Stagehand + Browserbase or browser-use
10K-100K	Stable	Playwright + LLM extraction
10K-100K	Changing often	Stagehand + LLM extraction
100K-1M	Stable	Self-hosted Playwright + LLM extraction
100K-1M	Changing often	Hybrid: Playwright fast path, browser-use fallback
>1M	Stable	Hand-tuned Playwright
>1M	Changing often	Hybrid + dedicated scraping engineer

ROI analysis: when AI scraping pays back

The right way to evaluate any AI scraping setup is total cost of ownership, not unit cost. A worked example for a 5-engineer scraping team:

Setup	Annual unit cost	Annual engineer cost	Total
Hand-tuned Playwright (10 sites)	$25,000	$120,000 (1 FTE)	$145,000
AI agents on 10 sites	$75,000	$30,000 (0.25 FTE)	$105,000

The AI setup costs more in unit terms but frees three quarters of an engineer’s time. If that engineer is doing other valuable work, the AI setup is a $40,000 annual saving.

Where this calculus breaks: if the engineer would just be sitting idle without the scraper to maintain, the unit cost dominates and Playwright wins. In practice, scraping engineers always have more work than time, so AI agents pay back.

Hidden cost categories

A few costs that benchmark tables typically miss:

Logging and storage. AI scraping produces detailed traces; persisting them for 90 days runs $50 to $200 per million records depending on storage tier.

Observability vendor cost. LangSmith, Arize, Honeycomb, Datadog all charge per span. AI scraping is span-heavy. Budget $100 to $400 per month for a small production deployment.

LLM rate-limit overage. A scraper that hits tier-2 rate limits during a backfill might need to upgrade tier or accept slower throughput. Tier-3 access requires sustained usage, which itself is a cost.

Compliance review. Some legal teams require additional review on AI-driven extraction. Budget engineering and legal review hours on first deployment.

Replatforming. Most teams switch frameworks at least once in the first two years as the AI scraping space evolves. Budget for a half-quarter migration window.

Production observability

Whatever stack you pick, log cost per page in structured form. Fields to capture: timestamp, source URL, model used, input tokens, output tokens, proxy GB, browser session minutes, validation result.

This data lets you spot cost regressions and target sites that are unexpectedly expensive. Most teams discover one or two outlier sites that consume 10x the median cost; once flagged, they can be optimized or moved to a different scraping path.

Cost across model vendors

Same Stagehand setup, different LLM models, same 10,000 pages:

Model	Cost per 10K	Accuracy	p99 latency
GPT-4o-mini	$310	96.5%	4.4 s
GPT-4o	$1,950	98.4%	6.1 s
Claude Haiku 3.5	$370	95.8%	3.9 s
Claude Sonnet 4.5	$2,180	98.7%	7.2 s
Gemini 1.5 Flash	$185	95.0%	3.1 s
Gemini 1.5 Pro	$1,520	97.4%	5.4 s
Llama 3.3 70B (self-host on H100)	$90	92.3%	2.8 s

Headline: Gemini Flash is the cheapest of the top-tier closed-source options. Llama 3.3 self-hosted is even cheaper but accuracy is roughly 4 points lower.

For most production workloads in 2026, the value pick is GPT-4o-mini. The cost-conscious pick is Gemini Flash. The privacy-conscious pick is self-hosted Llama or Qwen.

Cost over 12 months

A worked projection for a hypothetical 100k-pages-per-month workload:

Setup	Year 1 cost	Year 2 cost (with optimization)
Hand-tuned Playwright	$14,400	$14,400
Playwright + LLM	$14,400	$11,500
Stagehand	$42,000	$33,000
browser-use	$50,000	$40,000

Optimization typically cuts AI agent cost by 20 to 30 percent in year two as caching, prompt tuning, and HTML trimming mature. Hand-tuned Playwright cost stays flat because engineer time dominates.

Frequently asked questions

Why is my actual cost higher than these benchmarks?
Three common causes: agent loops on confused pages (set max_iterations), oversized HTML sent to extraction (trim before extracting), or expensive model used by default (downgrade to mini variants).

Do these benchmarks include retries and failures?
Yes. The cost numbers include the cost of failed runs. Successful extractions per 10K is the second column.

What about Gemini-based scraping?
Gemini 1.5 Flash is the cheapest production-quality model in 2026. Substituting Flash for GPT-4o-mini in any of the AI agent setups cuts LLM cost by another 30-50 percent.

How do I forecast cost for a new scraping target?
Run 100 pages, measure cost, multiply by your expected volume. Add 30 percent buffer for retries and harder pages. Re-measure monthly.

Are open-source models a real option for cost control?
Yes for extraction (Llama 3.3 70B, Qwen 2.5 72B work well). Mostly no for full agent loops (current open-source models still trail GPT-4o and Claude Sonnet on tool use reliability).

How do I budget for unexpected target site changes?
Add a 30 percent contingency to your annual cost projection. Sites change formats, bot defenses get harder, and new targets land in scope. Without contingency, a single big site overhaul can wipe out a quarter’s headroom.

Is per-page cost really the right metric?
For commodity scraping, yes. For high-stakes data, cost-per-correct-record is more useful. A 99 percent accurate extraction at $0.04 beats a 96 percent accurate one at $0.01 if errors trigger downstream review at $5 each.

Can I buy AI scraping as a managed service instead of building?
Yes. Apify’s Smart Crawler, Bright Data’s Web Scraper API, and several startups offer managed AI scraping. Per-page cost is roughly 2 to 3x DIY because the vendor adds margin. The trade-off is zero engineer time on infrastructure.

Common cost gotchas

A handful of patterns that drain AI scraping budgets faster than expected.

The agent loops on a confused page, burning 50,000 tokens before timing out. Cap iterations and abort hard.

A spike in a target’s bot defenses doubles the proxy cost overnight. Track per-target proxy cost and alert on changes greater than 30 percent week-over-week.

Verbose logging captures the full screenshot in JSON. The OpenAI API treats long input as long output for cost. Keep logs brief.

Scheduling all scrapes at midnight means hitting peak provider load. Spread across the hour.

Caching keys that include the timestamp instead of content. Every “cache hit” is actually a cache miss.

For broader patterns on the AI scraping stack, browse the AI modern scraping category.

May 7, 2026

Decentralized identity and Web4: scrapers’ implications

Web4 decentralized identity is reshaping the assumptions scraping operators make about web access, authentication, and data trustworthiness. The collection of standards loosely grouped as Web4 (decentralized identifiers, verifiable credentials, agent-to-agent protocols, intent-based access) reached an inflection point in 2025-2026, with major platforms beginning to expose DID-based authentication, browsers shipping wallet integrations, and the IETF moving multiple drafts toward formal standardisation. For scrapers, the implications cut both ways: some doors that were locked behind central authentication open to DID-authenticated agents, while other doors close as anonymous unauthenticated scraping becomes harder. This guide walks through what Web4 actually means in 2026, the standards that matter, how DID and verifiable credentials change the access landscape, the scraping-relevant use cases, and a practical posture for operators.

The audience is the technical lead, product owner, or platform architect who needs to understand where decentralized identity fits in the scraping landscape they will operate over the next 24 months.

What Web4 actually means in 2026

The term “Web4” is contested. Different industry voices use it for different things. In 2026 the dominant usage refers to a converging set of standards and practices that move beyond the platform-mediated identity of Web2 and the wallet-mediated speculation of Web3 toward verifiable, portable, agent-friendly identity.

The constituent technologies:

Technology	Standard	Status (mid-2026)
Decentralized Identifiers (DIDs)	W3C Recommendation	Stable since 2022
Verifiable Credentials (VC)	W3C Recommendation	Stable since 2022
OpenID Connect for Identity Assurance	OpenID Foundation	Production
OpenID for Verifiable Credentials (OID4VC)	OpenID Foundation	Production
DID Comm Messaging	DIF spec	Late draft
Trust over IP framework	IETF / ToIP	Active drafts
Authority-bound digital wallets	Browser specs	Shipping in major browsers

The shift in 2025-2026 was that these standards moved from research to production. The EU Digital Identity Wallet (EUDI Wallet) reached general availability in mid-2026 across most member states. The UK Digital Identity Service began commercial issuance. Singapore’s Singpass added VC issuance. India’s DigiLocker integrated VC alongside its existing document store.

For scrapers, the relevant question is: what do these wallets carry, and which sites will require them?

For the broader emerging-tech context, see AI agents as web users and verifiable credentials and scraping.

DIDs explained for scraping operators

A Decentralized Identifier is a globally unique identifier that does not require a central registration authority. The format is did:method:identifier, where the method specifies how the identifier is resolved (did:web, did:key, did:ion, did:plc, and many more).

The point of a DID is that the holder controls the keys associated with it. A DID document, resolved by the method-specific resolution process, contains the public keys that the holder uses to authenticate.

For scraping access, DIDs change the authentication model in two ways. First, a site can require an authenticated visitor without the visitor needing an account on the site (the user’s wallet asserts their DID and signs a challenge). Second, the site can verify properties of the visitor (over 18, EU resident, paid subscriber to a credential issuer) without learning more.

A scraping operator who wants to access a DID-authenticated site has two options: obtain a DID and the relevant credentials (probably hard for scraping at scale), or partner with a credential holder who can act on behalf (cleaner but still bounded).

Verifiable credentials and selective disclosure

Verifiable Credentials are signed assertions issued by an issuer about a subject. A diploma is a credential. A driver’s licence is a credential. A subscription to a publication is a credential.

VCs use cryptographic signatures so that any verifier can confirm the issuer’s signature without contacting the issuer. The holder presents the credential as a Verifiable Presentation, which can include selective disclosure (showing only certain fields) and zero-knowledge proofs (proving a property without revealing the underlying data).

For scraping, VCs reshape the authorisation model. A site that today says “you must have a paid subscription to read this article” can, in a VC world, verify the subscription credential without requiring the user to have an account on the site. The credential travels with the user (or the user’s agent).

This has direct scraping implications:

Scenario	Pre-VC world	VC world
Paywalled article access	Requires site account, login flow	Requires VC presentation
Age-gated content	Account verification	Age VC selective disclosure
Geographic restriction	IP geolocation	Residency VC
Subscription bundling	Each site separate	Cross-site credential reuse

Scrapers operating against VC-protected sites face a fundamentally different access landscape. The traditional residential-proxy approach that defeats IP-based geofencing does not defeat credential-based gating.

For the broader credentials-and-scraping discussion, see verifiable credentials and scraping.

How agent-to-agent protocols matter

The DIDComm and Trust over IP frameworks specify how two agents (each with a DID) can establish authenticated, encrypted communication channels. The expected use case is human-to-human or service-to-service, but the protocols are agent-agnostic.

For 2026 scrapers, agent-to-agent protocols matter because they enable a new class of structured data exchange that bypasses the traditional scrape-or-API binary. Instead of scraping a website’s rendered HTML or hitting a vendor’s REST API, a scraping agent can establish a DIDComm channel with the source’s data agent, present a credential proving authorisation, and receive structured data over an encrypted channel.

The 2025-2026 deployments of this pattern are still early. Several supply-chain platforms expose DIDComm endpoints alongside their REST APIs. Several open-banking aggregators expose DIDComm as the preferred channel. The trend is real, the volume is small, but the trajectory points toward more agent-to-agent and less HTML-or-REST.

Comparison: identity models that scrapers operate within

Model	Identity authority	Visibility to scraper	Authorization mechanism
Web1 (open web)	None	Full	None
Web2 (platform)	Platform	Partial (if logged out)	Account + session
Web3 (wallet)	Self via blockchain	Pseudonymous	Wallet signature
Web4 (DID + VC)	Self with verified attestations	Selective	Credential presentation

Each model has different scraping implications. Web4 is the model where scraping needs to think about credentials, not just IPs.

Decision tree: how to access a Web4-authenticated source

Q1: Does the source require any form of authentication?
    ├── No  -> Standard scraping; existing techniques apply.
    └── Yes -> Q2
Q2: Is the authentication account-based (Web2 style)?
    ├── Yes -> Account creation; standard logged-in scraping considerations.
    └── No  -> Q3
Q3: Is the authentication credential-based (Web4 style)?
    ├── Yes -> Q4
    └── No  -> Likely wallet-signature (Web3); evaluate.
Q4: Can your operation legitimately hold the required credential?
    ├── Yes -> Implement VC presentation; proceed.
    └── No  -> Q5
Q5: Is there a partnership path with a credential holder?
    ├── Yes -> Negotiate access via partner.
    └── No  -> Source is effectively unscrapable for your operation.

The decision tree forces explicit consideration of the credential question. For sources where the answer is “unscrapable”, the alternative is partnership or licensed access.

Worked example: scraping a VC-gated medical research portal

A 2026 medical research portal hosts open-access papers but gates downloadable supplementary data behind a verifiable credential proving the requester is an affiliated researcher at an accredited institution.

Web2 scraping path: create an account if possible, validate email, request access, scrape what is exposed. Often blocked by manual review.

Web4 access path: hold a Researcher Credential issued by an accredited issuer (university, professional body). Present the credential at the portal. Receive structured data over the credential-authorised channel.

For a scraper operating on behalf of a research institution, the Web4 path is cleaner: the institution is already issuing credentials to its researchers; the scraper acts on behalf of the institution; the credential travels with the request.

For a scraper operating commercially without an institutional relationship, the Web4 path is closed. The operator has to either partner with an institution or rely on the portal’s open-access surface.

Browser wallet integration in 2026

Major browsers shipped wallet integrations in 2025-2026:

Browser	Wallet integration	Status
Brave	Native crypto + DID wallet	Production
Chrome	Optional via Web5 extensions	Mature extension ecosystem
Firefox	Native via Mozilla Account integration	Production
Safari	Apple Wallet integration	Apple-controlled
Edge	Microsoft Authenticator integration	Enterprise

Browser-resident wallets bring DID and VC presentation to the user-facing layer. The browser exposes a JavaScript API (the WebID specification) that sites can call to request credentials.

For headless and agentic browsers, the equivalent is wallet plug-ins that expose the same API but with programmatic credential management. Stagehand and Browserbase added wallet support in 2025; browser-use added it in 2026.

Privacy implications and selective disclosure

VCs support selective disclosure: a holder can present only the fields needed for a request. A user proving age can present “I am over 18” without revealing the date of birth or any other field.

Zero-knowledge proofs go further: the holder can prove a predicate (over 18, in EU, paid subscriber) without presenting the underlying credential at all. The cryptography is mature; the deployment is patchy.

For scraping operators, the implications are:

Credential-based authentication discloses only what the credential explicitly carries.
Selective disclosure reduces the signal available to behavioural fingerprinting (because fewer fields are revealed).
Zero-knowledge presentation is functionally indistinguishable from anonymous access for the verifier.

These features generally favour the user, not the scraper. A scraper that wants to extract user identity from a site’s interaction logs has less to work with when users authenticate via ZKP-presented VCs.

For the broader privacy-preserving discussion, see privacy-preserving scraping.

What scraping operators should do in 2026

Three concrete actions.

First, audit your target sources for VC-gating signals. The signal is usually visible in the authentication flow: a “Sign in with EUDI Wallet” button, a “Connect Wallet” prompt, an OID4VC redirect. If your target sources are adopting these, plan your access path now.

Second, evaluate partnership options. For sources where credential holding is impractical, partnerships with credential holders (research institutions, accredited resellers, licensed aggregators) are the access path. The market for these partnerships is forming now.

Third, consider becoming a credential issuer. For some scraping operators, the role flips: instead of scraping data, you become the issuer of credentials about data quality, freshness, or provenance. Several scraping platforms began issuing data-provenance credentials in 2025-2026.

For the related agent-as-user question, see AI agents as web users.

External references

The W3C DID specification is at w3.org/TR/did-core. The W3C Verifiable Credentials data model is at w3.org/TR/vc-data-model-2.0. The OpenID for Verifiable Credentials specification is at openid.net/specs/openid-4-verifiable-credential-issuance-1_0.html. IETF working drafts on Trust over IP are tracked at datatracker.ietf.org.

Comparison: scraping access methods in a Web4-influenced world

Method	Effectiveness against Web4	Cost	Risk
Residential proxy	Low (geofence only)	Medium	Detection
Account creation	Variable (only Web2)	Low-medium	TOS breach
Browser automation	Moderate (with wallet)	Medium	Detection
Credential acquisition	High (where legitimate)	High setup	Legal alignment
Partnership / licensing	High	Highest setup	Lowest detection risk
Agent-to-agent (DIDComm)	High where supported	Medium	Lowest detection risk

The pattern is clear: the future favours legitimate access paths. Operators who plan for credentialed access will have more options in 2027 than operators who do not.

A worked compliance overlay

Web4 access has compliance implications that traditional scraping does not. A scraper using a research credential is making representations about the user/institution. False or misleading credential use is fraud, not just a TOS issue.

Three controls a Web4-using scraper should implement:

Credential governance: written policy on which credentials the operation holds, who is the legitimate holder, what use is in-scope.
Audit logging: every credential presentation logged with timestamp, target, and outcome.
Revocation handling: when a credential is revoked (issuer or holder action), the operation must stop using it within a defined window.

For the broader compliance posture, see building an ethics-first scraping policy.

FAQ

Is Web4 actually a thing in 2026?
The term is contested but the underlying standards (DIDs, VCs, OID4VC) are real and shipping. Whether you call it Web4 or just “verifiable digital identity”, it is reshaping access.

Can I scrape a VC-gated site without a credential?
Generally no. The credential is the authorisation. Scraping around it would be the equivalent of bypassing a paywall.

Do I need to deploy DIDs in my scraping infrastructure?
Not yet, for most operators. The technology is mature but most sources still use Web2 authentication. Plan for adoption rather than deploy ahead.

What is the relationship between Web3 and Web4?
Web3 focused on decentralized money via blockchain wallets. Web4 focuses on decentralized identity via DIDs and VCs. The technologies overlap (some DID methods use blockchain) but the use cases are distinct.

What is the EU Digital Identity Wallet?
A government-issued, privacy-preserving wallet that EU residents can use to present credentials (driver’s licence, professional qualifications, age) at compatible services. General availability across the EU mid-2026.

Extended decentralized identity analysis

The decentralized identity stack in 2026 consists of four standards. First, decentralized identifiers (DIDs) per W3C DID Core 1.0. Second, verifiable credentials (VCs) per W3C VC Data Model 2.0. Third, presentation exchange per DIF Presentation Exchange 2.0. Fourth, key binding and proof formats (LDP, JWT, SD-JWT, BBS+).

For scrapers DIDs and VCs reshape three things. First, identity-gated content moves from cookie-based session auth to credential-based access. Second, proof of personhood (PoP) credentials become a counter-bot signal. Third, content provenance shifts from platform-attested to creator-attested via signed credentials.

The 2024-2026 wave of EU eIDAS 2.0 and the European Digital Identity Wallet pushed DID and VC adoption from research to production. By 2026 several large platforms accept VC-based proof of age and proof of residence.

Implementation pattern: DID-aware fetcher

import json
from did_resolver import resolve_did
from vc_lib import verify_vc, present

async def fetch_with_vc(url, did, vc_token):
    did_doc = await resolve_did(did)
    presentation = present(vc_token, audience=url)
    headers = {
        "Authorization": f"VC {presentation}",
        "DID": did,
    }
    response = await http.get(url, headers=headers)
    return response

async def verify_inbound_vc(presentation, expected_audience):
    result = verify_vc(presentation)
    if not result.valid:
        return False
    if result.audience != expected_audience:
        return False
    return True

SD-JWT pattern for selective disclosure

Selective Disclosure JWT (SD-JWT) lets a holder reveal only a subset of credential claims to a verifier. Scrapers acting as verifiers can request only the claims they need (for example country of residence) without seeing the full credential. This is privacy-preserving and reduces compliance burden.

def select_disclosures(sd_jwt, claims_to_reveal):
    payload = parse_sd_jwt(sd_jwt)
    revealed = {k: v for k, v in payload.items() if k in claims_to_reveal}
    return reissue_with_disclosures(sd_jwt, revealed)

Comparison: identity models for scrapers

Model	Privacy	Provenance	Replay protection	Scraper effort
Cookie session	Low	None	Per-session	Low
OAuth bearer	Low	Issuer-attested	Per-token	Moderate
API key	Low	Issuer-attested	Per-key	Low
DID plus VC	High (with SD-JWT)	Issuer-attested with crypto proof	Per-presentation	High
zk-credentials	Highest	Crypto-attested	Per-proof	Highest

Web4 vocabulary for scrapers

Web4 is an evolving label that overlaps with decentralized identity, content authenticity (C2PA), and AI-agent-native protocols. For scrapers the practical Web4 surface includes four primitives.

C2PA content credentials embedded in media files for provenance.
did:web identifiers for site-level identity (a DID hosted at .well-known/did.json).
AI agent identity DIDs distinguishing automated traffic from human traffic.
Cross-platform trust frameworks built on top of DIF specifications.

Additional FAQ

Are DIDs replacing OAuth?
Not yet. OAuth remains dominant. DIDs are gaining ground for high-assurance use cases.

Do scrapers need their own DIDs?
For agentic browsers acting on behalf of a user, yes increasingly. The DID is how the scraper identifies itself to the target service.

What about C2PA?
C2PA content credentials are useful for scrapers that need to verify media provenance, particularly for AI training data curation.

How does this interact with bot detection?
A scraper presenting a verified personhood VC may be treated as human-equivalent. A scraper presenting an agent VC is identified as an agent and routed accordingly.

The W3C DID core specification in detail

The W3C DID Core 1.0 specification, recommended in July 2022, defines decentralized identifiers as a new type of identifier that is created and managed without reliance on a centralized registry. A DID resolves to a DID Document, which contains the verification methods, service endpoints, and other metadata associated with the identifier.

DIDs come in many methods. did:web is a method that uses a domain name as the basis. did:key uses a cryptographic key directly. did:ion uses the Sidetree protocol on Bitcoin. did:plc uses the Bluesky-developed Public Ledger of Credentials. Each method has different trade-offs in decentralization, performance, and cost.

For scrapers the most relevant methods are did:web (for site-level identity) and did:key (for ephemeral keys). did:web is essentially a DNS-based approach where a DID resolves via fetching the .well-known/did.json file at the domain. This is operationally simple and integrates with existing web infrastructure.

The DID Document specifies one or more verification methods, each of which is a public key and an associated algorithm. Authentication, assertion, key agreement, and capability invocation are different relationships a verification method can have to the DID. A scraper signing a request uses an authentication-relationship key.

Verifiable credential lifecycle

A verifiable credential has three actors: the issuer, the holder, and the verifier. The issuer creates and signs the credential. The holder stores the credential and presents it to verifiers. The verifier checks the credential’s signature, status, and contents.

The lifecycle proceeds in five steps. First, the issuer issues a credential to the holder, typically via OID4VCI. Second, the holder stores the credential in a wallet. Third, a verifier requests a presentation, typically via OID4VP. Fourth, the holder constructs a presentation (which may include selective disclosure) and sends it to the verifier. Fifth, the verifier validates the presentation and acts on it.

For scrapers acting as verifiers the verification step is the operational concern. Verification involves checking the cryptographic signature against the issuer’s verification method, checking the issuer against a trust list, checking the credential’s expiration, and checking the credential’s revocation status.

The trust list is the most operationally complex piece. The verifier must decide which issuers it trusts. Some trust lists are centralised (a government list of accredited issuers). Others are federated (a mutual recognition agreement among issuers). The decision is policy-driven and context-dependent.

Selective disclosure and zero-knowledge proofs

Selective disclosure is the ability to reveal a subset of credential claims without revealing the rest. SD-JWT is a 2024-stable format that achieves selective disclosure through hash-based blinding of individual claims.

Zero-knowledge proofs go further. A ZKP-based credential allows a holder to prove a statement about the credential (for example over 18) without revealing any specific claim. The holder constructs a proof that the verifier can check without seeing the underlying data.

For scrapers ZKP is operationally heavier but privacy-preserving. The 2026 pattern is to use SD-JWT for most cases (good privacy, modest compute) and ZKP for high-sensitivity cases (best privacy, higher compute).

The ZKP toolkit in 2026 includes AnonCreds (the Hyperledger flagship), BBS+ signatures (for ZK on standard VCs), and several research-grade systems. Production deployments are growing but remain a minority.

C2PA and content provenance

C2PA (Coalition for Content Provenance and Authenticity) is a parallel standard focused on media provenance rather than identity. A C2PA manifest, embedded in an image or video file, describes the file’s origin and edit history through signed assertions.

For scrapers harvesting media files the C2PA manifest is a useful provenance signal. A scraper feeding AI training data can use C2PA to filter out content that has been flagged by the creator as not for AI training. A scraper feeding a news aggregator can use C2PA to verify the file’s claimed source.

The C2PA ecosystem grew rapidly in 2024-2026. Major camera manufacturers ship C2PA-capable hardware. Major image editors embed C2PA manifests on save. Major social platforms display C2PA badges. The pattern is similar to TLS adoption in the early 2010s.

A 2026 best practice for scrapers is to read and preserve C2PA manifests at ingest. The manifest itself is small. Preservation enables downstream consumers to make their own provenance decisions.

Common pitfalls when adopting DIDs in a scraping pipeline

Three failure modes consistently bite teams that introduce DID-based identity into existing scraping infrastructure.

The first pitfall is choosing the wrong DID method for the use case. Teams default to did:web because it is simple, but did:web inherits all the trust limitations of DNS and TLS, including registrar takeover and CA mis-issuance. For high-assurance use cases like agent identity that crosses regulatory boundaries, did:key or did:ion provide stronger guarantees at the cost of more complex resolution. Map the trust requirement to the method before writing code.

The second pitfall is not rotating verification method keys. A DID is durable, but the keys associated with it are not. Most DID methods support adding new verification methods and retiring old ones via DID Document updates. A scraping operation that uses the same key for years exposes itself to key compromise with no recovery path. Build key rotation into the operational runbook from day one.

The third pitfall is conflating proof-of-personhood with proof-of-uniqueness. A personhood credential proves the holder is human; it does not prove the holder is unique to your platform. Sybil resistance requires additional signals like nullifier sets or federated uniqueness checks, which most off-the-shelf personhood credentials do not provide.

Next steps

The fastest first step is to audit your top sources for any wallet/credential signals in the authentication flow. If you find any, the time to plan your access path is now, before VC-gating becomes the default. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the verifiable credentials guide.

This guide is informational, not engineering or legal advice.

May 7, 2026

Scraping with vision models (GPT-4o, Claude 3.5, Gemini Pro)

Vision model scraping in 2026 has crossed the line from cool demo to legitimate production tool. The major LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro and Flash) all accept image input now, all do strong OCR and layout reasoning, and all return strict JSON when asked. For scraping work specifically, that means you can take a screenshot of any web page and extract structured data without writing a single CSS selector.

This guide covers when vision-model scraping wins, how to use each major model effectively, the cost picture, and the production patterns that keep latency and bills under control. Working code throughout.

Why vision model scraping matters

Three problems vision models solve that text-based extraction cannot.

First, sites that render content with images. PDF embeds, infographics, sites that ship product information as images for SEO reasons. Text scrapers see nothing. Vision models read the image.

Second, sites with bot defenses that mangle HTML. Cloudflare’s HTML scrambling, sites that randomize class names per request, sites that ship CSS sprites instead of text. Vision models bypass all of it because they read the rendered pixels.

Third, layout-driven extraction. When the same field name appears in two places (header price and main price), text extraction guesses. Vision extraction sees which one is bigger, more prominent, in the right region.

How vision-model scraping works

The pattern is consistent across all three providers:

Render the target page in a headless browser
Take a screenshot (full page or viewport)
Send the screenshot plus an extraction prompt and schema to the vision model
Validate and store the result

The browser is just a screenshot generator. No selectors. No DOM traversal. The model does all the layout reasoning.

When to skip vision entirely

Vision extraction is the wrong tool when the page is static text in a stable HTML structure. The cost of vision tokens dwarfs text tokens, and accuracy is no better. Reach for vision only when text extraction fails or hits one of the three winning conditions described later.

Implementation with GPT-4o

import asyncio
import base64
from openai import AsyncOpenAI
from playwright.async_api import async_playwright

client = AsyncOpenAI()

PRODUCT_SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "currency": {"type": "string"},
        "in_stock": {"type": "boolean"},
        "rating": {"type": ["number", "null"]},
        "review_count": {"type": ["integer", "null"]},
    },
    "required": ["title", "price", "currency", "in_stock", "rating", "review_count"],
    "additionalProperties": False,
}

async def screenshot_url(url: str) -> bytes:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page(viewport={"width": 1280, "height": 1024})
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=True)
        await browser.close()
        return png

async def extract_with_gpt4o(png_bytes: bytes) -> dict:
    b64 = base64.b64encode(png_bytes).decode()
    resp = await client.chat.completions.create(
        model="gpt-4o",
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "product", "schema": PRODUCT_SCHEMA, "strict": True},
        },
        messages=[
            {"role": "system", "content": "Extract product data from this screenshot."},
            {"role": "user", "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "high"}}
            ]},
        ],
    )
    import json
    return json.loads(resp.choices[0].message.content)

async def main():
    png = await screenshot_url("https://www.lazada.sg/products/example.html")
    print(await extract_with_gpt4o(png))

asyncio.run(main())

detail: "high" matters. The default auto downsamples large images and loses fine text. For product pages with small price labels, always use high.

Implementation with Claude 3.5 Sonnet

from anthropic import AsyncAnthropic
import base64

client = AsyncAnthropic()

async def extract_with_claude(png_bytes: bytes) -> dict:
    b64 = base64.b64encode(png_bytes).decode()
    resp = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=2000,
        tools=[{
            "name": "save_product",
            "description": "Save the extracted product",
            "input_schema": PRODUCT_SCHEMA,
        }],
        tool_choice={"type": "tool", "name": "save_product"},
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
                {"type": "text", "text": "Extract the product data from this screenshot."},
            ],
        }],
    )
    return resp.content[0].input

Claude does not have a detail: high flag because it always processes at full resolution. The trade-off is higher per-image cost than GPT-4o on large screenshots.

Implementation with Gemini 1.5 Pro

import google.generativeai as genai
import os
import json

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

model = genai.GenerativeModel(
    "gemini-1.5-pro",
    generation_config={"response_mime_type": "application/json", "response_schema": PRODUCT_SCHEMA},
)

async def extract_with_gemini(png_bytes: bytes) -> dict:
    response = await model.generate_content_async([
        "Extract the product data from this screenshot.",
        {"mime_type": "image/png", "data": png_bytes},
    ])
    return json.loads(response.text)

Gemini’s huge context window (2M tokens) and dedicated responseSchema parameter make it natural for vision extraction at scale. Cost per image is competitive with the others.

Side-by-side comparison

We ran 100 product page screenshots from Lazada, Amazon, and Best Buy through each model.

Metric	GPT-4o	Claude Sonnet 4.5	Gemini 1.5 Pro	Gemini 1.5 Flash
Cost per image	$0.027	$0.041	$0.024	$0.0035
Latency p50	2.4 s	3.1 s	2.0 s	1.2 s
Extraction accuracy	96%	97%	95%	91%
Best at	UI element recognition	Text-heavy pages	Long pages, multilingual	Cost-sensitive volume
Worst at	Very small text	Image-heavy without text	OCR on stylized fonts	Complex layouts

For most production scraping, GPT-4o or Claude Sonnet are the right pick. Gemini Flash is the value play when cost dominates over the last 5 percent of accuracy.

Comparing model behavior on the same screenshot

The same Lazada product page screenshot, three models, same JSON Schema:

GPT-4o output: clean extraction, occasional off-by-one on review counts when the displayed number includes a comma in non-US locale.

Claude Sonnet 4.5 output: most reliable on text-heavy pages, occasionally over-conservative on in_stock (returns false if any “out of stock” appears anywhere on the page, including in related products).

Gemini 1.5 Pro output: strongest at multilingual content, occasional layout confusion when the price is in a sidebar widget rather than the main panel.

Practical implication: if you have multilingual targets, lean Gemini. If you have text-heavy English ecommerce, lean Claude. If you have a mix of languages and want a balanced default, GPT-4o.

When vision wins over text extraction

Vision wins when one of three conditions holds:

The HTML is intentionally obfuscated (Cloudflare scrambling, randomized classes)
Critical content is rendered as image (price tags as PNGs, infographics)
Layout matters for disambiguation (multiple prices on the same page)

Vision loses when the HTML is clean and well-structured. Text extraction is 5-10x cheaper and just as accurate on those targets.

For more on text extraction patterns, see LLM extraction patterns: structured output from messy HTML.

Real failure modes

A few specific failure patterns observed in production:

The model reads a strikethrough price (the “old” price) instead of the current price. Mitigation: explicit instruction “extract the current price, not the strikethrough or comparison price.”

The model extracts a related product’s price when the main product price is hidden behind a button. Mitigation: instruct “extract only the main product on this page” and add a sentinel check (e.g. the title must contain a known keyword).

The model treats currency-only labels (just “$”) as full prices. Mitigation: validate that price > 0 and reject extractions where price is implausibly small.

The model fails on sites that render prices with web fonts containing custom glyphs (some bot defenses ship a font that maps numbers to other glyphs). Mitigation: a hybrid extraction with HTML, where the HTML still contains the real character codes.

Hybrid extraction: vision for hard fields only

The cost-optimal pattern for many sites is hybrid. Use cheap text extraction for the easy fields (title, description) and reserve vision for the fields that fail text extraction (price hidden behind dynamic rendering, stock indicator embedded in an SVG icon).

async def hybrid_extract(html: str, png: bytes) -> dict:
    text_result = await extract_text_with_4o_mini(html)
    if text_result.get("price") is None or text_result.get("currency") is None:
        vision_result = await extract_with_gpt4o(png)
        text_result["price"] = vision_result.get("price")
        text_result["currency"] = vision_result.get("currency")
    return text_result

This pattern saves significant cost over pure vision while catching the cases where text fails.

Full-page vs viewport screenshots

Full-page screenshots capture everything but produce huge PNGs that cost more to process and confuse models with too much content.

Viewport screenshots capture only the visible region but may miss below-the-fold content (reviews, related products).

The pragmatic default: viewport screenshot for the primary entity, scroll-and-snap for any below-the-fold field you specifically need.

async def screenshot_with_scroll(url: str, scroll_targets=None) -> list[bytes]:
    screenshots = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page(viewport={"width": 1280, "height": 1024})
        await page.goto(url, wait_until="networkidle")
        screenshots.append(await page.screenshot())

        if scroll_targets:
            for selector in scroll_targets:
                el = await page.locator(selector).first
                await el.scroll_into_view_if_needed()
                screenshots.append(await page.screenshot())

        await browser.close()
    return screenshots

Handling multiple entities per page

For pages with many entities (a search results page, a category listing), pass the screenshot with an array schema.

LISTING_SCHEMA = {
    "type": "object",
    "properties": {
        "items": {
            "type": "array",
            "items": PRODUCT_SCHEMA,
            "minItems": 0,
            "maxItems": 50,
        },
    },
    "required": ["items"],
    "additionalProperties": False,
}

async def extract_listing(png_bytes: bytes) -> dict:
    # use GPT-4o or Claude with the listing schema
    ...

Vision models handle arrays well. The cap on maxItems prevents runaway hallucination on confused inputs.

Adding proxies

Proxies live in your screenshot step, not the vision call. Configure the headless browser:

async def screenshot_with_proxy(url: str, proxy: str) -> bytes:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy={"server": proxy},
        )
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=True)
        await browser.close()
        return png

For ASEAN ecommerce specifically, Singapore mobile proxy carries clean carrier IPs that survive the strongest bot defenses. Pair with full-page screenshots for product listings on Lazada and Shopee.

Production patterns

Three patterns separate hobby vision scraping from production.

First, downsample appropriately. Vision models have an effective resolution they actually use. For GPT-4o, anything above 2048×2048 wastes tokens. Resize before sending.

from PIL import Image
import io

def resize_for_model(png_bytes: bytes, max_dim: int = 2048) -> bytes:
    img = Image.open(io.BytesIO(png_bytes))
    if max(img.size) > max_dim:
        ratio = max_dim / max(img.size)
        new_size = (int(img.size[0] * ratio), int(img.size[1] * ratio))
        img = img.resize(new_size, Image.LANCZOS)
    out = io.BytesIO()
    img.save(out, format="PNG", optimize=True)
    return out.getvalue()

Second, cache by image hash. Identical screenshots produce identical extractions. SHA-256 the PNG, key your cache on it.

Third, run two models in parallel for high-stakes data. GPT-4o and Claude Sonnet, take the agreement. Catches the rare hallucination at 2x cost.

Memory and disk considerations

Full-page screenshots can be large. A 4000-pixel-tall page at 2x DPR is roughly 8 MB as PNG, 600 KB as JPEG quality 80. For high-volume pipelines:

Compress to JPEG before sending. JPEG quality 85 is visually indistinguishable from PNG for typical web pages and cuts payload size by 90 percent.

Stream screenshots through a temporary buffer rather than holding them all in memory. A 100-worker pool with full-page PNGs can OOM a 16 GB host quickly.

Cache screenshots locally for replay. The screenshot is the source of truth for an extraction run; saving it lets you re-extract with a different model later without re-fetching.

def to_jpeg(png_bytes: bytes, quality: int = 85) -> bytes:
    img = Image.open(io.BytesIO(png_bytes)).convert("RGB")
    out = io.BytesIO()
    img.save(out, format="JPEG", quality=quality, optimize=True)
    return out.getvalue()

Real benchmarks across sites

100 product pages each, full-page screenshot, GPT-4o:

Site	Success rate	Avg cost per page
Lazada SG	98%	$0.029
Shopee SG	96%	$0.031
Amazon US	99%	$0.025
Walmart	97%	$0.027
Best Buy	95%	$0.030
Tokopedia	94%	$0.034

Add browser cost (Browserbase or self-hosted) at $0.002-$0.005 per page. Total per 1000 pages: $30-$40 with vision, vs $5-$10 with text-only extraction. Vision wins on accuracy and resilience; text wins on cost.

Token cost mechanics

Vision tokens are computed differently from text tokens. The mechanics matter for cost prediction.

GPT-4o computes vision tokens by splitting the image into 512×512 tiles, charging 170 tokens per tile, plus a fixed 85 tokens for the low-res view. A 1024×1024 image is 4 tiles plus the base = 765 tokens. A 1600×1024 image is 6 tiles plus base = 1105 tokens. detail: low uses only the 85 base tokens at the cost of accuracy.

Claude charges roughly 1.15 tokens per pixel up to a max, with an effective image cost around 1500 to 4000 tokens depending on size.

Gemini charges a flat 258 tokens per image regardless of size, which makes it dramatically cheaper for large screenshots.

The implication: if your scraper sends 5 MB full-page screenshots through GPT-4o, you are paying for 4 to 8 thousand vision tokens per image. Resize to 1280×800 and you cut that to under 1500 tokens with minimal accuracy loss.

Region cropping for high-stakes fields

For mission-critical fields (transaction prices, contract terms, regulatory disclosures), crop the image to the field region and send only the crop. This pushes accuracy from roughly 96 percent on full pages to over 99 percent on focused crops.

def crop_to_region(png_bytes: bytes, x: int, y: int, w: int, h: int) -> bytes:
    img = Image.open(io.BytesIO(png_bytes))
    cropped = img.crop((x, y, x + w, y + h))
    out = io.BytesIO()
    cropped.save(out, format="PNG")
    return out.getvalue()

# Use selectors or LLM observation to find the region first, then crop and re-extract

The two-step approach (full page first, then crop and re-extract critical fields) is the right pattern when accuracy matters more than cost.

Multimodal pipelines: combining vision and text

The strongest extraction pipelines combine HTML and screenshot in the same LLM call. The model uses the HTML as ground truth for structured fields and the screenshot for visual context.

async def multimodal_extract(html: str, png: bytes, schema: dict) -> dict:
    b64 = base64.b64encode(png).decode()
    return await client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_schema", "json_schema": {"name": "x", "schema": schema, "strict": True}},
        messages=[
            {"role": "system", "content": "Use the HTML for structured data and the screenshot for layout context."},
            {"role": "user", "content": [
                {"type": "text", "text": f"HTML:\n{html[:100000]}"},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "high"}},
            ]},
        ],
    )

This combination outperforms either alone on roughly 60 percent of pages we tested. Cost is higher than text-only by the vision token premium.

Vision-based crawling for shape discovery

A clever pattern uses vision to discover the shape of an unknown site. Take a few screenshots, ask the model to describe the layout in structured form (“this site has a header, a search bar, a product grid with 3 columns”), then use that description to build a Playwright scraper.

This bootstraps a deterministic scraper from a few vision calls, paying once for discovery instead of every scrape.

Common production gotchas

A few patterns that bite teams using vision models.

The model occasionally hallucinates fields that look plausible but are not on the page. Always validate extracted data against the source HTML or a deterministic check.

Different vendors handle base64 differently. OpenAI accepts a data: URL. Anthropic accepts the raw base64 with media_type. Gemini accepts the bytes directly with mime_type. Wrappers help but the bare APIs differ.

Image preprocessing libraries (PIL, OpenCV) introduce subtle artifacts that can change OCR output. Save the raw screenshot and the preprocessed version both, and prefer the raw if the preprocessing is not strictly necessary.

Vision token cost varies by model in non-obvious ways. Always benchmark on your actual screenshots; do not extrapolate from documented pricing alone.

Frequently asked questions

How do I evaluate which vision model is best for my specific target?
Hand-label 50 representative pages, run each model with the same schema, score against the gold set. Cost is roughly $5 per evaluation run; the data drives a multi-month decision.

Can vision models read tiny text like product SKUs?
Up to a point. GPT-4o and Claude Sonnet handle text down to about 8px reliably at high detail. For smaller text, crop to the relevant region before sending.

What about charts and tables?
All three models handle structured tables in screenshots well. Charts are mixed; line and bar charts work, complex multi-series charts often fail. Pass the underlying data if you can.

How do I handle international character sets?
Vision OCR for Chinese, Japanese, Korean, Thai, Arabic is strong on Gemini Pro and Claude Sonnet. GPT-4o is good but slightly behind on uncommon scripts. Test on your specific target.

Can I use vision to fill out forms?
Indirectly. Vision models can identify form fields and instruct your scraper. For actual form filling, browser automation (Playwright, browser-use) is the right tool.

What about cost-effective open-source vision models?
Llama 3.2 90B Vision and Qwen 2.5 VL 72B are the strongest open-source vision models in early 2026. Self-hosted on a 4xA100 machine, cost per image is around $0.001 if you have throughput to amortize. Below the major closed-source models on quality, especially on small text.

Can vision models extract from videos?
Indirectly. Sample frames at 1 fps, send each frame to the vision model, aggregate the extractions. For long videos, sampling every 5 seconds and aggregating works well.

How do vision models handle CAPTCHAs?
They will solve simple image CAPTCHAs (find traffic lights, identify text in a distorted image) reasonably well, but the major LLM providers refuse the obvious “solve this CAPTCHA” prompts. Phrasing matters. Solver services remain more reliable for production CAPTCHA workflows.

Can I extract from rendered PDFs as images?
Yes. Convert the PDF to images with pdf2image or similar, then run vision extraction on each page. For text-heavy PDFs, the modern LLM APIs accept PDFs directly which is faster and cheaper.

Is there a future where vision replaces selector-based scraping entirely?
For sites that change layout faster than engineers can update selectors, vision is already winning. For high-volume known-shape sites, the cost gap keeps selector-based extraction relevant. The real future is hybrid: vision for discovery and resilience, selectors for the bulk.

Can I run vision extraction on edge devices?
The smaller open-source vision models (Qwen 2 VL 2B, Llava-OneVision 7B) run on consumer GPUs. Quality is well below the major models but adequate for known-shape extraction.

For more on the broader AI scraping landscape, browse the AI modern scraping category.

May 7, 2026

The agentic browser revolution: Claude, OpenAI Operator, Stagehand

Agentic browser 2026 is no longer a research curiosity. The eighteen months between Anthropic’s Computer Use launch in October 2024 and the May 2026 state of the art produced a fundamentally different stack for browser automation. Claude Computer Use, OpenAI Operator, Stagehand from Browserbase, browser-use the open-source library, and the Browser MCP servers all matured into production-grade tools. For scraping operators, the change is structural: brittle CSS selectors give way to vision-grounded, intent-driven instructions; multi-step workflows that took weeks to build now take an afternoon; and the cost economics shifted from “engineering hours per scraper” to “agent tokens per task.” This guide walks through what each agentic browser actually does, the head-to-head comparison, the migration patterns from selector-based to agent-based scraping, the failure modes that still bite, and where the technology is heading.

The audience is the data engineer or scraping platform owner who needs to decide whether and how to adopt agentic browsing in 2026.

What an agentic browser actually is

An agentic browser is a system in which an LLM (typically vision-capable) drives a browser by interpreting user intent, observing the rendered page, and issuing actions (click, type, scroll, navigate). The “agentic” part is that the model decides what to do next based on what it sees, rather than executing a hard-coded script.

The minimum architecture has three components: a browser runtime (Chromium, Firefox, or a managed service), an action interface (the API by which the model issues clicks and keystrokes), and the model itself with vision capability.

The four major implementations in 2026:

Implementation	Vendor	Browser runtime	Model	Hosted?
Claude Computer Use	Anthropic	Local or remote VM	Claude 4.7 (vision)	Self-host
OpenAI Operator	OpenAI	OpenAI-managed	GPT-4o (vision) / o3	Hosted
Stagehand	Browserbase	Browserbase-managed Chromium	Pluggable (Claude, GPT, Gemini)	Hosted
browser-use	Open source	Local Chromium via Playwright	Pluggable	Self-host

Each takes a different position on hosted versus self-hosted, on the level of abstraction over the browser, and on which model providers it supports.

For the broader MCP integration story, see MCP for data engineers. For the AI-as-web-user concept, see AI agents as web users.

Claude Computer Use: the OS-level abstraction

Anthropic’s Computer Use is the lowest-level abstraction. The agent is given a sandboxed virtual machine with a screen, mouse, and keyboard, and it operates by taking screenshots, reasoning about pixel coordinates, and issuing mouse/keyboard events.

Strengths:
– Universal: anything a human can do with a desktop, the agent can do.
– Not browser-specific: works on installed apps, terminal, file manager.
– Vision-grounded: the model sees what the user sees.
– Self-hosted by default: full control over data and access.

Weaknesses:
– Higher latency: screenshot, reason, act, screenshot.
– Higher token cost: each step burns vision tokens.
– More fragile to layout shifts: pixel coordinates drift on responsive UIs.
– Operational overhead: you run the VM.

Best for: complex multi-app workflows, desktop automation, situations where browser isolation matters, controlled internal use.

A minimal Claude Computer Use loop in Python:

from anthropic import Anthropic
import base64

client = Anthropic()
def screenshot_b64():
    return base64.b64encode(open("screen.png", "rb").read()).decode()

response = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=2048,
    tools=[{"type": "computer_20250124", "name": "computer",
            "display_width_px": 1280, "display_height_px": 800}],
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Open the website and list all products."},
            {"type": "image", "source": {"type": "base64",
                                         "media_type": "image/png",
                                         "data": screenshot_b64()}},
        ],
    }],
)

The model returns tool-use blocks with action types (click, type, key, screenshot). Your loop executes them in the VM, takes a new screenshot, and calls the model again.

OpenAI Operator: the hosted browsing agent

OpenAI Operator launched in January 2025 as a hosted browsing agent built on a fine-tuned GPT-4o variant called CUA (Computer-Using Agent). Operator runs in OpenAI infrastructure and exposes an API for users to delegate browser tasks.

Strengths:
– Hosted: no infrastructure ownership.
– Tight integration with ChatGPT consumer surface.
– Rapid iteration: OpenAI continuously improves the underlying model.
– Cleanly framed for end-user delegation use cases.

Weaknesses:
– Hosted-only: no self-host option.
– Less granular control: the abstraction is “task” not “click”.
– Data leaves your environment.
– US-Europe regulatory exposure.

Best for: end-user productivity tasks, ChatGPT-integrated experiences, low-volume high-value workflows where the hosted convenience justifies the data exposure.

The Operator API call pattern:

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="computer-use-preview",
    tools=[{"type": "computer_use_preview",
            "display_width": 1280, "display_height": 800,
            "environment": "browser"}],
    input=[{"role": "user", "content": "Find the cheapest direct flight "
                                       "from SIN to TYO next Monday."}],
)

The Operator returns a sequence of actions; OpenAI executes them in its hosted browser; you receive structured progress events.

Stagehand: the developer-first abstraction

Stagehand from Browserbase is a TypeScript-first library that sits one level above the raw browser. It provides three high-level primitives: act (do something), extract (pull structured data), and observe (find an element). Each is backed by an LLM under the hood.

Strengths:
– Developer ergonomics: writing scrapers feels like writing tests.
– Pluggable model: choose Claude, GPT, or Gemini per call.
– Browserbase-hosted: managed Chromium with anti-bot built in.
– Strong observability: every action logged.
– Good TypeScript ergonomics; Python SDK matured in 2025.

Weaknesses:
– Hosted browser by default (Browserbase); local mode possible but less polished.
– Cost model: pay per browser session plus per LLM call.
– Less universal than OS-level approaches.

Best for: production scraping pipelines, situations where developer velocity and reliability matter, teams that want a managed browser without giving up control.

A minimal Stagehand session:

import { Stagehand } from "@browserbasehq/stagehand";

const stagehand = new Stagehand({ env: "BROWSERBASE" });
await stagehand.init();
await stagehand.page.goto("https://example.com/products");
await stagehand.act({ action: "filter products by category 'shoes'" });
const data = await stagehand.extract({
  instruction: "extract all product names and prices",
  schema: z.object({
    products: z.array(z.object({ name: z.string(), price: z.string() })),
  }),
});

Three primitives, structured output, no selector engineering.

For the head-to-head with Playwright, see Stagehand vs Playwright for AI-driven scraping.

browser-use: the open-source contender

browser-use is an open-source Python library that pairs Playwright with vision-capable LLMs. It launched in late 2024 and matured rapidly through 2025. By 2026 it is the most popular self-hosted agentic browsing library.

Strengths:
– Fully open source; MIT licence.
– Self-hosted; data and browser stay in your environment.
– Pluggable model: any vision-capable LLM via langchain-style adapters.
– Active community; rapid iteration.
– Cheaper at scale than hosted alternatives.

Weaknesses:
– More setup: you run the browser and the model.
– Less polished than commercial offerings.
– Documentation evolving.
– No built-in anti-bot infrastructure.

Best for: cost-sensitive teams, regulated environments, situations where the data must not leave, teams comfortable with open-source operational ownership.

For the broader self-hosted infrastructure story, see self-hosted proxy infrastructure.

Head-to-head comparison

Dimension	Computer Use	Operator	Stagehand	browser-use
Hosted?	Self-host	Hosted	Hosted (default)	Self-host
Browser runtime	VM you run	OpenAI-managed	Browserbase	Playwright local
Model	Claude only	OpenAI only (CUA)	Pluggable	Pluggable
Abstraction level	Pixel/coordinate	Task	Act/extract/observe	Action
Best language	Python	Python/TS	TypeScript (Python catching up)	Python
Anti-bot built in	No	Partial	Yes (Browserbase)	No
Cost model	Token + VM	Per session	Session + tokens	Token only
Suitable for production scraping	Moderate	Moderate	High	High
Suitable for desktop automation	High	Low	Low	Low
Suitable for end-user delegation	Low	High	Moderate	Low

Migration pattern: from selector-based to agentic

Most scraping teams in 2026 are migrating from selector-based pipelines (Scrapy, Playwright with explicit selectors) to agentic browsers. The migration pattern that works:

Identify the most-fragile scrapers (highest selector breakage rate, highest engineering time per maintenance).
Pick one as the migration pilot.
Build the agentic version side-by-side; do not retire the selector version.
Run both for two weeks; compare outputs, costs, latency, success rate.
If the agentic version wins on net (success rate matters more than cost in 2026), retire the selector version.
Repeat for the next-most-fragile scraper.

The pattern works because agentic browsers are dramatically more resilient to layout changes but cost more per page. The economics flip in favour of agentic when maintenance cost dominates.

Pipeline characteristic	Stay selector	Migrate to agent
Stable site, simple structure	Stay	–
Frequent layout changes	–	Migrate
High volume, low value per page	Stay	–
Low volume, high value per page	–	Migrate
Complex multi-step workflow	–	Migrate
Single-step extraction	Stay	–
Anti-bot heavy	Hybrid	Hybrid (use Stagehand or Browserbase)

Failure modes that still bite

Three failure modes show up consistently in 2026 production deployments.

The first is non-determinism. The same prompt against the same page can produce different action sequences. For workflows where audit and reproducibility matter (financial, compliance), this is a problem. The mitigation: use temperature zero, snapshot intermediate states, and validate outputs against schemas.

The second is hallucination. Vision-capable LLMs occasionally describe elements that are not present. They click on coordinates that do not contain a button. The mitigation: use the act-then-verify pattern, where every action is followed by an observation that confirms the expected state change.

The third is anti-bot detection. Vision-grounded clicks at pixel coordinates produce a behavioural signature different from human mouse movements. Bot management systems trained on human behaviour increasingly flag agentic browsing. The mitigation: use anti-bot-aware browsers (Browserbase, Bright Data Scraping Browser) or implement realistic mouse movement simulation.

For the broader anti-bot question, see DataDome vs PerimeterX vs Akamai.

Cost economics in 2026

A rough cost benchmark for a 100-step scraping task across the four implementations:

Implementation	Cost per task	Latency	Success rate
Claude Computer Use	USD 0.40-0.80	60-120s	85-92%
OpenAI Operator	USD 0.50-1.00	60-90s	88-94%
Stagehand	USD 0.30-0.60	30-60s	90-95%
browser-use	USD 0.15-0.40	30-90s	85-93%

The numbers shift weekly as model pricing changes. The pattern is stable: hosted offerings cost more but reduce operational overhead; self-hosted offerings cost less but require ownership. Stagehand sits at the favourable middle for production scraping.

For the deeper benchmark, see AI scraping cost benchmark.

External references

The Anthropic Computer Use documentation is at docs.anthropic.com/en/docs/agents-and-tools/computer-use. The OpenAI Operator launch announcement and developer documentation is at openai.com/index/introducing-operator. Stagehand’s open-source repository is at github.com/browserbase/stagehand. browser-use is at github.com/browser-use/browser-use.

Where the technology is heading

Three trends shape the 2026-2027 trajectory.

First, vision models are getting cheaper and faster. The cost per agentic action has fallen 70 percent year-over-year for the past two years, and that trend continues. Workflows that are uneconomic today become economic in six months.

Second, the abstraction is moving up. Stagehand-style act/extract/observe is replacing pixel-level coordinate reasoning for most use cases. Pixel-level work persists for edge cases (canvas-based UIs, custom desktop apps).

Third, anti-bot detection is adapting. The arms race between agentic browsers and bot management is the same as the proxy versus bot management arms race that ran for the past decade. Expect 2027 to bring purpose-built agent management products from DataDome, PerimeterX, Akamai, and Cloudflare.

For the longer-arc view of how AI agents become indistinguishable from human users, see AI agents as web users.

FAQ

Are agentic browsers replacing Playwright?
Not yet. Playwright remains the workhorse for stable, high-volume, simple scrapes. Agentic browsers win where layout changes are frequent or workflows are complex.

Which is best for production scraping?
Stagehand or browser-use. Stagehand if you want managed; browser-use if you want self-hosted.

Can I use one for desktop automation?
Claude Computer Use is the only one designed for that. Operator and Stagehand are browser-only.

How does anti-bot detection see agentic browsers?
Increasingly visible. Use anti-bot-aware browser providers or invest in behavioural realism.

What is the right stack for a 2026 greenfield scraping pipeline?
Stagehand on Browserbase for managed; browser-use on Playwright with residential proxies for self-hosted. Both with structured-output schemas and verification loops.

Extended agentic browser architecture analysis

The agentic browser stack in 2026 consists of four layers. First, the underlying browser engine (Chromium, Firefox, WebKit). Second, the automation protocol (CDP, WebDriver Classic, WebDriver BiDi). Third, the agent orchestration layer (a planner that decomposes tasks into actions). Fourth, the model that proposes the next action from the current page state.

The 2024-2026 wave of agentic browsers (Anthropic’s computer use, OpenAI’s operator, Browserbase, Stagehand, AgentQL) converged on three patterns. First, page state is captured as a combination of accessibility tree plus a screenshot. Second, actions are issued as a small typed vocabulary (click, type, scroll, wait, navigate). Third, the agent runs in a loop until task completion or a step budget is exhausted.

The accessibility-tree-plus-screenshot pattern beat pure screenshot grounding because the tree gives precise element identifiers while the screenshot gives layout context. The combination reduces hallucinated coordinates.

Production agentic browser pattern

from playwright.async_api import async_playwright

async def run_agent(task, max_steps=20):
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto("https://example.com")

        for step in range(max_steps):
            snapshot = await page.accessibility.snapshot()
            screenshot = await page.screenshot()
            action = await model_propose_action(task, snapshot, screenshot, step)
            if action["type"] == "done":
                return action["result"]
            await execute_action(page, action)
        return {"status": "step_budget_exhausted"}

async def execute_action(page, action):
    if action["type"] == "click":
        await page.click(action["selector"])
    elif action["type"] == "type":
        await page.fill(action["selector"], action["value"])
    elif action["type"] == "navigate":
        await page.goto(action["url"])
    elif action["type"] == "scroll":
        await page.evaluate(f"window.scrollBy(0, {action['delta']})")
    elif action["type"] == "wait":
        await page.wait_for_timeout(action["ms"])

Step-budget and termination patterns

A robust agentic browser sets four budgets per task.

Step budget (typically 20-50 actions).
Wall-clock budget (typically 5-15 minutes).
Token budget for the model (typically 100k tokens per task).
Cost budget in dollars.

Termination triggers when any budget is exhausted, when the model returns a done action, when an unrecoverable error occurs, or when a safety check fires.

Detection and counter-detection

Bot management vendors (Cloudflare Bot Management, Akamai Bot Manager, DataDome, PerimeterX) ship 2026 detectors that look for the following agent signals.

Headless Chromium fingerprints (missing navigator.webdriver, missing plugins, missing window.chrome).
CDP-specific runtime traces.
Mouse and keyboard event timing distributions that lack human jitter.
Action sequences that match common agent libraries.

Counter-detection in 2026 typically includes residential proxies, fingerprint patching, and human-jitter event timing. The arms race continues.

Comparison: agentic browser frameworks 2026

Framework	Underlying engine	Automation protocol	Best for
Playwright plus custom agent	Chromium, Firefox, WebKit	CDP, BiDi	Custom builds
Browserbase	Chromium	CDP	Hosted scaling
Stagehand	Chromium	CDP	LLM-native abstractions
AgentQL	Chromium	CDP	Schema-driven extraction
Anthropic computer use	OS-level	Pixel grounding	Cross-app workflows

Additional FAQ

How do agentic browsers handle CAPTCHAs?
They typically pause and request human intervention. Some integrate solving services. Production systems should treat CAPTCHA as a termination signal rather than a step to bypass.

What about session state?
Persist cookies and storage in a context per task. Reuse contexts only across related tasks for the same user.

How do I evaluate agentic browser performance?
Build a fixed test suite of tasks (web navigation, form filling, data extraction). Measure success rate, mean steps, mean wall-clock, mean cost. Track trend over model updates.

Is the agentic browser pattern replacing classical scraping?
For one-off tasks yes. For high-volume structured extraction classical scraping remains cheaper and more reliable.

Common pitfalls in production agentic browser deployments

Five failure modes recur across teams that move from pilot to production with agentic browsers in 2026.

The first pitfall is unbounded step budgets in production. A pilot script with no step ceiling will eventually encounter a page where the agent loops on a recoverable error and burns through hundreds of dollars in vision tokens before the wall-clock budget catches it. Always set a step budget, a wall-clock budget, and a hard cost ceiling per task, and alert when any task hits 50 percent of any budget.

The second pitfall is treating the agent’s natural-language reasoning as audit-grade output. The agent’s chain of thought may say it clicked the correct button when it actually clicked an adjacent element. Capture the post-action accessibility snapshot and verify the expected state change with deterministic checks, not the agent’s self-report.

The third pitfall is sharing browser contexts across tasks. Agents that reuse a single Chromium context accumulate cookies, storage, and history that leak between unrelated tasks. The leak shows up as mysterious cross-task contamination weeks into production. Use one context per task by default; share only when the workflow explicitly requires session continuity.

The fourth pitfall is failing to record screenshots and DOM snapshots for every action. When an agentic scraper produces wrong output, the only way to debug is to replay the visual state the agent saw at decision time. Storage is cheap; debugging without snapshots is impossible. Record everything for at least 30 days.

The fifth pitfall is ignoring model version drift. The same prompt against the same page can produce different action sequences when the underlying model is updated by the provider. Pin the model version explicitly, validate on a regression suite before adopting a new version, and never let a hosted offering silently upgrade your production pipeline.

The architecture shift from scripted to agentic automation

Classical web automation (Selenium, Puppeteer, Playwright in scripted mode) follows a deterministic recipe. The script knows the page structure, the selectors, and the expected response. When any of those changes, the script breaks. Engineers spend significant time maintaining selectors and recovery paths.

Agentic browser automation flips the model. The agent does not know the page structure. It receives a goal and a current page state, decides on the next action, and observes the result. The agent adapts to layout changes, follows alternative paths, and recovers from unexpected states.

The shift has implications for cost, capability, and reliability. Cost is higher because each step requires a model inference. Capability is broader because the agent can handle tasks the script author did not anticipate. Reliability is more variable because the agent occasionally chooses suboptimal actions.

The 2024-2026 pattern is to use scripted automation for high-volume, well-defined tasks (price scraping, sitemap crawling, structured data extraction) and agentic automation for low-volume, varied tasks (research, customer support automation, exploratory data gathering). The two patterns coexist in the same operation.

The accessibility tree as the agent’s primary input

The decision to use the accessibility tree (rather than the raw DOM or pure pixels) as the agent’s primary input was driven by three considerations. First, the tree is structured and parseable, supporting reliable selector generation. Second, the tree captures semantic information that the DOM does not (button roles, form labels, link purposes). Third, the tree is a small enough representation to fit in context windows.

The accessibility tree has limitations. Sites that use custom controls without ARIA attributes have impoverished trees. Single-page applications that update via JavaScript may have stale trees if the snapshot is taken at the wrong moment. Sites that intentionally obscure their structure (some bot-protected sites) have deliberately confusing trees.

The 2026 toolkit handles these cases through a combination of techniques. ARIA-deficient sites are augmented with screenshot grounding. Stale trees are addressed with explicit wait-for-load conditions. Obscured trees are addressed with vision-only fallback when the tree is unusable.

Step quality and the planning loop

The quality of an agentic browser depends on the quality of each step decision. Three factors drive step quality: the model’s understanding of the goal, the precision of the page state representation, and the appropriateness of the action vocabulary.

The 2026 patterns for improving step quality include explicit goal restatement at each step (preventing goal drift), structured action proposals with reasoning fields (improving the model’s articulation), and step-level critique by a separate model (catching obvious mistakes).

Planning loops can be flat (the model decides each step from scratch) or hierarchical (a planner decides sub-goals and a executor decides per-sub-goal actions). Hierarchical planning works better for complex multi-step tasks. Flat planning works better for short tasks. The 2026 best practice is to use hierarchical planning for tasks expected to take more than five steps.

Testing and evaluation of agentic browsers

A tested agentic browser is a more reliable agentic browser. The 2026 best practice is to maintain a fixed test suite of representative tasks with known expected outcomes. The suite is run on every model update and every framework update.

Evaluation metrics typically include task success rate, mean steps to completion, mean wall-clock time, mean cost, and recovery rate from injected failures. Each metric is tracked over time. Regressions trigger investigation.

The test suite must be maintained alongside the production tasks. As production tasks evolve, the test suite is updated. The test suite is the safety net that catches regressions before they affect production.

A 2026 best practice that is gaining traction is adversarial evaluation. The test suite includes deliberately misleading pages (decoy buttons, ambiguous instructions, time-pressured prompts) that probe the agent’s robustness. Performance on adversarial cases is a leading indicator of production reliability.

Next steps

If you have not piloted an agentic browser yet, the highest-leverage move this quarter is to pick your most fragile scraper and rebuild it in Stagehand or browser-use. The hour spent will tell you more than weeks of comparison reading. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the AI agents as web users guide.

This guide is informational, not engineering or legal advice.

May 7, 2026

LLM extraction patterns: structured output from messy HTML

LLM extraction structured output is the workhorse of modern scraping pipelines. Once the browser layer has rendered a page and given you HTML, the question is how to turn that messy DOM into clean JSON that your warehouse can ingest. In 2026 every major LLM provider ships strict JSON Schema mode, so the question is no longer “can I get JSON” but “what schema, what prompt, what model, what cost”.

This guide is the playbook. We cover schema design, prompt patterns, validation, retry strategy, cost control, and the model selection matrix across OpenAI, Anthropic, Google, and the open-source contenders. Every pattern is from production usage in 2026.

Why structured output matters

Three reasons.

First, downstream systems need typed data. A price field that is sometimes a number and sometimes a string with a currency symbol breaks every dashboard. Schema enforcement at the LLM boundary kills this class of bug.

Second, structured output is dramatically cheaper than freeform extraction over time. Freeform output requires post-processing logic that drifts with each new page format. Structured output forces the LLM to do the work once.

Third, structured output is the only path to reliable agentic loops. An agent that returns JSON can chain into the next step. An agent that returns prose breaks pipelines.

JSON Schema the right way

The single most common mistake in LLM extraction is loose schemas. A {"price": {"type": "number"}} field looks fine until the model returns null, the validation passes (because null is technically allowed without required), and your pipeline writes a row of garbage.

The right pattern is strict, required, and bounded.

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string", "minLength": 1, "maxLength": 500},
        "price": {"type": "number", "minimum": 0, "maximum": 1000000},
        "currency": {"type": "string", "pattern": "^[A-Z]{3}$"},
        "in_stock": {"type": "boolean"},
        "sku": {"type": ["string", "null"]},
    },
    "required": ["title", "price", "currency", "in_stock", "sku"],
    "additionalProperties": False,
}

additionalProperties: False and complete required lists are not optional. They are how you stop the model from inventing fields or skipping ones that should be present.

OpenAI’s Structured Outputs (GA in 2024) and Anthropic’s tool use (which doubles as structured output) both honor strict schemas. Google Gemini supports JSON mode with a similar shape via the responseSchema parameter.

Strict mode in OpenAI

from openai import AsyncOpenAI
import json

client = AsyncOpenAI()

async def extract_product(html: str) -> dict:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "product",
                "schema": schema,
                "strict": True,
            },
        },
        messages=[
            {"role": "system", "content": (
                "Extract product data from the HTML. If a field is not present, "
                "use null for sku. All other fields are required."
            )},
            {"role": "user", "content": html[:200000]},
        ],
    )
    return json.loads(resp.choices[0].message.content)

strict: True constrains the decoder so the model literally cannot output invalid JSON. Compliance is enforced at the token level. This is the gold standard.

Tool use in Anthropic

from anthropic import AsyncAnthropic
import json

client = AsyncAnthropic()

async def extract_product(html: str) -> dict:
    resp = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=2000,
        tools=[{
            "name": "save_product",
            "description": "Save the extracted product record",
            "input_schema": schema,
        }],
        tool_choice={"type": "tool", "name": "save_product"},
        messages=[{
            "role": "user",
            "content": f"Extract product data from this HTML:\n\n{html[:200000]}",
        }],
    )
    return resp.content[0].input

tool_choice: tool forces Claude to call the tool, which is how you guarantee structured output. The input_schema is JSON Schema and Claude validates against it before returning.

Prompt design for extraction

The system prompt matters more than people realize. Three rules from production.

First, name the entity explicitly. “Extract the product” is better than “extract structured data”. The model anchors on entity type.

Second, specify what to do when fields are missing. “Use null if not present” beats letting the model guess.

Third, if the HTML contains multiple candidates (multiple products, related items, ads), tell the model which one to extract. “The main product on this page” beats letting the model decide.

A solid system prompt template:

You are a precise data extractor. Extract the {entity} from the provided HTML.

Rules:
- Use the schema exactly. Do not add fields. Do not skip required fields.
- For missing fields, use null only if explicitly allowed.
- The entity to extract is: {entity_description}.
- Ignore related items, recommendations, advertisements, and footer content.
- Numeric fields must be numbers, not strings. Strip currency symbols and commas.

Gemini structured output

Google’s pattern uses responseSchema directly on the generation config:

import google.generativeai as genai

model = genai.GenerativeModel("gemini-1.5-flash-002")

resp = model.generate_content(
    f"Extract the product from this HTML:\n\n{html[:1000000]}",
    generation_config={
        "response_mime_type": "application/json",
        "response_schema": schema,
    },
)
data = json.loads(resp.text)

Gemini’s response_schema accepts JSON Schema with the same semantics as OpenAI’s strict mode. The 2-million token context window of Gemini Pro is the only place where you can pass an entire site’s product catalog HTML as a single extraction call.

Pre-processing HTML

Sending raw 800KB HTML to the model wastes tokens and confuses extraction. Trim aggressively before extraction.

from bs4 import BeautifulSoup
import re

def trim_html(html: str, target_tags=("title", "script[type='application/ld+json']", "meta")) -> str:
    soup = BeautifulSoup(html, "html.parser")
    # remove scripts (except JSON-LD), styles, navigation, footer
    for tag in soup(["style", "nav", "footer", "header", "iframe", "noscript"]):
        tag.decompose()
    for tag in soup("script"):
        if tag.get("type") != "application/ld+json":
            tag.decompose()
    text = str(soup)
    text = re.sub(r"\n\s*\n+", "\n\n", text)
    return text[:200000]

For ecommerce, JSON-LD Product markup is gold. Many sites embed full product data in <script type="application/ld+json"> and you can extract it with zero LLM cost.

import json
from bs4 import BeautifulSoup

def try_jsonld_product(html: str) -> dict | None:
    soup = BeautifulSoup(html, "html.parser")
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string or "")
            if isinstance(data, dict) and data.get("@type") == "Product":
                return data
            if isinstance(data, list):
                for item in data:
                    if isinstance(item, dict) and item.get("@type") == "Product":
                        return item
        except json.JSONDecodeError:
            continue
    return None

Always try JSON-LD first. Fall back to LLM extraction only if it fails.

Model selection matrix

For extraction specifically (not full agent loops):

Model	Cost per 1k extractions	Quality on messy HTML	Best fit
GPT-4o-mini	$0.30	High	Default for high-volume
GPT-4o	$5.00	Highest	Hard cases, large schemas
Claude Haiku 3.5	$0.40	High	Default if Anthropic-native
Claude Sonnet 4.5	$5.50	Highest	Hard cases, large schemas
Gemini 1.5 Flash	$0.20	High	Cost-sensitive volume
Gemini 1.5 Pro	$3.50	Highest	Long context (2M tokens)
Llama 3.3 70B (self-host)	$0.05	Medium-high	Privacy-critical
Qwen 2.5 72B (self-host)	$0.05	Medium-high	Asia language pages

GPT-4o-mini is the default pick in 2026 for English-language extraction at scale. It is cheap enough that you stop optimizing prompts to save tokens. Claude Haiku is the default pick if your stack is Anthropic-native. Gemini Flash is the cheapest of the strong options.

For multilingual extraction (Thai, Indonesian, Korean, Vietnamese), Gemini Pro and Claude Sonnet outperform GPT-4o on local-language pages. Anthropic and Google both invested heavily in Asian language quality through 2025.

Validation and retry

Schema enforcement is necessary but not sufficient. The model can return schema-valid garbage. Validate semantically.

from pydantic import BaseModel, Field, validator

class Product(BaseModel):
    title: str = Field(min_length=1, max_length=500)
    price: float = Field(gt=0, lt=1_000_000)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    in_stock: bool

    @validator("title")
    def title_not_placeholder(cls, v):
        if v.lower() in ("loading", "untitled", "n/a", "..."):
            raise ValueError("placeholder title")
        return v

On validation failure, retry with a different model (escalate from 4o-mini to 4o) or with a hint in the prompt (“the previous extraction had price=0 which is invalid; try harder to find the actual price”).

Two-pass extraction for hard pages

For very messy HTML, a two-pass extraction often beats a single-pass attempt.

Pass one: ask the model to find and extract just the relevant region.

Pass two: ask the model to extract the structured fields from that region.

async def two_pass_extract(html: str) -> dict:
    # pass 1: locate
    locate = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Find the main product section in the HTML and return only that section's HTML."},
            {"role": "user", "content": html[:200000]},
        ],
    )
    region = locate.choices[0].message.content

    # pass 2: extract
    return await extract_product(region)

This pattern doubles cost but cuts noise enough that quality on hard pages goes up by 10-20 percent.

Adding context to the prompt

When the model is missing context (you know the page is about wireless mice, the model has to guess), supply it.

async def extract_with_context(html: str, hint: dict) -> dict:
    return await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_schema", "json_schema": {"name": "product", "schema": schema, "strict": True}},
        messages=[
            {"role": "system", "content": "Extract the product."},
            {"role": "user", "content": f"Page context: {hint}\n\nHTML:\n{html[:200000]}"},
        ],
    )

Hint can include the URL, the breadcrumb category, the expected currency. The model uses these to disambiguate.

Few-shot examples in the prompt

For new sites where extraction quality is initially poor, two or three labeled examples in the prompt boost accuracy by 10 to 25 percent.

async def extract_with_examples(html: str, examples: list[tuple[str, dict]]) -> dict:
    example_text = "\n\n".join(
        f"Example HTML:\n{ex_html[:5000]}\nExtracted: {json.dumps(ex_data)}"
        for ex_html, ex_data in examples
    )
    return await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_schema", "json_schema": {"name": "x", "schema": schema, "strict": True}},
        messages=[
            {"role": "system", "content": (
                "Extract the product. Here are examples of correct extractions:\n\n"
                + example_text
            )},
            {"role": "user", "content": html[:200000]},
        ],
    )

The cost increase is the example tokens (a few thousand input tokens) versus accuracy gains. Worth it for any site you scrape regularly.

Cost control patterns

Three patterns that cut extraction cost without sacrificing quality.

Cache by content hash. Hash the trimmed HTML. If you have seen it before, reuse the prior extraction. For sites that change rarely, this is huge.

Schema-first model selection. Start with the cheapest model. Validate. Escalate to a stronger model only on failure. Most pages succeed on the cheap path.

Sample then scale. For new sites, run 10 pages on the strong model and 10 pages on the cheap model. If results match, scale on cheap. If they diverge, stay on strong.

Comparison to other extraction patterns

Pattern	Cost per 1k pages	Setup time	Adaptability
Hand-written CSS selectors	$0	4 hours per site	Low
XPath with auto-discovery	$0	1 hour per site	Low
LLM with strict schema	$0.30-$5	30 minutes per schema	High
Vision model (page screenshot)	$5-$20	30 minutes	Highest

For a deeper look at vision-model extraction, see our scraping with vision models guide.

Real-world benchmark across 1000 product pages

We ran the same extraction across 1000 mixed product pages from Lazada, Shopee, Amazon, Best Buy, and Mercado Libre. Schema enforced strict; pre-processing applied. Numbers from March 2026:

Model	Accuracy	Cost per 1000 pages	p50 latency
GPT-4o-mini	96.4%	$0.30	1.2 s
GPT-4o	98.1%	$5.20	1.8 s
Claude Haiku 3.5	95.7%	$0.45	1.4 s
Claude Sonnet 4.5	98.4%	$5.80	2.1 s
Gemini 1.5 Flash	95.2%	$0.22	1.1 s
Gemini 1.5 Pro	97.6%	$3.60	2.4 s
Llama 3.3 70B (vLLM)	91.2%	$0.06	0.9 s

Headline: GPT-4o-mini at $0.30 per 1000 pages with 96.4 percent accuracy is the value pick. Sonnet 4.5 wins on accuracy but the 19x cost is rarely justified unless the data is high-stakes.

The Llama row is the surprise. Self-hosted Llama 3.3 70B on a single H100 reaches 91 percent accuracy at one-fifth the cost. For high-volume teams with privacy requirements, this is the right pick despite the lower ceiling.

Storing extracted data

Extracted records should land in a typed schema. Postgres with JSONB plus extracted columns is the production pattern.

CREATE TABLE extractions (
    id BIGSERIAL PRIMARY KEY,
    source_url TEXT NOT NULL,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    title TEXT NOT NULL,
    price NUMERIC(12,2) NOT NULL,
    currency CHAR(3) NOT NULL,
    in_stock BOOLEAN NOT NULL,
    raw_jsonb JSONB NOT NULL
);

Raw JSONB preserves the full extraction for reprocessing if your schema evolves. Typed columns give you the indices and analytics performance.

Multi-entity extraction

Many pages contain multiple records (a search results page with 30 products, a job board with 50 listings). Two patterns work.

Pattern A, single call with array schema. Wrap the entity in an array.

schema = {
    "type": "object",
    "properties": {
        "results": {
            "type": "array",
            "items": product_schema,
            "minItems": 1,
            "maxItems": 50,
        }
    },
    "required": ["results"],
    "additionalProperties": False,
}

The model returns all matches in one call. Cheaper than N calls but loses partial-success granularity if extraction fails midway.

Pattern B, find then extract. First call locates the items (returns N HTML snippets), second call (per snippet) extracts the structured fields. More expensive but more reliable on long pages.

For listings under 30 items, pattern A is fine. Above 30, pattern B starts to win because the single-call approach starts losing items at the end of the response.

Schema evolution

Production extraction schemas evolve. Adding a field is easy; the model just produces nulls until you start populating. Removing a field is harder because old data still has it. Renaming a field requires migration.

Two practices that prevent pain:

Version your schema explicitly with a schema_version field. Old records keep their version; new records get the new one. Your warehouse can handle both.

Never delete fields. Mark them deprecated and stop reading them. Models that produced the old field get nulls or are ignored.

class ProductV3(BaseModel):
    schema_version: Literal["3.0"] = "3.0"
    title: str
    price: float
    currency: str
    in_stock: bool
    sku: Optional[str]
    # NEW in v3
    primary_image_url: Optional[str] = None
    # DEPRECATED in v3 (kept for back-compat reads)
    seller_name: Optional[str] = None

The result: schema changes never break the warehouse, and you can re-extract historical data on the new schema lazily.

Production observability

Log every extraction with: source URL, model used, tokens consumed, schema name and version, validation result, retry count. This data lets you spot model regressions, cost spikes, and pages that consistently fail.

When to skip the LLM entirely

Three scenarios where the LLM is overkill:

Site embeds JSON-LD Product markup. Already structured, parseable in 5 lines. No LLM needed.

Site has a public API (or an obvious internal one). Hit the API directly.

Site has stable selectors that have not changed in 12 months. A traditional Playwright selector script costs nothing per page.

The LLM is the right tool when the data is in messy HTML with no machine-readable alternative and the page format changes often enough that selector maintenance is expensive.

Frequently asked questions

Why does the model sometimes return null when the field is clearly on the page?
Three causes: schema allows null (tighten it), the prompt allows guessing (forbid it), or the relevant region was trimmed off (trim less aggressively).

How do I extract from non-English pages?
Add a language hint to the system prompt (“the page is in Thai”). Use a model with strong multilingual training. Gemini Pro and Claude Sonnet outperform GPT-4o on Asian languages in 2026.

Can I extract from PDFs, images, or videos?
Yes for PDFs (most LLM APIs accept PDFs directly). Yes for images (vision models). Videos require frame extraction first.

How do I handle nested or repeated entities (a list of variants on a product page)?
Use array fields in the schema with items as object schemas. The model handles arbitrary length cleanly.

Should I use one schema per site or one global schema?
Global schema with optional fields is the production pattern. Per-site schemas explode in maintenance cost.

How do I handle currency conversion in extraction?
Extract the original currency and price as the model sees them. Convert to a canonical currency in a downstream step using a daily FX rate snapshot. Mixing currency conversion into the extraction prompt makes the model less reliable.

How do I extract dates and times reliably?
Use a string field with format: "date-time" (ISO 8601). Add a system prompt instruction “convert all dates to ISO 8601 in UTC”. The model handles timezone conversion better than most teams expect.

Is JSON mode the same as strict structured output?
No. JSON mode just guarantees the output parses as JSON. Strict structured output guarantees it matches your schema. Always prefer strict.

How do I extract from sites that change schemas often?
Use a “best effort” outer schema with a free-form additional_data JSONB field that captures whatever the model finds beyond the strict fields. This is how you keep extracting useful data through schema drift.

Can I extract relationships (this product is a variant of that product)?
Yes. Add a parent_sku field. Or for richer graphs, run a separate relationship extraction pass after collecting the entities.

Common production gotchas

A few patterns bite repeatedly:

The model returns a price like 1,299.99 as a string because the page showed it that way. Schema validation should reject strings in number fields, and the prompt should explicitly tell the model to strip commas and currency symbols.

For very long pages (over 200k chars), you exceed the context window. Pre-trim aggressively or chunk the page and run the extraction per chunk, merging results.

Caching by URL alone misses content updates. Cache by content hash of the trimmed HTML, not URL.

The additionalProperties: False constraint occasionally rejects model output that included a useful extra field. Decide consciously whether you want strictness (reject) or flexibility (allow and ignore).

Validation libraries differ in date handling. Pydantic v2’s date parser is stricter than v1. Pin the version.

For more patterns on the AI extraction stack, see the AI data collection category.

May 7, 2026