Edge AI scraping: running models at the network edge

Edge AI scraping has moved from research into mainstream production through 2024-2026. The combination of edge compute platforms (Cloudflare Workers AI, Vercel Edge Functions, Fastly Compute@Edge, AWS Lambda@Edge) and increasingly capable small models (Llama 3.2 3B, Phi-3 Mini, Mistral 7B, Gemma 2B) made it economical to run inference close to the network rather than in centralised model APIs. For scraping operators, this matters because edge AI changes the cost structure, the latency characteristics, the privacy posture, and the operational model of AI-augmented scraping. This guide walks through what edge AI actually is for scraping, the platforms that matter in 2026, the model choices that work, the patterns that fit edge constraints, and a practical playbook for moving inference closer to the data.

The audience is the data engineer or platform owner running AI-augmented scraping who wants to understand where edge fits.

What edge AI means for scraping

Three things at once.

First, model inference runs at edge locations rather than in central regions. Instead of round-tripping every request to a US-east OpenAI endpoint, inference happens at one of dozens (Cloudflare 300+, Vercel 25+, Fastly 90+) of edge locations close to the requester or the data source.

Second, the model is typically smaller. Edge platforms support small-to-mid-sized models (under 10B parameters typically) due to memory and cold-start constraints. Frontier models still run centrally; edge runs supporting models.

Third, the edge platform absorbs operational complexity. The edge runtime handles routing, scaling, cold starts, and deployment. The developer writes a function; the platform runs it close to the user.

For scraping, the implication is that AI tasks adjacent to the scrape (classification, extraction, summarisation, language detection, content moderation, deduplication) can move to the edge while heavyweight reasoning stays central.

For the broader emerging tech context, see the agentic browser revolution and RAG over scraped data.

The 2026 edge AI platforms

Four platforms in production scraping use:

Platform	Runtime	Native AI	Model catalogue
Cloudflare Workers AI	V8 isolates, Wasm	Yes (Workers AI)	50+ pre-deployed (Llama, Mistral, Whisper, embedding models)
Vercel Edge Functions	V8 isolates	Through partners	OpenAI, Anthropic, fal.ai integrations
Fastly Compute@Edge	Wasm	Limited	Custom WASM models possible
AWS Lambda@Edge	Node.js, Python	Limited	Bedrock-adjacent integrations

Cloudflare Workers AI is the most scraping-relevant in 2026 because it includes a substantial model catalogue running natively at the edge with no cold-start tax. The pricing model (per neurons-per-month) makes inference economical at scale.

A worked example: edge classification before central LLM

A common pattern: a scraper ingests millions of pages per day. Most pages need only basic classification (language, content type, freshness signal). A small fraction (say 5 percent) require deep LLM analysis. Running the LLM on every page is wasteful; running classification on every page is necessary.

The edge solution: deploy a lightweight classifier at the edge that runs on every scraped page and forwards only the relevant pages to the central LLM.

// Cloudflare Worker: edge classification gate
export default {
  async fetch(request, env) {
    const { url, html } = await request.json();
    const text = extractMainContent(html).slice(0, 2000);

    const classifyResult = await env.AI.run(
      "@cf/meta/llama-3.2-3b-instruct",
      {
        prompt: `Classify the following page. Return JSON with fields:
                 {category: news|product|profile|other, language: ISO code,
                  freshness_signal: stale|fresh|unknown, requires_deep_analysis: boolean}.
                 Content: ${text}`,
        max_tokens: 100,
      }
    );

    const classification = JSON.parse(classifyResult.response);

    if (classification.requires_deep_analysis) {
      return Response.json({
        forward: true,
        classification,
        url,
      });
    }
    return Response.json({
      forward: false,
      classification,
      url,
    });
  },
};

The economic outcome: for a 1M-page-per-day pipeline, edge classification at fractions of a cent per page filters down to 50K pages per day requiring central LLM analysis, with the central LLM bill dropping by 95 percent.

Where edge AI fits in scraping pipelines

Six concrete patterns:

Pattern	Edge model	Saves
Page classification	3B model	Central LLM tokens for irrelevant pages
Language detection	Tiny model (FastText, Lingua)	Routing logic complexity
Extraction (structured)	Small instruct model	Central LLM for routine extraction
Embedding generation	bge-small, e5-small	Centralised embedding API costs
Deduplication (semantic)	Embedding + similarity	Central pipeline duplicate work
Content moderation	Small classifier	Manual review queue

Each pattern moves work that does not need frontier-model intelligence to the edge, where it runs cheaper and faster.

For the broader pipeline pattern, see building scraping pipelines with Prefect 3.

Model choices for edge inference

The 2026 small-model landscape has matured significantly. The models that perform well at the edge:

Model	Parameters	Strengths	Notes
Llama 3.2 3B Instruct	3B	General instruct, good multilingual	Cloudflare native
Llama 3.2 1B	1B	Tiny, fast, basic tasks	Cloudflare native
Phi-3 Mini	3.8B	Strong reasoning for size	Multiple platforms
Mistral 7B	7B	Balanced; production-tested	Most platforms
Gemma 2 2B	2B	Strong instruction-following	Multiple platforms
BGE-Small	Embedding	Multilingual	Cloudflare native
E5-Small	Embedding	English-strong	Cloudflare native
Whisper Tiny	ASR	Audio transcription	Cloudflare native

Picking the right model is the central engineering decision. The pattern: pick the smallest model that meets your quality bar, validate against your evaluation set, deploy.

Decision tree: should this AI task run at the edge?

Q1: Does the task happen on every scraped page?
    ├── Yes -> Edge candidate (volume justifies edge optimisation).
    └── No  -> Q2
Q2: Is the task latency-sensitive (sub-100ms)?
    ├── Yes -> Edge candidate (round-trip to central API too slow).
    └── No  -> Q3
Q3: Does the task require frontier model reasoning?
    ├── Yes -> Stay central. Edge cannot match frontier capability.
    └── No  -> Q4
Q4: Does the task need to run close to data (privacy, residency)?
    ├── Yes -> Edge candidate.
    └── No  -> Q5
Q5: Is the model size under 10B parameters and the prompt under 4K tokens?
    ├── Yes -> Edge candidate.
    └── No  -> Stay central or hybrid.

The decision tree captures the typical fit. Volume, latency, capability ceiling, residency, and size constraints all push toward or away from edge.

Cost economics at the edge

Rough cost benchmarks for 1M classifications of 1000-token inputs in mid-2026:

Approach	Cost (USD)	Latency p50	Latency p99
OpenAI GPT-4o-mini (central)	150	800ms	3000ms
Anthropic Haiku (central)	250	600ms	2500ms
Cloudflare Workers AI Llama 3B	30	200ms	800ms
Self-hosted Llama 3B (on-prem)	50 (compute)	300ms	1200ms
Self-hosted Llama 70B	600 (compute)	1000ms	4000ms

The pattern: edge AI on small models is the cost leader for high-volume routing-style tasks. Frontier models at central locations are the right choice for nuanced reasoning. The architecture combines both.

For the deeper cost discussion, see AI scraping cost benchmark.

Privacy and residency

Edge AI improves privacy in two ways.

First, data does not have to leave the region. A page scraped from an EU site can be classified at an EU edge location without the content reaching US-based central inference. For GDPR compliance (covered in the GDPR scraping compliance guide), this matters.

Second, the data lifecycle is shorter. Edge functions are stateless by default; the page content is processed and forgotten. Central inference often involves logging and retention.

The privacy improvement is real but not absolute. Most edge platforms still log requests for billing and observability. A scraping operator with strict residency requirements should verify the platform’s data processing terms.

Operational patterns: deployment and observability

Three patterns that work in production.

Pattern one: managed edge with platform AI. Cloudflare Workers AI or Vercel Edge with provider AI. Lowest operational overhead. Use when the platform’s model catalogue meets your needs.

Pattern two: managed edge with custom model. Deploy your own small model to the edge via the platform’s WASM/binary support. Higher complexity, but unlocks proprietary or fine-tuned models. Cloudflare WASM and Fastly Compute support this.

Pattern three: hybrid edge plus central. The most common production pattern. Edge handles classification, embedding, simple extraction. Central handles reasoning, summarisation, complex extraction. The edge function makes the routing decision.

For deployment specifics on running scrapers at the edge themselves (not just AI), see running scrapers on Cloudflare Workers.

Edge embeddings and semantic search

A specific high-leverage pattern: generate embeddings at the edge as part of the scrape, before the data ever reaches central infrastructure.

export default {
  async fetch(request, env) {
    const { url, text } = await request.json();
    const embedding = await env.AI.run(
      "@cf/baai/bge-base-en-v1.5",
      { text }
    );
    await env.VECTOR_INDEX.upsert([
      { id: hash(url), values: embedding.data[0],
        metadata: { url, scraped_at: new Date().toISOString() } },
    ]);
    return new Response("OK");
  },
};

The embedding generation, which would historically have run in a central worker after the scrape completed, now runs at the edge as part of the scrape. The latency saving is real (no round-trip to central embedding API) and the cost saving is substantial.

For the broader vector database integration, see vector databases for scraping pipelines.

Comparison: edge AI platforms for scraping

Platform	Native AI catalogue	Cold start	Egress cost	Best for
Cloudflare Workers AI	50+ models	None (V8 isolates)	Free	Most scraping AI use cases
Vercel Edge Functions	Provider integrations	Minimal	Per request	Vercel-stack scraping
Fastly Compute@Edge	Custom WASM	Minimal	Per request	Custom-model needs
AWS Lambda@Edge	Bedrock adjacency	Cold start risk	Per request + AWS-typical	AWS-stack scraping
Self-hosted edge	Anything	None (warm)	Variable	Maximum control

Cloudflare Workers AI dominates the 2026 scraping use case because of the native model catalogue, the cold-start-free runtime, and the pricing model. Vercel and Fastly are competitive for specific stacks.

Limitations and where edge AI does not fit

Three classes of task remain central-only:

1. Frontier-model reasoning. Claude Opus, GPT-4o, Gemini Ultra do not run at the edge in 2026. Tasks that need their capabilities stay central.

2. Long-context tasks. Edge runtimes typically have memory caps that limit context to 8K-32K tokens. Long-document analysis stays central.

3. Stateful workflows. Edge functions are stateless; multi-step agentic workflows that require memory across steps need central orchestration even if individual steps run at the edge.

The pragmatic 2026 architecture splits the work: edge for high-volume simple tasks, central for low-volume complex tasks, with the edge making the routing decision.

For the broader agentic context, see the agentic browser revolution.

External references

Cloudflare Workers AI documentation is at developers.cloudflare.com/workers-ai. Vercel Edge Functions docs are at vercel.com/docs/functions/edge-functions. Fastly Compute@Edge is at docs.fastly.com/products/compute. The Hugging Face small-model leaderboard is at huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.

Operational checklist

Item	Owner	Done when
Identify edge-eligible AI tasks in pipeline	Engineering	Inventory complete
Select edge platform	Platform	Decision documented
Pick edge model per task	ML lead	Eval results signed off
Implement edge function with logging	Engineering	Deployed in staging
Run quality eval against central baseline	ML lead	Quality within tolerance
Implement central fallback for edge failures	Engineering	Fallback tested
Wire monitoring (latency, error rate)	Platform	Dashboards live
Document privacy posture	Compliance	Privacy assessment complete
Cutover with shadow mode	Engineering	Old path retired after stable

FAQ

What is the smallest model that performs well at the edge?
For routing-style classification, Llama 3.2 1B or Gemma 2 2B work well. For extraction, Llama 3.2 3B or Phi-3 Mini. Validate against your eval set.

Can frontier models run at the edge?
Not in 2026. Frontier models exceed edge memory and runtime constraints. Edge handles small/medium models; frontier stays central.

Is edge AI cheaper than central API?
For high-volume tasks (embeddings, classification, simple extraction), yes by 5-10x. For low-volume nuanced tasks, the difference is marginal.

What happens during edge AI outages?
Most platforms have multi-region failover. Build central fallback for the same task to maintain pipeline operation during outages.

Can I run my own fine-tuned model at the edge?
On Cloudflare WASM and Fastly Compute, yes if you can compile your model to WASM. On Workers AI, only models in the platform catalogue.

Extended edge AI scraping analysis

Edge AI scraping moves the model inference closer to the data, reducing round-trip latency and enabling on-device privacy. By 2026 three deployment patterns dominate.

1. Browser-side inference using WebGPU plus ONNX Runtime Web or transformers.js.
2. Edge-worker inference using Cloudflare Workers AI, Vercel Edge, or Fastly Compute.
3. Device-side inference using llama.cpp, MLX, or Apple Neural Engine.

For scraping the use cases include in-page extraction without round-tripping HTML to a server, content classification at the edge before storage, and PII redaction before centralised aggregation.

Pattern: WebGPU classification of scraped pages

import { pipeline, env } from "@xenova/transformers";

env.backends.onnx.wasm.proxy = true;

const classifier = await pipeline(
  "text-classification",
  "Xenova/distilbert-base-uncased-finetuned-sst-2-english",
  { device: "webgpu" }
);

async function classifyPage(html) {
  const text = stripHtml(html).slice(0, 2000);
  const result = await classifier(text);
  return result;
}

Pattern: Cloudflare Workers AI for edge extraction

export default {
  async fetch(request, env) {
    const url = new URL(request.url).searchParams.get("u");
    const page = await fetch(url).then(r => r.text());
    const text = stripHtml(page).slice(0, 4000);
    const completion = await env.AI.run(
      "@cf/meta/llama-3.1-8b-instruct",
      {
        messages: [
          { role: "system", content: "Extract product name, price, and availability from the text. Return JSON only." },
          { role: "user", content: text },
        ],
      }
    );
    return new Response(completion.response, {
      headers: { "Content-Type": "application/json" },
    });
  },
};

Pattern: on-device inference with llama.cpp

from llama_cpp import Llama

llm = Llama(
    model_path="./models/Phi-3-mini-4k-instruct-q4.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,
)

def extract(text, schema):
    prompt = f"Extract per schema: {schema}\n\nText: {text}\n\nJSON:"
    output = llm(prompt, max_tokens=512, stop=["\n\n"], temperature=0.0)
    return output["choices"][0]["text"].strip()

Privacy and compliance benefits

Edge inference provides three compliance benefits.

1. Personal data can be redacted before leaving the user’s device.
2. Cross-border transfer obligations can be reduced because data never leaves the jurisdiction.
3. Aggregation can be done on derived signals rather than raw personal data.

Comparison: edge AI deployment options 2026

Option	Latency to first token	Cost model	Privacy posture
Browser WebGPU	100-300ms	Free (user device)	Strongest
Cloudflare Workers AI	50-200ms	Per-request	Moderate
Vercel Edge	100-300ms	Per-request	Moderate
AWS Lambda + Bedrock	200-500ms	Per-token	Moderate
On-device (mobile)	50-200ms	Free (user device)	Strongest
Centralised GPU server	50-100ms	Per-token plus infra	Weakest

Model size and quality tradeoffs

Edge deployment forces smaller models. The 2026 sweet spots are.

• 1-3B parameters for browser WebGPU (Phi-3, Llama 3.2 1B/3B).
• 7-13B for edge workers with hosted GPU (Mistral, Llama 3.1 8B).
• 70B+ remains centralised for complex tasks.

A pattern is to route by task complexity. Simple extraction goes to the 1-3B edge model. Complex synthesis goes to a 70B centralised model. The router decides per request.

Additional FAQ

Is edge AI mature enough for production scraping?
Yes for classification, redaction, and simple extraction. Complex multi-step reasoning still benefits from larger centralised models.

How do I update edge models?
For browser WebGPU, version the model file and use service worker caching. For edge workers, use the platform’s deployment pipeline. For on-device, follow the platform’s app-update mechanism.

What about quality?
Quantised small models (4-bit, 8-bit) achieve 90-95 percent of full-precision quality on extraction tasks. Validate per use case.

How does this interact with cost?
Edge AI shifts cost from inference per-token to development complexity. The break-even depends on volume. Above one million requests per month edge often wins.

Common pitfalls in edge AI scraping deployments

Three failure modes show up consistently when teams move edge AI from prototype to production.

The first pitfall is silent quality regression after a model update. Cloudflare and similar platforms periodically refresh hosted model weights, and a model identifier like @cf/meta/llama-3.1-8b-instruct can point to different underlying weights over time. Pin specific model revisions where the platform allows, and run a daily eval against a fixed regression set so quality drops are caught within hours rather than weeks.

The second pitfall is treating the edge as stateful. Edge workers spin up and down across regions, and any state held in worker memory disappears between invocations. Scrapers that try to dedupe URLs in worker-local memory will see duplicates because two simultaneous workers in different regions hold different state. Push deduplication and rate-limit state to a shared store like Workers KV, Durable Objects, or a regional Redis.

The third pitfall is assuming WebGPU works everywhere. WebGPU shipped to most browsers by 2026 but coverage on older Android, locked-down enterprise browsers, and some mobile Safari versions remains spotty. A scraper that depends on browser-side WebGPU inference must implement a server-side fallback path and detect WebGPU availability at runtime, otherwise the pipeline silently produces no output for a segment of users.

The economics of edge versus centralised inference

The decision to run inference at the edge versus in a centralised GPU cluster is increasingly an economic decision rather than a technical one. The break-even point depends on volume, latency requirements, and privacy requirements.

For low-volume workloads (under 1 million inferences per month) centralised inference via API is typically cheapest. The fixed costs of edge deployment (model packaging, deployment pipeline, monitoring) outweigh the per-inference savings.

For medium-volume workloads (1-100 million inferences per month) edge becomes competitive. Cloudflare Workers AI, AWS Lambda with smaller models, and Vercel Edge offer per-request pricing that compares favourably to centralised API pricing. The decision typically rests on latency and privacy preferences.

For high-volume workloads (over 100 million inferences per month) edge typically wins materially. The marginal cost per inference at the edge approaches zero (the user device or the platform’s already-allocated resources), while centralised costs scale linearly.

The 2026 inflection has moved many real workloads into the edge-favouring zone. Classification, extraction, redaction, and short-form generation are increasingly profitable at the edge.

The model size frontier for edge

Edge deployment is constrained by model size. Browser WebGPU realistically supports 1-3 billion parameter models. Edge workers with hosted GPU support 7-13 billion parameter models. On-device with modern mobile silicon supports 1-7 billion parameter models depending on the device.

The 2024-2026 wave of small high-quality models (Phi-3, Llama 3.2, Mistral, Gemma) raised the quality floor at every size tier. A 3B-parameter model in 2026 outperforms a 13B model from 2023 on many extraction and classification tasks. The trend means edge-deployable models are increasingly capable of production-quality work.

Quantisation extends the frontier further. A 7B-parameter model quantised to 4-bit fits in roughly 4 GB of memory, which is achievable on modern phones and on Cloudflare Workers AI. Quantisation costs 1-3 percent quality on most tasks, which is usually acceptable for production extraction workloads.

Browser WebGPU as a deployment target

Browser WebGPU is the most exotic edge deployment target but also the most privacy-friendly. Inference happens entirely on the user’s device. No data leaves the browser. The site cost is the model file size (typically 1-3 GB for useful models).

The 2026 toolkit for WebGPU inference includes transformers.js (the JavaScript port of Hugging Face transformers), ONNX Runtime Web, and several specialised libraries. Each ships pre-quantised models that load quickly and run on consumer GPUs.

The user experience considerations for WebGPU inference include the model download (long on first visit, cached afterwards), the GPU memory consumption (must be considered alongside the page’s other GPU usage), and the inference latency (typically 100-500 ms per generation step, slow compared to centralised GPU but fast enough for many use cases).

A 2026 pattern that is gaining adoption is hybrid inference. The page first attempts WebGPU inference. If unavailable or unacceptably slow, the page falls back to an edge worker or centralised API. The fallback is invisible to the user but provides graceful degradation.

On-device inference for mobile and desktop

Mobile and desktop applications can ship inference models directly. Apple’s Core ML, Android’s Neural Networks API, and the cross-platform llama.cpp and MLC-LLM libraries provide the deployment pipelines.

The 2026 best practice for mobile on-device inference is to ship a quantised 1-3B parameter model with the app. The model handles common tasks (classification, summarisation, simple extraction) without network round-trips. Larger or more complex tasks fall back to a server.

Desktop deployment is less constrained by memory and battery. A desktop app can ship a 7-13B model and use the host GPU. The capability available is closer to centralised inference, with the privacy and latency advantages of local execution.

The 2026 release of high-quality 1-3B models that fit easily on consumer hardware made on-device inference economically attractive for the first time. Many applications that previously required server inference can now run locally with comparable quality.

Next steps

The fastest first move is to identify one high-volume AI task in your pipeline (classification, embedding, language detection) and prototype an edge implementation in Cloudflare Workers AI. The cost saving and latency improvement will speak for themselves. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the RAG over scraped data guide.

This guide is informational, not engineering or legal advice.

How to scrape Coupang Korea: a practical 2026 guide

Scrape Coupang Korea correctly in 2026 and you have access to the dominant ecommerce platform in one of the most digitally mature markets in the world. Coupang serves over 22 million active customers in South Korea, runs the country’s largest same-day delivery network (Rocket Delivery), and indexes hundreds of millions of SKUs ranging from groceries to electronics to fashion. For brand managers tracking pricing, agencies running competitive intelligence, or product teams sizing demand, Coupang is non-negotiable.

This guide covers Coupang Korea scraping end-to-end: which endpoints to hit, how to handle the bot defenses Coupang has stacked since their NYSE listing, how to deal with Korean character encoding and KRW pricing nuances, and how to manage Korean mobile carrier proxies. Working Python code throughout.

What Coupang Korea exposes

Three surfaces matter:

Surface	URL pattern	Best for
Product detail page	coupang.com/vp/products/{product_id}	Full extraction with reviews
Internal API	coupang.com/vp/products/{product_id}/items/{item_id}/vendor-items	Variant-level data
Search results	coupang.com/np/search?q={query}	Discovery

Coupang’s structure is more nested than Lazada or Shopee. A “product” can have multiple “items” (variants), each with multiple “vendor items” (different sellers offering the same item). For complete competitive intelligence, you need vendor-item-level data, not just product-level.

Anti-bot defenses

Coupang uses a custom bot defense stack assembled by the Coupang security team:

1. Cloudflare protection on the public web pages
2. Aggressive IP reputation scoring; data center IPs are heavily challenged
3. Custom JavaScript challenges that defeat headless Chromium with default settings
4. Header-based fingerprinting (specific Accept-Language and User-Agent combinations expected)

The recommended path in 2026: Korean mobile carrier IPs (KT, SK Telecom, LG U+), real Chromium driven through CDP with a Korean locale, and patient throttling.

Working browser-based scraper

import asyncio
import json
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup

async def scrape_coupang_kr(product_url: str, proxy: dict | None = None) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy=proxy,
            args=["--disable-blink-features=AutomationControlled"],
        )
        ctx = await browser.new_context(
            user_agent="Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1",
            locale="ko-KR",
            timezone_id="Asia/Seoul",
            extra_http_headers={"Accept-Language": "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7"},
            viewport={"width": 390, "height": 844},
        )
        page = await ctx.new_page()
        await page.goto(product_url, wait_until="networkidle", timeout=45000)
        html = await page.content()
        await browser.close()

    return _parse_coupang_html(html)

def _parse_coupang_html(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    title = soup.select_one("h2.prod-buy-header__title")
    price_el = soup.select_one(".total-price strong")
    original_price = soup.select_one(".price-amount.origin-price")
    rating_el = soup.select_one(".rating-star-num")
    review_count_el = soup.select_one(".count")
    stock_el = soup.select_one(".out-of-stock")

    return {
        "title": title.text.strip() if title else None,
        "price_krw": _parse_krw(price_el.text) if price_el else None,
        "original_price_krw": _parse_krw(original_price.text) if original_price else None,
        "rating": float(rating_el.get("style", "").replace("width:", "").replace("%;", "")) / 20 if rating_el else None,
        "review_count": int(review_count_el.text.strip("()").replace(",", "")) if review_count_el else None,
        "in_stock": stock_el is None,
    }

def _parse_krw(text: str) -> float:
    import re
    digits = re.sub(r"[^\d]", "", text or "")
    return float(digits) if digits else 0.0

asyncio.run(scrape_coupang_kr("https://www.coupang.com/vp/products/1234567890"))

Mobile user agent matters more here than on most sites. Coupang serves a more API-friendly (smaller, JSON-heavy) version to mobile clients.

Capturing the internal product API

Coupang’s product detail page makes several internal API calls. Intercepting them gives cleaner JSON than parsing HTML.

async def scrape_with_api_capture(url: str, proxy: dict | None = None) -> dict:
    api_payloads = {}
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=proxy)
        ctx = await browser.new_context(locale="ko-KR")
        page = await ctx.new_page()

        async def handler(resp):
            if "/vp/products/" in resp.url and resp.status == 200:
                try:
                    if "application/json" in resp.headers.get("content-type", ""):
                        api_payloads[resp.url] = await resp.json()
                except Exception:
                    pass

        page.on("response", handler)
        await page.goto(url, wait_until="networkidle", timeout=45000)
        await asyncio.sleep(2)
        await browser.close()
    return api_payloads

The intercepted payloads include the structured price, stock, vendor, and review data without you having to parse HTML.

Korean Won price handling

Korean Won uses the symbol ₩ and is whole-number (no fractional units). Prices appear as “1,234,500원” or “₩1,234,500”. Strip everything that is not a digit to parse:

import re

def parse_krw(s: str) -> float:
    return float(re.sub(r"[^\d]", "", s) or 0)

Conversion rates fluctuate but rough USD ratio in 2026 is around 1,400 KRW per USD. Always store the raw KRW value; convert only for display.

Korean character handling

Korean uses Hangul (한글) which is well-supported by UTF-8. Two specific gotchas:

First, Hangul has both completed syllable blocks (가, 나) and decomposed forms (Jamo). Coupang uses completed forms. Make sure your storage layer normalizes via NFC.

import unicodedata

def normalize_korean(s: str) -> str:
    return unicodedata.normalize("NFC", s)

Second, product titles often mix Hangul, Latin (brand names like Samsung, LG, Apple), and CJK ideographs (some traditional terms). Storage and indexing should support all three.

Mobile proxy rotation

Korean mobile carrier IPs (KT, SK Telecom, LG U+) are the cleanest source for Coupang scraping. Korean residential IPs work for low volume; mobile is required for sustained throughput.

import random

KR_MOBILE_PROXIES = [
    {"server": "socks5://us:pw@kr-kt-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@kr-skt-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@kr-lgu-1.proxy.example.com:1080"},
]

async def scrape_with_proxy(url: str):
    proxy = random.choice(KR_MOBILE_PROXIES)
    return await scrape_coupang_kr(url, proxy=proxy)

For broader proxy strategy in Asia, see best mobile proxy providers 2026.

Discovering product URLs

Coupang’s category structure is deeply nested. Sitemap discovery works:

import httpx
import xml.etree.ElementTree as ET

async def list_coupang_sitemap_urls(limit: int = 5) -> list[str]:
    sitemap_index = "https://www.coupang.com/sitemap.xml"
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.get(sitemap_index)
        root = ET.fromstring(r.text)
        ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
        sitemaps = [s.find("sm:loc", ns).text for s in root.findall("sm:sitemap", ns)][:limit]

        urls = []
        for sm_url in sitemaps:
            r = await client.get(sm_url)
            sm_root = ET.fromstring(r.text)
            urls.extend(u.find("sm:loc", ns).text for u in sm_root.findall("sm:url", ns))
        return urls

Coupang category landing pages also expose paginated listings:

async def search_coupang(query: str, page: int = 1) -> list[dict]:
    url = f"https://www.coupang.com/np/search?q={query}&page={page}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="ko-KR")
        pg = await ctx.new_page()
        await pg.goto(url, wait_until="networkidle")
        items = await pg.locator(".search-product").all()
        results = []
        for item in items:
            href = await item.locator("a").first.get_attribute("href")
            title = await item.locator(".name").text_content()
            results.append({"url": f"https://www.coupang.com{href}", "title": title.strip() if title else ""})
        await browser.close()
    return results

Korean address and seller data

Korean ecommerce uses a unique address structure (시 / 도 / 군 / 구 / 동 hierarchy). Vendor location data on Coupang typically appears at the city or district level. For brand intelligence, normalize to a hierarchical structure:

KOREAN_REGIONS = {
    "Seoul": "서울특별시",
    "Busan": "부산광역시",
    "Gyeonggi": "경기도",
    # ...
}

def normalize_korean_region(text: str) -> str | None:
    for english, korean in KOREAN_REGIONS.items():
        if korean in text or english in text:
            return english
    return None

For PIPA compliance, store at city level, not specific addresses.

Comparison to other Asian markets

Market	Bot defense	Volume	Mobile proxy required
Coupang Korea	High	Largest in Korea	Yes
Naver Smart Store	High	Very large	Yes
Gmarket Korea	Medium	Large	Recommended
11Street Korea	Medium	Medium	Optional
Rakuten Japan	High	Largest in Japan	Yes
Amazon Japan	Medium	Largest in Japan	Optional

For Japan specifically, see our Rakuten Japan scraping guide.

Stealth fingerprint hardening for Coupang

Coupang’s Cloudflare integration trips on the standard headless Chromium fingerprint. Combine the AutomationControlled patch with realistic Korean mobile fingerprints:

context_init = """
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, 'languages', {get: () => ['ko-KR', 'ko', 'en']});
Object.defineProperty(navigator, 'platform', {get: () => 'iPhone'});
Object.defineProperty(screen, 'colorDepth', {get: () => 32});
"""

await ctx.add_init_script(context_init)

Additionally, Coupang weighs the order and casing of HTTP headers. Use extra_http_headers to send a Korean-realistic header set in the right order:

ctx = await browser.new_context(
    extra_http_headers={
        "Accept-Language": "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Sec-Ch-Ua-Platform": '"iOS"',
        "Sec-Ch-Ua-Mobile": "?1",
    },
)

These details push the bot score from “high” to “medium” on Coupang’s internal scoring, which is enough to keep the session alive.

Coupang Rocket vs Marketplace

Coupang sells in two modes. Coupang-fulfilled (Rocket Delivery) products are sold by Coupang directly. Marketplace products are sold by third parties through Coupang. Both appear on the same product page, often with multiple vendor offers.

For competitive intelligence, vendor-level data matters. A single product might have 20 different vendors offering it at 20 different prices. The product-level price is meaningless without the vendor breakdown.

async def scrape_coupang_vendors(product_id: int, item_id: int):
    url = f"https://www.coupang.com/vp/products/{product_id}/items/{item_id}/vendor-items"
    # fetch with browser session, parse vendor list
    pass

Cost optimization tactics

Three patterns specifically valuable for Coupang scraping:

Block image and font requests. Coupang product pages load 4 to 6 MB of imagery by default. Blocking via Playwright route interception cuts proxy bandwidth by 75 percent.

Cache vendor data per item. The vendor list rarely changes hourly. Refresh vendor data once per day for most items, more often only for hot SKUs.

Use the API capture pattern over HTML parsing. The intercepted JSON contains structured data; HTML parsing is brittle as Coupang ships frontend updates.

Combined, these cut typical per-page cost from $0.038 to $0.019, roughly half.

Korean ecommerce calendar awareness

Korean ecommerce has different peak periods than Western or ASEAN markets. Plan capacity around:

• Lunar New Year (Seollal): late January to mid-February. Surge in gift purchases.
• Pepero Day (November 11): minor spike (different from China’s Singles Day but on the same date).
• Coupang’s own anniversary sales: irregular schedule, usually late summer.
• Christmas and New Year: standard global peak.

During peak windows, expect 3x normal load on Coupang infrastructure plus more aggressive bot defense. Scale your IP pool by 2x and increase pacing margins.

Production patterns

Three patterns matter.

First, throttle conservatively. 1-2 requests per second per IP. Coupang challenges aggressive scrapers within minutes.

Second, capture warm sessions. Sessions that have visited the homepage, browsed a category, and visited an item have a much lower challenge rate than cold sessions.

Third, monitor for the Cloudflare interstitial. If your scraper starts hitting “Just a moment…” pages, your IP pool is being challenged. Pause and rotate.

Vendor-level data extraction

Coupang’s vendor-items endpoint is the only way to see all sellers offering a single SKU. The shape:

async def fetch_vendor_items(product_id: int, item_id: int, session_cookies: dict) -> list[dict]:
    url = (f"https://www.coupang.com/vp/products/{product_id}/items/{item_id}"
           f"/vendor-items")
    async with httpx.AsyncClient(cookies=session_cookies) as c:
        r = await c.get(url, headers={
            "Accept": "application/json",
            "User-Agent": "Mozilla/5.0 ...",
            "Referer": f"https://www.coupang.com/vp/products/{product_id}",
        })
        return r.json().get("vendorItems", [])

Each vendor item includes price, stock, shipping cost, vendor name, vendor rating, and delivery type. For brand intelligence (catching unauthorized resellers, monitoring grey-market pricing), this data is gold.

Real benchmarks

A March 2026 production run, 10,000 Coupang products with the API capture pattern:

Metric	Value
Success rate	91%
Median latency per item	5.8 s
p99 latency	18 s
Cost per 1000 items	$19
Cloudflare challenge rate	5.3%
429 throttle rate	1.4%

Cloudflare challenges are the leading failure cause. With proper stealth and IP rotation, you can keep the rate under 6 percent.

Storage schema

CREATE TABLE coupang_products (
    id BIGSERIAL PRIMARY KEY,
    product_id BIGINT NOT NULL,
    item_id BIGINT,
    vendor_item_id BIGINT,
    url TEXT NOT NULL,
    title TEXT NOT NULL,
    price_krw NUMERIC(12,0) NOT NULL,
    original_price_krw NUMERIC(12,0),
    rating NUMERIC(3,2),
    review_count INTEGER,
    in_stock BOOLEAN NOT NULL,
    is_rocket BOOLEAN DEFAULT FALSE,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    raw_jsonb JSONB,
    UNIQUE(product_id, item_id, vendor_item_id)
);
CREATE INDEX idx_coupang_extracted_at ON coupang_products(extracted_at);

AI-driven extraction fallback

For pages where the deterministic JSON interception fails (Coupang ships UI updates frequently), fall through to LLM extraction:

async def scrape_with_fallback(url: str) -> dict:
    try:
        return await scrape_with_api_capture(url)
    except (NoPayloadError, KeyError):
        html = await fetch_html(url)
        return await llm_extract_product(html, schema=PRODUCT_SCHEMA)

The LLM fallback runs at roughly 4x the cost per page but catches the cases where the deterministic path breaks. This hybrid keeps the happy path fast and cheap while staying resilient to frontend changes.

Cost expectations

10,000 Coupang Korea products per month with Korean mobile proxies:

Component	Cost
Korean mobile proxy traffic (~2.5MB/page)	$80-$130
Browser compute	$40
LLM extraction (optional)	$30
Total	$150-$200

Korean mobile IPs are slightly cheaper than Indonesian mobile, partly because Korean carrier infrastructure has more capacity.

Legal considerations

Korea’s Personal Information Protection Act (PIPA) is strict. Public commercial data (product listings, prices, vendor names) is not personal data. Customer reviews that include real names are personal data and require care; the typical compliance pattern is to extract only ratings and review counts, not review text or reviewer names.

The Coupang terms of service prohibit automated access. Civil enforcement only; no criminal exposure for scraping public commercial data.

For broader compliance reading, see GDPR compliance for web scraping, which covers many of the same principles applied to Korean PIPA.

Coupang-specific data quirks

Several Coupang-only data points that other ecommerce platforms do not expose:

Rocket Wow membership pricing. Members get different prices on many SKUs. The page renders both prices and Wow-only prices appear with a Wow badge. Capture both.

Coupang Card discount. Coupang’s branded credit card offers an automatic discount that appears on the product page. Capture as a separate field; it affects price comparison logic.

Same-day delivery flag. The “Rocket Delivery” badge indicates next-day or same-day delivery. For demand intelligence, this flag is correlated with sales velocity.

Origin country. Coupang labels imported products with origin country (China, Korea, USA, etc). For brand and trade intelligence, this is essential.

def extract_coupang_specific(page_data: dict) -> dict:
    return {
        "wow_price_krw": page_data.get("wowPrice"),
        "card_discount_krw": page_data.get("cardDiscountAmount"),
        "is_rocket_delivery": page_data.get("rocketDelivery", False),
        "origin_country": page_data.get("originCountry"),
    }

Review and rating extraction

Coupang reviews are paginated and load lazily. Each review includes star rating, text, photos, and a buyer-helpful counter. The endpoint:

async def fetch_reviews(product_id: int, page: int = 1, size: int = 30) -> dict:
    url = (f"https://www.coupang.com/vp/product/reviews"
           f"?productId={product_id}&page={page}&size={size}")
    # uses the same session cookies as product fetches
    ...

For sentiment analysis, capture the text plus rating. For authenticity (counterfeit detection), photos are a strong signal because genuine buyers post product photos and fake reviews rarely do.

Frequently asked questions

Can I use Coupang’s Partner API?
Coupang has an Affiliate Partner API for sellers and an Open API for partners. If you qualify, official APIs are the safe path. For competitive intelligence (you are not a seller), scraping is the practical option.

Why does my scraper work for an hour then start failing?
IP reputation degradation. Mobile IPs survive longer than residential, but every IP eventually gets flagged with sustained traffic. Rotate aggressively.

How does Coupang’s anti-bot compare to Naver Smart Store?
Naver is harder. Coupang relies on Cloudflare plus custom challenges; Naver has its own homegrown defense plus deep integration with Korean identity verification. For Naver scraping, expect 2x the cost and 30 percent lower success rate.

Can I scrape Coupang affiliate links?
The affiliate program API gives you tracked product URLs you can include in content. The scraping pattern for product data is the same; only the URL structure adds a tracking parameter.

Can I scrape Coupang Eats (food delivery)?
Yes with similar patterns. Coupang Eats has a mobile-first interface that works best with mobile user agents and Korean mobile IPs.

How do I detect when a Coupang product moves between Rocket and Marketplace?
Track the is_rocket flag over time. A change from true to false often signals supply chain or pricing changes that brand managers care about.

What about Coupang Play (streaming) metadata?
Title and synopsis data are scrapable. View counts and engagement data are not exposed publicly.

How do I handle the seller location data?
Vendor profiles include city-level location for marketplace sellers. Store at city granularity; scraping shop-level address details ventures into PIPA territory.

What about Coupang Fresh (groceries)?
Same scraping pattern with a slightly different URL structure (coupang.com/vp/products/{id} is universal but Fresh items have additional perishability and chilled-delivery flags).

Can I scrape Coupang from outside Korea?
Yes for the public web pages, but mobile carrier IPs from Korea perform dramatically better. From a US IP, expect a 3x challenge rate.

How do I track price changes accurately on Coupang?
Snapshot daily for stable products, hourly for hot deals. Coupang prices can change multiple times per day during 11.11-style sales.

Does Coupang have a search-suggest API I can use for keyword discovery?
Yes, at coupang.com/np/search/suggestion?q={prefix}. Useful for brand monitoring and trend tracking.

Common production gotchas

• The Cloudflare challenge cookie expires after 30 minutes. Sessions need refresh more often than on Lazada or Shopee.
• Korean character encoding in URLs uses %EC%-style percent-encoding. URL parsing libraries usually handle it but logging may show garbled text.
• Some Coupang pages require login for full pricing visibility (loyalty pricing). Scraping anonymously gets you the public price tier only.
• The mobile site (m.coupang.com) returns slightly different DOM than the desktop site. Pick one and stick with it.
• Vendor data updates more frequently than product data. Re-scrape vendors weekly even if products are stable.

Storing variant data

Coupang’s nested product/item/vendor-item structure deserves a normalized schema. A relational design that scales:

CREATE TABLE coupang_product_master (
    product_id BIGINT PRIMARY KEY,
    title TEXT NOT NULL,
    brand TEXT,
    category_id INTEGER,
    first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE coupang_items (
    item_id BIGINT PRIMARY KEY,
    product_id BIGINT REFERENCES coupang_product_master(product_id),
    variant_attributes JSONB NOT NULL,
    first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE coupang_vendor_items (
    vendor_item_id BIGINT PRIMARY KEY,
    item_id BIGINT REFERENCES coupang_items(item_id),
    vendor_id BIGINT NOT NULL,
    vendor_name TEXT,
    is_rocket BOOLEAN DEFAULT FALSE,
    first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE coupang_price_history (
    id BIGSERIAL PRIMARY KEY,
    vendor_item_id BIGINT REFERENCES coupang_vendor_items(vendor_item_id),
    price_krw NUMERIC(12,0) NOT NULL,
    in_stock BOOLEAN NOT NULL,
    captured_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_coupang_price_history_vendor_time
    ON coupang_price_history(vendor_item_id, captured_at);

This shape supports the most common queries (price over time per vendor, which vendors offer SKU X, average price across vendors) without requiring expensive joins.

For more Asian ecommerce coverage, browse the ecommerce category.

Vector databases for scraping pipelines in 2026

Vector databases scraping pipelines are inseparable in 2026. Almost every meaningful scraping operation that powers RAG, semantic search, recommendation, or AI-assisted analysis ends up writing embeddings to a vector store. The choice of vector database matters more than most teams initially recognise: it shapes ingestion throughput, query latency, hybrid retrieval support, operational overhead, cost economics, and the migration path when the system grows. The market consolidated around five serious options in 2024-2025, and the mid-2026 picture is clearer than ever. This guide walks through the production-grade vector databases, the comparison criteria that matter for scraping workloads, the deployment patterns that work, and a selection framework your team can apply.

The audience is the data engineer or platform owner choosing a vector database for a scraping-driven AI pipeline.

Why vector databases matter for scraping

Three reasons.

First, embedding storage and similarity search at scale require purpose-built infrastructure. A scraping operation that ingests 10 million documents produces tens of millions of vectors (one per chunk, often more). Storing and querying these in a general-purpose database does not scale.

Second, the retrieval pattern is different from traditional databases. Vector queries are nearest-neighbour searches over high-dimensional vectors, with hybrid sparse-plus-dense often required. The right database makes hybrid trivial; the wrong database makes it custom code.

Third, the operational characteristics matter: ingestion throughput, query latency at p99, memory footprint, replication, and cost per million vectors all shape the production experience.

For the broader RAG context, see RAG over scraped data production patterns. For the MCP integration, see MCP for data engineers.

The 2026 vector database landscape

Five production-grade options:

Database	Type	Open source	Hosted	Strength
Qdrant	Purpose-built	Yes (Apache 2.0)	Yes (Qdrant Cloud)	Performance + filters
Weaviate	Purpose-built	Yes (BSD)	Yes (Weaviate Cloud)	Modules + multi-modal
Pinecone	Purpose-built	No	Yes only	Operational simplicity
pgvector (Postgres extension)	Embedded	Yes	Yes (Supabase, Neon, RDS)	Postgres-native
Milvus	Purpose-built	Yes (Apache 2.0)	Yes (Zilliz Cloud)	Massive scale

The choice between them is rarely about raw performance. All five can serve millions of queries per day. The choice is about operational fit, ecosystem, and the rest of your stack.

Qdrant: the performance and filtering favourite

Qdrant is a purpose-built vector database written in Rust. It launched in 2021 and matured through 2023-2025 to become the production favourite for performance-sensitive workloads.

Strengths:
– Excellent query performance at scale (millions of vectors).
– Strong payload filtering: combine vector search with metadata filters efficiently.
– Open source with a permissive licence (Apache 2.0).
– Mature client libraries (Python, TypeScript, Go, Rust).
– Hybrid search (dense + sparse) supported natively as of 2024.

Weaknesses:
– Self-hosting requires more operational sophistication than pgvector.
– Hosted Qdrant Cloud is reasonably priced but not the cheapest.
– Less ecosystem integration than Weaviate (modules) or pgvector (Postgres).

Best for: production scraping pipelines where filter combination and query performance matter; teams comfortable with self-hosted infrastructure.

A minimal Qdrant ingestion in Python:

from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

client = QdrantClient(host="qdrant.internal", port=6333)
client.recreate_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
)
client.upsert(
    collection_name="docs",
    points=[
        PointStruct(id=i, vector=embedding,
                    payload={"url": url, "scraped_at": ts, "text": text[:200]})
        for i, (embedding, url, ts, text) in enumerate(rows)
    ],
)

Weaviate: the modules and multi-modal favourite

Weaviate is purpose-built, written in Go, and launched in 2019. It has a stronger orientation around modular pipelines (built-in embedding generation, reranking, summarisation) and multi-modal data (images, audio, video alongside text).

Strengths:
– Modules: built-in connectors for OpenAI, Cohere, Hugging Face, ColBERT, and many more.
– Multi-modal native: cleaner support for cross-modal queries.
– GraphQL query language: convenient for complex retrieval.
– Open source (BSD licence) with managed hosting.

Weaknesses:
– More opinionated; the modular design adds complexity to simple use cases.
– Performance similar to Qdrant but the operational characteristics differ.
– Smaller community than pgvector or Pinecone.

Best for: multi-modal pipelines, teams that benefit from built-in embedding/reranking modules, GraphQL-friendly stacks.

Pinecone: the operational-simplicity choice

Pinecone is the original commercial vector database, launched 2019. It is hosted-only and proprietary. Its value proposition is operational simplicity: a managed service with predictable pricing, zero infrastructure ownership.

Strengths:
– Zero-ops: no self-hosting required.
– Predictable pricing model.
– Strong production reliability.
– Clean Python SDK, well-documented.

Weaknesses:
– Hosted-only; no self-host option.
– More expensive at scale than self-hosted alternatives.
– Closed source; no inspection of internals.
– Filter performance has historically lagged Qdrant.

Best for: teams that want to outsource vector database operations entirely; pre-production, smaller teams; situations where vendor lock-in is acceptable.

pgvector: the Postgres-native option

pgvector is a Postgres extension that adds vector data types and similarity search. It launched in 2021 and matured significantly in 2024-2025 with HNSW index support and improved performance.

Strengths:
– Postgres native: reuse existing Postgres expertise, tooling, backups, observability.
– Cost-effective when Postgres is already in the stack.
– Transactional consistency with relational data.
– Hosted everywhere (RDS, Supabase, Neon, CloudSQL).
– Open source, free.

Weaknesses:
– Performance lower than purpose-built databases at large scale (50M+ vectors).
– Index types (IVFFlat, HNSW) and parameter tuning require expertise.
– No native sparse-plus-dense hybrid search; requires combining with full-text search manually.

Best for: teams already running Postgres, smaller corpora (under 50M vectors), use cases where transactional consistency with relational data is valuable.

A pgvector setup with HNSW:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
  id BIGSERIAL PRIMARY KEY,
  embedding vector(1024),
  url TEXT,
  scraped_at TIMESTAMPTZ,
  text TEXT
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Milvus: the massive-scale option

Milvus is open source, written in C++ and Go, and designed for extreme scale (billions of vectors). It launched in 2019 and matured through 2024-2025 into the standard for the largest deployments.

Strengths:
– Scale: production deployments at billions of vectors and 10K+ QPS.
– Distributed architecture: separate compute and storage scale independently.
– Multiple index types (IVF_FLAT, IVF_SQ8, HNSW, DiskANN).
– Strong China-region adoption with mature Mandarin documentation.

Weaknesses:
– Operational complexity: distributed Milvus requires real DevOps investment.
– For corpora under 100M vectors, the complexity is overkill.
– The hosted version (Zilliz Cloud) is mature but less ecosystem-adopted.

Best for: extreme-scale deployments, teams with mature DevOps, organisations with massive scraping operations producing billions of vectors.

For the deeper proxy infrastructure question, see self-hosted proxy infrastructure.

Comparison matrix

Dimension	Qdrant	Weaviate	Pinecone	pgvector	Milvus
Open source	Yes	Yes	No	Yes	Yes
Self-hosting	Yes	Yes	No	Yes (via Postgres)	Yes
Managed hosting	Yes	Yes	Yes only	Yes (Supabase, Neon, RDS)	Yes (Zilliz)
Hybrid search native	Yes	Yes	Limited	Manual	Yes
Multi-modal	Limited	Strong	Limited	Limited	Strong
Filter performance	Excellent	Good	Moderate	Good	Excellent
Scale ceiling	100M+ vectors	100M+ vectors	100M+ vectors	50M vectors	10B+ vectors
Ecosystem maturity	High	High	High	High (Postgres)	High
Best client lang	Python, TS, Go	Python, GraphQL	Python	SQL	Python, Java, Go
Cost at 10M vectors	Low (self) / Medium (cloud)	Low (self) / Medium (cloud)	Medium-High	Low (self)	Low (self)
Cost at 1B vectors	High (self)	High (self)	Highest (cloud only)	Not recommended	Best for scale

Decision tree: pick a vector database

Q1: Is your team running Postgres already?
    ├── Yes -> Q2
    └── No  -> Q3
Q2: Will the corpus stay under 50M vectors for the next 18 months?
    ├── Yes -> pgvector. Reuse the stack.
    └── No  -> Q3
Q3: Is operational simplicity (no self-host) the priority?
    ├── Yes -> Pinecone (commercial); Qdrant Cloud or Weaviate Cloud (open).
    └── No  -> Q4
Q4: Is the corpus at 1B+ vectors or expected to be?
    ├── Yes -> Milvus.
    └── No  -> Q5
Q5: Are filter combinations central to your queries?
    ├── Yes -> Qdrant.
    └── No  -> Q6
Q6: Do you need built-in modules or strong multi-modal?
    ├── Yes -> Weaviate.
    └── No  -> Qdrant (sensible default).

The decision tree handles 80 percent of cases. Edge cases (regulatory data residency, specific cloud provider lock-in, language-specific embedding tooling) override.

Production deployment patterns

Three patterns for production deployment.

Pattern one: managed cloud, single region. Pinecone, Qdrant Cloud, Weaviate Cloud, Zilliz Cloud, or Supabase pgvector. Lowest operational overhead. Suitable for most teams.

Pattern two: self-hosted on Kubernetes. Qdrant, Weaviate, Milvus, or pgvector deployed via Helm charts on EKS/GKE/AKS. Medium operational overhead. Required for data-residency or cost-sensitive deployments.

Pattern three: self-hosted on bare metal. Same databases as pattern two, deployed directly on dedicated hardware. Highest operational overhead, lowest cost per vector. Required for the largest deployments where cloud egress and managed-service margins dominate.

For the broader infrastructure question, see building scraping pipelines with Prefect 3.

Operational characteristics

Beyond the headline benchmark numbers, the operational characteristics that matter:

Characteristic	Why it matters
Ingestion throughput	Determines time to backfill a large corpus
Query p99 latency	Determines user-facing response time
Memory footprint	Determines hosting cost
Index build time	Determines time-to-first-query after data load
Replication and HA	Determines uptime SLO
Backup and restore	Determines disaster recovery RTO/RPO
Schema evolution	Determines pain when payload schema changes
Multi-tenancy	Determines whether you can run shared collections

A team that picks a database without evaluating these often discovers them painfully in month two of production.

Hybrid search implementation

Hybrid retrieval (dense embedding + sparse keyword) is the production default in 2026. The implementation differs across databases:

Database	Hybrid mechanism
Qdrant	Native sparse vectors; combine with dense via fusion
Weaviate	Native hybrid query mode
Pinecone	Sparse-dense indexes (sparse vectors with dense)
pgvector	Combine with Postgres full-text search; manual fusion
Milvus	Native hybrid via multi-field queries

A Qdrant hybrid query:

from qdrant_client.models import Prefetch, Fusion, FusionQuery

results = client.query_points(
    collection_name="docs",
    prefetch=[
        Prefetch(query=dense_vector, using="dense", limit=20),
        Prefetch(query=sparse_vector, using="sparse", limit=20),
    ],
    query=FusionQuery(fusion=Fusion.RRF),
    limit=5,
)

Reciprocal Rank Fusion (RRF) is the standard merger.

For the deeper retrieval discussion, see RAG over scraped data.

Cost economics at scale

Rough cost benchmarks for storing and querying 100M vectors of 1024 dimensions in mid-2026:

Option	Storage cost (monthly)	Query cost per 1M	Notes
Qdrant Cloud	USD 800-1500	Included up to volume	Predictable
Weaviate Cloud	USD 900-1800	Included up to volume	Module surcharges
Pinecone	USD 1200-2500	USD 0.40	Tier-based
pgvector on RDS	USD 600-1200	Included	Suboptimal at this scale
Milvus self-hosted	USD 400-800	Included	Plus DevOps overhead
Qdrant self-hosted	USD 300-600	Included	Plus DevOps overhead

Self-hosting wins on raw cost. Managed wins on total cost of ownership when DevOps capacity is constrained.

External references

The Qdrant documentation is at qdrant.tech/documentation. Weaviate’s documentation is at weaviate.io/developers/weaviate. Pinecone’s docs are at docs.pinecone.io. pgvector’s repository is at github.com/pgvector/pgvector. Milvus is at milvus.io. Vector database benchmarks are tracked at vectordbbench.com.

Migration patterns

Teams frequently migrate vector databases as scale grows. The pattern that works:

1. Start with pgvector if Postgres exists, or Qdrant if not.
2. Migrate when corpus or QPS exceeds the comfortable operating range (50M vectors for pgvector; 100M+ for purpose-built single-node).
3. Plan migration as: dual-write during transition; read-cutover after validation; old database retired after 2 weeks.
4. Embedding models do not need to change unless the migration coincides with a model upgrade. Vector dimensions must match.

A typical migration is one engineer-month of effort. The cost is real but predictable.

FAQ

Which vector database is best for scraping pipelines?
Qdrant is the sensible default for most. pgvector if you already run Postgres at sub-50M scale. Pinecone if zero-ops is paramount.

Do I need a vector database if I use OpenAI embeddings?
Yes. The embeddings need to be stored and searched somewhere. OpenAI provides embeddings; vector databases provide retrieval.

Is pgvector good enough for production?
Yes, up to about 50M vectors. Beyond that, the index build times and query latencies push toward purpose-built options.

What about hybrid sparse-plus-dense search?
Native in Qdrant, Weaviate, and Milvus. Manual in pgvector. Limited in Pinecone unless using sparse-dense indexes.

Can I run multi-tenant collections?
All five support some form of multi-tenancy via collection separation or payload filtering. Implementation differs.

Extended vector database analysis

The vector database market consolidated around several production-ready options in 2026. The choice depends on scale, latency, and operational preference.

• pgvector on PostgreSQL. Best for teams already running Postgres. Hits 100M vector scale comfortably with HNSW indexing. Strong filter performance via standard SQL.
• Qdrant. Rust-based, excellent filter performance, strong for hybrid search. Self-hosted or managed.
• Weaviate. Schema-driven, GraphQL surface, strong for multi-tenancy.
• Pinecone. Managed only, simplest operations, highest cost per vector.
• Milvus. High scale (billions of vectors), more operational complexity.
• LanceDB. Embedded, columnar, best for analytics-style workloads.

Production ingestion pattern with deduplication

import hashlib
from typing import List, Dict

class IngestionPipeline:
    def __init__(self, embedder, vector_store, dedupe_table):
        self.embedder = embedder
        self.vector_store = vector_store
        self.dedupe = dedupe_table

    def doc_hash(self, doc: Dict) -> str:
        content = f"{doc['url']}|{doc['text']}"
        return hashlib.sha256(content.encode()).hexdigest()

    async def ingest(self, docs: List[Dict]) -> int:
        new_docs = []
        for doc in docs:
            h = self.doc_hash(doc)
            if not await self.dedupe.exists(h):
                doc["_hash"] = h
                new_docs.append(doc)
        if not new_docs:
            return 0
        embeddings = await self.embedder.embed_batch(
            [d["text"] for d in new_docs], batch_size=64
        )
        records = [{
            "id": d["_hash"],
            "vector": e,
            "metadata": {
                "url": d["url"],
                "scraped_at": d.get("scraped_at"),
                "source": d.get("source"),
                "purpose": d.get("purpose"),
            },
        } for d, e in zip(new_docs, embeddings)]
        await self.vector_store.upsert(records)
        for d in new_docs:
            await self.dedupe.add(d["_hash"])
        return len(new_docs)

Index choice matters

The HNSW vs IVF-PQ vs DiskANN choice affects latency, recall, and memory.

• HNSW. In-memory graph index. Best recall and latency. RAM-bound.
• IVF-PQ. Quantised inverted file. Memory-efficient at moderate recall cost.
• DiskANN. Disk-based graph index. Scales beyond RAM. Slightly higher latency.
• ScaNN. Google’s hybrid. Strong recall and speed in benchmarks.

A 2026 production choice often pairs HNSW for the hot tier (last 30 days, in-memory) with IVF-PQ or DiskANN for the cold tier.

Filter performance: pre-filter vs post-filter

Pre-filtering applies the metadata filter before the vector search. Post-filtering applies it after. Pre-filter is correct but slower when filter selectivity is low. Post-filter is faster but may return fewer than k results.

A 2026 pattern is adaptive filtering. The query planner estimates filter selectivity and chooses pre or post per query.

def adaptive_search(query_vec, filter_dict, k=10, selectivity_threshold=0.05):
    estimated = estimate_selectivity(filter_dict)
    if estimated < selectivity_threshold:
        return vector_store.search(query_vec, k=k, filter=filter_dict, mode="pre")
    else:
        return vector_store.search(query_vec, k=k*3, filter=filter_dict, mode="post")[:k]

Comparison: vector databases 2026

DB	Max scale	Latency p95	Filter perf	Best for
pgvector	100M+	20-50ms	Excellent (SQL)	Postgres shops
Qdrant	1B+	10-30ms	Excellent	Hybrid search
Weaviate	500M+	15-40ms	Good	Multi-tenant
Pinecone	Multi-billion	30-100ms	Moderate	Managed simplicity
Milvus	10B+	10-50ms	Good	Massive scale
LanceDB	100M+	20-60ms	Good	Embedded analytics

Cost optimisation patterns

Vector database costs follow three drivers.

1. Storage (per million vectors per month).
2. Compute for index building and query.
3. Network egress for hosted services.

The 2026 cost-cutting patterns are.

• Quantisation (PQ, OPQ, scalar quantisation) to reduce vector size 4x to 32x.
• Dimensionality reduction via Matryoshka embeddings to truncate vectors at query time.
• Tiered storage with hot, warm, cold partitions.
• Embedding model swap to a smaller model for cost-sensitive workloads.

Additional FAQ

Should I use a vector DB or a SQL extension?
For most teams pgvector is enough up to 100M vectors. Beyond that consider a dedicated vector DB.

How do I handle incremental updates?
HNSW supports insert efficiently. Delete is harder; many production systems use tombstones plus periodic reindex.

What about hybrid search?
Most modern vector DBs support BM25-plus-vector fusion via reciprocal rank fusion or weighted scores. Use it.

How do I version embeddings?
When the embedding model changes you must re-embed. Maintain two indexes during the transition and dual-write.

The choice between embedded and dedicated vector databases

A team building a scraping-plus-RAG pipeline faces an early decision: embedded vector database or dedicated. Embedded options (LanceDB, Chroma in single-node mode, FAISS) live in the application process. Dedicated options (Qdrant, Weaviate, Milvus, Pinecone) live in their own service.

Embedded wins for prototyping, single-node deployments, and analytics-style workloads where the embedding store is read more than written. Dedicated wins for multi-service deployments, multi-team usage, high-write workloads, and operational requirements like backup and replication.

The 2026 pattern is to start embedded for the first 90 days of a project and migrate to dedicated when the project graduates to production. The migration is non-trivial but well-traveled. The embedded prototype validates the data model and the schema before the dedicated commitment.

A specific case worth calling out is pgvector. pgvector lives in PostgreSQL and benefits from the operational maturity of Postgres. For teams that already run Postgres, pgvector is often the right answer up to 100 million vectors. The savings on operating an additional database service often outweigh the marginal performance benefits of a dedicated vector DB at moderate scale.

The embedding model decision

The embedding model is the foundation of every vector store. A change of embedding model requires re-embedding the entire corpus, which is expensive at scale. The model choice should therefore be considered carefully.

The 2026 leaderboard (MTEB) lists models by retrieval quality on standard benchmarks. The current state of the art for English is around 70+ on the average MTEB score. The popular open-weight choices include the BGE family, the Jina embeddings v3, and the Cohere multilingual embeddings (proprietary but well-regarded).

Dimensionality matters for storage and latency. A 1536-dimensional embedding takes 6 KB per vector at float32. A 768-dimensional embedding takes 3 KB. Smaller dimensions also support faster nearest neighbour search. The 2024 Matryoshka representation learning approach lets a single model produce embeddings that can be truncated to smaller dimensions with graceful quality degradation.

For multilingual scraping the choice narrows. Models trained on diverse language data perform better on cross-lingual queries. The Cohere multilingual embeddings, the BGE-M3 model, and the Jina multilingual variants are the strong open choices.

The chunking strategy decision

Chunking is the most underrated decision in vector pipelines. The chunk size, the chunking strategy, and the overlap all materially affect retrieval quality.

Fixed-size chunking (split every 512 tokens) is simple but breaks semantic boundaries. Document-structure-aware chunking respects headings, paragraphs, code blocks, and tables. Semantic chunking uses a small model to find sentence boundaries that preserve meaning.

The 2026 pattern is to use document-structure-aware chunking as the default, with semantic chunking for prose-heavy content where structure is weak. Fixed-size chunking remains useful as a fallback for content with no exploitable structure.

Overlap improves retrieval quality at the cost of storage. A typical 2026 default is 10-20 percent overlap. Below 10 percent the boundary effects degrade retrieval. Above 20 percent the marginal benefit is small.

Cost optimisation in production

A production vector store at scale costs real money. Three optimisation patterns drive material savings.

The first is quantisation. Standard float32 vectors can be compressed to int8 (4x reduction) or int4 (8x reduction) with small recall impact. Product quantisation goes further (32x reduction) with more recall impact. The 2026 best practice is to ship int8 quantised indexes for hot data.

The second is tiered storage. Hot data (last 30 days) lives in HNSW in-memory. Warm data (last year) lives in a quantised disk-backed index. Cold data (older) lives in compressed object storage with on-demand re-indexing. The tier boundaries are workload-specific.

The third is matryoshka truncation. A model trained for matryoshka representation produces embeddings that work at multiple dimensionalities. The vector store can store the full dimension and serve at a smaller dimension when latency matters, or vice versa.

Each pattern stacks. A vector store using all three can be 50-100x cheaper than a naive deployment of the same workload, with modest recall impact. The 2026 best practice is to layer the techniques rather than choose one.

Next steps

If you have not picked a vector database yet, evaluate Qdrant and pgvector for your specific workload. An afternoon of prototyping with both will tell you more than weeks of comparison reading. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the RAG over scraped data guide.

This guide is informational, not engineering or legal advice.

How to scrape Shopee Indonesia in 2026

Scrape Shopee Indonesia at scale and you tap into the largest ecommerce market in Southeast Asia. Indonesia has more than 90 million Shopee monthly active users, the platform processes billions of dollars in GMV per quarter, and Shopee.co.id serves a different SKU mix than Shopee Singapore or Shopee Thailand. Brand managers, agencies, and price intelligence teams cannot get a complete ASEAN picture without it.

Shopee is also the hardest ASEAN ecommerce target to scrape. Sea Limited’s bot defense team has shipped some of the most aggressive anti-scraping infrastructure in the region, and what worked in 2023 stopped working months ago. This guide covers what works in 2026: the right endpoints, the right proxies, the right stealth defaults, and working Python code that produces clean structured records from Shopee Indonesia.

What Shopee Indonesia exposes

Three surfaces produce useful data:

Surface	URL pattern	Best for
Product detail page	shopee.co.id/{slug}-i.{shop_id}.{item_id}	Full single-product extraction
Internal API	shopee.co.id/api/v4/item/get	High-throughput product data
Shop page	shopee.co.id/{shop_username}	Seller catalog discovery

The internal /api/v4/item/get endpoint returns a clean JSON object with everything: title, price (in IDR), stock, ratings, variations, shop info. This is the highest-value endpoint and the one most defended.

Anti-bot defenses

Shopee uses a custom bot defense stack that combines:

1. PerimeterX (now HUMAN) bot management on the public web pages
2. Custom request signing on internal API endpoints (the X-API-SOURCE and X-Csrftoken headers)
3. Aggressive IP reputation scoring; data center IPs are nearly useless
4. JS-only fingerprinting with custom challenges that defeat headless Chromium with default settings

A clean extraction at scale needs three things in 2026: Indonesian mobile carrier IPs, a real (not headless-stealth-patched) Chromium driven through CDP, and either a fresh PerimeterX challenge solve or a captured warm session.

Working browser-based scraper

import asyncio
import json
import re
from playwright.async_api import async_playwright

async def scrape_shopee_id(item_url: str, proxy: dict | None = None) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy=proxy,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--disable-features=IsolateOrigins,site-per-process",
            ],
        )
        ctx = await browser.new_context(
            user_agent="Mozilla/5.0 (Linux; Android 13; SM-A546B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36",
            locale="id-ID",
            timezone_id="Asia/Jakarta",
            viewport={"width": 412, "height": 915},
        )
        page = await ctx.new_page()

        # intercept the internal API call that fires on page load
        api_payload = {}
        async def handle_response(resp):
            if "/api/v4/item/get" in resp.url:
                try:
                    api_payload["data"] = await resp.json()
                except Exception:
                    pass
        page.on("response", handle_response)

        await page.goto(item_url, wait_until="networkidle", timeout=45000)
        await asyncio.sleep(2)
        await browser.close()

    return _normalize_shopee_id(api_payload.get("data", {}).get("data", {}))

def _normalize_shopee_id(item: dict) -> dict:
    if not item:
        return {"error": "no_item_data"}
    return {
        "item_id": item.get("itemid"),
        "shop_id": item.get("shopid"),
        "title": item.get("name"),
        "brand": item.get("brand") or None,
        "price_idr": item.get("price", 0) // 100000,  # Shopee stores price in micros
        "original_price_idr": item.get("price_before_discount", 0) // 100000,
        "stock": item.get("stock"),
        "sold": item.get("historical_sold"),
        "rating": item.get("item_rating", {}).get("rating_star"),
        "review_count": item.get("item_rating", {}).get("rating_count", [None])[0],
        "in_stock": (item.get("stock") or 0) > 0,
    }

asyncio.run(scrape_shopee_id("https://shopee.co.id/example-i.12345678.987654321"))

The trick is intercepting the API response that fires when the page loads. This avoids reverse-engineering the request signing. The page does the signing for you.

Direct API path with X-Csrftoken

For teams willing to maintain the request signing, hitting /api/v4/item/get directly is dramatically faster than driving a browser. The endpoint requires:

• X-API-SOURCE: pc header
• X-Csrftoken header (extracted from the csrftoken cookie set on first page visit)
• A valid session cookie (SPC_EC and friends) from a warm session
• Correct Referer matching the product slug

async def fetch_item_api(item_id: str, shop_id: str, session_cookies: dict) -> dict:
    url = f"https://shopee.co.id/api/v4/item/get?itemid={item_id}&shopid={shop_id}"
    csrf = session_cookies.get("csrftoken", "")
    async with httpx.AsyncClient(cookies=session_cookies) as c:
        r = await c.get(url, headers={
            "X-API-SOURCE": "pc",
            "X-Csrftoken": csrf,
            "Referer": f"https://shopee.co.id/i.{shop_id}.{item_id}",
            "User-Agent": "Mozilla/5.0 ...",
        })
        return r.json()

The session cookies expire after roughly 2 hours of inactivity. Refresh via a browser warmup before the next batch.

Capturing warm session cookies

async def capture_session_cookies() -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=random.choice(ID_MOBILE_PROXIES))
        ctx = await browser.new_context(locale="id-ID")
        pg = await ctx.new_page()
        await pg.goto("https://shopee.co.id", wait_until="networkidle")
        await asyncio.sleep(3)  # let JS set all cookies
        cookies = {c["name"]: c["value"] for c in await ctx.cookies()}
        await browser.close()
    return cookies

A captured session is good for hundreds of API calls before getting throttled. Rotate sessions across IPs.

Indonesian Rupiah price handling

Shopee stores prices in micros: the integer field price is the price in Rupiah multiplied by 100,000. A 25,000 IDR product has price: 2_500_000_000. Always divide by 100,000 before display.

def parse_idr_micros(micros: int) -> float:
    return micros / 100_000

Indonesian Rupiah display formats use period as thousands separator and comma as decimal: 25.000,50 IDR. When you display extracted values back to users, format accordingly.

Indonesian language handling

Shopee Indonesia listings are predominantly in Bahasa Indonesia. Product titles often include both English brand name and Indonesian descriptors. Both are useful for matching across catalogs.

For language detection (useful for downstream analytics), langdetect works on Indonesian:

from langdetect import detect
title = "Sepatu Nike Air Max Original Pria"
print(detect(title))  # 'id'

Mobile proxy rotation

Indonesian mobile carriers (Telkomsel, Indosat, XL, Smartfren) provide the cleanest IP pool for Shopee Indonesia scraping. Mobile IPs from these carriers carry low suspicion scores.

import random

ID_MOBILE_PROXIES = [
    {"server": "socks5://us:pw@id-telkomsel-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@id-indosat-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@id-xl-1.proxy.example.com:1080"},
]

async def scrape_with_proxy(url: str):
    proxy = random.choice(ID_MOBILE_PROXIES)
    return await scrape_shopee_id(url, proxy=proxy)

For broader proxy strategy, see our best mobile proxy providers 2026 review.

Discovering product URLs at scale

Shopee provides category landing pages with paginated listings:

async def list_shopee_id_category(category_id: int, page: int = 0) -> list[dict]:
    url = f"https://shopee.co.id/api/v4/recommend/recommend?bundle=category_landing_page&cat_level=1&catid={category_id}&limit=60&offset={page*60}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="id-ID")
        pg = await ctx.new_page()
        # warm the session by visiting the category page first
        await pg.goto(f"https://shopee.co.id/Computer-Aksesoris-cat.{category_id}", wait_until="networkidle")
        # then fetch the recommend API
        resp = await pg.request.get(url)
        data = await resp.json()
        await browser.close()

    items = data.get("data", {}).get("sections", [{}])[0].get("data", {}).get("item", [])
    return [{"item_id": i["itemid"], "shop_id": i["shopid"], "name": i["name"]} for i in items]

The warm-up visit to the category page sets cookies that the recommend API requires.

Stealth fingerprint hardening for Shopee

Shopee’s PerimeterX (now HUMAN) defense is more aggressive than Lazada’s. The fingerprint hardening that works:

context_init_script = """
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, 'plugins', {get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'languages', {get: () => ['id-ID', 'id', 'en']});
window.chrome = {runtime: {}, app: {}};
"""

# inside context creation
await ctx.add_init_script(context_init_script)

These patches reduce the PerimeterX score from “high bot likelihood” to “low to medium”. Combined with mobile IPs and warm sessions, success rates climb from roughly 60 percent on bare Playwright to over 92 percent.

The other harder defense is mouse movement scoring. PerimeterX times mouse trajectories and flags too-linear patterns. For high-volume scraping, simulating curved mouse movements from screen edge to clicked element is worth the implementation effort.

async def humanlike_navigate(page, target_x, target_y):
    import math, random
    steps = 30
    start_x, start_y = random.randint(0, 100), random.randint(0, 100)
    for i in range(steps):
        t = i / steps
        x = start_x + (target_x - start_x) * t + random.uniform(-3, 3)
        y = start_y + (target_y - start_y) * t + math.sin(t * math.pi) * 40
        await page.mouse.move(x, y)

Comparison to other ASEAN markets

Market	Bot defense	Volume	Mobile proxy required
Shopee Indonesia	Highest	Largest	Yes
Shopee Thailand	Highest	Very high	Yes
Shopee Vietnam	High	High	Yes
Shopee Malaysia	High	Medium	Recommended
Shopee Singapore	Medium	Smaller	Optional
Shopee Philippines	High	High	Recommended

Indonesian Shopee is the largest market in the region and also the most defended. The patterns here translate directly to other Shopee markets with country-specific proxy pools.

For Lazada-specific patterns, see scrape Lazada Thailand.

Production patterns

Three patterns matter for sustained scraping.

First, persistent contexts. Shopee tracks session cookies. A warm session that has visited a category page and an item page survives longer than a cold session.

async def warm_session_loop(items: list[str]):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="id-ID")
        pg = await ctx.new_page()
        # warm up
        await pg.goto("https://shopee.co.id", wait_until="networkidle")
        await asyncio.sleep(1)
        # scrape items in same session
        for url in items:
            await pg.goto(url, wait_until="networkidle")
            yield await pg.content()
            await asyncio.sleep(random.uniform(2, 6))

Second, exponential backoff on 429 and 403 responses. Aggressive retries amplify bans.

Third, monitor the PerimeterX challenge rate. If your scraper starts hitting the captcha at higher than 5 percent of requests, your IP pool is getting flagged. Rotate or pause.

Shopee variations and SKUs

Many Shopee products have variants (size, color, model). The API exposes them in item.tier_variations and item.models. Parsing variations:

def parse_variations(item: dict) -> list[dict]:
    tiers = item.get("tier_variations", [])
    models = item.get("models", [])
    out = []
    for m in models:
        out.append({
            "model_id": m.get("modelid"),
            "name": m.get("name"),
            "price_idr": m.get("price", 0) // 100_000,
            "stock": m.get("stock"),
            "sku": m.get("extinfo", {}).get("seller_sku"),
        })
    return out

For competitive intelligence, variant-level pricing is critical. Rolling up to a single product price misses the long tail.

Cost optimization for Shopee specifically

Three patterns that cut Shopee Indonesia scraping cost meaningfully:

Block image and font requests. Shopee product pages load 3 to 5 MB of images by default. Blocking them via Playwright route interception cuts proxy bandwidth by 70 percent.

Use the API path with cached cookies for items you scrape repeatedly. Browser-based scraping is for first-time visits only.

Batch requests by shop_id. Cookies and rate limits are session-scoped, so processing 10 items from the same shop in one session is cheaper than 10 cold scrapes.

Combined, these cut typical per-page cost from $0.082 to $0.018, a 4x improvement.

Storage schema

CREATE TABLE shopee_id_products (
    id BIGSERIAL PRIMARY KEY,
    item_id BIGINT NOT NULL,
    shop_id BIGINT NOT NULL,
    url TEXT NOT NULL,
    title TEXT NOT NULL,
    brand TEXT,
    price_idr NUMERIC(12,2) NOT NULL,
    original_price_idr NUMERIC(12,2),
    stock INTEGER,
    sold INTEGER,
    rating NUMERIC(3,2),
    review_count INTEGER,
    in_stock BOOLEAN NOT NULL,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    raw_jsonb JSONB,
    UNIQUE(item_id, shop_id)
);
CREATE INDEX idx_shopee_id_extracted_at ON shopee_id_products(extracted_at);
CREATE INDEX idx_shopee_id_shop ON shopee_id_products(shop_id);

For variant-level data, normalize into a shopee_id_variants table referenced by (item_id, shop_id).

Real benchmarks

A March 2026 production run, 10,000 Shopee Indonesia products with the API path plus warm session cookies:

Metric	Value
Success rate	93%
Median latency per item	1.4 s (API path)
Median latency per item (browser path)	7.2 s
Cost per 1000 items (API path)	$18
Cost per 1000 items (browser path)	$82
Captcha rate	4.1%
429 throttle rate	1.8%

The API path is roughly 5x faster and 4x cheaper. The maintenance cost of keeping the session warmup and cookie capture working is real but worth it for high-volume teams.

Failure mode breakdown

Of the 7 percent failures:

• 4.1% PerimeterX captcha
• 1.8% rate-limit throttle
• 0.6% network timeout
• 0.3% legitimate 404 (item removed)
• 0.2% malformed response

Each failure mode needs different mitigation. Captcha rate is the leading indicator of pool health; if it climbs above 8 percent, rotate proxies.

Cost expectations

10,000 Shopee Indonesia products per month with mobile proxies:

Component	Cost
Indonesian mobile proxy traffic (~3MB/page)	$120-$200
Browser compute	$50
LLM extraction (optional)	$30
Total	$200-$280

Shopee Indonesia is on the higher end of cost per page in ASEAN because of the bot defense and the price of Indonesian mobile IPs.

Legal considerations

Indonesia’s Personal Data Protection Law (UU PDP) became fully enforceable in October 2024. Public commercial data (product listings, prices, store names) is not personal data. Buyer reviews that include real names or photos may be personal data and require care.

Shopee’s terms of service prohibit automated access. The terms apply to civil contractual obligations; criminal exposure for scraping public commercial data is minimal in Indonesia. For a deeper compliance discussion, see scraping EU sites: jurisdictional realities which covers the broader frameworks; Indonesia-specific guidance comes from local counsel.

Region-specific Shopee features

Shopee Indonesia ships features that other Shopee markets do not, which affect what you can scrape:

ShopeeFood is integrated into the main Shopee Indonesia app for restaurant delivery. Listings live at shopee.co.id/food. They expose menu items as separate “products” in the same item endpoint, with a food flag.

ShopeePay is the payment layer; for normal product scraping you do not interact with it, but its pricing tiers (e.g. ShopeePay-only deals) appear in product promotion blocks.

Shopee Live is a livestream commerce surface. Live broadcasts list featured products via a separate /api/v4/live endpoint with rapidly-changing item lists. For live commerce intelligence, this is a separate scraper pipeline.

ShopeeMall is the verified-brand tier, identifiable via shopee_verified and shopee_official flags on items. Treat as separate from marketplace listings for brand intelligence.

Promotion and voucher data

Shopee promotions are complex and important for accurate price tracking. The relevant fields:

• price_min and price_max: range across variants
• price_before_discount: original price before any promotion
• discount: percent string (“50%” or “Rp50,000 OFF”)
• flash_sale: boolean indicating active flash sale
• vouchers: array of voucher codes applicable to the listing

Capture all of these. A 30 percent off promotion plus a 10 percent voucher plus free shipping is a 45 percent effective discount, and that effective price is what consumers actually pay.

Reviews and Q&A

Shopee Indonesia’s review system is verbose. Reviews include text, star rating, photos, videos, and seller responses. The endpoint:

async def fetch_reviews(item_id: str, shop_id: str, offset: int = 0) -> list[dict]:
    url = (f"https://shopee.co.id/api/v2/item/get_ratings"
           f"?itemid={item_id}&shopid={shop_id}&type=0&limit=20&offset={offset}")
    # uses same X-Csrftoken pattern as item/get
    ...

For sentiment analysis, capture the text plus rating. For authenticity verification, photos in reviews are a strong signal (genuine buyers post photos; fake reviews rarely do).

Frequently asked questions

Can I use Shopee’s Open Platform API?
Yes if you are a registered seller or partner. The Open Platform API is well-documented, rate-limited, and the safest path. For competitive intelligence (you are not a seller), scraping remains the practical option.

How often does Shopee change its anti-bot logic?
Major changes every 4-8 weeks. Small tweaks more often. Self-healing scrapers (see our self-healing scraper guide) reduce maintenance burden.

Will Indonesian residential IPs work?
For low-volume scraping, yes. For sustained high volume, mobile carrier IPs are the only reliable path.

How does Shopee handle the relationship between item_id and shop_id in URLs?
Both are required to uniquely identify a listing because the same item_id can exist under different shops (different sellers selling identical products). Always store both as the composite key.

Can I scrape Shopee data while using a Shopee buyer account?
Authenticated scraping is possible but raises the legal stakes (you are violating ToS while logged in). For most intelligence purposes, anonymous public-page scraping is sufficient and lower risk.

Can I scrape live stream sales?
Yes for the listed product side. Shopee Live broadcast metadata (viewer count, host info) requires a different endpoint that changes frequently.

How do I scale to scraping 1 million products per month from Shopee Indonesia?
Pool 50 to 100 mobile IPs, run 20 to 30 worker processes with API-path scraping, and budget roughly $4,000 per month at current proxy and compute prices. Keep cookie rotation aggressive and monitor PerimeterX challenge rates.

What about Shopee Pay payment data?
Not exposed publicly. Payment information is restricted to seller-side reports through the Open Platform API.

How do I track flash sale countdowns?
The flash sale info is in the flash_sale block with start and end timestamps. Poll items flagged for flash sale every minute during the sale window to capture the price-and-stock dynamics that brands care most about.

Does Shopee Indonesia have a different API version than Shopee Singapore?
The endpoints are nearly identical (both use /api/v4/) but content shapes differ slightly. Field names like flash_sale.special_promo_label vary in capitalization. Test against each market.

What is the right pacing per IP?
Roughly 1 request every 4 to 8 seconds per IP, with random jitter. Faster than 4 seconds gets throttled.

Can I scrape Shopee at the SKU level over time for trend analysis?
Yes. Snapshot daily for stable products, hourly for flash-sale items, and you build a price history that supports robust trend analysis. Storage cost grows linearly so plan for partition by month.

Common production gotchas

• The internal API occasionally returns a “shop closed” or “item removed” payload that looks like normal data but with all fields nulled. Check item.status == 1 for live items.
• Shopee uses CDN-side image URLs that expire. If you persist images, download them, do not store the URLs.
• The historical_sold count is approximate and rounds aggressively. For exact unit movement, you need seller-side data via the Open Platform API.
• Mobile carrier IPs from Indonesia have very different latency characteristics. Telkomsel is the most reliable; Smartfren is faster but blocks more often.
• Verbose JSONB storage adds up. Compress raw payloads if you store them long-term.

Live commerce data

Shopee Live is a major sales channel in Indonesia. Scraping live broadcast metadata requires a separate endpoint.

async def fetch_live_streams() -> list[dict]:
    url = "https://shopee.co.id/api/v4/live/get_session_list?limit=20&offset=0"
    # same X-Csrftoken pattern as item endpoints
    ...

The data includes streamer name, viewer count, products featured, and current deals. Useful for influencer marketing intelligence and live commerce trend tracking.

Indonesia-specific compliance notes

Beyond UU PDP, Indonesian ecommerce scraping should consider Bank Indonesia regulations on payment data (which you do not access via scraping anyway) and Kominfo regulations on data residency for personal data of Indonesian users (relevant only if you store reviews containing personal data).

For brand intelligence projects, the relevant data (prices, listings, seller IDs at the public level) is squarely in the safe zone. For sentiment analysis using buyer reviews, anonymize before storing.

For more ASEAN scraping coverage, browse the ecommerce category.

RAG over scraped data: production patterns 2026

RAG scraped data is the most common production AI pattern of 2026: scrape a domain, embed it, store in a vector database, retrieve at query time, augment LLM generation. The architecture sounds simple. The production reality is full of choices that determine whether the system is useful, trustworthy, and economical or whether it becomes a recurring engineering tax. This guide walks through the production patterns that work in 2026, the chunking and embedding choices that matter, the retrieval techniques that have matured beyond cosine similarity, the freshness and provenance disciplines that distinguish serious systems from prototypes, and a reliability checklist your team can apply.

The audience is the data engineer or applied AI lead building a RAG product on top of scraped or aggregated content.

What changed in 2024-2026 RAG practice

Three shifts.

First, hybrid retrieval (combining dense embeddings with sparse keyword search and reranking) replaced naive cosine-only retrieval as the default. Pure-cosine systems consistently underperform hybrid by 15-30 percent on retrieval recall in 2026 benchmarks.

Second, evaluation moved from “does the model answer plausibly” to “does the system retrieve and ground correctly”. RAGAS, ARES, and custom domain evals became the discipline. Production teams in 2026 ship eval suites with the system, not after.

Third, freshness and provenance became hard requirements. The 2024 era of “scrape once, embed, forget” produced systems that confidently cited stale or inaccurate data. The 2026 systems track when each chunk was scraped, what its source URL was, what the source’s freshness signal was, and they re-rank or down-weight stale content.

For the related vector database discussion, see vector databases for scraping pipelines. For the related MCP integration, see MCP for data engineers.

The production RAG pipeline in 2026

A working pipeline has seven stages:

Stage	Purpose	Common tooling 2026
Ingestion	Scrape and normalise source content	Scrapy, Stagehand, Firecrawl
Chunking	Split into retrievable units	LangChain, LlamaIndex, custom
Embedding	Vectorise chunks	OpenAI, Cohere, Voyage AI, BGE
Indexing	Store in vector + sparse index	Qdrant, Weaviate, Pinecone, pgvector
Retrieval	Hybrid dense + sparse lookup	Custom or framework
Reranking	Reorder for relevance	Cohere Rerank, Jina, custom
Generation	LLM call with retrieved context	OpenAI, Anthropic, local

Each stage has its own choices. Get any one badly wrong and the whole system suffers.

Ingestion: getting clean signal in

Scraped HTML is dirty. Production RAG ingestion is mostly cleanup. The patterns that work in 2026:

1. Parse with a structured extractor (Trafilatura, Newspaper3k, or LLM-based extraction with Stagehand) rather than crude HTML stripping.
2. Strip boilerplate (navigation, footers, ads, cookie banners) aggressively. These pollute embeddings.
3. Preserve structural metadata (heading hierarchy, paragraph boundaries, list structure). Chunk-aware structure improves retrieval.
4. Normalise text (Unicode, whitespace, punctuation). Embedding models are sensitive to noise.
5. Capture metadata at ingestion: source URL, scrape timestamp, content publish date if extractable, content language, content hash for change detection.

import trafilatura
from datetime import datetime

def ingest(url: str, html: str) -> dict:
    extracted = trafilatura.extract(
        html, include_comments=False, include_tables=True,
        favor_recall=False, output_format="json",
        with_metadata=True,
    )
    if not extracted:
        return None
    doc = json.loads(extracted)
    return {
        "source_url": url,
        "scraped_at": datetime.utcnow().isoformat(),
        "published_at": doc.get("date"),
        "title": doc.get("title"),
        "text": doc.get("text"),
        "content_hash": hashlib.sha256(doc["text"].encode()).hexdigest(),
        "language": doc.get("language"),
    }

For the broader scraping technique discussion, see building scraping pipelines with Prefect 3.

Chunking: more art than science

Chunking is where most teams get it wrong. The naive approach (fixed-size 500-token chunks with 50-token overlap) works adequately for many use cases but is rarely optimal. The 2026 patterns:

Strategy	When to use	Tradeoffs
Fixed-size with overlap	Default starting point	Simple; can split mid-thought
Sentence-aware	Most natural text	Better boundaries; slower
Heading-aware (markdown/structured)	Technical docs, articles	Best for structured content
Semantic (embedding-similarity-based)	Conceptual content	More expensive; better cohesion
Hierarchical (parent-child)	Long documents needing context	Most complex; best for QA
Late chunking (chunk after embed)	Very long contexts (1M+ tokens)	New 2025 approach; promising

A heading-aware chunker for markdown-extracted content:

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

headers_to_split = [("#", "h1"), ("##", "h2"), ("###", "h3")]
md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split)
char_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800, chunk_overlap=80,
    separators=["\n\n", "\n", ". ", " "],
)

def chunk_doc(text: str, metadata: dict) -> list:
    sections = md_splitter.split_text(text)
    chunks = []
    for section in sections:
        sub_chunks = char_splitter.split_text(section.page_content)
        for sub in sub_chunks:
            chunks.append({
                "text": sub,
                "metadata": {**metadata, **section.metadata},
            })
    return chunks

The result: chunks that preserve heading context, do not split mid-sentence, and stay within an embedding-friendly token range.

Embeddings: the model choice matters

The embedding model choice has a large effect on retrieval quality. The 2026 leaderboard is dominated by:

Model	Provider	Dimensions	Notes
text-embedding-3-large	OpenAI	3072	Strong general performance; pricing favourable
voyage-3-large	Voyage AI	1024	Top performance on technical content
Cohere Embed v4	Cohere	1024	Strong multilingual; good reranking pair
BGE-M3	BAAI (open)	1024	Best open-source option; multilingual
Nomic Embed v2	Nomic	768	Open; good cost-performance
jina-embeddings-v3	Jina	1024	Strong long-context

The right choice depends on language coverage, domain (technical vs general), and cost sensitivity. For most English-only general-purpose RAG in 2026, text-embedding-3-large or voyage-3-large produce the best results.

A practical recommendation: do not optimise embedding choice in isolation. Run your eval suite (next section) with two or three candidate models. The right model for your domain is rarely the same as the leaderboard winner.

For the deeper vector database choice question, see vector databases for scraping pipelines.

Hybrid retrieval: dense plus sparse plus rerank

Pure cosine retrieval misses keyword matches that humans expect. Pure BM25 misses semantic matches. The hybrid pattern (run both, combine, rerank) is the production default in 2026.

from qdrant_client import QdrantClient
import cohere

qdrant = QdrantClient(...)
co = cohere.Client(...)

def retrieve(query: str, top_k: int = 20, final_k: int = 5):
    dense_hits = qdrant.search(
        collection_name="docs",
        query_vector=embed(query),
        limit=top_k,
    )
    sparse_hits = bm25_search(query, top_k)
    merged = reciprocal_rank_fusion([dense_hits, sparse_hits])

    rerank_input = [hit.payload["text"] for hit in merged[:top_k]]
    reranked = co.rerank(
        model="rerank-3.5",
        query=query,
        documents=rerank_input,
        top_n=final_k,
    )
    return [merged[r.index] for r in reranked.results]

Reciprocal rank fusion is a simple, effective merger. The reranker (Cohere Rerank, Jina Rerank, or custom cross-encoder) provides the final relevance ordering using a model that can compare query and document directly.

Freshness and provenance

A scraped corpus goes stale. The 2026 production discipline:

1. Track scrape timestamp on every chunk.
2. Track source publish date where extractable.
3. Build a freshness scorer that down-weights old chunks for time-sensitive queries.
4. Re-scrape on a scheduled cadence per source.
5. Detect content changes (content hash comparison) and re-embed only changed content.
6. Surface provenance in answers: cite source URLs, show scrape date, allow user verification.

A typical freshness-aware retrieval modifies the score:

import math
from datetime import datetime

def freshness_weight(scraped_at: str, half_life_days: int = 90) -> float:
    age = (datetime.utcnow() - datetime.fromisoformat(scraped_at)).days
    return math.exp(-age / half_life_days * math.log(2))

def score_with_freshness(hit, query_is_time_sensitive: bool):
    base = hit.score
    if not query_is_time_sensitive:
        return base
    weight = freshness_weight(hit.payload["scraped_at"])
    return base * (0.6 + 0.4 * weight)  # Floor at 60% to preserve relevance

For the broader scraping freshness discipline, see building scraping pipelines with Prefect 3.

Evaluation: the discipline that separates production from prototype

A 2026 RAG system without an eval suite is a prototype, not a product. The minimum eval discipline:

Eval type	Measures	Tooling
Retrieval recall	Did we retrieve relevant docs?	Per-query labelled set
Retrieval precision	Were retrieved docs relevant?	Per-query labelled set
Faithfulness	Did the answer use only retrieved info?	RAGAS, ARES
Answer relevance	Did the answer address the query?	RAGAS, custom rubric
Groundedness	Are answer claims supported?	Custom claim-checker
Latency	Is the system fast enough?	Production tracing
Cost	What does each query cost?	Production tracing

A working eval set has 100-500 query-and-expected-answer pairs from your domain, refreshed quarterly. Run it on every meaningful change.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset

def run_eval(test_set):
    results = []
    for q in test_set:
        retrieved = retrieve(q["question"])
        answer = generate(q["question"], retrieved)
        results.append({
            "question": q["question"],
            "answer": answer,
            "contexts": [r.payload["text"] for r in retrieved],
            "ground_truth": q["expected_answer"],
        })
    ds = Dataset.from_list(results)
    return evaluate(ds, metrics=[faithfulness, answer_relevancy, context_precision])

Decision tree: production RAG checklist

Q1: Is the corpus stable, slowly changing, or fast-moving?
    ├── Stable -> Standard ingestion; quarterly re-scrape.
    ├── Slow -> Monthly re-scrape; freshness scoring optional.
    └── Fast -> Daily or hourly re-scrape; freshness scoring mandatory.
Q2: Is the use case general QA or domain-specific?
    ├── General -> OpenAI/Voyage default embeddings.
    └── Specific -> Test domain-specific embeddings or fine-tune.
Q3: Is multilingual coverage required?
    ├── Yes -> BGE-M3 or Cohere Embed v4.
    └── No  -> English-optimised models.
Q4: What is the typical document length?
    ├── Short (article) -> Standard chunking.
    ├── Long (manual, book) -> Hierarchical chunking with parent-child.
    └── Very long (1M+ token doc) -> Late chunking experiments.
Q5: What is the answer-faithfulness requirement?
    ├── Strict (legal, medical) -> Reranker + groundedness check + citations.
    └── Tolerant (casual) -> Standard pipeline.

Comparison: RAG architecture choices

Architecture	Pros	Cons	Best for
Naive (single embed + cosine)	Simple	Underperforms	Prototype only
Hybrid (dense + sparse)	Better recall	Two indexes	Most production
Hybrid + rerank	Best quality	Higher latency	Quality-sensitive
Hierarchical (parent-child)	Long doc context	Complex	Long-doc QA
Multi-query expansion	Better recall on terse queries	Higher LLM cost	User-facing search
Agent-driven (multi-step retrieval)	Can resolve ambiguity	Highest cost and latency	Open-domain assistants

External references

The RAGAS evaluation framework is at github.com/explodinggradients/ragas. The Trafilatura content extractor is at trafilatura.readthedocs.io. The Cohere Rerank documentation is at docs.cohere.com/docs/rerank. The MTEB leaderboard for embedding model comparison is at huggingface.co/spaces/mteb/leaderboard.

Operational patterns: cost, latency, observability

A 2026 production RAG system tracks cost per query (embedding, vector lookup, rerank, generation), latency per stage, and quality scores per query (sampled). Without these, capacity planning and quality regression detection are impossible.

A typical cost breakdown for a single query in mid-2026:

Stage	Cost (USD)	Latency (ms)
Query embedding	0.0001	50
Vector lookup	0.0001	30
Sparse lookup	0.00005	20
Rerank	0.001	200
Generation (Sonnet)	0.005	2000
Total	~0.006	~2300

These numbers move with model pricing changes. The pattern is stable: generation dominates cost; rerank dominates retrieval latency.

Compliance and provenance for AI training adjacent use

A RAG system that retrieves from a corpus is not training a model on that corpus. But the line is sometimes contested. Three practices that keep RAG defensibly distinct from training:

1. Retrieve at query time, not ahead. Do not pre-compute or memorise.
2. Cite sources in answers. Make the retrieval visible.
3. Honour content opt-outs. If a source has revoked permission, remove from the corpus on the next refresh.

For the broader fair-use discussion, see fair use and copyright for AI training data.

FAQ

What is the simplest production RAG architecture?
Hybrid dense plus sparse retrieval, with a reranker, and an LLM with retrieved context. Three components: index, retriever-reranker, generator.

Which vector database should I pick?
Qdrant for general use, Weaviate for managed convenience, pgvector if you already run Postgres. See the vector database guide for full comparison.

How often should I re-scrape?
Daily for fast-moving content (news, prices), weekly for moderate (documentation), monthly for slow (reference). Faster than your users notice staleness.

What is the most common production failure?
Stale or wrong content surfaced confidently. Mitigate with freshness scoring, citations, and a faithfulness eval.

Is RAG dead now that LLMs have long context?
No. Long context complements RAG; it does not replace it. RAG handles corpora that exceed any context window.

Extended production RAG analysis

The 2024-2026 evolution of production RAG converged on a six-stage pipeline. Each stage has measurable inputs, outputs, and quality signals. The stages are ingestion, normalisation, chunking, embedding, retrieval, and synthesis.

A production-grade RAG system in 2026 typically achieves the following on a real workload.

• Faithfulness above 0.85 on Ragas-style evals.
• Answer relevance above 0.80.
• Context precision above 0.70.
• p95 latency below 2 seconds.
• Cost per answered question below USD 0.01.

Achieving those numbers requires hybrid retrieval (dense plus sparse plus rerank), a freshness signal, and an eval suite that runs on every change.

Hybrid retrieval pattern with reranking

from sentence_transformers import CrossEncoder
from rank_bm25 import BM25Okapi

class HybridRetriever:
    def __init__(self, vector_store, corpus):
        self.vector_store = vector_store
        self.bm25 = BM25Okapi([doc.split() for doc in corpus])
        self.corpus = corpus
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

    def retrieve(self, query, k=10):
        dense = self.vector_store.similarity_search(query, k=k*2)
        sparse_scores = self.bm25.get_scores(query.split())
        sparse_idx = sorted(range(len(sparse_scores)), key=lambda i: -sparse_scores[i])[:k*2]
        sparse = [self.corpus[i] for i in sparse_idx]
        candidates = list({d.page_content: d for d in dense + sparse}.values())
        pairs = [(query, c.page_content) for c in candidates]
        scores = self.reranker.predict(pairs)
        ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
        return [c for c, _ in ranked[:k]]

Chunking strategies that work in production

Chunking is the most underrated decision in RAG. The 2026 patterns that ship are.

1. Semantic chunking with a 512-1024 token target window and 10-20 percent overlap.
2. Document-structure-aware chunking that respects headings, tables, and code blocks.
3. Late chunking (Jina v3 style) where embeddings are computed on the full document and pooled per chunk.
4. Per-document chunking strategy that varies by content type (code, prose, tables).

def semantic_chunk(text, target_tokens=768, overlap_tokens=96):
    sentences = split_sentences(text)
    chunks = []
    current = []
    current_tokens = 0
    for sent in sentences:
        sent_tokens = count_tokens(sent)
        if current_tokens + sent_tokens > target_tokens and current:
            chunks.append(" ".join(current))
            overlap_count = 0
            overlap_chunk = []
            for s in reversed(current):
                if overlap_count + count_tokens(s) > overlap_tokens:
                    break
                overlap_chunk.insert(0, s)
                overlap_count += count_tokens(s)
            current = overlap_chunk
            current_tokens = overlap_count
        current.append(sent)
        current_tokens += sent_tokens
    if current:
        chunks.append(" ".join(current))
    return chunks

Evaluation harness pattern

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

def run_eval(qa_dataset, rag_chain):
    samples = []
    for item in qa_dataset:
        result = rag_chain.invoke(item["question"])
        samples.append({
            "question": item["question"],
            "answer": result["answer"],
            "contexts": [c.page_content for c in result["contexts"]],
            "ground_truth": item["ground_truth"],
        })
    scores = evaluate(samples, [faithfulness, answer_relevancy, context_precision])
    return scores

Comparison: RAG architectures by use case

Use case	Architecture	Cost per query	Latency p95
Customer support	Hybrid retrieval plus rerank	USD 0.005-0.01	1-2 sec
Research synthesis	Multi-step agentic RAG	USD 0.05-0.20	10-30 sec
Code search	Dense retrieval on code-trained embeddings	USD 0.002-0.005	0.5-1 sec
Compliance Q and A	Hybrid plus citation enforcement	USD 0.01-0.02	2-3 sec
Real-time data Q and A	Streaming retrieval plus freshness boost	USD 0.01-0.03	1-3 sec

Observability for RAG pipelines

Production RAG should emit five signals per query.

1. Retrieval count and per-stage latency.
2. Reranker score distribution.
3. Context relevance score (LLM judge or Ragas).
4. Answer faithfulness score.
5. User feedback (thumbs up or down) where available.

Additional FAQ

How do I handle freshness?
Tag every document with a timestamp at ingest. At query time boost recent documents and decay older ones. Define decay per use case.

How do I prevent hallucination?
Faithfulness eval as a release gate. Citation requirement in the prompt. Refusal when retrieval returns low-confidence results.

What about multi-modal RAG?
2026 stable patterns include image embeddings (CLIP, SigLIP) and document AI for tables. Treat each modality with its own pipeline and retrieval index.

When does fine-tuning beat RAG?
For stable, narrow domains where the corpus rarely changes and latency matters. RAG wins for evolving corpora and citability.

The RAG quality plateau and how to break through

Most RAG systems plateau at a similar level of quality. The plateau symptoms are answers that are mostly right but occasionally wrong, citations that are mostly accurate but occasionally hallucinated, and latency that is acceptable but not impressive. Breaking through the plateau requires investment in five specific areas.

The first is reranking. Most plateaued RAG systems retrieve top-k documents by vector similarity and pass them directly to the model. Adding a cross-encoder reranker between retrieval and generation typically lifts answer quality by 10-20 percent in evaluations. The reranker is a small model that scores query-document pairs more accurately than vector similarity.

The second is query rewriting. The user’s question is often imprecise or contextually loaded. A query rewriter expands the question into a search-optimised form. A 2026 pattern is to generate three to five rewrites, retrieve for each, and merge results.

The third is context construction. Most plateaued systems concatenate retrieved chunks into a flat context. A higher-quality system structures the context with provenance markers, dedupes overlapping chunks, and orders by relevance.

The fourth is evaluation. A system without an eval harness improves randomly. A system with a fixed eval set improves systematically. The eval harness should run on every change, with thresholds that block regressions.

The fifth is observability. A system that logs every query, every retrieval, every rerank, and every output enables retrospective analysis. The retrospective is where the next round of improvements is found.

The freshness problem in production RAG

Many real corpora are not static. News articles are published continuously. Product pages change. Documentation evolves. A RAG system that ignores freshness gives confidently outdated answers.

The 2026 pattern for freshness handling has three layers. The first is timestamping every document at ingest. Every chunk has a created_at and updated_at field. The retrieval layer can filter or boost by recency.

The second is freshness-aware retrieval. The retriever applies a recency boost to scores, with the magnitude tunable per query type. A query about current events gets a strong recency boost. A query about historical context gets little or no boost. The classifier that decides the boost can be a small model or a heuristic.

The third is freshness-aware generation. The model is told the publication dates of the retrieved documents and is instructed to flag potentially outdated information. The model can also be instructed to refuse confident answers when all retrieved documents are stale.

The freshness machinery adds cost (the timestamp filter) and complexity (the recency boost and the model prompting). The benefit is fewer confidently wrong answers. For evolving corpora the trade is favourable.

Citations and grounding

A RAG system that does not cite sources is hard to trust. Users cannot verify claims, debug errors, or trace provenance. The 2026 best practice is to require citations in every generated answer, with a 1-to-1 mapping from claim to source.

Citation enforcement is implemented in three layers. The prompt instructs the model to cite. The output parser validates that every claim has a citation. The eval harness measures citation accuracy on a held-out set.

Citation accuracy is a separable metric from answer accuracy. An answer can be correct but cite the wrong source, or cite a source that does not actually support the claim. A high-quality RAG system measures both.

The 2026 advanced patterns include grounded generation (where the model is constrained to only state claims that the retrieved context supports) and citation post-processing (where a separate model checks each claim against its cited source). Both improve citation accuracy at the cost of additional latency and compute.

Beyond RAG: agentic retrieval and self-querying

The classical RAG pattern is single-shot retrieval followed by single-shot generation. The 2026 evolution is multi-step agentic retrieval, where a planner decides what to retrieve, observes the result, and decides whether to retrieve more.

Agentic retrieval handles questions that require multiple sources, where the relevance of subsequent sources depends on the content of earlier sources. Examples include comparison questions (find documents about X, then find documents about Y, then compare), multi-hop questions (find what John works on, then find what John’s team works on), and exploratory research (find papers on topic, then drill into the most cited paper).

The cost of agentic retrieval is higher latency and cost, since each retrieval step is a separate model call. The benefit is the ability to answer questions that single-shot RAG cannot. The 2026 pattern is to route queries between single-shot and agentic based on a complexity classifier, balancing cost and capability.

Operational maturity stages for RAG teams

Production RAG systems progress through four maturity stages. Stage zero is a prototype that works on a small fixed corpus. Stage one is a pilot that handles real users with manual evaluation. Stage two is a production deployment with an automated eval suite and freshness handling. Stage three is a mature deployment with hybrid retrieval, reranking, citation enforcement, agentic retrieval for complex queries, and continuous improvement workflows.

Most teams plateau at stage one. The transition from stage one to stage two is the highest-leverage investment, because the eval suite is what enables systematic improvement. The transition from stage two to stage three is incremental, with each component adding measurable quality.

The cost profile changes with maturity. Stage zero is essentially free. Stage one costs a few thousand dollars per month for moderate traffic. Stage two costs ten to twenty thousand dollars per month for moderate traffic. Stage three costs more, but the cost-per-quality-unit decreases because the additional spend buys disproportionate quality.

A 2026 best practice for teams entering stage two is to establish a quality target before launching. The target might be faithfulness above 0.85, citation accuracy above 0.90, and p95 latency below 2 seconds. The team commits to the target publicly and reports progress. The discipline accelerates the maturity transition.

Document parsing as a quality bottleneck

The quality ceiling of a RAG system is bounded by the quality of its document parsing. A pipeline that ingests garbage HTML produces garbage chunks regardless of how good the embedding model is. The 2026 best practice is to invest in document parsing as a first-class concern.

For HTML the 2026 toolkit includes Trafilatura for content extraction, Readability variants for boilerplate removal, and dedicated parsers for tables and code blocks. For PDFs the toolkit includes PyMuPDF, pdfplumber, and the various LayoutLM-derived models for structured extraction. For office documents the unstructured library aggregates many formats.

The cost of high-quality parsing is meaningful at scale. A 2026 pipeline parsing 10 million pages might spend 500-2000 dollars per million on parsing alone. The investment is worthwhile because parsing quality compounds through the rest of the pipeline.

A 2026 trend is multimodal parsing using vision-language models. A model that can read a page screenshot and extract structured content handles edge cases that text-only parsers miss. The cost is higher per page, but the quality gain on hard pages can be 10-30 percent in retrieval evaluations.

Next steps

The fastest improvement to most existing RAG systems is to add a reranker and an eval suite. Both are an afternoon of work and improve quality measurably. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the vector databases guide.

This guide is informational, not engineering or legal advice.

Verifiable credentials and scraping in 2026

Verifiable credentials scraping is the access pattern that did not exist commercially three years ago and is starting to appear at production scale in 2026. Verifiable Credentials (VCs), defined by the W3C, are signed digital attestations that travel with the user and that a verifier can validate without contacting the issuer. The standardisation finished in 2022; the production deployment ramped in 2024-2025; the pivotal moment was the EU Digital Identity Wallet rollout across 2025-2026. For scraping operators, VCs reshape both the access landscape (sources will increasingly require credential presentation) and the operations landscape (some scrapers will themselves issue or hold credentials). This guide walks through what VCs actually are, the OID4VC presentation flow, the scraping-relevant credential ecosystems, the access patterns that work, and a practical operator playbook.

The audience is the technical lead, security architect, or compliance partner who needs to plan for a world where verifiable credentials gate the data they need.

What a verifiable credential actually is

A Verifiable Credential is a tamper-evident signed assertion. The data model is JSON-LD or JWT-encoded and contains three things:

1. The issuer (a DID or an X.509 certificate identifying the authority).
2. The subject (the entity the credential describes, typically a DID).
3. The claims (the assertions being made, e.g., “this subject is over 18”, “this subject is a licensed driver”).

The credential is signed by the issuer’s key. Any verifier can validate the signature using the issuer’s public key. The verifier does not need to contact the issuer at validation time, which is the protocol’s central innovation. This makes VCs offline-verifiable, fast, and privacy-preserving.

A holder (typically the user via their wallet) presents the credential by signing a Verifiable Presentation, which can include the entire credential or only selected claims. Selective disclosure is supported via two main mechanisms: BBS+ signatures (mature, computationally heavy) and SD-JWT (lighter, widely deployed in 2026).

For the broader Web4 context, see decentralized identity and Web4 scrapers.

The OID4VC presentation flow

OpenID for Verifiable Credentials (OID4VC) is the protocol family that puts VCs into production HTTP flows. There are two main specs: OID4VCI (issuance) and OID4VP (presentation).

The OID4VP flow that a scraping operator interacts with:

1. The verifier (the site you want to access) presents an authorisation request, typically as a QR code or a deep link, asking for specific credentials.
2. The user’s wallet retrieves the request, identifies matching credentials, and asks the user to authorise presentation.
3. The wallet builds a Verifiable Presentation containing the requested claims (with selective disclosure if applicable).
4. The wallet POSTs the presentation to the verifier’s endpoint.
5. The verifier validates the signature, checks the issuer, checks revocation, and grants access.

The flow is HTTP-native and integrates cleanly with existing OAuth 2.0/OIDC infrastructure. From the verifier’s perspective, it is a sign-in flow; from the wallet’s perspective, it is a credential presentation; from the user’s perspective, it is one approval click.

For scraping operators, the flow has implications. A scraper that wants to access an OID4VP-gated source needs to fit into this flow: either be the wallet (programmatic credential presentation) or partner with a wallet holder (delegated access).

The 2026 credential ecosystems

Ecosystem	Issuer authority	Common credentials	Status
EU Digital Identity Wallet	Member states	National ID, driver’s licence, professional qualifications, age	General availability
UK Digital Identity	UK Government Digital Service	National ID-equivalent, age, residency	Production
Singpass (Singapore)	Singapore Government	National ID, qualifications, residence	Production with VC issuance
DigiLocker (India)	Indian Government	National ID, education, employment	VC alongside document store
Apple Wallet	Apple, partner issuers	State IDs, transit, payment, loyalty	US state IDs in selected states
Google Wallet	Google, partner issuers	State IDs, transit, payment	Equivalent to Apple in coverage
OpenAttestation (Singapore)	Open standard, multi-issuer	Trade documents, qualifications	Mature for trade and education
Hyperledger AnonCreds	Open standard	Various sectoral	Self-sovereign identity

For scraping, the EU and Singapore ecosystems are the most relevant in 2026. Both have substantive VC issuance with mature verification infrastructure.

Compliance and trust framework

The trust in a VC-gated system rests on three layers:

Layer	Question	Mechanism
Issuer trust	Is the issuer who they claim to be?	Trust list (e.g., EU LOTL), DID resolution
Cryptographic integrity	Is the signature valid?	Standard cryptography
Revocation	Has the credential been revoked?	Status list 2021, OCSP-like protocols

A verifier checks all three. A scraper presenting a credential needs all three to pass. For commercial scrapers without legitimate credential issuance, the trust framework is a hard barrier; falsifying a credential requires breaking cryptography or compromising an issuer key.

The legitimate access paths for scrapers:

1. Hold credentials in your own right (research institutions, regulated industries).
2. Partner with credential holders (delegated access via signed authorisation).
3. Become a credential issuer (issue credentials about your data, not about your identity).
4. Use the public, non-credentialed surface of the source.

Decision tree: how to scrape a VC-gated source

Q1: Does the source require VC presentation?
    ├── No  -> Standard access; no VC work needed.
    └── Yes -> Q2
Q2: Which credential is required?
    ├── National ID / age -> Likely user-specific; partnership path only.
    ├── Professional qualification -> Operator may hold legitimately.
    ├── Subscription / payment -> Operator can typically obtain.
    └── Other -> Evaluate per credential type.
Q3: Can your operation hold the credential legitimately?
    ├── Yes -> Implement OID4VP client; present credential.
    └── No  -> Q4
Q4: Is partnership with a credential holder feasible?
    ├── Yes -> Negotiate; implement delegated access.
    └── No  -> Source is effectively unscrapable; explore licensed access.

Worked example: scraping an OpenAttestation-protected supply-chain document portal

A 2026 supply-chain portal hosts customs documents and bills of lading, accessible to authorised supply-chain participants. Access requires an OpenAttestation credential proving the requester is a licensed customs broker.

Web2 access path: register an account, submit business verification, wait for human approval. Often months.

VC access path: a customs broker holds a credential issued by Singapore’s Customs authority. The broker authorises the scraping operation to use the credential on their behalf. The scraping operation runs an OID4VP client that presents the credential at the portal’s verifier endpoint. Access is granted.

For the scraping operation, the credential acquisition and partnership path is real engineering work but bounded. Once in place, ongoing access is automated.

For the broader supply-chain scraping discussion, see scraping crypto exchange order books.

Selective disclosure and minimisation

A Verifiable Presentation can include only the claims the verifier needs. A credential carrying multiple claims (name, date of birth, nationality, address) can be presented with only “over 18” or “EU resident” disclosed.

This matters for scrapers in two ways:

1. As a verifier (if your operation is the verifier of credentials presented by other parties), request only the claims you need. Excess collection is a GDPR/CCPA violation.
2. As a holder (if your operation presents credentials), use selective disclosure to limit what the source learns about you.

The selective disclosure mechanisms in 2026 are SD-JWT (most common, simpler) and BBS+ (more cryptographically powerful, less common). Both are well-documented and implemented in major wallet SDKs.

Comparison: VC presentation vs traditional authentication

Dimension	Username/password	OAuth 2.0 / OIDC	OID4VP / VC presentation
Account at site required	Yes	Conditional	No
Site learns user identity	Yes	Yes	Only what credential discloses
Credential portable across sites	No	Limited	Yes
Selective disclosure	No	No	Yes (SD-JWT, BBS+)
Offline verification	No	No	Yes
Phishing resistance	Low	Moderate	High (cryptographic)
Adoption (mid-2026)	Mature	Mature	Early production

The VC pattern is structurally superior on privacy and portability. Adoption is the constraint, not capability.

Operations: becoming a credential issuer

A 2026 trend that scraping operators should not miss: some operators are themselves becoming credential issuers. Instead of issuing credentials about identity, they issue credentials about data. Examples:

Issuer type	Credentials issued	Use case
Data marketplace	Provenance VCs (this dataset was scraped from X on Y)	Buyer trust
Aggregator	Quality VCs (this record passes 99 percent quality checks)	Downstream confidence
Compliance auditor	Compliance VCs (this dataset was processed under GDPR)	Regulator-facing assurance
Research operator	Replication VCs (this experiment used this dataset)	Academic integrity

Becoming an issuer requires: a DID, an issuance service, a revocation registry, and (for trust) inclusion on relevant trust lists. The setup cost is real (engineering plus legal) but the resulting product differentiation is meaningful.

For the parallel discussion of how this fits into RAG-over-scraped-data products, see RAG over scraped data.

Implementation: an OID4VP verifier in Python

A minimal OID4VP verifier endpoint in Python (FastAPI) for accepting a presented Verifiable Presentation:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from didkit import verify_presentation
import json

app = FastAPI()

class Presentation(BaseModel):
    vp_token: str
    presentation_submission: dict

@app.post("/verify")
async def verify(p: Presentation):
    options = json.dumps({"proofPurpose": "authentication"})
    result_str = await verify_presentation(p.vp_token, options)
    result = json.loads(result_str)
    if result.get("errors"):
        raise HTTPException(400, detail=result["errors"])

    vp = json.loads(p.vp_token) if p.vp_token.startswith("{") else None
    holder = vp.get("holder") if vp else None
    return {"verified": True, "holder": holder}

The verifier validates signatures, issuer trust, and revocation status. Production deployment adds replay protection (nonce checking) and audience binding (verifier-specific challenges).

External references

The W3C Verifiable Credentials data model is at w3.org/TR/vc-data-model-2.0. The OpenID for Verifiable Presentations specification is at openid.net/specs/openid-4-verifiable-presentations-1_0.html. The EU Digital Identity Wallet architecture and reference framework is at ec.europa.eu/digital-building-blocks/sites/display/EUDIGITALIDENTITYWALLET.

Comparison: SD-JWT vs BBS+ for selective disclosure

Dimension	SD-JWT	BBS+
Computational cost	Low	High
Predicate proofs (over 18 without DOB)	No (requires explicit claim)	Yes
Wallet support (mid-2026)	Wide	Limited
Verifier support (mid-2026)	Wide	Limited
Revocation handling	Standard	Standard
Production maturity	High	Moderate

SD-JWT is the workhorse for production OID4VC in 2026. BBS+ is the path forward for use cases that need true zero-knowledge predicate proofs.

Where this is heading

Three trajectories.

First, more sectors adopt VC-gated access. Healthcare, finance, government, professional services, and supply chain are leading. Consumer sites are slower but trending.

Second, agent wallets become standard. Agentic browsers (Stagehand, browser-use, Operator) are adding wallet capabilities to programmatically present credentials. The boundary between user wallets and agent wallets blurs.

Third, the credential ecosystem fragments before it consolidates. The 2026 landscape has dozens of credential formats and trust frameworks. The next two years bring consolidation as larger ecosystems (EUDI Wallet, Singpass, US Wallets) absorb smaller ones.

For the broader trajectory of access patterns, see the agentic browser revolution.

Compliance overlay for VC-using scrapers

A scraper that uses VCs has compliance implications that traditional scraping does not. Four controls:

1. Credential governance: written policy on which credentials the operation holds, who legitimately holds them, what use is in scope.
2. Audit logging: every credential presentation logged with timestamp, target, claim selection, and outcome.
3. Revocation handling: when a credential is revoked, the operation must stop using it within a defined window.
4. Misuse safeguards: technical controls preventing inappropriate use of held credentials.

For the broader policy build, see building an ethics-first scraping policy.

FAQ

Is VC-gated access widespread in 2026?
It is in early production for several major sectors (EU public services, Singapore, regulated industries) but not yet ubiquitous. Adoption is accelerating.

Can a scraper hold a verifiable credential?
Yes if the credential is appropriate to the operation (research credential for a research scraper, broker credential for a broker-affiliated scraper). Falsifying a credential is fraud, full stop.

What is the difference between OAuth and OID4VP?
OAuth is account-based and platform-mediated. OID4VP is credential-based and portable across sites without per-site accounts.

What if I just want to bypass the VC requirement?
Bypass is generally infeasible (cryptographic) and almost certainly fraudulent (misrepresentation). The legitimate paths are partnership, delegated access, or licensed access.

Are VCs only for humans?
No. VCs can be issued to organisations, software agents, IoT devices, or any entity with a DID. Agent-targeted credentials are a growing category in 2026.

Extended verifiable credentials analysis

Verifiable credentials (VCs) per the W3C VC Data Model 2.0 specification became a production technology between 2023 and 2026. The eIDAS 2.0 European Digital Identity Wallet, the US California mDL programme, and several Singapore Singpass pilots all shipped VC-based credentials by mid-2026.

For scrapers VCs change three things. First, gated content moves from cookie auth to credential presentation. Second, content provenance can be verified cryptographically rather than through platform attestation. Third, scrapers themselves can present credentials to identify their operator and purpose.

The 2026 stable VC stack consists of four layers.

1. Issuance protocol (OpenID for Verifiable Credential Issuance, OID4VCI).
2. Presentation protocol (OpenID for Verifiable Presentations, OID4VP).
3. Credential format (JWT-VC, SD-JWT-VC, mDoc).
4. Status mechanism (Status List 2021, Bitstring Status List).

Implementation pattern: VC verification at fetch

from vc_lib import verify_presentation, check_status

async def verify_vc_presentation(presentation_jwt, expected_issuer, expected_claims):
    result = verify_presentation(presentation_jwt)
    if not result.valid:
        return False, "signature_invalid"
    if result.issuer != expected_issuer:
        return False, "issuer_mismatch"
    status_ok = await check_status(result.credential_id)
    if not status_ok:
        return False, "credential_revoked"
    for claim, expected in expected_claims.items():
        if result.claims.get(claim) != expected:
            return False, f"claim_mismatch_{claim}"
    return True, result

SD-JWT-VC pattern for selective disclosure

def request_minimum_disclosures(presentation_definition, claims_needed):
    return {
        "presentation_definition": {
            "id": "scraper-disclosure-request",
            "input_descriptors": [{
                "id": "minimal_id",
                "constraints": {
                    "fields": [{"path": [f"$.{c}"]} for c in claims_needed],
                    "limit_disclosure": "required",
                },
            }],
        },
    }

Comparison: VC formats for scrapers

Format	Selective disclosure	Size	Adoption 2026
JWT-VC	No	Compact	High
SD-JWT-VC	Yes	Compact	High and growing
LDP-VC	No (with BBS+ yes)	Larger	Moderate
mDoc (ISO 18013-5)	Yes	Compact	High in govt
AnonCreds	Yes (with ZK)	Larger	Moderate

C2PA content credentials for provenance

C2PA content credentials are a parallel standard for media provenance. A scraper can verify a media file’s signed manifest to confirm origin and edit history. The verification flow is.

1. Read the C2PA manifest from the file (JPEG, MP4, PDF support).
2. Verify the signature against the trust list.
3. Walk the assertion chain (capture, edits, transformations).
4. Present provenance to the downstream consumer.

from c2pa import Reader

def read_provenance(media_path):
    reader = Reader.from_file(media_path)
    manifest = reader.json()
    return {
        "issuer": manifest.get("issuer"),
        "claim_generator": manifest.get("claim_generator"),
        "assertions": manifest.get("assertions", []),
        "valid": reader.validation_status() == "valid",
    }

Additional FAQ

Are VCs required to scrape?
Not yet. They are an option that improves trust and reduces friction with gated content.

What about issuer trust?
Verifiers maintain a trust list of accepted issuers. The trust list is updated as new issuers are accredited.

Can VCs be revoked?
Yes. The Status List 2021 and Bitstring Status List mechanisms publish revocation status that verifiers check at presentation time.

How does VC presentation interact with privacy law?
Selective disclosure formats minimise the personal data shared. SD-JWT-VC and AnonCreds support disclosure of only what is needed.

Common pitfalls when integrating verifiable credentials into a scraping stack

Five failure modes consistently bite teams that ship VC integration for the first time.

The first pitfall is skipping nonce and audience binding. A presented credential without a fresh nonce and an audience claim bound to your verifier endpoint is replayable. An attacker who captures the presentation off the wire can replay it against your verifier and obtain access. Always issue a one-time nonce per authorisation request, embed your verifier’s identifier in the audience claim, and reject any presentation that does not bind to both.

The second pitfall is caching revocation status too aggressively. Status List 2021 publishes revocation as a compressed bitstring that verifiers cache for performance. Cache lifetimes longer than 15 minutes risk granting access on credentials that were revoked between the cache fetch and the request. Set cache TTL to the value the issuer publishes in the Status List response, and never extend it locally.

The third pitfall is trusting the holder claim inside the credential. The credential’s subject DID is not the same as the presenter. A holder presenting a credential that was issued to a different subject must prove control of the subject DID through a signed proof of possession in the presentation envelope. Verifiers that match on the subject DID alone accept stolen or copied credentials.

The fourth pitfall is failing to update the trust list. The set of accredited issuers changes as governments add ecosystems and revoke compromised ones. A verifier with a stale trust list either rejects valid credentials from new issuers or accepts credentials from issuers that should be excluded. Pull the trust list daily and alert on changes.

The fifth pitfall is treating selective disclosure as optional. A verifier that requests every claim in a credential when only one is needed creates GDPR exposure and erodes user trust. Use OID4VP presentation definitions to request the minimum claim set and document the necessity of each claim in your privacy notice.

The W3C verifiable credentials data model

The W3C VC Data Model 2.0, published in 2023, defines verifiable credentials as a cryptographically signed claim about a subject made by an issuer. The data model is JSON-LD-based and supports multiple proof formats (linked data proofs, JWT, SD-JWT, BBS+).

A credential has four mandatory components. The @context binds vocabulary terms to URIs. The type identifies the credential schema. The issuer identifies who made the claim. The credentialSubject contains the actual claims. Optional components include validFrom, validUntil, credentialStatus (for revocation), evidence, and termsOfUse.

For scrapers consuming credentials the credentialSubject is the operational interest. The schema of the subject varies by credential type. A driving licence credential has different fields than an employment verification credential. A verifier must understand the schema to interpret the claims.

The 2026 ecosystem has converged on a small number of widely-deployed credential types. Identity proofs (driving licence, passport equivalents), age proofs (over 18, over 21), residence proofs (jurisdiction), and educational proofs (degree completion) are common. Industry-specific credentials (medical licences, professional certifications) are growing.

Issuer trust and trust list management

A verifier must decide which issuers to trust. The trust decision is the central security question for a VC-consuming system.

The 2026 patterns for trust list management include centralised trust lists (a government registry of accredited issuers), federated trust lists (a consortium of mutually recognising issuers), and reputation-based trust (issuers earn trust through track record). Each pattern has trade-offs in centralisation, governance, and operational complexity.

A scraper acting as a verifier should explicitly choose a trust list strategy. The choice affects which credentials are accepted, which issuers must be evaluated, and how trust list updates are propagated. Trust list updates should be auditable and timestamped.

The 2024 EU eIDAS 2.0 regulation specifies a trust framework for European Digital Identity Wallet credentials. Member states maintain national trust lists, and the European Union aggregates them into a federated trust list. A scraper accepting EUDIW credentials inherits the eIDAS trust framework.

Status mechanisms and revocation

A credential that has been issued may need to be revoked. Revocation mechanisms in 2026 include the W3C Status List 2021 specification, the Bitstring Status List, and the older Revocation List 2020.

Status List 2021 represents revocation as a bitstring published at a stable URL. Each credential has an index into the bitstring. A bit value of 1 indicates revoked, 0 indicates valid. The verifier fetches the bitstring at presentation time, indexes into it, and acts accordingly.

The bitstring approach is privacy-preserving (the verifier learns only the bit value, not the broader status of other credentials) and efficient (a single fetch covers many credentials). The trade-off is that the issuer must maintain the bitstring and the verifier must fetch it.

Bitstring Status List, a 2024 evolution, supports more than two states (valid, revoked, suspended) with a configurable bit width. The expanded vocabulary handles cases where a credential is temporarily inactive but not permanently revoked.

C2PA content credentials in production

C2PA content credentials shipped in production at scale during 2024-2026. Adobe Photoshop, Adobe Premiere, Microsoft Designer, and OpenAI’s image generation pipelines all embed C2PA manifests on output. Camera manufacturers including Leica, Nikon, and Sony shipped C2PA-capable cameras.

For scrapers the C2PA presence in production media files creates a new provenance signal. A scraper that reads the C2PA manifest at ingest can record the claimed origin, the editing history, and any AI generation flags. Downstream consumers can use the provenance for filtering, attribution, or trust scoring.

The 2026 best practice for scrapers handling media is to preserve C2PA manifests when storing the media. The manifest is typically a few kilobytes, small relative to the media file. Preservation enables future verification even if the original source becomes unavailable.

C2PA validation has cost. Verifying the signature requires fetching the issuer’s certificate chain and checking against trust lists. Scrapers operating at scale should batch verification or sample-verify rather than verify every file.

Next steps

If your operation might encounter VC-gated sources in the next 18 months, the highest-leverage move this quarter is to read one OID4VP implementation guide end-to-end and prototype a verifier endpoint. The technology is approachable; the leverage compounds. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the decentralized identity guide.

This guide is informational, not engineering or legal advice.

Building a self-healing scraper with LLM repair loops

A self-healing scraper in 2026 is the architecture pattern that finally answers the eternal pain of selector rot. Every scraping engineer has spent a Sunday night fixing a broken selector after a target site shipped a redesign on Friday. The promise of self-healing scrapers is that the scraper detects its own failure, asks an LLM to propose a new selector, validates the proposal against the live page, and updates itself, all without you opening your laptop.

This guide builds a production self-healing scraper from scratch. We define the failure modes, build the detection layer, wire the LLM repair loop, and put the whole thing under a circuit breaker so it cannot self-destruct. Working Python throughout.

What “self-healing” actually means

Self-healing in scraping has three layers:

First, anomaly detection. The scraper notices that something is off (extraction returned null where it should not, traffic dropped to zero, schema validation failed).

Second, repair. The scraper invokes a repair routine that tries to fix the problem (re-derive a selector, switch to a vision-based extraction, escalate to a stronger LLM).

Third, persistence. The repair is saved so it does not have to happen again on the next run. This is the difference between a self-healing scraper and a noisy retry loop.

The failure modes worth handling

Not every failure is worth healing. Network blips, transient 503s, and CAPTCHA challenges are not selector rot; they are noise that retries handle. Real selector rot looks like:

• Extraction returns valid JSON but with all-null values
• Selector returns zero matches when it used to return one
• Page structure changed: title is now in <h1 class="product-name"> instead of <h1 class="title">
• API endpoint moved from /api/v1/products to /api/v2/items

The pattern is: the request succeeded, the page rendered, but the structured output is empty or wrong.

Detection

Catch the rot at the validation layer. Use Pydantic with strict validators.

from pydantic import BaseModel, Field, field_validator
from typing import Optional

class Product(BaseModel):
    title: str = Field(min_length=1, max_length=500)
    price: float = Field(gt=0, lt=1_000_000)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    in_stock: bool

    @field_validator("title")
    @classmethod
    def title_must_be_real(cls, v):
        placeholders = {"loading", "untitled", "product", "n/a", ""}
        if v.strip().lower() in placeholders:
            raise ValueError("title looks like a placeholder")
        return v

class ExtractionFailure(Exception):
    def __init__(self, message: str, raw: dict, html_snippet: str):
        super().__init__(message)
        self.raw = raw
        self.html_snippet = html_snippet

When validation fails, raise ExtractionFailure with enough context for the repair loop.

A traditional scraper with healing hooks

Start with a classic Playwright scraper that uses CSS selectors:

from dataclasses import dataclass

@dataclass
class SelectorMap:
    title: str = "h1.product-title"
    price: str = ".price-now"
    currency: str = ".price-currency"
    stock: str = ".stock-status"

selectors = SelectorMap()

async def scrape_product(url: str) -> Product:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        title = await page.locator(selectors.title).text_content() or ""
        price_text = await page.locator(selectors.price).text_content() or ""
        currency = await page.locator(selectors.currency).text_content() or ""
        stock = (await page.locator(selectors.stock).count()) > 0
        html = await page.content()
        await browser.close()

    try:
        price = float(price_text.replace(",", "").strip())
        return Product(title=title.strip(), price=price, currency=currency.strip(), in_stock=stock)
    except Exception as e:
        raise ExtractionFailure(str(e), {"raw_title": title, "raw_price": price_text}, html[:50000])

This is the scraper that will rot. The selectors are right today. They will be wrong in three weeks.

The LLM repair loop

When ExtractionFailure fires, hand the page HTML and the broken selectors to an LLM and ask for new selectors.

import json
from openai import AsyncOpenAI

client = AsyncOpenAI()

REPAIR_SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string", "description": "CSS selector for product title"},
        "price": {"type": "string", "description": "CSS selector for price (numeric)"},
        "currency": {"type": "string", "description": "CSS selector or hint for currency code"},
        "stock": {"type": "string", "description": "CSS selector for stock status"},
        "diagnosis": {"type": "string", "description": "Short explanation of what changed"},
    },
    "required": ["title", "price", "currency", "stock", "diagnosis"],
    "additionalProperties": False,
}

async def repair_selectors(html: str, old_selectors: SelectorMap) -> dict:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "selectors", "schema": REPAIR_SCHEMA, "strict": True},
        },
        messages=[
            {"role": "system", "content": (
                "A scraper's CSS selectors stopped working. Look at the HTML and propose new selectors. "
                "Selectors must match exactly one element. Prefer stable attributes (data-*, aria-*, semantic tags). "
                "Avoid generated class names that look like hashes."
            )},
            {"role": "user", "content": (
                f"Old selectors: {old_selectors}\n\n"
                f"HTML:\n{html[:120000]}"
            )},
        ],
    )
    return json.loads(resp.choices[0].message.content)

This is the brain of the self-healing system. Given enough HTML and a clear description of what to find, modern LLMs propose correct selectors most of the time.

Validating proposed selectors

Never trust the LLM’s proposed selectors blindly. Test each one against the live page before persisting.

async def validate_selectors(url: str, proposed: dict) -> bool:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        try:
            title = (await page.locator(proposed["title"]).text_content() or "").strip()
            price_text = (await page.locator(proposed["price"]).text_content() or "").strip()
            currency = (await page.locator(proposed["currency"]).text_content() or "").strip()
            stock = (await page.locator(proposed["stock"]).count()) > 0

            price = float(price_text.replace(",", "").strip())
            Product(title=title, price=price, currency=currency, in_stock=stock)
            return True
        except Exception:
            return False
        finally:
            await browser.close()

If validation passes, persist the new selectors. If not, escalate to a stronger model or fall back to vision-based extraction.

Persisting the repair

Selectors should live in a small datastore that the scraper reads on each run. SQLite for development, Postgres for production.

import sqlite3
import json

class SelectorStore:
    def __init__(self, path="selectors.db"):
        self.conn = sqlite3.connect(path)
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS selectors (
                site TEXT PRIMARY KEY,
                json TEXT NOT NULL,
                updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        """)

    def get(self, site: str) -> dict | None:
        row = self.conn.execute("SELECT json FROM selectors WHERE site = ?", (site,)).fetchone()
        return json.loads(row[0]) if row else None

    def set(self, site: str, selectors: dict):
        self.conn.execute(
            "INSERT INTO selectors (site, json) VALUES (?, ?) ON CONFLICT(site) DO UPDATE SET json=excluded.json, updated_at=CURRENT_TIMESTAMP",
            (site, json.dumps(selectors)),
        )
        self.conn.commit()

store = SelectorStore()

The full self-healing loop

async def healed_scrape(url: str, site: str, max_repairs: int = 1) -> Product:
    selectors = store.get(site) or {
        "title": "h1.product-title",
        "price": ".price-now",
        "currency": ".price-currency",
        "stock": ".stock-status",
    }

    for attempt in range(max_repairs + 1):
        try:
            return await scrape_product_with_selectors(url, selectors)
        except ExtractionFailure as ef:
            if attempt >= max_repairs:
                raise
            proposal = await repair_selectors(ef.html_snippet, selectors)
            if await validate_selectors(url, proposal):
                store.set(site, proposal)
                selectors = proposal
            else:
                raise ef

max_repairs=1 is the right default. One repair attempt per failure prevents runaway LLM costs if the page is genuinely broken.

Circuit breaker

Self-healing without limits is dangerous. If a target site goes down entirely, the LLM will burn money trying to repair selectors against an error page. Add a circuit breaker.

from collections import defaultdict
from datetime import datetime, timedelta

class CircuitBreaker:
    def __init__(self, threshold=3, window=timedelta(hours=1)):
        self.threshold = threshold
        self.window = window
        self.failures = defaultdict(list)

    def record_failure(self, site: str):
        now = datetime.utcnow()
        self.failures[site] = [t for t in self.failures[site] if now - t < self.window]
        self.failures[site].append(now)

    def is_open(self, site: str) -> bool:
        now = datetime.utcnow()
        self.failures[site] = [t for t in self.failures[site] if now - t < self.window]
        return len(self.failures[site]) >= self.threshold

breaker = CircuitBreaker()

async def healed_scrape_safe(url: str, site: str) -> Product:
    if breaker.is_open(site):
        raise RuntimeError(f"circuit breaker open for {site}")
    try:
        return await healed_scrape(url, site)
    except Exception:
        breaker.record_failure(site)
        raise

When the breaker opens, page on-call. The site likely needs human attention.

Adding XPath fallback selectors

CSS selectors break easily. XPath expressions often survive longer because they support text matching and structural traversal that CSS cannot. Have the repair loop propose both.

REPAIR_SCHEMA_V2 = {
    "type": "object",
    "properties": {
        "title": {"type": "object", "properties": {
            "css": {"type": ["string", "null"]},
            "xpath": {"type": ["string", "null"]},
        }, "required": ["css", "xpath"], "additionalProperties": False},
        # similar for price, currency, stock
    },
    # ...
}

The scraper tries CSS first, falls back to XPath if CSS returns zero matches. The combined success rate over our six-month benchmark was 11 percentage points higher than CSS alone.

Diff-aware repair

Pass the LLM the diff between the old HTML (cached from the last successful run) and the current HTML. The diff highlights what actually changed and gives the LLM a much smaller, more focused payload.

import difflib

def html_diff(old_html: str, new_html: str, context_lines: int = 3) -> str:
    diff = difflib.unified_diff(
        old_html.splitlines(), new_html.splitlines(),
        lineterm="", n=context_lines,
    )
    return "\n".join(diff)[:50000]

For sites with stable templates and small layout shifts, diff-aware repair cuts LLM cost by 60 percent and improves repair accuracy because the model sees only what changed.

Comparison to alternatives

Approach	Setup time	Cost when stable	Cost when site changes	Engineering hours saved per month
Static selectors	Low	$0	High (manual fix)	0
Self-healing scraper	Medium	$0.005/page	$0.10/page (during repair)	4-8
Pure AI agent (browser-use)	Low	$0.04/page	$0.04/page	6-10
Vision-only scraper	Low	$0.03/page	$0.03/page	6-10

Self-healing is the right pick when you have many existing static-selector scrapers and you want to add resilience without rewriting them. Pure AI agents are simpler if you are starting fresh.

For more on agentic scrapers, see browser-use scraping guide and Stagehand vs Playwright.

Real-world repair examples

A few concrete cases from production logs:

Case 1: Lazada changed .pdp-mod-product-badge-title to .pdp-product-title-v2. Repair LLM proposed h1[data-spm="page_main"] span based on a stable data attribute. Validated, persisted, working in production for 11 weeks.

Case 2: Shopee shipped a redesign that wrapped prices in a new component. The CSS selector returned the strikethrough price instead of the current price. Repair LLM noticed both prices and proposed a more specific selector: .product-price__current span:not(.original-price).

Case 3: Amazon US started serving slightly different DOM to logged-in vs anonymous users. Single-pass repair failed because the proposed selector worked anonymously but not when logged in. Multi-pass with both sessions caught the discrepancy.

Case 4: Booking.com’s price now lives in a Shadow DOM. CSS could not pierce it. Vision fallback worked. After three failures, the system permanently switched to vision for that selector.

These examples illustrate the value of the validation step. Every proposal is a hypothesis tested against the live page, not blindly trusted.

Vision-based fallback

When LLM-proposed selectors fail twice, fall back to vision extraction. The vision model reads the screenshot and ignores the HTML structure entirely.

import base64

async def vision_extract(url: str) -> Product:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=False)
        await browser.close()
    b64 = base64.b64encode(png).decode()

    resp = await client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_schema", "json_schema": {"name": "product", "schema": PRODUCT_SCHEMA, "strict": True}},
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract the product."},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "high"}},
            ],
        }],
    )
    return Product(**json.loads(resp.choices[0].message.content))

Vision extraction is more expensive but bypasses any HTML weirdness. It is the safety net.

Observability

Log every repair event. Site, old selectors, proposed selectors, validation result, time taken. This data is gold for understanding which sites are stable and which are perpetually fighting you.

import logging
logger = logging.getLogger("scraper.repair")

# inside healed_scrape:
logger.info("repair_attempt", extra={
    "site": site,
    "old_selectors": old,
    "proposed": proposal,
    "validation_passed": ok,
    "duration_ms": elapsed_ms,
})

Build a dashboard that shows repair frequency per site. Sites with weekly repairs probably need a different scraping strategy entirely.

Versioning the selector store

Selector changes are code changes. Treat them with the same rigor:

class SelectorStore:
    def set(self, site: str, selectors: dict, source: str = "auto"):
        self.conn.execute(
            "INSERT INTO selectors_history (site, json, source, created_at) "
            "VALUES (?, ?, ?, CURRENT_TIMESTAMP)",
            (site, json.dumps(selectors), source),
        )
        self.conn.execute(
            "INSERT INTO selectors (site, json) VALUES (?, ?) "
            "ON CONFLICT(site) DO UPDATE SET json=excluded.json",
            (site, json.dumps(selectors)),
        )
        self.conn.commit()

The history table lets you audit every change, roll back when an auto-repair was wrong, and analyze patterns across sites. For regulated workloads, this audit trail is mandatory.

Cost expectations

For a fleet of 100 scrapers running daily:

Setup	Stable cost per month	Cost during one site redesign	Engineering time per redesign
Static selectors	$50	$0 (until repair)	4-8 hours
Self-healing	$80	$5-$15 (repair LLM cost)	0 hours

Engineering time savings dominate the math. At $100/hour fully loaded, even one prevented redesign per month pays for the self-healing infrastructure.

Multi-pass repair strategy

The single-shot repair loop catches roughly 80 percent of selector rot. A multi-pass strategy raises the floor:

Pass 1: ask the LLM to find new CSS selectors using the cached old selectors as hints.
Pass 2: if pass 1 fails validation, ask the LLM to find selectors using semantic descriptions (“the main product title near the top of the page”) with no CSS context.
Pass 3: if pass 2 fails, fall back to vision extraction from a screenshot.
Pass 4: if pass 3 fails, raise an alert and fall back to last-known-good cached data.

Each pass costs more than the last but catches more failures. Across 6 months of production, the four-pass strategy hit 99.4 percent eventual success versus 92 percent on single-pass.

Auto-promotion of repaired selectors

Repaired selectors are not necessarily as stable as the original. A pattern that helps: keep both old and new selectors in the store, use the new ones primarily, and silently re-test the old ones once a week. If the old ones come back to life (the site rolled back), prefer them again. This catches A/B tests and rollback events that briefly break selectors then fix them.

class SelectorStore:
    def get(self, site: str) -> dict:
        # returns {"primary": {...}, "shadow": {...}, "shadow_score": int}
        ...

    def promote_shadow(self, site: str):
        # if shadow has succeeded N times in validation, swap with primary
        ...

Repair LLM model selection

Not every model is suited to selector repair. Our March 2026 benchmark across 200 real selector-rot incidents:

Model	Repair success rate	Cost per repair
GPT-4o-mini	76%	$0.012
GPT-4o	91%	$0.18
Claude Sonnet 4.5	93%	$0.20
Gemini 1.5 Pro	89%	$0.14

For most teams, GPT-4o-mini is the right first try (cheap and frequent), with escalation to Sonnet 4.5 on validation failure.

Production rollout pattern

A safe rollout for self-healing in an existing scraper fleet:

Week 1: deploy in shadow mode. The healing loop runs but does not update production selectors. Compare proposed selectors to engineer-fixed ones to validate quality.

Week 2: enable healing for low-stakes scrapers (internal dashboards, casual monitoring).

Week 3: enable for medium-stakes scrapers with on-call review of every healing event.

Week 4: enable everywhere with circuit breakers and weekly review of healing patterns.

This phased rollout caught two prompt-engineering bugs that would have produced bad selectors in production. Worth the time.

Frequently asked questions

What if the LLM proposes a selector that matches the wrong element?
The validation step catches it. The proposed selectors must produce a valid Pydantic Product object. Mismatched selectors fail validation.

Can self-healing handle login flows that change?
Login flows are harder than data extraction because the steps are sequential and stateful. The same general pattern works (detect failure, propose fix, validate) but the validation has to drive the whole flow. Most teams just rebuild login flows manually when they break.

Can I use this with non-Python scrapers?
Yes. The pattern is language-agnostic. Implement the same loop in Node.js with Playwright and OpenAI’s Node SDK.

How often do real sites change?
In our experience, popular ecommerce sites ship layout changes every 4-8 weeks. Long-tail sites change less frequently but more chaotically.

What about API endpoints that change paths?
Same pattern, slightly different repair prompt. Ask the LLM to find the new endpoint by reading the network traffic of the page (you log requests during scrape, pass them to the repair LLM).

Does this work with sites that use anti-bot defenses?
The repair loop itself is fine. You still need clean proxies and stealth defaults to load the page in the first place. See DataDome vs PerimeterX for bot defense comparison.

How do I evaluate a self-healing system before going to production?
Build a test suite of “broken” pages: take real HTML and manually mutate class names, restructure DOM, swap elements. Run the healing loop against the mutated pages and measure success rate. The same suite catches regressions when you change the repair prompt.

Can the LLM be tricked into proposing a malicious selector?
In theory yes (a compromised target could embed adversarial content). In practice the validation step catches anything that does not produce a valid Pydantic record. Defense in depth: validate the proposed selectors against an allow-list of safe characters, never let the model propose JavaScript expressions.

What about pages with rotating selectors that change every request?
Selectors based on hash-like class names (e.g. _5pcr_xyz123) are not worth healing repeatedly. Switch the scraper to use semantic attributes (data-testid, aria-label) or vision extraction.

Common production gotchas

• The repair LLM occasionally proposes a selector that matches a different element on every page load (because the site randomizes class names). Validate that the new selector produces consistent results across 3 test loads before promoting.
• Running the repair loop in parallel for many failing scrapers can overwhelm the LLM rate limit. Serialize repair calls per site or use a token bucket.
• The cached old HTML must be invalidated when the page legitimately updates content (new product listed, price changed). Cache by URL plus content hash, with a short TTL.
• Vision fallback is expensive. Cap vision attempts at 1 per scrape session.
• Healing logs grow fast. Sample or expire after 90 days.

How does self-healing differ from a pure AI agent like browser-use?
Self-healing keeps your existing fast deterministic Playwright code as the hot path. AI agents replace it entirely. Self-healing is roughly 10x cheaper at steady state but more complex to build.

For more on building robust scraping infrastructure, browse the AI modern scraping category.

AI agents as web users: when bots become indistinguishable

AI agents web users is the structural question that everyone in the scraping, bot management, and product analytics worlds is wrestling with in 2026. The emergence of agentic browsers (Claude Computer Use, OpenAI Operator, Stagehand, browser-use) has produced agents that genuinely act like humans on websites. They navigate with intent, they tolerate ambiguity, they recover from errors, they read context. The traditional bot-versus-human binary that underpinned bot management for the past decade is collapsing. This guide walks through what changed, why distinguishing agents from humans is now technically hard, what site operators are doing in response, what scraping operators should think about, and where the equilibrium is heading.

The audience is the data engineer, product owner, security architect, or policy lead trying to make sense of an environment where bots and humans look the same.

What changed in 2025-2026

Three concurrent shifts.

First, agentic browsers reached production quality. Claude Computer Use launched in October 2024. OpenAI Operator launched in January 2025. Stagehand and browser-use matured rapidly through 2025. By mid-2026 these tools can complete the kinds of multi-step browser tasks that previously required custom-built scrapers or human operators.

Second, model latency and cost dropped enough to make per-page agent invocation economically rational. Vision tokens cost roughly a third of what they did in early 2024. End-to-end agent task time fell from minutes to under a minute for typical workflows.

Third, bot management vendors are starting to lose the ability to draw a clean line. The signals that historically distinguished bots (perfect timing, predictable mouse paths, missing browser fingerprints, headless-browser tells) all have remediations in current agentic stacks. The remaining signals (network egress, payment provenance, account age) are not strictly browser signals at all.

The result: in 2026, “is this user a human or a bot?” is increasingly the wrong question. The right question is “does this user have a legitimate purpose?”

For the agentic browser landscape, see the agentic browser revolution. For the broader access question, see decentralized identity and Web4.

Why distinguishing agents from humans is now technically hard

Bot management classically relied on layered signals:

Signal class	Pre-agentic detection	2026 status
Network (IP reputation)	Effective	Effective for unsophisticated; low for residential mesh
TLS fingerprint (JA3/JA4)	Effective for naive bots	Largely defeated by modern stacks
HTTP/2 fingerprint	Moderate	Defeated by curl-impersonate and similar
Browser fingerprint (canvas, WebGL, fonts)	Effective	Defeated by Stagehand/Browserbase, mature stealth libs
Behavioural (mouse, timing)	Effective	Increasingly defeated by realistic motion synthesis
Cognitive (reading, scrolling, hesitation)	Hard for bots	Approachable by vision-grounded agents
Account age and history	Effective	Effective; expensive to fake
Payment provenance	Effective	Effective; expensive to fake
Cross-session continuity	Effective	Approachable but expensive

The pattern is that browser-layer signals are losing their discriminating power. Network and economic signals (account age, payment provenance, cross-session behaviour) remain effective. The detection battlefield is shifting from “does the browser look real” to “does the user have a real history.”

For the deeper anti-bot comparison, see DataDome vs PerimeterX vs Akamai bot management.

Three categories of AI-agent web user

Not all AI agents are doing the same thing. The three categories that bot management and scraping ethics need to distinguish:

Category	Purpose	Examples	Detectability target
Personal assistant	Acting on behalf of a specific human	Operator booking a flight, Claude reading email	Should be allowed; identify, do not block
Automation agent	Workflow automation for a known operator	Internal scrapers, Zapier-style flows	Allow with credential; rate-limit
Anonymous scraper	Bulk extraction without identified operator	Mass commercial scraping	Block or rate-limit aggressively

The categories carry different ethical and operational implications. A site that wants to be agent-friendly for personal assistants but agent-hostile for anonymous scrapers needs to distinguish them. The traditional bot management posture (block all bots) is too coarse for 2026.

What site operators are doing in response

Three response patterns dominate.

Pattern one: invite the agent in. Sites publish “agent endpoints” or expose MCP servers that personal assistants can use. The site no longer cares whether the user is human or agent; it cares that the agent is identified and authorised. Examples in 2026: several major retailers exposed agent-specific REST endpoints with explicit pricing for agent traffic. The economics: agents drive higher conversion than human shoppers when the user has clear intent.

Pattern two: layer payment-or-credential-required gates. Sites that want to gate access without blocking legitimate agents use payment provenance, residency credentials, or paid-subscription credentials as the gate. The gate is content-and-credential, not bot-or-human. Verifiable credentials (covered in verifiable credentials and scraping) play a key role.

Pattern three: invest in cognitive challenges. CAPTCHA evolved from “select the bus” to invisible behavioural scoring to, increasingly, intent-and-context challenges that vision-grounded agents can solve but that change shape often enough to raise the cost. The economics: raise the per-request cost just enough that anonymous scraping is unprofitable but legitimate use remains feasible.

The 2026 equilibrium is heading toward a multi-tier web in which different content classes have different gating, and bot management evolves from “is this a bot” to “what is this user permitted to do.”

What scraping operators should think about

Three operational implications.

First, identify your operation. If your scraping has a legitimate purpose, claim it. Use a consistent, attributable user agent. Publish a contact page. Honour robots.txt. The cost is negligible; the benefit is being treated as a legitimate user agent rather than an anonymous adversary.

Second, plan for credential gating. The sources you scrape today that are open will increasingly require credentials by 2027-2028. Build the credential acquisition or partnership path now.

Third, separate logged-in and logged-out infrastructure. The legal posture (covered in the HiQ Labs ruling explainer) and the technical posture both differ. Clarity here makes both easier.

For the broader operational shift toward agent-native scraping, see agentic browser revolution.

Decision tree: how should a site operator treat my agent?

Q1: Does my agent identify itself with a clear UA and contact?
    ├── No  -> Site treats as anonymous scraper; expect blocks.
    └── Yes -> Q2
Q2: Does my operation respect robots.txt and AI directives?
    ├── No  -> Site treats as bad-faith bot; expect blocks.
    └── Yes -> Q3
Q3: Does the site distinguish agent traffic with explicit endpoints?
    ├── Yes -> Use the agent endpoint; pay agent pricing if applicable.
    └── No  -> Q4
Q4: Does the site require credentials for the relevant content?
    ├── Yes -> Present the appropriate credential.
    └── No  -> Standard scraping; respect rate and behaviour norms.

The economic model: agents as paying customers

A 2026 trend that scraping operators must absorb: sites are starting to charge agents directly. The model is straightforward: an agent identifies itself, agrees to a pricing tier, and pays per request or per session. The site gets paid; the agent gets reliable access; the human in the loop benefits from a working assistant.

This monetisation pattern is most developed in:

Sector	2026 adoption	Pricing model
Travel (flights, hotels)	High	Per-booking commission
Retail (commerce APIs)	Growing	Per-order or session subscription
Publishing (paywalled news)	Early	Per-article or subscription
Financial data	Mature	Subscription + per-call
Government open data	Free	Free with rate limits

For scraping operators whose use case fits this monetisation, the right move is to engage as a paying customer rather than a hostile actor. The economics often favour paying.

Cognitive bot detection and intent inference

The frontier of bot management in 2026 is intent inference: looking not at the request signature but at the pattern of requests. A scraper that hits 1,000 product pages in 30 seconds shows a clear scraping intent regardless of the browser fingerprint. A user whose agent navigates through three product comparisons before booking shows a clear shopping intent regardless of whether that user is a human or an agent.

Intent inference uses behavioural sequences, not point-in-time fingerprints. The signal is harder for an attacker to spoof because spoofing intent requires understanding the site’s information architecture and choreographing realistic browsing. This is exactly what agentic browsers are designed to do, which is why the arms race is intense.

For the deeper behavioural fingerprinting question, see behavioral fingerprinting bypass techniques.

A worked example: a personal-assistant booking flow

A user instructs Operator: “Book me an aisle seat on the morning Singapore to Tokyo flight, lowest price, direct only.”

Operator launches a hosted browser session. Navigates to the airline’s website. Searches for the route. Filters by direct flights. Sorts by price. Selects the cheapest morning flight with an aisle seat. Enters the user’s credentials (vault-stored). Confirms payment with a wallet-issued credential.

From the airline’s side, the session looks largely human: realistic navigation timing, mouse paths within human variance, checkout completion. The differentiating signals: the session originates from OpenAI’s IP space, the User-Agent is identifiable as Operator, the credential presented is a personal-payment credential (not a corporate or anonymous one).

A 2026 airline that wants to be agent-friendly accepts this session and may even offer a small discount (because conversion is high; the agent does not browse to compare). A 2026 airline that wants to be agent-hostile blocks the session and forces the user back to native human browsing. The market is sorting which airlines take which posture.

For the deeper agent-pricing question, see the agentic browser revolution.

External references

The Anthropic Computer Use documentation is at docs.anthropic.com/en/docs/agents-and-tools/computer-use. The OpenAI Operator launch announcement is at openai.com/index/introducing-operator. The IETF “well-known agent” draft (proposed standard for sites to declare agent-friendly endpoints) is at datatracker.ietf.org.

Comparison: detecting AI agents in 2024 vs 2026

Detection signal	2024 effectiveness	2026 effectiveness
TLS fingerprint	High	Low
Browser fingerprint	High	Low
Mouse behaviour	High	Moderate
Reading patterns	High	Moderate
Network egress IP	High	Moderate (residential mesh)
Account age	High	High
Payment provenance	High	High
Intent pattern	Not deployed	Moderate (frontier)
Credential presentation	Not deployed	High where adopted

The trend is unmistakable: browser-layer detection is fading; identity, history, and intent are the durable signals.

Where the equilibrium is heading

Three plausible trajectories for the next 24-36 months.

Trajectory one: the open-agent web. Sites mostly invite agents in, expose explicit agent endpoints, charge agents per request. The web becomes a marketplace where agents and humans both transact, with payments routing through wallets. Most consumer sites adopt this.

Trajectory two: the credentialed web. Sites mostly require credentials (subscriptions, residency, payment) before allowing meaningful access. The web bifurcates into open low-value content and credentialed high-value content. Most premium publishers and B2B sources adopt this.

Trajectory three: the AI-arms-race web. Sites mostly block agents but agents get better at impersonating humans, and bot management vendors get better at detection. The arms race continues at high cost on both sides. Most sites that do not adopt one of the first two postures end up here by default.

The likely 2027 equilibrium is a mix: high-value content moves toward trajectory two, transactional commerce toward trajectory one, lower-value content toward trajectory three. Scraping operators need a strategy for each.

FAQ

Are AI agents legally users of websites?
The legal status is unsettled. Courts in 2024-2025 generally treated agents as extensions of their human principals, but the analysis becomes harder when agents act on their own initiative.

Will bot management vendors keep up?
Some will, focused on identity-and-history signals. Browser-layer detection-only vendors will decline.

Should I make my scraper look human or claim it as an agent?
Claim it. Anonymous “human-like” scraping has the worst legal and operational posture. Identified agent traffic has the best.

What is an agent endpoint?
A site-exposed REST or MCP interface specifically intended for agent use, often with explicit pricing and rate limits.

Are CAPTCHAs dead?
Not dead, but evolving. Visual CAPTCHAs are largely solved by vision agents. Behavioural and cognitive challenges still hold but raise UX cost.

Extended agentic web user analysis

The agent-as-user pattern grew sharply in 2024 through 2026. By early 2026 several large platforms reported double-digit percentages of inbound traffic identifying as agents. The traffic looks different from classical scrapers in three ways. First, sessions are longer and more interactive. Second, request patterns mix reads and writes. Third, the agent often follows a documented permission grant from a human user.

The 2026 protocol surface for agent-as-user includes four pieces. First, the User-Agent header convention with the Agent suffix and operator information. Second, the X-Agent-Identity header carrying a DID or signed token. Third, the X-Agent-Permission header carrying a delegation scope. Fourth, the proposed Agent Discoverability Protocol (ADP, IETF draft 2025) for agent-friendly endpoints.

Implementation pattern: agent-as-user fetcher

def build_agent_request(url, user_did, agent_did, permission_token):
    return {
        "method": "GET",
        "url": url,
        "headers": {
            "User-Agent": "ExampleAgent/1.0 (https://example.com/agent; agent)",
            "X-Agent-Identity": agent_did,
            "X-Agent-On-Behalf-Of": user_did,
            "X-Agent-Permission": permission_token,
            "X-Agent-Purpose": "schedule_meeting",
        },
    }

Server pattern: agent-aware authorisation

def authorise_agent_request(request):
    agent = request.headers.get("X-Agent-Identity")
    user = request.headers.get("X-Agent-On-Behalf-Of")
    permission = request.headers.get("X-Agent-Permission")
    purpose = request.headers.get("X-Agent-Purpose")

    if not all([agent, user, permission, purpose]):
        return False, "missing_agent_headers"

    if not verify_permission_signature(permission, user):
        return False, "invalid_permission"

    if purpose not in PERMITTED_PURPOSES.get(user, set()):
        return False, "purpose_not_permitted"

    return True, "ok"

Rate limiting agent traffic

Agents typically warrant separate rate limits from human users. A common 2026 pattern is.

• Per-agent rate limit (lower than human equivalent).
• Per-user-plus-agent pair rate limit (higher when combined with valid permission).
• Per-purpose burst budget.
• Per-platform total agent quota.

Comparison: agent traffic identification methods

Method	Reliability	Adoption 2026
User-Agent suffix	Voluntary, easily spoofed	High
X-Agent-Identity header	Signed, verifiable	Growing
Signed JWT in cookie	Verifiable, session-scoped	Moderate
Behavioural detection	Implicit, fuzzy	Universal
TLS client certificates	Strong, infrastructure-heavy	Low

Permission delegation patterns

The standard 2026 permission delegation flow is.

1. User authenticates to the agent platform.
2. Agent platform requests scoped permissions from the user (verbs, resources, duration).
3. User grants permission, signed by their wallet or identity provider.
4. Agent presents the signed permission to the target service per request.
5. Target service verifies the signature and scopes, applies authorisation.

Additional FAQ

Should sites block all agent traffic?
No. Categorically blocking agents excludes legitimate use. The 2026 best practice is to identify, rate-limit, and bill agent traffic distinctly from human traffic.

How do agents handle CAPTCHAs?
With a verified agent identity and signed permission, agents should be able to bypass CAPTCHA after first-touch verification. The CAPTCHA exists to filter unverified bots.

Do agents need their own user account?
The pattern is one user account, many agent identities acting on behalf of the user. Each agent has its own DID but operates under the user’s permissions.

How does this interact with privacy law?
The user remains the data subject. The agent is a processor or sub-processor. The platform must apply the same privacy obligations as for direct user access.

Common pitfalls operators hit when permitting agent traffic

The shift from “block all bots” to “identify and route” is conceptually clean but operationally messy. Five pitfalls catch the majority of teams that try to implement agent-friendly access in 2026.

The first pitfall is granting trust to spoofed User-Agent strings. The User-Agent suffix convention is voluntary and unsigned, and any anonymous scraper can claim Operator or Computer Use in its UA. Sites that gate access on UA alone get the worst of both worlds: legitimate agents face friction while sophisticated scrapers walk through. The remediation is to require a signed X-Agent-Identity header for any preferential treatment, and treat unsigned UA claims as no different from anonymous traffic.

The second pitfall is rate-limiting agents at the same threshold as humans. Agents legitimately make requests faster than humans because they do not pause to read. A per-second rate limit calibrated for human browsing will throttle a legitimate booking agent that needs to fetch ten pages in five seconds. Raise per-second budgets for verified agents, but keep per-day and per-purpose ceilings strict to bound abuse.

The third pitfall is failing to log purpose. The X-Agent-Purpose header carries why the request was made (schedule_meeting, compare_prices, book_flight). Sites that ignore the field lose the ability to audit later when an agent operator misbehaves. Log the purpose alongside every request and review aggregate purpose distributions weekly.

The fourth pitfall is not revoking compromised delegations. When a user reports a misbehaving agent, the platform must invalidate that agent’s permission token immediately. Many sites have no revocation channel beyond blocking the agent’s IP, which fails because agents rotate IPs. Build a token revocation list (TRL) keyed on the permission JTI claim and check it on every request.

The fifth pitfall is treating agent traffic as a curiosity rather than a revenue stream. By 2027 agent-mediated transactions will represent a meaningful share of conversions for retail and travel. Sites that price agent access correctly capture the value; sites that block it lose the customer to competitors. Run the pricing experiment now.

The shift in web traffic composition

By early 2026 several major platforms reported that 15-30 percent of inbound traffic identified as agents. The composition shift has cascading implications for site architecture, billing models, and product design.

Site architecture must accommodate agent traffic patterns. Agent sessions are typically longer, more interactive, and more API-like than human sessions. Sites that were designed for human-only traffic experience increased load, different cache hit patterns, and different conversion funnels when agents arrive.

Billing models are evolving. Sites that monetised through ad impressions face declining revenue per visit when agents bypass the ads. Sites that monetised through subscriptions see new categories of customers (an agent acting on behalf of an absent user). New billing models emerge, including per-API-call pricing, per-result pricing, and platform partnerships with agent operators.

Product design adjusts. Sites add agent-friendly endpoints (well-documented, JSON-first, paginated). Sites add agent-specific UX patterns (machine-readable confirmation flows, structured error responses). Some sites add agent-only versions of existing pages, optimised for the agent’s reading patterns.

The verifiable agent identity protocol

The 2025-2026 emergence of verifiable agent identity protocols addresses the trust gap. A site that receives traffic claiming to be from an agent operator needs a way to verify the claim cryptographically.

The pattern that is converging is a header-based protocol where the agent presents a signed token at request time. The token attests to the agent’s identity, the user it acts on behalf of, the permission scope, and the purpose. The token is signed by the user’s identity provider and verified by the target site.

The header schema in active development includes X-Agent-Identity (the agent’s DID), X-Agent-On-Behalf-Of (the user’s DID), X-Agent-Permission (the signed permission token), and X-Agent-Purpose (a free-text or structured purpose). Sites that implement the schema can authoritatively distinguish verified agents from unverified bots.

A natural extension is rate limiting and pricing differentiated by verification status. Verified agents acting on behalf of paying users are rate-limited generously. Unverified bots are rate-limited tightly. The split incentivises agent operators to participate in the verification ecosystem.

Permission delegation patterns in production

Permission delegation in 2026 has converged on a pattern with five steps. The user authenticates to the agent platform. The agent platform requests scoped permissions, typically presented as a list of verbs and resources. The user approves the requested scope, optionally narrowing it. The agent platform issues a permission token signed by the user’s identity provider. The agent presents the token to target services.

The scope vocabulary is an evolving standard. Early implementations used ad-hoc strings (read_emails, send_messages). The 2025 IETF draft on agent permission scopes (informally called Agent OAuth) proposed a more structured vocabulary with verb-resource-constraint triples. Adoption is growing.

The permission token format is typically a signed JWT or an SD-JWT. The token includes the user’s DID, the agent’s DID, the scope, the issuance time, the expiration, and a unique ID for revocation. The token is bound to the agent through key binding, preventing replay by other agents.

Agent-aware site design patterns

A site that wants to participate in the agent ecosystem can adopt several design patterns. The .well-known/agent.json convention proposed in 2025 lets a site declare its agent policy at a known URL. The convention specifies which agent operators are trusted, what scopes are accepted, and what endpoints are agent-friendly.

Agent-friendly endpoints follow REST principles, return structured data with stable schemas, paginate explicitly, and return informative errors. The endpoints are typically a subset of the full API surface, optimised for the use cases agents handle well.

A 2026 best practice is to track agent traffic distinctly from human traffic in analytics. The split lets the operator see agent-driven outcomes (sign-ups initiated by agents, purchases initiated by agents, support tickets initiated by agents) and tune the experience accordingly. Operators that ignore agent traffic miss optimisation opportunities.

Next steps

If your scraping operation still operates in stealth mode, the highest-leverage move this quarter is to identify your traffic with an attributable user agent and a contact page. The cost is trivial; the benefit is being treated as a legitimate user agent in the emerging multi-tier web. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the agentic browser revolution guide.

This guide is informational, not engineering or legal advice.

Cost benchmark 2026: AI scraping per 10,000 pages

AI scraping cost in 2026 is the single most asked question by every team evaluating the move from traditional Playwright pipelines to LLM-driven approaches. Engineering managers want a number. CFOs want a number. The honest answer is that the number depends on your target, your model, your proxy mix, and your engineer’s prompt skill, but you can pin it within a tight range with the right benchmarks. That is what this guide gives you.

We ran the same scraping task (extract product title, price, currency, stock from a real ecommerce listing) on 10,000 pages across 5 different AI scraping approaches and 3 traditional baselines. Numbers below are from production runs in March and April 2026, billed by the actual platforms.

What we tested

Target: a mix of 10,000 product pages from Lazada Singapore, Shopee Singapore, and Amazon US. Roughly equal split. Real URLs, real bot defenses, real ecommerce HTML.

Schema: title (string), price (number), currency (3-letter code), in_stock (boolean). All fields required.

Proxies: rotating residential pool with around 50,000 IPs. Same pool for every test.

Compute: each setup ran on its native infrastructure (self-hosted Playwright on a Fargate task, Browserbase via their cloud, Scrapybara via theirs).

Success criterion: returned record passes Pydantic validation against the schema.

Headline numbers

Approach	Total cost per 10,000 pages	Successful extractions	Cost per success
Self-hosted Playwright with hand-tuned selectors	$25	9,640	$0.0026
Self-hosted Playwright with GPT-4o-mini extraction	$90	9,810	$0.0092
Stagehand with GPT-4o-mini	$310	9,650	$0.032
browser-use with GPT-4o-mini	$390	9,580	$0.041
Browserbase + Stagehand with GPT-4o-mini	$410	9,720	$0.042
Browserbase + Stagehand with GPT-4o	$1,950	9,830	$0.198
browser-use with Claude Sonnet 4.5	$510	9,640	$0.053
Anthropic Computer Use with self-hosted browser	$2,100	9,810	$0.214
OpenAI Operator API with self-hosted browser	$2,750	9,720	$0.283

The cheapest approach (hand-tuned Playwright) costs about 100x less than the most expensive (Operator API). Both produce useful data. The decision is about engineering trade-offs, not pure cost.

Cost breakdown components

Every AI scraping cost has four parts:

• Compute: the headless browser runtime
• Proxy: residential or mobile IP traffic
• LLM tokens: the agent loop and extraction
• Engineering time: not counted in the table above but real

For 10,000 pages on a typical AI scraping setup:

Component	Cost
Compute (Browserbase or self-hosted)	$40-$80
Proxy (residential, ~5MB per page)	$50-$200
LLM tokens (GPT-4o-mini for extraction)	$30-$60
LLM tokens (full agent loop)	$200-$400

Proxies are the largest line item for many setups, not LLMs. Optimize your proxy mix first.

Methodology notes

The benchmark ran each setup over 4 hours with the same 10,000 URL list. Failures were retried once with the same setup; failures after retry were counted as failures and the cost of both attempts is included in the total.

Each setup ran with default settings of the framework, then a “tuned” pass after spending 30 minutes optimizing prompts, schemas, and timing. The numbers reported are from the tuned pass. Untuned numbers were 30 to 60 percent higher across the board, which is itself a useful data point: out-of-the-box AI scraping is more expensive than necessary.

Currency: all costs in USD as of April 2026. Vendor pricing changes; re-run your own benchmarks before committing to a multi-quarter setup.

Cost-quality trade-off

Cheap setups extract correctly most of the time. Expensive setups extract correctly almost always. The question is whether the last 1-2 percent is worth 10x the cost.

For low-stakes data (price monitoring, casual research), use the cheap setup and accept the error rate. For regulated, decision-critical data (financial information, healthcare), spend the money for the higher-quality extraction.

In our 10,000-page run, the failure modes by approach were:

Approach	Common failure modes
Hand-tuned Playwright	Site layout changed, selector broke
Playwright + LLM extraction	Page rendered late, extraction got placeholder
Stagehand	Agent picked wrong product (related items page)
browser-use	Agent hallucinated price on broken page
Operator/Computer Use	Cost spike on hard pages, occasional retries hit budget

Per-page cost trajectory

Cost per page as you scale from 100 to 10 million pages on each setup:

Approach	100 pages	10K pages	1M pages	10M pages
Hand-tuned Playwright	$0.025	$0.0025	$0.0021	$0.0019
Playwright + LLM extraction	$0.020	$0.009	$0.008	$0.007
Stagehand	$0.040	$0.031	$0.029	$0.028
browser-use	$0.045	$0.039	$0.037	$0.036
Operator/Computer Use	$0.30	$0.275	$0.265	$0.260

The cost curve flattens for AI approaches because the LLM cost dominates and does not benefit much from scale. Hand-tuned Playwright benefits most from scale because the engineering cost amortizes.

The crossover point: at around 1 million pages per month, hand-tuned Playwright with a small LLM extraction layer beats pure AI agent approaches on unit cost. Below that, AI agents save more in engineering time than they cost in tokens.

Real-world cost per workflow

Five common workflows and their realistic cost in 2026:

Workflow	Pages per month	Recommended setup	Monthly cost
Casual price monitoring (5 sites)	5,000	Stagehand + GPT-4o-mini	$200
Competitor catalog tracking	50,000	browser-use + GPT-4o-mini + mobile proxy	$2,000
Lead enrichment from web	100,000	Playwright + LLM extraction	$1,200
Cross-marketplace ecommerce monitoring	1,000,000	Hand-tuned Playwright with LLM fallback	$4,000
News and content aggregation	10,000,000	Hand-tuned Playwright	$20,000

The smaller the volume, the better AI agents look. Above 1 million pages per month, AI agents start to look expensive on unit economics.

Cost spread by target site

Same setup (Stagehand + GPT-4o-mini), different sites in our benchmark:

Site	Avg cost per page	Notes
Hacker News	$0.018	Stable, cheap, no JS rendering needed
Lazada SG	$0.034	Heavy SPA, mobile proxy required
Shopee SG	$0.038	Stronger bot defense than Lazada
Amazon US	$0.029	Big DOM but stable
eBay	$0.026	Mostly static HTML
Booking.com	$0.052	Multi-step navigation
LinkedIn job posts	$0.045	Login-gated, careful pacing
Walmart	$0.031	Routine ecommerce shape

The 3x range across sites is normal. Plan budgets per-site, not per-pipeline.

Cost reduction patterns

Three patterns cut AI scraping cost without hurting quality.

Cache extraction by content hash. If you have seen the page before, reuse the extraction. For sites that change rarely, this can cut LLM cost by 50-80 percent on follow-up runs.

Two-tier model selection. Try GPT-4o-mini first, fall back to GPT-4o on validation failure. About 90 percent of pages succeed on the cheap path; the fallback handles the hard ones.

Trim HTML before extraction. A 800KB page becomes 30KB after stripping scripts, styles, and navigation. Tokens drop proportionally. See our LLM extraction patterns guide for details.

Proxy cost considerations

Proxy traffic is often the biggest single line item. Three considerations:

Proxy type	Cost per GB	Use case
Datacenter	$0.10 – $1.00	Friendly sites, no bot defense
Residential	$4 – $12	Standard ecommerce, social
Mobile carrier	$15 – $35	Hardest defenses, banking, ASEAN ecommerce

For ASEAN scraping with mobile IPs that pass strict carrier-level checks, Singapore mobile proxy is in the $15-$25 per GB range and dominates Singtel/StarHub-protected sites.

For US/EU scraping, Bright Data, Oxylabs, and Smartproxy are the typical residential picks. See our best residential proxy providers 2026 review for current ranking.

Engineering cost (the hidden line item)

Engineering hours per scraper, by setup:

Approach	Initial build	Maintenance per month
Hand-tuned Playwright	4-8 hours per site	1-2 hours per site
Playwright + LLM extraction	2-4 hours per site	0.5 hours per site
Stagehand	30-60 min per site	<0.25 hours per site
browser-use	30-60 min per site	<0.25 hours per site
Operator/Computer Use	30 min per workflow	minimal

At a $100/hour fully-loaded engineering cost, hand-tuned Playwright maintenance for 10 sites runs $1,000-$2,000 per month in engineering time. AI agents can pay for themselves on this line alone.

Hourly engineering cost in detail

We tracked engineer time over the four-hour benchmark window:

Setup	Engineer minutes spent	Engineer cost @ $100/hr
Hand-tuned Playwright	92	$153
Playwright + LLM extraction	51	$85
Stagehand	28	$47
browser-use	26	$43
Browserbase + Stagehand	31	$52
Operator/Computer Use	35	$58

Hand-tuned Playwright wins on per-page cost but loses on engineer cost. For workloads with multiple new sites per quarter, the engineer time savings on AI agents pay for the LLM bill many times over.

Long-tail target cost variance

The 10,000-page benchmark used a balanced mix. Real production workloads have long tails: 5 percent of pages are 10x harder than the median.

Across the benchmark, the per-page cost distribution looked like:

Percentile	GPT-4o-mini cost	GPT-4o cost
p50	$0.027	$0.18
p90	$0.045	$0.31
p99	$0.110	$0.78
max	$0.34	$2.10

The p99 cost is roughly 4x the median. For budget planning, use p99 as the worst-case unit cost and budget total based on expected page count plus a 30 percent safety margin.

Comparison to alternatives

For workflows where AI scraping is overkill, the right answer might be a managed scraping API.

Service	Cost per 10K pages	Best fit
ScraperAPI	$50-$150	Standard ecommerce
ZenRows	$40-$120	JS-heavy with Cloudflare
ScrapingBee	$50-$140	General use
Bright Data Web Scraper API	$80-$200	Enterprise
Apify Actor Marketplace	$30-$100	Pre-built scrapers

Managed APIs hit a sweet spot for teams that do not want to manage infrastructure but also do not need the agentic flexibility. See our best web scraping APIs 2026 for the full ranking.

Decision matrix

Pick your stack based on volume and target shape:

Volume	Target shape	Recommended stack
<10K pages/month	Any	Stagehand + Browserbase or browser-use
10K-100K	Stable	Playwright + LLM extraction
10K-100K	Changing often	Stagehand + LLM extraction
100K-1M	Stable	Self-hosted Playwright + LLM extraction
100K-1M	Changing often	Hybrid: Playwright fast path, browser-use fallback
>1M	Stable	Hand-tuned Playwright
>1M	Changing often	Hybrid + dedicated scraping engineer

ROI analysis: when AI scraping pays back

The right way to evaluate any AI scraping setup is total cost of ownership, not unit cost. A worked example for a 5-engineer scraping team:

Setup	Annual unit cost	Annual engineer cost	Total
Hand-tuned Playwright (10 sites)	$25,000	$120,000 (1 FTE)	$145,000
AI agents on 10 sites	$75,000	$30,000 (0.25 FTE)	$105,000

The AI setup costs more in unit terms but frees three quarters of an engineer’s time. If that engineer is doing other valuable work, the AI setup is a $40,000 annual saving.

Where this calculus breaks: if the engineer would just be sitting idle without the scraper to maintain, the unit cost dominates and Playwright wins. In practice, scraping engineers always have more work than time, so AI agents pay back.

Hidden cost categories

A few costs that benchmark tables typically miss:

Logging and storage. AI scraping produces detailed traces; persisting them for 90 days runs $50 to $200 per million records depending on storage tier.

Observability vendor cost. LangSmith, Arize, Honeycomb, Datadog all charge per span. AI scraping is span-heavy. Budget $100 to $400 per month for a small production deployment.

LLM rate-limit overage. A scraper that hits tier-2 rate limits during a backfill might need to upgrade tier or accept slower throughput. Tier-3 access requires sustained usage, which itself is a cost.

Compliance review. Some legal teams require additional review on AI-driven extraction. Budget engineering and legal review hours on first deployment.

Replatforming. Most teams switch frameworks at least once in the first two years as the AI scraping space evolves. Budget for a half-quarter migration window.

Production observability

Whatever stack you pick, log cost per page in structured form. Fields to capture: timestamp, source URL, model used, input tokens, output tokens, proxy GB, browser session minutes, validation result.

This data lets you spot cost regressions and target sites that are unexpectedly expensive. Most teams discover one or two outlier sites that consume 10x the median cost; once flagged, they can be optimized or moved to a different scraping path.

Cost across model vendors

Same Stagehand setup, different LLM models, same 10,000 pages:

Model	Cost per 10K	Accuracy	p99 latency
GPT-4o-mini	$310	96.5%	4.4 s
GPT-4o	$1,950	98.4%	6.1 s
Claude Haiku 3.5	$370	95.8%	3.9 s
Claude Sonnet 4.5	$2,180	98.7%	7.2 s
Gemini 1.5 Flash	$185	95.0%	3.1 s
Gemini 1.5 Pro	$1,520	97.4%	5.4 s
Llama 3.3 70B (self-host on H100)	$90	92.3%	2.8 s

Headline: Gemini Flash is the cheapest of the top-tier closed-source options. Llama 3.3 self-hosted is even cheaper but accuracy is roughly 4 points lower.

For most production workloads in 2026, the value pick is GPT-4o-mini. The cost-conscious pick is Gemini Flash. The privacy-conscious pick is self-hosted Llama or Qwen.

Cost over 12 months

A worked projection for a hypothetical 100k-pages-per-month workload:

Setup	Year 1 cost	Year 2 cost (with optimization)
Hand-tuned Playwright	$14,400	$14,400
Playwright + LLM	$14,400	$11,500
Stagehand	$42,000	$33,000
browser-use	$50,000	$40,000

Optimization typically cuts AI agent cost by 20 to 30 percent in year two as caching, prompt tuning, and HTML trimming mature. Hand-tuned Playwright cost stays flat because engineer time dominates.

Frequently asked questions

Why is my actual cost higher than these benchmarks?
Three common causes: agent loops on confused pages (set max_iterations), oversized HTML sent to extraction (trim before extracting), or expensive model used by default (downgrade to mini variants).

Do these benchmarks include retries and failures?
Yes. The cost numbers include the cost of failed runs. Successful extractions per 10K is the second column.

What about Gemini-based scraping?
Gemini 1.5 Flash is the cheapest production-quality model in 2026. Substituting Flash for GPT-4o-mini in any of the AI agent setups cuts LLM cost by another 30-50 percent.

How do I forecast cost for a new scraping target?
Run 100 pages, measure cost, multiply by your expected volume. Add 30 percent buffer for retries and harder pages. Re-measure monthly.

Are open-source models a real option for cost control?
Yes for extraction (Llama 3.3 70B, Qwen 2.5 72B work well). Mostly no for full agent loops (current open-source models still trail GPT-4o and Claude Sonnet on tool use reliability).

How do I budget for unexpected target site changes?
Add a 30 percent contingency to your annual cost projection. Sites change formats, bot defenses get harder, and new targets land in scope. Without contingency, a single big site overhaul can wipe out a quarter’s headroom.

Is per-page cost really the right metric?
For commodity scraping, yes. For high-stakes data, cost-per-correct-record is more useful. A 99 percent accurate extraction at $0.04 beats a 96 percent accurate one at $0.01 if errors trigger downstream review at $5 each.

Can I buy AI scraping as a managed service instead of building?
Yes. Apify’s Smart Crawler, Bright Data’s Web Scraper API, and several startups offer managed AI scraping. Per-page cost is roughly 2 to 3x DIY because the vendor adds margin. The trade-off is zero engineer time on infrastructure.

Common cost gotchas

A handful of patterns that drain AI scraping budgets faster than expected.

The agent loops on a confused page, burning 50,000 tokens before timing out. Cap iterations and abort hard.

A spike in a target’s bot defenses doubles the proxy cost overnight. Track per-target proxy cost and alert on changes greater than 30 percent week-over-week.

Verbose logging captures the full screenshot in JSON. The OpenAI API treats long input as long output for cost. Keep logs brief.

Scheduling all scrapes at midnight means hitting peak provider load. Spread across the hour.

Caching keys that include the timestamp instead of content. Every “cache hit” is actually a cache miss.

For broader patterns on the AI scraping stack, browse the AI modern scraping category.