Decentralized identity and Web4: scrapers’ implications

Web4 decentralized identity is reshaping the assumptions scraping operators make about web access, authentication, and data trustworthiness. The collection of standards loosely grouped as Web4 (decentralized identifiers, verifiable credentials, agent-to-agent protocols, intent-based access) reached an inflection point in 2025-2026, with major platforms beginning to expose DID-based authentication, browsers shipping wallet integrations, and the IETF moving multiple drafts toward formal standardisation. For scrapers, the implications cut both ways: some doors that were locked behind central authentication open to DID-authenticated agents, while other doors close as anonymous unauthenticated scraping becomes harder. This guide walks through what Web4 actually means in 2026, the standards that matter, how DID and verifiable credentials change the access landscape, the scraping-relevant use cases, and a practical posture for operators.

The audience is the technical lead, product owner, or platform architect who needs to understand where decentralized identity fits in the scraping landscape they will operate over the next 24 months.

What Web4 actually means in 2026

The term “Web4” is contested. Different industry voices use it for different things. In 2026 the dominant usage refers to a converging set of standards and practices that move beyond the platform-mediated identity of Web2 and the wallet-mediated speculation of Web3 toward verifiable, portable, agent-friendly identity.

The constituent technologies:

Technology	Standard	Status (mid-2026)
Decentralized Identifiers (DIDs)	W3C Recommendation	Stable since 2022
Verifiable Credentials (VC)	W3C Recommendation	Stable since 2022
OpenID Connect for Identity Assurance	OpenID Foundation	Production
OpenID for Verifiable Credentials (OID4VC)	OpenID Foundation	Production
DID Comm Messaging	DIF spec	Late draft
Trust over IP framework	IETF / ToIP	Active drafts
Authority-bound digital wallets	Browser specs	Shipping in major browsers

The shift in 2025-2026 was that these standards moved from research to production. The EU Digital Identity Wallet (EUDI Wallet) reached general availability in mid-2026 across most member states. The UK Digital Identity Service began commercial issuance. Singapore’s Singpass added VC issuance. India’s DigiLocker integrated VC alongside its existing document store.

For scrapers, the relevant question is: what do these wallets carry, and which sites will require them?

For the broader emerging-tech context, see AI agents as web users and verifiable credentials and scraping.

DIDs explained for scraping operators

A Decentralized Identifier is a globally unique identifier that does not require a central registration authority. The format is did:method:identifier, where the method specifies how the identifier is resolved (did:web, did:key, did:ion, did:plc, and many more).

The point of a DID is that the holder controls the keys associated with it. A DID document, resolved by the method-specific resolution process, contains the public keys that the holder uses to authenticate.

For scraping access, DIDs change the authentication model in two ways. First, a site can require an authenticated visitor without the visitor needing an account on the site (the user’s wallet asserts their DID and signs a challenge). Second, the site can verify properties of the visitor (over 18, EU resident, paid subscriber to a credential issuer) without learning more.

A scraping operator who wants to access a DID-authenticated site has two options: obtain a DID and the relevant credentials (probably hard for scraping at scale), or partner with a credential holder who can act on behalf (cleaner but still bounded).

Verifiable credentials and selective disclosure

Verifiable Credentials are signed assertions issued by an issuer about a subject. A diploma is a credential. A driver’s licence is a credential. A subscription to a publication is a credential.

VCs use cryptographic signatures so that any verifier can confirm the issuer’s signature without contacting the issuer. The holder presents the credential as a Verifiable Presentation, which can include selective disclosure (showing only certain fields) and zero-knowledge proofs (proving a property without revealing the underlying data).

For scraping, VCs reshape the authorisation model. A site that today says “you must have a paid subscription to read this article” can, in a VC world, verify the subscription credential without requiring the user to have an account on the site. The credential travels with the user (or the user’s agent).

This has direct scraping implications:

Scenario	Pre-VC world	VC world
Paywalled article access	Requires site account, login flow	Requires VC presentation
Age-gated content	Account verification	Age VC selective disclosure
Geographic restriction	IP geolocation	Residency VC
Subscription bundling	Each site separate	Cross-site credential reuse

Scrapers operating against VC-protected sites face a fundamentally different access landscape. The traditional residential-proxy approach that defeats IP-based geofencing does not defeat credential-based gating.

For the broader credentials-and-scraping discussion, see verifiable credentials and scraping.

How agent-to-agent protocols matter

The DIDComm and Trust over IP frameworks specify how two agents (each with a DID) can establish authenticated, encrypted communication channels. The expected use case is human-to-human or service-to-service, but the protocols are agent-agnostic.

For 2026 scrapers, agent-to-agent protocols matter because they enable a new class of structured data exchange that bypasses the traditional scrape-or-API binary. Instead of scraping a website’s rendered HTML or hitting a vendor’s REST API, a scraping agent can establish a DIDComm channel with the source’s data agent, present a credential proving authorisation, and receive structured data over an encrypted channel.

The 2025-2026 deployments of this pattern are still early. Several supply-chain platforms expose DIDComm endpoints alongside their REST APIs. Several open-banking aggregators expose DIDComm as the preferred channel. The trend is real, the volume is small, but the trajectory points toward more agent-to-agent and less HTML-or-REST.

Comparison: identity models that scrapers operate within

Model	Identity authority	Visibility to scraper	Authorization mechanism
Web1 (open web)	None	Full	None
Web2 (platform)	Platform	Partial (if logged out)	Account + session
Web3 (wallet)	Self via blockchain	Pseudonymous	Wallet signature
Web4 (DID + VC)	Self with verified attestations	Selective	Credential presentation

Each model has different scraping implications. Web4 is the model where scraping needs to think about credentials, not just IPs.

Decision tree: how to access a Web4-authenticated source

Q1: Does the source require any form of authentication?
    ├── No  -> Standard scraping; existing techniques apply.
    └── Yes -> Q2
Q2: Is the authentication account-based (Web2 style)?
    ├── Yes -> Account creation; standard logged-in scraping considerations.
    └── No  -> Q3
Q3: Is the authentication credential-based (Web4 style)?
    ├── Yes -> Q4
    └── No  -> Likely wallet-signature (Web3); evaluate.
Q4: Can your operation legitimately hold the required credential?
    ├── Yes -> Implement VC presentation; proceed.
    └── No  -> Q5
Q5: Is there a partnership path with a credential holder?
    ├── Yes -> Negotiate access via partner.
    └── No  -> Source is effectively unscrapable for your operation.

The decision tree forces explicit consideration of the credential question. For sources where the answer is “unscrapable”, the alternative is partnership or licensed access.

Worked example: scraping a VC-gated medical research portal

A 2026 medical research portal hosts open-access papers but gates downloadable supplementary data behind a verifiable credential proving the requester is an affiliated researcher at an accredited institution.

Web2 scraping path: create an account if possible, validate email, request access, scrape what is exposed. Often blocked by manual review.

Web4 access path: hold a Researcher Credential issued by an accredited issuer (university, professional body). Present the credential at the portal. Receive structured data over the credential-authorised channel.

For a scraper operating on behalf of a research institution, the Web4 path is cleaner: the institution is already issuing credentials to its researchers; the scraper acts on behalf of the institution; the credential travels with the request.

For a scraper operating commercially without an institutional relationship, the Web4 path is closed. The operator has to either partner with an institution or rely on the portal’s open-access surface.

Browser wallet integration in 2026

Major browsers shipped wallet integrations in 2025-2026:

Browser	Wallet integration	Status
Brave	Native crypto + DID wallet	Production
Chrome	Optional via Web5 extensions	Mature extension ecosystem
Firefox	Native via Mozilla Account integration	Production
Safari	Apple Wallet integration	Apple-controlled
Edge	Microsoft Authenticator integration	Enterprise

Browser-resident wallets bring DID and VC presentation to the user-facing layer. The browser exposes a JavaScript API (the WebID specification) that sites can call to request credentials.

For headless and agentic browsers, the equivalent is wallet plug-ins that expose the same API but with programmatic credential management. Stagehand and Browserbase added wallet support in 2025; browser-use added it in 2026.

Privacy implications and selective disclosure

VCs support selective disclosure: a holder can present only the fields needed for a request. A user proving age can present “I am over 18” without revealing the date of birth or any other field.

Zero-knowledge proofs go further: the holder can prove a predicate (over 18, in EU, paid subscriber) without presenting the underlying credential at all. The cryptography is mature; the deployment is patchy.

For scraping operators, the implications are:

1. Credential-based authentication discloses only what the credential explicitly carries.
2. Selective disclosure reduces the signal available to behavioural fingerprinting (because fewer fields are revealed).
3. Zero-knowledge presentation is functionally indistinguishable from anonymous access for the verifier.

These features generally favour the user, not the scraper. A scraper that wants to extract user identity from a site’s interaction logs has less to work with when users authenticate via ZKP-presented VCs.

For the broader privacy-preserving discussion, see privacy-preserving scraping.

What scraping operators should do in 2026

Three concrete actions.

First, audit your target sources for VC-gating signals. The signal is usually visible in the authentication flow: a “Sign in with EUDI Wallet” button, a “Connect Wallet” prompt, an OID4VC redirect. If your target sources are adopting these, plan your access path now.

Second, evaluate partnership options. For sources where credential holding is impractical, partnerships with credential holders (research institutions, accredited resellers, licensed aggregators) are the access path. The market for these partnerships is forming now.

Third, consider becoming a credential issuer. For some scraping operators, the role flips: instead of scraping data, you become the issuer of credentials about data quality, freshness, or provenance. Several scraping platforms began issuing data-provenance credentials in 2025-2026.

For the related agent-as-user question, see AI agents as web users.

External references

The W3C DID specification is at w3.org/TR/did-core. The W3C Verifiable Credentials data model is at w3.org/TR/vc-data-model-2.0. The OpenID for Verifiable Credentials specification is at openid.net/specs/openid-4-verifiable-credential-issuance-1_0.html. IETF working drafts on Trust over IP are tracked at datatracker.ietf.org.

Comparison: scraping access methods in a Web4-influenced world

Method	Effectiveness against Web4	Cost	Risk
Residential proxy	Low (geofence only)	Medium	Detection
Account creation	Variable (only Web2)	Low-medium	TOS breach
Browser automation	Moderate (with wallet)	Medium	Detection
Credential acquisition	High (where legitimate)	High setup	Legal alignment
Partnership / licensing	High	Highest setup	Lowest detection risk
Agent-to-agent (DIDComm)	High where supported	Medium	Lowest detection risk

The pattern is clear: the future favours legitimate access paths. Operators who plan for credentialed access will have more options in 2027 than operators who do not.

A worked compliance overlay

Web4 access has compliance implications that traditional scraping does not. A scraper using a research credential is making representations about the user/institution. False or misleading credential use is fraud, not just a TOS issue.

Three controls a Web4-using scraper should implement:

1. Credential governance: written policy on which credentials the operation holds, who is the legitimate holder, what use is in-scope.
2. Audit logging: every credential presentation logged with timestamp, target, and outcome.
3. Revocation handling: when a credential is revoked (issuer or holder action), the operation must stop using it within a defined window.

For the broader compliance posture, see building an ethics-first scraping policy.

FAQ

Is Web4 actually a thing in 2026?
The term is contested but the underlying standards (DIDs, VCs, OID4VC) are real and shipping. Whether you call it Web4 or just “verifiable digital identity”, it is reshaping access.

Can I scrape a VC-gated site without a credential?
Generally no. The credential is the authorisation. Scraping around it would be the equivalent of bypassing a paywall.

Do I need to deploy DIDs in my scraping infrastructure?
Not yet, for most operators. The technology is mature but most sources still use Web2 authentication. Plan for adoption rather than deploy ahead.

What is the relationship between Web3 and Web4?
Web3 focused on decentralized money via blockchain wallets. Web4 focuses on decentralized identity via DIDs and VCs. The technologies overlap (some DID methods use blockchain) but the use cases are distinct.

What is the EU Digital Identity Wallet?
A government-issued, privacy-preserving wallet that EU residents can use to present credentials (driver’s licence, professional qualifications, age) at compatible services. General availability across the EU mid-2026.

Extended decentralized identity analysis

The decentralized identity stack in 2026 consists of four standards. First, decentralized identifiers (DIDs) per W3C DID Core 1.0. Second, verifiable credentials (VCs) per W3C VC Data Model 2.0. Third, presentation exchange per DIF Presentation Exchange 2.0. Fourth, key binding and proof formats (LDP, JWT, SD-JWT, BBS+).

For scrapers DIDs and VCs reshape three things. First, identity-gated content moves from cookie-based session auth to credential-based access. Second, proof of personhood (PoP) credentials become a counter-bot signal. Third, content provenance shifts from platform-attested to creator-attested via signed credentials.

The 2024-2026 wave of EU eIDAS 2.0 and the European Digital Identity Wallet pushed DID and VC adoption from research to production. By 2026 several large platforms accept VC-based proof of age and proof of residence.

Implementation pattern: DID-aware fetcher

import json
from did_resolver import resolve_did
from vc_lib import verify_vc, present

async def fetch_with_vc(url, did, vc_token):
    did_doc = await resolve_did(did)
    presentation = present(vc_token, audience=url)
    headers = {
        "Authorization": f"VC {presentation}",
        "DID": did,
    }
    response = await http.get(url, headers=headers)
    return response

async def verify_inbound_vc(presentation, expected_audience):
    result = verify_vc(presentation)
    if not result.valid:
        return False
    if result.audience != expected_audience:
        return False
    return True

SD-JWT pattern for selective disclosure

Selective Disclosure JWT (SD-JWT) lets a holder reveal only a subset of credential claims to a verifier. Scrapers acting as verifiers can request only the claims they need (for example country of residence) without seeing the full credential. This is privacy-preserving and reduces compliance burden.

def select_disclosures(sd_jwt, claims_to_reveal):
    payload = parse_sd_jwt(sd_jwt)
    revealed = {k: v for k, v in payload.items() if k in claims_to_reveal}
    return reissue_with_disclosures(sd_jwt, revealed)

Comparison: identity models for scrapers

Model	Privacy	Provenance	Replay protection	Scraper effort
Cookie session	Low	None	Per-session	Low
OAuth bearer	Low	Issuer-attested	Per-token	Moderate
API key	Low	Issuer-attested	Per-key	Low
DID plus VC	High (with SD-JWT)	Issuer-attested with crypto proof	Per-presentation	High
zk-credentials	Highest	Crypto-attested	Per-proof	Highest

Web4 vocabulary for scrapers

Web4 is an evolving label that overlaps with decentralized identity, content authenticity (C2PA), and AI-agent-native protocols. For scrapers the practical Web4 surface includes four primitives.

1. C2PA content credentials embedded in media files for provenance.
2. did:web identifiers for site-level identity (a DID hosted at .well-known/did.json).
3. AI agent identity DIDs distinguishing automated traffic from human traffic.
4. Cross-platform trust frameworks built on top of DIF specifications.

Additional FAQ

Are DIDs replacing OAuth?
Not yet. OAuth remains dominant. DIDs are gaining ground for high-assurance use cases.

Do scrapers need their own DIDs?
For agentic browsers acting on behalf of a user, yes increasingly. The DID is how the scraper identifies itself to the target service.

What about C2PA?
C2PA content credentials are useful for scrapers that need to verify media provenance, particularly for AI training data curation.

How does this interact with bot detection?
A scraper presenting a verified personhood VC may be treated as human-equivalent. A scraper presenting an agent VC is identified as an agent and routed accordingly.

The W3C DID core specification in detail

The W3C DID Core 1.0 specification, recommended in July 2022, defines decentralized identifiers as a new type of identifier that is created and managed without reliance on a centralized registry. A DID resolves to a DID Document, which contains the verification methods, service endpoints, and other metadata associated with the identifier.

DIDs come in many methods. did:web is a method that uses a domain name as the basis. did:key uses a cryptographic key directly. did:ion uses the Sidetree protocol on Bitcoin. did:plc uses the Bluesky-developed Public Ledger of Credentials. Each method has different trade-offs in decentralization, performance, and cost.

For scrapers the most relevant methods are did:web (for site-level identity) and did:key (for ephemeral keys). did:web is essentially a DNS-based approach where a DID resolves via fetching the .well-known/did.json file at the domain. This is operationally simple and integrates with existing web infrastructure.

The DID Document specifies one or more verification methods, each of which is a public key and an associated algorithm. Authentication, assertion, key agreement, and capability invocation are different relationships a verification method can have to the DID. A scraper signing a request uses an authentication-relationship key.

Verifiable credential lifecycle

A verifiable credential has three actors: the issuer, the holder, and the verifier. The issuer creates and signs the credential. The holder stores the credential and presents it to verifiers. The verifier checks the credential’s signature, status, and contents.

The lifecycle proceeds in five steps. First, the issuer issues a credential to the holder, typically via OID4VCI. Second, the holder stores the credential in a wallet. Third, a verifier requests a presentation, typically via OID4VP. Fourth, the holder constructs a presentation (which may include selective disclosure) and sends it to the verifier. Fifth, the verifier validates the presentation and acts on it.

For scrapers acting as verifiers the verification step is the operational concern. Verification involves checking the cryptographic signature against the issuer’s verification method, checking the issuer against a trust list, checking the credential’s expiration, and checking the credential’s revocation status.

The trust list is the most operationally complex piece. The verifier must decide which issuers it trusts. Some trust lists are centralised (a government list of accredited issuers). Others are federated (a mutual recognition agreement among issuers). The decision is policy-driven and context-dependent.

Selective disclosure and zero-knowledge proofs

Selective disclosure is the ability to reveal a subset of credential claims without revealing the rest. SD-JWT is a 2024-stable format that achieves selective disclosure through hash-based blinding of individual claims.

Zero-knowledge proofs go further. A ZKP-based credential allows a holder to prove a statement about the credential (for example over 18) without revealing any specific claim. The holder constructs a proof that the verifier can check without seeing the underlying data.

For scrapers ZKP is operationally heavier but privacy-preserving. The 2026 pattern is to use SD-JWT for most cases (good privacy, modest compute) and ZKP for high-sensitivity cases (best privacy, higher compute).

The ZKP toolkit in 2026 includes AnonCreds (the Hyperledger flagship), BBS+ signatures (for ZK on standard VCs), and several research-grade systems. Production deployments are growing but remain a minority.

C2PA and content provenance

C2PA (Coalition for Content Provenance and Authenticity) is a parallel standard focused on media provenance rather than identity. A C2PA manifest, embedded in an image or video file, describes the file’s origin and edit history through signed assertions.

For scrapers harvesting media files the C2PA manifest is a useful provenance signal. A scraper feeding AI training data can use C2PA to filter out content that has been flagged by the creator as not for AI training. A scraper feeding a news aggregator can use C2PA to verify the file’s claimed source.

The C2PA ecosystem grew rapidly in 2024-2026. Major camera manufacturers ship C2PA-capable hardware. Major image editors embed C2PA manifests on save. Major social platforms display C2PA badges. The pattern is similar to TLS adoption in the early 2010s.

A 2026 best practice for scrapers is to read and preserve C2PA manifests at ingest. The manifest itself is small. Preservation enables downstream consumers to make their own provenance decisions.

Common pitfalls when adopting DIDs in a scraping pipeline

Three failure modes consistently bite teams that introduce DID-based identity into existing scraping infrastructure.

The first pitfall is choosing the wrong DID method for the use case. Teams default to did:web because it is simple, but did:web inherits all the trust limitations of DNS and TLS, including registrar takeover and CA mis-issuance. For high-assurance use cases like agent identity that crosses regulatory boundaries, did:key or did:ion provide stronger guarantees at the cost of more complex resolution. Map the trust requirement to the method before writing code.

The second pitfall is not rotating verification method keys. A DID is durable, but the keys associated with it are not. Most DID methods support adding new verification methods and retiring old ones via DID Document updates. A scraping operation that uses the same key for years exposes itself to key compromise with no recovery path. Build key rotation into the operational runbook from day one.

The third pitfall is conflating proof-of-personhood with proof-of-uniqueness. A personhood credential proves the holder is human; it does not prove the holder is unique to your platform. Sybil resistance requires additional signals like nullifier sets or federated uniqueness checks, which most off-the-shelf personhood credentials do not provide.

Next steps

The fastest first step is to audit your top sources for any wallet/credential signals in the authentication flow. If you find any, the time to plan your access path is now, before VC-gating becomes the default. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the verifiable credentials guide.

This guide is informational, not engineering or legal advice.

Scraping with vision models (GPT-4o, Claude 3.5, Gemini Pro)

Vision model scraping in 2026 has crossed the line from cool demo to legitimate production tool. The major LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro and Flash) all accept image input now, all do strong OCR and layout reasoning, and all return strict JSON when asked. For scraping work specifically, that means you can take a screenshot of any web page and extract structured data without writing a single CSS selector.

This guide covers when vision-model scraping wins, how to use each major model effectively, the cost picture, and the production patterns that keep latency and bills under control. Working code throughout.

Why vision model scraping matters

Three problems vision models solve that text-based extraction cannot.

First, sites that render content with images. PDF embeds, infographics, sites that ship product information as images for SEO reasons. Text scrapers see nothing. Vision models read the image.

Second, sites with bot defenses that mangle HTML. Cloudflare’s HTML scrambling, sites that randomize class names per request, sites that ship CSS sprites instead of text. Vision models bypass all of it because they read the rendered pixels.

Third, layout-driven extraction. When the same field name appears in two places (header price and main price), text extraction guesses. Vision extraction sees which one is bigger, more prominent, in the right region.

How vision-model scraping works

The pattern is consistent across all three providers:

1. Render the target page in a headless browser
2. Take a screenshot (full page or viewport)
3. Send the screenshot plus an extraction prompt and schema to the vision model
4. Validate and store the result

The browser is just a screenshot generator. No selectors. No DOM traversal. The model does all the layout reasoning.

When to skip vision entirely

Vision extraction is the wrong tool when the page is static text in a stable HTML structure. The cost of vision tokens dwarfs text tokens, and accuracy is no better. Reach for vision only when text extraction fails or hits one of the three winning conditions described later.

Implementation with GPT-4o

import asyncio
import base64
from openai import AsyncOpenAI
from playwright.async_api import async_playwright

client = AsyncOpenAI()

PRODUCT_SCHEMA = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "price": {"type": "number"},
        "currency": {"type": "string"},
        "in_stock": {"type": "boolean"},
        "rating": {"type": ["number", "null"]},
        "review_count": {"type": ["integer", "null"]},
    },
    "required": ["title", "price", "currency", "in_stock", "rating", "review_count"],
    "additionalProperties": False,
}

async def screenshot_url(url: str) -> bytes:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page(viewport={"width": 1280, "height": 1024})
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=True)
        await browser.close()
        return png

async def extract_with_gpt4o(png_bytes: bytes) -> dict:
    b64 = base64.b64encode(png_bytes).decode()
    resp = await client.chat.completions.create(
        model="gpt-4o",
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "product", "schema": PRODUCT_SCHEMA, "strict": True},
        },
        messages=[
            {"role": "system", "content": "Extract product data from this screenshot."},
            {"role": "user", "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "high"}}
            ]},
        ],
    )
    import json
    return json.loads(resp.choices[0].message.content)

async def main():
    png = await screenshot_url("https://www.lazada.sg/products/example.html")
    print(await extract_with_gpt4o(png))

asyncio.run(main())

detail: "high" matters. The default auto downsamples large images and loses fine text. For product pages with small price labels, always use high.

Implementation with Claude 3.5 Sonnet

from anthropic import AsyncAnthropic
import base64

client = AsyncAnthropic()

async def extract_with_claude(png_bytes: bytes) -> dict:
    b64 = base64.b64encode(png_bytes).decode()
    resp = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=2000,
        tools=[{
            "name": "save_product",
            "description": "Save the extracted product",
            "input_schema": PRODUCT_SCHEMA,
        }],
        tool_choice={"type": "tool", "name": "save_product"},
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
                {"type": "text", "text": "Extract the product data from this screenshot."},
            ],
        }],
    )
    return resp.content[0].input

Claude does not have a detail: high flag because it always processes at full resolution. The trade-off is higher per-image cost than GPT-4o on large screenshots.

Implementation with Gemini 1.5 Pro

import google.generativeai as genai
import os
import json

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

model = genai.GenerativeModel(
    "gemini-1.5-pro",
    generation_config={"response_mime_type": "application/json", "response_schema": PRODUCT_SCHEMA},
)

async def extract_with_gemini(png_bytes: bytes) -> dict:
    response = await model.generate_content_async([
        "Extract the product data from this screenshot.",
        {"mime_type": "image/png", "data": png_bytes},
    ])
    return json.loads(response.text)

Gemini’s huge context window (2M tokens) and dedicated responseSchema parameter make it natural for vision extraction at scale. Cost per image is competitive with the others.

Side-by-side comparison

We ran 100 product page screenshots from Lazada, Amazon, and Best Buy through each model.

Metric	GPT-4o	Claude Sonnet 4.5	Gemini 1.5 Pro	Gemini 1.5 Flash
Cost per image	$0.027	$0.041	$0.024	$0.0035
Latency p50	2.4 s	3.1 s	2.0 s	1.2 s
Extraction accuracy	96%	97%	95%	91%
Best at	UI element recognition	Text-heavy pages	Long pages, multilingual	Cost-sensitive volume
Worst at	Very small text	Image-heavy without text	OCR on stylized fonts	Complex layouts

For most production scraping, GPT-4o or Claude Sonnet are the right pick. Gemini Flash is the value play when cost dominates over the last 5 percent of accuracy.

Comparing model behavior on the same screenshot

The same Lazada product page screenshot, three models, same JSON Schema:

GPT-4o output: clean extraction, occasional off-by-one on review counts when the displayed number includes a comma in non-US locale.

Claude Sonnet 4.5 output: most reliable on text-heavy pages, occasionally over-conservative on in_stock (returns false if any “out of stock” appears anywhere on the page, including in related products).

Gemini 1.5 Pro output: strongest at multilingual content, occasional layout confusion when the price is in a sidebar widget rather than the main panel.

Practical implication: if you have multilingual targets, lean Gemini. If you have text-heavy English ecommerce, lean Claude. If you have a mix of languages and want a balanced default, GPT-4o.

When vision wins over text extraction

Vision wins when one of three conditions holds:

1. The HTML is intentionally obfuscated (Cloudflare scrambling, randomized classes)
2. Critical content is rendered as image (price tags as PNGs, infographics)
3. Layout matters for disambiguation (multiple prices on the same page)

Vision loses when the HTML is clean and well-structured. Text extraction is 5-10x cheaper and just as accurate on those targets.

For more on text extraction patterns, see LLM extraction patterns: structured output from messy HTML.

Real failure modes

A few specific failure patterns observed in production:

The model reads a strikethrough price (the “old” price) instead of the current price. Mitigation: explicit instruction “extract the current price, not the strikethrough or comparison price.”

The model extracts a related product’s price when the main product price is hidden behind a button. Mitigation: instruct “extract only the main product on this page” and add a sentinel check (e.g. the title must contain a known keyword).

The model treats currency-only labels (just “$”) as full prices. Mitigation: validate that price > 0 and reject extractions where price is implausibly small.

The model fails on sites that render prices with web fonts containing custom glyphs (some bot defenses ship a font that maps numbers to other glyphs). Mitigation: a hybrid extraction with HTML, where the HTML still contains the real character codes.

Hybrid extraction: vision for hard fields only

The cost-optimal pattern for many sites is hybrid. Use cheap text extraction for the easy fields (title, description) and reserve vision for the fields that fail text extraction (price hidden behind dynamic rendering, stock indicator embedded in an SVG icon).

async def hybrid_extract(html: str, png: bytes) -> dict:
    text_result = await extract_text_with_4o_mini(html)
    if text_result.get("price") is None or text_result.get("currency") is None:
        vision_result = await extract_with_gpt4o(png)
        text_result["price"] = vision_result.get("price")
        text_result["currency"] = vision_result.get("currency")
    return text_result

This pattern saves significant cost over pure vision while catching the cases where text fails.

Full-page vs viewport screenshots

Full-page screenshots capture everything but produce huge PNGs that cost more to process and confuse models with too much content.

Viewport screenshots capture only the visible region but may miss below-the-fold content (reviews, related products).

The pragmatic default: viewport screenshot for the primary entity, scroll-and-snap for any below-the-fold field you specifically need.

async def screenshot_with_scroll(url: str, scroll_targets=None) -> list[bytes]:
    screenshots = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page(viewport={"width": 1280, "height": 1024})
        await page.goto(url, wait_until="networkidle")
        screenshots.append(await page.screenshot())

        if scroll_targets:
            for selector in scroll_targets:
                el = await page.locator(selector).first
                await el.scroll_into_view_if_needed()
                screenshots.append(await page.screenshot())

        await browser.close()
    return screenshots

Handling multiple entities per page

For pages with many entities (a search results page, a category listing), pass the screenshot with an array schema.

LISTING_SCHEMA = {
    "type": "object",
    "properties": {
        "items": {
            "type": "array",
            "items": PRODUCT_SCHEMA,
            "minItems": 0,
            "maxItems": 50,
        },
    },
    "required": ["items"],
    "additionalProperties": False,
}

async def extract_listing(png_bytes: bytes) -> dict:
    # use GPT-4o or Claude with the listing schema
    ...

Vision models handle arrays well. The cap on maxItems prevents runaway hallucination on confused inputs.

Adding proxies

Proxies live in your screenshot step, not the vision call. Configure the headless browser:

async def screenshot_with_proxy(url: str, proxy: str) -> bytes:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy={"server": proxy},
        )
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=True)
        await browser.close()
        return png

For ASEAN ecommerce specifically, Singapore mobile proxy carries clean carrier IPs that survive the strongest bot defenses. Pair with full-page screenshots for product listings on Lazada and Shopee.

Production patterns

Three patterns separate hobby vision scraping from production.

First, downsample appropriately. Vision models have an effective resolution they actually use. For GPT-4o, anything above 2048×2048 wastes tokens. Resize before sending.

from PIL import Image
import io

def resize_for_model(png_bytes: bytes, max_dim: int = 2048) -> bytes:
    img = Image.open(io.BytesIO(png_bytes))
    if max(img.size) > max_dim:
        ratio = max_dim / max(img.size)
        new_size = (int(img.size[0] * ratio), int(img.size[1] * ratio))
        img = img.resize(new_size, Image.LANCZOS)
    out = io.BytesIO()
    img.save(out, format="PNG", optimize=True)
    return out.getvalue()

Second, cache by image hash. Identical screenshots produce identical extractions. SHA-256 the PNG, key your cache on it.

Third, run two models in parallel for high-stakes data. GPT-4o and Claude Sonnet, take the agreement. Catches the rare hallucination at 2x cost.

Memory and disk considerations

Full-page screenshots can be large. A 4000-pixel-tall page at 2x DPR is roughly 8 MB as PNG, 600 KB as JPEG quality 80. For high-volume pipelines:

Compress to JPEG before sending. JPEG quality 85 is visually indistinguishable from PNG for typical web pages and cuts payload size by 90 percent.

Stream screenshots through a temporary buffer rather than holding them all in memory. A 100-worker pool with full-page PNGs can OOM a 16 GB host quickly.

Cache screenshots locally for replay. The screenshot is the source of truth for an extraction run; saving it lets you re-extract with a different model later without re-fetching.

def to_jpeg(png_bytes: bytes, quality: int = 85) -> bytes:
    img = Image.open(io.BytesIO(png_bytes)).convert("RGB")
    out = io.BytesIO()
    img.save(out, format="JPEG", quality=quality, optimize=True)
    return out.getvalue()

Real benchmarks across sites

100 product pages each, full-page screenshot, GPT-4o:

Site	Success rate	Avg cost per page
Lazada SG	98%	$0.029
Shopee SG	96%	$0.031
Amazon US	99%	$0.025
Walmart	97%	$0.027
Best Buy	95%	$0.030
Tokopedia	94%	$0.034

Add browser cost (Browserbase or self-hosted) at $0.002-$0.005 per page. Total per 1000 pages: $30-$40 with vision, vs $5-$10 with text-only extraction. Vision wins on accuracy and resilience; text wins on cost.

Token cost mechanics

Vision tokens are computed differently from text tokens. The mechanics matter for cost prediction.

GPT-4o computes vision tokens by splitting the image into 512×512 tiles, charging 170 tokens per tile, plus a fixed 85 tokens for the low-res view. A 1024×1024 image is 4 tiles plus the base = 765 tokens. A 1600×1024 image is 6 tiles plus base = 1105 tokens. detail: low uses only the 85 base tokens at the cost of accuracy.

Claude charges roughly 1.15 tokens per pixel up to a max, with an effective image cost around 1500 to 4000 tokens depending on size.

Gemini charges a flat 258 tokens per image regardless of size, which makes it dramatically cheaper for large screenshots.

The implication: if your scraper sends 5 MB full-page screenshots through GPT-4o, you are paying for 4 to 8 thousand vision tokens per image. Resize to 1280×800 and you cut that to under 1500 tokens with minimal accuracy loss.

Region cropping for high-stakes fields

For mission-critical fields (transaction prices, contract terms, regulatory disclosures), crop the image to the field region and send only the crop. This pushes accuracy from roughly 96 percent on full pages to over 99 percent on focused crops.

def crop_to_region(png_bytes: bytes, x: int, y: int, w: int, h: int) -> bytes:
    img = Image.open(io.BytesIO(png_bytes))
    cropped = img.crop((x, y, x + w, y + h))
    out = io.BytesIO()
    cropped.save(out, format="PNG")
    return out.getvalue()

# Use selectors or LLM observation to find the region first, then crop and re-extract

The two-step approach (full page first, then crop and re-extract critical fields) is the right pattern when accuracy matters more than cost.

Multimodal pipelines: combining vision and text

The strongest extraction pipelines combine HTML and screenshot in the same LLM call. The model uses the HTML as ground truth for structured fields and the screenshot for visual context.

async def multimodal_extract(html: str, png: bytes, schema: dict) -> dict:
    b64 = base64.b64encode(png).decode()
    return await client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_schema", "json_schema": {"name": "x", "schema": schema, "strict": True}},
        messages=[
            {"role": "system", "content": "Use the HTML for structured data and the screenshot for layout context."},
            {"role": "user", "content": [
                {"type": "text", "text": f"HTML:\n{html[:100000]}"},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "high"}},
            ]},
        ],
    )

This combination outperforms either alone on roughly 60 percent of pages we tested. Cost is higher than text-only by the vision token premium.

Vision-based crawling for shape discovery

A clever pattern uses vision to discover the shape of an unknown site. Take a few screenshots, ask the model to describe the layout in structured form (“this site has a header, a search bar, a product grid with 3 columns”), then use that description to build a Playwright scraper.

This bootstraps a deterministic scraper from a few vision calls, paying once for discovery instead of every scrape.

Common production gotchas

A few patterns that bite teams using vision models.

The model occasionally hallucinates fields that look plausible but are not on the page. Always validate extracted data against the source HTML or a deterministic check.

Different vendors handle base64 differently. OpenAI accepts a data: URL. Anthropic accepts the raw base64 with media_type. Gemini accepts the bytes directly with mime_type. Wrappers help but the bare APIs differ.

Image preprocessing libraries (PIL, OpenCV) introduce subtle artifacts that can change OCR output. Save the raw screenshot and the preprocessed version both, and prefer the raw if the preprocessing is not strictly necessary.

Vision token cost varies by model in non-obvious ways. Always benchmark on your actual screenshots; do not extrapolate from documented pricing alone.

Frequently asked questions

How do I evaluate which vision model is best for my specific target?
Hand-label 50 representative pages, run each model with the same schema, score against the gold set. Cost is roughly $5 per evaluation run; the data drives a multi-month decision.

Can vision models read tiny text like product SKUs?
Up to a point. GPT-4o and Claude Sonnet handle text down to about 8px reliably at high detail. For smaller text, crop to the relevant region before sending.

What about charts and tables?
All three models handle structured tables in screenshots well. Charts are mixed; line and bar charts work, complex multi-series charts often fail. Pass the underlying data if you can.

How do I handle international character sets?
Vision OCR for Chinese, Japanese, Korean, Thai, Arabic is strong on Gemini Pro and Claude Sonnet. GPT-4o is good but slightly behind on uncommon scripts. Test on your specific target.

Can I use vision to fill out forms?
Indirectly. Vision models can identify form fields and instruct your scraper. For actual form filling, browser automation (Playwright, browser-use) is the right tool.

What about cost-effective open-source vision models?
Llama 3.2 90B Vision and Qwen 2.5 VL 72B are the strongest open-source vision models in early 2026. Self-hosted on a 4xA100 machine, cost per image is around $0.001 if you have throughput to amortize. Below the major closed-source models on quality, especially on small text.

Can vision models extract from videos?
Indirectly. Sample frames at 1 fps, send each frame to the vision model, aggregate the extractions. For long videos, sampling every 5 seconds and aggregating works well.

How do vision models handle CAPTCHAs?
They will solve simple image CAPTCHAs (find traffic lights, identify text in a distorted image) reasonably well, but the major LLM providers refuse the obvious “solve this CAPTCHA” prompts. Phrasing matters. Solver services remain more reliable for production CAPTCHA workflows.

Can I extract from rendered PDFs as images?
Yes. Convert the PDF to images with pdf2image or similar, then run vision extraction on each page. For text-heavy PDFs, the modern LLM APIs accept PDFs directly which is faster and cheaper.

Is there a future where vision replaces selector-based scraping entirely?
For sites that change layout faster than engineers can update selectors, vision is already winning. For high-volume known-shape sites, the cost gap keeps selector-based extraction relevant. The real future is hybrid: vision for discovery and resilience, selectors for the bulk.

Can I run vision extraction on edge devices?
The smaller open-source vision models (Qwen 2 VL 2B, Llava-OneVision 7B) run on consumer GPUs. Quality is well below the major models but adequate for known-shape extraction.

For more on the broader AI scraping landscape, browse the AI modern scraping category.

The agentic browser revolution: Claude, OpenAI Operator, Stagehand

Agentic browser 2026 is no longer a research curiosity. The eighteen months between Anthropic’s Computer Use launch in October 2024 and the May 2026 state of the art produced a fundamentally different stack for browser automation. Claude Computer Use, OpenAI Operator, Stagehand from Browserbase, browser-use the open-source library, and the Browser MCP servers all matured into production-grade tools. For scraping operators, the change is structural: brittle CSS selectors give way to vision-grounded, intent-driven instructions; multi-step workflows that took weeks to build now take an afternoon; and the cost economics shifted from “engineering hours per scraper” to “agent tokens per task.” This guide walks through what each agentic browser actually does, the head-to-head comparison, the migration patterns from selector-based to agent-based scraping, the failure modes that still bite, and where the technology is heading.

The audience is the data engineer or scraping platform owner who needs to decide whether and how to adopt agentic browsing in 2026.

What an agentic browser actually is

An agentic browser is a system in which an LLM (typically vision-capable) drives a browser by interpreting user intent, observing the rendered page, and issuing actions (click, type, scroll, navigate). The “agentic” part is that the model decides what to do next based on what it sees, rather than executing a hard-coded script.

The minimum architecture has three components: a browser runtime (Chromium, Firefox, or a managed service), an action interface (the API by which the model issues clicks and keystrokes), and the model itself with vision capability.

The four major implementations in 2026:

Implementation	Vendor	Browser runtime	Model	Hosted?
Claude Computer Use	Anthropic	Local or remote VM	Claude 4.7 (vision)	Self-host
OpenAI Operator	OpenAI	OpenAI-managed	GPT-4o (vision) / o3	Hosted
Stagehand	Browserbase	Browserbase-managed Chromium	Pluggable (Claude, GPT, Gemini)	Hosted
browser-use	Open source	Local Chromium via Playwright	Pluggable	Self-host

Each takes a different position on hosted versus self-hosted, on the level of abstraction over the browser, and on which model providers it supports.

For the broader MCP integration story, see MCP for data engineers. For the AI-as-web-user concept, see AI agents as web users.

Claude Computer Use: the OS-level abstraction

Anthropic’s Computer Use is the lowest-level abstraction. The agent is given a sandboxed virtual machine with a screen, mouse, and keyboard, and it operates by taking screenshots, reasoning about pixel coordinates, and issuing mouse/keyboard events.

Strengths:
– Universal: anything a human can do with a desktop, the agent can do.
– Not browser-specific: works on installed apps, terminal, file manager.
– Vision-grounded: the model sees what the user sees.
– Self-hosted by default: full control over data and access.

Weaknesses:
– Higher latency: screenshot, reason, act, screenshot.
– Higher token cost: each step burns vision tokens.
– More fragile to layout shifts: pixel coordinates drift on responsive UIs.
– Operational overhead: you run the VM.

Best for: complex multi-app workflows, desktop automation, situations where browser isolation matters, controlled internal use.

A minimal Claude Computer Use loop in Python:

from anthropic import Anthropic
import base64

client = Anthropic()
def screenshot_b64():
    return base64.b64encode(open("screen.png", "rb").read()).decode()

response = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=2048,
    tools=[{"type": "computer_20250124", "name": "computer",
            "display_width_px": 1280, "display_height_px": 800}],
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Open the website and list all products."},
            {"type": "image", "source": {"type": "base64",
                                         "media_type": "image/png",
                                         "data": screenshot_b64()}},
        ],
    }],
)

The model returns tool-use blocks with action types (click, type, key, screenshot). Your loop executes them in the VM, takes a new screenshot, and calls the model again.

OpenAI Operator: the hosted browsing agent

OpenAI Operator launched in January 2025 as a hosted browsing agent built on a fine-tuned GPT-4o variant called CUA (Computer-Using Agent). Operator runs in OpenAI infrastructure and exposes an API for users to delegate browser tasks.

Strengths:
– Hosted: no infrastructure ownership.
– Tight integration with ChatGPT consumer surface.
– Rapid iteration: OpenAI continuously improves the underlying model.
– Cleanly framed for end-user delegation use cases.

Weaknesses:
– Hosted-only: no self-host option.
– Less granular control: the abstraction is “task” not “click”.
– Data leaves your environment.
– US-Europe regulatory exposure.

Best for: end-user productivity tasks, ChatGPT-integrated experiences, low-volume high-value workflows where the hosted convenience justifies the data exposure.

The Operator API call pattern:

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="computer-use-preview",
    tools=[{"type": "computer_use_preview",
            "display_width": 1280, "display_height": 800,
            "environment": "browser"}],
    input=[{"role": "user", "content": "Find the cheapest direct flight "
                                       "from SIN to TYO next Monday."}],
)

The Operator returns a sequence of actions; OpenAI executes them in its hosted browser; you receive structured progress events.

Stagehand: the developer-first abstraction

Stagehand from Browserbase is a TypeScript-first library that sits one level above the raw browser. It provides three high-level primitives: act (do something), extract (pull structured data), and observe (find an element). Each is backed by an LLM under the hood.

Strengths:
– Developer ergonomics: writing scrapers feels like writing tests.
– Pluggable model: choose Claude, GPT, or Gemini per call.
– Browserbase-hosted: managed Chromium with anti-bot built in.
– Strong observability: every action logged.
– Good TypeScript ergonomics; Python SDK matured in 2025.

Weaknesses:
– Hosted browser by default (Browserbase); local mode possible but less polished.
– Cost model: pay per browser session plus per LLM call.
– Less universal than OS-level approaches.

Best for: production scraping pipelines, situations where developer velocity and reliability matter, teams that want a managed browser without giving up control.

A minimal Stagehand session:

import { Stagehand } from "@browserbasehq/stagehand";

const stagehand = new Stagehand({ env: "BROWSERBASE" });
await stagehand.init();
await stagehand.page.goto("https://example.com/products");
await stagehand.act({ action: "filter products by category 'shoes'" });
const data = await stagehand.extract({
  instruction: "extract all product names and prices",
  schema: z.object({
    products: z.array(z.object({ name: z.string(), price: z.string() })),
  }),
});

Three primitives, structured output, no selector engineering.

For the head-to-head with Playwright, see Stagehand vs Playwright for AI-driven scraping.

browser-use: the open-source contender

browser-use is an open-source Python library that pairs Playwright with vision-capable LLMs. It launched in late 2024 and matured rapidly through 2025. By 2026 it is the most popular self-hosted agentic browsing library.

Strengths:
– Fully open source; MIT licence.
– Self-hosted; data and browser stay in your environment.
– Pluggable model: any vision-capable LLM via langchain-style adapters.
– Active community; rapid iteration.
– Cheaper at scale than hosted alternatives.

Weaknesses:
– More setup: you run the browser and the model.
– Less polished than commercial offerings.
– Documentation evolving.
– No built-in anti-bot infrastructure.

Best for: cost-sensitive teams, regulated environments, situations where the data must not leave, teams comfortable with open-source operational ownership.

For the broader self-hosted infrastructure story, see self-hosted proxy infrastructure.

Head-to-head comparison

Dimension	Computer Use	Operator	Stagehand	browser-use
Hosted?	Self-host	Hosted	Hosted (default)	Self-host
Browser runtime	VM you run	OpenAI-managed	Browserbase	Playwright local
Model	Claude only	OpenAI only (CUA)	Pluggable	Pluggable
Abstraction level	Pixel/coordinate	Task	Act/extract/observe	Action
Best language	Python	Python/TS	TypeScript (Python catching up)	Python
Anti-bot built in	No	Partial	Yes (Browserbase)	No
Cost model	Token + VM	Per session	Session + tokens	Token only
Suitable for production scraping	Moderate	Moderate	High	High
Suitable for desktop automation	High	Low	Low	Low
Suitable for end-user delegation	Low	High	Moderate	Low

Migration pattern: from selector-based to agentic

Most scraping teams in 2026 are migrating from selector-based pipelines (Scrapy, Playwright with explicit selectors) to agentic browsers. The migration pattern that works:

1. Identify the most-fragile scrapers (highest selector breakage rate, highest engineering time per maintenance).
2. Pick one as the migration pilot.
3. Build the agentic version side-by-side; do not retire the selector version.
4. Run both for two weeks; compare outputs, costs, latency, success rate.
5. If the agentic version wins on net (success rate matters more than cost in 2026), retire the selector version.
6. Repeat for the next-most-fragile scraper.

The pattern works because agentic browsers are dramatically more resilient to layout changes but cost more per page. The economics flip in favour of agentic when maintenance cost dominates.

Pipeline characteristic	Stay selector	Migrate to agent
Stable site, simple structure	Stay	–
Frequent layout changes	–	Migrate
High volume, low value per page	Stay	–
Low volume, high value per page	–	Migrate
Complex multi-step workflow	–	Migrate
Single-step extraction	Stay	–
Anti-bot heavy	Hybrid	Hybrid (use Stagehand or Browserbase)

Failure modes that still bite

Three failure modes show up consistently in 2026 production deployments.

The first is non-determinism. The same prompt against the same page can produce different action sequences. For workflows where audit and reproducibility matter (financial, compliance), this is a problem. The mitigation: use temperature zero, snapshot intermediate states, and validate outputs against schemas.

The second is hallucination. Vision-capable LLMs occasionally describe elements that are not present. They click on coordinates that do not contain a button. The mitigation: use the act-then-verify pattern, where every action is followed by an observation that confirms the expected state change.

The third is anti-bot detection. Vision-grounded clicks at pixel coordinates produce a behavioural signature different from human mouse movements. Bot management systems trained on human behaviour increasingly flag agentic browsing. The mitigation: use anti-bot-aware browsers (Browserbase, Bright Data Scraping Browser) or implement realistic mouse movement simulation.

For the broader anti-bot question, see DataDome vs PerimeterX vs Akamai.

Cost economics in 2026

A rough cost benchmark for a 100-step scraping task across the four implementations:

Implementation	Cost per task	Latency	Success rate
Claude Computer Use	USD 0.40-0.80	60-120s	85-92%
OpenAI Operator	USD 0.50-1.00	60-90s	88-94%
Stagehand	USD 0.30-0.60	30-60s	90-95%
browser-use	USD 0.15-0.40	30-90s	85-93%

The numbers shift weekly as model pricing changes. The pattern is stable: hosted offerings cost more but reduce operational overhead; self-hosted offerings cost less but require ownership. Stagehand sits at the favourable middle for production scraping.

For the deeper benchmark, see AI scraping cost benchmark.

External references

The Anthropic Computer Use documentation is at docs.anthropic.com/en/docs/agents-and-tools/computer-use. The OpenAI Operator launch announcement and developer documentation is at openai.com/index/introducing-operator. Stagehand’s open-source repository is at github.com/browserbase/stagehand. browser-use is at github.com/browser-use/browser-use.

Where the technology is heading

Three trends shape the 2026-2027 trajectory.

First, vision models are getting cheaper and faster. The cost per agentic action has fallen 70 percent year-over-year for the past two years, and that trend continues. Workflows that are uneconomic today become economic in six months.

Second, the abstraction is moving up. Stagehand-style act/extract/observe is replacing pixel-level coordinate reasoning for most use cases. Pixel-level work persists for edge cases (canvas-based UIs, custom desktop apps).

Third, anti-bot detection is adapting. The arms race between agentic browsers and bot management is the same as the proxy versus bot management arms race that ran for the past decade. Expect 2027 to bring purpose-built agent management products from DataDome, PerimeterX, Akamai, and Cloudflare.

For the longer-arc view of how AI agents become indistinguishable from human users, see AI agents as web users.

FAQ

Are agentic browsers replacing Playwright?
Not yet. Playwright remains the workhorse for stable, high-volume, simple scrapes. Agentic browsers win where layout changes are frequent or workflows are complex.

Which is best for production scraping?
Stagehand or browser-use. Stagehand if you want managed; browser-use if you want self-hosted.

Can I use one for desktop automation?
Claude Computer Use is the only one designed for that. Operator and Stagehand are browser-only.

How does anti-bot detection see agentic browsers?
Increasingly visible. Use anti-bot-aware browser providers or invest in behavioural realism.

What is the right stack for a 2026 greenfield scraping pipeline?
Stagehand on Browserbase for managed; browser-use on Playwright with residential proxies for self-hosted. Both with structured-output schemas and verification loops.

Extended agentic browser architecture analysis

The agentic browser stack in 2026 consists of four layers. First, the underlying browser engine (Chromium, Firefox, WebKit). Second, the automation protocol (CDP, WebDriver Classic, WebDriver BiDi). Third, the agent orchestration layer (a planner that decomposes tasks into actions). Fourth, the model that proposes the next action from the current page state.

The 2024-2026 wave of agentic browsers (Anthropic’s computer use, OpenAI’s operator, Browserbase, Stagehand, AgentQL) converged on three patterns. First, page state is captured as a combination of accessibility tree plus a screenshot. Second, actions are issued as a small typed vocabulary (click, type, scroll, wait, navigate). Third, the agent runs in a loop until task completion or a step budget is exhausted.

The accessibility-tree-plus-screenshot pattern beat pure screenshot grounding because the tree gives precise element identifiers while the screenshot gives layout context. The combination reduces hallucinated coordinates.

Production agentic browser pattern

from playwright.async_api import async_playwright

async def run_agent(task, max_steps=20):
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto("https://example.com")

        for step in range(max_steps):
            snapshot = await page.accessibility.snapshot()
            screenshot = await page.screenshot()
            action = await model_propose_action(task, snapshot, screenshot, step)
            if action["type"] == "done":
                return action["result"]
            await execute_action(page, action)
        return {"status": "step_budget_exhausted"}

async def execute_action(page, action):
    if action["type"] == "click":
        await page.click(action["selector"])
    elif action["type"] == "type":
        await page.fill(action["selector"], action["value"])
    elif action["type"] == "navigate":
        await page.goto(action["url"])
    elif action["type"] == "scroll":
        await page.evaluate(f"window.scrollBy(0, {action['delta']})")
    elif action["type"] == "wait":
        await page.wait_for_timeout(action["ms"])

Step-budget and termination patterns

A robust agentic browser sets four budgets per task.

1. Step budget (typically 20-50 actions).
2. Wall-clock budget (typically 5-15 minutes).
3. Token budget for the model (typically 100k tokens per task).
4. Cost budget in dollars.

Termination triggers when any budget is exhausted, when the model returns a done action, when an unrecoverable error occurs, or when a safety check fires.

Detection and counter-detection

Bot management vendors (Cloudflare Bot Management, Akamai Bot Manager, DataDome, PerimeterX) ship 2026 detectors that look for the following agent signals.

• Headless Chromium fingerprints (missing navigator.webdriver, missing plugins, missing window.chrome).
• CDP-specific runtime traces.
• Mouse and keyboard event timing distributions that lack human jitter.
• Action sequences that match common agent libraries.

Counter-detection in 2026 typically includes residential proxies, fingerprint patching, and human-jitter event timing. The arms race continues.

Comparison: agentic browser frameworks 2026

Framework	Underlying engine	Automation protocol	Best for
Playwright plus custom agent	Chromium, Firefox, WebKit	CDP, BiDi	Custom builds
Browserbase	Chromium	CDP	Hosted scaling
Stagehand	Chromium	CDP	LLM-native abstractions
AgentQL	Chromium	CDP	Schema-driven extraction
Anthropic computer use	OS-level	Pixel grounding	Cross-app workflows

Additional FAQ

How do agentic browsers handle CAPTCHAs?
They typically pause and request human intervention. Some integrate solving services. Production systems should treat CAPTCHA as a termination signal rather than a step to bypass.

What about session state?
Persist cookies and storage in a context per task. Reuse contexts only across related tasks for the same user.

How do I evaluate agentic browser performance?
Build a fixed test suite of tasks (web navigation, form filling, data extraction). Measure success rate, mean steps, mean wall-clock, mean cost. Track trend over model updates.

Is the agentic browser pattern replacing classical scraping?
For one-off tasks yes. For high-volume structured extraction classical scraping remains cheaper and more reliable.

Common pitfalls in production agentic browser deployments

Five failure modes recur across teams that move from pilot to production with agentic browsers in 2026.

The first pitfall is unbounded step budgets in production. A pilot script with no step ceiling will eventually encounter a page where the agent loops on a recoverable error and burns through hundreds of dollars in vision tokens before the wall-clock budget catches it. Always set a step budget, a wall-clock budget, and a hard cost ceiling per task, and alert when any task hits 50 percent of any budget.

The second pitfall is treating the agent’s natural-language reasoning as audit-grade output. The agent’s chain of thought may say it clicked the correct button when it actually clicked an adjacent element. Capture the post-action accessibility snapshot and verify the expected state change with deterministic checks, not the agent’s self-report.

The third pitfall is sharing browser contexts across tasks. Agents that reuse a single Chromium context accumulate cookies, storage, and history that leak between unrelated tasks. The leak shows up as mysterious cross-task contamination weeks into production. Use one context per task by default; share only when the workflow explicitly requires session continuity.

The fourth pitfall is failing to record screenshots and DOM snapshots for every action. When an agentic scraper produces wrong output, the only way to debug is to replay the visual state the agent saw at decision time. Storage is cheap; debugging without snapshots is impossible. Record everything for at least 30 days.

The fifth pitfall is ignoring model version drift. The same prompt against the same page can produce different action sequences when the underlying model is updated by the provider. Pin the model version explicitly, validate on a regression suite before adopting a new version, and never let a hosted offering silently upgrade your production pipeline.

The architecture shift from scripted to agentic automation

Classical web automation (Selenium, Puppeteer, Playwright in scripted mode) follows a deterministic recipe. The script knows the page structure, the selectors, and the expected response. When any of those changes, the script breaks. Engineers spend significant time maintaining selectors and recovery paths.

Agentic browser automation flips the model. The agent does not know the page structure. It receives a goal and a current page state, decides on the next action, and observes the result. The agent adapts to layout changes, follows alternative paths, and recovers from unexpected states.

The shift has implications for cost, capability, and reliability. Cost is higher because each step requires a model inference. Capability is broader because the agent can handle tasks the script author did not anticipate. Reliability is more variable because the agent occasionally chooses suboptimal actions.

The 2024-2026 pattern is to use scripted automation for high-volume, well-defined tasks (price scraping, sitemap crawling, structured data extraction) and agentic automation for low-volume, varied tasks (research, customer support automation, exploratory data gathering). The two patterns coexist in the same operation.

The accessibility tree as the agent’s primary input

The decision to use the accessibility tree (rather than the raw DOM or pure pixels) as the agent’s primary input was driven by three considerations. First, the tree is structured and parseable, supporting reliable selector generation. Second, the tree captures semantic information that the DOM does not (button roles, form labels, link purposes). Third, the tree is a small enough representation to fit in context windows.

The accessibility tree has limitations. Sites that use custom controls without ARIA attributes have impoverished trees. Single-page applications that update via JavaScript may have stale trees if the snapshot is taken at the wrong moment. Sites that intentionally obscure their structure (some bot-protected sites) have deliberately confusing trees.

The 2026 toolkit handles these cases through a combination of techniques. ARIA-deficient sites are augmented with screenshot grounding. Stale trees are addressed with explicit wait-for-load conditions. Obscured trees are addressed with vision-only fallback when the tree is unusable.

Step quality and the planning loop

The quality of an agentic browser depends on the quality of each step decision. Three factors drive step quality: the model’s understanding of the goal, the precision of the page state representation, and the appropriateness of the action vocabulary.

The 2026 patterns for improving step quality include explicit goal restatement at each step (preventing goal drift), structured action proposals with reasoning fields (improving the model’s articulation), and step-level critique by a separate model (catching obvious mistakes).

Planning loops can be flat (the model decides each step from scratch) or hierarchical (a planner decides sub-goals and a executor decides per-sub-goal actions). Hierarchical planning works better for complex multi-step tasks. Flat planning works better for short tasks. The 2026 best practice is to use hierarchical planning for tasks expected to take more than five steps.

Testing and evaluation of agentic browsers

A tested agentic browser is a more reliable agentic browser. The 2026 best practice is to maintain a fixed test suite of representative tasks with known expected outcomes. The suite is run on every model update and every framework update.

Evaluation metrics typically include task success rate, mean steps to completion, mean wall-clock time, mean cost, and recovery rate from injected failures. Each metric is tracked over time. Regressions trigger investigation.

The test suite must be maintained alongside the production tasks. As production tasks evolve, the test suite is updated. The test suite is the safety net that catches regressions before they affect production.

A 2026 best practice that is gaining traction is adversarial evaluation. The test suite includes deliberately misleading pages (decoy buttons, ambiguous instructions, time-pressured prompts) that probe the agent’s robustness. Performance on adversarial cases is a leading indicator of production reliability.

Next steps

If you have not piloted an agentic browser yet, the highest-leverage move this quarter is to pick your most fragile scraper and rebuild it in Stagehand or browser-use. The hour spent will tell you more than weeks of comparison reading. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the AI agents as web users guide.

This guide is informational, not engineering or legal advice.

LLM extraction patterns: structured output from messy HTML

LLM extraction structured output is the workhorse of modern scraping pipelines. Once the browser layer has rendered a page and given you HTML, the question is how to turn that messy DOM into clean JSON that your warehouse can ingest. In 2026 every major LLM provider ships strict JSON Schema mode, so the question is no longer “can I get JSON” but “what schema, what prompt, what model, what cost”.

This guide is the playbook. We cover schema design, prompt patterns, validation, retry strategy, cost control, and the model selection matrix across OpenAI, Anthropic, Google, and the open-source contenders. Every pattern is from production usage in 2026.

Why structured output matters

Three reasons.

First, downstream systems need typed data. A price field that is sometimes a number and sometimes a string with a currency symbol breaks every dashboard. Schema enforcement at the LLM boundary kills this class of bug.

Second, structured output is dramatically cheaper than freeform extraction over time. Freeform output requires post-processing logic that drifts with each new page format. Structured output forces the LLM to do the work once.

Third, structured output is the only path to reliable agentic loops. An agent that returns JSON can chain into the next step. An agent that returns prose breaks pipelines.

JSON Schema the right way

The single most common mistake in LLM extraction is loose schemas. A {"price": {"type": "number"}} field looks fine until the model returns null, the validation passes (because null is technically allowed without required), and your pipeline writes a row of garbage.

The right pattern is strict, required, and bounded.

schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string", "minLength": 1, "maxLength": 500},
        "price": {"type": "number", "minimum": 0, "maximum": 1000000},
        "currency": {"type": "string", "pattern": "^[A-Z]{3}$"},
        "in_stock": {"type": "boolean"},
        "sku": {"type": ["string", "null"]},
    },
    "required": ["title", "price", "currency", "in_stock", "sku"],
    "additionalProperties": False,
}

additionalProperties: False and complete required lists are not optional. They are how you stop the model from inventing fields or skipping ones that should be present.

OpenAI’s Structured Outputs (GA in 2024) and Anthropic’s tool use (which doubles as structured output) both honor strict schemas. Google Gemini supports JSON mode with a similar shape via the responseSchema parameter.

Strict mode in OpenAI

from openai import AsyncOpenAI
import json

client = AsyncOpenAI()

async def extract_product(html: str) -> dict:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "product",
                "schema": schema,
                "strict": True,
            },
        },
        messages=[
            {"role": "system", "content": (
                "Extract product data from the HTML. If a field is not present, "
                "use null for sku. All other fields are required."
            )},
            {"role": "user", "content": html[:200000]},
        ],
    )
    return json.loads(resp.choices[0].message.content)

strict: True constrains the decoder so the model literally cannot output invalid JSON. Compliance is enforced at the token level. This is the gold standard.

Tool use in Anthropic

from anthropic import AsyncAnthropic
import json

client = AsyncAnthropic()

async def extract_product(html: str) -> dict:
    resp = await client.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=2000,
        tools=[{
            "name": "save_product",
            "description": "Save the extracted product record",
            "input_schema": schema,
        }],
        tool_choice={"type": "tool", "name": "save_product"},
        messages=[{
            "role": "user",
            "content": f"Extract product data from this HTML:\n\n{html[:200000]}",
        }],
    )
    return resp.content[0].input

tool_choice: tool forces Claude to call the tool, which is how you guarantee structured output. The input_schema is JSON Schema and Claude validates against it before returning.

Prompt design for extraction

The system prompt matters more than people realize. Three rules from production.

First, name the entity explicitly. “Extract the product” is better than “extract structured data”. The model anchors on entity type.

Second, specify what to do when fields are missing. “Use null if not present” beats letting the model guess.

Third, if the HTML contains multiple candidates (multiple products, related items, ads), tell the model which one to extract. “The main product on this page” beats letting the model decide.

A solid system prompt template:

You are a precise data extractor. Extract the {entity} from the provided HTML.

Rules:
- Use the schema exactly. Do not add fields. Do not skip required fields.
- For missing fields, use null only if explicitly allowed.
- The entity to extract is: {entity_description}.
- Ignore related items, recommendations, advertisements, and footer content.
- Numeric fields must be numbers, not strings. Strip currency symbols and commas.

Gemini structured output

Google’s pattern uses responseSchema directly on the generation config:

import google.generativeai as genai

model = genai.GenerativeModel("gemini-1.5-flash-002")

resp = model.generate_content(
    f"Extract the product from this HTML:\n\n{html[:1000000]}",
    generation_config={
        "response_mime_type": "application/json",
        "response_schema": schema,
    },
)
data = json.loads(resp.text)

Gemini’s response_schema accepts JSON Schema with the same semantics as OpenAI’s strict mode. The 2-million token context window of Gemini Pro is the only place where you can pass an entire site’s product catalog HTML as a single extraction call.

Pre-processing HTML

Sending raw 800KB HTML to the model wastes tokens and confuses extraction. Trim aggressively before extraction.

from bs4 import BeautifulSoup
import re

def trim_html(html: str, target_tags=("title", "script[type='application/ld+json']", "meta")) -> str:
    soup = BeautifulSoup(html, "html.parser")
    # remove scripts (except JSON-LD), styles, navigation, footer
    for tag in soup(["style", "nav", "footer", "header", "iframe", "noscript"]):
        tag.decompose()
    for tag in soup("script"):
        if tag.get("type") != "application/ld+json":
            tag.decompose()
    text = str(soup)
    text = re.sub(r"\n\s*\n+", "\n\n", text)
    return text[:200000]

For ecommerce, JSON-LD Product markup is gold. Many sites embed full product data in <script type="application/ld+json"> and you can extract it with zero LLM cost.

import json
from bs4 import BeautifulSoup

def try_jsonld_product(html: str) -> dict | None:
    soup = BeautifulSoup(html, "html.parser")
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string or "")
            if isinstance(data, dict) and data.get("@type") == "Product":
                return data
            if isinstance(data, list):
                for item in data:
                    if isinstance(item, dict) and item.get("@type") == "Product":
                        return item
        except json.JSONDecodeError:
            continue
    return None

Always try JSON-LD first. Fall back to LLM extraction only if it fails.

Model selection matrix

For extraction specifically (not full agent loops):

Model	Cost per 1k extractions	Quality on messy HTML	Best fit
GPT-4o-mini	$0.30	High	Default for high-volume
GPT-4o	$5.00	Highest	Hard cases, large schemas
Claude Haiku 3.5	$0.40	High	Default if Anthropic-native
Claude Sonnet 4.5	$5.50	Highest	Hard cases, large schemas
Gemini 1.5 Flash	$0.20	High	Cost-sensitive volume
Gemini 1.5 Pro	$3.50	Highest	Long context (2M tokens)
Llama 3.3 70B (self-host)	$0.05	Medium-high	Privacy-critical
Qwen 2.5 72B (self-host)	$0.05	Medium-high	Asia language pages

GPT-4o-mini is the default pick in 2026 for English-language extraction at scale. It is cheap enough that you stop optimizing prompts to save tokens. Claude Haiku is the default pick if your stack is Anthropic-native. Gemini Flash is the cheapest of the strong options.

For multilingual extraction (Thai, Indonesian, Korean, Vietnamese), Gemini Pro and Claude Sonnet outperform GPT-4o on local-language pages. Anthropic and Google both invested heavily in Asian language quality through 2025.

Validation and retry

Schema enforcement is necessary but not sufficient. The model can return schema-valid garbage. Validate semantically.

from pydantic import BaseModel, Field, validator

class Product(BaseModel):
    title: str = Field(min_length=1, max_length=500)
    price: float = Field(gt=0, lt=1_000_000)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    in_stock: bool

    @validator("title")
    def title_not_placeholder(cls, v):
        if v.lower() in ("loading", "untitled", "n/a", "..."):
            raise ValueError("placeholder title")
        return v

On validation failure, retry with a different model (escalate from 4o-mini to 4o) or with a hint in the prompt (“the previous extraction had price=0 which is invalid; try harder to find the actual price”).

Two-pass extraction for hard pages

For very messy HTML, a two-pass extraction often beats a single-pass attempt.

Pass one: ask the model to find and extract just the relevant region.

Pass two: ask the model to extract the structured fields from that region.

async def two_pass_extract(html: str) -> dict:
    # pass 1: locate
    locate = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Find the main product section in the HTML and return only that section's HTML."},
            {"role": "user", "content": html[:200000]},
        ],
    )
    region = locate.choices[0].message.content

    # pass 2: extract
    return await extract_product(region)

This pattern doubles cost but cuts noise enough that quality on hard pages goes up by 10-20 percent.

Adding context to the prompt

When the model is missing context (you know the page is about wireless mice, the model has to guess), supply it.

async def extract_with_context(html: str, hint: dict) -> dict:
    return await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_schema", "json_schema": {"name": "product", "schema": schema, "strict": True}},
        messages=[
            {"role": "system", "content": "Extract the product."},
            {"role": "user", "content": f"Page context: {hint}\n\nHTML:\n{html[:200000]}"},
        ],
    )

Hint can include the URL, the breadcrumb category, the expected currency. The model uses these to disambiguate.

Few-shot examples in the prompt

For new sites where extraction quality is initially poor, two or three labeled examples in the prompt boost accuracy by 10 to 25 percent.

async def extract_with_examples(html: str, examples: list[tuple[str, dict]]) -> dict:
    example_text = "\n\n".join(
        f"Example HTML:\n{ex_html[:5000]}\nExtracted: {json.dumps(ex_data)}"
        for ex_html, ex_data in examples
    )
    return await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_schema", "json_schema": {"name": "x", "schema": schema, "strict": True}},
        messages=[
            {"role": "system", "content": (
                "Extract the product. Here are examples of correct extractions:\n\n"
                + example_text
            )},
            {"role": "user", "content": html[:200000]},
        ],
    )

The cost increase is the example tokens (a few thousand input tokens) versus accuracy gains. Worth it for any site you scrape regularly.

Cost control patterns

Three patterns that cut extraction cost without sacrificing quality.

Cache by content hash. Hash the trimmed HTML. If you have seen it before, reuse the prior extraction. For sites that change rarely, this is huge.

Schema-first model selection. Start with the cheapest model. Validate. Escalate to a stronger model only on failure. Most pages succeed on the cheap path.

Sample then scale. For new sites, run 10 pages on the strong model and 10 pages on the cheap model. If results match, scale on cheap. If they diverge, stay on strong.

Comparison to other extraction patterns

Pattern	Cost per 1k pages	Setup time	Adaptability
Hand-written CSS selectors	$0	4 hours per site	Low
XPath with auto-discovery	$0	1 hour per site	Low
LLM with strict schema	$0.30-$5	30 minutes per schema	High
Vision model (page screenshot)	$5-$20	30 minutes	Highest

For a deeper look at vision-model extraction, see our scraping with vision models guide.

Real-world benchmark across 1000 product pages

We ran the same extraction across 1000 mixed product pages from Lazada, Shopee, Amazon, Best Buy, and Mercado Libre. Schema enforced strict; pre-processing applied. Numbers from March 2026:

Model	Accuracy	Cost per 1000 pages	p50 latency
GPT-4o-mini	96.4%	$0.30	1.2 s
GPT-4o	98.1%	$5.20	1.8 s
Claude Haiku 3.5	95.7%	$0.45	1.4 s
Claude Sonnet 4.5	98.4%	$5.80	2.1 s
Gemini 1.5 Flash	95.2%	$0.22	1.1 s
Gemini 1.5 Pro	97.6%	$3.60	2.4 s
Llama 3.3 70B (vLLM)	91.2%	$0.06	0.9 s

Headline: GPT-4o-mini at $0.30 per 1000 pages with 96.4 percent accuracy is the value pick. Sonnet 4.5 wins on accuracy but the 19x cost is rarely justified unless the data is high-stakes.

The Llama row is the surprise. Self-hosted Llama 3.3 70B on a single H100 reaches 91 percent accuracy at one-fifth the cost. For high-volume teams with privacy requirements, this is the right pick despite the lower ceiling.

Storing extracted data

Extracted records should land in a typed schema. Postgres with JSONB plus extracted columns is the production pattern.

CREATE TABLE extractions (
    id BIGSERIAL PRIMARY KEY,
    source_url TEXT NOT NULL,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    title TEXT NOT NULL,
    price NUMERIC(12,2) NOT NULL,
    currency CHAR(3) NOT NULL,
    in_stock BOOLEAN NOT NULL,
    raw_jsonb JSONB NOT NULL
);

Raw JSONB preserves the full extraction for reprocessing if your schema evolves. Typed columns give you the indices and analytics performance.

Multi-entity extraction

Many pages contain multiple records (a search results page with 30 products, a job board with 50 listings). Two patterns work.

Pattern A, single call with array schema. Wrap the entity in an array.

schema = {
    "type": "object",
    "properties": {
        "results": {
            "type": "array",
            "items": product_schema,
            "minItems": 1,
            "maxItems": 50,
        }
    },
    "required": ["results"],
    "additionalProperties": False,
}

The model returns all matches in one call. Cheaper than N calls but loses partial-success granularity if extraction fails midway.

Pattern B, find then extract. First call locates the items (returns N HTML snippets), second call (per snippet) extracts the structured fields. More expensive but more reliable on long pages.

For listings under 30 items, pattern A is fine. Above 30, pattern B starts to win because the single-call approach starts losing items at the end of the response.

Schema evolution

Production extraction schemas evolve. Adding a field is easy; the model just produces nulls until you start populating. Removing a field is harder because old data still has it. Renaming a field requires migration.

Two practices that prevent pain:

Version your schema explicitly with a schema_version field. Old records keep their version; new records get the new one. Your warehouse can handle both.

Never delete fields. Mark them deprecated and stop reading them. Models that produced the old field get nulls or are ignored.

class ProductV3(BaseModel):
    schema_version: Literal["3.0"] = "3.0"
    title: str
    price: float
    currency: str
    in_stock: bool
    sku: Optional[str]
    # NEW in v3
    primary_image_url: Optional[str] = None
    # DEPRECATED in v3 (kept for back-compat reads)
    seller_name: Optional[str] = None

The result: schema changes never break the warehouse, and you can re-extract historical data on the new schema lazily.

Production observability

Log every extraction with: source URL, model used, tokens consumed, schema name and version, validation result, retry count. This data lets you spot model regressions, cost spikes, and pages that consistently fail.

When to skip the LLM entirely

Three scenarios where the LLM is overkill:

Site embeds JSON-LD Product markup. Already structured, parseable in 5 lines. No LLM needed.

Site has a public API (or an obvious internal one). Hit the API directly.

Site has stable selectors that have not changed in 12 months. A traditional Playwright selector script costs nothing per page.

The LLM is the right tool when the data is in messy HTML with no machine-readable alternative and the page format changes often enough that selector maintenance is expensive.

Frequently asked questions

Why does the model sometimes return null when the field is clearly on the page?
Three causes: schema allows null (tighten it), the prompt allows guessing (forbid it), or the relevant region was trimmed off (trim less aggressively).

How do I extract from non-English pages?
Add a language hint to the system prompt (“the page is in Thai”). Use a model with strong multilingual training. Gemini Pro and Claude Sonnet outperform GPT-4o on Asian languages in 2026.

Can I extract from PDFs, images, or videos?
Yes for PDFs (most LLM APIs accept PDFs directly). Yes for images (vision models). Videos require frame extraction first.

How do I handle nested or repeated entities (a list of variants on a product page)?
Use array fields in the schema with items as object schemas. The model handles arbitrary length cleanly.

Should I use one schema per site or one global schema?
Global schema with optional fields is the production pattern. Per-site schemas explode in maintenance cost.

How do I handle currency conversion in extraction?
Extract the original currency and price as the model sees them. Convert to a canonical currency in a downstream step using a daily FX rate snapshot. Mixing currency conversion into the extraction prompt makes the model less reliable.

How do I extract dates and times reliably?
Use a string field with format: "date-time" (ISO 8601). Add a system prompt instruction “convert all dates to ISO 8601 in UTC”. The model handles timezone conversion better than most teams expect.

Is JSON mode the same as strict structured output?
No. JSON mode just guarantees the output parses as JSON. Strict structured output guarantees it matches your schema. Always prefer strict.

How do I extract from sites that change schemas often?
Use a “best effort” outer schema with a free-form additional_data JSONB field that captures whatever the model finds beyond the strict fields. This is how you keep extracting useful data through schema drift.

Can I extract relationships (this product is a variant of that product)?
Yes. Add a parent_sku field. Or for richer graphs, run a separate relationship extraction pass after collecting the entities.

Common production gotchas

A few patterns bite repeatedly:

The model returns a price like 1,299.99 as a string because the page showed it that way. Schema validation should reject strings in number fields, and the prompt should explicitly tell the model to strip commas and currency symbols.

For very long pages (over 200k chars), you exceed the context window. Pre-trim aggressively or chunk the page and run the extraction per chunk, merging results.

Caching by URL alone misses content updates. Cache by content hash of the trimmed HTML, not URL.

The additionalProperties: False constraint occasionally rejects model output that included a useful extra field. Decide consciously whether you want strictness (reject) or flexibility (allow and ignore).

Validation libraries differ in date handling. Pydantic v2’s date parser is stricter than v1. Pin the version.

For more patterns on the AI extraction stack, see the AI data collection category.

Model Context Protocol (MCP) for data engineers in 2026

MCP for data engineers has shifted from a curiosity to a foundational protocol in eighteen months. Anthropic announced the Model Context Protocol (MCP) in November 2024 as an open standard for connecting LLMs to external data sources and tools. By mid-2026, MCP is supported across Claude Desktop, Claude Code, ChatGPT, Cursor, Windsurf, Continue, Sourcegraph Cody, and a long list of independent agent frameworks. For data engineers running scraping, ETL, and RAG pipelines, MCP changed how agents access data, how data engineering work surfaces to non-engineers, and how downstream products integrate with proprietary datasets. This guide walks through what MCP actually is, the server and client architecture, worked Python and TypeScript implementations for common data-engineering use cases, deployment patterns, and where MCP fits versus other patterns like function calling or RAG-only.

The audience is the data engineer or platform team responsible for making proprietary or scraped data available to AI agents in 2026.

What MCP actually is and is not

MCP is a JSON-RPC 2.0 based protocol that defines how an MCP host (typically an AI agent runtime like Claude Desktop) discovers and calls capabilities exposed by an MCP server. The capabilities come in three classes: tools (callable functions), resources (read-only data), and prompts (reusable prompt templates).

The protocol is intentionally minimal. It does not specify the LLM. It does not specify the transport at the application layer (the spec defines stdio, HTTP, and SSE transports). It does not require Anthropic infrastructure (the spec is open, the SDK is MIT-licensed). What it does is standardise the metadata format and request/response shape so that any client can talk to any server without bespoke integration.

For a data engineer, MCP is most useful as a way to expose datasets, query interfaces, and pipeline triggers to AI agents in a way that does not require rebuilding integration per agent platform.

For the broader agent landscape, see the agentic browser revolution and AI agents as web users.

The MCP architecture in three roles

MCP has three roles: host, client, server.

The host is the agent runtime. Claude Desktop is a host. Claude Code is a host. Cursor is a host. The host is responsible for managing user trust, presenting the agent’s context to the user, and routing tool/resource calls to the appropriate server.

The client is a session within a host that connects to one specific MCP server. A host can have many concurrent clients, each connected to a different server.

The server is the process that exposes capabilities. A server can expose any combination of tools, resources, and prompts. Servers can run as local subprocesses (stdio transport) or as remote services (HTTP/SSE transport).

For a data engineer, the server is the unit of work. You write a server. You publish it. Your agent users (or your customers’ agent users) install or connect to it.

A minimal MCP server for a scraping pipeline

A typical scraping pipeline exposes four operations: list known sources, trigger a scrape, retrieve scraped records, and check pipeline status. Here is a minimal Python implementation using the official anthropic-mcp Python SDK.

from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import Tool, TextContent, Resource
import asyncio
import json

server = Server("scraping-pipeline")

@server.list_tools()
async def list_tools():
    return [
        Tool(
            name="list_sources",
            description="List all known scraping sources.",
            inputSchema={"type": "object", "properties": {}},
        ),
        Tool(
            name="trigger_scrape",
            description="Queue a scrape for a specific source URL.",
            inputSchema={
                "type": "object",
                "properties": {
                    "source_id": {"type": "string"},
                    "max_pages": {"type": "integer", "default": 10},
                },
                "required": ["source_id"],
            },
        ),
        Tool(
            name="get_records",
            description="Fetch scraped records for a source within a date range.",
            inputSchema={
                "type": "object",
                "properties": {
                    "source_id": {"type": "string"},
                    "since": {"type": "string", "format": "date"},
                    "limit": {"type": "integer", "default": 100},
                },
                "required": ["source_id"],
            },
        ),
        Tool(
            name="pipeline_status",
            description="Show pipeline health and recent run summary.",
            inputSchema={"type": "object", "properties": {}},
        ),
    ]

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "list_sources":
        result = await db.list_sources()
        return [TextContent(type="text", text=json.dumps(result))]
    elif name == "trigger_scrape":
        job_id = await scheduler.enqueue(
            arguments["source_id"], arguments.get("max_pages", 10)
        )
        return [TextContent(type="text", text=f"Queued job {job_id}")]
    elif name == "get_records":
        records = await db.fetch_records(
            arguments["source_id"],
            arguments.get("since"),
            arguments.get("limit", 100),
        )
        return [TextContent(type="text", text=json.dumps(records))]
    elif name == "pipeline_status":
        status = await monitor.summary()
        return [TextContent(type="text", text=json.dumps(status))]

async def main():
    async with stdio_server() as (read, write):
        await server.run(read, write, server.create_initialization_options())

if __name__ == "__main__":
    asyncio.run(main())

The server is roughly 60 lines of code. An agent connected to it can list sources, trigger scrapes, fetch records, and check status without any agent-specific integration code.

Resources versus tools: when to use each

MCP servers can expose resources alongside tools. The distinction is intentional and matters for how agents reason.

Tools are imperative: they take arguments and return results. They are appropriate for actions (trigger a scrape, send an email, write a file).

Resources are declarative: they have a URI and a content type, and the host can read them on demand. They are appropriate for browsable content (a database table, a file in a blob store, a record from a CRM).

For a scraping pipeline, the typical pattern is:

Capability	Type	Why
trigger_scrape	Tool	It is an action with side effects
pipeline_status	Tool	It is a runtime query, not browsable
list_sources	Resource	Sources are a browsable list
get_records	Resource (per-source)	Records are browsable content

Mixing tools and resources gives the agent a richer mental model of your data surface. Tools-only servers feel like a set of CLI commands. Resource-rich servers feel like a database the agent can query.

Prompts: the under-used third capability

The third MCP capability, prompts, is the least understood. A prompt is a reusable template that a host can offer to its user. The user invokes it (typically via a slash command), and the host injects the rendered prompt into the conversation.

For a data engineer, prompts are useful for canonical workflows: “summarise yesterday’s scrape”, “diff today’s records against last week’s”, “draft a compliance report for source X”. The prompt definition lives on the server; the user invokes it by name; the host renders the templated content with arguments.

@server.list_prompts()
async def list_prompts():
    return [
        Prompt(
            name="summarise_yesterday",
            description="Summarise yesterday's scrape activity for the team.",
            arguments=[],
        ),
        Prompt(
            name="diff_records",
            description="Show changes in records since a specified date.",
            arguments=[
                PromptArgument(name="source_id", required=True),
                PromptArgument(name="since", required=True),
            ],
        ),
    ]

@server.get_prompt()
async def get_prompt(name: str, arguments: dict):
    if name == "summarise_yesterday":
        return GetPromptResult(
            messages=[
                PromptMessage(
                    role="user",
                    content=TextContent(
                        type="text",
                        text="Use the pipeline_status tool to get yesterday's "
                             "activity. Then summarise the new sources, "
                             "successful runs, and any failures.",
                    ),
                )
            ]
        )

Prompts close the loop: tools provide capability, resources provide browsable content, prompts provide canonical workflows.

Deployment patterns: stdio vs HTTP

MCP supports stdio and HTTP/SSE transports. The choice shapes deployment.

Stdio servers run as subprocesses spawned by the host. They are great for personal tools (the user installs the server and connects via Claude Desktop config). They are awful for shared infrastructure (every user runs their own instance, no shared state, no central observability).

HTTP/SSE servers run as long-lived services. They are great for shared infrastructure (one server, many users, shared state, central monitoring). They require authentication, networking, and operational ownership.

For a data engineering team, the typical deployment is HTTP/SSE behind your existing auth gateway. Add MCP to your existing service mesh; route /mcp/* to the MCP server; reuse your existing OIDC or token-based auth.

Transport	Best for	Setup time	Operational overhead
Stdio	Personal/desktop tools	Minutes	None
HTTP/SSE	Team/shared infrastructure	Hours	Standard service ops
Streamable HTTP (2025 addition)	Hybrid; better browser support	Hours	Standard service ops

The 2025 addition of streamable HTTP simplified browser-based hosts and is becoming the default for new HTTP servers.

MCP versus function calling: when to use each

Both MCP and function calling let an LLM invoke external capabilities. The difference is portability.

Function calling is per-LLM-provider. A tool defined for OpenAI’s function calling does not work with Anthropic’s tool use without translation. A change to one schema requires updates to all clients.

MCP is provider-neutral. A tool exposed via MCP works with any MCP-compatible host. The schema is declared once, used everywhere.

Dimension	Function calling	MCP
Portability	Per provider	Cross provider
Discovery	Static registration	Dynamic at session start
Resources	Not standardised	First-class
Prompts	Not standardised	First-class
Agent platform reuse	Low	High
Maintenance overhead	Per provider	Once
2026 ecosystem	Mature per provider	Rapidly growing cross provider

A data engineering team that picks function calling locks itself into one provider. A team that picks MCP gets cross-provider reach for slightly more upfront work.

Worked use case: RAG over scraped data via MCP

A common pattern in 2026 is to expose a RAG corpus over scraped data through an MCP server, so any agent host can query the corpus naturally. The architecture:

1. Scraping pipeline ingests source URLs, normalises HTML, embeds chunks, stores in a vector database.
2. MCP server exposes a search_corpus tool with arguments for query and filters.
3. Agent host connects to MCP server; user asks a question; agent calls search_corpus; corpus returns relevant chunks; agent synthesises an answer.

A minimal Python tool implementation:

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "search_corpus":
        query = arguments["query"]
        filters = arguments.get("filters", {})
        embedding = await embed(query)
        results = await vectordb.query(
            embedding, top_k=arguments.get("top_k", 5), filters=filters
        )
        return [TextContent(type="text", text=json.dumps(results))]

The MCP server is twenty lines. The vector database, embedder, and ingestion pipeline are independent. The server is the integration surface.

For the deeper RAG-over-scraped-data discussion, see RAG over scraped data production patterns and vector databases for scraping pipelines.

Security and trust model

MCP defines a trust boundary at the host-server connection. The server trusts the host to authenticate the user. The host trusts the server to honour its declared capabilities.

For HTTP transports, authentication is the server’s responsibility. The 2025 spec update added explicit guidance for OAuth 2.1 with PKCE, which is the recommended pattern for enterprise deployments. Bearer tokens work for service-to-service. Mutual TLS works for high-security environments.

Authorisation is the server’s responsibility. A server should enforce per-user permissions; the host’s user identity flows through the authentication layer. A scraping MCP server typically restricts trigger_scrape to authorised users while allowing get_records broadly.

Audit logging is the server’s responsibility. Every tool call should be logged with user identity, timestamp, arguments, and outcome. This is your defence against an agent calling the wrong thing on the wrong dataset.

Deployment checklist

Step	Owner	Done when
Define capabilities (tools, resources, prompts)	Engineering	Schema agreed, written
Implement server (stdio for prototype)	Engineering	Local Claude Desktop test passes
Migrate to HTTP for shared use	Engineering	Reachable behind auth gateway
Wire authentication (OAuth 2.1 / Bearer)	Security	Unauthenticated calls rejected
Wire authorisation (per-tool, per-user)	Security	Permission matrix enforced
Add audit logging	Engineering	Every call logged with identity
Document for users	Product	Runbook published
Publish for installation (registry / docs)	Marketing	Discoverable in MCP registry
Monitor and observe	Platform	Metrics dashboard live

External references

The MCP specification is at modelcontextprotocol.io. The Python SDK is at github.com/anthropics/python-mcp. The TypeScript SDK is at github.com/anthropics/typescript-mcp. Anthropic’s MCP servers reference at github.com/modelcontextprotocol/servers.

Comparison: MCP vs LangChain Tools vs OpenAI Function Calling

Dimension	MCP	LangChain Tools	OpenAI Function Calling
Cross-provider	Yes	Partial (with adapters)	No
Cross-host	Yes	LangChain runtime only	OpenAI agents only
Resource browsability	Yes	No	No
Prompts as first class	Yes	No	No
Streaming	Yes	Yes	Yes
Auth pattern	Spec-defined (OAuth 2.1)	App responsibility	OpenAI’s
2026 ecosystem maturity	High and growing	Mature within LangChain	Mature within OpenAI
Discovery	Dynamic at session	Static at runtime	Static per request

MCP is the cross-provider winner. LangChain Tools are the most flexible if you accept LangChain runtime lock-in. OpenAI function calling is the simplest if you only target OpenAI hosts.

FAQ

Do I need Anthropic infrastructure to use MCP?
No. The protocol is open, the SDKs are MIT-licensed, and any host can implement the protocol.

Can I expose existing REST APIs through MCP?
Yes. A thin MCP server can wrap any HTTP API and expose it as tools and resources.

How does MCP compare to OpenAPI?
MCP is at the agent integration layer; OpenAPI is at the HTTP API description layer. They are complementary; many MCP servers wrap OpenAPI-described services.

What is the right transport for production?
HTTP/SSE or streamable HTTP. Stdio is fine for personal/desktop scenarios, but production sharing needs HTTP.

How do I authenticate users on an HTTP MCP server?
OAuth 2.1 with PKCE is the recommended pattern. Bearer tokens work for service-to-service.

Extended MCP architecture analysis

The Model Context Protocol matured rapidly between its November 2024 launch and 2026. The protocol specification at v1.2 (early 2026) covers four primitive types, namely tools, resources, prompts, and sampling. For data engineers the tools and resources primitives are central. Tools expose callable functions to a model. Resources expose readable, addressable data.

A data-engineering MCP server typically wraps three layers. First, connection management (database connections, API clients, cache). Second, the tool surface (query, insert, transform, validate). Third, the observability surface (logs, metrics, request IDs).

The protocol is transport-agnostic. The two common transports are stdio (process-spawned servers, lowest latency) and SSE plus HTTP (network-deployed servers, fan-out across clients). For data pipelines stdio is preferred for local agents and SSE for shared infrastructure.

Production-ready MCP server pattern

from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import Tool, TextContent
import asyncio
import asyncpg

server = Server("data-pipeline")
pool = None

@server.list_tools()
async def list_tools():
    return [
        Tool(
            name="run_query",
            description="Execute a read-only SQL query against the warehouse",
            inputSchema={
                "type": "object",
                "properties": {
                    "sql": {"type": "string"},
                    "limit": {"type": "integer", "default": 1000},
                },
                "required": ["sql"],
            },
        ),
        Tool(
            name="describe_table",
            description="Return schema for a warehouse table",
            inputSchema={
                "type": "object",
                "properties": {"table": {"type": "string"}},
                "required": ["table"],
            },
        ),
    ]

@server.call_tool()
async def call_tool(name, arguments):
    if name == "run_query":
        sql = arguments["sql"]
        if not sql.strip().lower().startswith("select"):
            return [TextContent(type="text", text="Read-only access. SELECT only.")]
        async with pool.acquire() as conn:
            rows = await conn.fetch(sql + f" LIMIT {arguments.get('limit', 1000)}")
        return [TextContent(type="text", text="\n".join(str(r) for r in rows))]
    if name == "describe_table":
        async with pool.acquire() as conn:
            rows = await conn.fetch(
                "SELECT column_name, data_type FROM information_schema.columns WHERE table_name = $1",
                arguments["table"],
            )
        return [TextContent(type="text", text="\n".join(f"{r['column_name']}: {r['data_type']}" for r in rows))]

async def main():
    global pool
    pool = await asyncpg.create_pool("postgresql://...")
    async with stdio_server() as (read, write):
        await server.run(read, write, server.create_initialization_options())

if __name__ == "__main__":
    asyncio.run(main())

Tool surface design patterns

Effective MCP tool surfaces follow five rules.

1. Read and write are separate tools. Never bundle them.
2. Every destructive tool requires a confirm token in the input schema.
3. Pagination is explicit (cursor or offset/limit) rather than streaming everything.
4. Errors return structured content with hints for the model on next steps.
5. Long-running tools return a job ID and a separate status tool checks progress.

Comparison: MCP transport choices

Transport	Latency	Fan-out	Best for
stdio	Low (process IPC)	One client	Local agents, dev tooling
SSE plus HTTP	Moderate (network)	Many clients	Shared infrastructure
WebSocket (proposed)	Low	Bidirectional	Real-time dashboards

Observability for MCP servers

Production MCP servers should emit four signals.

1. Request count by tool name.
2. Request latency histogram by tool name.
3. Error rate by tool name and error class.
4. Token consumption per request (input plus output).

A simple approach is OpenTelemetry plus a Prometheus exporter. The MCP request lifecycle maps cleanly onto OTel spans.

Additional FAQ

Is MCP only for Anthropic models?
No. The protocol is open and other model vendors have shipped MCP support.

Can MCP replace REST APIs?
For agent-facing surfaces yes. For human-facing surfaces REST and GraphQL remain better fits.

How do I version an MCP tool surface?
Use semantic versioning on the server, expose the version in the initialization handshake, and add new tools rather than mutating existing ones.

What about authentication?
Transport-level auth (TLS, mutual TLS, signed headers) is the current pattern. The protocol does not prescribe an auth scheme.

When MCP wins versus when it loses

MCP is not the right answer for every data engineering integration. The protocol shines when the consumer is a model or an agent, and loses when the consumer is a deterministic application or a high-throughput batch job.

MCP wins when the access pattern is exploratory, when the schema is not known in advance, when the operations involve natural language, and when the consumer needs metadata to interpret the data. Examples include a model querying a warehouse for ad-hoc analysis, an agent investigating a customer support issue, and a copilot helping an analyst draft a report.

MCP loses when the access pattern is fixed, when the schema is well-known, when throughput requirements are high, and when latency budgets are tight. Examples include an ETL pipeline ingesting transactions, a real-time dashboard refreshing every second, and a microservice serving a known query at high QPS. For these cases REST, GraphQL, or direct database access remain the right choice.

The decision rule for a data engineering team is to expose the warehouse via REST or GraphQL for application consumers, and to layer MCP on top for agent and copilot consumers. The two surfaces share the underlying connection management and data layer, but expose different abstractions to different audiences.

Tool surface design beyond the basics

A first-pass MCP tool surface tends to expose run_query and describe_table. A production-quality tool surface goes further. The patterns that ship in 2026 include search_tables (for discovery when the agent does not know table names), suggest_join (for analytical queries that span tables), explain_query (for surfacing query plans), and validate_data (for checking data quality assumptions).

Each additional tool reduces the number of round-trips the agent needs to complete a task. An agent that can search, describe, query, and validate in four tool calls is materially more capable than an agent that can only query. The design goal is to anticipate the agent’s needs and expose primitives that satisfy them.

A countervailing concern is tool sprawl. An MCP server with too many tools confuses the agent. The 2026 sweet spot is somewhere between five and twenty tools, with clear non-overlapping purposes. Beyond twenty tools the agent’s planning quality degrades.

Security model for MCP servers

An MCP server that exposes warehouse access creates a powerful attack surface. The security model must address authentication, authorisation, audit, and rate limiting.

Authentication establishes who is connecting. For stdio transport the parent process is implicitly trusted. For SSE plus HTTP transport the connection should require a token, ideally short-lived and tied to a specific agent identity.

Authorisation determines what the authenticated principal can do. The 2026 pattern is to map the agent’s permissions onto the same role-based model used for human users. An agent acting on behalf of an analyst inherits the analyst’s permissions, plus additional restrictions specific to agent traffic.

Audit captures every tool call with the principal, the arguments, the result summary, and the timestamp. The audit log is the forensic record for incident investigation. The 2026 best practice is to retain MCP audit logs for at least ninety days.

Rate limiting prevents runaway agents from consuming resources. Per-tool, per-principal, and per-server quotas should each be enforced. The 2026 default is conservative quotas that can be raised on request.

MCP versioning and evolution

MCP is a young protocol, and breaking changes are expected as it matures. The 2026 best practice for MCP server operators is to follow semantic versioning, expose the version in the initialization handshake, and support the previous major version for at least six months after a breaking change.

For tool surfaces the rule is to add new tools rather than mutate existing ones. A tool that needs a new argument should be deprecated and replaced with a v2 variant. Existing agents continue to use the v1 tool until they are updated.

For data shapes the rule is similar. A response schema that needs to add a field is safe. A response schema that needs to remove or rename a field requires a versioned response. Many MCP servers expose a content_version metadata field on responses to allow agents to detect and adapt.

Next steps

The fastest first step is to wrap your most-used internal data interface in a minimal MCP server, run it via Claude Desktop, and see how it changes the team’s interaction. The integration overhead is low, the leverage is high. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the agentic browser revolution and RAG over scraped data guides.

This guide is informational, not engineering or legal advice.

Scraping JavaScript-heavy SPAs with AI agents in 2026

Scraping SPA AI agents pipelines have become the dominant pattern for any modern web target because the entire ecommerce, SaaS, and content stack has converged on React, Vue, and Next.js. Server-rendered HTML is the exception; client-rendered, hydrated, lazy-loaded SPAs are the rule. Traditional fetch-and-parse scrapers crash on these targets. AI agents that drive a real headless browser succeed.

This guide covers the patterns that work in 2026 for scraping SPAs with AI agents. We cover detection, rendering strategy, wait conditions, structured extraction, proxy integration, and the edge cases that bite if you miss them. Code in Python and TypeScript throughout.

Why SPAs break traditional scrapers

A SPA serves a near-empty HTML shell that loads JavaScript bundles, calls APIs, and renders the actual content client-side. If you fetch the URL with requests or httpx, you get the shell. The data you want is generated milliseconds to seconds later by JavaScript that never ran in your scraper.

The fix is to render the page in a real browser. The fix used to be Puppeteer or Selenium with brittle wait conditions and per-site selector maintenance. In 2026, the better fix is to drive a browser with an AI agent that watches the page render in real time and decides when to extract.

Detecting SPA targets

Before reaching for an AI agent, check whether the target actually needs one. Many sites that look like SPAs in DevTools are actually server-rendered or hybrid.

Quick detection script:

import httpx
from bs4 import BeautifulSoup

def is_spa(url: str) -> bool:
    r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"})
    soup = BeautifulSoup(r.text, "html.parser")
    text_chars = len(soup.get_text(strip=True))

    has_react_root = bool(soup.find(id="root") or soup.find(id="__next"))
    has_app_div = bool(soup.find("div", attrs={"id": "app"}))
    has_low_text = text_chars < 1000

    return (has_react_root or has_app_div) and has_low_text

print(is_spa("https://www.lazada.sg/"))  # True
print(is_spa("https://news.ycombinator.com/"))  # False

If is_spa returns False, use a normal HTTP scraper. If True, you need a real browser.

The agentic browser pattern

The pattern that wins in 2026 looks like this:

1. Launch a headless Chromium with stealth defaults
2. Navigate to the target URL
3. Wait for visual stability (network idle plus a small grace period)
4. Take a screenshot and dump the rendered HTML
5. Pass both to an LLM with a strict JSON Schema
6. Validate the result, retry if needed

In code, with browser-use as the agent driver:

import asyncio
from browser_use import Agent, Browser, BrowserConfig, Controller
from langchain_openai import ChatOpenAI
from pydantic import BaseModel

class Product(BaseModel):
    title: str
    price: float
    currency: str
    in_stock: bool

controller = Controller(output_model=Product)

async def scrape_spa_product(url: str) -> Product:
    browser = Browser(config=BrowserConfig(
        headless=True,
        extra_chromium_args=["--disable-blink-features=AutomationControlled"],
    ))

    agent = Agent(
        task=(
            f"Visit {url}, wait for the product details to fully load, "
            f"and return the title, price (number), currency code, and stock status."
        ),
        llm=ChatOpenAI(model="gpt-4o-mini"),
        browser=browser,
        controller=controller,
        max_failures=3,
    )

    history = await agent.run()
    return Product.model_validate_json(history.final_result())

product = asyncio.run(scrape_spa_product("https://www.lazada.sg/products/example-12345.html"))
print(product)

The agent watches the page render and decides when to extract. No selector maintenance. No wait condition tuning per site.

For more on browser-use specifically, see our browser-use scraping guide.

SPA framework cheat sheet

Different frameworks fingerprint differently. Quick recognition guide:

Framework	Tells
Next.js (App Router)	__next div, __next-build-id meta, _next/static/... script URLs
Next.js (Pages Router)	__NEXT_DATA__ script tag, _next/static/chunks/pages/...
Remix	__remixContext script, data-route on root
SvelteKit	__sveltekit_* global, data-sveltekit-* attrs
Nuxt	__NUXT__ script, _nuxt/... script URLs
Astro	astro-island custom elements, mixed SSR with client islands
React + Vite	<div id="root"> + /src/main.tsx script ref
Vue + Vite	<div id="app"> + Vue devtools meta
Angular	<app-root> element, ng-version attr

Knowing the framework tells you whether the data is already in the initial HTML payload. Next.js with App Router and Remix often serve fully rendered HTML; Astro typically does too. SPA frameworks with strict client rendering (vanilla React + Vite, vanilla Vue) almost always need a real browser.

Wait conditions that actually work

The trickiest part of SPA scraping is waiting long enough for content to hydrate without waiting forever. Three patterns:

Network idle plus grace period. Wait for networkidle (no requests for 500ms) then sleep an additional 1-2 seconds. Catches most React Suspense boundaries.

DOM-stability detection. Watch the DOM mutation count, wait until it stabilizes for 1-2 seconds.

Sentinel selector. Wait for a known element that only appears after content loads (price, product image, review count). Most reliable when you know the site.

For agentic scraping, the agent handles this implicitly. You just give it enough time budget per page.

Stagehand example with explicit wait:

import { Stagehand } from "@browserbasehq/stagehand";

const stagehand = new Stagehand({ env: "LOCAL", modelName: "gpt-4o-mini" });
await stagehand.init();
const page = stagehand.page;

await page.goto("https://www.lazada.sg/products/example.html", { waitUntil: "networkidle" });
await page.waitForTimeout(1500);  // grace period for React hydration

const data = await page.extract({
  instruction: "Extract product title, price, currency, in-stock status",
  schema: z.object({ title: z.string(), price: z.number(), currency: z.string(), inStock: z.boolean() }),
});

When networkidle lies

networkidle is a useful default but it lies on three common patterns. Long-poll connections never go idle, so a chat widget keeps the network busy forever. Analytics beacons that fire every 5 seconds prevent idle from triggering. Web sockets that retry every few seconds also prevent the idle state.

The mitigation is to combine networkidle with a hard timeout cap and a DOM-stability check. If networkidle has not fired within 10 seconds but the visible content has stopped changing, extract anyway.

async function waitForContent(page: Page, timeoutMs = 15000) {
  const start = Date.now();
  let lastDomSize = 0;
  let stableCount = 0;
  while (Date.now() - start < timeoutMs) {
    const size = await page.evaluate(() => document.body.innerText.length);
    if (size === lastDomSize && size > 500) {
      stableCount++;
      if (stableCount >= 3) return;  // 1.5s of stability
    } else {
      stableCount = 0;
    }
    lastDomSize = size;
    await page.waitForTimeout(500);
  }
}

This pattern beats raw networkidle on roughly 30 percent of SPAs we tested.

Handling infinite scroll and lazy loading

Many SPAs render content lazily as the user scrolls. Two patterns to handle this.

Scroll loop. Scroll to bottom in a loop until the page height stops growing.

async function scrollUntilStable(page: Page, maxScrolls = 20) {
  let lastHeight = 0;
  for (let i = 0; i < maxScrolls; i++) {
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(1000);
    const height = await page.evaluate(() => document.body.scrollHeight);
    if (height === lastHeight) return;
    lastHeight = height;
  }
}

Intercept the underlying API. Many SPAs lazy-load by calling a paginated JSON endpoint. Open DevTools, find the call, and hit it directly. Skip the browser entirely. This is the highest-throughput pattern when it applies.

import httpx

async def fetch_lazada_listings(category: str, page: int):
    url = f"https://www.lazada.sg/api/listing?category={category}&page={page}"
    headers = {"x-csrf-token": "...", "User-Agent": "Mozilla/5.0"}
    async with httpx.AsyncClient() as c:
        r = await c.get(url, headers=headers)
        return r.json()

When you can find and hit the underlying API, do that. AI agents are the fallback when the API is hidden, signed, or rate-limited too aggressively for direct access.

Comparison of SPA scraping approaches

Approach	Cost per page	Reliability on changing layouts	Maintenance	Best fit
Direct API interception	$0.001	High	Medium	When you can find the API
Playwright with custom selectors	$0.005	Low	High	Stable known-shape sites
Stagehand extract	$0.04	High	Low	Long-tail SPA targets
browser-use full agent	$0.04	High	Low	Multi-step SPA flows
Operator/Computer Use	$0.20	Highest	Lowest	Hardest targets

For 80 percent of SPA scraping work in 2026, Stagehand’s extract primitive plus a 1.5-second grace period is the sweet spot. It is cheap enough to scale, reliable enough to ignore most layout changes, and easy enough to write that a junior engineer can ship a new scraper in an hour.

Hydration race conditions

A common bug: you take a screenshot at the wrong moment and the agent extracts placeholder data (“Loading…”, skeleton boxes, default values).

Two defenses:

First, validate the output. Reject any extraction where price equals zero or title contains “loading”. Retry with a longer wait.

def validate_product(p: dict) -> bool:
    if not p.get("title") or "loading" in p["title"].lower():
        return False
    if p.get("price", 0) <= 0:
        return False
    return True

Second, take two screenshots 500ms apart and compare. If they differ significantly, the page is still rendering. Wait, retry, repeat.

Adding proxy rotation

SPAs are typically served by sites with strong bot defenses (Cloudflare, DataDome, Akamai). Mobile or residential proxies are mandatory.

In browser-use:

from browser_use import Browser, BrowserConfig

browser = Browser(config=BrowserConfig(
    headless=True,
    proxy={"server": "http://proxy.example.com:8000", "username": "u", "password": "p"},
))

In Stagehand:

const stagehand = new Stagehand({
  env: "LOCAL",
  localBrowserLaunchOptions: {
    proxy: { server: "http://proxy.example.com:8000", username: "u", password: "p" },
  },
});

For ASEAN ecommerce SPAs (Lazada, Shopee, Tokopedia), Singapore mobile proxy carries real Singtel and StarHub IPs that avoid the data-center blocks these sites apply.

Structured extraction at the end

The right pattern is to drive the agent only to reach the right page and dump the rendered HTML, then run structured extraction on a cheaper model.

# step 1: navigate with agent
agent = Agent(task=f"Reach the product page at {url} and extract the full HTML",
              llm=ChatOpenAI(model="gpt-4o-mini"))
result = await agent.run()
html = await agent.browser.context.pages[0].content()

# step 2: cheap structured extraction
import json
from openai import AsyncOpenAI
client = AsyncOpenAI()
extract = await client.chat.completions.create(
    model="gpt-4o-mini",
    response_format={"type": "json_schema", "json_schema": {
        "name": "product",
        "schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "number"},
                "currency": {"type": "string"},
                "in_stock": {"type": "boolean"},
            },
            "required": ["title", "price", "currency", "in_stock"],
            "additionalProperties": False,
        },
        "strict": True,
    }},
    messages=[{"role": "user", "content": html[:200000]}],
)
product = json.loads(extract.choices[0].message.content)

This split typically cuts cost by half compared to one big agent loop. For more, see LLM extraction patterns.

Network interception for hidden data

When the underlying API is hidden but exists, intercept the network requests directly. Both Playwright and Stagehand expose a request/response listener that captures everything the browser fetches.

const apiResponses: Record<string, unknown> = {};

page.on("response", async (response) => {
  const url = response.url();
  if (url.includes("/api/product/") && response.headers()["content-type"]?.includes("json")) {
    try {
      apiResponses[url] = await response.json();
    } catch {}
  }
});

await page.goto(productUrl);
await waitForContent(page);

// apiResponses now contains the underlying JSON payloads

This pattern often gets you cleaner data than DOM extraction because the API payload is the source of truth that the SPA renders. Once you find the API, you can hit it directly and skip the browser entirely.

SPA scraping with vision-only extraction

For SPAs where the DOM is obfuscated (CSS-in-JS with random class names, shadow DOM components, canvas-rendered text), vision extraction can succeed where DOM extraction fails.

from openai import AsyncOpenAI
client = AsyncOpenAI()

async def extract_from_screenshot(png_b64: str, schema: dict) -> dict:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_schema", "json_schema": {"name": "x", "schema": schema, "strict": True}},
        messages=[{"role": "user", "content": [
            {"type": "text", "text": "Extract data from this screenshot per the schema"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{png_b64}"}},
        ]}],
    )
    return json.loads(resp.choices[0].message.content)

Vision extraction is roughly 3x more expensive per page than DOM extraction but works on a handful of sites where DOM extraction is essentially impossible. For more, see scraping with vision models 2026.

Production patterns

Three patterns separate hobby SPA scrapers from production ones.

First, cap per-page wall clock. Even agents can spin if the page is broken. Set timeout: 60000 and treat exceeded timeouts as failures.

Second, monitor cost per page. SPA scraping costs add up fast. Log tokens consumed per scrape and alert if a single page exceeds 30,000 tokens (likely a confused agent).

Third, keep a fallback. When the agent fails, fall through to a static Playwright scraper with cached selectors. Catches the cases where the agent is wrong and the deterministic code is right.

Cookie banners and modal interruptions

The single most common cause of stuck SPA scrapes is a cookie banner blocking the content. The agent sees the banner, the LLM does not understand to dismiss it, and the agent gives up.

Hardcode a banner-handling preamble in your task:

"If a cookie banner, age verification modal, region selector, or login prompt
is visible, dismiss it (click 'Accept all', 'I'm 18+', 'Close', or similar).
Then proceed with the main task."

This single instruction moves success rates on European retailer SPAs by 15 to 25 percentage points.

For Stagehand, you can also use act to dismiss known modals before extract:

await page.act("If a cookie banner is visible, click the 'Accept all' button");
const data = await page.extract({ instruction: "...", schema: ... });

Common SPA scraping pitfalls

A handful of failure modes worth memorizing.

The agent sees server-rendered placeholder content and extracts the placeholder. Mitigation: detect placeholders by content patterns (“loading”, skeleton boxes, default 0 values).

The page redirects to a login wall after a few page loads. Mitigation: rotate session cookies and IPs, or warm up the session with realistic browsing before scraping.

The site uses cursor-based pagination with opaque tokens. Mitigation: simulate the user click that triggers the next page rather than constructing the URL yourself.

The site delivers different markup based on user agent or viewport. Mitigation: use a realistic UA and a desktop viewport (1440×900), not the Playwright defaults.

The site lazy-loads images that block extraction because the LLM expects them. Mitigation: prefer text-only extractions, and if you need images, scroll first.

Real benchmarks on common SPAs

100 product pages each, GPT-4o-mini extraction:

Target	Approach	Success rate	Avg time per page
Lazada SG	browser-use + mobile proxy	98%	7.1 s
Shopee SG	Stagehand + mobile proxy	96%	6.4 s
Amazon US	Playwright + residential	94%	3.2 s
Best Buy	browser-use + residential	91%	8.5 s
Booking.com	Stagehand + residential	88%	12 s

For more on Lazada specifically, see our Lazada Thailand scraping guide. For Shopee, see Shopee Indonesia scraping.

Hydration timing across SPA frameworks

Different frameworks hydrate at different speeds. Average time from domcontentloaded to “fully interactive” on a typical product page:

Framework	Median hydration	p99 hydration
Next.js App Router (RSC)	250 ms	1.4 s
Next.js Pages Router	600 ms	2.8 s
Remix	350 ms	1.6 s
SvelteKit	200 ms	1.1 s
Nuxt 3	480 ms	2.4 s
React + Vite SPA	1.2 s	4.5 s
Vue + Vite SPA	1.0 s	4.2 s
Astro with islands	200 ms (mostly SSR)	1.0 s

For most production scrapers, a 2-second wait is sufficient. For React + Vite SPAs, bump to 4 seconds. The cost in latency is small compared to the cost in failed extractions.

Frequently asked questions

Can I scrape an SPA without a real browser?
Sometimes. If the SPA uses Next.js with getServerSideProps or React Server Components, the initial HTML may already contain the data you need. Check by curl-ing the URL. Otherwise, a real browser is required.

What about hydration mismatches?
Hydration mismatches happen when client-rendered HTML differs from server HTML. Wait until after hydration completes (typically 500-1500ms) before extracting.

How do I handle authentication on SPAs?
Most SPAs use cookie-based session auth. Log in once, save the storage state, replay on each scrape. Both Playwright and Stagehand expose storageState config for this.

Can I run an SPA scraper on AWS Lambda?
Cold starts kill latency. Lambda with the Chromium layer works for occasional jobs. For high throughput, run on Fargate or a dedicated VPS. Browserbase is the pay-per-minute alternative.

Why is my agent extracting “0” for prices?
Almost always a hydration timing issue. The agent extracted while the React app still showed the skeleton placeholder. Add a longer wait or a stability check before extraction.

How do I parallelize SPA scraping when each page takes 8 seconds?
Run multiple browser contexts in parallel within one Chromium process (cheaper than multiple browsers). Cap concurrency at roughly 1 per CPU core to avoid thrashing the browser’s rendering pipeline.

How do I handle SPAs that detect headless browsers?
Use headless=False with a virtual display (Xvfb) on Linux, or use Browserbase’s stealth mode. The single biggest tell is the navigator.webdriver flag, which Chromium sets to true in headless mode and most stealth plugins patch.

Can I use the Beautiful Soup HTML output from a Playwright render?
Yes. After page.content(), you have the rendered HTML as a string and can pass it to BeautifulSoup or lxml for traditional parsing. The combination of headless render plus traditional parser is a real production pattern.

What about WebSockets and Server-Sent Events?
Real-time data over WebSocket or SSE is harder to scrape because the data flows continuously. Use Playwright’s page.on('websocket') event listener to capture frames as they arrive, or intercept the underlying API connection.

How do I scrape an SPA behind a paywall I have access to?
Save the browser storage state (cookies, localStorage) after manual login, then load it on each scrape via storageState. Refresh the state when it expires.

Are there SPAs that just cannot be scraped reliably?
Yes. Sites with strong client-side encryption (some financial dashboards), sites that gate data behind interactive verification (real-time KYC), and sites with anti-replay tokens that bind to a specific browser fingerprint. For these, manual data export or partner APIs are the only realistic paths.

For broader patterns on the agentic browser stack, see our AI modern scraping category.

Building an ethics-first scraping policy for your team

Scraping ethics policy is the artefact that ties everything together: legal posture, technical controls, customer expectations, employee onboarding, and incident response. Most teams have informal practices but no written policy, and the gap shows up the first time something goes wrong (a regulator letter, a customer compliance question, a journalist inquiry, an internal escalation). A written policy is not paperwork. It is the document that prevents most of those failures from becoming crises. This guide walks through what a working scraping ethics policy contains, how to operationalise it across compliance regimes, the team workflow for adoption and maintenance, and a template structure your team can adapt this quarter.

The audience is the engineering lead, product owner, or compliance partner who needs to move from informal scraping practice to a defensible, written, lived policy.

Why a written policy matters operationally

Three reasons.

First, regulators ask for it. The EDPB, the CPPA, the PDPC, and the DPB all explicitly look for written policies during investigations. Their absence is itself evidence of insufficient organisational maturity, which weighs against you in penalty assessments. Their presence shifts the burden: investigators read your policy first and assess deviation.

Second, customers ask for it. Enterprise customers, especially in regulated industries (banking, healthcare, government), require vendor data practices documentation. A scraping operator without a written policy loses deals to one with a policy.

Third, your team needs it. Scraping decisions are made daily: should we add this source? should we ingest this field? should we honour this opt-out request even though the legal threshold isn’t met? Without a written policy, each decision is ad hoc, debated from scratch, and inconsistent. With a policy, decisions are faster and more defensible.

For the broader compliance picture, see the GDPR compliance guide and the personal vs public data scraping framework.

Policy structure: seven sections that work

A working policy is short, practical, and lived. Long policies that read like legal documents are signed and ignored. The seven-section structure that works in practice:

1. Stated principles (one page maximum)
2. Scope and applicability
3. Allowed and disallowed activities
4. Compliance regime alignment
5. Operational controls
6. Incident response
7. Review and accountability

Each section has a clear owner, a review cadence, and a link to the operational artefacts that implement it.

Stated principles

The principles section is the most important. It is what stays with employees long after they forget the procedural details. A working set of principles for a 2026 scraping operator:

1. We collect only the data we need for the purposes we have stated.
2. We respect site operator preferences expressed through robots.txt and AI-specific directives.
3. We honour data subject rights regardless of jurisdiction.
4. We treat publicly available data with the same care we would apply to data we collected directly.
5. We document our decisions and review them quarterly.
6. We disclose data breaches promptly, internally and externally as required.
7. We do not scrape behind technical access controls we did not lawfully bypass.
8. We provide a clear, monitored channel for site operators and data subjects to contact us.

Eight principles, each one sentence, all action-oriented. Print them. Post them. Reference them in performance reviews. They become culture.

Scope and applicability

The scope section answers: who does this policy bind, and which activities does it cover?

A working scope statement: “This policy applies to all employees, contractors, and engaged service providers of [Company] who design, build, operate, or use any scraping pipeline, automated browser, or data collection workflow that touches third-party websites or APIs. The policy applies to all scraping activities regardless of jurisdiction, target, scale, or commercial purpose.”

The breadth is intentional. A narrow scope creates loopholes that bite later.

Allowed and disallowed activities

The allowed/disallowed section is the most operational. It removes ambiguity from common decisions.

Allowed activities (default):
– Scraping logged-out, publicly accessible URLs that respect robots.txt
– Scraping with our published, attributable user agent
– Honouring published opt-out signals (TDM-Reservation, AI-bot disallow)
– Caching robots.txt with a maximum 24-hour TTL
– Polling published APIs at documented rate limits

Disallowed activities (default; require leadership approval):
– Bypassing CAPTCHAs, IP blocks, or fingerprinting checks
– Creating accounts on target platforms for the purpose of scraping
– Scraping behind paywalls or other access controls
– Scraping personal data of children
– Scraping sensitive personal information (health, biometric, sex life, political)
– Reselling raw scraped personal data to third parties
– Operating without an opt-out mechanism

Conditionally allowed (require documented assessment):
– Scraping personal data of identifiable individuals (LIA required)
– Scraping for AI training (training manifest required)
– Scraping for cross-border resale (Article 27 / SCC compliance required)

The trick is to be specific. “Honour applicable law” is not a policy; it is a wish.

Compliance regime alignment

This section maps the policy to the specific regulations that apply to your operation. A worked example:

Regime	In scope?	Owner	Key artefacts
GDPR (EU)	Yes	DPO	LIA, Article 30 register, Article 27 representative
UK GDPR	Yes	DPO	Same as GDPR plus UK addendum
CCPA (California)	Yes	DPO	Privacy notice, GPC handler, deletion inbox
PDPA (Singapore)	Yes	DPO	Notice, DPO appointment, grievance mechanism
DPDP (India)	Yes	DPO	Consent records, grievance mechanism
LGPD (Brazil)	Conditional	DPO	Same as GDPR analogue
US state laws (others)	Conditional	DPO	Map per state; align to CCPA where stricter
EU AI Act	If training	ML lead	Training data summary, transparency report

The map gets reviewed annually. New regimes get added. New jurisdictions get evaluated.

For the per-regime detail, see the GDPR, CCPA, PDPA, and DPDP guides.

Operational controls

The operational controls section translates principles and compliance maps into specific technical and procedural controls. A working control set:

Control	Owner	Implementation
robots.txt parser	Engineering	Protego middleware on every scraper
User agent identification	Engineering	Fixed UA string per pipeline; logged
Rate limiting	Engineering	Per-domain, with exponential backoff
TDM-Reservation parser	Engineering	HTTP and meta tag
Field-level data minimisation	Engineering + product	Storage schema review per pipeline
Retention enforcement	Engineering	Automated purge jobs per source
Pseudonymisation	Engineering	Per-pipeline token replacement
Encryption at rest and transit	Security	TLS 1.3, AES-256 storage
Access controls	Security	Role-based, audit-logged
Audit logging	Security	Per-request, retained 12 months
Privacy notice publication	Compliance	Public page, multi-language
Opt-out and deletion inbox	Compliance	Monitored daily
DPO appointment	Compliance	Named, contactable, in scope of role
Article 27 representative	Compliance	EU-based, contracted
Vendor DPAs	Procurement	Per provider, reviewed annually
Training manifest (if AI)	ML lead	Per dataset, per training run

Controls are concrete and assigned. A control without an owner is aspirational.

Incident response

The incident response section answers: what happens when something goes wrong?

A working incident response process:

1. Detection: any team member who notices an issue (regulator letter, journalist inquiry, customer escalation, internal anomaly) reports to incidents@yourcompany.com within 24 hours.

2. Triage: the on-call compliance partner (rotating role) classifies the incident as low, medium, high, or critical within 24 hours of detection.

3. Containment: high or critical incidents trigger immediate containment (pause the affected scraper, freeze the affected dataset, restrict access).

4. Notification: regulator notification under GDPR Article 33 within 72 hours of awareness for personal data breaches likely to result in risk to data subjects. Other regimes have varying timelines.

5. Investigation: a written incident report within 7 days of triage, naming the cause, the scope, the affected parties, and the remediation.

6. Remediation: changes to controls, policy, or training to prevent recurrence.

7. Post-mortem: within 30 days, a blameless review with the team, documented lessons.

The incident response process is the part of the policy most teams skip. Build it before you need it.

Decision tree: policy alignment for a new scrape

Q1: Is the proposed scrape within the allowed activities list?
    ├── Yes -> Proceed with standard controls.
    └── No  -> Q2
Q2: Is it within the conditionally allowed list?
    ├── Yes -> Conduct documented assessment; obtain DPO sign-off.
    └── No  -> Q3
Q3: Is it on the disallowed list?
    ├── Yes -> Stop. Escalate to leadership for explicit override.
    └── No  -> Add to allowed/disallowed list during next policy review.

The decision tree forces the conversation early, before engineering effort is committed.

Review and accountability

The review section answers: who is responsible for keeping the policy alive?

A working accountability map:

• Policy owner: the DPO, with executive sponsor.
• Quarterly review: the DPO and engineering lead review the policy, the controls, and the audit log.
• Annual review: a full review including the compliance regime map, the allowed/disallowed list, and the incident report log.
• Trigger reviews: any new jurisdiction, any new high-risk source, any incident classified medium or higher, any new product line.

The policy is treated as a living document. Versioned in git or a comparable system. Each version dated and signed.

A worked policy implementation timeline

For a team adopting an ethics-first scraping policy from scratch, a 12-week rollout:

Week	Deliverable
1-2	Stated principles drafted with leadership
3-4	Scope, allowed/disallowed lists, compliance regime map
5-6	Operational controls inventory; gap analysis against current state
7-8	Build missing controls (robots.txt middleware, opt-out inbox, retention purge)
9-10	DPO appointed, Article 27 representative engaged, privacy notice published
11-12	Incident response runbook, blameless review template, training delivered

After week 12, the policy is in steady state. Quarterly reviews keep it current.

For the technical control implementation patterns, see robots.txt and modern scraping ethics.

External references

For sample policy structures, the EDPB Code of Conduct registry at edpb.europa.eu/our-work-tools/accountability-tools/register-codes-conduct-amendments-and-extensions lists approved industry codes. The OECD Privacy Guidelines (1980, revised 2013) at oecd.org/sti/ieconomy/oecdguidelinesontheprotectionofprivacyandtransborderflowsofpersonaldata.htm provide the foundational principles.

Comparison: ethics-first vs compliance-only vs principles-only

Dimension	Ethics-first policy	Compliance-only	Principles-only
Stated principles	Yes	Optional	Yes
Compliance regime map	Yes	Yes	No
Operational controls	Yes	Yes	No
Customer trust signal	High	Medium	Low
Regulator response posture	Strong	Adequate	Weak
Team consistency	High	Medium	Variable
Maintenance overhead	Moderate	High	Low
Defensibility	High	High	Low
Cultural fit	Best for engineering teams	Best for legal-heavy teams	Worst

Ethics-first is the most demanding to set up but the easiest to maintain because the principles drive the rest.

A template policy starter

Below is a minimal starter that a team can adapt. The full version runs to about 6-8 pages; this is a one-page condensed version suitable for week-one circulation.

[Company] Scraping Ethics Policy v1.0

Principles:
1. We collect only what we need.
2. We respect robots.txt and AI directives.
3. We honour data subject rights everywhere.
4. We treat public data with private-data care.
5. We document and review quarterly.
6. We disclose breaches promptly.
7. We do not bypass technical controls.
8. We provide a clear contact channel.

Scope: All employees, contractors, service providers; all scraping;
all jurisdictions; all targets.

Allowed: Logged-out public scraping; published API polling; UA-attributed
crawling; robots.txt-respecting fetching.

Disallowed: CAPTCHA bypass; account creation for scraping; paywall
bypass; children's data; sensitive personal info; raw personal data
resale; operation without opt-out mechanism.

Conditionally allowed (DPO sign-off): Personal data scraping (LIA);
AI training (manifest); cross-border resale (SCC).

Owner: DPO. Reviewed quarterly. Incidents to incidents@[company].com.

Adopt the spirit, adapt the specifics. The output is a document that fits in one email.

FAQ

Do small teams really need a written policy?
Yes. The first regulator letter or enterprise compliance questionnaire is the wrong moment to discover you don’t have one. Even a one-page policy is far better than none.

How often should the policy be reviewed?
Quarterly review of controls and audit log; annual full review. Trigger reviews for new jurisdictions, sources, or incidents.

Who should own the policy?
The Data Protection Officer (formal title or designated equivalent), with an executive sponsor.

What if our team is too small to have a DPO?
Designate a current employee as DPO; the role can be combined with other duties. The PDPA, GDPR, and DPDP all permit this for smaller organisations.

How do we handle disagreement between principles and commercial pressure?
The principles win, every time. That is the entire point of writing them down. If the commercial pressure persistently overrides the principles, escalate to leadership; if the principles persistently lose, the company has a culture problem that the policy alone cannot fix.

Extended policy implementation analysis

An ethics-first scraping policy succeeds or fails on three dimensions, namely measurability, accountability, and adaptability. The 2024-2026 wave of regulator activity (CPPA enforcement advisories, Italian Garante decisions, Singapore PDPC AI Model Governance Framework second edition, India DPDP rules) all reward operators with documented, measured, and reviewed policies. They penalise operators with policies that exist on paper but cannot be evidenced in operation.

A policy is measurable when each principle has a quantitative or binary indicator. For example transparency is measurable as does the privacy notice cover the scraping operation, yes or no. Proportionality is measurable as percentage of fields collected versus fields available. Rights-honouring is measurable as median response time to a verified rights request.

A policy is accountable when each section has a named owner and a review cadence. The owner does not need to be a senior executive. They need to be the person who is asked first if the principle is violated.

A policy is adaptable when it is reviewed at minimum annually and reissued with a one-page diff against the prior version. Change is the only constant in this space, and a policy that has not been touched since 2023 is no longer a current policy.

Implementation patterns for the seven sections

The seven-section template generally follows this structure.

1. Stated principles. One sentence per principle. Do not exceed seven.
2. Scope and applicability. Which products, teams, regions, and data classes are covered.
3. Allowed and disallowed activities. A bright-line list with examples.
4. Compliance regime alignment. The mapping table to GDPR, CCPA, PDPA, DPDP, and others.
5. Operational controls. The technical list (robots.txt handling, rate limits, retention, pseudonymisation, audit logs).
6. Incident response. The decision tree for breaches, complaints, and rights requests.
7. Review and accountability. The owners and the review cadence.

Code pattern: policy compliance check at ingest

class PolicyGate:
    def __init__(self, policy):
        self.policy = policy

    def allow(self, target_url, purpose, jurisdiction):
        if target_url in self.policy.disallowed_domains:
            return False, "domain_disallowed"
        if purpose not in self.policy.allowed_purposes:
            return False, "purpose_not_listed"
        if jurisdiction in self.policy.consent_required and not self.has_consent(target_url):
            return False, "consent_missing"
        return True, "ok"

Worked policy implementation timeline expanded

A first-time rollout typically takes six weeks.

• Week 1. Draft principles and scope. Run a tabletop exercise against three real scrape targets.
• Week 2. Map to compliance regimes. Write the LIA template and the privacy notice updates.
• Week 3. Write the operational controls list. Identify gaps in current tooling.
• Week 4. Stand up the rights-request inbox and the breach-notification runbook.
• Week 5. Train the team. Run a second tabletop with the new policy in hand.
• Week 6. Publish the policy and the changelog. Set the next review date.

The annual review typically takes one engineering week and one legal week.

Comparison: policy maturity by stage

Stage	Indicator	Risk posture
Stage 0 (no policy)	Nothing written	High
Stage 1 (paper policy)	Document exists, not enforced	Moderate to high
Stage 2 (enforced policy)	Document exists, controls implemented	Moderate
Stage 3 (measured policy)	Controls measured monthly	Low to moderate
Stage 4 (adaptive policy)	Annual review with diff, regulator-grade evidence	Low

Additional FAQ

Who should sign off the policy?
At minimum the head of engineering, the head of legal, and the data protection officer if one exists. Board-level sign-off is appropriate for organisations above 100 employees.

Should the policy be public?
A summary should be public for transparency. The full operational policy can remain internal.

What if our scraping is small-scale and ad hoc?
The policy should still exist. A two-page version is acceptable for small operations. The principles do not change with scale.

How often should the policy change?
Annually at minimum. Out-of-cycle updates are appropriate after major regulatory developments or after an internal incident.

Why ethics-first beats compliance-only

A compliance-only posture treats the policy as a checklist of regulatory requirements. The policy says do these specific things to satisfy GDPR, CCPA, PDPA, and DPDP. The policy is silent on cases the statutes do not specifically address.

An ethics-first posture starts from principles (transparency, proportionality, respect for data subject rights) and derives behaviour from the principles. The policy speaks to cases the statutes have not yet addressed, and tends to anticipate regulator priorities a year or two before they crystallise into rules.

The 2024-2026 regulator activity reinforces the ethics-first advantage. The Italian Garante’s Replika decision turned on transparency and proportionality, principles that an ethics-first policy would already cover. The CPPA’s enforcement on Global Privacy Control turned on respecting consumer signals, a principle that ethics-first policies typically include before specific rule-making.

A compliance-only policy that lacks an ethics-first overlay is more vulnerable to surprise. When a regulator extends an existing rule to new facts, the compliance-only policy must be amended. The ethics-first policy already addresses the new facts because the underlying principle was already in scope.

The role of internal champions

A policy without internal champions tends to atrophy. The named owner for each section must be empowered to enforce the policy, and the broader engineering and product teams must understand the policy’s relevance to their work.

The 2026 best practice is to designate a Data Stewardship Council with rotating membership. The Council reviews the policy annually, fields questions from product teams, and makes recommendations on policy amendments. The Council includes representation from engineering, legal, product, and customer support.

The Council’s most important function is the case-by-case advisory role. Product teams considering a new scrape submit a brief to the Council. The Council reviews against the policy and either approves, rejects, or returns with conditions. The decisions accumulate as a body of internal precedent that informs future cases.

The Council does not need to be heavy-weight. A typical Council meets once a quarter for sixty minutes plus async case reviews. The cost is modest relative to the regulatory and reputational exposure that the Council prevents.

Tabletop exercises and incident drills

A policy that has not been tested under stress is not a real policy. The 2026 best practice is to run quarterly tabletop exercises that simulate realistic incidents.

Useful tabletop scenarios include a regulator inquiry alleging insufficient lawful basis, a high-profile data subject objection that becomes a media story, a downstream customer requesting data the policy does not allow disclosing, an internal employee scraping outside the policy, and a vendor breach exposing scraped data.

Each scenario is run for sixty to ninety minutes with the relevant team members. The team works through the response, identifies gaps, and updates the policy or the runbook as needed. The output of each tabletop is a one-page summary with three to five action items.

The cumulative effect of quarterly tabletops is that the team develops muscle memory for incidents. When a real incident occurs the response is faster, more measured, and better documented. Several 2025 enforcement decisions explicitly cited the absence of incident drills as evidence of insufficient governance.

Next steps

The fastest first step is to draft the eight stated principles, get them signed by your engineering and product leads, and circulate them this week. The rest of the policy follows. For the underlying compliance posture, head to the DRT compliance hub and pair this guide with the GDPR, CCPA, PDPA, and DPDP guides.

This guide is informational, not legal advice.

OpenAI Operator vs Anthropic Computer Use for scraping

Operator vs Computer Use scraping has become a real engineering choice in 2026 now that both products have matured past their initial preview releases. OpenAI Operator launched in January 2025 as a consumer Pro feature and added an API in mid-2025. Anthropic’s Computer Use went GA on the API in late 2024 and shipped through Claude 3.5 Sonnet, then 3.7, then the current 4.x line. Both let an LLM drive a real computer the way a human does: screenshots in, mouse and keyboard out.

This guide compares the two for scraping work specifically. Architecture, code samples, cost, where each one wins.

What each product actually is

OpenAI Operator is a hosted agent product. You give it a task, it opens a Chromium browser inside OpenAI’s infrastructure, and it executes the task end to end. You do not provision the compute. You do not write the loop. You write the prompt and you collect the result.

Anthropic Computer Use is an API capability, not a product. The API exposes computer, bash, and text_editor tools that the model can call. You provide the compute (a browser, a desktop, a sandbox), you implement the tool execution, you run the loop. Anthropic gives you the brain. You build the body.

For scraping, that distinction matters. Operator is a black-box product with low setup cost. Computer Use is a building block you assemble.

Operator API basics

OpenAI’s Operator API is part of the Responses API surface. You declare an ComputerUsePreview tool, send a user message, and receive a stream of actions to execute.

Wait, actually, in early 2026 OpenAI’s hosted Operator runs the loop for you when you use the consumer product. The API exposes computer-use-preview model which still runs in your VM. So both Operator (hosted) and the API exist.

For the API:

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="computer-use-preview",
    tools=[{
        "type": "computer_use_preview",
        "display_width": 1024,
        "display_height": 768,
        "environment": "browser",
    }],
    input=[{
        "role": "user",
        "content": "Go to news.ycombinator.com and return the top 5 story titles as a JSON array."
    }],
    truncation="auto",
)

# response.output contains a list of computer-use actions
# Execute each action against your browser, screenshot, send back as new input

You implement the loop yourself, executing each action and feeding screenshots back. The OpenAI cookbook has a complete reference implementation.

Anthropic Computer Use basics

Anthropic’s API exposes the computer tool similarly. You declare it, the model returns tool calls, you execute, you feed the screenshot back.

from anthropic import Anthropic

client = Anthropic()

messages = [{
    "role": "user",
    "content": "Go to news.ycombinator.com and return the top 5 story titles as JSON."
}]

while True:
    response = client.beta.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=4096,
        tools=[{
            "type": "computer_20250124",
            "name": "computer",
            "display_width_px": 1024,
            "display_height_px": 768,
        }],
        messages=messages,
        betas=["computer-use-2025-01-24"],
    )

    tool_use_blocks = [b for b in response.content if b.type == "tool_use"]
    if not tool_use_blocks:
        break

    for tu in tool_use_blocks:
        screenshot_b64 = execute_action(tu.input)  # your browser driver
        messages.append({"role": "assistant", "content": response.content})
        messages.append({
            "role": "user",
            "content": [{
                "type": "tool_result",
                "tool_use_id": tu.id,
                "content": [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64}}],
            }],
        })

Same shape, different model. The key difference is execution quality, which we benchmark below.

A complete loop in 60 lines

For a working reference, here is a complete Computer Use loop that drives a Playwright browser. Drop this into a script and it works against any target.

from anthropic import Anthropic
from playwright.sync_api import sync_playwright
import base64

client = Anthropic()

def run_task(task: str, start_url: str = "about:blank") -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page(viewport={"width": 1024, "height": 768})
        page.goto(start_url)

        def screenshot_b64() -> str:
            return base64.b64encode(page.screenshot()).decode()

        def execute(action: dict) -> dict:
            t = action["action"]
            if t == "screenshot":
                pass
            elif t == "left_click":
                x, y = action["coordinate"]
                page.mouse.click(x, y)
            elif t == "type":
                page.keyboard.type(action["text"])
            elif t == "key":
                page.keyboard.press(action["text"])
            elif t == "scroll":
                page.mouse.wheel(0, action.get("scroll_amount", 5) * 100)
            page.wait_for_timeout(500)
            return {
                "type": "image",
                "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64()},
            }

        messages = [{"role": "user", "content": task}]
        final_text = ""
        while True:
            r = client.beta.messages.create(
                model="claude-sonnet-4-5-20250929",
                max_tokens=4096,
                tools=[{"type": "computer_20250124", "name": "computer",
                        "display_width_px": 1024, "display_height_px": 768}],
                messages=messages,
                betas=["computer-use-2025-01-24"],
            )
            tool_uses = [b for b in r.content if b.type == "tool_use"]
            text_blocks = [b for b in r.content if b.type == "text"]
            if text_blocks:
                final_text = text_blocks[-1].text
            if not tool_uses:
                browser.close()
                return final_text
            messages.append({"role": "assistant", "content": r.content})
            messages.append({"role": "user", "content": [
                {"type": "tool_result", "tool_use_id": tu.id, "content": [execute(tu.input)]}
                for tu in tool_uses
            ]})

The Operator equivalent is structurally similar but uses the Responses API and the computer_use_preview tool type.

Where to run the actual computer

Both APIs require you to provide the compute. Three common choices:

Compute target	Best for	Setup time
Local Playwright Chromium	Development, single-user	30 minutes
Browserbase	Browser-only production	1 hour
Scrapybara	Full desktop production	2 hours
Self-hosted X11 + Chromium	Full control, custom apps	1 day

For scraping specifically, Browserbase is the typical pick because the platform is built for this. For agents that need to manipulate downloaded files, Scrapybara is required. For comparison, see our Scrapybara vs Browserbase guide.

Side-by-side capability comparison

Capability	OpenAI Operator	Anthropic Computer Use
Hosted product	Yes (ChatGPT Pro)	No
API availability	Yes (computer-use-preview)	Yes (computer_20250124)
Default model	computer-use-preview	claude-sonnet-4-5
Visual reasoning quality	Strong on UI elements	Strong on text-heavy pages
Screenshot rate	Per action	Per action
Bash tool	No	Yes
Text editor tool	No	Yes
Multi-app workflows	Browser only	Full desktop
Cost per 1000 actions	$40-$60	$35-$70
Best fit	Browser scraping with hosted convenience	Multi-tool agentic workflows

The honest takeaway: Operator is more polished for browser-only scraping. Computer Use is more flexible because it ships bash and text editor alongside computer. If your scraping involves running CLI tools, processing files, or interacting with non-browser apps, Computer Use is the only real choice.

Latency and cost benchmarks

Same task, both platforms: scrape Hacker News top 5 stories with title, score, submitter.

Metric	Operator	Computer Use
Average steps to complete	6	5
Wall clock time	32 s	28 s
LLM tokens per run	18,000	14,000
LLM cost per run	$0.27	$0.18
Compute cost (Browserbase)	$0.004	$0.004
Total per run	$0.27	$0.19

Computer Use was slightly faster and cheaper on this specific task, primarily because Claude Sonnet 4.5 is more efficient on visual reasoning per token than the current Operator preview model.

Numbers shift on harder tasks. For complex multi-step scrapes (login, filter, paginate, extract), Operator’s tighter loop won in our testing. For ambiguous pages where the model needs to think through what to do, Computer Use won.

Reliability on common scraping targets

We ran 50 trials of three scraping tasks against each platform.

Target	Operator success	Computer Use success
Hacker News top 5 (easy)	50/50	50/50
Lazada product page (medium)	47/50	49/50
Cloudflare-protected site (hard)	31/50	38/50
Booking.com flight search (multi-step)	42/50	39/50

The platforms are close on easy and medium tasks. Computer Use edges ahead on bot-defended sites. Operator edges ahead on multi-step flows where the model has to maintain longer working state.

Integration patterns

For production scraping, neither platform is a drop-in replacement for traditional scrapers. The right pattern is to use them for high-value, hard-to-parse targets and use traditional Playwright for everything else.

Use Operator for:

• Sites that change layout often
• Sites where you only need a few thousand pages a month
• Workflows that benefit from a polished hosted experience

Use Computer Use for:

• Workflows that need bash or file manipulation alongside browser
• Multi-tool agentic pipelines
• Sites where you need maximum reasoning quality on the extraction step

Use traditional Playwright for:

• Known-shape, high-volume scraping where unit cost matters
• Sites with stable selectors

Bash and text editor advantages

Anthropic’s bash and text_editor tools open workflows that Operator cannot match.

A scraping pipeline that needs to download a PDF, run pdftotext on it, and extract structured data can do all three steps in one Computer Use loop:

# inside the loop, the model can call:
# computer.left_click on the download button
# bash: pdftotext /tmp/downloaded.pdf -
# text_editor: parse output

Operator can only drive the browser. The PDF processing must happen outside the loop, in your code, after Operator finishes. The result is more glue code and more state to track.

For pure browser scraping the difference does not matter. For agents that touch any non-browser tool, the difference is decisive.

Pairing with structured extraction

Both platforms benefit enormously from a structured-output extraction step at the end. Rather than ask the agent to return JSON directly, have it copy the relevant page region (or take a final screenshot), then pass that to a cheaper model with strict JSON Schema.

This pattern cuts cost by 30 to 50 percent because the structured-output call is much cheaper than another agent step.

For more on structured extraction, see LLM extraction patterns: structured output from messy HTML.

Action atom-level breakdown

What the agent actually does step by step on a typical scrape:

1. Take screenshot
2. Identify search box, click it
3. Type query
4. Press Enter
5. Wait for results
6. Take screenshot
7. Extract result list, return JSON

Each numbered step is one or more LLM calls. Operator and Computer Use both reach this in roughly 5 to 8 atoms for simple tasks. The variance comes from how aggressively each model takes “extra look” screenshots, which Operator does more often than Computer Use.

For a 50-step debugging session, the screenshot count alone dominates token usage. A frequent optimization is to downscale screenshots to 800×600 before sending to the model, which cuts vision tokens by roughly 35 percent with minimal quality loss on simple pages.

Adding proxies

Operator (the hosted product) does not expose proxy configuration; the API does, through the underlying browser you provide. Computer Use is the same; you provide the compute, you provide the proxies.

For any production scraping that needs IP diversity, route the underlying browser through a residential or mobile proxy. Singapore mobile proxy is the right pick for ASEAN. Bright Data and Oxylabs cover the rest of the world.

Vendor pricing in detail

Per-1M-token pricing as of mid-2026 for the relevant models:

Model	Input	Output	Vision tokens (per 1M)
computer-use-preview	$3	$12	$3
Claude Sonnet 4.5	$3	$15	$3
Claude Haiku 4	$0.80	$4	$0.80

A typical 5-screenshot task at 1024×768 burns roughly 9,000 input tokens (vision-heavy) and 600 output tokens. Per task: roughly $0.03 to $0.06 in raw LLM cost, before browser compute.

These numbers move quarter to quarter. Re-check vendor pricing pages before budgeting a campaign.

Comparison with browser-use and Stagehand agent

Approach	Setup	Cost per page	Quality	Best fit
OpenAI Operator API	Medium	$0.27	High	Browser scraping with OpenAI ecosystem
Anthropic Computer Use	Medium	$0.19	High	Multi-tool agents with Claude
browser-use	Easy	$0.04	Medium-high	Quick OSS prototype
Stagehand agent	Easy	$0.05	Medium-high	TypeScript-first AI scraping

The OSS frameworks (browser-use, Stagehand agent) are cheaper because they make many smaller LLM calls rather than one big computer-use loop. They produce comparable quality on most targets. The hosted Computer Use APIs win on harder targets where reasoning quality matters.

For more, see our browser-use guide.

Cost engineering

The biggest cost lever in either platform is the screenshot. A 1024×768 PNG is roughly 1,500 vision tokens for Claude and around 1,800 for OpenAI’s preview model. Multiply by 5 to 10 screenshots per task and you see why per-task cost is in the dollar range.

Three optimizations that work:

Downscale screenshots to 800×600 before sending to the model. Cuts vision tokens by 35 percent, accuracy drops by less than 2 percent on most tasks.

Crop to the relevant viewport. If the task is in the top half of the page, send only the top half. Cuts another 30 percent.

Skip screenshots when the action does not change the page. After typing into a field, the page rarely changes meaningfully, so skip the next screenshot and let the next “real” screenshot capture the click result.

Combined, these cut typical per-task cost by 50 percent or more without significant accuracy loss.

Reliability patterns

Both APIs occasionally fail mid-loop. The most common failure modes:

The model gets stuck in a loop, repeatedly clicking the same element. Mitigation: track recent actions, refuse to repeat the same action three times, escalate to a clarifying prompt.

The model produces an action with bad coordinates. Mitigation: validate coordinates against the screenshot dimensions, reject and re-prompt if they fall outside.

The browser navigates to an unexpected page. Mitigation: track URL changes, stop the agent if it leaves the expected domain or follows an unrelated link.

The agent times out without completing. Mitigation: cap total wall-clock time at 90 seconds and total LLM calls at 25, then return whatever partial state exists.

Production recommendation

Pick Anthropic Computer Use if you are building a real agent product, want bash and text editor alongside the browser, and need the strongest reasoning quality on hard targets.

Pick OpenAI Operator if your team is OpenAI-native, your scraping is browser-only, and you value the hosted product polish.

Pick neither if you are doing high-volume known-shape scraping where unit cost matters. browser-use, Stagehand, or raw Playwright is the right choice there.

Multi-tab and multi-page handling

Scraping tasks that span multiple tabs (e.g. open a search result, capture data, return to results, open the next) trip up both APIs. The model has to track which tab is foreground and the screenshot only shows the active tab.

Workarounds:

Limit to one tab. Most scraping tasks are doable in a single tab if the model navigates carefully.

Tag tabs in screenshots. Add a small overlay in the screenshot showing the tab index, so the model can include “switch to tab 2” as an action.

For Computer Use, leverage bash to inspect the browser process and list open windows. Operator does not expose this.

Decision matrix

Your situation	Pick
Browser-only, OpenAI-native team	Operator API
Browser plus file or terminal manipulation	Computer Use
Highest reasoning quality on weird sites	Computer Use (Sonnet 4.5)
Lowest setup time	Operator hosted (consumer)
High-volume known-shape	Neither, use Playwright
Cost-sensitive long tail	browser-use or Stagehand
Multi-LLM comparison	Run both, take majority

Frequently asked questions

Does Operator’s consumer product have an API?
The hosted ChatGPT Pro Operator is human-facing only. The API surface is computer-use-preview, which is the underlying capability you call yourself.

Can Computer Use run on a serverless container?
Yes, with caveats. You need a Chromium binary, X11 (or virtual display), and enough memory. AWS Lambda with the Chromium layer works for short tasks. Fargate with a 2vCPU 4GB task is more practical.

Which one supports vision better, screenshots aside?
Both consume page screenshots. Claude Sonnet 4.5 produces sharper reasoning on text-heavy pages. The Operator preview model is better at recognizing UI elements like dropdowns and modals.

Can I use both in parallel for resilience?
Yes. Run the same task on both, compare outputs, take the agreement. Triples your cost but cuts your error rate substantially. Useful for high-stakes data extraction.

What about Gemini Computer Use?
Google added a comparable capability in Gemini 2.5 in Q1 2026. It is closer to Operator in shape and ships through the Gemini API. Worth comparing if your stack is Google-native.

Can the model get stuck in an infinite click loop?
Yes. The mitigation is a per-task action history with a deduplication rule that refuses to repeat the same action three times in a row. Both APIs let you set this in your loop.

Does either API support batching?
No, both are single-task. For batching, run multiple tasks in parallel asyncio coroutines, each with its own browser instance.

What is the rate limit story?
Anthropic’s tier-2 default is 50,000 input tokens per minute on Sonnet 4.5. OpenAI’s computer-use-preview ships with similar tier-based limits. Plan for 10 to 20 concurrent tasks per tier-2 account.

Common production gotchas

The model occasionally hallucinates an element that is not on the page. Always validate that the click coordinate corresponds to a visible element before executing.

Browser context state leaks across tasks if you reuse the browser. Either start a fresh context per task or explicitly clear cookies and storage.

The screenshot encoding is base64 PNG. For high-throughput pipelines, the base64 overhead adds up; consider compressing the screenshot first if you need to ship it elsewhere.

The platforms charge for retries. A loop that keeps misclicking burns cost fast. Set a budget per task and abort cleanly when hit.

If the target site uses sticky session cookies tied to fingerprint, switching browsers mid-task breaks the session. Pin the browser instance to the task lifecycle.

For broader context on the agentic browser space, see our agentic browser revolution guide.

India DPDP Act for scrapers: 2026 compliance

DPDP Act India scraping compliance is the newest major data protection regime in Asia, with the Digital Personal Data Protection Act 2023 substantially in force from 2025 and the supporting rules issued in early 2026. India is one of the world’s largest sources of digital personal data, and any scraper touching Indian users now operates under the DPDP. The regime is consent-default like GDPR, but with several India-specific design choices: the data fiduciary versus data principal terminology, the Significant Data Fiduciary tier, the Data Protection Board (rather than independent regulators), and the cross-border transfer model based on a positive whitelist. This guide walks through the structure, the consent and notification rules, the operator obligations, and a practical compliance checklist for scrapers.

The audience is the technical lead or in-house counsel responsible for a scraping pipeline that touches Indian residents, or one based in India that touches anywhere.

What the DPDP Act actually covers in scraping context

The DPDP Act applies to the processing of digital personal data in India where the data is collected in digital form, or in non-digital form and subsequently digitised. It also applies to processing outside India that is in connection with offering goods or services to data principals in India. Like GDPR Article 3, the DPDP has extraterritorial reach, and scraping operations that systematically target Indian residents are in scope.

Personal data under Section 2(t) is any data about an individual who is identifiable by or in relation to such data. The “digital” qualifier is critical: the DPDP only covers digital personal data, which means voice, paper, and physical-only data falls outside the regime.

Data principal is the individual to whom the data relates. Data fiduciary is the entity that determines the purpose and means of processing. Data processor processes on behalf of the fiduciary. The terminology mirrors GDPR’s data subject and data controller but is locally distinct.

The Data Protection Board (DPB) was constituted in 2024 and operates as an adjudicatory body rather than a Western-style regulator. The DPB receives complaints, conducts inquiries, and imposes penalties. The maximum penalty under the DPDP is INR 250 crore per breach (approximately USD 30 million), making it one of the highest-penalty regimes in Asia.

For comparison with the Singapore PDPA, see the Singapore PDPA for scrapers guide. For the GDPR parallel, see the GDPR compliance guide.

The consent default and its narrow exceptions

The DPDP Act’s default rule is that personal data may be processed only with the consent of the data principal. Consent must be free, specific, informed, unconditional, unambiguous, and given through a clear affirmative action. The consent request must be presented in a manner that is clear, in plain language, and accompanied by a notice describing the personal data being collected and the purpose of processing.

This consent default makes life harder for scrapers than the corresponding GDPR Article 6(1)(f) Legitimate Interests basis. The DPDP does not include a general legitimate interests provision. It does, however, provide for “legitimate uses” under Section 7, which permits processing without consent in specific enumerated cases:

• The data principal voluntarily provided the data and has not indicated an objection.
• Processing by the State for the provision of any subsidy, benefit, service, certificate, licence, or permit.
• Compliance with a court order, judgement, or legal obligation.
• Medical emergency, threat to life, or public health response.
• Employer-employee relationship purposes.
• Compliance with a legal obligation to disclose.

For scrapers, the operationally relevant cases are extremely narrow. The “voluntarily provided” exception is the closest analogue to a general scraping basis, but it requires the data principal to have voluntarily provided the data and applies only to the purpose for which it was provided. A user who voluntarily posts on a public forum did not voluntarily provide the data to a downstream scraper.

The publicly-available exception is also narrower than PDPA Singapore: only personal data made publicly available by the data principal themselves, or by another person obligated under law to make it available, is excluded. Publicly available data that ended up online through breach, leak, or third-party aggregation is not covered.

Compliance checklist for scrapers

Control	What it requires	Why it matters
Consent or legitimate use basis	Per-source documentation	Section 6 / Section 7
Notice to data principal	At or before collection	Section 5
Purpose limitation	Use only for notified purposes	Section 5(2)
Data minimisation	Collect only what is needed	Section 4
Accuracy obligation	Reasonable steps to maintain accuracy	Section 8(3)
Reasonable security safeguards	Section 8(5)	Mandatory
Breach notification	To DPB and affected principals	Section 8(6)
Erasure on consent withdrawal	Section 8(7)	Right to be forgotten
Data Protection Officer (if SDF)	Mandatory for SDFs	Section 10(2)
Independent data auditor (if SDF)	Annual	Section 10(2)
DPIA (if SDF)	Periodic	Section 10(2)
Cross-border transfer	Whitelist mechanism	Section 16
Children’s data extra protections	Verifiable parental consent	Section 9
Grievance redressal mechanism	Easily accessible	Section 8(10)

A scraper operating at scale in India needs every row, with extra controls if classed as a Significant Data Fiduciary.

The Significant Data Fiduciary tier

Section 10 introduces the Significant Data Fiduciary (SDF) classification. The Central Government may notify any data fiduciary or class of fiduciaries as significant based on the volume and sensitivity of personal data processed, risk to electoral democracy, security of the State, public order, or other prescribed factors.

SDFs face additional obligations:
– Appoint a Data Protection Officer based in India.
– Engage an independent data auditor to conduct annual audits.
– Conduct periodic Data Protection Impact Assessments.
– Comply with such other measures as the Central Government may prescribe.

The 2025-2026 notifications classified several large-scale data processors as SDFs, including some that scrape and aggregate at industrial scale. A scraper that crosses certain volume thresholds or operates AI training pipelines on Indian personal data should expect SDF designation in the near term.

For the AI training overlay, see fair use and copyright for AI training data.

Cross-border transfer and the whitelist model

Section 16 of the DPDP gives the Central Government the power to restrict transfer of personal data to specified countries or territories. This is a positive whitelist model: by default, transfer is permitted to any country, and the government can list countries that are restricted (or, depending on rules interpretation, list permitted countries).

The 2026 rules clarified that the government would publish a list of restricted destinations rather than a list of permitted destinations. This is operator-friendly: scrapers can transfer freely to any unlisted country until and unless that country is added to the restricted list.

As of mid-2026, the restricted list is short and primarily targets countries with no diplomatic relations with India or specific national security concerns. Major scraping destinations (US, EU, Singapore, UK, Canada, Australia) are all unrestricted.

This is the most operator-friendly element of the DPDP. The flexibility makes Indian-data pipelines easier to architect than EU-data pipelines.

Decision tree: is this scrape DPDP-compliant?

Q1: Is the source data made publicly available by the data principal themselves?
    ├── Yes -> Outside DPDP scope; verify and document.
    └── No  -> Q2
Q2: Have you obtained consent from the data principal?
    ├── Yes -> Document consent; standard obligations apply.
    └── No  -> Q3
Q3: Does a Section 7 legitimate use apply?
    ├── Yes -> Document; standard obligations apply.
    └── No  -> Stop; restructure to obtain consent or change scope.

For scrapers, the consent default and narrow legitimate use list make the DPDP much more restrictive than PDPA Singapore. The publicly-available carve-out is narrower than EU GDPR’s. Plan around this; do not assume.

Data principal rights

Sections 11-14 grant data principals four core rights: right to information about processing, right to correction and erasure, right to grievance redressal, and right to nominate (a person who can exercise rights on the principal’s behalf).

The right to erasure is broader than GDPR’s because it applies on consent withdrawal and is not subject to the same balancing tests. A scraper who relies on consent and the principal withdraws consent must erase, full stop.

The right of grievance redressal requires the data fiduciary to maintain an easily accessible grievance mechanism. The DPB only accepts complaints after the principal has exhausted the fiduciary’s grievance mechanism. A scraper without a public, responsive grievance inbox is exposed.

For the worked operational pattern, see the personal vs public data scraping framework.

Children’s data and verifiable parental consent

Section 9 imposes additional obligations for data of children (under 18 in India). A data fiduciary must obtain verifiable consent from the parent or lawful guardian before processing children’s data. Tracking, behavioural monitoring, and targeted advertising directed at children are prohibited.

For scrapers, this is a major exposure if any source contains children’s data. Social media platforms, gaming platforms, education platforms, and parts of the public web all contain user-generated content from minors. The DPDP’s verifiable parental consent requirement is not a “best efforts” standard; it requires a specific verifiable mechanism.

The 2026 rules specified acceptable verification mechanisms including Aadhaar-based verification for parents and acceptable third-party verification services. Scrapers who cannot verify must avoid processing children’s data, which in practice means filtering aggressively at intake.

How DPDP enforcement is shaping up in 2025-2026

The DPB began operations in 2024 and has issued its first round of decisions in 2025-2026. The early pattern: focus on consent quality, breach notification timing, and grievance redressal mechanisms. Several mid-sized fines (INR 5-25 crore range) were issued for missing or non-functional grievance redressal.

Larger fines are anticipated as the DPB completes its first investigations of SDFs. The maximum INR 250 crore penalty per breach has not yet been imposed but is reserved for systematic violations affecting large numbers of data principals.

The DPB is more litigation-focused than the PDPC or the EDPB; it is an adjudicatory body. Operators who engage early and constructively in proceedings tend to settle for substantially reduced penalties.

External references

The DPDP Act 2023 full text is at meity.gov.in/dpdp-act-2023. The Digital Personal Data Protection Rules 2026 are at the same MeitY portal. Data Protection Board notices and decisions are published at dpb.gov.in (note: as of mid-2026 the official site is in active build-out).

Comparison: DPDP vs PDPA vs GDPR for scrapers

Dimension	DPDP India	PDPA Singapore	GDPR EU
Default basis	Consent	Consent (with carve-outs)	Lawful basis (six options)
Public data carve-out	Narrow (data principal disclosed)	Broad (generally available)	Narrow
Legitimate Interests	Limited (Section 7 list)	Yes (since 2020)	Yes (Article 6(1)(f))
Cross-border transfer	Whitelist (operator-friendly)	Comparable protection	SCCs / adequacy
Right to erasure	Yes (on consent withdrawal)	Limited	Yes (Article 17)
DPO mandatory	For SDFs only	All organisations	Conditional
Maximum fine	INR 250 crore (~USD 30M)	SGD 1M or 10% turnover	EUR 20M or 4% turnover
Children’s data extra	Verifiable parental consent	Standard rules	Article 8 conditions
AI training friendly	Not articulated	DIP since 2024	EU AI Act layers on
Adjudicatory body	DPB (litigation-style)	PDPC (regulator-style)	National DPAs (regulator-style)

The DPDP is broadly stricter than PDPA on default basis but more flexible on cross-border transfer. It is comparable to GDPR on overall stringency but with different operational textures.

A worked example: scraping Indian ecommerce listings

A scraper collects publicly available product listings from major Indian ecommerce sites (Flipkart, Myntra, Meesho, Amazon India). The dataset includes product name, price, seller name, seller location, and seller rating.

Classification: seller name and location are personal data if the seller is an individual (many Indian ecommerce sellers are sole proprietors). Product name and price are not personal data.

Basis: the publicly-available carve-out applies if the seller voluntarily made their information public on the platform. Most platform terms require sellers to display their contact information. A defensible basis exists.

Notice: a public privacy notice on the scraper’s website describing collection, purpose, retention.

Grievance: an easily accessible grievance redressal mechanism (preferably an inbox, ideally a portal).

Cross-border: if data is transferred to non-restricted countries (US, EU, Singapore), no additional mechanism required as of mid-2026.

Outcome: defensible posture, with low ongoing overhead, with the publicly-available basis documented and a grievance mechanism live.

For a parallel ecommerce-targeted scrape, see scraping Flipkart India product data.

Special cases: AI training and political data

The DPDP Act does not yet have a specific AI training carve-out or framework. The current operator posture is to obtain consent or rely on Section 7 legitimate uses where possible, and to maintain documented training data manifests in anticipation of future rules.

Political data (party affiliation, voting behaviour, election-related profiling) is treated with extra caution by the DPB, with multiple early enforcement actions targeting election-time profiling operations. Scrapers should avoid political data entirely unless they have a defensible democratic-interest basis, which is hard to construct.

FAQ

Is publicly available data exempt from DPDP?
Only if the data was made publicly available by the data principal themselves or by a person obligated under law. Third-party publication does not qualify.

Do I need consent to scrape Indian data?
The default is yes. The Section 7 legitimate uses list is narrow and rarely applies to scraping operations.

Does DPDP apply if I am outside India?
Yes if your processing relates to offering goods or services to data principals in India. Extraterritorial reach is similar to GDPR.

Do I need a local DPO?
Only if classified as a Significant Data Fiduciary. SDFs must appoint an India-based DPO.

What is the cross-border transfer rule?
A positive whitelist model: transfer is permitted by default to any country not on the government’s restricted list. Major scraping destinations are unrestricted as of mid-2026.

Extended DPDP Act enforcement analysis

The Digital Personal Data Protection Act 2023 received presidential assent in August 2023. The Data Protection Board of India was constituted in 2024, and the Act’s substantive provisions came into force in phased fashion through 2025 and 2026. The DPDP rules notified in early 2025 fleshed out consent manager registration, breach notification timelines, and cross-border transfer mechanics.

The Act’s territorial reach (Section 3) covers processing of digital personal data in India, plus processing outside India if it relates to offering goods or services to data principals in India. The latter prong directly captures scrapers harvesting India-resident data from anywhere in the world.

The lawful-basis architecture differs from GDPR. The DPDP recognises two pathways. First, consent under Section 6, which must be free, specific, informed, unconditional, and unambiguous, and given through clear affirmative action. Second, certain legitimate uses under Section 7, including performance of state functions, employment, medical emergencies, disaster response, and certain specified purposes. There is no general legitimate-interest catch-all of the GDPR Article 6(1)(f) kind.

For scrapers the practical implication is that consent is the dominant pathway, and consent for third-party scraping is rarely available. The narrow Section 7 routes do not generally cover commercial scraping. Operators must therefore consider whether DPDP applies and design accordingly.

Implementation patterns for India-touching scraping

A 2026 DPDP-aware scraping pipeline should implement eight controls.

1. Identify India-resident data principals at ingest.
2. Distinguish between processing under consent and under Section 7 legitimate use, with documentation per record.
3. Honour withdrawal of consent and erasure requests.
4. Maintain a grievance officer and process within the statutory time limit.
5. Apply notified-country transfer rules for cross-border movement.
6. Apply enhanced safeguards for children’s data (under 18) including verifiable parental consent.
7. Maintain breach notification within the prescribed window to the DPB.
8. Maintain a data protection impact assessment for significant data fiduciary status if applicable.

Code pattern: India identification and consent gate

import re

IN_PHONE = re.compile(r"\+?91[\s-]?\d{5}[\s-]?\d{5}")
IN_DOMAINS = {"in", "co.in", "org.in", "ac.in", "gov.in"}

def is_india_subject(record):
    if IN_PHONE.search(record.get("text", "")):
        return True
    if any(record.get("email","").endswith("." + d) for d in IN_DOMAINS):
        return True
    if record.get("country_iso") == "IN":
        return True
    return False

def consent_gate(record):
    if not is_india_subject(record):
        return True
    if record.get("consent_token"):
        return True
    if record.get("section_7_basis") in {"employment", "medical_emergency", "disaster", "state_function"}:
        return True
    return False

Comparison: DPDP vs GDPR for scrapers

Question	India DPDP	EU GDPR
Lawful bases	Consent plus narrow Section 7 list	Six bases including legitimate interest
Territorial reach	India processing plus targeting	EU processing plus targeting plus monitoring
Children threshold	Under 18	Under 16 default, member states can lower to 13
Cross-border transfer	Notified country list	SCCs, adequacy, derogations
Max fine	INR 250 crore	EUR 20 million or 4 percent global turnover
Breach notification	DPB notification window	72 hours to DPA, individuals where high risk
Data principal rights	Access, correction, erasure, grievance, nominate	Access, rectification, erasure, restriction, portability, object

Additional FAQ

Does DPDP have a legitimate-interest catch-all?
No. Section 7 specifies discrete legitimate uses. Commercial scraping of public data does not fit any listed category.

What is a Significant Data Fiduciary?
A class designated by the central government based on volume, sensitivity, risk to electoral democracy or public order, sovereignty, or India’s security. Significant Data Fiduciaries face enhanced obligations.

How does DPDP treat AI training data?
The Act does not have AI-specific provisions, but consent or a Section 7 ground is required for any personal data processing including training.

What happens to non-personal data?
DPDP regulates only digital personal data. A separate non-personal data framework has been discussed but not enacted as of 2026.

The DPDP Act’s consent architecture

The DPDP Act places consent at the centre of the lawful processing analysis. Section 6 specifies that consent must be free, specific, informed, unconditional, and unambiguous, and must be given through clear affirmative action. The Act introduced the concept of a Consent Manager (Section 6(7)), a registered intermediary that can manage consent on behalf of data principals.

For scrapers the consent requirement is generally not satisfiable. Third-party scraping involves no direct interaction with the data principal, so affirmative consent cannot be obtained. The consent pathway is therefore typically unavailable for commercial scraping.

The Consent Manager mechanism is a 2026 innovation that may eventually create new pathways. A data principal who has consented through a Consent Manager can have that consent presented to downstream services. The infrastructure is still maturing, and broad scraper-friendly consent flows have not emerged. Scrapers should monitor the development but should not rely on it for current operations.

Section 7 legitimate uses in detail

Section 7 of the DPDP Act lists the legitimate uses for which processing may occur without consent. The list is closed, meaning it cannot be expanded by regulator interpretation. The legitimate uses include: processing for state functions; medical emergency; disaster response; employment-related processing; and a few specific public interest categories.

Critically the list does not include a general legitimate-interest catch-all comparable to GDPR Article 6(1)(f). This makes the DPDP regime more restrictive for scrapers than the GDPR. A scrape that would pass an LIA under GDPR may have no available basis under DPDP.

The 2025 DPDP Rules provided some additional clarification on Section 7 application. The rules tightened the definition of employment-related processing and specified that the disaster response basis is limited to time-bound emergencies declared by competent authority. Neither change benefits scrapers.

Significant Data Fiduciary obligations

Section 10 of the DPDP Act allows the central government to designate certain data fiduciaries as Significant Data Fiduciaries (SDFs) based on volume, sensitivity, risk to electoral democracy, public order, sovereignty, or India’s security. SDFs face enhanced obligations including DPIA, data protection officer designation, and independent audit.

The 2025 government notification of initial SDF categories included social media platforms above thresholds, e-commerce platforms above thresholds, and certain AI service providers. Future notifications may extend to data brokers and aggregators.

For scrapers the practical implication is that operating at scale in India increases the risk of SDF designation. A scraper crossing the volume threshold or feeding sensitive AI applications should plan for SDF obligations. The compliance uplift is non-trivial, including a designated DPO, mandatory DPIAs, and annual audits.

Cross-border transfer under DPDP

The DPDP Act takes a different approach to cross-border transfer than GDPR. Section 16 permits transfer to any country except those specifically restricted by the central government through notification. The starting position is permissive, with restrictions added as needed.

The 2025 DPDP Rules clarified the transfer framework. As of 2026 only a small number of countries are on the restricted list, and the criteria for addition are based on national security and public order rather than data protection adequacy.

For scrapers the implication is that exporting India-resident data to most jurisdictions is permissible under the DPDP, but the source-side consent or Section 7 basis must still be established. The DPDP transfer rules do not create a lawful basis where none otherwise exists.

Next steps

The fastest path to DPDP compliance is to publish a privacy notice and grievance redressal mechanism, document the basis (preferably the narrow publicly-available carve-out where it applies), and prepare for SDF designation if your volume warrants it. For broader Asia compliance, head to the DRT compliance hub and pair this with the PDPA Singapore guide.

This guide is informational, not legal advice.

Scrapybara vs Browserbase for agentic workflows

The Scrapybara vs Browserbase decision has become the cleanest fork in the road for any team building agentic workflows in 2026. Both are managed cloud platforms designed for AI agents to drive real browsers and operating systems. They overlap on the surface and diverge significantly in philosophy. Browserbase is a browser cloud, period. Scrapybara is a virtual desktop cloud that happens to include a browser.

That difference shows up everywhere. This guide compares both platforms through the lens of an engineer building a scraping or research agent in 2026, with code, benchmarks, and a clear picture of which one fits which problem.

What each platform actually is

Browserbase gives you a Chromium browser session over a Playwright-compatible WebSocket endpoint. You write Playwright or Stagehand code, you connect to a managed session, you scrape. The platform handles fingerprinting, proxies, CAPTCHA solving, and observability. Browser-only.

Scrapybara gives you a full Ubuntu desktop in the cloud with a browser, a terminal, a file system, and X11. Your agent can launch any application, not just a browser. The platform exposes a Computer Use API directly compatible with Anthropic’s Claude Computer Use and OpenAI’s Operator.

If your agent only needs the web, Browserbase is the leaner choice. If your agent needs to download a file, manipulate it in a desktop application, or run a CLI tool, Scrapybara is the only real option.

Pricing in 2026

Both are usage-based but with different units.

Browserbase:

Plan	Cost	Browser-minutes
Developer	$39/mo	5,000 included, then $0.0078/min
Startup	$399/mo	70,000 included, then $0.0057/min

Scrapybara:

Plan	Cost	Compute-hours
Hobby	Pay-as-you-go	$0.10/hour
Pro	$99/mo	100 hours included, then $0.06/hour
Scale	Custom	Custom

A like-for-like comparison: 1000 sessions averaging 3 minutes each.

Platform	Compute cost	Proxy cost (residential)	Total
Browserbase Developer	$23.40	$4.50	$27.90
Browserbase Startup	$17.10	$4.50	$21.60
Scrapybara Hobby	$5.00	$4.50 (BYO)	$9.50
Scrapybara Pro	$3.00	$4.50 (BYO)	$7.50

Scrapybara is meaningfully cheaper on compute. Browserbase costs more but includes residential proxies, CAPTCHA solving, and the Stagehand framework as native primitives.

Mental model in one sentence each

Scrapybara is “give me a Linux box my agent can drive.” Browserbase is “give me a Chrome tab my agent can drive.”

If you find yourself wishing the agent could apt install ffmpeg and then run a CLI tool on a downloaded video, Scrapybara fits. If you find yourself wishing the agent could click around five sites and emit JSON, Browserbase fits.

Setup speed

Both platforms ship in minutes.

Browserbase:

import { Browserbase } from "@browserbasehq/sdk";
import { chromium } from "playwright";

const bb = new Browserbase({ apiKey: process.env.BROWSERBASE_API_KEY! });
const session = await bb.sessions.create({ projectId: process.env.BROWSERBASE_PROJECT_ID! });
const browser = await chromium.connectOverCDP(session.connectUrl);
const page = browser.contexts()[0].pages()[0];

await page.goto("https://example.com");
console.log(await page.title());

await browser.close();

Scrapybara:

from scrapybara import Scrapybara

client = Scrapybara(api_key="scrapy_...")

instance = client.start_ubuntu()
instance.bash(command="echo hello")
instance.computer.screenshot()  # returns base64 PNG of the desktop
instance.browser.start()
instance.browser.goto("https://example.com")
title = instance.browser.evaluate("document.title")
print(title)

instance.stop()

Browserbase ships a Playwright-compatible API, so any existing Playwright code drops in. Scrapybara ships a richer API with desktop primitives, but your existing Playwright code needs adaptation.

Latency to first action

Cold-start times measured March 2026:

Platform	Cold start	Warm	First navigation
Browserbase	2.1 s	0.4 s	0.8 s after start
Scrapybara Ubuntu	8.4 s	2.1 s	1.4 s after start

Browserbase wins on latency because it boots only Chromium. Scrapybara boots a whole desktop. For pure browser workloads, the latency cost on Scrapybara is real. For multi-app workloads, the latency is amortized over a longer session.

Computer Use integration

This is where Scrapybara pulls ahead for agentic workflows.

Scrapybara is built specifically as a Computer Use target. Anthropic’s Claude Computer Use API and OpenAI’s Operator both plug into Scrapybara as the underlying compute environment.

from scrapybara import Scrapybara
from anthropic import Anthropic

client = Scrapybara(api_key="scrapy_...")
instance = client.start_ubuntu()
anthropic = Anthropic()

response = anthropic.beta.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=4096,
    tools=[{
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px": 1024,
        "display_height_px": 768,
    }],
    messages=[{
        "role": "user",
        "content": "Open the browser, search Google for 'browserbase pricing', and return the first result URL."
    }],
    betas=["computer-use-2024-10-22"],
)

# loop: feed Claude's tool calls into instance.computer.* and feed back

Browserbase, by contrast, supports Computer Use only through the Stagehand agent primitive, and only for the browser. If your agent needs to open Excel, manipulate files in a terminal, or interact with desktop apps, Browserbase cannot help.

For more on Computer Use vs Operator, see our OpenAI Operator vs Anthropic Computer Use comparison.

Browser primitive depth

Where Browserbase is more polished is the browser experience. Stagehand is theirs, and it shows.

Browserbase + Stagehand:

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const stagehand = new Stagehand({ env: "BROWSERBASE", modelName: "gpt-4o-mini" });
await stagehand.init();
const page = stagehand.page;
await page.goto("https://news.ycombinator.com");

const top = await page.extract({
  instruction: "Top 5 stories with title, score, submitter",
  schema: z.object({
    stories: z.array(z.object({ title: z.string(), score: z.number(), submitter: z.string() })),
  }),
});
console.log(top);
await stagehand.close();

Scrapybara has a browser API but no equivalent of Stagehand’s extract. You bring your own LLM extraction layer.

For the Stagehand vs Playwright story, see Stagehand vs Playwright for AI scraping.

Side-by-side comparison

Dimension	Scrapybara	Browserbase
Compute target	Full Ubuntu desktop	Chromium browser only
Native API surface	Browser, terminal, file system, GUI	Browser only
Computer Use support	First-class for Claude and Operator	Browser-only via Stagehand agent
AI extraction primitives	BYO	Stagehand native
Proxy support	BYO	Built-in residential and stealth
CAPTCHA solving	BYO	Built-in for major types
Session replay	Yes (video plus event trace)	Yes (video plus event trace)
Cost per 1000 short sessions	$7 to $10	$20 to $28
Best fit	Agents that need full OS access	Browser-only AI workflows

The two products are complementary more than competitive. Scrapybara is the right pick when your agent needs to do anything outside a browser. Browserbase is the right pick when your work fits in a browser and you want the most polished AI-native experience.

Computer Use action loop on Scrapybara

The actual loop that drives Claude Computer Use against a Scrapybara instance is simple but worth seeing in full.

def computer_action(action_input):
    if action_input["action"] == "screenshot":
        return instance.computer.screenshot()
    elif action_input["action"] == "left_click":
        x, y = action_input["coordinate"]
        return instance.computer.left_click(x=x, y=y)
    elif action_input["action"] == "type":
        return instance.computer.type(text=action_input["text"])
    elif action_input["action"] == "key":
        return instance.computer.key(text=action_input["text"])
    # ... more actions

messages = [{"role": "user", "content": "Open the browser and search Hacker News"}]
while True:
    resp = anthropic.beta.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        tools=[{"type": "computer_20241022", "name": "computer",
                "display_width_px": 1024, "display_height_px": 768}],
        messages=messages,
        betas=["computer-use-2024-10-22"],
    )
    if resp.stop_reason == "end_turn":
        break
    for block in resp.content:
        if block.type == "tool_use":
            result = computer_action(block.input)
            messages.append({"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": [{
                "type": "tool_result", "tool_use_id": block.id, "content": result,
            }]})

The loop is roughly 30 lines and works for any Computer Use task. Browserbase has an analogous loop but only for browser actions.

Real production patterns

We ran two pipelines in parallel to compare.

Pipeline A: scrape Lazada and Shopee product listings, extract to Postgres. Pure browser work.

Pipeline B: download invoice PDFs from a supplier portal, extract line items with a vision model, push to a procurement system. Needs file download and PDF processing.

Pipeline A on Browserbase + Stagehand:

• Time to working code: 2 hours
• Cost per 1000 products: $6 (compute) + $4 (residential proxy) + $2 (LLM)
• Maintenance burden: low

Pipeline A on Scrapybara:

• Time to working code: 4 hours (had to wire LLM extraction layer)
• Cost per 1000 products: $4 (compute) + $4 (BYO proxy) + $2 (LLM)
• Maintenance burden: medium

Pipeline B on Browserbase: not feasible. The browser cannot natively process the downloaded PDFs.

Pipeline B on Scrapybara:

• Time to working code: 5 hours
• Cost per 1000 invoices: $9 (compute, longer sessions) + $3 (proxy) + $5 (vision model)
• Maintenance burden: medium

Detailed pipeline metrics

The pipelines above ran for 30 days each. Aggregate numbers:

Metric	Pipeline A on Browserbase	Pipeline A on Scrapybara	Pipeline B on Scrapybara
Pages or invoices processed	90,000 products	90,000 products	12,000 invoices
Total cost	$1,080	$720	$204
Per-unit cost	$0.012	$0.008	$0.017
Median latency	7.4 s	9.1 s	38 s
Success rate	97.4%	95.1%	91.2%
On-call incidents	1	4	5

Browserbase’s higher cost was offset by lower incident count. Scrapybara’s lower cost came with more operational overhead because the BYO proxy layer needed its own monitoring. The pipeline B numbers are for a fundamentally different workload (PDF processing) where Scrapybara was the only option.

Where each platform showed weakness

Browserbase’s main weakness was the proxy markup. Halfway through the test, we switched the highest-volume target to BYO proxies through the session API, which dropped the proxy cost line by roughly 60 percent. The platform supports BYO but the docs do not lead with it.

Scrapybara’s main weakness was the lack of an extraction primitive. We had to wire OpenAI structured output ourselves, which added roughly two hours of development time per new target. The Scrapybara team has hinted at native extraction primitives in 2026 H2 but as of writing it is still BYO.

Adding proxies

Both platforms support proxies, but the integration depth differs.

Browserbase ships first-party residential proxies billed at $8/GB and stealth (datacenter) at $0.30/GB. Set them at session creation:

const session = await bb.sessions.create({
  projectId: process.env.BROWSERBASE_PROJECT_ID!,
  proxies: true, // or pass an explicit proxy config
});

Scrapybara expects you to bring your own. Configure inside the instance:

instance.browser.start(proxy={"server": "http://proxy.example.com:8000", "username": "u", "password": "p"})

For ASEAN scraping where mobile carrier IPs matter, Singapore mobile proxy plugs into either platform with one config block.

Geographic coverage

Browserbase: US East, US West, EU West, APAC South (Singapore). Compute always in one of these regions; proxies cover roughly 195 countries.

Scrapybara: US East, US West, EU West. APAC region is on the roadmap as of mid-2026. For Asia-targeted scraping, latency is higher than Browserbase.

CAPTCHA story in detail

Browserbase ships built-in solvers for reCAPTCHA v2/v3, hCaptcha, and Cloudflare Turnstile, with quoted success rates of 88 to 95 percent depending on type.

Scrapybara has no built-in CAPTCHA solving. You wire 2Captcha, CapSolver, or a similar service. The integration takes maybe 50 lines of code, but it is friction the Browserbase user does not have.

For teams that scrape CAPTCHA-heavy targets (LinkedIn, Indeed, several banking portals), Browserbase’s built-in solver is a real time-saver. For targets without CAPTCHAs (most B2B SaaS apps you have legitimate credentials for), the difference is irrelevant.

Observability

Both ship session replay with video and event traces. Browserbase’s UI is more polished and the search across sessions is faster. Scrapybara’s replay shows the full desktop, which is more useful when your agent uses non-browser apps.

Both export OpenTelemetry spans for trace correlation with your existing observability stack.

Security and isolation

Browserbase sessions run in ephemeral containers with full process isolation. Cookies, storage, and any cached state are wiped at session end unless you opt into context persistence. Network egress is funneled through the platform’s IP pool by default.

Scrapybara instances are full Ubuntu VMs with disk persistence per instance ID. This is more powerful and more dangerous: a compromised agent that gets shell access on a Scrapybara instance can persist files there, install software, and reuse the state across runs. For sensitive workloads, treat Scrapybara instances as ephemeral and tear them down explicitly.

Both platforms hold SOC 2 Type II reports as of 2026 and offer DPAs for GDPR-relevant workloads. Neither is yet HIPAA-eligible, so healthcare scraping requires self-hosting.

SDKs and language ecosystem

Browserbase’s primary SDK is TypeScript. The Python SDK exists but lags by roughly one release. For TypeScript-heavy teams, the platform feels native; for Python teams, less so.

Scrapybara’s primary SDK is Python. The TypeScript SDK exists but covers fewer features. For Python-heavy teams (most data science and AI shops), this is the natural fit.

If your team is split or polyglot, Browserbase wins because its WebSocket endpoint speaks any Playwright client. Scrapybara is more SDK-locked because the Computer Use primitives are first-class only in their SDKs.

Production recommendations

Use Browserbase if:

• Your scraping is browser-only
• You want Stagehand as your AI primitive layer
• You value built-in CAPTCHA solving
• You prefer a TypeScript-first SDK

Use Scrapybara if:

• Your agent needs file system, terminal, or desktop access
• You are running Claude Computer Use or OpenAI Operator
• You want lower per-hour compute cost and BYO proxy
• You prefer a Python-first SDK

The mature pattern in 2026 is to use both: Browserbase for the high-volume browser scraping, Scrapybara for the agent workflows that go beyond the browser. They cost together about what one Browserbase Startup plan costs, and you cover both shapes of work.

For broader context on the agentic browser space, see our agentic browser revolution guide.

Decision matrix

Your situation	Pick
Browser-only scraping, want fastest path	Browserbase + Stagehand
Browser-only scraping, want lowest cost	Self-hosted Playwright (not Scrapybara, not Browserbase)
Agent needs file system or terminal	Scrapybara
Running Claude Computer Use or OpenAI Operator	Scrapybara
TypeScript-first team, multi-site scraping	Browserbase
Python-first team, complex multi-app workflow	Scrapybara
Need built-in CAPTCHA solving	Browserbase
Need built-in residential proxies, simple billing	Browserbase
Need to install custom CLI tools at runtime	Scrapybara
One-off prototype, single site	Either, slight Browserbase edge

Frequently asked questions

Can I run my own LLM extraction prompts on Scrapybara?
Yes. Scrapybara just exposes the underlying compute. Wrap your favorite LLM (OpenAI, Anthropic, local) and call it from your agent code.

Are there any Scrapybara features Browserbase has copied or is likely to copy?
Both platforms ship session replay, both ship persistent contexts. Browserbase has hinted at adding limited terminal access on Scale plans but has not shipped. Scrapybara has hinted at native extraction primitives. Convergence is happening slowly.

Does Browserbase have a Python SDK?
Yes, but the TypeScript SDK is more feature-complete. For the latest features, TypeScript is the better choice in early 2026.

Can Scrapybara run alongside other browser frameworks?
Yes. The Scrapybara browser is a real Chromium that speaks CDP. You can drive it from Playwright, Puppeteer, or any browser automation library.

What about Hyperbrowser, Steel.dev, and other competitors?
Hyperbrowser is closest to Browserbase in shape but smaller. Steel.dev sits between Browserbase and Scrapybara on capabilities. Both are valid alternatives if pricing or feature set fits better.

How does authentication work for sites that need login?
Both platforms support persistent contexts that store cookies and localStorage. Save once after manual login, reload on each session. For higher-throughput pipelines, store the context state encrypted in your secret store.

Can I expose a Scrapybara instance to the public internet?
Not directly. The instance is private to the Scrapybara network. To expose a service running on the instance, tunnel through ngrok or a similar reverse proxy from inside the instance.

What are the per-instance resource limits on Scrapybara?
Default Hobby tier: 2 vCPU, 4 GB RAM, 20 GB disk. Pro: 4 vCPU, 8 GB RAM, 50 GB disk. Scale: negotiated. For most browser-driving tasks, Pro is sufficient. For video processing or large data jobs, request a custom config.

Can either platform handle file uploads to a target site?
Both can. Browserbase exposes the standard Playwright setInputFiles API. Scrapybara supports the same plus drag-and-drop from the desktop file system.

What about secrets management?
Neither platform provides a secrets vault. Pass secrets via environment variables to your code, or pull from your own secrets manager (AWS Secrets Manager, HashiCorp Vault, Doppler) before passing to the platform.

Can I use both platforms in the same workflow?
Yes. A common pattern is to use Browserbase for the high-volume scraping leg and Scrapybara for the post-processing leg that involves file manipulation. They are complementary by design.

Common production gotchas

For Browserbase: forgetting to release sessions burns minutes; mismatched proxy and target region triggers cloaking; the LLM bill hides separately on your OpenAI account.

For Scrapybara: the desktop is a real Ubuntu so resource leaks (lingering processes, large temp files) accumulate across long-lived instances; explicitly stop instances when done; mount your file output to a known location for export rather than relying on the temp file system.

For both: session replay retention is finite. Export critical traces if you need them long-term. The platforms do not retain everything indefinitely.

Six-month verdict

After running both side by side for six months: Browserbase wins for any team where 100 percent of work is browser-based. The polish and Stagehand integration are worth the premium. Scrapybara wins for any team where even 20 percent of work goes outside the browser, because that 20 percent is otherwise blocking. The right answer for many teams is to use Browserbase as the default and reach for Scrapybara when the task explicitly needs OS access.

If you are deciding between agentic platforms, the AI modern scraping category has more reviews and head-to-head comparisons that can help anchor the decision.