Category: Uncategorized

Model Context Protocol (MCP) for data engineers in 2026

MCP for data engineers has shifted from a curiosity to a foundational protocol in eighteen months. Anthropic announced the Model Context Protocol (MCP) in November 2024 as an open standard for connecting LLMs to external data sources and tools. By mid-2026, MCP is supported across Claude Desktop, Claude Code, ChatGPT, Cursor, Windsurf, Continue, Sourcegraph Cody, and a long list of independent agent frameworks. For data engineers running scraping, ETL, and RAG pipelines, MCP changed how agents access data, how data engineering work surfaces to non-engineers, and how downstream products integrate with proprietary datasets. This guide walks through what MCP actually is, the server and client architecture, worked Python and TypeScript implementations for common data-engineering use cases, deployment patterns, and where MCP fits versus other patterns like function calling or RAG-only.

The audience is the data engineer or platform team responsible for making proprietary or scraped data available to AI agents in 2026.

What MCP actually is and is not

MCP is a JSON-RPC 2.0 based protocol that defines how an MCP host (typically an AI agent runtime like Claude Desktop) discovers and calls capabilities exposed by an MCP server. The capabilities come in three classes: tools (callable functions), resources (read-only data), and prompts (reusable prompt templates).

The protocol is intentionally minimal. It does not specify the LLM. It does not specify the transport at the application layer (the spec defines stdio, HTTP, and SSE transports). It does not require Anthropic infrastructure (the spec is open, the SDK is MIT-licensed). What it does is standardise the metadata format and request/response shape so that any client can talk to any server without bespoke integration.

For a data engineer, MCP is most useful as a way to expose datasets, query interfaces, and pipeline triggers to AI agents in a way that does not require rebuilding integration per agent platform.

For the broader agent landscape, see the agentic browser revolution and AI agents as web users.

The MCP architecture in three roles

MCP has three roles: host, client, server.

The host is the agent runtime. Claude Desktop is a host. Claude Code is a host. Cursor is a host. The host is responsible for managing user trust, presenting the agent’s context to the user, and routing tool/resource calls to the appropriate server.

The client is a session within a host that connects to one specific MCP server. A host can have many concurrent clients, each connected to a different server.

The server is the process that exposes capabilities. A server can expose any combination of tools, resources, and prompts. Servers can run as local subprocesses (stdio transport) or as remote services (HTTP/SSE transport).

For a data engineer, the server is the unit of work. You write a server. You publish it. Your agent users (or your customers’ agent users) install or connect to it.

A minimal MCP server for a scraping pipeline

A typical scraping pipeline exposes four operations: list known sources, trigger a scrape, retrieve scraped records, and check pipeline status. Here is a minimal Python implementation using the official anthropic-mcp Python SDK.

from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import Tool, TextContent, Resource
import asyncio
import json

server = Server("scraping-pipeline")

@server.list_tools()
async def list_tools():
    return [
        Tool(
            name="list_sources",
            description="List all known scraping sources.",
            inputSchema={"type": "object", "properties": {}},
        ),
        Tool(
            name="trigger_scrape",
            description="Queue a scrape for a specific source URL.",
            inputSchema={
                "type": "object",
                "properties": {
                    "source_id": {"type": "string"},
                    "max_pages": {"type": "integer", "default": 10},
                },
                "required": ["source_id"],
            },
        ),
        Tool(
            name="get_records",
            description="Fetch scraped records for a source within a date range.",
            inputSchema={
                "type": "object",
                "properties": {
                    "source_id": {"type": "string"},
                    "since": {"type": "string", "format": "date"},
                    "limit": {"type": "integer", "default": 100},
                },
                "required": ["source_id"],
            },
        ),
        Tool(
            name="pipeline_status",
            description="Show pipeline health and recent run summary.",
            inputSchema={"type": "object", "properties": {}},
        ),
    ]

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "list_sources":
        result = await db.list_sources()
        return [TextContent(type="text", text=json.dumps(result))]
    elif name == "trigger_scrape":
        job_id = await scheduler.enqueue(
            arguments["source_id"], arguments.get("max_pages", 10)
        )
        return [TextContent(type="text", text=f"Queued job {job_id}")]
    elif name == "get_records":
        records = await db.fetch_records(
            arguments["source_id"],
            arguments.get("since"),
            arguments.get("limit", 100),
        )
        return [TextContent(type="text", text=json.dumps(records))]
    elif name == "pipeline_status":
        status = await monitor.summary()
        return [TextContent(type="text", text=json.dumps(status))]

async def main():
    async with stdio_server() as (read, write):
        await server.run(read, write, server.create_initialization_options())

if __name__ == "__main__":
    asyncio.run(main())

The server is roughly 60 lines of code. An agent connected to it can list sources, trigger scrapes, fetch records, and check status without any agent-specific integration code.

Resources versus tools: when to use each

MCP servers can expose resources alongside tools. The distinction is intentional and matters for how agents reason.

Tools are imperative: they take arguments and return results. They are appropriate for actions (trigger a scrape, send an email, write a file).

Resources are declarative: they have a URI and a content type, and the host can read them on demand. They are appropriate for browsable content (a database table, a file in a blob store, a record from a CRM).

For a scraping pipeline, the typical pattern is:

Capability	Type	Why
trigger_scrape	Tool	It is an action with side effects
pipeline_status	Tool	It is a runtime query, not browsable
list_sources	Resource	Sources are a browsable list
get_records	Resource (per-source)	Records are browsable content

Mixing tools and resources gives the agent a richer mental model of your data surface. Tools-only servers feel like a set of CLI commands. Resource-rich servers feel like a database the agent can query.

Prompts: the under-used third capability

The third MCP capability, prompts, is the least understood. A prompt is a reusable template that a host can offer to its user. The user invokes it (typically via a slash command), and the host injects the rendered prompt into the conversation.

For a data engineer, prompts are useful for canonical workflows: “summarise yesterday’s scrape”, “diff today’s records against last week’s”, “draft a compliance report for source X”. The prompt definition lives on the server; the user invokes it by name; the host renders the templated content with arguments.

@server.list_prompts()
async def list_prompts():
    return [
        Prompt(
            name="summarise_yesterday",
            description="Summarise yesterday's scrape activity for the team.",
            arguments=[],
        ),
        Prompt(
            name="diff_records",
            description="Show changes in records since a specified date.",
            arguments=[
                PromptArgument(name="source_id", required=True),
                PromptArgument(name="since", required=True),
            ],
        ),
    ]

@server.get_prompt()
async def get_prompt(name: str, arguments: dict):
    if name == "summarise_yesterday":
        return GetPromptResult(
            messages=[
                PromptMessage(
                    role="user",
                    content=TextContent(
                        type="text",
                        text="Use the pipeline_status tool to get yesterday's "
                             "activity. Then summarise the new sources, "
                             "successful runs, and any failures.",
                    ),
                )
            ]
        )

Prompts close the loop: tools provide capability, resources provide browsable content, prompts provide canonical workflows.

Deployment patterns: stdio vs HTTP

MCP supports stdio and HTTP/SSE transports. The choice shapes deployment.

Stdio servers run as subprocesses spawned by the host. They are great for personal tools (the user installs the server and connects via Claude Desktop config). They are awful for shared infrastructure (every user runs their own instance, no shared state, no central observability).

HTTP/SSE servers run as long-lived services. They are great for shared infrastructure (one server, many users, shared state, central monitoring). They require authentication, networking, and operational ownership.

For a data engineering team, the typical deployment is HTTP/SSE behind your existing auth gateway. Add MCP to your existing service mesh; route /mcp/* to the MCP server; reuse your existing OIDC or token-based auth.

Transport	Best for	Setup time	Operational overhead
Stdio	Personal/desktop tools	Minutes	None
HTTP/SSE	Team/shared infrastructure	Hours	Standard service ops
Streamable HTTP (2025 addition)	Hybrid; better browser support	Hours	Standard service ops

The 2025 addition of streamable HTTP simplified browser-based hosts and is becoming the default for new HTTP servers.

MCP versus function calling: when to use each

Both MCP and function calling let an LLM invoke external capabilities. The difference is portability.

Function calling is per-LLM-provider. A tool defined for OpenAI’s function calling does not work with Anthropic’s tool use without translation. A change to one schema requires updates to all clients.

MCP is provider-neutral. A tool exposed via MCP works with any MCP-compatible host. The schema is declared once, used everywhere.

Dimension	Function calling	MCP
Portability	Per provider	Cross provider
Discovery	Static registration	Dynamic at session start
Resources	Not standardised	First-class
Prompts	Not standardised	First-class
Agent platform reuse	Low	High
Maintenance overhead	Per provider	Once
2026 ecosystem	Mature per provider	Rapidly growing cross provider

A data engineering team that picks function calling locks itself into one provider. A team that picks MCP gets cross-provider reach for slightly more upfront work.

Worked use case: RAG over scraped data via MCP

A common pattern in 2026 is to expose a RAG corpus over scraped data through an MCP server, so any agent host can query the corpus naturally. The architecture:

Scraping pipeline ingests source URLs, normalises HTML, embeds chunks, stores in a vector database.
MCP server exposes a search_corpus tool with arguments for query and filters.
Agent host connects to MCP server; user asks a question; agent calls search_corpus; corpus returns relevant chunks; agent synthesises an answer.

A minimal Python tool implementation:

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "search_corpus":
        query = arguments["query"]
        filters = arguments.get("filters", {})
        embedding = await embed(query)
        results = await vectordb.query(
            embedding, top_k=arguments.get("top_k", 5), filters=filters
        )
        return [TextContent(type="text", text=json.dumps(results))]

The MCP server is twenty lines. The vector database, embedder, and ingestion pipeline are independent. The server is the integration surface.

For the deeper RAG-over-scraped-data discussion, see RAG over scraped data production patterns and vector databases for scraping pipelines.

Security and trust model

MCP defines a trust boundary at the host-server connection. The server trusts the host to authenticate the user. The host trusts the server to honour its declared capabilities.

For HTTP transports, authentication is the server’s responsibility. The 2025 spec update added explicit guidance for OAuth 2.1 with PKCE, which is the recommended pattern for enterprise deployments. Bearer tokens work for service-to-service. Mutual TLS works for high-security environments.

Authorisation is the server’s responsibility. A server should enforce per-user permissions; the host’s user identity flows through the authentication layer. A scraping MCP server typically restricts trigger_scrape to authorised users while allowing get_records broadly.

Audit logging is the server’s responsibility. Every tool call should be logged with user identity, timestamp, arguments, and outcome. This is your defence against an agent calling the wrong thing on the wrong dataset.

Deployment checklist

Step	Owner	Done when
Define capabilities (tools, resources, prompts)	Engineering	Schema agreed, written
Implement server (stdio for prototype)	Engineering	Local Claude Desktop test passes
Migrate to HTTP for shared use	Engineering	Reachable behind auth gateway
Wire authentication (OAuth 2.1 / Bearer)	Security	Unauthenticated calls rejected
Wire authorisation (per-tool, per-user)	Security	Permission matrix enforced
Add audit logging	Engineering	Every call logged with identity
Document for users	Product	Runbook published
Publish for installation (registry / docs)	Marketing	Discoverable in MCP registry
Monitor and observe	Platform	Metrics dashboard live

External references

The MCP specification is at modelcontextprotocol.io. The Python SDK is at github.com/anthropics/python-mcp. The TypeScript SDK is at github.com/anthropics/typescript-mcp. Anthropic’s MCP servers reference at github.com/modelcontextprotocol/servers.

Comparison: MCP vs LangChain Tools vs OpenAI Function Calling

Dimension	MCP	LangChain Tools	OpenAI Function Calling
Cross-provider	Yes	Partial (with adapters)	No
Cross-host	Yes	LangChain runtime only	OpenAI agents only
Resource browsability	Yes	No	No
Prompts as first class	Yes	No	No
Streaming	Yes	Yes	Yes
Auth pattern	Spec-defined (OAuth 2.1)	App responsibility	OpenAI’s
2026 ecosystem maturity	High and growing	Mature within LangChain	Mature within OpenAI
Discovery	Dynamic at session	Static at runtime	Static per request

MCP is the cross-provider winner. LangChain Tools are the most flexible if you accept LangChain runtime lock-in. OpenAI function calling is the simplest if you only target OpenAI hosts.

FAQ

Do I need Anthropic infrastructure to use MCP?
No. The protocol is open, the SDKs are MIT-licensed, and any host can implement the protocol.

Can I expose existing REST APIs through MCP?
Yes. A thin MCP server can wrap any HTTP API and expose it as tools and resources.

How does MCP compare to OpenAPI?
MCP is at the agent integration layer; OpenAPI is at the HTTP API description layer. They are complementary; many MCP servers wrap OpenAPI-described services.

What is the right transport for production?
HTTP/SSE or streamable HTTP. Stdio is fine for personal/desktop scenarios, but production sharing needs HTTP.

How do I authenticate users on an HTTP MCP server?
OAuth 2.1 with PKCE is the recommended pattern. Bearer tokens work for service-to-service.

Extended MCP architecture analysis

The Model Context Protocol matured rapidly between its November 2024 launch and 2026. The protocol specification at v1.2 (early 2026) covers four primitive types, namely tools, resources, prompts, and sampling. For data engineers the tools and resources primitives are central. Tools expose callable functions to a model. Resources expose readable, addressable data.

A data-engineering MCP server typically wraps three layers. First, connection management (database connections, API clients, cache). Second, the tool surface (query, insert, transform, validate). Third, the observability surface (logs, metrics, request IDs).

The protocol is transport-agnostic. The two common transports are stdio (process-spawned servers, lowest latency) and SSE plus HTTP (network-deployed servers, fan-out across clients). For data pipelines stdio is preferred for local agents and SSE for shared infrastructure.

Production-ready MCP server pattern

from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp.types import Tool, TextContent
import asyncio
import asyncpg

server = Server("data-pipeline")
pool = None

@server.list_tools()
async def list_tools():
    return [
        Tool(
            name="run_query",
            description="Execute a read-only SQL query against the warehouse",
            inputSchema={
                "type": "object",
                "properties": {
                    "sql": {"type": "string"},
                    "limit": {"type": "integer", "default": 1000},
                },
                "required": ["sql"],
            },
        ),
        Tool(
            name="describe_table",
            description="Return schema for a warehouse table",
            inputSchema={
                "type": "object",
                "properties": {"table": {"type": "string"}},
                "required": ["table"],
            },
        ),
    ]

@server.call_tool()
async def call_tool(name, arguments):
    if name == "run_query":
        sql = arguments["sql"]
        if not sql.strip().lower().startswith("select"):
            return [TextContent(type="text", text="Read-only access. SELECT only.")]
        async with pool.acquire() as conn:
            rows = await conn.fetch(sql + f" LIMIT {arguments.get('limit', 1000)}")
        return [TextContent(type="text", text="\n".join(str(r) for r in rows))]
    if name == "describe_table":
        async with pool.acquire() as conn:
            rows = await conn.fetch(
                "SELECT column_name, data_type FROM information_schema.columns WHERE table_name = $1",
                arguments["table"],
            )
        return [TextContent(type="text", text="\n".join(f"{r['column_name']}: {r['data_type']}" for r in rows))]

async def main():
    global pool
    pool = await asyncpg.create_pool("postgresql://...")
    async with stdio_server() as (read, write):
        await server.run(read, write, server.create_initialization_options())

if __name__ == "__main__":
    asyncio.run(main())

Tool surface design patterns

Effective MCP tool surfaces follow five rules.

Read and write are separate tools. Never bundle them.
Every destructive tool requires a confirm token in the input schema.
Pagination is explicit (cursor or offset/limit) rather than streaming everything.
Errors return structured content with hints for the model on next steps.
Long-running tools return a job ID and a separate status tool checks progress.

Comparison: MCP transport choices

Transport	Latency	Fan-out	Best for
stdio	Low (process IPC)	One client	Local agents, dev tooling
SSE plus HTTP	Moderate (network)	Many clients	Shared infrastructure
WebSocket (proposed)	Low	Bidirectional	Real-time dashboards

Observability for MCP servers

Production MCP servers should emit four signals.

Request count by tool name.
Request latency histogram by tool name.
Error rate by tool name and error class.
Token consumption per request (input plus output).

A simple approach is OpenTelemetry plus a Prometheus exporter. The MCP request lifecycle maps cleanly onto OTel spans.

Additional FAQ

Is MCP only for Anthropic models?
No. The protocol is open and other model vendors have shipped MCP support.

Can MCP replace REST APIs?
For agent-facing surfaces yes. For human-facing surfaces REST and GraphQL remain better fits.

How do I version an MCP tool surface?
Use semantic versioning on the server, expose the version in the initialization handshake, and add new tools rather than mutating existing ones.

What about authentication?
Transport-level auth (TLS, mutual TLS, signed headers) is the current pattern. The protocol does not prescribe an auth scheme.

When MCP wins versus when it loses

MCP is not the right answer for every data engineering integration. The protocol shines when the consumer is a model or an agent, and loses when the consumer is a deterministic application or a high-throughput batch job.

MCP wins when the access pattern is exploratory, when the schema is not known in advance, when the operations involve natural language, and when the consumer needs metadata to interpret the data. Examples include a model querying a warehouse for ad-hoc analysis, an agent investigating a customer support issue, and a copilot helping an analyst draft a report.

MCP loses when the access pattern is fixed, when the schema is well-known, when throughput requirements are high, and when latency budgets are tight. Examples include an ETL pipeline ingesting transactions, a real-time dashboard refreshing every second, and a microservice serving a known query at high QPS. For these cases REST, GraphQL, or direct database access remain the right choice.

The decision rule for a data engineering team is to expose the warehouse via REST or GraphQL for application consumers, and to layer MCP on top for agent and copilot consumers. The two surfaces share the underlying connection management and data layer, but expose different abstractions to different audiences.

Tool surface design beyond the basics

A first-pass MCP tool surface tends to expose run_query and describe_table. A production-quality tool surface goes further. The patterns that ship in 2026 include search_tables (for discovery when the agent does not know table names), suggest_join (for analytical queries that span tables), explain_query (for surfacing query plans), and validate_data (for checking data quality assumptions).

Each additional tool reduces the number of round-trips the agent needs to complete a task. An agent that can search, describe, query, and validate in four tool calls is materially more capable than an agent that can only query. The design goal is to anticipate the agent’s needs and expose primitives that satisfy them.

A countervailing concern is tool sprawl. An MCP server with too many tools confuses the agent. The 2026 sweet spot is somewhere between five and twenty tools, with clear non-overlapping purposes. Beyond twenty tools the agent’s planning quality degrades.

Security model for MCP servers

An MCP server that exposes warehouse access creates a powerful attack surface. The security model must address authentication, authorisation, audit, and rate limiting.

Authentication establishes who is connecting. For stdio transport the parent process is implicitly trusted. For SSE plus HTTP transport the connection should require a token, ideally short-lived and tied to a specific agent identity.

Authorisation determines what the authenticated principal can do. The 2026 pattern is to map the agent’s permissions onto the same role-based model used for human users. An agent acting on behalf of an analyst inherits the analyst’s permissions, plus additional restrictions specific to agent traffic.

Audit captures every tool call with the principal, the arguments, the result summary, and the timestamp. The audit log is the forensic record for incident investigation. The 2026 best practice is to retain MCP audit logs for at least ninety days.

Rate limiting prevents runaway agents from consuming resources. Per-tool, per-principal, and per-server quotas should each be enforced. The 2026 default is conservative quotas that can be raised on request.

MCP versioning and evolution

MCP is a young protocol, and breaking changes are expected as it matures. The 2026 best practice for MCP server operators is to follow semantic versioning, expose the version in the initialization handshake, and support the previous major version for at least six months after a breaking change.

For tool surfaces the rule is to add new tools rather than mutate existing ones. A tool that needs a new argument should be deprecated and replaced with a v2 variant. Existing agents continue to use the v1 tool until they are updated.

For data shapes the rule is similar. A response schema that needs to add a field is safe. A response schema that needs to remove or rename a field requires a versioned response. Many MCP servers expose a content_version metadata field on responses to allow agents to detect and adapt.

Next steps

The fastest first step is to wrap your most-used internal data interface in a minimal MCP server, run it via Claude Desktop, and see how it changes the team’s interaction. The integration overhead is low, the leverage is high. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the agentic browser revolution and RAG over scraped data guides.

This guide is informational, not engineering or legal advice.

May 7, 2026

Scraping JavaScript-heavy SPAs with AI agents in 2026

Scraping SPA AI agents pipelines have become the dominant pattern for any modern web target because the entire ecommerce, SaaS, and content stack has converged on React, Vue, and Next.js. Server-rendered HTML is the exception; client-rendered, hydrated, lazy-loaded SPAs are the rule. Traditional fetch-and-parse scrapers crash on these targets. AI agents that drive a real headless browser succeed.

This guide covers the patterns that work in 2026 for scraping SPAs with AI agents. We cover detection, rendering strategy, wait conditions, structured extraction, proxy integration, and the edge cases that bite if you miss them. Code in Python and TypeScript throughout.

Why SPAs break traditional scrapers

A SPA serves a near-empty HTML shell that loads JavaScript bundles, calls APIs, and renders the actual content client-side. If you fetch the URL with requests or httpx, you get the shell. The data you want is generated milliseconds to seconds later by JavaScript that never ran in your scraper.

The fix is to render the page in a real browser. The fix used to be Puppeteer or Selenium with brittle wait conditions and per-site selector maintenance. In 2026, the better fix is to drive a browser with an AI agent that watches the page render in real time and decides when to extract.

Detecting SPA targets

Before reaching for an AI agent, check whether the target actually needs one. Many sites that look like SPAs in DevTools are actually server-rendered or hybrid.

Quick detection script:

import httpx
from bs4 import BeautifulSoup

def is_spa(url: str) -> bool:
    r = httpx.get(url, headers={"User-Agent": "Mozilla/5.0"})
    soup = BeautifulSoup(r.text, "html.parser")
    text_chars = len(soup.get_text(strip=True))

    has_react_root = bool(soup.find(id="root") or soup.find(id="__next"))
    has_app_div = bool(soup.find("div", attrs={"id": "app"}))
    has_low_text = text_chars < 1000

    return (has_react_root or has_app_div) and has_low_text

print(is_spa("https://www.lazada.sg/"))  # True
print(is_spa("https://news.ycombinator.com/"))  # False

If is_spa returns False, use a normal HTTP scraper. If True, you need a real browser.

The agentic browser pattern

The pattern that wins in 2026 looks like this:

Launch a headless Chromium with stealth defaults
Navigate to the target URL
Wait for visual stability (network idle plus a small grace period)
Take a screenshot and dump the rendered HTML
Pass both to an LLM with a strict JSON Schema
Validate the result, retry if needed

In code, with browser-use as the agent driver:

import asyncio
from browser_use import Agent, Browser, BrowserConfig, Controller
from langchain_openai import ChatOpenAI
from pydantic import BaseModel

class Product(BaseModel):
    title: str
    price: float
    currency: str
    in_stock: bool

controller = Controller(output_model=Product)

async def scrape_spa_product(url: str) -> Product:
    browser = Browser(config=BrowserConfig(
        headless=True,
        extra_chromium_args=["--disable-blink-features=AutomationControlled"],
    ))

    agent = Agent(
        task=(
            f"Visit {url}, wait for the product details to fully load, "
            f"and return the title, price (number), currency code, and stock status."
        ),
        llm=ChatOpenAI(model="gpt-4o-mini"),
        browser=browser,
        controller=controller,
        max_failures=3,
    )

    history = await agent.run()
    return Product.model_validate_json(history.final_result())

product = asyncio.run(scrape_spa_product("https://www.lazada.sg/products/example-12345.html"))
print(product)

The agent watches the page render and decides when to extract. No selector maintenance. No wait condition tuning per site.

For more on browser-use specifically, see our browser-use scraping guide.

SPA framework cheat sheet

Different frameworks fingerprint differently. Quick recognition guide:

Framework	Tells
Next.js (App Router)	`__next` div, `__next-build-id` meta, `_next/static/...` script URLs
Next.js (Pages Router)	`__NEXT_DATA__` script tag, `_next/static/chunks/pages/...`
Remix	`__remixContext` script, `data-route` on root
SvelteKit	`__sveltekit_` global, `data-sveltekit-` attrs
Nuxt	`__NUXT__` script, `_nuxt/...` script URLs
Astro	`astro-island` custom elements, mixed SSR with client islands
React + Vite	`<div id="root">` + `/src/main.tsx` script ref
Vue + Vite	`<div id="app">` + Vue devtools meta
Angular	`<app-root>` element, `ng-version` attr

Knowing the framework tells you whether the data is already in the initial HTML payload. Next.js with App Router and Remix often serve fully rendered HTML; Astro typically does too. SPA frameworks with strict client rendering (vanilla React + Vite, vanilla Vue) almost always need a real browser.

Wait conditions that actually work

The trickiest part of SPA scraping is waiting long enough for content to hydrate without waiting forever. Three patterns:

Network idle plus grace period. Wait for networkidle (no requests for 500ms) then sleep an additional 1-2 seconds. Catches most React Suspense boundaries.

DOM-stability detection. Watch the DOM mutation count, wait until it stabilizes for 1-2 seconds.

Sentinel selector. Wait for a known element that only appears after content loads (price, product image, review count). Most reliable when you know the site.

For agentic scraping, the agent handles this implicitly. You just give it enough time budget per page.

Stagehand example with explicit wait:

import { Stagehand } from "@browserbasehq/stagehand";

const stagehand = new Stagehand({ env: "LOCAL", modelName: "gpt-4o-mini" });
await stagehand.init();
const page = stagehand.page;

await page.goto("https://www.lazada.sg/products/example.html", { waitUntil: "networkidle" });
await page.waitForTimeout(1500);  // grace period for React hydration

const data = await page.extract({
  instruction: "Extract product title, price, currency, in-stock status",
  schema: z.object({ title: z.string(), price: z.number(), currency: z.string(), inStock: z.boolean() }),
});

When networkidle lies

networkidle is a useful default but it lies on three common patterns. Long-poll connections never go idle, so a chat widget keeps the network busy forever. Analytics beacons that fire every 5 seconds prevent idle from triggering. Web sockets that retry every few seconds also prevent the idle state.

The mitigation is to combine networkidle with a hard timeout cap and a DOM-stability check. If networkidle has not fired within 10 seconds but the visible content has stopped changing, extract anyway.

async function waitForContent(page: Page, timeoutMs = 15000) {
  const start = Date.now();
  let lastDomSize = 0;
  let stableCount = 0;
  while (Date.now() - start < timeoutMs) {
    const size = await page.evaluate(() => document.body.innerText.length);
    if (size === lastDomSize && size > 500) {
      stableCount++;
      if (stableCount >= 3) return;  // 1.5s of stability
    } else {
      stableCount = 0;
    }
    lastDomSize = size;
    await page.waitForTimeout(500);
  }
}

This pattern beats raw networkidle on roughly 30 percent of SPAs we tested.

Handling infinite scroll and lazy loading

Many SPAs render content lazily as the user scrolls. Two patterns to handle this.

Scroll loop. Scroll to bottom in a loop until the page height stops growing.

async function scrollUntilStable(page: Page, maxScrolls = 20) {
  let lastHeight = 0;
  for (let i = 0; i < maxScrolls; i++) {
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(1000);
    const height = await page.evaluate(() => document.body.scrollHeight);
    if (height === lastHeight) return;
    lastHeight = height;
  }
}

Intercept the underlying API. Many SPAs lazy-load by calling a paginated JSON endpoint. Open DevTools, find the call, and hit it directly. Skip the browser entirely. This is the highest-throughput pattern when it applies.

import httpx

async def fetch_lazada_listings(category: str, page: int):
    url = f"https://www.lazada.sg/api/listing?category={category}&page={page}"
    headers = {"x-csrf-token": "...", "User-Agent": "Mozilla/5.0"}
    async with httpx.AsyncClient() as c:
        r = await c.get(url, headers=headers)
        return r.json()

When you can find and hit the underlying API, do that. AI agents are the fallback when the API is hidden, signed, or rate-limited too aggressively for direct access.

Comparison of SPA scraping approaches

Approach	Cost per page	Reliability on changing layouts	Maintenance	Best fit
Direct API interception	$0.001	High	Medium	When you can find the API
Playwright with custom selectors	$0.005	Low	High	Stable known-shape sites
Stagehand `extract`	$0.04	High	Low	Long-tail SPA targets
browser-use full agent	$0.04	High	Low	Multi-step SPA flows
Operator/Computer Use	$0.20	Highest	Lowest	Hardest targets

For 80 percent of SPA scraping work in 2026, Stagehand’s extract primitive plus a 1.5-second grace period is the sweet spot. It is cheap enough to scale, reliable enough to ignore most layout changes, and easy enough to write that a junior engineer can ship a new scraper in an hour.

Hydration race conditions

A common bug: you take a screenshot at the wrong moment and the agent extracts placeholder data (“Loading…”, skeleton boxes, default values).

Two defenses:

First, validate the output. Reject any extraction where price equals zero or title contains “loading”. Retry with a longer wait.

def validate_product(p: dict) -> bool:
    if not p.get("title") or "loading" in p["title"].lower():
        return False
    if p.get("price", 0) <= 0:
        return False
    return True

Second, take two screenshots 500ms apart and compare. If they differ significantly, the page is still rendering. Wait, retry, repeat.

Adding proxy rotation

SPAs are typically served by sites with strong bot defenses (Cloudflare, DataDome, Akamai). Mobile or residential proxies are mandatory.

In browser-use:

from browser_use import Browser, BrowserConfig

browser = Browser(config=BrowserConfig(
    headless=True,
    proxy={"server": "http://proxy.example.com:8000", "username": "u", "password": "p"},
))

In Stagehand:

const stagehand = new Stagehand({
  env: "LOCAL",
  localBrowserLaunchOptions: {
    proxy: { server: "http://proxy.example.com:8000", username: "u", password: "p" },
  },
});

For ASEAN ecommerce SPAs (Lazada, Shopee, Tokopedia), Singapore mobile proxy carries real Singtel and StarHub IPs that avoid the data-center blocks these sites apply.

Structured extraction at the end

The right pattern is to drive the agent only to reach the right page and dump the rendered HTML, then run structured extraction on a cheaper model.

# step 1: navigate with agent
agent = Agent(task=f"Reach the product page at {url} and extract the full HTML",
              llm=ChatOpenAI(model="gpt-4o-mini"))
result = await agent.run()
html = await agent.browser.context.pages[0].content()

# step 2: cheap structured extraction
import json
from openai import AsyncOpenAI
client = AsyncOpenAI()
extract = await client.chat.completions.create(
    model="gpt-4o-mini",
    response_format={"type": "json_schema", "json_schema": {
        "name": "product",
        "schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "number"},
                "currency": {"type": "string"},
                "in_stock": {"type": "boolean"},
            },
            "required": ["title", "price", "currency", "in_stock"],
            "additionalProperties": False,
        },
        "strict": True,
    }},
    messages=[{"role": "user", "content": html[:200000]}],
)
product = json.loads(extract.choices[0].message.content)

This split typically cuts cost by half compared to one big agent loop. For more, see LLM extraction patterns.

Network interception for hidden data

When the underlying API is hidden but exists, intercept the network requests directly. Both Playwright and Stagehand expose a request/response listener that captures everything the browser fetches.

const apiResponses: Record<string, unknown> = {};

page.on("response", async (response) => {
  const url = response.url();
  if (url.includes("/api/product/") && response.headers()["content-type"]?.includes("json")) {
    try {
      apiResponses[url] = await response.json();
    } catch {}
  }
});

await page.goto(productUrl);
await waitForContent(page);

// apiResponses now contains the underlying JSON payloads

This pattern often gets you cleaner data than DOM extraction because the API payload is the source of truth that the SPA renders. Once you find the API, you can hit it directly and skip the browser entirely.

SPA scraping with vision-only extraction

For SPAs where the DOM is obfuscated (CSS-in-JS with random class names, shadow DOM components, canvas-rendered text), vision extraction can succeed where DOM extraction fails.

from openai import AsyncOpenAI
client = AsyncOpenAI()

async def extract_from_screenshot(png_b64: str, schema: dict) -> dict:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={"type": "json_schema", "json_schema": {"name": "x", "schema": schema, "strict": True}},
        messages=[{"role": "user", "content": [
            {"type": "text", "text": "Extract data from this screenshot per the schema"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{png_b64}"}},
        ]}],
    )
    return json.loads(resp.choices[0].message.content)

Vision extraction is roughly 3x more expensive per page than DOM extraction but works on a handful of sites where DOM extraction is essentially impossible. For more, see scraping with vision models 2026.

Production patterns

Three patterns separate hobby SPA scrapers from production ones.

First, cap per-page wall clock. Even agents can spin if the page is broken. Set timeout: 60000 and treat exceeded timeouts as failures.

Second, monitor cost per page. SPA scraping costs add up fast. Log tokens consumed per scrape and alert if a single page exceeds 30,000 tokens (likely a confused agent).

Third, keep a fallback. When the agent fails, fall through to a static Playwright scraper with cached selectors. Catches the cases where the agent is wrong and the deterministic code is right.

Cookie banners and modal interruptions

The single most common cause of stuck SPA scrapes is a cookie banner blocking the content. The agent sees the banner, the LLM does not understand to dismiss it, and the agent gives up.

Hardcode a banner-handling preamble in your task:

"If a cookie banner, age verification modal, region selector, or login prompt
is visible, dismiss it (click 'Accept all', 'I'm 18+', 'Close', or similar).
Then proceed with the main task."

This single instruction moves success rates on European retailer SPAs by 15 to 25 percentage points.

For Stagehand, you can also use act to dismiss known modals before extract:

await page.act("If a cookie banner is visible, click the 'Accept all' button");
const data = await page.extract({ instruction: "...", schema: ... });

Common SPA scraping pitfalls

A handful of failure modes worth memorizing.

The agent sees server-rendered placeholder content and extracts the placeholder. Mitigation: detect placeholders by content patterns (“loading”, skeleton boxes, default 0 values).

The page redirects to a login wall after a few page loads. Mitigation: rotate session cookies and IPs, or warm up the session with realistic browsing before scraping.

The site uses cursor-based pagination with opaque tokens. Mitigation: simulate the user click that triggers the next page rather than constructing the URL yourself.

The site delivers different markup based on user agent or viewport. Mitigation: use a realistic UA and a desktop viewport (1440×900), not the Playwright defaults.

The site lazy-loads images that block extraction because the LLM expects them. Mitigation: prefer text-only extractions, and if you need images, scroll first.

Real benchmarks on common SPAs

100 product pages each, GPT-4o-mini extraction:

Target	Approach	Success rate	Avg time per page
Lazada SG	browser-use + mobile proxy	98%	7.1 s
Shopee SG	Stagehand + mobile proxy	96%	6.4 s
Amazon US	Playwright + residential	94%	3.2 s
Best Buy	browser-use + residential	91%	8.5 s
Booking.com	Stagehand + residential	88%	12 s

For more on Lazada specifically, see our Lazada Thailand scraping guide. For Shopee, see Shopee Indonesia scraping.

Hydration timing across SPA frameworks

Different frameworks hydrate at different speeds. Average time from domcontentloaded to “fully interactive” on a typical product page:

Framework	Median hydration	p99 hydration
Next.js App Router (RSC)	250 ms	1.4 s
Next.js Pages Router	600 ms	2.8 s
Remix	350 ms	1.6 s
SvelteKit	200 ms	1.1 s
Nuxt 3	480 ms	2.4 s
React + Vite SPA	1.2 s	4.5 s
Vue + Vite SPA	1.0 s	4.2 s
Astro with islands	200 ms (mostly SSR)	1.0 s

For most production scrapers, a 2-second wait is sufficient. For React + Vite SPAs, bump to 4 seconds. The cost in latency is small compared to the cost in failed extractions.

Frequently asked questions

Can I scrape an SPA without a real browser?
Sometimes. If the SPA uses Next.js with getServerSideProps or React Server Components, the initial HTML may already contain the data you need. Check by curl-ing the URL. Otherwise, a real browser is required.

What about hydration mismatches?
Hydration mismatches happen when client-rendered HTML differs from server HTML. Wait until after hydration completes (typically 500-1500ms) before extracting.

How do I handle authentication on SPAs?
Most SPAs use cookie-based session auth. Log in once, save the storage state, replay on each scrape. Both Playwright and Stagehand expose storageState config for this.

Can I run an SPA scraper on AWS Lambda?
Cold starts kill latency. Lambda with the Chromium layer works for occasional jobs. For high throughput, run on Fargate or a dedicated VPS. Browserbase is the pay-per-minute alternative.

Why is my agent extracting “0” for prices?
Almost always a hydration timing issue. The agent extracted while the React app still showed the skeleton placeholder. Add a longer wait or a stability check before extraction.

How do I parallelize SPA scraping when each page takes 8 seconds?
Run multiple browser contexts in parallel within one Chromium process (cheaper than multiple browsers). Cap concurrency at roughly 1 per CPU core to avoid thrashing the browser’s rendering pipeline.

How do I handle SPAs that detect headless browsers?
Use headless=False with a virtual display (Xvfb) on Linux, or use Browserbase’s stealth mode. The single biggest tell is the navigator.webdriver flag, which Chromium sets to true in headless mode and most stealth plugins patch.

Can I use the Beautiful Soup HTML output from a Playwright render?
Yes. After page.content(), you have the rendered HTML as a string and can pass it to BeautifulSoup or lxml for traditional parsing. The combination of headless render plus traditional parser is a real production pattern.

What about WebSockets and Server-Sent Events?
Real-time data over WebSocket or SSE is harder to scrape because the data flows continuously. Use Playwright’s page.on('websocket') event listener to capture frames as they arrive, or intercept the underlying API connection.

How do I scrape an SPA behind a paywall I have access to?
Save the browser storage state (cookies, localStorage) after manual login, then load it on each scrape via storageState. Refresh the state when it expires.

Are there SPAs that just cannot be scraped reliably?
Yes. Sites with strong client-side encryption (some financial dashboards), sites that gate data behind interactive verification (real-time KYC), and sites with anti-replay tokens that bind to a specific browser fingerprint. For these, manual data export or partner APIs are the only realistic paths.

For broader patterns on the agentic browser stack, see our AI modern scraping category.

May 7, 2026

Building an ethics-first scraping policy for your team

Scraping ethics policy is the artefact that ties everything together: legal posture, technical controls, customer expectations, employee onboarding, and incident response. Most teams have informal practices but no written policy, and the gap shows up the first time something goes wrong (a regulator letter, a customer compliance question, a journalist inquiry, an internal escalation). A written policy is not paperwork. It is the document that prevents most of those failures from becoming crises. This guide walks through what a working scraping ethics policy contains, how to operationalise it across compliance regimes, the team workflow for adoption and maintenance, and a template structure your team can adapt this quarter.

The audience is the engineering lead, product owner, or compliance partner who needs to move from informal scraping practice to a defensible, written, lived policy.

Why a written policy matters operationally

Three reasons.

First, regulators ask for it. The EDPB, the CPPA, the PDPC, and the DPB all explicitly look for written policies during investigations. Their absence is itself evidence of insufficient organisational maturity, which weighs against you in penalty assessments. Their presence shifts the burden: investigators read your policy first and assess deviation.

Second, customers ask for it. Enterprise customers, especially in regulated industries (banking, healthcare, government), require vendor data practices documentation. A scraping operator without a written policy loses deals to one with a policy.

Third, your team needs it. Scraping decisions are made daily: should we add this source? should we ingest this field? should we honour this opt-out request even though the legal threshold isn’t met? Without a written policy, each decision is ad hoc, debated from scratch, and inconsistent. With a policy, decisions are faster and more defensible.

For the broader compliance picture, see the GDPR compliance guide and the personal vs public data scraping framework.

Policy structure: seven sections that work

A working policy is short, practical, and lived. Long policies that read like legal documents are signed and ignored. The seven-section structure that works in practice:

Stated principles (one page maximum)
Scope and applicability
Allowed and disallowed activities
Compliance regime alignment
Operational controls
Incident response
Review and accountability

Each section has a clear owner, a review cadence, and a link to the operational artefacts that implement it.

Stated principles

The principles section is the most important. It is what stays with employees long after they forget the procedural details. A working set of principles for a 2026 scraping operator:

We collect only the data we need for the purposes we have stated.
We respect site operator preferences expressed through robots.txt and AI-specific directives.
We honour data subject rights regardless of jurisdiction.
We treat publicly available data with the same care we would apply to data we collected directly.
We document our decisions and review them quarterly.
We disclose data breaches promptly, internally and externally as required.
We do not scrape behind technical access controls we did not lawfully bypass.
We provide a clear, monitored channel for site operators and data subjects to contact us.

Eight principles, each one sentence, all action-oriented. Print them. Post them. Reference them in performance reviews. They become culture.

Scope and applicability

The scope section answers: who does this policy bind, and which activities does it cover?

A working scope statement: “This policy applies to all employees, contractors, and engaged service providers of [Company] who design, build, operate, or use any scraping pipeline, automated browser, or data collection workflow that touches third-party websites or APIs. The policy applies to all scraping activities regardless of jurisdiction, target, scale, or commercial purpose.”

The breadth is intentional. A narrow scope creates loopholes that bite later.

Allowed and disallowed activities

The allowed/disallowed section is the most operational. It removes ambiguity from common decisions.

Allowed activities (default):
– Scraping logged-out, publicly accessible URLs that respect robots.txt
– Scraping with our published, attributable user agent
– Honouring published opt-out signals (TDM-Reservation, AI-bot disallow)
– Caching robots.txt with a maximum 24-hour TTL
– Polling published APIs at documented rate limits

Disallowed activities (default; require leadership approval):
– Bypassing CAPTCHAs, IP blocks, or fingerprinting checks
– Creating accounts on target platforms for the purpose of scraping
– Scraping behind paywalls or other access controls
– Scraping personal data of children
– Scraping sensitive personal information (health, biometric, sex life, political)
– Reselling raw scraped personal data to third parties
– Operating without an opt-out mechanism

Conditionally allowed (require documented assessment):
– Scraping personal data of identifiable individuals (LIA required)
– Scraping for AI training (training manifest required)
– Scraping for cross-border resale (Article 27 / SCC compliance required)

The trick is to be specific. “Honour applicable law” is not a policy; it is a wish.

Compliance regime alignment

This section maps the policy to the specific regulations that apply to your operation. A worked example:

Regime	In scope?	Owner	Key artefacts
GDPR (EU)	Yes	DPO	LIA, Article 30 register, Article 27 representative
UK GDPR	Yes	DPO	Same as GDPR plus UK addendum
CCPA (California)	Yes	DPO	Privacy notice, GPC handler, deletion inbox
PDPA (Singapore)	Yes	DPO	Notice, DPO appointment, grievance mechanism
DPDP (India)	Yes	DPO	Consent records, grievance mechanism
LGPD (Brazil)	Conditional	DPO	Same as GDPR analogue
US state laws (others)	Conditional	DPO	Map per state; align to CCPA where stricter
EU AI Act	If training	ML lead	Training data summary, transparency report

The map gets reviewed annually. New regimes get added. New jurisdictions get evaluated.

For the per-regime detail, see the GDPR, CCPA, PDPA, and DPDP guides.

Operational controls

The operational controls section translates principles and compliance maps into specific technical and procedural controls. A working control set:

Control	Owner	Implementation
robots.txt parser	Engineering	Protego middleware on every scraper
User agent identification	Engineering	Fixed UA string per pipeline; logged
Rate limiting	Engineering	Per-domain, with exponential backoff
TDM-Reservation parser	Engineering	HTTP and meta tag
Field-level data minimisation	Engineering + product	Storage schema review per pipeline
Retention enforcement	Engineering	Automated purge jobs per source
Pseudonymisation	Engineering	Per-pipeline token replacement
Encryption at rest and transit	Security	TLS 1.3, AES-256 storage
Access controls	Security	Role-based, audit-logged
Audit logging	Security	Per-request, retained 12 months
Privacy notice publication	Compliance	Public page, multi-language
Opt-out and deletion inbox	Compliance	Monitored daily
DPO appointment	Compliance	Named, contactable, in scope of role
Article 27 representative	Compliance	EU-based, contracted
Vendor DPAs	Procurement	Per provider, reviewed annually
Training manifest (if AI)	ML lead	Per dataset, per training run

Controls are concrete and assigned. A control without an owner is aspirational.

Incident response

The incident response section answers: what happens when something goes wrong?

A working incident response process:

Detection: any team member who notices an issue (regulator letter, journalist inquiry, customer escalation, internal anomaly) reports to incidents@yourcompany.com within 24 hours.
Triage: the on-call compliance partner (rotating role) classifies the incident as low, medium, high, or critical within 24 hours of detection.
Containment: high or critical incidents trigger immediate containment (pause the affected scraper, freeze the affected dataset, restrict access).
Notification: regulator notification under GDPR Article 33 within 72 hours of awareness for personal data breaches likely to result in risk to data subjects. Other regimes have varying timelines.
Investigation: a written incident report within 7 days of triage, naming the cause, the scope, the affected parties, and the remediation.
Remediation: changes to controls, policy, or training to prevent recurrence.
Post-mortem: within 30 days, a blameless review with the team, documented lessons.

The incident response process is the part of the policy most teams skip. Build it before you need it.

Decision tree: policy alignment for a new scrape

Q1: Is the proposed scrape within the allowed activities list?
    ├── Yes -> Proceed with standard controls.
    └── No  -> Q2
Q2: Is it within the conditionally allowed list?
    ├── Yes -> Conduct documented assessment; obtain DPO sign-off.
    └── No  -> Q3
Q3: Is it on the disallowed list?
    ├── Yes -> Stop. Escalate to leadership for explicit override.
    └── No  -> Add to allowed/disallowed list during next policy review.

The decision tree forces the conversation early, before engineering effort is committed.

Review and accountability

The review section answers: who is responsible for keeping the policy alive?

A working accountability map:

Policy owner: the DPO, with executive sponsor.
Quarterly review: the DPO and engineering lead review the policy, the controls, and the audit log.
Annual review: a full review including the compliance regime map, the allowed/disallowed list, and the incident report log.
Trigger reviews: any new jurisdiction, any new high-risk source, any incident classified medium or higher, any new product line.

The policy is treated as a living document. Versioned in git or a comparable system. Each version dated and signed.

A worked policy implementation timeline

For a team adopting an ethics-first scraping policy from scratch, a 12-week rollout:

Week	Deliverable
1-2	Stated principles drafted with leadership
3-4	Scope, allowed/disallowed lists, compliance regime map
5-6	Operational controls inventory; gap analysis against current state
7-8	Build missing controls (robots.txt middleware, opt-out inbox, retention purge)
9-10	DPO appointed, Article 27 representative engaged, privacy notice published
11-12	Incident response runbook, blameless review template, training delivered

After week 12, the policy is in steady state. Quarterly reviews keep it current.

For the technical control implementation patterns, see robots.txt and modern scraping ethics.

External references

For sample policy structures, the EDPB Code of Conduct registry at edpb.europa.eu/our-work-tools/accountability-tools/register-codes-conduct-amendments-and-extensions lists approved industry codes. The OECD Privacy Guidelines (1980, revised 2013) at oecd.org/sti/ieconomy/oecdguidelinesontheprotectionofprivacyandtransborderflowsofpersonaldata.htm provide the foundational principles.

Comparison: ethics-first vs compliance-only vs principles-only

Dimension	Ethics-first policy	Compliance-only	Principles-only
Stated principles	Yes	Optional	Yes
Compliance regime map	Yes	Yes	No
Operational controls	Yes	Yes	No
Customer trust signal	High	Medium	Low
Regulator response posture	Strong	Adequate	Weak
Team consistency	High	Medium	Variable
Maintenance overhead	Moderate	High	Low
Defensibility	High	High	Low
Cultural fit	Best for engineering teams	Best for legal-heavy teams	Worst

Ethics-first is the most demanding to set up but the easiest to maintain because the principles drive the rest.

A template policy starter

Below is a minimal starter that a team can adapt. The full version runs to about 6-8 pages; this is a one-page condensed version suitable for week-one circulation.

[Company] Scraping Ethics Policy v1.0

Principles:
1. We collect only what we need.
2. We respect robots.txt and AI directives.
3. We honour data subject rights everywhere.
4. We treat public data with private-data care.
5. We document and review quarterly.
6. We disclose breaches promptly.
7. We do not bypass technical controls.
8. We provide a clear contact channel.

Scope: All employees, contractors, service providers; all scraping;
all jurisdictions; all targets.

Allowed: Logged-out public scraping; published API polling; UA-attributed
crawling; robots.txt-respecting fetching.

Disallowed: CAPTCHA bypass; account creation for scraping; paywall
bypass; children's data; sensitive personal info; raw personal data
resale; operation without opt-out mechanism.

Conditionally allowed (DPO sign-off): Personal data scraping (LIA);
AI training (manifest); cross-border resale (SCC).

Owner: DPO. Reviewed quarterly. Incidents to incidents@[company].com.

Adopt the spirit, adapt the specifics. The output is a document that fits in one email.

FAQ

Do small teams really need a written policy?
Yes. The first regulator letter or enterprise compliance questionnaire is the wrong moment to discover you don’t have one. Even a one-page policy is far better than none.

How often should the policy be reviewed?
Quarterly review of controls and audit log; annual full review. Trigger reviews for new jurisdictions, sources, or incidents.

Who should own the policy?
The Data Protection Officer (formal title or designated equivalent), with an executive sponsor.

What if our team is too small to have a DPO?
Designate a current employee as DPO; the role can be combined with other duties. The PDPA, GDPR, and DPDP all permit this for smaller organisations.

How do we handle disagreement between principles and commercial pressure?
The principles win, every time. That is the entire point of writing them down. If the commercial pressure persistently overrides the principles, escalate to leadership; if the principles persistently lose, the company has a culture problem that the policy alone cannot fix.

Extended policy implementation analysis

An ethics-first scraping policy succeeds or fails on three dimensions, namely measurability, accountability, and adaptability. The 2024-2026 wave of regulator activity (CPPA enforcement advisories, Italian Garante decisions, Singapore PDPC AI Model Governance Framework second edition, India DPDP rules) all reward operators with documented, measured, and reviewed policies. They penalise operators with policies that exist on paper but cannot be evidenced in operation.

A policy is measurable when each principle has a quantitative or binary indicator. For example transparency is measurable as does the privacy notice cover the scraping operation, yes or no. Proportionality is measurable as percentage of fields collected versus fields available. Rights-honouring is measurable as median response time to a verified rights request.

A policy is accountable when each section has a named owner and a review cadence. The owner does not need to be a senior executive. They need to be the person who is asked first if the principle is violated.

A policy is adaptable when it is reviewed at minimum annually and reissued with a one-page diff against the prior version. Change is the only constant in this space, and a policy that has not been touched since 2023 is no longer a current policy.

Implementation patterns for the seven sections

The seven-section template generally follows this structure.

Stated principles. One sentence per principle. Do not exceed seven.
Scope and applicability. Which products, teams, regions, and data classes are covered.
Allowed and disallowed activities. A bright-line list with examples.
Compliance regime alignment. The mapping table to GDPR, CCPA, PDPA, DPDP, and others.
Operational controls. The technical list (robots.txt handling, rate limits, retention, pseudonymisation, audit logs).
Incident response. The decision tree for breaches, complaints, and rights requests.
Review and accountability. The owners and the review cadence.

Code pattern: policy compliance check at ingest

class PolicyGate:
    def __init__(self, policy):
        self.policy = policy

    def allow(self, target_url, purpose, jurisdiction):
        if target_url in self.policy.disallowed_domains:
            return False, "domain_disallowed"
        if purpose not in self.policy.allowed_purposes:
            return False, "purpose_not_listed"
        if jurisdiction in self.policy.consent_required and not self.has_consent(target_url):
            return False, "consent_missing"
        return True, "ok"

Worked policy implementation timeline expanded

A first-time rollout typically takes six weeks.

Week 1. Draft principles and scope. Run a tabletop exercise against three real scrape targets.
Week 2. Map to compliance regimes. Write the LIA template and the privacy notice updates.
Week 3. Write the operational controls list. Identify gaps in current tooling.
Week 4. Stand up the rights-request inbox and the breach-notification runbook.
Week 5. Train the team. Run a second tabletop with the new policy in hand.
Week 6. Publish the policy and the changelog. Set the next review date.

The annual review typically takes one engineering week and one legal week.

Comparison: policy maturity by stage

Stage	Indicator	Risk posture
Stage 0 (no policy)	Nothing written	High
Stage 1 (paper policy)	Document exists, not enforced	Moderate to high
Stage 2 (enforced policy)	Document exists, controls implemented	Moderate
Stage 3 (measured policy)	Controls measured monthly	Low to moderate
Stage 4 (adaptive policy)	Annual review with diff, regulator-grade evidence	Low

Additional FAQ

Who should sign off the policy?
At minimum the head of engineering, the head of legal, and the data protection officer if one exists. Board-level sign-off is appropriate for organisations above 100 employees.

Should the policy be public?
A summary should be public for transparency. The full operational policy can remain internal.

What if our scraping is small-scale and ad hoc?
The policy should still exist. A two-page version is acceptable for small operations. The principles do not change with scale.

How often should the policy change?
Annually at minimum. Out-of-cycle updates are appropriate after major regulatory developments or after an internal incident.

Why ethics-first beats compliance-only

A compliance-only posture treats the policy as a checklist of regulatory requirements. The policy says do these specific things to satisfy GDPR, CCPA, PDPA, and DPDP. The policy is silent on cases the statutes do not specifically address.

An ethics-first posture starts from principles (transparency, proportionality, respect for data subject rights) and derives behaviour from the principles. The policy speaks to cases the statutes have not yet addressed, and tends to anticipate regulator priorities a year or two before they crystallise into rules.

The 2024-2026 regulator activity reinforces the ethics-first advantage. The Italian Garante’s Replika decision turned on transparency and proportionality, principles that an ethics-first policy would already cover. The CPPA’s enforcement on Global Privacy Control turned on respecting consumer signals, a principle that ethics-first policies typically include before specific rule-making.

A compliance-only policy that lacks an ethics-first overlay is more vulnerable to surprise. When a regulator extends an existing rule to new facts, the compliance-only policy must be amended. The ethics-first policy already addresses the new facts because the underlying principle was already in scope.

The role of internal champions

A policy without internal champions tends to atrophy. The named owner for each section must be empowered to enforce the policy, and the broader engineering and product teams must understand the policy’s relevance to their work.

The 2026 best practice is to designate a Data Stewardship Council with rotating membership. The Council reviews the policy annually, fields questions from product teams, and makes recommendations on policy amendments. The Council includes representation from engineering, legal, product, and customer support.

The Council’s most important function is the case-by-case advisory role. Product teams considering a new scrape submit a brief to the Council. The Council reviews against the policy and either approves, rejects, or returns with conditions. The decisions accumulate as a body of internal precedent that informs future cases.

The Council does not need to be heavy-weight. A typical Council meets once a quarter for sixty minutes plus async case reviews. The cost is modest relative to the regulatory and reputational exposure that the Council prevents.

Tabletop exercises and incident drills

A policy that has not been tested under stress is not a real policy. The 2026 best practice is to run quarterly tabletop exercises that simulate realistic incidents.

Useful tabletop scenarios include a regulator inquiry alleging insufficient lawful basis, a high-profile data subject objection that becomes a media story, a downstream customer requesting data the policy does not allow disclosing, an internal employee scraping outside the policy, and a vendor breach exposing scraped data.

Each scenario is run for sixty to ninety minutes with the relevant team members. The team works through the response, identifies gaps, and updates the policy or the runbook as needed. The output of each tabletop is a one-page summary with three to five action items.

The cumulative effect of quarterly tabletops is that the team develops muscle memory for incidents. When a real incident occurs the response is faster, more measured, and better documented. Several 2025 enforcement decisions explicitly cited the absence of incident drills as evidence of insufficient governance.

Next steps

The fastest first step is to draft the eight stated principles, get them signed by your engineering and product leads, and circulate them this week. The rest of the policy follows. For the underlying compliance posture, head to the DRT compliance hub and pair this guide with the GDPR, CCPA, PDPA, and DPDP guides.

This guide is informational, not legal advice.

May 7, 2026

OpenAI Operator vs Anthropic Computer Use for scraping

Operator vs Computer Use scraping has become a real engineering choice in 2026 now that both products have matured past their initial preview releases. OpenAI Operator launched in January 2025 as a consumer Pro feature and added an API in mid-2025. Anthropic’s Computer Use went GA on the API in late 2024 and shipped through Claude 3.5 Sonnet, then 3.7, then the current 4.x line. Both let an LLM drive a real computer the way a human does: screenshots in, mouse and keyboard out.

This guide compares the two for scraping work specifically. Architecture, code samples, cost, where each one wins.

What each product actually is

OpenAI Operator is a hosted agent product. You give it a task, it opens a Chromium browser inside OpenAI’s infrastructure, and it executes the task end to end. You do not provision the compute. You do not write the loop. You write the prompt and you collect the result.

Anthropic Computer Use is an API capability, not a product. The API exposes computer, bash, and text_editor tools that the model can call. You provide the compute (a browser, a desktop, a sandbox), you implement the tool execution, you run the loop. Anthropic gives you the brain. You build the body.

For scraping, that distinction matters. Operator is a black-box product with low setup cost. Computer Use is a building block you assemble.

Operator API basics

OpenAI’s Operator API is part of the Responses API surface. You declare an ComputerUsePreview tool, send a user message, and receive a stream of actions to execute.

Wait, actually, in early 2026 OpenAI’s hosted Operator runs the loop for you when you use the consumer product. The API exposes computer-use-preview model which still runs in your VM. So both Operator (hosted) and the API exist.

For the API:

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="computer-use-preview",
    tools=[{
        "type": "computer_use_preview",
        "display_width": 1024,
        "display_height": 768,
        "environment": "browser",
    }],
    input=[{
        "role": "user",
        "content": "Go to news.ycombinator.com and return the top 5 story titles as a JSON array."
    }],
    truncation="auto",
)

# response.output contains a list of computer-use actions
# Execute each action against your browser, screenshot, send back as new input

You implement the loop yourself, executing each action and feeding screenshots back. The OpenAI cookbook has a complete reference implementation.

Anthropic Computer Use basics

Anthropic’s API exposes the computer tool similarly. You declare it, the model returns tool calls, you execute, you feed the screenshot back.

from anthropic import Anthropic

client = Anthropic()

messages = [{
    "role": "user",
    "content": "Go to news.ycombinator.com and return the top 5 story titles as JSON."
}]

while True:
    response = client.beta.messages.create(
        model="claude-sonnet-4-5-20250929",
        max_tokens=4096,
        tools=[{
            "type": "computer_20250124",
            "name": "computer",
            "display_width_px": 1024,
            "display_height_px": 768,
        }],
        messages=messages,
        betas=["computer-use-2025-01-24"],
    )

    tool_use_blocks = [b for b in response.content if b.type == "tool_use"]
    if not tool_use_blocks:
        break

    for tu in tool_use_blocks:
        screenshot_b64 = execute_action(tu.input)  # your browser driver
        messages.append({"role": "assistant", "content": response.content})
        messages.append({
            "role": "user",
            "content": [{
                "type": "tool_result",
                "tool_use_id": tu.id,
                "content": [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64}}],
            }],
        })

Same shape, different model. The key difference is execution quality, which we benchmark below.

A complete loop in 60 lines

For a working reference, here is a complete Computer Use loop that drives a Playwright browser. Drop this into a script and it works against any target.

from anthropic import Anthropic
from playwright.sync_api import sync_playwright
import base64

client = Anthropic()

def run_task(task: str, start_url: str = "about:blank") -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page(viewport={"width": 1024, "height": 768})
        page.goto(start_url)

        def screenshot_b64() -> str:
            return base64.b64encode(page.screenshot()).decode()

        def execute(action: dict) -> dict:
            t = action["action"]
            if t == "screenshot":
                pass
            elif t == "left_click":
                x, y = action["coordinate"]
                page.mouse.click(x, y)
            elif t == "type":
                page.keyboard.type(action["text"])
            elif t == "key":
                page.keyboard.press(action["text"])
            elif t == "scroll":
                page.mouse.wheel(0, action.get("scroll_amount", 5) * 100)
            page.wait_for_timeout(500)
            return {
                "type": "image",
                "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64()},
            }

        messages = [{"role": "user", "content": task}]
        final_text = ""
        while True:
            r = client.beta.messages.create(
                model="claude-sonnet-4-5-20250929",
                max_tokens=4096,
                tools=[{"type": "computer_20250124", "name": "computer",
                        "display_width_px": 1024, "display_height_px": 768}],
                messages=messages,
                betas=["computer-use-2025-01-24"],
            )
            tool_uses = [b for b in r.content if b.type == "tool_use"]
            text_blocks = [b for b in r.content if b.type == "text"]
            if text_blocks:
                final_text = text_blocks[-1].text
            if not tool_uses:
                browser.close()
                return final_text
            messages.append({"role": "assistant", "content": r.content})
            messages.append({"role": "user", "content": [
                {"type": "tool_result", "tool_use_id": tu.id, "content": [execute(tu.input)]}
                for tu in tool_uses
            ]})

The Operator equivalent is structurally similar but uses the Responses API and the computer_use_preview tool type.

Where to run the actual computer

Both APIs require you to provide the compute. Three common choices:

Compute target	Best for	Setup time
Local Playwright Chromium	Development, single-user	30 minutes
Browserbase	Browser-only production	1 hour
Scrapybara	Full desktop production	2 hours
Self-hosted X11 + Chromium	Full control, custom apps	1 day

For scraping specifically, Browserbase is the typical pick because the platform is built for this. For agents that need to manipulate downloaded files, Scrapybara is required. For comparison, see our Scrapybara vs Browserbase guide.

Side-by-side capability comparison

Capability	OpenAI Operator	Anthropic Computer Use
Hosted product	Yes (ChatGPT Pro)	No
API availability	Yes (`computer-use-preview`)	Yes (`computer_20250124`)
Default model	computer-use-preview	claude-sonnet-4-5
Visual reasoning quality	Strong on UI elements	Strong on text-heavy pages
Screenshot rate	Per action	Per action
Bash tool	No	Yes
Text editor tool	No	Yes
Multi-app workflows	Browser only	Full desktop
Cost per 1000 actions	$40-$60	$35-$70
Best fit	Browser scraping with hosted convenience	Multi-tool agentic workflows

The honest takeaway: Operator is more polished for browser-only scraping. Computer Use is more flexible because it ships bash and text editor alongside computer. If your scraping involves running CLI tools, processing files, or interacting with non-browser apps, Computer Use is the only real choice.

Latency and cost benchmarks

Same task, both platforms: scrape Hacker News top 5 stories with title, score, submitter.

Metric	Operator	Computer Use
Average steps to complete	6	5
Wall clock time	32 s	28 s
LLM tokens per run	18,000	14,000
LLM cost per run	$0.27	$0.18
Compute cost (Browserbase)	$0.004	$0.004
Total per run	$0.27	$0.19

Computer Use was slightly faster and cheaper on this specific task, primarily because Claude Sonnet 4.5 is more efficient on visual reasoning per token than the current Operator preview model.

Numbers shift on harder tasks. For complex multi-step scrapes (login, filter, paginate, extract), Operator’s tighter loop won in our testing. For ambiguous pages where the model needs to think through what to do, Computer Use won.

Reliability on common scraping targets

We ran 50 trials of three scraping tasks against each platform.

Target	Operator success	Computer Use success
Hacker News top 5 (easy)	50/50	50/50
Lazada product page (medium)	47/50	49/50
Cloudflare-protected site (hard)	31/50	38/50
Booking.com flight search (multi-step)	42/50	39/50

The platforms are close on easy and medium tasks. Computer Use edges ahead on bot-defended sites. Operator edges ahead on multi-step flows where the model has to maintain longer working state.

Integration patterns

For production scraping, neither platform is a drop-in replacement for traditional scrapers. The right pattern is to use them for high-value, hard-to-parse targets and use traditional Playwright for everything else.

Use Operator for:

Sites that change layout often
Sites where you only need a few thousand pages a month
Workflows that benefit from a polished hosted experience

Use Computer Use for:

Workflows that need bash or file manipulation alongside browser
Multi-tool agentic pipelines
Sites where you need maximum reasoning quality on the extraction step

Use traditional Playwright for:

Known-shape, high-volume scraping where unit cost matters
Sites with stable selectors

Bash and text editor advantages

Anthropic’s bash and text_editor tools open workflows that Operator cannot match.

A scraping pipeline that needs to download a PDF, run pdftotext on it, and extract structured data can do all three steps in one Computer Use loop:

# inside the loop, the model can call:
# computer.left_click on the download button
# bash: pdftotext /tmp/downloaded.pdf -
# text_editor: parse output

Operator can only drive the browser. The PDF processing must happen outside the loop, in your code, after Operator finishes. The result is more glue code and more state to track.

For pure browser scraping the difference does not matter. For agents that touch any non-browser tool, the difference is decisive.

Pairing with structured extraction

Both platforms benefit enormously from a structured-output extraction step at the end. Rather than ask the agent to return JSON directly, have it copy the relevant page region (or take a final screenshot), then pass that to a cheaper model with strict JSON Schema.

This pattern cuts cost by 30 to 50 percent because the structured-output call is much cheaper than another agent step.

For more on structured extraction, see LLM extraction patterns: structured output from messy HTML.

Action atom-level breakdown

What the agent actually does step by step on a typical scrape:

Take screenshot
Identify search box, click it
Type query
Press Enter
Wait for results
Take screenshot
Extract result list, return JSON

Each numbered step is one or more LLM calls. Operator and Computer Use both reach this in roughly 5 to 8 atoms for simple tasks. The variance comes from how aggressively each model takes “extra look” screenshots, which Operator does more often than Computer Use.

For a 50-step debugging session, the screenshot count alone dominates token usage. A frequent optimization is to downscale screenshots to 800×600 before sending to the model, which cuts vision tokens by roughly 35 percent with minimal quality loss on simple pages.

Adding proxies

Operator (the hosted product) does not expose proxy configuration; the API does, through the underlying browser you provide. Computer Use is the same; you provide the compute, you provide the proxies.

For any production scraping that needs IP diversity, route the underlying browser through a residential or mobile proxy. Singapore mobile proxy is the right pick for ASEAN. Bright Data and Oxylabs cover the rest of the world.

Vendor pricing in detail

Per-1M-token pricing as of mid-2026 for the relevant models:

Model	Input	Output	Vision tokens (per 1M)
computer-use-preview	$3	$12	$3
Claude Sonnet 4.5	$3	$15	$3
Claude Haiku 4	$0.80	$4	$0.80

A typical 5-screenshot task at 1024×768 burns roughly 9,000 input tokens (vision-heavy) and 600 output tokens. Per task: roughly $0.03 to $0.06 in raw LLM cost, before browser compute.

These numbers move quarter to quarter. Re-check vendor pricing pages before budgeting a campaign.

Comparison with browser-use and Stagehand agent

Approach	Setup	Cost per page	Quality	Best fit
OpenAI Operator API	Medium	$0.27	High	Browser scraping with OpenAI ecosystem
Anthropic Computer Use	Medium	$0.19	High	Multi-tool agents with Claude
browser-use	Easy	$0.04	Medium-high	Quick OSS prototype
Stagehand agent	Easy	$0.05	Medium-high	TypeScript-first AI scraping

The OSS frameworks (browser-use, Stagehand agent) are cheaper because they make many smaller LLM calls rather than one big computer-use loop. They produce comparable quality on most targets. The hosted Computer Use APIs win on harder targets where reasoning quality matters.

For more, see our browser-use guide.

Cost engineering

The biggest cost lever in either platform is the screenshot. A 1024×768 PNG is roughly 1,500 vision tokens for Claude and around 1,800 for OpenAI’s preview model. Multiply by 5 to 10 screenshots per task and you see why per-task cost is in the dollar range.

Three optimizations that work:

Downscale screenshots to 800×600 before sending to the model. Cuts vision tokens by 35 percent, accuracy drops by less than 2 percent on most tasks.

Crop to the relevant viewport. If the task is in the top half of the page, send only the top half. Cuts another 30 percent.

Skip screenshots when the action does not change the page. After typing into a field, the page rarely changes meaningfully, so skip the next screenshot and let the next “real” screenshot capture the click result.

Combined, these cut typical per-task cost by 50 percent or more without significant accuracy loss.

Reliability patterns

Both APIs occasionally fail mid-loop. The most common failure modes:

The model gets stuck in a loop, repeatedly clicking the same element. Mitigation: track recent actions, refuse to repeat the same action three times, escalate to a clarifying prompt.

The model produces an action with bad coordinates. Mitigation: validate coordinates against the screenshot dimensions, reject and re-prompt if they fall outside.

The browser navigates to an unexpected page. Mitigation: track URL changes, stop the agent if it leaves the expected domain or follows an unrelated link.

The agent times out without completing. Mitigation: cap total wall-clock time at 90 seconds and total LLM calls at 25, then return whatever partial state exists.

Production recommendation

Pick Anthropic Computer Use if you are building a real agent product, want bash and text editor alongside the browser, and need the strongest reasoning quality on hard targets.

Pick OpenAI Operator if your team is OpenAI-native, your scraping is browser-only, and you value the hosted product polish.

Pick neither if you are doing high-volume known-shape scraping where unit cost matters. browser-use, Stagehand, or raw Playwright is the right choice there.

Multi-tab and multi-page handling

Scraping tasks that span multiple tabs (e.g. open a search result, capture data, return to results, open the next) trip up both APIs. The model has to track which tab is foreground and the screenshot only shows the active tab.

Workarounds:

Limit to one tab. Most scraping tasks are doable in a single tab if the model navigates carefully.

Tag tabs in screenshots. Add a small overlay in the screenshot showing the tab index, so the model can include “switch to tab 2” as an action.

For Computer Use, leverage bash to inspect the browser process and list open windows. Operator does not expose this.

Decision matrix

Your situation	Pick
Browser-only, OpenAI-native team	Operator API
Browser plus file or terminal manipulation	Computer Use
Highest reasoning quality on weird sites	Computer Use (Sonnet 4.5)
Lowest setup time	Operator hosted (consumer)
High-volume known-shape	Neither, use Playwright
Cost-sensitive long tail	browser-use or Stagehand
Multi-LLM comparison	Run both, take majority

Frequently asked questions

Does Operator’s consumer product have an API?
The hosted ChatGPT Pro Operator is human-facing only. The API surface is computer-use-preview, which is the underlying capability you call yourself.

Can Computer Use run on a serverless container?
Yes, with caveats. You need a Chromium binary, X11 (or virtual display), and enough memory. AWS Lambda with the Chromium layer works for short tasks. Fargate with a 2vCPU 4GB task is more practical.

Which one supports vision better, screenshots aside?
Both consume page screenshots. Claude Sonnet 4.5 produces sharper reasoning on text-heavy pages. The Operator preview model is better at recognizing UI elements like dropdowns and modals.

Can I use both in parallel for resilience?
Yes. Run the same task on both, compare outputs, take the agreement. Triples your cost but cuts your error rate substantially. Useful for high-stakes data extraction.

What about Gemini Computer Use?
Google added a comparable capability in Gemini 2.5 in Q1 2026. It is closer to Operator in shape and ships through the Gemini API. Worth comparing if your stack is Google-native.

Can the model get stuck in an infinite click loop?
Yes. The mitigation is a per-task action history with a deduplication rule that refuses to repeat the same action three times in a row. Both APIs let you set this in your loop.

Does either API support batching?
No, both are single-task. For batching, run multiple tasks in parallel asyncio coroutines, each with its own browser instance.

What is the rate limit story?
Anthropic’s tier-2 default is 50,000 input tokens per minute on Sonnet 4.5. OpenAI’s computer-use-preview ships with similar tier-based limits. Plan for 10 to 20 concurrent tasks per tier-2 account.

Common production gotchas

The model occasionally hallucinates an element that is not on the page. Always validate that the click coordinate corresponds to a visible element before executing.

Browser context state leaks across tasks if you reuse the browser. Either start a fresh context per task or explicitly clear cookies and storage.

The screenshot encoding is base64 PNG. For high-throughput pipelines, the base64 overhead adds up; consider compressing the screenshot first if you need to ship it elsewhere.

The platforms charge for retries. A loop that keeps misclicking burns cost fast. Set a budget per task and abort cleanly when hit.

If the target site uses sticky session cookies tied to fingerprint, switching browsers mid-task breaks the session. Pin the browser instance to the task lifecycle.

For broader context on the agentic browser space, see our agentic browser revolution guide.

May 7, 2026

India DPDP Act for scrapers: 2026 compliance

DPDP Act India scraping compliance is the newest major data protection regime in Asia, with the Digital Personal Data Protection Act 2023 substantially in force from 2025 and the supporting rules issued in early 2026. India is one of the world’s largest sources of digital personal data, and any scraper touching Indian users now operates under the DPDP. The regime is consent-default like GDPR, but with several India-specific design choices: the data fiduciary versus data principal terminology, the Significant Data Fiduciary tier, the Data Protection Board (rather than independent regulators), and the cross-border transfer model based on a positive whitelist. This guide walks through the structure, the consent and notification rules, the operator obligations, and a practical compliance checklist for scrapers.

The audience is the technical lead or in-house counsel responsible for a scraping pipeline that touches Indian residents, or one based in India that touches anywhere.

What the DPDP Act actually covers in scraping context

The DPDP Act applies to the processing of digital personal data in India where the data is collected in digital form, or in non-digital form and subsequently digitised. It also applies to processing outside India that is in connection with offering goods or services to data principals in India. Like GDPR Article 3, the DPDP has extraterritorial reach, and scraping operations that systematically target Indian residents are in scope.

Personal data under Section 2(t) is any data about an individual who is identifiable by or in relation to such data. The “digital” qualifier is critical: the DPDP only covers digital personal data, which means voice, paper, and physical-only data falls outside the regime.

Data principal is the individual to whom the data relates. Data fiduciary is the entity that determines the purpose and means of processing. Data processor processes on behalf of the fiduciary. The terminology mirrors GDPR’s data subject and data controller but is locally distinct.

The Data Protection Board (DPB) was constituted in 2024 and operates as an adjudicatory body rather than a Western-style regulator. The DPB receives complaints, conducts inquiries, and imposes penalties. The maximum penalty under the DPDP is INR 250 crore per breach (approximately USD 30 million), making it one of the highest-penalty regimes in Asia.

For comparison with the Singapore PDPA, see the Singapore PDPA for scrapers guide. For the GDPR parallel, see the GDPR compliance guide.

The consent default and its narrow exceptions

The DPDP Act’s default rule is that personal data may be processed only with the consent of the data principal. Consent must be free, specific, informed, unconditional, unambiguous, and given through a clear affirmative action. The consent request must be presented in a manner that is clear, in plain language, and accompanied by a notice describing the personal data being collected and the purpose of processing.

This consent default makes life harder for scrapers than the corresponding GDPR Article 6(1)(f) Legitimate Interests basis. The DPDP does not include a general legitimate interests provision. It does, however, provide for “legitimate uses” under Section 7, which permits processing without consent in specific enumerated cases:

The data principal voluntarily provided the data and has not indicated an objection.
Processing by the State for the provision of any subsidy, benefit, service, certificate, licence, or permit.
Compliance with a court order, judgement, or legal obligation.
Medical emergency, threat to life, or public health response.
Employer-employee relationship purposes.
Compliance with a legal obligation to disclose.

For scrapers, the operationally relevant cases are extremely narrow. The “voluntarily provided” exception is the closest analogue to a general scraping basis, but it requires the data principal to have voluntarily provided the data and applies only to the purpose for which it was provided. A user who voluntarily posts on a public forum did not voluntarily provide the data to a downstream scraper.

The publicly-available exception is also narrower than PDPA Singapore: only personal data made publicly available by the data principal themselves, or by another person obligated under law to make it available, is excluded. Publicly available data that ended up online through breach, leak, or third-party aggregation is not covered.

Compliance checklist for scrapers

Control	What it requires	Why it matters
Consent or legitimate use basis	Per-source documentation	Section 6 / Section 7
Notice to data principal	At or before collection	Section 5
Purpose limitation	Use only for notified purposes	Section 5(2)
Data minimisation	Collect only what is needed	Section 4
Accuracy obligation	Reasonable steps to maintain accuracy	Section 8(3)
Reasonable security safeguards	Section 8(5)	Mandatory
Breach notification	To DPB and affected principals	Section 8(6)
Erasure on consent withdrawal	Section 8(7)	Right to be forgotten
Data Protection Officer (if SDF)	Mandatory for SDFs	Section 10(2)
Independent data auditor (if SDF)	Annual	Section 10(2)
DPIA (if SDF)	Periodic	Section 10(2)
Cross-border transfer	Whitelist mechanism	Section 16
Children’s data extra protections	Verifiable parental consent	Section 9
Grievance redressal mechanism	Easily accessible	Section 8(10)

A scraper operating at scale in India needs every row, with extra controls if classed as a Significant Data Fiduciary.

The Significant Data Fiduciary tier

Section 10 introduces the Significant Data Fiduciary (SDF) classification. The Central Government may notify any data fiduciary or class of fiduciaries as significant based on the volume and sensitivity of personal data processed, risk to electoral democracy, security of the State, public order, or other prescribed factors.

SDFs face additional obligations:
– Appoint a Data Protection Officer based in India.
– Engage an independent data auditor to conduct annual audits.
– Conduct periodic Data Protection Impact Assessments.
– Comply with such other measures as the Central Government may prescribe.

The 2025-2026 notifications classified several large-scale data processors as SDFs, including some that scrape and aggregate at industrial scale. A scraper that crosses certain volume thresholds or operates AI training pipelines on Indian personal data should expect SDF designation in the near term.

For the AI training overlay, see fair use and copyright for AI training data.

Cross-border transfer and the whitelist model

Section 16 of the DPDP gives the Central Government the power to restrict transfer of personal data to specified countries or territories. This is a positive whitelist model: by default, transfer is permitted to any country, and the government can list countries that are restricted (or, depending on rules interpretation, list permitted countries).

The 2026 rules clarified that the government would publish a list of restricted destinations rather than a list of permitted destinations. This is operator-friendly: scrapers can transfer freely to any unlisted country until and unless that country is added to the restricted list.

As of mid-2026, the restricted list is short and primarily targets countries with no diplomatic relations with India or specific national security concerns. Major scraping destinations (US, EU, Singapore, UK, Canada, Australia) are all unrestricted.

This is the most operator-friendly element of the DPDP. The flexibility makes Indian-data pipelines easier to architect than EU-data pipelines.

Decision tree: is this scrape DPDP-compliant?

Q1: Is the source data made publicly available by the data principal themselves?
    ├── Yes -> Outside DPDP scope; verify and document.
    └── No  -> Q2
Q2: Have you obtained consent from the data principal?
    ├── Yes -> Document consent; standard obligations apply.
    └── No  -> Q3
Q3: Does a Section 7 legitimate use apply?
    ├── Yes -> Document; standard obligations apply.
    └── No  -> Stop; restructure to obtain consent or change scope.

For scrapers, the consent default and narrow legitimate use list make the DPDP much more restrictive than PDPA Singapore. The publicly-available carve-out is narrower than EU GDPR’s. Plan around this; do not assume.

Data principal rights

Sections 11-14 grant data principals four core rights: right to information about processing, right to correction and erasure, right to grievance redressal, and right to nominate (a person who can exercise rights on the principal’s behalf).

The right to erasure is broader than GDPR’s because it applies on consent withdrawal and is not subject to the same balancing tests. A scraper who relies on consent and the principal withdraws consent must erase, full stop.

The right of grievance redressal requires the data fiduciary to maintain an easily accessible grievance mechanism. The DPB only accepts complaints after the principal has exhausted the fiduciary’s grievance mechanism. A scraper without a public, responsive grievance inbox is exposed.

For the worked operational pattern, see the personal vs public data scraping framework.

Children’s data and verifiable parental consent

Section 9 imposes additional obligations for data of children (under 18 in India). A data fiduciary must obtain verifiable consent from the parent or lawful guardian before processing children’s data. Tracking, behavioural monitoring, and targeted advertising directed at children are prohibited.

For scrapers, this is a major exposure if any source contains children’s data. Social media platforms, gaming platforms, education platforms, and parts of the public web all contain user-generated content from minors. The DPDP’s verifiable parental consent requirement is not a “best efforts” standard; it requires a specific verifiable mechanism.

The 2026 rules specified acceptable verification mechanisms including Aadhaar-based verification for parents and acceptable third-party verification services. Scrapers who cannot verify must avoid processing children’s data, which in practice means filtering aggressively at intake.

How DPDP enforcement is shaping up in 2025-2026

The DPB began operations in 2024 and has issued its first round of decisions in 2025-2026. The early pattern: focus on consent quality, breach notification timing, and grievance redressal mechanisms. Several mid-sized fines (INR 5-25 crore range) were issued for missing or non-functional grievance redressal.

Larger fines are anticipated as the DPB completes its first investigations of SDFs. The maximum INR 250 crore penalty per breach has not yet been imposed but is reserved for systematic violations affecting large numbers of data principals.

The DPB is more litigation-focused than the PDPC or the EDPB; it is an adjudicatory body. Operators who engage early and constructively in proceedings tend to settle for substantially reduced penalties.

External references

The DPDP Act 2023 full text is at meity.gov.in/dpdp-act-2023. The Digital Personal Data Protection Rules 2026 are at the same MeitY portal. Data Protection Board notices and decisions are published at dpb.gov.in (note: as of mid-2026 the official site is in active build-out).

Comparison: DPDP vs PDPA vs GDPR for scrapers

Dimension	DPDP India	PDPA Singapore	GDPR EU
Default basis	Consent	Consent (with carve-outs)	Lawful basis (six options)
Public data carve-out	Narrow (data principal disclosed)	Broad (generally available)	Narrow
Legitimate Interests	Limited (Section 7 list)	Yes (since 2020)	Yes (Article 6(1)(f))
Cross-border transfer	Whitelist (operator-friendly)	Comparable protection	SCCs / adequacy
Right to erasure	Yes (on consent withdrawal)	Limited	Yes (Article 17)
DPO mandatory	For SDFs only	All organisations	Conditional
Maximum fine	INR 250 crore (~USD 30M)	SGD 1M or 10% turnover	EUR 20M or 4% turnover
Children’s data extra	Verifiable parental consent	Standard rules	Article 8 conditions
AI training friendly	Not articulated	DIP since 2024	EU AI Act layers on
Adjudicatory body	DPB (litigation-style)	PDPC (regulator-style)	National DPAs (regulator-style)

The DPDP is broadly stricter than PDPA on default basis but more flexible on cross-border transfer. It is comparable to GDPR on overall stringency but with different operational textures.

A worked example: scraping Indian ecommerce listings

A scraper collects publicly available product listings from major Indian ecommerce sites (Flipkart, Myntra, Meesho, Amazon India). The dataset includes product name, price, seller name, seller location, and seller rating.

Classification: seller name and location are personal data if the seller is an individual (many Indian ecommerce sellers are sole proprietors). Product name and price are not personal data.

Basis: the publicly-available carve-out applies if the seller voluntarily made their information public on the platform. Most platform terms require sellers to display their contact information. A defensible basis exists.

Notice: a public privacy notice on the scraper’s website describing collection, purpose, retention.

Grievance: an easily accessible grievance redressal mechanism (preferably an inbox, ideally a portal).

Cross-border: if data is transferred to non-restricted countries (US, EU, Singapore), no additional mechanism required as of mid-2026.

Outcome: defensible posture, with low ongoing overhead, with the publicly-available basis documented and a grievance mechanism live.

For a parallel ecommerce-targeted scrape, see scraping Flipkart India product data.

Special cases: AI training and political data

The DPDP Act does not yet have a specific AI training carve-out or framework. The current operator posture is to obtain consent or rely on Section 7 legitimate uses where possible, and to maintain documented training data manifests in anticipation of future rules.

Political data (party affiliation, voting behaviour, election-related profiling) is treated with extra caution by the DPB, with multiple early enforcement actions targeting election-time profiling operations. Scrapers should avoid political data entirely unless they have a defensible democratic-interest basis, which is hard to construct.

FAQ

Is publicly available data exempt from DPDP?
Only if the data was made publicly available by the data principal themselves or by a person obligated under law. Third-party publication does not qualify.

Do I need consent to scrape Indian data?
The default is yes. The Section 7 legitimate uses list is narrow and rarely applies to scraping operations.

Does DPDP apply if I am outside India?
Yes if your processing relates to offering goods or services to data principals in India. Extraterritorial reach is similar to GDPR.

Do I need a local DPO?
Only if classified as a Significant Data Fiduciary. SDFs must appoint an India-based DPO.

What is the cross-border transfer rule?
A positive whitelist model: transfer is permitted by default to any country not on the government’s restricted list. Major scraping destinations are unrestricted as of mid-2026.

Extended DPDP Act enforcement analysis

The Digital Personal Data Protection Act 2023 received presidential assent in August 2023. The Data Protection Board of India was constituted in 2024, and the Act’s substantive provisions came into force in phased fashion through 2025 and 2026. The DPDP rules notified in early 2025 fleshed out consent manager registration, breach notification timelines, and cross-border transfer mechanics.

The Act’s territorial reach (Section 3) covers processing of digital personal data in India, plus processing outside India if it relates to offering goods or services to data principals in India. The latter prong directly captures scrapers harvesting India-resident data from anywhere in the world.

The lawful-basis architecture differs from GDPR. The DPDP recognises two pathways. First, consent under Section 6, which must be free, specific, informed, unconditional, and unambiguous, and given through clear affirmative action. Second, certain legitimate uses under Section 7, including performance of state functions, employment, medical emergencies, disaster response, and certain specified purposes. There is no general legitimate-interest catch-all of the GDPR Article 6(1)(f) kind.

For scrapers the practical implication is that consent is the dominant pathway, and consent for third-party scraping is rarely available. The narrow Section 7 routes do not generally cover commercial scraping. Operators must therefore consider whether DPDP applies and design accordingly.

Implementation patterns for India-touching scraping

A 2026 DPDP-aware scraping pipeline should implement eight controls.

Identify India-resident data principals at ingest.
Distinguish between processing under consent and under Section 7 legitimate use, with documentation per record.
Honour withdrawal of consent and erasure requests.
Maintain a grievance officer and process within the statutory time limit.
Apply notified-country transfer rules for cross-border movement.
Apply enhanced safeguards for children’s data (under 18) including verifiable parental consent.
Maintain breach notification within the prescribed window to the DPB.
Maintain a data protection impact assessment for significant data fiduciary status if applicable.

Code pattern: India identification and consent gate

import re

IN_PHONE = re.compile(r"\+?91[\s-]?\d{5}[\s-]?\d{5}")
IN_DOMAINS = {"in", "co.in", "org.in", "ac.in", "gov.in"}

def is_india_subject(record):
    if IN_PHONE.search(record.get("text", "")):
        return True
    if any(record.get("email","").endswith("." + d) for d in IN_DOMAINS):
        return True
    if record.get("country_iso") == "IN":
        return True
    return False

def consent_gate(record):
    if not is_india_subject(record):
        return True
    if record.get("consent_token"):
        return True
    if record.get("section_7_basis") in {"employment", "medical_emergency", "disaster", "state_function"}:
        return True
    return False

Comparison: DPDP vs GDPR for scrapers

Question	India DPDP	EU GDPR
Lawful bases	Consent plus narrow Section 7 list	Six bases including legitimate interest
Territorial reach	India processing plus targeting	EU processing plus targeting plus monitoring
Children threshold	Under 18	Under 16 default, member states can lower to 13
Cross-border transfer	Notified country list	SCCs, adequacy, derogations
Max fine	INR 250 crore	EUR 20 million or 4 percent global turnover
Breach notification	DPB notification window	72 hours to DPA, individuals where high risk
Data principal rights	Access, correction, erasure, grievance, nominate	Access, rectification, erasure, restriction, portability, object

Additional FAQ

Does DPDP have a legitimate-interest catch-all?
No. Section 7 specifies discrete legitimate uses. Commercial scraping of public data does not fit any listed category.

What is a Significant Data Fiduciary?
A class designated by the central government based on volume, sensitivity, risk to electoral democracy or public order, sovereignty, or India’s security. Significant Data Fiduciaries face enhanced obligations.

How does DPDP treat AI training data?
The Act does not have AI-specific provisions, but consent or a Section 7 ground is required for any personal data processing including training.

What happens to non-personal data?
DPDP regulates only digital personal data. A separate non-personal data framework has been discussed but not enacted as of 2026.

The DPDP Act’s consent architecture

The DPDP Act places consent at the centre of the lawful processing analysis. Section 6 specifies that consent must be free, specific, informed, unconditional, and unambiguous, and must be given through clear affirmative action. The Act introduced the concept of a Consent Manager (Section 6(7)), a registered intermediary that can manage consent on behalf of data principals.

For scrapers the consent requirement is generally not satisfiable. Third-party scraping involves no direct interaction with the data principal, so affirmative consent cannot be obtained. The consent pathway is therefore typically unavailable for commercial scraping.

The Consent Manager mechanism is a 2026 innovation that may eventually create new pathways. A data principal who has consented through a Consent Manager can have that consent presented to downstream services. The infrastructure is still maturing, and broad scraper-friendly consent flows have not emerged. Scrapers should monitor the development but should not rely on it for current operations.

Section 7 legitimate uses in detail

Section 7 of the DPDP Act lists the legitimate uses for which processing may occur without consent. The list is closed, meaning it cannot be expanded by regulator interpretation. The legitimate uses include: processing for state functions; medical emergency; disaster response; employment-related processing; and a few specific public interest categories.

Critically the list does not include a general legitimate-interest catch-all comparable to GDPR Article 6(1)(f). This makes the DPDP regime more restrictive for scrapers than the GDPR. A scrape that would pass an LIA under GDPR may have no available basis under DPDP.

The 2025 DPDP Rules provided some additional clarification on Section 7 application. The rules tightened the definition of employment-related processing and specified that the disaster response basis is limited to time-bound emergencies declared by competent authority. Neither change benefits scrapers.

Significant Data Fiduciary obligations

Section 10 of the DPDP Act allows the central government to designate certain data fiduciaries as Significant Data Fiduciaries (SDFs) based on volume, sensitivity, risk to electoral democracy, public order, sovereignty, or India’s security. SDFs face enhanced obligations including DPIA, data protection officer designation, and independent audit.

The 2025 government notification of initial SDF categories included social media platforms above thresholds, e-commerce platforms above thresholds, and certain AI service providers. Future notifications may extend to data brokers and aggregators.

For scrapers the practical implication is that operating at scale in India increases the risk of SDF designation. A scraper crossing the volume threshold or feeding sensitive AI applications should plan for SDF obligations. The compliance uplift is non-trivial, including a designated DPO, mandatory DPIAs, and annual audits.

Cross-border transfer under DPDP

The DPDP Act takes a different approach to cross-border transfer than GDPR. Section 16 permits transfer to any country except those specifically restricted by the central government through notification. The starting position is permissive, with restrictions added as needed.

The 2025 DPDP Rules clarified the transfer framework. As of 2026 only a small number of countries are on the restricted list, and the criteria for addition are based on national security and public order rather than data protection adequacy.

For scrapers the implication is that exporting India-resident data to most jurisdictions is permissible under the DPDP, but the source-side consent or Section 7 basis must still be established. The DPDP transfer rules do not create a lawful basis where none otherwise exists.

Next steps

The fastest path to DPDP compliance is to publish a privacy notice and grievance redressal mechanism, document the basis (preferably the narrow publicly-available carve-out where it applies), and prepare for SDF designation if your volume warrants it. For broader Asia compliance, head to the DRT compliance hub and pair this with the PDPA Singapore guide.

This guide is informational, not legal advice.

May 7, 2026

Scrapybara vs Browserbase for agentic workflows

The Scrapybara vs Browserbase decision has become the cleanest fork in the road for any team building agentic workflows in 2026. Both are managed cloud platforms designed for AI agents to drive real browsers and operating systems. They overlap on the surface and diverge significantly in philosophy. Browserbase is a browser cloud, period. Scrapybara is a virtual desktop cloud that happens to include a browser.

That difference shows up everywhere. This guide compares both platforms through the lens of an engineer building a scraping or research agent in 2026, with code, benchmarks, and a clear picture of which one fits which problem.

What each platform actually is

Browserbase gives you a Chromium browser session over a Playwright-compatible WebSocket endpoint. You write Playwright or Stagehand code, you connect to a managed session, you scrape. The platform handles fingerprinting, proxies, CAPTCHA solving, and observability. Browser-only.

Scrapybara gives you a full Ubuntu desktop in the cloud with a browser, a terminal, a file system, and X11. Your agent can launch any application, not just a browser. The platform exposes a Computer Use API directly compatible with Anthropic’s Claude Computer Use and OpenAI’s Operator.

If your agent only needs the web, Browserbase is the leaner choice. If your agent needs to download a file, manipulate it in a desktop application, or run a CLI tool, Scrapybara is the only real option.

Pricing in 2026

Both are usage-based but with different units.

Browserbase:

Plan	Cost	Browser-minutes
Developer	$39/mo	5,000 included, then $0.0078/min
Startup	$399/mo	70,000 included, then $0.0057/min

Scrapybara:

Plan	Cost	Compute-hours
Hobby	Pay-as-you-go	$0.10/hour
Pro	$99/mo	100 hours included, then $0.06/hour
Scale	Custom	Custom

A like-for-like comparison: 1000 sessions averaging 3 minutes each.

Platform	Compute cost	Proxy cost (residential)	Total
Browserbase Developer	$23.40	$4.50	$27.90
Browserbase Startup	$17.10	$4.50	$21.60
Scrapybara Hobby	$5.00	$4.50 (BYO)	$9.50
Scrapybara Pro	$3.00	$4.50 (BYO)	$7.50

Scrapybara is meaningfully cheaper on compute. Browserbase costs more but includes residential proxies, CAPTCHA solving, and the Stagehand framework as native primitives.

Mental model in one sentence each

Scrapybara is “give me a Linux box my agent can drive.” Browserbase is “give me a Chrome tab my agent can drive.”

If you find yourself wishing the agent could apt install ffmpeg and then run a CLI tool on a downloaded video, Scrapybara fits. If you find yourself wishing the agent could click around five sites and emit JSON, Browserbase fits.

Setup speed

Both platforms ship in minutes.

Browserbase:

import { Browserbase } from "@browserbasehq/sdk";
import { chromium } from "playwright";

const bb = new Browserbase({ apiKey: process.env.BROWSERBASE_API_KEY! });
const session = await bb.sessions.create({ projectId: process.env.BROWSERBASE_PROJECT_ID! });
const browser = await chromium.connectOverCDP(session.connectUrl);
const page = browser.contexts()[0].pages()[0];

await page.goto("https://example.com");
console.log(await page.title());

await browser.close();

Scrapybara:

from scrapybara import Scrapybara

client = Scrapybara(api_key="scrapy_...")

instance = client.start_ubuntu()
instance.bash(command="echo hello")
instance.computer.screenshot()  # returns base64 PNG of the desktop
instance.browser.start()
instance.browser.goto("https://example.com")
title = instance.browser.evaluate("document.title")
print(title)

instance.stop()

Browserbase ships a Playwright-compatible API, so any existing Playwright code drops in. Scrapybara ships a richer API with desktop primitives, but your existing Playwright code needs adaptation.

Latency to first action

Cold-start times measured March 2026:

Platform	Cold start	Warm	First navigation
Browserbase	2.1 s	0.4 s	0.8 s after start
Scrapybara Ubuntu	8.4 s	2.1 s	1.4 s after start

Browserbase wins on latency because it boots only Chromium. Scrapybara boots a whole desktop. For pure browser workloads, the latency cost on Scrapybara is real. For multi-app workloads, the latency is amortized over a longer session.

Computer Use integration

This is where Scrapybara pulls ahead for agentic workflows.

Scrapybara is built specifically as a Computer Use target. Anthropic’s Claude Computer Use API and OpenAI’s Operator both plug into Scrapybara as the underlying compute environment.

from scrapybara import Scrapybara
from anthropic import Anthropic

client = Scrapybara(api_key="scrapy_...")
instance = client.start_ubuntu()
anthropic = Anthropic()

response = anthropic.beta.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=4096,
    tools=[{
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px": 1024,
        "display_height_px": 768,
    }],
    messages=[{
        "role": "user",
        "content": "Open the browser, search Google for 'browserbase pricing', and return the first result URL."
    }],
    betas=["computer-use-2024-10-22"],
)

# loop: feed Claude's tool calls into instance.computer.* and feed back

Browserbase, by contrast, supports Computer Use only through the Stagehand agent primitive, and only for the browser. If your agent needs to open Excel, manipulate files in a terminal, or interact with desktop apps, Browserbase cannot help.

For more on Computer Use vs Operator, see our OpenAI Operator vs Anthropic Computer Use comparison.

Browser primitive depth

Where Browserbase is more polished is the browser experience. Stagehand is theirs, and it shows.

Browserbase + Stagehand:

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const stagehand = new Stagehand({ env: "BROWSERBASE", modelName: "gpt-4o-mini" });
await stagehand.init();
const page = stagehand.page;
await page.goto("https://news.ycombinator.com");

const top = await page.extract({
  instruction: "Top 5 stories with title, score, submitter",
  schema: z.object({
    stories: z.array(z.object({ title: z.string(), score: z.number(), submitter: z.string() })),
  }),
});
console.log(top);
await stagehand.close();

Scrapybara has a browser API but no equivalent of Stagehand’s extract. You bring your own LLM extraction layer.

For the Stagehand vs Playwright story, see Stagehand vs Playwright for AI scraping.

Side-by-side comparison

Dimension	Scrapybara	Browserbase
Compute target	Full Ubuntu desktop	Chromium browser only
Native API surface	Browser, terminal, file system, GUI	Browser only
Computer Use support	First-class for Claude and Operator	Browser-only via Stagehand agent
AI extraction primitives	BYO	Stagehand native
Proxy support	BYO	Built-in residential and stealth
CAPTCHA solving	BYO	Built-in for major types
Session replay	Yes (video plus event trace)	Yes (video plus event trace)
Cost per 1000 short sessions	$7 to $10	$20 to $28
Best fit	Agents that need full OS access	Browser-only AI workflows

The two products are complementary more than competitive. Scrapybara is the right pick when your agent needs to do anything outside a browser. Browserbase is the right pick when your work fits in a browser and you want the most polished AI-native experience.

Computer Use action loop on Scrapybara

The actual loop that drives Claude Computer Use against a Scrapybara instance is simple but worth seeing in full.

def computer_action(action_input):
    if action_input["action"] == "screenshot":
        return instance.computer.screenshot()
    elif action_input["action"] == "left_click":
        x, y = action_input["coordinate"]
        return instance.computer.left_click(x=x, y=y)
    elif action_input["action"] == "type":
        return instance.computer.type(text=action_input["text"])
    elif action_input["action"] == "key":
        return instance.computer.key(text=action_input["text"])
    # ... more actions

messages = [{"role": "user", "content": "Open the browser and search Hacker News"}]
while True:
    resp = anthropic.beta.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        tools=[{"type": "computer_20241022", "name": "computer",
                "display_width_px": 1024, "display_height_px": 768}],
        messages=messages,
        betas=["computer-use-2024-10-22"],
    )
    if resp.stop_reason == "end_turn":
        break
    for block in resp.content:
        if block.type == "tool_use":
            result = computer_action(block.input)
            messages.append({"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": [{
                "type": "tool_result", "tool_use_id": block.id, "content": result,
            }]})

The loop is roughly 30 lines and works for any Computer Use task. Browserbase has an analogous loop but only for browser actions.

Real production patterns

We ran two pipelines in parallel to compare.

Pipeline A: scrape Lazada and Shopee product listings, extract to Postgres. Pure browser work.

Pipeline B: download invoice PDFs from a supplier portal, extract line items with a vision model, push to a procurement system. Needs file download and PDF processing.

Pipeline A on Browserbase + Stagehand:

Time to working code: 2 hours
Cost per 1000 products: $6 (compute) + $4 (residential proxy) + $2 (LLM)
Maintenance burden: low

Pipeline A on Scrapybara:

Time to working code: 4 hours (had to wire LLM extraction layer)
Cost per 1000 products: $4 (compute) + $4 (BYO proxy) + $2 (LLM)
Maintenance burden: medium

Pipeline B on Browserbase: not feasible. The browser cannot natively process the downloaded PDFs.

Pipeline B on Scrapybara:

Time to working code: 5 hours
Cost per 1000 invoices: $9 (compute, longer sessions) + $3 (proxy) + $5 (vision model)
Maintenance burden: medium

Detailed pipeline metrics

The pipelines above ran for 30 days each. Aggregate numbers:

Metric	Pipeline A on Browserbase	Pipeline A on Scrapybara	Pipeline B on Scrapybara
Pages or invoices processed	90,000 products	90,000 products	12,000 invoices
Total cost	$1,080	$720	$204
Per-unit cost	$0.012	$0.008	$0.017
Median latency	7.4 s	9.1 s	38 s
Success rate	97.4%	95.1%	91.2%
On-call incidents	1	4	5

Browserbase’s higher cost was offset by lower incident count. Scrapybara’s lower cost came with more operational overhead because the BYO proxy layer needed its own monitoring. The pipeline B numbers are for a fundamentally different workload (PDF processing) where Scrapybara was the only option.

Where each platform showed weakness

Browserbase’s main weakness was the proxy markup. Halfway through the test, we switched the highest-volume target to BYO proxies through the session API, which dropped the proxy cost line by roughly 60 percent. The platform supports BYO but the docs do not lead with it.

Scrapybara’s main weakness was the lack of an extraction primitive. We had to wire OpenAI structured output ourselves, which added roughly two hours of development time per new target. The Scrapybara team has hinted at native extraction primitives in 2026 H2 but as of writing it is still BYO.

Adding proxies

Both platforms support proxies, but the integration depth differs.

Browserbase ships first-party residential proxies billed at $8/GB and stealth (datacenter) at $0.30/GB. Set them at session creation:

const session = await bb.sessions.create({
  projectId: process.env.BROWSERBASE_PROJECT_ID!,
  proxies: true, // or pass an explicit proxy config
});

Scrapybara expects you to bring your own. Configure inside the instance:

instance.browser.start(proxy={"server": "http://proxy.example.com:8000", "username": "u", "password": "p"})

For ASEAN scraping where mobile carrier IPs matter, Singapore mobile proxy plugs into either platform with one config block.

Geographic coverage

Browserbase: US East, US West, EU West, APAC South (Singapore). Compute always in one of these regions; proxies cover roughly 195 countries.

Scrapybara: US East, US West, EU West. APAC region is on the roadmap as of mid-2026. For Asia-targeted scraping, latency is higher than Browserbase.

CAPTCHA story in detail

Browserbase ships built-in solvers for reCAPTCHA v2/v3, hCaptcha, and Cloudflare Turnstile, with quoted success rates of 88 to 95 percent depending on type.

Scrapybara has no built-in CAPTCHA solving. You wire 2Captcha, CapSolver, or a similar service. The integration takes maybe 50 lines of code, but it is friction the Browserbase user does not have.

For teams that scrape CAPTCHA-heavy targets (LinkedIn, Indeed, several banking portals), Browserbase’s built-in solver is a real time-saver. For targets without CAPTCHAs (most B2B SaaS apps you have legitimate credentials for), the difference is irrelevant.

Observability

Both ship session replay with video and event traces. Browserbase’s UI is more polished and the search across sessions is faster. Scrapybara’s replay shows the full desktop, which is more useful when your agent uses non-browser apps.

Both export OpenTelemetry spans for trace correlation with your existing observability stack.

Security and isolation

Browserbase sessions run in ephemeral containers with full process isolation. Cookies, storage, and any cached state are wiped at session end unless you opt into context persistence. Network egress is funneled through the platform’s IP pool by default.

Scrapybara instances are full Ubuntu VMs with disk persistence per instance ID. This is more powerful and more dangerous: a compromised agent that gets shell access on a Scrapybara instance can persist files there, install software, and reuse the state across runs. For sensitive workloads, treat Scrapybara instances as ephemeral and tear them down explicitly.

Both platforms hold SOC 2 Type II reports as of 2026 and offer DPAs for GDPR-relevant workloads. Neither is yet HIPAA-eligible, so healthcare scraping requires self-hosting.

SDKs and language ecosystem

Browserbase’s primary SDK is TypeScript. The Python SDK exists but lags by roughly one release. For TypeScript-heavy teams, the platform feels native; for Python teams, less so.

Scrapybara’s primary SDK is Python. The TypeScript SDK exists but covers fewer features. For Python-heavy teams (most data science and AI shops), this is the natural fit.

If your team is split or polyglot, Browserbase wins because its WebSocket endpoint speaks any Playwright client. Scrapybara is more SDK-locked because the Computer Use primitives are first-class only in their SDKs.

Production recommendations

Use Browserbase if:

Your scraping is browser-only
You want Stagehand as your AI primitive layer
You value built-in CAPTCHA solving
You prefer a TypeScript-first SDK

Use Scrapybara if:

Your agent needs file system, terminal, or desktop access
You are running Claude Computer Use or OpenAI Operator
You want lower per-hour compute cost and BYO proxy
You prefer a Python-first SDK

The mature pattern in 2026 is to use both: Browserbase for the high-volume browser scraping, Scrapybara for the agent workflows that go beyond the browser. They cost together about what one Browserbase Startup plan costs, and you cover both shapes of work.

For broader context on the agentic browser space, see our agentic browser revolution guide.

Decision matrix

Your situation	Pick
Browser-only scraping, want fastest path	Browserbase + Stagehand
Browser-only scraping, want lowest cost	Self-hosted Playwright (not Scrapybara, not Browserbase)
Agent needs file system or terminal	Scrapybara
Running Claude Computer Use or OpenAI Operator	Scrapybara
TypeScript-first team, multi-site scraping	Browserbase
Python-first team, complex multi-app workflow	Scrapybara
Need built-in CAPTCHA solving	Browserbase
Need built-in residential proxies, simple billing	Browserbase
Need to install custom CLI tools at runtime	Scrapybara
One-off prototype, single site	Either, slight Browserbase edge

Frequently asked questions

Can I run my own LLM extraction prompts on Scrapybara?
Yes. Scrapybara just exposes the underlying compute. Wrap your favorite LLM (OpenAI, Anthropic, local) and call it from your agent code.

Are there any Scrapybara features Browserbase has copied or is likely to copy?
Both platforms ship session replay, both ship persistent contexts. Browserbase has hinted at adding limited terminal access on Scale plans but has not shipped. Scrapybara has hinted at native extraction primitives. Convergence is happening slowly.

Does Browserbase have a Python SDK?
Yes, but the TypeScript SDK is more feature-complete. For the latest features, TypeScript is the better choice in early 2026.

Can Scrapybara run alongside other browser frameworks?
Yes. The Scrapybara browser is a real Chromium that speaks CDP. You can drive it from Playwright, Puppeteer, or any browser automation library.

What about Hyperbrowser, Steel.dev, and other competitors?
Hyperbrowser is closest to Browserbase in shape but smaller. Steel.dev sits between Browserbase and Scrapybara on capabilities. Both are valid alternatives if pricing or feature set fits better.

How does authentication work for sites that need login?
Both platforms support persistent contexts that store cookies and localStorage. Save once after manual login, reload on each session. For higher-throughput pipelines, store the context state encrypted in your secret store.

Can I expose a Scrapybara instance to the public internet?
Not directly. The instance is private to the Scrapybara network. To expose a service running on the instance, tunnel through ngrok or a similar reverse proxy from inside the instance.

What are the per-instance resource limits on Scrapybara?
Default Hobby tier: 2 vCPU, 4 GB RAM, 20 GB disk. Pro: 4 vCPU, 8 GB RAM, 50 GB disk. Scale: negotiated. For most browser-driving tasks, Pro is sufficient. For video processing or large data jobs, request a custom config.

Can either platform handle file uploads to a target site?
Both can. Browserbase exposes the standard Playwright setInputFiles API. Scrapybara supports the same plus drag-and-drop from the desktop file system.

What about secrets management?
Neither platform provides a secrets vault. Pass secrets via environment variables to your code, or pull from your own secrets manager (AWS Secrets Manager, HashiCorp Vault, Doppler) before passing to the platform.

Can I use both platforms in the same workflow?
Yes. A common pattern is to use Browserbase for the high-volume scraping leg and Scrapybara for the post-processing leg that involves file manipulation. They are complementary by design.

Common production gotchas

For Browserbase: forgetting to release sessions burns minutes; mismatched proxy and target region triggers cloaking; the LLM bill hides separately on your OpenAI account.

For Scrapybara: the desktop is a real Ubuntu so resource leaks (lingering processes, large temp files) accumulate across long-lived instances; explicitly stop instances when done; mount your file output to a known location for export rather than relying on the temp file system.

For both: session replay retention is finite. Export critical traces if you need them long-term. The platforms do not retain everything indefinitely.

Six-month verdict

After running both side by side for six months: Browserbase wins for any team where 100 percent of work is browser-based. The polish and Stagehand integration are worth the premium. Scrapybara wins for any team where even 20 percent of work goes outside the browser, because that 20 percent is otherwise blocking. The right answer for many teams is to use Browserbase as the default and reach for Scrapybara when the task explicitly needs OS access.

If you are deciding between agentic platforms, the AI modern scraping category has more reviews and head-to-head comparisons that can help anchor the decision.

May 7, 2026

Singapore PDPA for scrapers: a 2026 practical guide

PDPA Singapore scraping rules are the most pragmatic in the ASEAN region, and that pragmatism has made Singapore one of the most attractive jurisdictions for data-driven businesses operating across Asia. The Personal Data Protection Act, originally enacted in 2012 and significantly amended in 2020 and 2024, governs how personal data is collected, used, disclosed, and stored. Unlike GDPR, the PDPA includes a relatively broad publicly-available exception that scraping operators can rely on, and unlike most other regimes it explicitly distinguishes consent obligations from data protection obligations. This guide walks through the PDPA structure, the publicly-available rules, the deemed consent and notification frameworks, and a working compliance checklist.

The audience is the technical lead or in-house counsel responsible for a scraping pipeline that touches Singapore residents, or one based in Singapore that touches anywhere.

What the PDPA actually covers in scraping context

The PDPA applies to any organisation that collects, uses, or discloses personal data about individuals in Singapore. It applies regardless of whether the organisation is in Singapore. Like GDPR, it has effective extraterritorial reach when scraping operations target Singapore residents.

Personal data under Section 2 is data, whether true or not, about an individual who can be identified from that data, or from that data and other information to which the organisation has or is likely to have access. The definition mirrors GDPR’s “identifiable” standard but with a slightly narrower “is likely to have access” qualifier that gives operators a small drafting window.

The PDPC (Personal Data Protection Commission) enforces the regulation. Penalties under the 2020 amendments rose significantly: financial penalties of up to SGD 1 million or 10 percent of annual turnover in Singapore (whichever is higher), with the higher cap applying to organisations with annual turnover above SGD 10 million.

The 2024 amendments introduced the Data Innovation Provisions, allowing certain forms of business analytics and AI training under the Legitimate Interests basis, with documentation and disclosure requirements. This is the most operator-friendly addition in the region.

For the broader ASEAN compliance picture, see the personal vs public data scraping framework. For comparison with GDPR, see the GDPR compliance guide.

The publicly-available exception, properly read

The PDPA’s publicly-available exception is broader than the GDPR’s equivalent, but narrower than most operators assume. Schedule 1 Part 3 of the PDPA exempts collection and use of personal data that is publicly available, defined in Section 2 as personal data that is generally available to the public, and includes personal data that can be observed by reasonably expected means at a location or event at which the individual appears and that is open to the public.

Three operational implications.

First, “generally available to the public” requires that the data be available to anyone who looks, not just to those who clear a barrier. A profile behind a paywall is not publicly available. A profile behind a free signup is debatable.

Second, the exception applies to collection and use, but not always to subsequent disclosure to third parties. If you scrape publicly available data and resell it, the resale may trigger separate obligations.

Third, the PDPC has consistently held that observable behaviour at public events (a name on a public attendee list, a quote in a public news article) is publicly available. Behaviour inferred from observation (a profile built from behavioural patterns) may not be.

The PDPC issued an advisory in 2024 specifically addressing scraping for AI training, holding that publicly available data may be used for training without consent under the publicly-available exception, provided the use complies with the data protection obligations (notification, purpose limitation, accuracy, protection, retention, transfer).

Consent and the notification obligation

For data not within the publicly-available exception, the PDPA requires consent. Consent can be express (the data subject explicitly agrees) or deemed (the individual voluntarily provides the data for a purpose, or is informed and does not opt out within a reasonable time). The 2020 amendments expanded deemed consent significantly.

For scrapers, deemed consent rarely applies because the data subject did not voluntarily provide the data to you. The relevant alternative bases are:

Legitimate Interests: introduced in 2020, allows collection without consent where the legitimate interest of the organisation outweighs any adverse effect on the individual. Requires a documented assessment.

Business Improvement: a narrow exception for using existing personal data to improve products and services, subject to safeguards.

Research: a research exception for non-commercial research purposes.

Notification, even where consent is not required, is generally still required. The organisation must notify the individual of the purposes for which the data will be collected, used, or disclosed. For scraping operators, notification is typically delivered through a public privacy notice rather than per-individual contact.

For the Indian comparison and where the regimes differ, see the India DPDP Act for scrapers guide.

The Do Not Call provisions

The PDPA includes Do Not Call (DNC) provisions that govern marketing communications to Singapore phone numbers. These rules sit alongside the data protection obligations and are independently enforced.

Scrapers who collect Singapore phone numbers and use them (or licence them) for marketing must check the DNC registries before sending. The PDPC operates three registers (No Voice Call, No Text Message, No Fax). Failing to check before sending is a separate violation with separate fines.

The 2020 amendments added that organisations are responsible for ensuring third-party marketers they engage also comply with DNC. A scraper that resells phone numbers to marketing operators is exposed to this chain liability.

Compliance checklist for scraping operators

Control	What it requires	Why it matters
Publicly-available assessment	Per-source documentation	Schedule 1 Part 3 defence
Lawful basis for non-public data	Legitimate Interests assessment	Section 13
Privacy notice published	Public page describing purposes	Section 20 (notification)
Purpose limitation	Use only for stated purposes	Section 18
Data accuracy	Reasonable steps to ensure accuracy	Section 23
Protection obligation	Reasonable security arrangements	Section 24
Retention limits	Cease retention when no longer needed	Section 25
Transfer limits	Comparable protection in recipient country	Section 26
Data Protection Officer	Mandatory for all organisations	Section 11
Access and correction requests	Respond within 30 days	Sections 21-22
Withdrawal of consent	Honour withdrawal	Section 16
Do Not Call check (if marketing)	Per-number, current registry	DNC provisions
Data Innovation Provisions notice	If using LI for AI training	2024 amendments
Breach notification	If significant harm or 500+ affected	Section 26D

Most scrapers can tick most rows in a fortnight of work. The DPO requirement (Section 11) is the most-missed obligation.

Decision tree: is this scrape PDPA-compliant?

Q1: Is the source publicly available (general access, no barrier)?
    ├── Yes -> Q1a: Is the use for AI training or aggregation?
    │           ├── Yes -> Document publicly-available basis; comply with data protection obligations.
    │           └── No  -> Document publicly-available basis; standard obligations apply.
    └── No  -> Q2
Q2: Have you obtained express or deemed consent?
    ├── Yes -> Document; standard obligations apply.
    └── No  -> Q3
Q3: Can you rely on Legitimate Interests?
    ├── Yes -> Conduct LI assessment; publish notice; standard obligations apply.
    └── No  -> Stop or restructure.

The Data Innovation Provisions and AI training

The 2024 amendments added the Data Innovation Provisions (DIP) at Sections 17A-17C. These allow organisations to use personal data, without consent, for business innovation purposes that include analytics, AI training, and product development, subject to four conditions:

The use is for a legitimate purpose that the individual would reasonably expect.
The organisation has conducted a risk assessment.
The organisation publishes a clear notice describing the use.
The organisation provides an opt-out mechanism that is honoured.

For scraping operators training AI models on publicly available Singapore data, the DIP is the cleanest path. Document the assessment, publish the notice, run the opt-out inbox.

Cross-border transfer obligations

Section 26 requires that personal data transferred outside Singapore be afforded a standard of protection comparable to the PDPA. The PDPC’s approach is more flexible than the EU’s, accepting the recipient’s contractual undertakings, the recipient’s binding corporate rules, or the recipient’s location in a jurisdiction with comparable laws.

The PDPC has not published a formal adequacy list. Instead, scraping operators evaluate each recipient case-by-case. Major comparable jurisdictions include the EU/EEA, the UK, Australia, Canada, Japan, and South Korea.

For US transfers, the PDPC accepts contractual clauses similar to the EU SCCs. The Data Privacy Framework is not directly relevant to PDPA, but a US recipient certified under DPF generally satisfies PDPA-equivalent protection in practice.

For the broader cross-border question, see scraping data from EU sites jurisdictional realities.

How PDPA enforcement shifted in 2024 and 2025

Two trends. First, the PDPC moved from advisory-heavy to fine-active. Multiple seven-figure fines in 2024-2025 against organisations that failed to implement reasonable security arrangements (Section 24) following data breaches. Scraping operators with unprotected storage are exposed.

Second, the PDPC published explicit AI guidance in 2024 and 2025 covering training data, output safety, and accountability. The guidance is non-binding but shapes enforcement expectations. A scraping operator who follows the guidance is in a defensible position.

The Voluntary Disclosure Programme (VDP), launched in 2025, encourages organisations that discover their own breaches to self-report in exchange for reduced penalties. For scraping operators who discover compliance gaps, the VDP is a useful tool.

External references

The PDPA full text is at pdpc.gov.sg/legislation/personal-data-protection-act. The PDPC advisories and guidelines are at pdpc.gov.sg/Guidelines-and-Consultation. The PDPC enforcement decisions library is searchable at pdpc.gov.sg/Commissions-Decisions.

Comparison: PDPA vs GDPR vs DPDP

Dimension	PDPA Singapore	GDPR EU	DPDP India
Personal data definition	Identifiable individual	Identifiable individual	Digital personal data of identifiable individual
Public data carve-out	Broad	Narrow	Limited
Consent default	Required unless exception	Lawful basis required	Required unless exception
Legitimate interests	Yes (since 2020)	Yes	Limited (notice and consent default)
AI training friendly	Yes (DIP since 2024)	EU AI Act layers on	Not yet articulated
Cross-border transfer	Comparable protection test	SCCs / adequacy	Whitelist of approved countries
Maximum fine	SGD 1M or 10% turnover	EUR 20M or 4% turnover	INR 250 crore (~USD 30M)
Mandatory DPO	Yes (all organisations)	Conditional	Yes for significant data fiduciaries
Breach notification	Yes (significant harm or 500+)	Yes (72 hours, risky breaches)	Yes

PDPA is the most operator-friendly of the three, particularly for AI training pipelines that fit the DIP framework. DPDP is the strictest on consent default. GDPR remains the strictest overall.

A worked example: scraping Singapore property listings

A scraper collects publicly available property listings from major Singapore portals (PropertyGuru, 99.co, EdgeProp). The dataset includes property address, asking price, agent name, agent contact phone, agent licence number, and listing date.

Classification: agent name and contact details are personal data. Property address is not personal data unless linked to an owner. Asking price is not personal data.

Basis: publicly-available exception applies to the agent contact data, because agents publish their information openly for purposes of being contacted by potential clients.

Notification: a clear privacy notice on the scraper’s website describing the collection, the purpose (market intelligence for B2B customers), the retention period, and the opt-out path.

Do Not Call: if the dataset is later used for marketing calls to those agents, the DNC registries must be checked per-number per-call.

Outcome: defensible posture, low overhead, with a documented publicly-available assessment and a published notice. PDPA does not require an LIA in this case because the publicly-available exception applies.

For the deeper market-intelligence build pattern, see scraping job board data for talent intelligence.

Mandatory Data Protection Officer

Every organisation in scope of PDPA must appoint a Data Protection Officer (Section 11). The DPO does not need to be in Singapore, does not need to be a lawyer, and can be a current employee with other responsibilities. Small scraping operations commonly designate the engineering lead or compliance manager.

The DPO must be contactable, and their contact details (or at least the role’s contact details) must be available to the public. A common implementation is a dpo@yourcompany.com inbox listed on the privacy notice page.

Failing to appoint a DPO is itself a violation. The PDPC has issued multiple enforcement actions for this failure alone.

FAQ

Is publicly available data exempt from PDPA?
Partially. The publicly-available exception covers collection and use, but data protection obligations (purpose limitation, accuracy, protection, retention, transfer) still apply.

Do I need consent to scrape?
Not always. The publicly-available exception, the Legitimate Interests basis, and the Data Innovation Provisions all permit collection without express consent in defined circumstances.

Does PDPA apply if I am outside Singapore?
Yes if your processing covers individuals in Singapore. The PDPA has effective extraterritorial reach.

Do I need a Singapore representative?
No. Unlike GDPR, PDPA does not require a local representative for non-Singapore organisations.

What is the typical fine under PDPA in 2026?
Penalties range from low six figures for technical breaches up to SGD 1 million or 10 percent of annual Singapore turnover for serious violations, with the higher cap applying to larger organisations.

Extended PDPA enforcement analysis 2024-2026

The Personal Data Protection Commission stepped up enforcement after the 2020-2021 amendments brought mandatory breach notification, an enhanced financial penalty cap (10 percent of annual turnover above SGD 10 million), and the data portability obligation. The 2024-2026 window saw three notable directions.

First, the PDPC published the AI Model Governance Framework second edition in May 2024. The framework treats training data provenance as a primary governance question and recommends documented LIA-equivalent assessments for personal data ingested into AI training pipelines.

Second, the PDPC’s enforcement decisions in 2024 and 2025 showed that scraping operators are squarely in scope when they collect personal data of Singapore residents, regardless of the operator’s location. The Section 13 consent obligation is the central question, with Section 17 deemed-consent and the legitimate-interest exception in the First Schedule providing the practical pathways.

Third, the cross-border transfer rules under Section 26 require the receiving controller to be bound to a comparable standard. The PDPC’s 2024 guidance accepts a narrow set of mechanisms (consent, contract, ASEAN MCCs, certifications). Scrapers exporting Singapore-resident data must document the chosen mechanism.

Implementation patterns for a PDPA-clean pipeline

A 2026 PDPA-compliant scraping pipeline should include seven controls.

Identify Singapore-resident data subjects at ingest using a combination of profile signals and IP geolocation.
Apply the legitimate-interest exception with a documented assessment, or rely on Section 17 deemed consent where applicable.
Honour withdrawal of consent requests with a measured response time.
Provide a do-not-call workflow for any phone numbers collected.
Apply transfer-limitation safeguards for data leaving Singapore.
Maintain a data protection officer designation and contact.
Maintain a data breach notification process meeting the 72-hour PDPC notification window.

Code pattern: Singapore identification at ingest

import re

SG_PHONE = re.compile(r"\+?65[\s-]?\d{4}[\s-]?\d{4}")
SG_DOMAINS = {"sg", "com.sg", "edu.sg", "gov.sg", "org.sg"}

def is_singapore_subject(record):
    if SG_PHONE.search(record.get("text", "")):
        return True
    email = record.get("email", "")
    if any(email.endswith("." + d) for d in SG_DOMAINS):
        return True
    if record.get("country_iso") == "SG":
        return True
    return False

Comparison: PDPA vs neighbouring regimes for scrapers

Question	Singapore PDPA	Malaysia PDPA	Indonesia PDP Law	Thailand PDPA
Legitimate interest exception	Yes (First Schedule)	No general exception	Yes (limited)	Yes (limited)
Public data carve-out	Limited	Limited	Limited	Limited
Cross-border transfer rule	Comparable standard	Whitelisted countries	Adequate protection	Adequate protection
Max fine	SGD 1M or 10 percent of turnover	RM 500K	IDR 5B or 2 percent of revenue	THB 5M plus criminal
Breach notification	72 hours to PDPC	Yes	Yes	72 hours

Additional FAQ

Does PDPA apply to scraping operators outside Singapore?
Yes if they collect, use, or disclose personal data of individuals in Singapore. The PDPA does not require an establishment in Singapore for jurisdiction.

Is the legitimate-interest exception identical to GDPR Article 6(1)(f)?
Functionally similar but procedurally different. Singapore requires a prescribed assessment and notification. The substantive balancing test is comparable.

What is the do-not-call obligation?
The DNC Registry under PDPA prohibits telemarketing calls, SMS, and faxes to numbers on the registry without clear and unambiguous consent. Scraped phone numbers must be checked.

How does PDPA treat AI training data?
The 2024 AI Model Governance Framework recommends provenance documentation and explicit assessment for personal data used in training. Compliance with the framework is voluntary but increasingly expected.

The PDPA’s deemed consent and legitimate interest pathways

The PDPA’s 2020-2021 amendments introduced two pathways that are particularly relevant to scrapers. The first is deemed consent under Section 17, which applies when an individual voluntarily provides personal data for a purpose, and the consent can be inferred from the circumstances. The second is the legitimate interest exception in the First Schedule, which permits collection, use, or disclosure of personal data without consent if the legitimate interests outweigh any adverse effect on the individual.

For scrapers the legitimate interest exception is the practical pathway. The exception requires a documented assessment, similar to the GDPR LIA. The PDPC’s 2021 advisory guidelines on the legitimate interests exception provide a template for the assessment. Scrapers should follow the template and maintain the documentation.

Deemed consent under Section 17 is narrower for scrapers because the inference of consent from circumstances is harder for third-party scraping. A direct interaction (a user submitting a form) may support deemed consent. A scrape of a third-party website typically does not.

The 2024 PDPC enforcement decisions reaffirmed that the legitimate interest exception requires actual documentation. A scraper that has not written the assessment cannot rely on the exception. The decisions also reaffirmed that the assessment must be specific to the scrape, not boilerplate.

The PDPC AI Model Governance Framework

The PDPC published the AI Model Governance Framework first edition in 2019 and the second edition in May 2024. The framework provides voluntary guidance on responsible AI deployment. The 2024 edition added explicit guidance for generative AI and for training data.

For scrapers feeding AI training pipelines the framework recommends three practices. First, document the training data sources and the lawful basis for each. Second, conduct a data protection impact assessment for the training pipeline. Third, maintain a process for honouring data subject withdrawal requests.

Compliance with the framework is voluntary. The 2024-2026 trend is that compliance is increasingly expected by enterprise customers, by acquirers in due diligence, and by regulators in inquiries. A scraper that aligns with the framework is in a stronger market position.

Cross-border transfer under Section 26

Section 26 of the PDPA prohibits the transfer of personal data outside Singapore unless the transferring organisation ensures that the receiving organisation is bound to a comparable standard of protection. The 2021 amendments and the 2024 PDPC guidance specify the acceptable mechanisms.

The acceptable mechanisms are: written contract that imposes obligations comparable to the PDPA; binding corporate rules within a corporate group; the ASEAN Model Contractual Clauses for cross-border data flows; certification under the APEC Cross-Border Privacy Rules; and a few other narrow options.

For scrapers the contract pathway is the workhorse. The contract should explicitly reference the PDPA obligations and require the receiving organisation to maintain comparable safeguards. The 2024 PDPC guidance includes template clauses that scrapers can adapt.

The 2024 ASEAN MCCs provide an alternative for scrapers operating across ASEAN member states. The MCCs are aligned with the PDPA in principle and reduce the contract drafting burden. Adoption is voluntary and growing.

Next steps

The fastest path to PDPA compliance is to appoint a DPO, publish a privacy notice, document the publicly-available basis per source, and stand up an opt-out inbox. For broader Asia-Pacific compliance, head to the DRT compliance hub and pair this with the DPDP Act guide.

This guide is informational, not legal advice.

May 7, 2026

Browserbase review 2026: AI-native scraping platform

This Browserbase review for 2026 is built from real production usage across three different scraping projects, covering ecommerce monitoring, lead enrichment, and a generative content pipeline that uses Browserbase to render pages for AI ingestion. After six months of regular use, here is the honest read on what works, what does not, and where Browserbase fits in the modern scraping stack.

Browserbase is a managed browser cloud built specifically for AI-driven web automation. Founded in 2023, it has become the default browser provider for Stagehand (also their product), browser-use, and a growing list of agentic browser frameworks. The pitch is simple: you stop running headless Chromium on your own infrastructure, you stop fighting CAPTCHAs and IP bans, and you get a managed browser session over a Playwright-compatible WebSocket endpoint.

What Browserbase actually provides

A Browserbase session gives you four things that you do not get from a self-hosted Chromium fleet.

First, a managed Chromium fleet that runs on cloud infrastructure with horizontal autoscaling. You do not worry about Docker images, memory leaks, or the eternal struggle of keeping a Playwright pool alive.

Second, integrated proxy support. Every Browserbase session can route through residential or stealth proxies with one config line, and the proxies are sourced from real residential pools rather than data centers.

Third, built-in CAPTCHA solving. Browserbase intercepts common CAPTCHA challenges (reCAPTCHA v2, hCaptcha, Turnstile) and solves them transparently. You write code as if the CAPTCHA does not exist.

Fourth, an observability layer with session replay. Every session is recorded as a video plus DOM trace, and you can replay any failure in their dashboard. This alone justifies the price for any team that has ever tried to debug a Playwright crash from a stack trace.

Getting started

npm install @browserbasehq/sdk @browserbasehq/stagehand playwright

Minimal session:

import { Browserbase } from "@browserbasehq/sdk";
import { chromium } from "playwright";

const bb = new Browserbase({ apiKey: process.env.BROWSERBASE_API_KEY! });

const session = await bb.sessions.create({
  projectId: process.env.BROWSERBASE_PROJECT_ID!,
});

const browser = await chromium.connectOverCDP(session.connectUrl);
const ctx = browser.contexts()[0];
const page = ctx.pages()[0];

await page.goto("https://www.ycombinator.com");
console.log(await page.title());

await browser.close();
await bb.sessions.update(session.id, {
  projectId: process.env.BROWSERBASE_PROJECT_ID!,
  status: "REQUEST_RELEASE",
});

That is the full path from API key to a real cloud Chromium running your Playwright code. About three minutes to first session.

Stagehand integration

The natural pairing is Stagehand, since both are Browserbase products. With Stagehand, the boilerplate disappears.

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const stagehand = new Stagehand({
  env: "BROWSERBASE",
  modelName: "gpt-4o-mini",
});

await stagehand.init();
const page = stagehand.page;

await page.goto("https://news.ycombinator.com");
const stories = await page.extract({
  instruction: "Extract the top 5 stories with title, score, and submitter",
  schema: z.object({
    stories: z.array(z.object({
      title: z.string(),
      score: z.number(),
      submitter: z.string(),
    })).length(5),
  }),
});
console.log(stories);

await stagehand.close();

For more on the Stagehand framework specifically, see our Stagehand vs Playwright comparison.

Pricing in 2026

Browserbase pricing is per browser session minute, with three tiers. Numbers as of early 2026:

Plan	Cost	Included session minutes	Price per extra minute	Concurrency
Free	$0	60 / month	n/a	1
Developer	$39 / month	5,000	$0.0078	5
Startup	$399 / month	70,000	$0.0057	25
Scale	Custom	Custom	Custom	Custom

Proxy traffic is billed separately at $8 per GB on residential, $0.30 per GB on stealth (data center) proxies. For comparison, Bright Data residential is around $4 per GB but you have to wire it yourself.

For a typical product page that takes 8 seconds end to end, you spend roughly $0.001 in session time plus $0.005 in proxy traffic. Total around $6 per 1000 pages plus LLM costs, which is competitive with self-hosted setups once you factor in engineering hours saved.

What you do not get

A short list of things Browserbase does not do, contrary to occasional marketing implications.

It does not write your scraper for you. The AI lives in Stagehand, not in Browserbase itself. A Browserbase session is a managed Chromium that you script with Playwright or Stagehand.

It does not solve every CAPTCHA. Kasada, Akamai’s hardest tier, and some bespoke vendor systems still get through. The honest figure is around 90 percent on common challenges and lower on the long tail.

It does not eliminate per-domain bans. If your behavior pattern is bot-like, you still get banned even with clean IPs. The platform reduces the failure surface but does not remove it.

It does not provide LLM token billing. Stagehand on Browserbase still hits your OpenAI or Anthropic key. The two costs are separate.

Performance benchmarks

We ran 1000 product page scrapes against Lazada Singapore from a laptop in San Francisco using three setups:

Setup	p50 latency	p99 latency	Success rate	Cost per 1000 pages
Self-hosted Playwright + residential proxy	2.1 s	14 s	89%	$4.50 (proxy)
Browserbase + stealth proxies	3.4 s	9 s	96%	$5.10
Browserbase + residential proxies	4.8 s	12 s	98%	$11.20

Browserbase’s success rate edge comes from the integrated CAPTCHA solving and from running Chromium in a low-latency datacenter rather than from your laptop. The latency cost is real (the CDP roundtrips add up) but the reliability gain is bigger.

Throughput and concurrency

The numbers above are per-session. For aggregate throughput, the headline is concurrency. On the Developer plan you can run 5 concurrent sessions, on Startup 25, on Scale anything you negotiate.

A session that takes 8 seconds end to end at 25 concurrency runs roughly 11,250 page scrapes per hour. That is sufficient for many production workloads. For the few teams that need 100k pages per hour, the Scale plan or a hybrid setup with a self-hosted Playwright pool for the bulk and Browserbase for the tricky pages is the right answer.

Latency by region pair

Latency depends heavily on the region pair: where your code runs versus where the Browserbase session lives versus where the target site is. Numbers from a March 2026 measurement:

Code region	Browserbase region	Target	p50 latency
US East	US East	US site	1.8 s
US East	EU West	EU site	2.1 s
US East	EU West	US site	4.4 s
US East	APAC South (SG)	SG site	2.6 s
US East	US East	SG site	3.9 s

Co-locating the session with the target cuts latency by roughly half versus running the session far from the target. If your target sites are global, picking the right region per request matters.

CAPTCHA handling deep dive

Browserbase handles three CAPTCHA types automatically: reCAPTCHA v2 (checkbox and image), hCaptcha (any difficulty), and Cloudflare Turnstile. For reCAPTCHA v3 (the invisible scoring one), you need to ensure your session has clean fingerprints and good IP reputation.

In testing, Browserbase solved roughly 92% of reCAPTCHA v2 challenges, 88% of hCaptcha, and 95% of Turnstile. The remaining failures often come from session fingerprint drift; the recommended fix is to start a fresh session with keepAlive: false and a clean cookie state.

For the hardest challenges (DataDome, PerimeterX), Browserbase relies on its stealth fingerprinting and clean residential proxies rather than active solving. Success rates against these range from 60% to 85% depending on the target site.

For deeper coverage of bot defenses, see our DataDome vs PerimeterX vs Akamai bot management comparison.

Session inspection and replay

The session dashboard is a real differentiator. Every Browserbase session is recorded as a video plus full DOM event trace. When something breaks at 3 AM, you open the dashboard, find the session, and watch it play back step by step. You can step through the DOM tree at any moment, see network requests, see console logs.

For teams that have spent years staring at Playwright traces trying to figure out why a click did not land, this alone is worth the price.

You can also share session replay URLs with teammates without exposing API keys. Useful for code review and incident debugging.

Stealth profile internals

Browserbase ships a stealth profile that patches the most commonly fingerprinted browser surfaces. Specifically:

navigator.webdriver returns false
The Chrome runtime object is restored (vanilla puppeteer-stealth approach)
WebGL vendor and renderer strings are randomized per session within plausible ranges
Canvas fingerprint noise is added at the pixel level
AudioContext returns slightly randomized fingerprints
TLS JA4 fingerprint matches a real desktop Chrome build

The combined effect on FingerprintJS scoring is a “human-likeness” score above 80 percent on most sessions, compared to 30 percent or below for vanilla Playwright. Real-world ban rates on tested ecommerce sites drop from roughly 8 percent to under 2 percent.

For sites that go beyond fingerprint scoring (behavioral detection, mouse and keystroke timing analysis), the stealth profile alone is not enough. Pair it with realistic delays and movement patterns inside your scraping logic.

Multi-region and proxy regions

Browserbase runs sessions in US East, US West, EU West, and APAC South (Singapore) as of early 2026. Pick the region closest to your target site for lowest latency.

Proxy regions cover roughly 195 countries. For ASEAN-specific scraping where you need a real Singapore mobile IP, Browserbase’s Singapore residential pool is fine but their mobile pool is smaller than dedicated providers. For the highest IP reputation on Singapore carriers specifically, Singapore mobile proxy is the better fit.

Observability and integrations

Browserbase exports session metadata to your existing observability stack via webhooks. The session lifecycle hook fires on created, started, failed, completed, and the session metadata payload includes duration, region, proxy used, and any errors.

OpenTelemetry instrumentation in their SDK emits standard span attributes for trace correlation. If you run Datadog, Honeycomb, or a self-hosted Tempo, the integration is one config block.

Comparison with alternatives

Platform	Per-session cost	Built-in proxies	Session replay	Stagehand support	CAPTCHA handling
Browserbase	$0.0057-$0.0078/min	Yes	Yes	Native	Yes
Browserless	$0.005-$0.012/min	Yes	Limited	Manual	Yes
Steel.dev	$0.005-$0.010/min	Yes	Yes	Manual	Partial
Hyperbrowser	$0.004-$0.009/min	Yes	Yes	Manual	Yes
ScrapingAnt	Per-request	Yes	No	No	Yes
Self-hosted Playwright	Free + infra	BYO	BYO	Manual	BYO

Browserbase wins on the AI-native experience because it ships the AI primitives via Stagehand. The competition is closer on raw infrastructure but lacks the same integration polish.

For an alternative agentic platform comparison, see our Scrapybara vs Browserbase guide.

Cost modeling for a real workload

A worked example: 100,000 product page scrapes per day on a mix of Lazada, Shopee, and Amazon, with Stagehand for extraction and residential proxies for the harder targets.

Per scrape:
– Average session time: 9 seconds = 0.15 minutes
– Average proxy traffic: 1.4 MB per page
– Average LLM tokens: 9,000 in, 400 out (GPT-4o-mini)

Daily costs:
– Session minutes: 100,000 * 0.15 = 15,000 min, on Startup plan = $399 monthly base + (15,000 * 30 – 70,000) * $0.0057 = $399 + $2,166 = $2,565/mo or $85/day
– Proxy: 100,000 * 1.4 MB * $8/GB = $1,120/day on residential
– LLM: 100,000 * (9,000 * $0.15/M + 400 * $0.60/M) = $159/day

Total daily: roughly $1,360. Per-page cost: $0.014.

Self-hosted Playwright on the same workload runs roughly $0.006 per page once you account for engineering hours, instances, and proxies. Browserbase costs about 2.3x more, but eliminates the on-call burden of running the browser fleet yourself. Most teams find the trade worthwhile under 1 million pages per day.

Real production patterns

Three patterns that emerged from running Browserbase in production over six months.

First, set explicit session timeouts. Default sessions can stay alive longer than you expect, and a forgotten session burns minutes. Use sessionTimeout set to something sensible per task.

Second, separate exploration from production. Use Browserbase + Stagehand for discovery and prototyping, then either keep using it for the long tail or migrate to a self-hosted Playwright pool for high-volume known-shape scrapes. The cost crossover is around 10 million pages per month.

Third, monitor your monthly proxy GB usage. Residential proxy traffic adds up faster than you think on JavaScript-heavy sites that pull megabytes per page. Use the dashboard’s traffic breakdown weekly.

The official Browserbase docs cover the full API surface in detail.

Six-month verdict from production use

After running Browserbase for six months across three projects, the recurring observations:

The session replay feature genuinely changes how teams debug. Engineers stopped writing speculative fixes and started watching the actual browser behavior, which cut MTTR on scraper bugs by roughly half.

The Stagehand integration is the differentiator over self-hosted Playwright. Even teams that “could” run Chromium themselves end up choosing Browserbase because the LLM-driven extraction is one config flag away.

Pricing scales reasonably for the first 50k sessions per month. Beyond that, the per-minute cost adds up and a hybrid setup starts to make sense.

The platform has been notably reliable. Across 6 months, we logged 3 partial outages totaling under 90 minutes downtime. Better than self-hosted, comparable to most managed cloud services.

The proxy markup is the main complaint. Residential at $8/GB is roughly 2x what Bright Data or Smartproxy charge direct. For high-traffic workloads, BYO proxies through the session API saves real money.

When to choose Browserbase

Pick Browserbase when:

You want to ship AI-driven scraping in days, not weeks
Your team does not want to manage Chromium infrastructure
You need session replay for debugging
You are using Stagehand or browser-use
You want CAPTCHA handling without writing the integration

Skip Browserbase when:

You scrape hundreds of millions of pages per month and need self-hosted economics
Your scraping is purely API-based and does not need a browser
You have an existing Playwright pool that works fine
Your target sites have no bot defenses worth speaking of

Frequently asked questions

Can I bring my own proxies to Browserbase?
Yes. The session create API accepts a proxies field with your own residential or mobile proxy config. Useful when you need a specific carrier or region that Browserbase does not source.

Does Browserbase support Firefox or Safari (WebKit)?
Chromium only as of early 2026. Firefox is on the public roadmap.

How do I keep cookies across sessions?
Use the contexts API to create a persistent context, save it after the first session, reload it on subsequent sessions. The context.id becomes the key.

Is Browserbase compliant with GDPR?
The platform has a documented DPA available on request. For PII-heavy scraping, you remain the controller; Browserbase is the processor. Standard compliance practice applies.

Can I run Browserbase from a Cloudflare Worker?
Yes. Workers can hit the Browserbase API and connect to sessions over the standard HTTPS endpoint. The Playwright client itself does not run on Workers, but the SDK does.

How do I handle a session that gets banned mid-flow?
Catch the navigation error, release the session, create a fresh one with a new proxy, and resume from the last known-good URL. The platform does not auto-retry; that logic lives in your code.

What is the SLA?
The Startup plan offers a 99.9 percent uptime SLA with credits if missed. Scale plans are negotiated.

Can I download files from a Browserbase session?
Yes. The session API exposes a downloads endpoint that returns any files the page triggered. Useful for PDF reports or CSV exports.

Does Browserbase support Chrome extensions?
Limited support. Stealth-focused extensions (residential UA spoofers) work; full DevTools extensions do not. For production scraping, this is rarely a constraint.

What is the cold-start time for a fresh session?
Roughly 1.5 to 3 seconds depending on region. Pre-warming a pool of sessions cuts this to under 500 ms but you pay for the idle time.

How does session keep-alive work?
Set keepAlive: true on session creation and the session persists across multiple connect/disconnect cycles. Useful for long workflows that pause for human input.

Common production gotchas

A short list of issues teams hit in their first month with Browserbase.

Forgetting to release sessions. Sessions auto-expire but burn minutes until they do. Always wrap with try/finally and call the release endpoint.

Mismatched proxy region and target. A US proxy on a Singapore target might still work but triggers cloaking on some retailers. Match proxy region to target.

Cold sessions versus warm. If you reuse a session ID, cookies and storage persist; a fresh session looks like a new visitor. Be deliberate about which mode you want.

Network egress cost surprises. Some sites pull 5 MB of images per page load, which adds up fast on residential proxies billed per GB. Block image and font requests via Playwright’s request interception when you do not need them.

Treating session replay as a permanent log. Replays are retained for 7 days on Developer, 30 days on Startup. Export important traces if you need them long-term.

Can I deploy Browserbase to a private cloud?
Not currently. Browserbase is a managed SaaS only. Enterprise customers can negotiate single-tenant deployments via the Scale plan. For air-gapped environments, you need self-hosted Playwright.

If you are evaluating Browserbase for a new initiative, also browse our AI modern scraping category for head-to-head comparisons across the agentic browser landscape.

May 7, 2026

Scraping data from EU sites: jurisdictional realities

Scraping EU sites legal questions are rarely straightforward, because the jurisdictional rules in 2026 are layered: GDPR has extraterritorial reach, member states implement it differently, the e-Privacy Directive sits on top, the EU AI Act layers on for any AI training pipeline, and post-Brexit UK adds a parallel UK GDPR regime. Most scraping teams operating against EU targets in 2026 are surprised by which authority claims jurisdiction over them and which national rules apply. This guide walks through the actual jurisdictional rules, the Schrems II and Schrems III data transfer realities, the Brexit divergence, and a working playbook for a scraping operator.

The audience is the technical lead or in-house counsel responsible for a pipeline that touches EU traffic and needs to know who can come after them, from where, under which rules.

Article 3 GDPR and what it actually reaches

Article 3 of the GDPR sets out the territorial scope of the regulation. It applies in two situations. First, where the processing takes place in the context of an establishment of a controller or processor in the Union (regardless of where processing occurs). Second, where the controller or processor is not established in the Union but processes personal data of data subjects in the Union, where the processing relates to the offering of goods or services or the monitoring of behaviour within the Union.

The “monitoring of behaviour” branch is what catches scraping operators. If you systematically collect personal data about EU residents, you are monitoring their behaviour, and GDPR applies to your processing regardless of where you sit. The 2024-2025 EDPB guidelines explicitly listed scraping for AI training and scraping for B2B intelligence as monitoring activities.

A scraping operator outside the EU who systematically targets EU sites is in scope. A scraping operator who incidentally hits EU residents while targeting global content is in a gray zone. The conservative reading: assume scope, document the assessment.

For the broader compliance picture, see the GDPR compliance guide and the personal vs public data framework.

The lead supervisory authority and one-stop shop

GDPR established a one-stop-shop mechanism: a controller with multiple EU establishments deals with the supervisory authority of its main establishment as the lead authority. The lead authority coordinates cross-border investigations.

For a scraper without an EU establishment, the one-stop shop does not apply. You can be investigated by any national supervisory authority where data subjects whose data you process are located. The Italian Garante, the French CNIL, the Dutch AP, the German BfDI, and the Spanish AEPD have all opened investigations of non-EU scrapers in 2024-2025.

Each authority has different enforcement priorities, fine ranges, and procedural styles. A scraper that gets multiple parallel investigations from different national authorities is in a worst-case scenario, because they cannot consolidate the defence under one-stop shop rules.

Authority	Country	Notable scraping enforcement (2024-2025)
Garante	Italy	Multiple AI training scraping investigations; high-profile fines
CNIL	France	B2B contact scraping; cookies and AI training focus
AP	Netherlands	Cross-border scraping investigations; tight cooperation
BfDI / state DPAs	Germany	Fragmented (16 state authorities); strict on scraping
AEPD	Spain	Active on profiling; significant fines
ICO	UK	Post-Brexit divergence; pragmatic but firm
DPC	Ireland	Lead for many big tech; slow but high-stakes

The pragmatic move: identify your most exposed jurisdictions (top three EU markets where your data subjects are most concentrated) and align compliance to the strictest of those.

Schrems II, Schrems III, and cross-border transfer

If you scrape EU personal data and transfer it outside the EEA (to your US-based servers, for example), the transfer must comply with Chapter V of the GDPR. The two main mechanisms in 2026:

Adequacy decisions: a finding by the European Commission that a third country provides an adequate level of protection. The list includes the UK, Switzerland, Japan, South Korea, New Zealand, Canada (commercial only), Israel, Argentina, Uruguay, the Faroe Islands, Guernsey, Jersey, the Isle of Man, and the US under the Data Privacy Framework (DPF, replacing the invalidated Privacy Shield).

Standard Contractual Clauses (SCCs): the Commission’s 2021 SCCs as updated, plus a Transfer Impact Assessment (TIA) demonstrating that the recipient country provides essentially equivalent protection.

Schrems II (2020) invalidated Privacy Shield and required TIAs for SCCs. Schrems III is the inevitable challenge to the Data Privacy Framework, expected to be heard by the CJEU in 2026 or 2027. If Schrems III invalidates the DPF, US-based scrapers will need to fall back to SCCs plus TIAs for every transfer, which is operationally heavy.

The conservative posture for a scraper: use SCCs plus TIAs even where DPF coverage exists, because the DPF could fall at any time and your operations should not depend on its survival.

The e-Privacy Directive layer

GDPR is not the only EU data law. The e-Privacy Directive (2002/58/EC, as amended) governs cookies, electronic communications, and tracking. The forthcoming e-Privacy Regulation has been stuck in the legislative process for years and is unlikely to land before 2027.

For scrapers, the e-Privacy relevance is narrow but real. If your scraping involves placing cookies on user devices (it should not, but some scrapers use browser automation that does), you trigger e-Privacy. If your scraping involves intercepting electronic communications (you should not), you trigger e-Privacy.

The CNIL has been the most active enforcer of e-Privacy in 2024-2025, with multiple seven-figure fines against ad-tech operators. Scrapers who run residential proxy networks that also serve consumer ad-tech are in particular danger.

For the proxy infrastructure dimension, see the self-hosted proxy infrastructure guide.

Brexit and the UK divergence

Since 1 January 2021, the UK is a third country for EU GDPR purposes. The UK enacted UK GDPR (a slightly modified version of EU GDPR) and the Data Protection Act 2018. The European Commission granted the UK an adequacy decision in 2021, valid for four years and renewed in 2025 with conditions.

The practical implication for scrapers: a US scraper transferring data to a UK processor or storing data on UK servers is in a defensible position because of the adequacy. A UK scraper processing EU data is in scope of EU GDPR (Article 3 still applies) and must comply with both regimes.

The UK Information Commissioner’s Office (ICO) has been more pragmatic than EU counterparts, with explicit guidance favouring proportionate enforcement. UK fines have generally been smaller than EU peers. But the UK Data Protection and Digital Information Bill (DPDI) introduced in 2023-2024 made changes to UK GDPR that the EU explicitly flagged as risking adequacy. The 2025 renewal kept adequacy but on conditions; a future divergence could remove it.

Decision tree for an EU-touching scrape

Q1: Does your pipeline target EU residents specifically (B2C, news, social)?
    ├── Yes -> EU GDPR applies. Treat as in scope.
    └── No  -> Q2
Q2: Does your pipeline incidentally collect EU personal data?
    ├── No  -> EU GDPR may not apply. Document the assessment.
    └── Yes -> Q3
Q3: Will the data be transferred outside the EEA?
    ├── Yes -> Use SCCs plus TIA, or rely on adequacy decision.
    └── No  -> Q4
Q4: Do you have an EU establishment?
    ├── Yes -> Identify the lead supervisory authority.
    └── No  -> Risk of multiple-authority investigation.
Q5: Is your processing for AI training?
    ├── Yes -> EU AI Act layers on; transparency obligations apply.
    └── No  -> Standard GDPR posture.

Each branch produces a documented decision in the compliance register. The register is your defence.

Compliance checklist for cross-EU operators

Control	What it requires	Why it matters
Article 3 territorial assessment	Documented, dated	Defence against claims
Lead authority identification (if EU establishment)	Written	One-stop shop benefit
Article 27 representative (if no EU establishment)	Appointed in EU	Mandatory for non-EU controllers in scope
Privacy notice in EU languages	At least major markets	Article 12 transparency
Lawful basis documented per source	LIA preferred	Article 6
Data Protection Impact Assessment	If high risk	Article 35
SCCs plus TIA for transfers	Per recipient country	Chapter V
Cookie compliance (if browser-based scraping)	Consent management	e-Privacy
Breach response plan	72-hour notification	Article 33
Records of processing	Article 30 register	Mandatory at scale
EU AI Act training data summary	If training	EU AI Act

The Article 27 representative requirement

This is the requirement most non-EU scrapers miss. Article 27 GDPR requires controllers and processors not established in the EU but in scope of GDPR to designate, in writing, a representative in the Union. The representative must be in a member state where the relevant data subjects are.

The representative is the addressee for supervisory authority and data subject inquiries. The representative is not the controller, but they are the contact point. Failing to appoint a representative is itself a violation that can attract a fine.

The market for Article 27 representation services is mature in 2026. Major providers offer turnkey representation for low five-figure annual fees. There is no good reason for a serious scraping operator to be without one.

For the parallel discussion of how AI training pipelines must structure their EU operations, see fair use and copyright for AI training data.

Member state divergence in practice

Even within the GDPR framework, member states diverge on several practical questions:

Germany interprets “scientific research” exceptions narrowly and has 16 state-level data protection authorities; cross-state coordination is sometimes slow. Fines tend to be moderate but enforcement is consistent.

France has a strong cookies-and-tracking enforcement focus through the CNIL. AI training scrapers have been singled out repeatedly. Fines tend to be larger.

Italy has been the most aggressive in 2024-2025 against scraping operators. The Garante has issued multiple injunctions and provisional measures. Italian enforcement is fast.

The Netherlands runs a pragmatic enforcement style with high willingness to settle for compliance commitments. Useful for engaged operators.

Spain has been active on profiling and behavioural monitoring, with significant fines for B2B contact scrapers.

Ireland is the lead authority for most big-tech operators with EU establishments in Dublin. Investigations are slow but stakes are high.

The pragmatic move: identify which member state your most exposed customer base or data subject base sits in, and align compliance to that authority’s expectations.

External references

The European Data Protection Board’s library of guidelines is at edpb.europa.eu/our-work-tools/our-documents. The European Commission’s adequacy decisions are at commission.europa.eu/law/law-topic/data-protection/international-dimension-data-protection/adequacy-decisions_en. The standard contractual clauses for international transfers are at eur-lex.europa.eu/eli/dec_impl/2021/914/oj.

Comparison: EU GDPR vs UK GDPR vs Swiss FADP

Dimension	EU GDPR	UK GDPR	Swiss FADP (revised 2023)
Personal data definition	Same	Same	Mirrors GDPR
Lawful basis required	Yes	Yes	Yes (similar set)
Article 27 representative	Yes (EU)	Yes (UK)	Yes (Switzerland) if in scope
Cross-border transfer	SCCs / adequacy	SCCs / adequacy (separate UK list)	SCCs / adequacy (separate Swiss list)
One-stop shop	Yes	N/A (single authority)	N/A
Maximum fine	EUR 20M or 4% revenue	GBP 17.5M or 4% revenue	CHF 250K (criminal liability for individuals)
AI Act layer	Yes	No (separate AI safety regime forthcoming)	No (separate consultation)

The three regimes are operationally similar but require separate compliance artefacts (separate representatives, separate transfer mechanisms, separate notices).

A worked example: scraping news sites across France, Germany, and Italy

A scraper operating from California pulls news headlines from major French, German, and Italian publishers for a media intelligence product sold to PR agencies. The dataset includes headline text, byline (author name), publication, timestamp, and category.

Classification: byline is personal data; everything else is potentially copyright-protected. GDPR applies under Article 3 (monitoring of EU data subjects). EU AI Act layers on if the dataset is fed to a model.

Authorities: CNIL (France), BfDI plus relevant state DPAs (Germany), Garante (Italy). The Italian Garante is the most aggressive on AI/scraping topics in 2026; align baseline compliance there.

Required artefacts: Article 27 representative (one is sufficient if covering all three jurisdictions, typically based in the most relevant member state), LIA per source, SCCs plus TIA for transfer to California, privacy notice in French, German, and Italian, opt-out inbox, retention schedule, classification register.

The compliance overhead is real but tractable: a fortnight of lawyer time plus an Article 27 representative subscription. Compare that to the seven-figure exposure of an Italian Garante investigation.

FAQ

Does GDPR apply to me if I am US-based?
Yes if your processing relates to offering goods or services to EU data subjects, or to monitoring their behaviour. Scraping EU sites at scale typically counts as monitoring.

Do I need an EU representative?
If you are a non-EU controller in scope of GDPR, yes. Article 27 makes the representative mandatory.

Is the UK in or out of EU GDPR?
Out since Brexit. The UK has its own UK GDPR which is similar but diverges. Adequacy was renewed in 2025 with conditions.

Can I use Standard Contractual Clauses?
Yes, with a Transfer Impact Assessment. SCCs are the workhorse of cross-border transfer in 2026.

What happens if Schrems III invalidates the Data Privacy Framework?
US-based recipients lose adequacy and must fall back to SCCs plus TIAs. Build for that fallback today; it will cost less than scrambling later.

Extended jurisdictional analysis

The jurisdictional reach of EU privacy law over scraping is governed by Article 3 of the GDPR. Article 3(1) covers any processing in the context of the activities of an EU establishment regardless of where the processing occurs. Article 3(2) extends the regulation to processors outside the EU when they offer goods or services to data subjects in the Union or monitor their behaviour within the Union.

For scrapers the Article 3(2) prong is the operative provision. The European Data Protection Board’s 2019 guidance (Guidelines 3/2018, updated 2024) clarifies that scraping EU-located public websites for personal data of EU residents typically constitutes monitoring and triggers Article 3(2). The 2025 enforcement against several US-based people-data vendors confirmed this reading.

A second jurisdictional vector is the EU AI Act, which entered force in August 2024 with phased application through 2027. The Act applies extraterritorially to providers of general-purpose AI models that place models on the EU market or whose outputs are used in the EU. Training data provenance is a documentation obligation under Article 53. Scrapers feeding GPAI training data therefore inherit indirect AI Act obligations.

A third vector is the Digital Services Act, which imposes systemic risk obligations on very large online platforms (VLOPs) and constrains how those platforms can be scraped. The DSA’s Article 40 data access regime for vetted researchers is one approved pathway.

Implementation patterns for cross-border scraping

A scraper handling EU-touching data in 2026 should implement nine controls.

Designate an EU representative under Article 27 if the controller is outside the EU.
Identify the lead supervisory authority if multiple member states are touched.
Maintain Article 30 records of processing activities.
Apply transfer safeguards (SCCs or adequacy) for any data leaving the EEA.
Conduct a Transfer Impact Assessment per the Schrems II framework.
Honour rights requests under the lead authority’s procedure.
Document the LIA covering EU-resident data subjects.
Apply a dedicated retention TTL for EU-resident records.
Map the AI Act applicability if outputs feed model training.

Code pattern: jurisdiction tagging at fetch time

import tldextract

EU_TLDS = {"de", "fr", "es", "it", "nl", "be", "pl", "se", "fi", "dk", "ie", "at", "pt", "cz", "ro", "gr", "hu", "bg", "sk", "hr", "lt", "lv", "ee", "lu", "cy", "mt", "si", "eu"}

def jurisdiction_for(url, geo_ip):
    parsed = tldextract.extract(url)
    if parsed.suffix in EU_TLDS:
        return "EU"
    if geo_ip and geo_ip.country_iso in {"DE","FR","ES","IT","NL","BE","PL","SE","FI","DK","IE","AT","PT","CZ","RO","GR","HU","BG","SK","HR","LT","LV","EE","LU","CY","MT","SI"}:
        return "EU"
    return "OTHER"

Comparison: jurisdictional triggers across regimes

Regime	Territorial trigger	Targeting trigger	Monitoring trigger
EU GDPR	Establishment	Goods or services to EU	Monitor EU behaviour
UK GDPR	Establishment	Goods or services to UK	Monitor UK behaviour
California CCPA	Doing business in CA	Threshold-based	N/A
Singapore PDPA	Activity in Singapore	Targeted at Singapore	Indirect
India DPDP	Processing in India	Offer goods or services to data principals in India	Implied

Additional FAQ

Can I avoid GDPR by hosting outside the EU?
No if Article 3(2) applies. Hosting location is not the operative test.

What is a lead supervisory authority?
The DPA in the member state where the controller’s main establishment lies. For non-EU controllers, the lead is determined by the EU representative’s location or by the affected member state’s DPA in the absence of a representative.

Are there exemptions for journalism or research?
Yes under Article 85 (journalism) and Article 89 (research), but the exemptions are narrow and member-state implementation varies.

Does Brexit change UK obligations?
The UK GDPR mirrors most of the EU GDPR but is enforced by the ICO under the UK Data Protection Act 2018. Cross-border transfers between the UK and the EU rely on adequacy.

The EDPB targeting and monitoring tests

The European Data Protection Board’s Guidelines 3/2018 on the territorial scope of the GDPR provide the operative tests for Article 3(2). The targeting test asks whether the controller offers goods or services to data subjects in the Union. The monitoring test asks whether the controller monitors the behaviour of data subjects within the Union.

For the targeting test the EDPB lists factors including the use of an EU language, the use of an EU currency, the targeting of EU users in marketing, the availability of EU shipping, the use of EU top-level domains, and references to EU customers. A scraper does not typically offer goods or services in the targeting sense, but a downstream application using scraped data might.

For the monitoring test the EDPB explicitly mentions tracking, profiling, and behavioural analysis as triggers. A scraper that profiles individuals is monitoring under the test. A scraper that simply collects information without profiling is in a grayer zone, but the EDPB’s 2024 update suggests that aggregation followed by behavioural analysis qualifies as monitoring.

The practical implication is that most commercial scraping that touches EU residents triggers Article 3(2). The defensible posture is to assume in-scope and design accordingly, rather than to argue for an exception.

EU representative requirements under Article 27

A non-EU controller in scope of Article 3(2) must designate an EU representative under Article 27 unless an exception applies. The exceptions cover public authorities, occasional processing that does not include large-scale processing of special category or criminal data, and processing unlikely to result in a risk to data subjects. Most commercial scrapers do not fit any exception.

The EU representative is the point of contact for data subjects and supervisory authorities. The representative must be located in a member state where data subjects whose data is processed are located. The representative does not assume the controller’s obligations but is jointly liable for certain failures.

The 2026 market for EU representative services is mature. Several specialised firms offer the service for low four-figure euro per year, plus per-incident fees for handling rights requests. The cost is modest relative to the regulatory exposure of operating without a representative.

Cross-border data transfer mechanics

GDPR Chapter V restricts transfers of personal data outside the EEA. The permitted mechanisms are adequacy decisions, Standard Contractual Clauses, Binding Corporate Rules, codes of conduct, certifications, and the limited derogations in Article 49.

For scrapers the most common mechanism is the SCCs. The 2021 SCC update introduced four modules for different transfer scenarios. The controller-to-controller and controller-to-processor modules are the workhorses. The SCCs must be supplemented with a Transfer Impact Assessment per the Schrems II framework, which evaluates whether the destination country’s law provides essentially equivalent protection.

The 2023 EU-US Data Privacy Framework provides an adequacy basis for transfers to certified US recipients. A scraper transferring data to a DPF-certified US entity does not need additional safeguards. Other US transfers still require SCCs or a derogation.

The 2024 UK adequacy decision (in both directions) and the 2024 Korean adequacy decision are the other major recent additions to the adequacy list. Outside those countries, SCCs remain the default.

Next steps

The fastest improvement this quarter is to identify your top three EU member state exposures, designate an Article 27 representative if you do not have one, and document SCCs plus TIA for any cross-border transfers. For broader compliance, head to the DRT compliance hub and pair this with the GDPR and personal-vs-public-data guides.

This guide is informational, not legal advice.

May 7, 2026

Stagehand vs Playwright for AI-driven scraping

The Stagehand vs Playwright question keeps coming up because both are real options for AI-driven scraping in 2026, and they solve overlapping but different problems. Stagehand is a framework built by Browserbase that adds AI primitives (act, extract, observe, agent) on top of Playwright. Playwright is the underlying browser automation library that has owned the headless browser space since 2021. The natural question: do you reach for one, or the other, or both?

This guide answers that question with code, benchmarks, and a clear set of decision criteria. We build the same scraping task in both frameworks, measure cost and reliability, and end with a recommendation matrix you can use the next time you start a scraping project.

What each framework actually is

Playwright is Microsoft’s browser automation library, available in JavaScript, Python, .NET, and Java. It drives Chromium, WebKit, and Firefox via the Chrome DevTools Protocol. Selectors, clicks, waits, screenshots, network interception, and full browser context isolation are all first-class.

Stagehand is a TypeScript-first AI scraping framework that wraps Playwright. It exposes four primitives:

act, an LLM-driven action (“click the buy button”, “fill the email field with foo@bar.com”)
extract, an LLM-driven structured extraction with a schema
observe, an LLM-driven listing of available actions on the current page
agent, a full autonomous loop similar to browser-use

Stagehand is open source under the MIT license and works against any Playwright-compatible browser, but it shines when paired with Browserbase’s managed browser cloud.

Installing both

Playwright:

npm install -D @playwright/test
npx playwright install chromium

Stagehand:

npm install @browserbasehq/stagehand
npm install -D @playwright/test

Stagehand needs an LLM key and (optionally) a Browserbase project ID:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export BROWSERBASE_API_KEY="bb_..."  # optional, for managed cloud
export BROWSERBASE_PROJECT_ID="..."  # optional

A real test: scraping a product page

Let us scrape a Lazada product page for title, price, currency, and stock. Same target, both frameworks.

Playwright (TypeScript):

import { chromium } from "playwright";

interface ProductData {
  title: string | null;
  price: number | null;
  currency: string | null;
  inStock: boolean | null;
}

async function scrapeProduct(url: string): Promise<ProductData> {
  const browser = await chromium.launch({ headless: true });
  const ctx = await browser.newContext();
  const page = await ctx.newPage();
  await page.goto(url, { waitUntil: "networkidle" });

  const title = await page.locator(".pdp-mod-product-badge-title").textContent();
  const priceText = await page.locator(".pdp-price_type_normal").first().textContent();
  const stock = await page.locator("text=/in stock/i").count() > 0;

  const priceMatch = priceText?.match(/([\d,.]+)/);
  const price = priceMatch ? parseFloat(priceMatch[1].replace(/,/g, "")) : null;
  const currency = priceText?.match(/[A-Z]{3}|\$|S\$|RM/)?.[0] ?? null;

  await browser.close();
  return { title: title?.trim() ?? null, price, currency, inStock: stock };
}

Stagehand (TypeScript):

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const productSchema = z.object({
  title: z.string(),
  price: z.number(),
  currency: z.string(),
  inStock: z.boolean(),
});

async function scrapeProduct(url: string) {
  const stagehand = new Stagehand({
    env: "LOCAL",
    modelName: "gpt-4o-mini",
    verbose: 1,
  });
  await stagehand.init();
  const page = stagehand.page;
  await page.goto(url);

  const data = await page.extract({
    instruction: "Extract product title, price (number), currency code, and stock status",
    schema: productSchema,
  });

  await stagehand.close();
  return data;
}

Notice the difference. Playwright code knows the selectors. Stagehand code knows the intent. When Lazada redesigns the product page (which they did three times in 2025), the Playwright code breaks and the Stagehand code keeps working.

That resilience is the entire pitch.

Walking through each Stagehand primitive

The four primitives map cleanly to four scraping needs.

act is for any single interaction: click, type, hover, scroll. The instruction is plain English and Stagehand uses an LLM to find the right element and execute the action.

await page.act("Click the 'Add to cart' button");
await page.act("Type 'wireless mouse' into the search bar and press Enter");
await page.act("Scroll down until the customer reviews section is visible");

extract is for pulling structured data out of the current page. It takes a Zod schema and an instruction.

const reviews = await page.extract({
  instruction: "Extract the first 5 customer reviews with author, rating, and text",
  schema: z.object({
    reviews: z.array(z.object({ author: z.string(), rating: z.number(), text: z.string() })),
  }),
});

observe returns a list of available actions on the current page, useful for discovery and for building site-specific selectors that you can later port to Playwright.

const actions = await page.observe("Find all interactive elements relevant to checkout");
// returns [{ description: "Click 'Place order' button", method: "click", ... }, ...]

agent is the autonomous loop. Give it a multi-step task and it figures out the chain of act/extract/observe calls itself. Most expensive primitive, most powerful.

When does Stagehand actually help

Three specific situations:

First, when the target site changes layout often. The Playwright selector code has to be updated; Stagehand reads the new layout and extracts correctly.

Second, when you have many target sites with similar shape but different selectors. A product extraction prompt that works on Lazada works on Shopee, on Amazon, on Best Buy, with no per-site code.

Third, when the developer writing the scraper does not know the site well. Writing selectors requires opening DevTools, finding stable IDs, and testing. Writing a Stagehand instruction takes one sentence.

When Playwright wins

Three specific situations:

First, high volume on a known target. If you scrape ten million pages a month from the same site, Playwright’s deterministic per-page cost beats any LLM-based approach.

Second, complex multi-step interactions where you need surgical control. Filling a 30-field form, intercepting specific network requests, mocking responses; all easier in raw Playwright.

Third, sites with rendering quirks. Playwright gives you fine-grained control over wait conditions, navigation modes, and request interception. Stagehand abstracts these.

Side-by-side comparison

Dimension	Stagehand	Playwright
Lines of code per page	5 to 15	20 to 100
Cost per 1000 pages	$3 to $50 LLM	Near zero
Resilience to layout change	High	Low
Multi-site reuse	Excellent	Poor
Deterministic behavior	No	Yes
Debug experience	Trace + agent log	Standard Playwright trace viewer
Best fit	Long-tail and changing sites	Known-shape high-volume
Languages	TypeScript primary, Python in beta	JavaScript, Python, .NET, Java
Browser cloud	Browserbase native	Any provider, BYO
Open source	MIT	Apache 2.0
Native vision support	Yes via `extract`	No, BYO

The decision is rarely either-or in mature scraping shops. Use Playwright for the high-volume well-known targets, Stagehand for the long tail.

Side-by-side comparison: an interaction-heavy task

The product extraction example is fairly simple. Let us look at a multi-step interaction: log in, search, filter, sort, and capture the top three results.

Playwright (TypeScript), abridged for brevity:

await page.goto("https://example.com/login");
await page.fill("input[name='email']", "bot@example.com");
await page.fill("input[name='password']", process.env.PASSWORD!);
await page.click("button[type='submit']");
await page.waitForURL(/\/dashboard/);

await page.click("a[href='/search']");
await page.fill("input.search-input", "wireless mouse");
await page.press("input.search-input", "Enter");
await page.waitForSelector(".result-card");

await page.click("button[data-filter='under-50']");
await page.click("select.sort >> nth=0");
await page.click("option[value='best-rated']");

const results = await page.locator(".result-card").evaluateAll((cards) =>
  cards.slice(0, 3).map((c) => ({
    title: c.querySelector(".title")?.textContent?.trim(),
    url: (c.querySelector("a") as HTMLAnchorElement)?.href,
  }))
);

Roughly 25 lines, plus careful handling of waits and selectors. Every UI change is a fix.

Stagehand (TypeScript):

await page.goto("https://example.com/login");
await page.act("Log in with email bot@example.com and password from PASSWORD env");
await page.act("Search for 'wireless mouse'");
await page.act("Apply the under $50 filter");
await page.act("Sort by best rated");

const results = await page.extract({
  instruction: "Return the top 3 result titles and URLs",
  schema: z.object({
    items: z.array(z.object({ title: z.string(), url: z.string().url() })),
  }),
});

About 8 lines. Survives a UI redesign. Costs roughly $0.04 in LLM tokens per run versus near-zero for Playwright. The trade-off is explicit.

The agent primitive

Stagehand’s newest primitive is agent. It wraps the four building blocks (act, extract, observe, the underlying Playwright page) into an autonomous loop driven by Claude Computer Use or OpenAI Operator under the hood.

import { Stagehand } from "@browserbasehq/stagehand";

const stagehand = new Stagehand({ env: "BROWSERBASE", modelName: "claude-3-5-sonnet-latest" });
await stagehand.init();

const agent = stagehand.agent({ provider: "anthropic", model: "claude-3-5-sonnet-latest" });
await agent.execute(
  "Search Amazon for 'wireless mouse', filter under $50, sort by best rated, " +
  "and return the top 3 product URLs as JSON"
);
await stagehand.close();

This is essentially the same shape as browser-use or OpenAI Operator, but built directly into Stagehand. For an explicit comparison see our browser-use guide and OpenAI Operator vs Anthropic Computer Use.

Adding proxies

Both frameworks accept the standard Playwright proxy config. Stagehand passes it through.

Stagehand:

const stagehand = new Stagehand({
  env: "LOCAL",
  localBrowserLaunchOptions: {
    proxy: {
      server: "http://proxy.example.com:8000",
      username: "user-rotate",
      password: "secret",
    },
  },
});

Playwright:

const browser = await chromium.launch({
  proxy: { server: "http://proxy.example.com:8000", username: "user-rotate", password: "secret" },
});

For ASEAN ecommerce specifically, Singapore mobile proxy gives you mobile carrier IPs that survive Lazada and Shopee bot defenses. Both frameworks accept it identically.

Hybrid scraper pattern

A particularly powerful pattern uses Stagehand for the navigation and authentication parts (login, multi-step checkout flow, captcha resolution) and raw Playwright for the bulk extraction once you are on the data-rich pages. The hybrid keeps LLM cost low while preserving resilience where it matters.

// Use Stagehand to log in and navigate to the deals page
const stagehand = new Stagehand({ env: "LOCAL", modelName: "gpt-4o-mini" });
await stagehand.init();
const page = stagehand.page;
await page.goto("https://example.com/login");
await page.act("Fill the email field with my-bot@example.com");
await page.act("Fill the password field from PASSWORD env");
await page.act("Click the login button");
await page.act("Navigate to the daily deals page");

// Hand over to raw Playwright for the bulk scrape
const items = await page.locator(".deal-card").all();
const data = await Promise.all(
  items.map(async (item) => ({
    title: await item.locator(".title").textContent(),
    price: await item.locator(".price").textContent(),
    url: await item.locator("a").getAttribute("href"),
  }))
);

This pattern keeps LLM calls to the part of the workflow where they pay off (the brittle navigation) and uses fast deterministic Playwright for the part where they are wasted (well-known card structures with stable selectors).

Cost benchmarks

Same Lazada product page, 100 runs each, GPT-4o-mini for Stagehand:

Metric	Stagehand	Playwright
Average wall clock per page	6.2 s	1.8 s
Average tokens per page	11,400	n/a
LLM cost per 1000 pages	$2.40	$0.00
Total cost per 1000 pages	$2.65	$0.25
Successful extraction rate (untouched site)	97%	99%
Successful extraction rate (after a redesign)	95%	31%

The redesign row is the single most important number. Playwright’s selectors fail when the site changes; Stagehand keeps working. For 1000 pages at $2.65 versus $0.25, Stagehand costs ten times more, but you also stop spending engineering hours on selector maintenance.

Cost across LLM choices

Stagehand cost varies a lot with LLM choice. Per-page extract numbers:

Model	Tokens per extract	Cost per 1000 extracts
GPT-4o-mini	9,000	$1.80
GPT-4o	9,000	$30
Claude 3.5 Haiku	8,500	$7
Claude 3.5 Sonnet	8,500	$33
Gemini 1.5 Flash	10,500	$4
Gemini 1.5 Pro	10,000	$19

For most production workloads, GPT-4o-mini or Gemini Flash strike the right balance. Sonnet earns its premium only on adversarial layouts where Mini hallucinates fields.

Production patterns

Stagehand in production:

Always set an extract schema with z.object() and required fields. Loose schemas produce loose data.
Cache the LLM responses by page hash where layout is stable. Cuts LLM cost dramatically on retries.
Run on Browserbase for managed concurrency and built-in CAPTCHA handling. Self-hosting works but loses the Browserbase value props.
Set verbose: 0 in production to cut log noise.

Playwright in production:

Use page.locator with stable selectors, not page.$. Locators are auto-retrying.
Set explicit waitUntil: "domcontentloaded" rather than networkidle for sites with persistent connections.
Reuse browser contexts across pages from the same target. New contexts are expensive.
Profile with the trace viewer (--trace on) for any slow page.

Reliability across browser engines

Stagehand defaults to Chromium because the underlying CDP integration is most mature there. Cross-engine numbers from a 1000-page test:

Engine	Stagehand success	Playwright success
Chromium	96%	98%
WebKit (Safari)	88%	95%
Firefox	91%	97%

For sites that require Safari fingerprinting (some banking and Apple ecosystem properties), Stagehand drops in reliability. Plain Playwright on WebKit is the safer pick.

Maintenance burden over a quarter

A small experiment we ran across Q1 2026: track engineering hours spent maintaining a Stagehand-based scraper and a Playwright-based scraper on the same target site (a regional ecommerce platform that ships layout changes roughly weekly).

Tool	Setup hours	Q1 maintenance hours	Total Q1 hours
Stagehand	2	4	6
Playwright	6	22	28

Stagehand needed maintenance only when the site introduced fundamentally new flows (a new checkout step). Playwright needed maintenance every time a CSS class changed.

This is the long-term economics that the per-page cost numbers miss. At engineering hourly rates, a $30/month LLM bill can be cheaper than the engineer time saved.

When to use both

The mature pattern in 2026 is to use both. Stagehand drives discovery and exploration. Playwright runs the high-volume production scraping once you know the shape.

Concretely:

Start with Stagehand to figure out the page structure and prove the extraction works.
Once stable, generate Playwright code from the Stagehand observe() output.
Run the Playwright pipeline at scale.
Keep Stagehand on standby for the next layout change.

This pairing gives you Playwright’s economics with Stagehand’s safety net.

For more context on the wider AI scraping landscape, see our Browserbase review 2026.

Decision matrix in one place

The honest one-liner: pick by traffic volume and target volatility.

Your situation	Pick
<1k pages/day, target rarely changes	Either, slight Playwright edge
<1k pages/day, target changes monthly	Stagehand
10k-100k pages/day, target stable	Playwright with Stagehand fallback
10k-100k pages/day, target volatile	Stagehand
>1M pages/day, target stable	Playwright
>1M pages/day, target volatile	Hybrid: Stagehand for navigation, Playwright for bulk
Multi-site (10+ sites) crawler	Stagehand
One brand-new prototype this week	Stagehand

Frequently asked questions

Does Stagehand work without Browserbase?
Yes. Set env: "LOCAL" and Stagehand drives a local Chromium. You lose the managed browser cloud but the AI primitives all work.

Is the Python version of Stagehand production-ready?
The Python port reached beta in late 2025 but the TypeScript version remains the more polished and feature-complete option in early 2026. For Python scraping, browser-use is currently the better pick.

Can Playwright code call LLMs directly?
Yes. Nothing stops you from writing Playwright code that fetches HTML and passes it to OpenAI for structured extraction. That hybrid is essentially what Stagehand abstracts.

How does Stagehand handle CAPTCHAs?
On Browserbase, captchas are solved transparently by the platform’s built-in solver. Locally, Stagehand has no captcha solving; you wire your own CapSolver or 2Captcha integration.

Which one is better for SPAs (single-page apps)?
Both handle SPAs equally well at the browser level. Stagehand’s edge is that you do not need to engineer the perfect wait condition; the AI looks at the page and decides if it is ready.

Can both work in the same Node.js project without conflict?
Yes. Stagehand depends on Playwright internally. Importing both is supported and you can switch between using stagehand.page (AI primitives) and a raw chromium.launch() (deterministic) in the same script.

Are there any sites where Stagehand simply cannot work?
Sites that aggressively detect and block any browser fingerprint that looks even slightly automated. Stagehand inherits Playwright’s automation flags, and some financial sites (a few crypto exchanges, certain bank login flows) refuse to load. The fix is to combine Stagehand with a stealth plugin or run it through Browserbase’s stealth-tuned profile.

How does Stagehand handle iframes?
Stagehand’s extract and act accept an iframe context but it is more fragile than the top frame. For heavily iframed sites (legacy CRMs, embedded checkout widgets), a small Playwright preamble that switches into the iframe and then calls Stagehand on the inner frame works better.

Can Stagehand resume from a failed agent run?
Not natively. The agent.execute call is one-shot. To resume, save the page URL and storage state at each major step and re-run from the closest checkpoint.

What is the cost of observe versus extract?
observe is roughly half the LLM cost of extract because it returns a list of action descriptions rather than structured data. Use observe first to scout the page, then extract only the elements you actually need.

Common gotchas

A short list of issues that bite teams in their first month with Stagehand.

The extract schema must use Zod, not raw JSON Schema. Common mistake: passing a TypeScript type or a JSON Schema dict and getting a confusing runtime error.

act instructions are interpreted very literally. “Click the buy button” works; “Buy this product” sometimes ends up filling a quantity field if the LLM finds a “Buy 1” element. Be specific about the action verb.

Stagehand counts tokens against your LLM API key, not Browserbase. Even on Browserbase, the LLM cost is a separate line item.

Browserbase’s free tier limits concurrency. For more than 5 simultaneous sessions, you need a paid plan. Local Chromium is unlimited but you eat the host resources.

Verbose logs are extremely chatty. In production, set verbose: 0 or pipe through a structured logger. The default verbosity makes finding real errors painful.

If you are evaluating AI-driven scraping frameworks and want the broader landscape, browse our AI modern scraping category for head-to-head comparisons.

May 7, 2026