India DPDP Act for scrapers: 2026 compliance

DPDP Act India scraping compliance is the newest major data protection regime in Asia, with the Digital Personal Data Protection Act 2023 substantially in force from 2025 and the supporting rules issued in early 2026. India is one of the world’s largest sources of digital personal data, and any scraper touching Indian users now operates under the DPDP. The regime is consent-default like GDPR, but with several India-specific design choices: the data fiduciary versus data principal terminology, the Significant Data Fiduciary tier, the Data Protection Board (rather than independent regulators), and the cross-border transfer model based on a positive whitelist. This guide walks through the structure, the consent and notification rules, the operator obligations, and a practical compliance checklist for scrapers.

The audience is the technical lead or in-house counsel responsible for a scraping pipeline that touches Indian residents, or one based in India that touches anywhere.

What the DPDP Act actually covers in scraping context

The DPDP Act applies to the processing of digital personal data in India where the data is collected in digital form, or in non-digital form and subsequently digitised. It also applies to processing outside India that is in connection with offering goods or services to data principals in India. Like GDPR Article 3, the DPDP has extraterritorial reach, and scraping operations that systematically target Indian residents are in scope.

Personal data under Section 2(t) is any data about an individual who is identifiable by or in relation to such data. The “digital” qualifier is critical: the DPDP only covers digital personal data, which means voice, paper, and physical-only data falls outside the regime.

Data principal is the individual to whom the data relates. Data fiduciary is the entity that determines the purpose and means of processing. Data processor processes on behalf of the fiduciary. The terminology mirrors GDPR’s data subject and data controller but is locally distinct.

The Data Protection Board (DPB) was constituted in 2024 and operates as an adjudicatory body rather than a Western-style regulator. The DPB receives complaints, conducts inquiries, and imposes penalties. The maximum penalty under the DPDP is INR 250 crore per breach (approximately USD 30 million), making it one of the highest-penalty regimes in Asia.

For comparison with the Singapore PDPA, see the Singapore PDPA for scrapers guide. For the GDPR parallel, see the GDPR compliance guide.

The consent default and its narrow exceptions

The DPDP Act’s default rule is that personal data may be processed only with the consent of the data principal. Consent must be free, specific, informed, unconditional, unambiguous, and given through a clear affirmative action. The consent request must be presented in a manner that is clear, in plain language, and accompanied by a notice describing the personal data being collected and the purpose of processing.

This consent default makes life harder for scrapers than the corresponding GDPR Article 6(1)(f) Legitimate Interests basis. The DPDP does not include a general legitimate interests provision. It does, however, provide for “legitimate uses” under Section 7, which permits processing without consent in specific enumerated cases:

• The data principal voluntarily provided the data and has not indicated an objection.
• Processing by the State for the provision of any subsidy, benefit, service, certificate, licence, or permit.
• Compliance with a court order, judgement, or legal obligation.
• Medical emergency, threat to life, or public health response.
• Employer-employee relationship purposes.
• Compliance with a legal obligation to disclose.

For scrapers, the operationally relevant cases are extremely narrow. The “voluntarily provided” exception is the closest analogue to a general scraping basis, but it requires the data principal to have voluntarily provided the data and applies only to the purpose for which it was provided. A user who voluntarily posts on a public forum did not voluntarily provide the data to a downstream scraper.

The publicly-available exception is also narrower than PDPA Singapore: only personal data made publicly available by the data principal themselves, or by another person obligated under law to make it available, is excluded. Publicly available data that ended up online through breach, leak, or third-party aggregation is not covered.

Compliance checklist for scrapers

Control	What it requires	Why it matters
Consent or legitimate use basis	Per-source documentation	Section 6 / Section 7
Notice to data principal	At or before collection	Section 5
Purpose limitation	Use only for notified purposes	Section 5(2)
Data minimisation	Collect only what is needed	Section 4
Accuracy obligation	Reasonable steps to maintain accuracy	Section 8(3)
Reasonable security safeguards	Section 8(5)	Mandatory
Breach notification	To DPB and affected principals	Section 8(6)
Erasure on consent withdrawal	Section 8(7)	Right to be forgotten
Data Protection Officer (if SDF)	Mandatory for SDFs	Section 10(2)
Independent data auditor (if SDF)	Annual	Section 10(2)
DPIA (if SDF)	Periodic	Section 10(2)
Cross-border transfer	Whitelist mechanism	Section 16
Children’s data extra protections	Verifiable parental consent	Section 9
Grievance redressal mechanism	Easily accessible	Section 8(10)

A scraper operating at scale in India needs every row, with extra controls if classed as a Significant Data Fiduciary.

The Significant Data Fiduciary tier

Section 10 introduces the Significant Data Fiduciary (SDF) classification. The Central Government may notify any data fiduciary or class of fiduciaries as significant based on the volume and sensitivity of personal data processed, risk to electoral democracy, security of the State, public order, or other prescribed factors.

SDFs face additional obligations:
– Appoint a Data Protection Officer based in India.
– Engage an independent data auditor to conduct annual audits.
– Conduct periodic Data Protection Impact Assessments.
– Comply with such other measures as the Central Government may prescribe.

The 2025-2026 notifications classified several large-scale data processors as SDFs, including some that scrape and aggregate at industrial scale. A scraper that crosses certain volume thresholds or operates AI training pipelines on Indian personal data should expect SDF designation in the near term.

For the AI training overlay, see fair use and copyright for AI training data.

Cross-border transfer and the whitelist model

Section 16 of the DPDP gives the Central Government the power to restrict transfer of personal data to specified countries or territories. This is a positive whitelist model: by default, transfer is permitted to any country, and the government can list countries that are restricted (or, depending on rules interpretation, list permitted countries).

The 2026 rules clarified that the government would publish a list of restricted destinations rather than a list of permitted destinations. This is operator-friendly: scrapers can transfer freely to any unlisted country until and unless that country is added to the restricted list.

As of mid-2026, the restricted list is short and primarily targets countries with no diplomatic relations with India or specific national security concerns. Major scraping destinations (US, EU, Singapore, UK, Canada, Australia) are all unrestricted.

This is the most operator-friendly element of the DPDP. The flexibility makes Indian-data pipelines easier to architect than EU-data pipelines.

Decision tree: is this scrape DPDP-compliant?

Q1: Is the source data made publicly available by the data principal themselves?
    ├── Yes -> Outside DPDP scope; verify and document.
    └── No  -> Q2
Q2: Have you obtained consent from the data principal?
    ├── Yes -> Document consent; standard obligations apply.
    └── No  -> Q3
Q3: Does a Section 7 legitimate use apply?
    ├── Yes -> Document; standard obligations apply.
    └── No  -> Stop; restructure to obtain consent or change scope.

For scrapers, the consent default and narrow legitimate use list make the DPDP much more restrictive than PDPA Singapore. The publicly-available carve-out is narrower than EU GDPR’s. Plan around this; do not assume.

Data principal rights

Sections 11-14 grant data principals four core rights: right to information about processing, right to correction and erasure, right to grievance redressal, and right to nominate (a person who can exercise rights on the principal’s behalf).

The right to erasure is broader than GDPR’s because it applies on consent withdrawal and is not subject to the same balancing tests. A scraper who relies on consent and the principal withdraws consent must erase, full stop.

The right of grievance redressal requires the data fiduciary to maintain an easily accessible grievance mechanism. The DPB only accepts complaints after the principal has exhausted the fiduciary’s grievance mechanism. A scraper without a public, responsive grievance inbox is exposed.

For the worked operational pattern, see the personal vs public data scraping framework.

Children’s data and verifiable parental consent

Section 9 imposes additional obligations for data of children (under 18 in India). A data fiduciary must obtain verifiable consent from the parent or lawful guardian before processing children’s data. Tracking, behavioural monitoring, and targeted advertising directed at children are prohibited.

For scrapers, this is a major exposure if any source contains children’s data. Social media platforms, gaming platforms, education platforms, and parts of the public web all contain user-generated content from minors. The DPDP’s verifiable parental consent requirement is not a “best efforts” standard; it requires a specific verifiable mechanism.

The 2026 rules specified acceptable verification mechanisms including Aadhaar-based verification for parents and acceptable third-party verification services. Scrapers who cannot verify must avoid processing children’s data, which in practice means filtering aggressively at intake.

How DPDP enforcement is shaping up in 2025-2026

The DPB began operations in 2024 and has issued its first round of decisions in 2025-2026. The early pattern: focus on consent quality, breach notification timing, and grievance redressal mechanisms. Several mid-sized fines (INR 5-25 crore range) were issued for missing or non-functional grievance redressal.

Larger fines are anticipated as the DPB completes its first investigations of SDFs. The maximum INR 250 crore penalty per breach has not yet been imposed but is reserved for systematic violations affecting large numbers of data principals.

The DPB is more litigation-focused than the PDPC or the EDPB; it is an adjudicatory body. Operators who engage early and constructively in proceedings tend to settle for substantially reduced penalties.

External references

The DPDP Act 2023 full text is at meity.gov.in/dpdp-act-2023. The Digital Personal Data Protection Rules 2026 are at the same MeitY portal. Data Protection Board notices and decisions are published at dpb.gov.in (note: as of mid-2026 the official site is in active build-out).

Comparison: DPDP vs PDPA vs GDPR for scrapers

Dimension	DPDP India	PDPA Singapore	GDPR EU
Default basis	Consent	Consent (with carve-outs)	Lawful basis (six options)
Public data carve-out	Narrow (data principal disclosed)	Broad (generally available)	Narrow
Legitimate Interests	Limited (Section 7 list)	Yes (since 2020)	Yes (Article 6(1)(f))
Cross-border transfer	Whitelist (operator-friendly)	Comparable protection	SCCs / adequacy
Right to erasure	Yes (on consent withdrawal)	Limited	Yes (Article 17)
DPO mandatory	For SDFs only	All organisations	Conditional
Maximum fine	INR 250 crore (~USD 30M)	SGD 1M or 10% turnover	EUR 20M or 4% turnover
Children’s data extra	Verifiable parental consent	Standard rules	Article 8 conditions
AI training friendly	Not articulated	DIP since 2024	EU AI Act layers on
Adjudicatory body	DPB (litigation-style)	PDPC (regulator-style)	National DPAs (regulator-style)

The DPDP is broadly stricter than PDPA on default basis but more flexible on cross-border transfer. It is comparable to GDPR on overall stringency but with different operational textures.

A worked example: scraping Indian ecommerce listings

A scraper collects publicly available product listings from major Indian ecommerce sites (Flipkart, Myntra, Meesho, Amazon India). The dataset includes product name, price, seller name, seller location, and seller rating.

Classification: seller name and location are personal data if the seller is an individual (many Indian ecommerce sellers are sole proprietors). Product name and price are not personal data.

Basis: the publicly-available carve-out applies if the seller voluntarily made their information public on the platform. Most platform terms require sellers to display their contact information. A defensible basis exists.

Notice: a public privacy notice on the scraper’s website describing collection, purpose, retention.

Grievance: an easily accessible grievance redressal mechanism (preferably an inbox, ideally a portal).

Cross-border: if data is transferred to non-restricted countries (US, EU, Singapore), no additional mechanism required as of mid-2026.

Outcome: defensible posture, with low ongoing overhead, with the publicly-available basis documented and a grievance mechanism live.

For a parallel ecommerce-targeted scrape, see scraping Flipkart India product data.

Special cases: AI training and political data

The DPDP Act does not yet have a specific AI training carve-out or framework. The current operator posture is to obtain consent or rely on Section 7 legitimate uses where possible, and to maintain documented training data manifests in anticipation of future rules.

Political data (party affiliation, voting behaviour, election-related profiling) is treated with extra caution by the DPB, with multiple early enforcement actions targeting election-time profiling operations. Scrapers should avoid political data entirely unless they have a defensible democratic-interest basis, which is hard to construct.

FAQ

Is publicly available data exempt from DPDP?
Only if the data was made publicly available by the data principal themselves or by a person obligated under law. Third-party publication does not qualify.

Do I need consent to scrape Indian data?
The default is yes. The Section 7 legitimate uses list is narrow and rarely applies to scraping operations.

Does DPDP apply if I am outside India?
Yes if your processing relates to offering goods or services to data principals in India. Extraterritorial reach is similar to GDPR.

Do I need a local DPO?
Only if classified as a Significant Data Fiduciary. SDFs must appoint an India-based DPO.

What is the cross-border transfer rule?
A positive whitelist model: transfer is permitted by default to any country not on the government’s restricted list. Major scraping destinations are unrestricted as of mid-2026.

Extended DPDP Act enforcement analysis

The Digital Personal Data Protection Act 2023 received presidential assent in August 2023. The Data Protection Board of India was constituted in 2024, and the Act’s substantive provisions came into force in phased fashion through 2025 and 2026. The DPDP rules notified in early 2025 fleshed out consent manager registration, breach notification timelines, and cross-border transfer mechanics.

The Act’s territorial reach (Section 3) covers processing of digital personal data in India, plus processing outside India if it relates to offering goods or services to data principals in India. The latter prong directly captures scrapers harvesting India-resident data from anywhere in the world.

The lawful-basis architecture differs from GDPR. The DPDP recognises two pathways. First, consent under Section 6, which must be free, specific, informed, unconditional, and unambiguous, and given through clear affirmative action. Second, certain legitimate uses under Section 7, including performance of state functions, employment, medical emergencies, disaster response, and certain specified purposes. There is no general legitimate-interest catch-all of the GDPR Article 6(1)(f) kind.

For scrapers the practical implication is that consent is the dominant pathway, and consent for third-party scraping is rarely available. The narrow Section 7 routes do not generally cover commercial scraping. Operators must therefore consider whether DPDP applies and design accordingly.

Implementation patterns for India-touching scraping

A 2026 DPDP-aware scraping pipeline should implement eight controls.

1. Identify India-resident data principals at ingest.
2. Distinguish between processing under consent and under Section 7 legitimate use, with documentation per record.
3. Honour withdrawal of consent and erasure requests.
4. Maintain a grievance officer and process within the statutory time limit.
5. Apply notified-country transfer rules for cross-border movement.
6. Apply enhanced safeguards for children’s data (under 18) including verifiable parental consent.
7. Maintain breach notification within the prescribed window to the DPB.
8. Maintain a data protection impact assessment for significant data fiduciary status if applicable.

Code pattern: India identification and consent gate

import re

IN_PHONE = re.compile(r"\+?91[\s-]?\d{5}[\s-]?\d{5}")
IN_DOMAINS = {"in", "co.in", "org.in", "ac.in", "gov.in"}

def is_india_subject(record):
    if IN_PHONE.search(record.get("text", "")):
        return True
    if any(record.get("email","").endswith("." + d) for d in IN_DOMAINS):
        return True
    if record.get("country_iso") == "IN":
        return True
    return False

def consent_gate(record):
    if not is_india_subject(record):
        return True
    if record.get("consent_token"):
        return True
    if record.get("section_7_basis") in {"employment", "medical_emergency", "disaster", "state_function"}:
        return True
    return False

Comparison: DPDP vs GDPR for scrapers

Question	India DPDP	EU GDPR
Lawful bases	Consent plus narrow Section 7 list	Six bases including legitimate interest
Territorial reach	India processing plus targeting	EU processing plus targeting plus monitoring
Children threshold	Under 18	Under 16 default, member states can lower to 13
Cross-border transfer	Notified country list	SCCs, adequacy, derogations
Max fine	INR 250 crore	EUR 20 million or 4 percent global turnover
Breach notification	DPB notification window	72 hours to DPA, individuals where high risk
Data principal rights	Access, correction, erasure, grievance, nominate	Access, rectification, erasure, restriction, portability, object

Additional FAQ

Does DPDP have a legitimate-interest catch-all?
No. Section 7 specifies discrete legitimate uses. Commercial scraping of public data does not fit any listed category.

What is a Significant Data Fiduciary?
A class designated by the central government based on volume, sensitivity, risk to electoral democracy or public order, sovereignty, or India’s security. Significant Data Fiduciaries face enhanced obligations.

How does DPDP treat AI training data?
The Act does not have AI-specific provisions, but consent or a Section 7 ground is required for any personal data processing including training.

What happens to non-personal data?
DPDP regulates only digital personal data. A separate non-personal data framework has been discussed but not enacted as of 2026.

The DPDP Act’s consent architecture

The DPDP Act places consent at the centre of the lawful processing analysis. Section 6 specifies that consent must be free, specific, informed, unconditional, and unambiguous, and must be given through clear affirmative action. The Act introduced the concept of a Consent Manager (Section 6(7)), a registered intermediary that can manage consent on behalf of data principals.

For scrapers the consent requirement is generally not satisfiable. Third-party scraping involves no direct interaction with the data principal, so affirmative consent cannot be obtained. The consent pathway is therefore typically unavailable for commercial scraping.

The Consent Manager mechanism is a 2026 innovation that may eventually create new pathways. A data principal who has consented through a Consent Manager can have that consent presented to downstream services. The infrastructure is still maturing, and broad scraper-friendly consent flows have not emerged. Scrapers should monitor the development but should not rely on it for current operations.

Section 7 legitimate uses in detail

Section 7 of the DPDP Act lists the legitimate uses for which processing may occur without consent. The list is closed, meaning it cannot be expanded by regulator interpretation. The legitimate uses include: processing for state functions; medical emergency; disaster response; employment-related processing; and a few specific public interest categories.

Critically the list does not include a general legitimate-interest catch-all comparable to GDPR Article 6(1)(f). This makes the DPDP regime more restrictive for scrapers than the GDPR. A scrape that would pass an LIA under GDPR may have no available basis under DPDP.

The 2025 DPDP Rules provided some additional clarification on Section 7 application. The rules tightened the definition of employment-related processing and specified that the disaster response basis is limited to time-bound emergencies declared by competent authority. Neither change benefits scrapers.

Significant Data Fiduciary obligations

Section 10 of the DPDP Act allows the central government to designate certain data fiduciaries as Significant Data Fiduciaries (SDFs) based on volume, sensitivity, risk to electoral democracy, public order, sovereignty, or India’s security. SDFs face enhanced obligations including DPIA, data protection officer designation, and independent audit.

The 2025 government notification of initial SDF categories included social media platforms above thresholds, e-commerce platforms above thresholds, and certain AI service providers. Future notifications may extend to data brokers and aggregators.

For scrapers the practical implication is that operating at scale in India increases the risk of SDF designation. A scraper crossing the volume threshold or feeding sensitive AI applications should plan for SDF obligations. The compliance uplift is non-trivial, including a designated DPO, mandatory DPIAs, and annual audits.

Cross-border transfer under DPDP

The DPDP Act takes a different approach to cross-border transfer than GDPR. Section 16 permits transfer to any country except those specifically restricted by the central government through notification. The starting position is permissive, with restrictions added as needed.

The 2025 DPDP Rules clarified the transfer framework. As of 2026 only a small number of countries are on the restricted list, and the criteria for addition are based on national security and public order rather than data protection adequacy.

For scrapers the implication is that exporting India-resident data to most jurisdictions is permissible under the DPDP, but the source-side consent or Section 7 basis must still be established. The DPDP transfer rules do not create a lawful basis where none otherwise exists.

Next steps

The fastest path to DPDP compliance is to publish a privacy notice and grievance redressal mechanism, document the basis (preferably the narrow publicly-available carve-out where it applies), and prepare for SDF designation if your volume warrants it. For broader Asia compliance, head to the DRT compliance hub and pair this with the PDPA Singapore guide.

This guide is informational, not legal advice.

Scrapybara vs Browserbase for agentic workflows

The Scrapybara vs Browserbase decision has become the cleanest fork in the road for any team building agentic workflows in 2026. Both are managed cloud platforms designed for AI agents to drive real browsers and operating systems. They overlap on the surface and diverge significantly in philosophy. Browserbase is a browser cloud, period. Scrapybara is a virtual desktop cloud that happens to include a browser.

That difference shows up everywhere. This guide compares both platforms through the lens of an engineer building a scraping or research agent in 2026, with code, benchmarks, and a clear picture of which one fits which problem.

What each platform actually is

Browserbase gives you a Chromium browser session over a Playwright-compatible WebSocket endpoint. You write Playwright or Stagehand code, you connect to a managed session, you scrape. The platform handles fingerprinting, proxies, CAPTCHA solving, and observability. Browser-only.

Scrapybara gives you a full Ubuntu desktop in the cloud with a browser, a terminal, a file system, and X11. Your agent can launch any application, not just a browser. The platform exposes a Computer Use API directly compatible with Anthropic’s Claude Computer Use and OpenAI’s Operator.

If your agent only needs the web, Browserbase is the leaner choice. If your agent needs to download a file, manipulate it in a desktop application, or run a CLI tool, Scrapybara is the only real option.

Pricing in 2026

Both are usage-based but with different units.

Browserbase:

Plan	Cost	Browser-minutes
Developer	$39/mo	5,000 included, then $0.0078/min
Startup	$399/mo	70,000 included, then $0.0057/min

Scrapybara:

Plan	Cost	Compute-hours
Hobby	Pay-as-you-go	$0.10/hour
Pro	$99/mo	100 hours included, then $0.06/hour
Scale	Custom	Custom

A like-for-like comparison: 1000 sessions averaging 3 minutes each.

Platform	Compute cost	Proxy cost (residential)	Total
Browserbase Developer	$23.40	$4.50	$27.90
Browserbase Startup	$17.10	$4.50	$21.60
Scrapybara Hobby	$5.00	$4.50 (BYO)	$9.50
Scrapybara Pro	$3.00	$4.50 (BYO)	$7.50

Scrapybara is meaningfully cheaper on compute. Browserbase costs more but includes residential proxies, CAPTCHA solving, and the Stagehand framework as native primitives.

Mental model in one sentence each

Scrapybara is “give me a Linux box my agent can drive.” Browserbase is “give me a Chrome tab my agent can drive.”

If you find yourself wishing the agent could apt install ffmpeg and then run a CLI tool on a downloaded video, Scrapybara fits. If you find yourself wishing the agent could click around five sites and emit JSON, Browserbase fits.

Setup speed

Both platforms ship in minutes.

Browserbase:

import { Browserbase } from "@browserbasehq/sdk";
import { chromium } from "playwright";

const bb = new Browserbase({ apiKey: process.env.BROWSERBASE_API_KEY! });
const session = await bb.sessions.create({ projectId: process.env.BROWSERBASE_PROJECT_ID! });
const browser = await chromium.connectOverCDP(session.connectUrl);
const page = browser.contexts()[0].pages()[0];

await page.goto("https://example.com");
console.log(await page.title());

await browser.close();

Scrapybara:

from scrapybara import Scrapybara

client = Scrapybara(api_key="scrapy_...")

instance = client.start_ubuntu()
instance.bash(command="echo hello")
instance.computer.screenshot()  # returns base64 PNG of the desktop
instance.browser.start()
instance.browser.goto("https://example.com")
title = instance.browser.evaluate("document.title")
print(title)

instance.stop()

Browserbase ships a Playwright-compatible API, so any existing Playwright code drops in. Scrapybara ships a richer API with desktop primitives, but your existing Playwright code needs adaptation.

Latency to first action

Cold-start times measured March 2026:

Platform	Cold start	Warm	First navigation
Browserbase	2.1 s	0.4 s	0.8 s after start
Scrapybara Ubuntu	8.4 s	2.1 s	1.4 s after start

Browserbase wins on latency because it boots only Chromium. Scrapybara boots a whole desktop. For pure browser workloads, the latency cost on Scrapybara is real. For multi-app workloads, the latency is amortized over a longer session.

Computer Use integration

This is where Scrapybara pulls ahead for agentic workflows.

Scrapybara is built specifically as a Computer Use target. Anthropic’s Claude Computer Use API and OpenAI’s Operator both plug into Scrapybara as the underlying compute environment.

from scrapybara import Scrapybara
from anthropic import Anthropic

client = Scrapybara(api_key="scrapy_...")
instance = client.start_ubuntu()
anthropic = Anthropic()

response = anthropic.beta.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=4096,
    tools=[{
        "type": "computer_20241022",
        "name": "computer",
        "display_width_px": 1024,
        "display_height_px": 768,
    }],
    messages=[{
        "role": "user",
        "content": "Open the browser, search Google for 'browserbase pricing', and return the first result URL."
    }],
    betas=["computer-use-2024-10-22"],
)

# loop: feed Claude's tool calls into instance.computer.* and feed back

Browserbase, by contrast, supports Computer Use only through the Stagehand agent primitive, and only for the browser. If your agent needs to open Excel, manipulate files in a terminal, or interact with desktop apps, Browserbase cannot help.

For more on Computer Use vs Operator, see our OpenAI Operator vs Anthropic Computer Use comparison.

Browser primitive depth

Where Browserbase is more polished is the browser experience. Stagehand is theirs, and it shows.

Browserbase + Stagehand:

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const stagehand = new Stagehand({ env: "BROWSERBASE", modelName: "gpt-4o-mini" });
await stagehand.init();
const page = stagehand.page;
await page.goto("https://news.ycombinator.com");

const top = await page.extract({
  instruction: "Top 5 stories with title, score, submitter",
  schema: z.object({
    stories: z.array(z.object({ title: z.string(), score: z.number(), submitter: z.string() })),
  }),
});
console.log(top);
await stagehand.close();

Scrapybara has a browser API but no equivalent of Stagehand’s extract. You bring your own LLM extraction layer.

For the Stagehand vs Playwright story, see Stagehand vs Playwright for AI scraping.

Side-by-side comparison

Dimension	Scrapybara	Browserbase
Compute target	Full Ubuntu desktop	Chromium browser only
Native API surface	Browser, terminal, file system, GUI	Browser only
Computer Use support	First-class for Claude and Operator	Browser-only via Stagehand agent
AI extraction primitives	BYO	Stagehand native
Proxy support	BYO	Built-in residential and stealth
CAPTCHA solving	BYO	Built-in for major types
Session replay	Yes (video plus event trace)	Yes (video plus event trace)
Cost per 1000 short sessions	$7 to $10	$20 to $28
Best fit	Agents that need full OS access	Browser-only AI workflows

The two products are complementary more than competitive. Scrapybara is the right pick when your agent needs to do anything outside a browser. Browserbase is the right pick when your work fits in a browser and you want the most polished AI-native experience.

Computer Use action loop on Scrapybara

The actual loop that drives Claude Computer Use against a Scrapybara instance is simple but worth seeing in full.

def computer_action(action_input):
    if action_input["action"] == "screenshot":
        return instance.computer.screenshot()
    elif action_input["action"] == "left_click":
        x, y = action_input["coordinate"]
        return instance.computer.left_click(x=x, y=y)
    elif action_input["action"] == "type":
        return instance.computer.type(text=action_input["text"])
    elif action_input["action"] == "key":
        return instance.computer.key(text=action_input["text"])
    # ... more actions

messages = [{"role": "user", "content": "Open the browser and search Hacker News"}]
while True:
    resp = anthropic.beta.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        tools=[{"type": "computer_20241022", "name": "computer",
                "display_width_px": 1024, "display_height_px": 768}],
        messages=messages,
        betas=["computer-use-2024-10-22"],
    )
    if resp.stop_reason == "end_turn":
        break
    for block in resp.content:
        if block.type == "tool_use":
            result = computer_action(block.input)
            messages.append({"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": [{
                "type": "tool_result", "tool_use_id": block.id, "content": result,
            }]})

The loop is roughly 30 lines and works for any Computer Use task. Browserbase has an analogous loop but only for browser actions.

Real production patterns

We ran two pipelines in parallel to compare.

Pipeline A: scrape Lazada and Shopee product listings, extract to Postgres. Pure browser work.

Pipeline B: download invoice PDFs from a supplier portal, extract line items with a vision model, push to a procurement system. Needs file download and PDF processing.

Pipeline A on Browserbase + Stagehand:

• Time to working code: 2 hours
• Cost per 1000 products: $6 (compute) + $4 (residential proxy) + $2 (LLM)
• Maintenance burden: low

Pipeline A on Scrapybara:

• Time to working code: 4 hours (had to wire LLM extraction layer)
• Cost per 1000 products: $4 (compute) + $4 (BYO proxy) + $2 (LLM)
• Maintenance burden: medium

Pipeline B on Browserbase: not feasible. The browser cannot natively process the downloaded PDFs.

Pipeline B on Scrapybara:

• Time to working code: 5 hours
• Cost per 1000 invoices: $9 (compute, longer sessions) + $3 (proxy) + $5 (vision model)
• Maintenance burden: medium

Detailed pipeline metrics

The pipelines above ran for 30 days each. Aggregate numbers:

Metric	Pipeline A on Browserbase	Pipeline A on Scrapybara	Pipeline B on Scrapybara
Pages or invoices processed	90,000 products	90,000 products	12,000 invoices
Total cost	$1,080	$720	$204
Per-unit cost	$0.012	$0.008	$0.017
Median latency	7.4 s	9.1 s	38 s
Success rate	97.4%	95.1%	91.2%
On-call incidents	1	4	5

Browserbase’s higher cost was offset by lower incident count. Scrapybara’s lower cost came with more operational overhead because the BYO proxy layer needed its own monitoring. The pipeline B numbers are for a fundamentally different workload (PDF processing) where Scrapybara was the only option.

Where each platform showed weakness

Browserbase’s main weakness was the proxy markup. Halfway through the test, we switched the highest-volume target to BYO proxies through the session API, which dropped the proxy cost line by roughly 60 percent. The platform supports BYO but the docs do not lead with it.

Scrapybara’s main weakness was the lack of an extraction primitive. We had to wire OpenAI structured output ourselves, which added roughly two hours of development time per new target. The Scrapybara team has hinted at native extraction primitives in 2026 H2 but as of writing it is still BYO.

Adding proxies

Both platforms support proxies, but the integration depth differs.

Browserbase ships first-party residential proxies billed at $8/GB and stealth (datacenter) at $0.30/GB. Set them at session creation:

const session = await bb.sessions.create({
  projectId: process.env.BROWSERBASE_PROJECT_ID!,
  proxies: true, // or pass an explicit proxy config
});

Scrapybara expects you to bring your own. Configure inside the instance:

instance.browser.start(proxy={"server": "http://proxy.example.com:8000", "username": "u", "password": "p"})

For ASEAN scraping where mobile carrier IPs matter, Singapore mobile proxy plugs into either platform with one config block.

Geographic coverage

Browserbase: US East, US West, EU West, APAC South (Singapore). Compute always in one of these regions; proxies cover roughly 195 countries.

Scrapybara: US East, US West, EU West. APAC region is on the roadmap as of mid-2026. For Asia-targeted scraping, latency is higher than Browserbase.

CAPTCHA story in detail

Browserbase ships built-in solvers for reCAPTCHA v2/v3, hCaptcha, and Cloudflare Turnstile, with quoted success rates of 88 to 95 percent depending on type.

Scrapybara has no built-in CAPTCHA solving. You wire 2Captcha, CapSolver, or a similar service. The integration takes maybe 50 lines of code, but it is friction the Browserbase user does not have.

For teams that scrape CAPTCHA-heavy targets (LinkedIn, Indeed, several banking portals), Browserbase’s built-in solver is a real time-saver. For targets without CAPTCHAs (most B2B SaaS apps you have legitimate credentials for), the difference is irrelevant.

Observability

Both ship session replay with video and event traces. Browserbase’s UI is more polished and the search across sessions is faster. Scrapybara’s replay shows the full desktop, which is more useful when your agent uses non-browser apps.

Both export OpenTelemetry spans for trace correlation with your existing observability stack.

Security and isolation

Browserbase sessions run in ephemeral containers with full process isolation. Cookies, storage, and any cached state are wiped at session end unless you opt into context persistence. Network egress is funneled through the platform’s IP pool by default.

Scrapybara instances are full Ubuntu VMs with disk persistence per instance ID. This is more powerful and more dangerous: a compromised agent that gets shell access on a Scrapybara instance can persist files there, install software, and reuse the state across runs. For sensitive workloads, treat Scrapybara instances as ephemeral and tear them down explicitly.

Both platforms hold SOC 2 Type II reports as of 2026 and offer DPAs for GDPR-relevant workloads. Neither is yet HIPAA-eligible, so healthcare scraping requires self-hosting.

SDKs and language ecosystem

Browserbase’s primary SDK is TypeScript. The Python SDK exists but lags by roughly one release. For TypeScript-heavy teams, the platform feels native; for Python teams, less so.

Scrapybara’s primary SDK is Python. The TypeScript SDK exists but covers fewer features. For Python-heavy teams (most data science and AI shops), this is the natural fit.

If your team is split or polyglot, Browserbase wins because its WebSocket endpoint speaks any Playwright client. Scrapybara is more SDK-locked because the Computer Use primitives are first-class only in their SDKs.

Production recommendations

Use Browserbase if:

• Your scraping is browser-only
• You want Stagehand as your AI primitive layer
• You value built-in CAPTCHA solving
• You prefer a TypeScript-first SDK

Use Scrapybara if:

• Your agent needs file system, terminal, or desktop access
• You are running Claude Computer Use or OpenAI Operator
• You want lower per-hour compute cost and BYO proxy
• You prefer a Python-first SDK

The mature pattern in 2026 is to use both: Browserbase for the high-volume browser scraping, Scrapybara for the agent workflows that go beyond the browser. They cost together about what one Browserbase Startup plan costs, and you cover both shapes of work.

For broader context on the agentic browser space, see our agentic browser revolution guide.

Decision matrix

Your situation	Pick
Browser-only scraping, want fastest path	Browserbase + Stagehand
Browser-only scraping, want lowest cost	Self-hosted Playwright (not Scrapybara, not Browserbase)
Agent needs file system or terminal	Scrapybara
Running Claude Computer Use or OpenAI Operator	Scrapybara
TypeScript-first team, multi-site scraping	Browserbase
Python-first team, complex multi-app workflow	Scrapybara
Need built-in CAPTCHA solving	Browserbase
Need built-in residential proxies, simple billing	Browserbase
Need to install custom CLI tools at runtime	Scrapybara
One-off prototype, single site	Either, slight Browserbase edge

Frequently asked questions

Can I run my own LLM extraction prompts on Scrapybara?
Yes. Scrapybara just exposes the underlying compute. Wrap your favorite LLM (OpenAI, Anthropic, local) and call it from your agent code.

Are there any Scrapybara features Browserbase has copied or is likely to copy?
Both platforms ship session replay, both ship persistent contexts. Browserbase has hinted at adding limited terminal access on Scale plans but has not shipped. Scrapybara has hinted at native extraction primitives. Convergence is happening slowly.

Does Browserbase have a Python SDK?
Yes, but the TypeScript SDK is more feature-complete. For the latest features, TypeScript is the better choice in early 2026.

Can Scrapybara run alongside other browser frameworks?
Yes. The Scrapybara browser is a real Chromium that speaks CDP. You can drive it from Playwright, Puppeteer, or any browser automation library.

What about Hyperbrowser, Steel.dev, and other competitors?
Hyperbrowser is closest to Browserbase in shape but smaller. Steel.dev sits between Browserbase and Scrapybara on capabilities. Both are valid alternatives if pricing or feature set fits better.

How does authentication work for sites that need login?
Both platforms support persistent contexts that store cookies and localStorage. Save once after manual login, reload on each session. For higher-throughput pipelines, store the context state encrypted in your secret store.

Can I expose a Scrapybara instance to the public internet?
Not directly. The instance is private to the Scrapybara network. To expose a service running on the instance, tunnel through ngrok or a similar reverse proxy from inside the instance.

What are the per-instance resource limits on Scrapybara?
Default Hobby tier: 2 vCPU, 4 GB RAM, 20 GB disk. Pro: 4 vCPU, 8 GB RAM, 50 GB disk. Scale: negotiated. For most browser-driving tasks, Pro is sufficient. For video processing or large data jobs, request a custom config.

Can either platform handle file uploads to a target site?
Both can. Browserbase exposes the standard Playwright setInputFiles API. Scrapybara supports the same plus drag-and-drop from the desktop file system.

What about secrets management?
Neither platform provides a secrets vault. Pass secrets via environment variables to your code, or pull from your own secrets manager (AWS Secrets Manager, HashiCorp Vault, Doppler) before passing to the platform.

Can I use both platforms in the same workflow?
Yes. A common pattern is to use Browserbase for the high-volume scraping leg and Scrapybara for the post-processing leg that involves file manipulation. They are complementary by design.

Common production gotchas

For Browserbase: forgetting to release sessions burns minutes; mismatched proxy and target region triggers cloaking; the LLM bill hides separately on your OpenAI account.

For Scrapybara: the desktop is a real Ubuntu so resource leaks (lingering processes, large temp files) accumulate across long-lived instances; explicitly stop instances when done; mount your file output to a known location for export rather than relying on the temp file system.

For both: session replay retention is finite. Export critical traces if you need them long-term. The platforms do not retain everything indefinitely.

Six-month verdict

After running both side by side for six months: Browserbase wins for any team where 100 percent of work is browser-based. The polish and Stagehand integration are worth the premium. Scrapybara wins for any team where even 20 percent of work goes outside the browser, because that 20 percent is otherwise blocking. The right answer for many teams is to use Browserbase as the default and reach for Scrapybara when the task explicitly needs OS access.

If you are deciding between agentic platforms, the AI modern scraping category has more reviews and head-to-head comparisons that can help anchor the decision.

Singapore PDPA for scrapers: a 2026 practical guide

PDPA Singapore scraping rules are the most pragmatic in the ASEAN region, and that pragmatism has made Singapore one of the most attractive jurisdictions for data-driven businesses operating across Asia. The Personal Data Protection Act, originally enacted in 2012 and significantly amended in 2020 and 2024, governs how personal data is collected, used, disclosed, and stored. Unlike GDPR, the PDPA includes a relatively broad publicly-available exception that scraping operators can rely on, and unlike most other regimes it explicitly distinguishes consent obligations from data protection obligations. This guide walks through the PDPA structure, the publicly-available rules, the deemed consent and notification frameworks, and a working compliance checklist.

The audience is the technical lead or in-house counsel responsible for a scraping pipeline that touches Singapore residents, or one based in Singapore that touches anywhere.

What the PDPA actually covers in scraping context

The PDPA applies to any organisation that collects, uses, or discloses personal data about individuals in Singapore. It applies regardless of whether the organisation is in Singapore. Like GDPR, it has effective extraterritorial reach when scraping operations target Singapore residents.

Personal data under Section 2 is data, whether true or not, about an individual who can be identified from that data, or from that data and other information to which the organisation has or is likely to have access. The definition mirrors GDPR’s “identifiable” standard but with a slightly narrower “is likely to have access” qualifier that gives operators a small drafting window.

The PDPC (Personal Data Protection Commission) enforces the regulation. Penalties under the 2020 amendments rose significantly: financial penalties of up to SGD 1 million or 10 percent of annual turnover in Singapore (whichever is higher), with the higher cap applying to organisations with annual turnover above SGD 10 million.

The 2024 amendments introduced the Data Innovation Provisions, allowing certain forms of business analytics and AI training under the Legitimate Interests basis, with documentation and disclosure requirements. This is the most operator-friendly addition in the region.

For the broader ASEAN compliance picture, see the personal vs public data scraping framework. For comparison with GDPR, see the GDPR compliance guide.

The publicly-available exception, properly read

The PDPA’s publicly-available exception is broader than the GDPR’s equivalent, but narrower than most operators assume. Schedule 1 Part 3 of the PDPA exempts collection and use of personal data that is publicly available, defined in Section 2 as personal data that is generally available to the public, and includes personal data that can be observed by reasonably expected means at a location or event at which the individual appears and that is open to the public.

Three operational implications.

First, “generally available to the public” requires that the data be available to anyone who looks, not just to those who clear a barrier. A profile behind a paywall is not publicly available. A profile behind a free signup is debatable.

Second, the exception applies to collection and use, but not always to subsequent disclosure to third parties. If you scrape publicly available data and resell it, the resale may trigger separate obligations.

Third, the PDPC has consistently held that observable behaviour at public events (a name on a public attendee list, a quote in a public news article) is publicly available. Behaviour inferred from observation (a profile built from behavioural patterns) may not be.

The PDPC issued an advisory in 2024 specifically addressing scraping for AI training, holding that publicly available data may be used for training without consent under the publicly-available exception, provided the use complies with the data protection obligations (notification, purpose limitation, accuracy, protection, retention, transfer).

Consent and the notification obligation

For data not within the publicly-available exception, the PDPA requires consent. Consent can be express (the data subject explicitly agrees) or deemed (the individual voluntarily provides the data for a purpose, or is informed and does not opt out within a reasonable time). The 2020 amendments expanded deemed consent significantly.

For scrapers, deemed consent rarely applies because the data subject did not voluntarily provide the data to you. The relevant alternative bases are:

Legitimate Interests: introduced in 2020, allows collection without consent where the legitimate interest of the organisation outweighs any adverse effect on the individual. Requires a documented assessment.

Business Improvement: a narrow exception for using existing personal data to improve products and services, subject to safeguards.

Research: a research exception for non-commercial research purposes.

Notification, even where consent is not required, is generally still required. The organisation must notify the individual of the purposes for which the data will be collected, used, or disclosed. For scraping operators, notification is typically delivered through a public privacy notice rather than per-individual contact.

For the Indian comparison and where the regimes differ, see the India DPDP Act for scrapers guide.

The Do Not Call provisions

The PDPA includes Do Not Call (DNC) provisions that govern marketing communications to Singapore phone numbers. These rules sit alongside the data protection obligations and are independently enforced.

Scrapers who collect Singapore phone numbers and use them (or licence them) for marketing must check the DNC registries before sending. The PDPC operates three registers (No Voice Call, No Text Message, No Fax). Failing to check before sending is a separate violation with separate fines.

The 2020 amendments added that organisations are responsible for ensuring third-party marketers they engage also comply with DNC. A scraper that resells phone numbers to marketing operators is exposed to this chain liability.

Compliance checklist for scraping operators

Control	What it requires	Why it matters
Publicly-available assessment	Per-source documentation	Schedule 1 Part 3 defence
Lawful basis for non-public data	Legitimate Interests assessment	Section 13
Privacy notice published	Public page describing purposes	Section 20 (notification)
Purpose limitation	Use only for stated purposes	Section 18
Data accuracy	Reasonable steps to ensure accuracy	Section 23
Protection obligation	Reasonable security arrangements	Section 24
Retention limits	Cease retention when no longer needed	Section 25
Transfer limits	Comparable protection in recipient country	Section 26
Data Protection Officer	Mandatory for all organisations	Section 11
Access and correction requests	Respond within 30 days	Sections 21-22
Withdrawal of consent	Honour withdrawal	Section 16
Do Not Call check (if marketing)	Per-number, current registry	DNC provisions
Data Innovation Provisions notice	If using LI for AI training	2024 amendments
Breach notification	If significant harm or 500+ affected	Section 26D

Most scrapers can tick most rows in a fortnight of work. The DPO requirement (Section 11) is the most-missed obligation.

Decision tree: is this scrape PDPA-compliant?

Q1: Is the source publicly available (general access, no barrier)?
    ├── Yes -> Q1a: Is the use for AI training or aggregation?
    │           ├── Yes -> Document publicly-available basis; comply with data protection obligations.
    │           └── No  -> Document publicly-available basis; standard obligations apply.
    └── No  -> Q2
Q2: Have you obtained express or deemed consent?
    ├── Yes -> Document; standard obligations apply.
    └── No  -> Q3
Q3: Can you rely on Legitimate Interests?
    ├── Yes -> Conduct LI assessment; publish notice; standard obligations apply.
    └── No  -> Stop or restructure.

The Data Innovation Provisions and AI training

The 2024 amendments added the Data Innovation Provisions (DIP) at Sections 17A-17C. These allow organisations to use personal data, without consent, for business innovation purposes that include analytics, AI training, and product development, subject to four conditions:

1. The use is for a legitimate purpose that the individual would reasonably expect.
2. The organisation has conducted a risk assessment.
3. The organisation publishes a clear notice describing the use.
4. The organisation provides an opt-out mechanism that is honoured.

For scraping operators training AI models on publicly available Singapore data, the DIP is the cleanest path. Document the assessment, publish the notice, run the opt-out inbox.

Cross-border transfer obligations

Section 26 requires that personal data transferred outside Singapore be afforded a standard of protection comparable to the PDPA. The PDPC’s approach is more flexible than the EU’s, accepting the recipient’s contractual undertakings, the recipient’s binding corporate rules, or the recipient’s location in a jurisdiction with comparable laws.

The PDPC has not published a formal adequacy list. Instead, scraping operators evaluate each recipient case-by-case. Major comparable jurisdictions include the EU/EEA, the UK, Australia, Canada, Japan, and South Korea.

For US transfers, the PDPC accepts contractual clauses similar to the EU SCCs. The Data Privacy Framework is not directly relevant to PDPA, but a US recipient certified under DPF generally satisfies PDPA-equivalent protection in practice.

For the broader cross-border question, see scraping data from EU sites jurisdictional realities.

How PDPA enforcement shifted in 2024 and 2025

Two trends. First, the PDPC moved from advisory-heavy to fine-active. Multiple seven-figure fines in 2024-2025 against organisations that failed to implement reasonable security arrangements (Section 24) following data breaches. Scraping operators with unprotected storage are exposed.

Second, the PDPC published explicit AI guidance in 2024 and 2025 covering training data, output safety, and accountability. The guidance is non-binding but shapes enforcement expectations. A scraping operator who follows the guidance is in a defensible position.

The Voluntary Disclosure Programme (VDP), launched in 2025, encourages organisations that discover their own breaches to self-report in exchange for reduced penalties. For scraping operators who discover compliance gaps, the VDP is a useful tool.

External references

The PDPA full text is at pdpc.gov.sg/legislation/personal-data-protection-act. The PDPC advisories and guidelines are at pdpc.gov.sg/Guidelines-and-Consultation. The PDPC enforcement decisions library is searchable at pdpc.gov.sg/Commissions-Decisions.

Comparison: PDPA vs GDPR vs DPDP

Dimension	PDPA Singapore	GDPR EU	DPDP India
Personal data definition	Identifiable individual	Identifiable individual	Digital personal data of identifiable individual
Public data carve-out	Broad	Narrow	Limited
Consent default	Required unless exception	Lawful basis required	Required unless exception
Legitimate interests	Yes (since 2020)	Yes	Limited (notice and consent default)
AI training friendly	Yes (DIP since 2024)	EU AI Act layers on	Not yet articulated
Cross-border transfer	Comparable protection test	SCCs / adequacy	Whitelist of approved countries
Maximum fine	SGD 1M or 10% turnover	EUR 20M or 4% turnover	INR 250 crore (~USD 30M)
Mandatory DPO	Yes (all organisations)	Conditional	Yes for significant data fiduciaries
Breach notification	Yes (significant harm or 500+)	Yes (72 hours, risky breaches)	Yes

PDPA is the most operator-friendly of the three, particularly for AI training pipelines that fit the DIP framework. DPDP is the strictest on consent default. GDPR remains the strictest overall.

A worked example: scraping Singapore property listings

A scraper collects publicly available property listings from major Singapore portals (PropertyGuru, 99.co, EdgeProp). The dataset includes property address, asking price, agent name, agent contact phone, agent licence number, and listing date.

Classification: agent name and contact details are personal data. Property address is not personal data unless linked to an owner. Asking price is not personal data.

Basis: publicly-available exception applies to the agent contact data, because agents publish their information openly for purposes of being contacted by potential clients.

Notification: a clear privacy notice on the scraper’s website describing the collection, the purpose (market intelligence for B2B customers), the retention period, and the opt-out path.

Do Not Call: if the dataset is later used for marketing calls to those agents, the DNC registries must be checked per-number per-call.

Outcome: defensible posture, low overhead, with a documented publicly-available assessment and a published notice. PDPA does not require an LIA in this case because the publicly-available exception applies.

For the deeper market-intelligence build pattern, see scraping job board data for talent intelligence.

Mandatory Data Protection Officer

Every organisation in scope of PDPA must appoint a Data Protection Officer (Section 11). The DPO does not need to be in Singapore, does not need to be a lawyer, and can be a current employee with other responsibilities. Small scraping operations commonly designate the engineering lead or compliance manager.

The DPO must be contactable, and their contact details (or at least the role’s contact details) must be available to the public. A common implementation is a dpo@yourcompany.com inbox listed on the privacy notice page.

Failing to appoint a DPO is itself a violation. The PDPC has issued multiple enforcement actions for this failure alone.

FAQ

Is publicly available data exempt from PDPA?
Partially. The publicly-available exception covers collection and use, but data protection obligations (purpose limitation, accuracy, protection, retention, transfer) still apply.

Do I need consent to scrape?
Not always. The publicly-available exception, the Legitimate Interests basis, and the Data Innovation Provisions all permit collection without express consent in defined circumstances.

Does PDPA apply if I am outside Singapore?
Yes if your processing covers individuals in Singapore. The PDPA has effective extraterritorial reach.

Do I need a Singapore representative?
No. Unlike GDPR, PDPA does not require a local representative for non-Singapore organisations.

What is the typical fine under PDPA in 2026?
Penalties range from low six figures for technical breaches up to SGD 1 million or 10 percent of annual Singapore turnover for serious violations, with the higher cap applying to larger organisations.

Extended PDPA enforcement analysis 2024-2026

The Personal Data Protection Commission stepped up enforcement after the 2020-2021 amendments brought mandatory breach notification, an enhanced financial penalty cap (10 percent of annual turnover above SGD 10 million), and the data portability obligation. The 2024-2026 window saw three notable directions.

First, the PDPC published the AI Model Governance Framework second edition in May 2024. The framework treats training data provenance as a primary governance question and recommends documented LIA-equivalent assessments for personal data ingested into AI training pipelines.

Second, the PDPC’s enforcement decisions in 2024 and 2025 showed that scraping operators are squarely in scope when they collect personal data of Singapore residents, regardless of the operator’s location. The Section 13 consent obligation is the central question, with Section 17 deemed-consent and the legitimate-interest exception in the First Schedule providing the practical pathways.

Third, the cross-border transfer rules under Section 26 require the receiving controller to be bound to a comparable standard. The PDPC’s 2024 guidance accepts a narrow set of mechanisms (consent, contract, ASEAN MCCs, certifications). Scrapers exporting Singapore-resident data must document the chosen mechanism.

Implementation patterns for a PDPA-clean pipeline

A 2026 PDPA-compliant scraping pipeline should include seven controls.

1. Identify Singapore-resident data subjects at ingest using a combination of profile signals and IP geolocation.
2. Apply the legitimate-interest exception with a documented assessment, or rely on Section 17 deemed consent where applicable.
3. Honour withdrawal of consent requests with a measured response time.
4. Provide a do-not-call workflow for any phone numbers collected.
5. Apply transfer-limitation safeguards for data leaving Singapore.
6. Maintain a data protection officer designation and contact.
7. Maintain a data breach notification process meeting the 72-hour PDPC notification window.

Code pattern: Singapore identification at ingest

import re

SG_PHONE = re.compile(r"\+?65[\s-]?\d{4}[\s-]?\d{4}")
SG_DOMAINS = {"sg", "com.sg", "edu.sg", "gov.sg", "org.sg"}

def is_singapore_subject(record):
    if SG_PHONE.search(record.get("text", "")):
        return True
    email = record.get("email", "")
    if any(email.endswith("." + d) for d in SG_DOMAINS):
        return True
    if record.get("country_iso") == "SG":
        return True
    return False

Comparison: PDPA vs neighbouring regimes for scrapers

Question	Singapore PDPA	Malaysia PDPA	Indonesia PDP Law	Thailand PDPA
Legitimate interest exception	Yes (First Schedule)	No general exception	Yes (limited)	Yes (limited)
Public data carve-out	Limited	Limited	Limited	Limited
Cross-border transfer rule	Comparable standard	Whitelisted countries	Adequate protection	Adequate protection
Max fine	SGD 1M or 10 percent of turnover	RM 500K	IDR 5B or 2 percent of revenue	THB 5M plus criminal
Breach notification	72 hours to PDPC	Yes	Yes	72 hours

Additional FAQ

Does PDPA apply to scraping operators outside Singapore?
Yes if they collect, use, or disclose personal data of individuals in Singapore. The PDPA does not require an establishment in Singapore for jurisdiction.

Is the legitimate-interest exception identical to GDPR Article 6(1)(f)?
Functionally similar but procedurally different. Singapore requires a prescribed assessment and notification. The substantive balancing test is comparable.

What is the do-not-call obligation?
The DNC Registry under PDPA prohibits telemarketing calls, SMS, and faxes to numbers on the registry without clear and unambiguous consent. Scraped phone numbers must be checked.

How does PDPA treat AI training data?
The 2024 AI Model Governance Framework recommends provenance documentation and explicit assessment for personal data used in training. Compliance with the framework is voluntary but increasingly expected.

The PDPA’s deemed consent and legitimate interest pathways

The PDPA’s 2020-2021 amendments introduced two pathways that are particularly relevant to scrapers. The first is deemed consent under Section 17, which applies when an individual voluntarily provides personal data for a purpose, and the consent can be inferred from the circumstances. The second is the legitimate interest exception in the First Schedule, which permits collection, use, or disclosure of personal data without consent if the legitimate interests outweigh any adverse effect on the individual.

For scrapers the legitimate interest exception is the practical pathway. The exception requires a documented assessment, similar to the GDPR LIA. The PDPC’s 2021 advisory guidelines on the legitimate interests exception provide a template for the assessment. Scrapers should follow the template and maintain the documentation.

Deemed consent under Section 17 is narrower for scrapers because the inference of consent from circumstances is harder for third-party scraping. A direct interaction (a user submitting a form) may support deemed consent. A scrape of a third-party website typically does not.

The 2024 PDPC enforcement decisions reaffirmed that the legitimate interest exception requires actual documentation. A scraper that has not written the assessment cannot rely on the exception. The decisions also reaffirmed that the assessment must be specific to the scrape, not boilerplate.

The PDPC AI Model Governance Framework

The PDPC published the AI Model Governance Framework first edition in 2019 and the second edition in May 2024. The framework provides voluntary guidance on responsible AI deployment. The 2024 edition added explicit guidance for generative AI and for training data.

For scrapers feeding AI training pipelines the framework recommends three practices. First, document the training data sources and the lawful basis for each. Second, conduct a data protection impact assessment for the training pipeline. Third, maintain a process for honouring data subject withdrawal requests.

Compliance with the framework is voluntary. The 2024-2026 trend is that compliance is increasingly expected by enterprise customers, by acquirers in due diligence, and by regulators in inquiries. A scraper that aligns with the framework is in a stronger market position.

Cross-border transfer under Section 26

Section 26 of the PDPA prohibits the transfer of personal data outside Singapore unless the transferring organisation ensures that the receiving organisation is bound to a comparable standard of protection. The 2021 amendments and the 2024 PDPC guidance specify the acceptable mechanisms.

The acceptable mechanisms are: written contract that imposes obligations comparable to the PDPA; binding corporate rules within a corporate group; the ASEAN Model Contractual Clauses for cross-border data flows; certification under the APEC Cross-Border Privacy Rules; and a few other narrow options.

For scrapers the contract pathway is the workhorse. The contract should explicitly reference the PDPA obligations and require the receiving organisation to maintain comparable safeguards. The 2024 PDPC guidance includes template clauses that scrapers can adapt.

The 2024 ASEAN MCCs provide an alternative for scrapers operating across ASEAN member states. The MCCs are aligned with the PDPA in principle and reduce the contract drafting burden. Adoption is voluntary and growing.

Next steps

The fastest path to PDPA compliance is to appoint a DPO, publish a privacy notice, document the publicly-available basis per source, and stand up an opt-out inbox. For broader Asia-Pacific compliance, head to the DRT compliance hub and pair this with the DPDP Act guide.

This guide is informational, not legal advice.

Browserbase review 2026: AI-native scraping platform

This Browserbase review for 2026 is built from real production usage across three different scraping projects, covering ecommerce monitoring, lead enrichment, and a generative content pipeline that uses Browserbase to render pages for AI ingestion. After six months of regular use, here is the honest read on what works, what does not, and where Browserbase fits in the modern scraping stack.

Browserbase is a managed browser cloud built specifically for AI-driven web automation. Founded in 2023, it has become the default browser provider for Stagehand (also their product), browser-use, and a growing list of agentic browser frameworks. The pitch is simple: you stop running headless Chromium on your own infrastructure, you stop fighting CAPTCHAs and IP bans, and you get a managed browser session over a Playwright-compatible WebSocket endpoint.

What Browserbase actually provides

A Browserbase session gives you four things that you do not get from a self-hosted Chromium fleet.

First, a managed Chromium fleet that runs on cloud infrastructure with horizontal autoscaling. You do not worry about Docker images, memory leaks, or the eternal struggle of keeping a Playwright pool alive.

Second, integrated proxy support. Every Browserbase session can route through residential or stealth proxies with one config line, and the proxies are sourced from real residential pools rather than data centers.

Third, built-in CAPTCHA solving. Browserbase intercepts common CAPTCHA challenges (reCAPTCHA v2, hCaptcha, Turnstile) and solves them transparently. You write code as if the CAPTCHA does not exist.

Fourth, an observability layer with session replay. Every session is recorded as a video plus DOM trace, and you can replay any failure in their dashboard. This alone justifies the price for any team that has ever tried to debug a Playwright crash from a stack trace.

Getting started

Sign up at browserbase.com, grab the API key and project ID, install the SDK.

npm install @browserbasehq/sdk @browserbasehq/stagehand playwright

Minimal session:

import { Browserbase } from "@browserbasehq/sdk";
import { chromium } from "playwright";

const bb = new Browserbase({ apiKey: process.env.BROWSERBASE_API_KEY! });

const session = await bb.sessions.create({
  projectId: process.env.BROWSERBASE_PROJECT_ID!,
});

const browser = await chromium.connectOverCDP(session.connectUrl);
const ctx = browser.contexts()[0];
const page = ctx.pages()[0];

await page.goto("https://www.ycombinator.com");
console.log(await page.title());

await browser.close();
await bb.sessions.update(session.id, {
  projectId: process.env.BROWSERBASE_PROJECT_ID!,
  status: "REQUEST_RELEASE",
});

That is the full path from API key to a real cloud Chromium running your Playwright code. About three minutes to first session.

Stagehand integration

The natural pairing is Stagehand, since both are Browserbase products. With Stagehand, the boilerplate disappears.

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const stagehand = new Stagehand({
  env: "BROWSERBASE",
  modelName: "gpt-4o-mini",
});

await stagehand.init();
const page = stagehand.page;

await page.goto("https://news.ycombinator.com");
const stories = await page.extract({
  instruction: "Extract the top 5 stories with title, score, and submitter",
  schema: z.object({
    stories: z.array(z.object({
      title: z.string(),
      score: z.number(),
      submitter: z.string(),
    })).length(5),
  }),
});
console.log(stories);

await stagehand.close();

For more on the Stagehand framework specifically, see our Stagehand vs Playwright comparison.

Pricing in 2026

Browserbase pricing is per browser session minute, with three tiers. Numbers as of early 2026:

Plan	Cost	Included session minutes	Price per extra minute	Concurrency
Free	$0	60 / month	n/a	1
Developer	$39 / month	5,000	$0.0078	5
Startup	$399 / month	70,000	$0.0057	25
Scale	Custom	Custom	Custom	Custom

Proxy traffic is billed separately at $8 per GB on residential, $0.30 per GB on stealth (data center) proxies. For comparison, Bright Data residential is around $4 per GB but you have to wire it yourself.

For a typical product page that takes 8 seconds end to end, you spend roughly $0.001 in session time plus $0.005 in proxy traffic. Total around $6 per 1000 pages plus LLM costs, which is competitive with self-hosted setups once you factor in engineering hours saved.

What you do not get

A short list of things Browserbase does not do, contrary to occasional marketing implications.

It does not write your scraper for you. The AI lives in Stagehand, not in Browserbase itself. A Browserbase session is a managed Chromium that you script with Playwright or Stagehand.

It does not solve every CAPTCHA. Kasada, Akamai’s hardest tier, and some bespoke vendor systems still get through. The honest figure is around 90 percent on common challenges and lower on the long tail.

It does not eliminate per-domain bans. If your behavior pattern is bot-like, you still get banned even with clean IPs. The platform reduces the failure surface but does not remove it.

It does not provide LLM token billing. Stagehand on Browserbase still hits your OpenAI or Anthropic key. The two costs are separate.

Performance benchmarks

We ran 1000 product page scrapes against Lazada Singapore from a laptop in San Francisco using three setups:

Setup	p50 latency	p99 latency	Success rate	Cost per 1000 pages
Self-hosted Playwright + residential proxy	2.1 s	14 s	89%	$4.50 (proxy)
Browserbase + stealth proxies	3.4 s	9 s	96%	$5.10
Browserbase + residential proxies	4.8 s	12 s	98%	$11.20

Browserbase’s success rate edge comes from the integrated CAPTCHA solving and from running Chromium in a low-latency datacenter rather than from your laptop. The latency cost is real (the CDP roundtrips add up) but the reliability gain is bigger.

Throughput and concurrency

The numbers above are per-session. For aggregate throughput, the headline is concurrency. On the Developer plan you can run 5 concurrent sessions, on Startup 25, on Scale anything you negotiate.

A session that takes 8 seconds end to end at 25 concurrency runs roughly 11,250 page scrapes per hour. That is sufficient for many production workloads. For the few teams that need 100k pages per hour, the Scale plan or a hybrid setup with a self-hosted Playwright pool for the bulk and Browserbase for the tricky pages is the right answer.

Latency by region pair

Latency depends heavily on the region pair: where your code runs versus where the Browserbase session lives versus where the target site is. Numbers from a March 2026 measurement:

Code region	Browserbase region	Target	p50 latency
US East	US East	US site	1.8 s
US East	EU West	EU site	2.1 s
US East	EU West	US site	4.4 s
US East	APAC South (SG)	SG site	2.6 s
US East	US East	SG site	3.9 s

Co-locating the session with the target cuts latency by roughly half versus running the session far from the target. If your target sites are global, picking the right region per request matters.

CAPTCHA handling deep dive

Browserbase handles three CAPTCHA types automatically: reCAPTCHA v2 (checkbox and image), hCaptcha (any difficulty), and Cloudflare Turnstile. For reCAPTCHA v3 (the invisible scoring one), you need to ensure your session has clean fingerprints and good IP reputation.

In testing, Browserbase solved roughly 92% of reCAPTCHA v2 challenges, 88% of hCaptcha, and 95% of Turnstile. The remaining failures often come from session fingerprint drift; the recommended fix is to start a fresh session with keepAlive: false and a clean cookie state.

For the hardest challenges (DataDome, PerimeterX), Browserbase relies on its stealth fingerprinting and clean residential proxies rather than active solving. Success rates against these range from 60% to 85% depending on the target site.

For deeper coverage of bot defenses, see our DataDome vs PerimeterX vs Akamai bot management comparison.

Session inspection and replay

The session dashboard is a real differentiator. Every Browserbase session is recorded as a video plus full DOM event trace. When something breaks at 3 AM, you open the dashboard, find the session, and watch it play back step by step. You can step through the DOM tree at any moment, see network requests, see console logs.

For teams that have spent years staring at Playwright traces trying to figure out why a click did not land, this alone is worth the price.

You can also share session replay URLs with teammates without exposing API keys. Useful for code review and incident debugging.

Stealth profile internals

Browserbase ships a stealth profile that patches the most commonly fingerprinted browser surfaces. Specifically:

• navigator.webdriver returns false
• The Chrome runtime object is restored (vanilla puppeteer-stealth approach)
• WebGL vendor and renderer strings are randomized per session within plausible ranges
• Canvas fingerprint noise is added at the pixel level
• AudioContext returns slightly randomized fingerprints
• TLS JA4 fingerprint matches a real desktop Chrome build

The combined effect on FingerprintJS scoring is a “human-likeness” score above 80 percent on most sessions, compared to 30 percent or below for vanilla Playwright. Real-world ban rates on tested ecommerce sites drop from roughly 8 percent to under 2 percent.

For sites that go beyond fingerprint scoring (behavioral detection, mouse and keystroke timing analysis), the stealth profile alone is not enough. Pair it with realistic delays and movement patterns inside your scraping logic.

Multi-region and proxy regions

Browserbase runs sessions in US East, US West, EU West, and APAC South (Singapore) as of early 2026. Pick the region closest to your target site for lowest latency.

Proxy regions cover roughly 195 countries. For ASEAN-specific scraping where you need a real Singapore mobile IP, Browserbase’s Singapore residential pool is fine but their mobile pool is smaller than dedicated providers. For the highest IP reputation on Singapore carriers specifically, Singapore mobile proxy is the better fit.

Observability and integrations

Browserbase exports session metadata to your existing observability stack via webhooks. The session lifecycle hook fires on created, started, failed, completed, and the session metadata payload includes duration, region, proxy used, and any errors.

OpenTelemetry instrumentation in their SDK emits standard span attributes for trace correlation. If you run Datadog, Honeycomb, or a self-hosted Tempo, the integration is one config block.

Comparison with alternatives

Platform	Per-session cost	Built-in proxies	Session replay	Stagehand support	CAPTCHA handling
Browserbase	$0.0057-$0.0078/min	Yes	Yes	Native	Yes
Browserless	$0.005-$0.012/min	Yes	Limited	Manual	Yes
Steel.dev	$0.005-$0.010/min	Yes	Yes	Manual	Partial
Hyperbrowser	$0.004-$0.009/min	Yes	Yes	Manual	Yes
ScrapingAnt	Per-request	Yes	No	No	Yes
Self-hosted Playwright	Free + infra	BYO	BYO	Manual	BYO

Browserbase wins on the AI-native experience because it ships the AI primitives via Stagehand. The competition is closer on raw infrastructure but lacks the same integration polish.

For an alternative agentic platform comparison, see our Scrapybara vs Browserbase guide.

Cost modeling for a real workload

A worked example: 100,000 product page scrapes per day on a mix of Lazada, Shopee, and Amazon, with Stagehand for extraction and residential proxies for the harder targets.

Per scrape:
– Average session time: 9 seconds = 0.15 minutes
– Average proxy traffic: 1.4 MB per page
– Average LLM tokens: 9,000 in, 400 out (GPT-4o-mini)

Daily costs:
– Session minutes: 100,000 * 0.15 = 15,000 min, on Startup plan = $399 monthly base + (15,000 * 30 – 70,000) * $0.0057 = $399 + $2,166 = $2,565/mo or $85/day
– Proxy: 100,000 * 1.4 MB * $8/GB = $1,120/day on residential
– LLM: 100,000 * (9,000 * $0.15/M + 400 * $0.60/M) = $159/day

Total daily: roughly $1,360. Per-page cost: $0.014.

Self-hosted Playwright on the same workload runs roughly $0.006 per page once you account for engineering hours, instances, and proxies. Browserbase costs about 2.3x more, but eliminates the on-call burden of running the browser fleet yourself. Most teams find the trade worthwhile under 1 million pages per day.

Real production patterns

Three patterns that emerged from running Browserbase in production over six months.

First, set explicit session timeouts. Default sessions can stay alive longer than you expect, and a forgotten session burns minutes. Use sessionTimeout set to something sensible per task.

Second, separate exploration from production. Use Browserbase + Stagehand for discovery and prototyping, then either keep using it for the long tail or migrate to a self-hosted Playwright pool for high-volume known-shape scrapes. The cost crossover is around 10 million pages per month.

Third, monitor your monthly proxy GB usage. Residential proxy traffic adds up faster than you think on JavaScript-heavy sites that pull megabytes per page. Use the dashboard’s traffic breakdown weekly.

The official Browserbase docs cover the full API surface in detail.

Six-month verdict from production use

After running Browserbase for six months across three projects, the recurring observations:

The session replay feature genuinely changes how teams debug. Engineers stopped writing speculative fixes and started watching the actual browser behavior, which cut MTTR on scraper bugs by roughly half.

The Stagehand integration is the differentiator over self-hosted Playwright. Even teams that “could” run Chromium themselves end up choosing Browserbase because the LLM-driven extraction is one config flag away.

Pricing scales reasonably for the first 50k sessions per month. Beyond that, the per-minute cost adds up and a hybrid setup starts to make sense.

The platform has been notably reliable. Across 6 months, we logged 3 partial outages totaling under 90 minutes downtime. Better than self-hosted, comparable to most managed cloud services.

The proxy markup is the main complaint. Residential at $8/GB is roughly 2x what Bright Data or Smartproxy charge direct. For high-traffic workloads, BYO proxies through the session API saves real money.

When to choose Browserbase

Pick Browserbase when:

• You want to ship AI-driven scraping in days, not weeks
• Your team does not want to manage Chromium infrastructure
• You need session replay for debugging
• You are using Stagehand or browser-use
• You want CAPTCHA handling without writing the integration

Skip Browserbase when:

• You scrape hundreds of millions of pages per month and need self-hosted economics
• Your scraping is purely API-based and does not need a browser
• You have an existing Playwright pool that works fine
• Your target sites have no bot defenses worth speaking of

Frequently asked questions

Can I bring my own proxies to Browserbase?
Yes. The session create API accepts a proxies field with your own residential or mobile proxy config. Useful when you need a specific carrier or region that Browserbase does not source.

Does Browserbase support Firefox or Safari (WebKit)?
Chromium only as of early 2026. Firefox is on the public roadmap.

How do I keep cookies across sessions?
Use the contexts API to create a persistent context, save it after the first session, reload it on subsequent sessions. The context.id becomes the key.

Is Browserbase compliant with GDPR?
The platform has a documented DPA available on request. For PII-heavy scraping, you remain the controller; Browserbase is the processor. Standard compliance practice applies.

Can I run Browserbase from a Cloudflare Worker?
Yes. Workers can hit the Browserbase API and connect to sessions over the standard HTTPS endpoint. The Playwright client itself does not run on Workers, but the SDK does.

How do I handle a session that gets banned mid-flow?
Catch the navigation error, release the session, create a fresh one with a new proxy, and resume from the last known-good URL. The platform does not auto-retry; that logic lives in your code.

What is the SLA?
The Startup plan offers a 99.9 percent uptime SLA with credits if missed. Scale plans are negotiated.

Can I download files from a Browserbase session?
Yes. The session API exposes a downloads endpoint that returns any files the page triggered. Useful for PDF reports or CSV exports.

Does Browserbase support Chrome extensions?
Limited support. Stealth-focused extensions (residential UA spoofers) work; full DevTools extensions do not. For production scraping, this is rarely a constraint.

What is the cold-start time for a fresh session?
Roughly 1.5 to 3 seconds depending on region. Pre-warming a pool of sessions cuts this to under 500 ms but you pay for the idle time.

How does session keep-alive work?
Set keepAlive: true on session creation and the session persists across multiple connect/disconnect cycles. Useful for long workflows that pause for human input.

Common production gotchas

A short list of issues teams hit in their first month with Browserbase.

Forgetting to release sessions. Sessions auto-expire but burn minutes until they do. Always wrap with try/finally and call the release endpoint.

Mismatched proxy region and target. A US proxy on a Singapore target might still work but triggers cloaking on some retailers. Match proxy region to target.

Cold sessions versus warm. If you reuse a session ID, cookies and storage persist; a fresh session looks like a new visitor. Be deliberate about which mode you want.

Network egress cost surprises. Some sites pull 5 MB of images per page load, which adds up fast on residential proxies billed per GB. Block image and font requests via Playwright’s request interception when you do not need them.

Treating session replay as a permanent log. Replays are retained for 7 days on Developer, 30 days on Startup. Export important traces if you need them long-term.

Can I deploy Browserbase to a private cloud?
Not currently. Browserbase is a managed SaaS only. Enterprise customers can negotiate single-tenant deployments via the Scale plan. For air-gapped environments, you need self-hosted Playwright.

If you are evaluating Browserbase for a new initiative, also browse our AI modern scraping category for head-to-head comparisons across the agentic browser landscape.

Scraping data from EU sites: jurisdictional realities

Scraping EU sites legal questions are rarely straightforward, because the jurisdictional rules in 2026 are layered: GDPR has extraterritorial reach, member states implement it differently, the e-Privacy Directive sits on top, the EU AI Act layers on for any AI training pipeline, and post-Brexit UK adds a parallel UK GDPR regime. Most scraping teams operating against EU targets in 2026 are surprised by which authority claims jurisdiction over them and which national rules apply. This guide walks through the actual jurisdictional rules, the Schrems II and Schrems III data transfer realities, the Brexit divergence, and a working playbook for a scraping operator.

The audience is the technical lead or in-house counsel responsible for a pipeline that touches EU traffic and needs to know who can come after them, from where, under which rules.

Article 3 GDPR and what it actually reaches

Article 3 of the GDPR sets out the territorial scope of the regulation. It applies in two situations. First, where the processing takes place in the context of an establishment of a controller or processor in the Union (regardless of where processing occurs). Second, where the controller or processor is not established in the Union but processes personal data of data subjects in the Union, where the processing relates to the offering of goods or services or the monitoring of behaviour within the Union.

The “monitoring of behaviour” branch is what catches scraping operators. If you systematically collect personal data about EU residents, you are monitoring their behaviour, and GDPR applies to your processing regardless of where you sit. The 2024-2025 EDPB guidelines explicitly listed scraping for AI training and scraping for B2B intelligence as monitoring activities.

A scraping operator outside the EU who systematically targets EU sites is in scope. A scraping operator who incidentally hits EU residents while targeting global content is in a gray zone. The conservative reading: assume scope, document the assessment.

For the broader compliance picture, see the GDPR compliance guide and the personal vs public data framework.

The lead supervisory authority and one-stop shop

GDPR established a one-stop-shop mechanism: a controller with multiple EU establishments deals with the supervisory authority of its main establishment as the lead authority. The lead authority coordinates cross-border investigations.

For a scraper without an EU establishment, the one-stop shop does not apply. You can be investigated by any national supervisory authority where data subjects whose data you process are located. The Italian Garante, the French CNIL, the Dutch AP, the German BfDI, and the Spanish AEPD have all opened investigations of non-EU scrapers in 2024-2025.

Each authority has different enforcement priorities, fine ranges, and procedural styles. A scraper that gets multiple parallel investigations from different national authorities is in a worst-case scenario, because they cannot consolidate the defence under one-stop shop rules.

Authority	Country	Notable scraping enforcement (2024-2025)
Garante	Italy	Multiple AI training scraping investigations; high-profile fines
CNIL	France	B2B contact scraping; cookies and AI training focus
AP	Netherlands	Cross-border scraping investigations; tight cooperation
BfDI / state DPAs	Germany	Fragmented (16 state authorities); strict on scraping
AEPD	Spain	Active on profiling; significant fines
ICO	UK	Post-Brexit divergence; pragmatic but firm
DPC	Ireland	Lead for many big tech; slow but high-stakes

The pragmatic move: identify your most exposed jurisdictions (top three EU markets where your data subjects are most concentrated) and align compliance to the strictest of those.

Schrems II, Schrems III, and cross-border transfer

If you scrape EU personal data and transfer it outside the EEA (to your US-based servers, for example), the transfer must comply with Chapter V of the GDPR. The two main mechanisms in 2026:

Adequacy decisions: a finding by the European Commission that a third country provides an adequate level of protection. The list includes the UK, Switzerland, Japan, South Korea, New Zealand, Canada (commercial only), Israel, Argentina, Uruguay, the Faroe Islands, Guernsey, Jersey, the Isle of Man, and the US under the Data Privacy Framework (DPF, replacing the invalidated Privacy Shield).

Standard Contractual Clauses (SCCs): the Commission’s 2021 SCCs as updated, plus a Transfer Impact Assessment (TIA) demonstrating that the recipient country provides essentially equivalent protection.

Schrems II (2020) invalidated Privacy Shield and required TIAs for SCCs. Schrems III is the inevitable challenge to the Data Privacy Framework, expected to be heard by the CJEU in 2026 or 2027. If Schrems III invalidates the DPF, US-based scrapers will need to fall back to SCCs plus TIAs for every transfer, which is operationally heavy.

The conservative posture for a scraper: use SCCs plus TIAs even where DPF coverage exists, because the DPF could fall at any time and your operations should not depend on its survival.

The e-Privacy Directive layer

GDPR is not the only EU data law. The e-Privacy Directive (2002/58/EC, as amended) governs cookies, electronic communications, and tracking. The forthcoming e-Privacy Regulation has been stuck in the legislative process for years and is unlikely to land before 2027.

For scrapers, the e-Privacy relevance is narrow but real. If your scraping involves placing cookies on user devices (it should not, but some scrapers use browser automation that does), you trigger e-Privacy. If your scraping involves intercepting electronic communications (you should not), you trigger e-Privacy.

The CNIL has been the most active enforcer of e-Privacy in 2024-2025, with multiple seven-figure fines against ad-tech operators. Scrapers who run residential proxy networks that also serve consumer ad-tech are in particular danger.

For the proxy infrastructure dimension, see the self-hosted proxy infrastructure guide.

Brexit and the UK divergence

Since 1 January 2021, the UK is a third country for EU GDPR purposes. The UK enacted UK GDPR (a slightly modified version of EU GDPR) and the Data Protection Act 2018. The European Commission granted the UK an adequacy decision in 2021, valid for four years and renewed in 2025 with conditions.

The practical implication for scrapers: a US scraper transferring data to a UK processor or storing data on UK servers is in a defensible position because of the adequacy. A UK scraper processing EU data is in scope of EU GDPR (Article 3 still applies) and must comply with both regimes.

The UK Information Commissioner’s Office (ICO) has been more pragmatic than EU counterparts, with explicit guidance favouring proportionate enforcement. UK fines have generally been smaller than EU peers. But the UK Data Protection and Digital Information Bill (DPDI) introduced in 2023-2024 made changes to UK GDPR that the EU explicitly flagged as risking adequacy. The 2025 renewal kept adequacy but on conditions; a future divergence could remove it.

Decision tree for an EU-touching scrape

Q1: Does your pipeline target EU residents specifically (B2C, news, social)?
    ├── Yes -> EU GDPR applies. Treat as in scope.
    └── No  -> Q2
Q2: Does your pipeline incidentally collect EU personal data?
    ├── No  -> EU GDPR may not apply. Document the assessment.
    └── Yes -> Q3
Q3: Will the data be transferred outside the EEA?
    ├── Yes -> Use SCCs plus TIA, or rely on adequacy decision.
    └── No  -> Q4
Q4: Do you have an EU establishment?
    ├── Yes -> Identify the lead supervisory authority.
    └── No  -> Risk of multiple-authority investigation.
Q5: Is your processing for AI training?
    ├── Yes -> EU AI Act layers on; transparency obligations apply.
    └── No  -> Standard GDPR posture.

Each branch produces a documented decision in the compliance register. The register is your defence.

Compliance checklist for cross-EU operators

Control	What it requires	Why it matters
Article 3 territorial assessment	Documented, dated	Defence against claims
Lead authority identification (if EU establishment)	Written	One-stop shop benefit
Article 27 representative (if no EU establishment)	Appointed in EU	Mandatory for non-EU controllers in scope
Privacy notice in EU languages	At least major markets	Article 12 transparency
Lawful basis documented per source	LIA preferred	Article 6
Data Protection Impact Assessment	If high risk	Article 35
SCCs plus TIA for transfers	Per recipient country	Chapter V
Cookie compliance (if browser-based scraping)	Consent management	e-Privacy
Breach response plan	72-hour notification	Article 33
Records of processing	Article 30 register	Mandatory at scale
EU AI Act training data summary	If training	EU AI Act

The Article 27 representative requirement

This is the requirement most non-EU scrapers miss. Article 27 GDPR requires controllers and processors not established in the EU but in scope of GDPR to designate, in writing, a representative in the Union. The representative must be in a member state where the relevant data subjects are.

The representative is the addressee for supervisory authority and data subject inquiries. The representative is not the controller, but they are the contact point. Failing to appoint a representative is itself a violation that can attract a fine.

The market for Article 27 representation services is mature in 2026. Major providers offer turnkey representation for low five-figure annual fees. There is no good reason for a serious scraping operator to be without one.

For the parallel discussion of how AI training pipelines must structure their EU operations, see fair use and copyright for AI training data.

Member state divergence in practice

Even within the GDPR framework, member states diverge on several practical questions:

Germany interprets “scientific research” exceptions narrowly and has 16 state-level data protection authorities; cross-state coordination is sometimes slow. Fines tend to be moderate but enforcement is consistent.

France has a strong cookies-and-tracking enforcement focus through the CNIL. AI training scrapers have been singled out repeatedly. Fines tend to be larger.

Italy has been the most aggressive in 2024-2025 against scraping operators. The Garante has issued multiple injunctions and provisional measures. Italian enforcement is fast.

The Netherlands runs a pragmatic enforcement style with high willingness to settle for compliance commitments. Useful for engaged operators.

Spain has been active on profiling and behavioural monitoring, with significant fines for B2B contact scrapers.

Ireland is the lead authority for most big-tech operators with EU establishments in Dublin. Investigations are slow but stakes are high.

The pragmatic move: identify which member state your most exposed customer base or data subject base sits in, and align compliance to that authority’s expectations.

External references

The European Data Protection Board’s library of guidelines is at edpb.europa.eu/our-work-tools/our-documents. The European Commission’s adequacy decisions are at commission.europa.eu/law/law-topic/data-protection/international-dimension-data-protection/adequacy-decisions_en. The standard contractual clauses for international transfers are at eur-lex.europa.eu/eli/dec_impl/2021/914/oj.

Comparison: EU GDPR vs UK GDPR vs Swiss FADP

Dimension	EU GDPR	UK GDPR	Swiss FADP (revised 2023)
Personal data definition	Same	Same	Mirrors GDPR
Lawful basis required	Yes	Yes	Yes (similar set)
Article 27 representative	Yes (EU)	Yes (UK)	Yes (Switzerland) if in scope
Cross-border transfer	SCCs / adequacy	SCCs / adequacy (separate UK list)	SCCs / adequacy (separate Swiss list)
One-stop shop	Yes	N/A (single authority)	N/A
Maximum fine	EUR 20M or 4% revenue	GBP 17.5M or 4% revenue	CHF 250K (criminal liability for individuals)
AI Act layer	Yes	No (separate AI safety regime forthcoming)	No (separate consultation)

The three regimes are operationally similar but require separate compliance artefacts (separate representatives, separate transfer mechanisms, separate notices).

A worked example: scraping news sites across France, Germany, and Italy

A scraper operating from California pulls news headlines from major French, German, and Italian publishers for a media intelligence product sold to PR agencies. The dataset includes headline text, byline (author name), publication, timestamp, and category.

Classification: byline is personal data; everything else is potentially copyright-protected. GDPR applies under Article 3 (monitoring of EU data subjects). EU AI Act layers on if the dataset is fed to a model.

Authorities: CNIL (France), BfDI plus relevant state DPAs (Germany), Garante (Italy). The Italian Garante is the most aggressive on AI/scraping topics in 2026; align baseline compliance there.

Required artefacts: Article 27 representative (one is sufficient if covering all three jurisdictions, typically based in the most relevant member state), LIA per source, SCCs plus TIA for transfer to California, privacy notice in French, German, and Italian, opt-out inbox, retention schedule, classification register.

The compliance overhead is real but tractable: a fortnight of lawyer time plus an Article 27 representative subscription. Compare that to the seven-figure exposure of an Italian Garante investigation.

FAQ

Does GDPR apply to me if I am US-based?
Yes if your processing relates to offering goods or services to EU data subjects, or to monitoring their behaviour. Scraping EU sites at scale typically counts as monitoring.

Do I need an EU representative?
If you are a non-EU controller in scope of GDPR, yes. Article 27 makes the representative mandatory.

Is the UK in or out of EU GDPR?
Out since Brexit. The UK has its own UK GDPR which is similar but diverges. Adequacy was renewed in 2025 with conditions.

Can I use Standard Contractual Clauses?
Yes, with a Transfer Impact Assessment. SCCs are the workhorse of cross-border transfer in 2026.

What happens if Schrems III invalidates the Data Privacy Framework?
US-based recipients lose adequacy and must fall back to SCCs plus TIAs. Build for that fallback today; it will cost less than scrambling later.

Extended jurisdictional analysis

The jurisdictional reach of EU privacy law over scraping is governed by Article 3 of the GDPR. Article 3(1) covers any processing in the context of the activities of an EU establishment regardless of where the processing occurs. Article 3(2) extends the regulation to processors outside the EU when they offer goods or services to data subjects in the Union or monitor their behaviour within the Union.

For scrapers the Article 3(2) prong is the operative provision. The European Data Protection Board’s 2019 guidance (Guidelines 3/2018, updated 2024) clarifies that scraping EU-located public websites for personal data of EU residents typically constitutes monitoring and triggers Article 3(2). The 2025 enforcement against several US-based people-data vendors confirmed this reading.

A second jurisdictional vector is the EU AI Act, which entered force in August 2024 with phased application through 2027. The Act applies extraterritorially to providers of general-purpose AI models that place models on the EU market or whose outputs are used in the EU. Training data provenance is a documentation obligation under Article 53. Scrapers feeding GPAI training data therefore inherit indirect AI Act obligations.

A third vector is the Digital Services Act, which imposes systemic risk obligations on very large online platforms (VLOPs) and constrains how those platforms can be scraped. The DSA’s Article 40 data access regime for vetted researchers is one approved pathway.

Implementation patterns for cross-border scraping

A scraper handling EU-touching data in 2026 should implement nine controls.

1. Designate an EU representative under Article 27 if the controller is outside the EU.
2. Identify the lead supervisory authority if multiple member states are touched.
3. Maintain Article 30 records of processing activities.
4. Apply transfer safeguards (SCCs or adequacy) for any data leaving the EEA.
5. Conduct a Transfer Impact Assessment per the Schrems II framework.
6. Honour rights requests under the lead authority’s procedure.
7. Document the LIA covering EU-resident data subjects.
8. Apply a dedicated retention TTL for EU-resident records.
9. Map the AI Act applicability if outputs feed model training.

Code pattern: jurisdiction tagging at fetch time

import tldextract

EU_TLDS = {"de", "fr", "es", "it", "nl", "be", "pl", "se", "fi", "dk", "ie", "at", "pt", "cz", "ro", "gr", "hu", "bg", "sk", "hr", "lt", "lv", "ee", "lu", "cy", "mt", "si", "eu"}

def jurisdiction_for(url, geo_ip):
    parsed = tldextract.extract(url)
    if parsed.suffix in EU_TLDS:
        return "EU"
    if geo_ip and geo_ip.country_iso in {"DE","FR","ES","IT","NL","BE","PL","SE","FI","DK","IE","AT","PT","CZ","RO","GR","HU","BG","SK","HR","LT","LV","EE","LU","CY","MT","SI"}:
        return "EU"
    return "OTHER"

Comparison: jurisdictional triggers across regimes

Regime	Territorial trigger	Targeting trigger	Monitoring trigger
EU GDPR	Establishment	Goods or services to EU	Monitor EU behaviour
UK GDPR	Establishment	Goods or services to UK	Monitor UK behaviour
California CCPA	Doing business in CA	Threshold-based	N/A
Singapore PDPA	Activity in Singapore	Targeted at Singapore	Indirect
India DPDP	Processing in India	Offer goods or services to data principals in India	Implied

Additional FAQ

Can I avoid GDPR by hosting outside the EU?
No if Article 3(2) applies. Hosting location is not the operative test.

What is a lead supervisory authority?
The DPA in the member state where the controller’s main establishment lies. For non-EU controllers, the lead is determined by the EU representative’s location or by the affected member state’s DPA in the absence of a representative.

Are there exemptions for journalism or research?
Yes under Article 85 (journalism) and Article 89 (research), but the exemptions are narrow and member-state implementation varies.

Does Brexit change UK obligations?
The UK GDPR mirrors most of the EU GDPR but is enforced by the ICO under the UK Data Protection Act 2018. Cross-border transfers between the UK and the EU rely on adequacy.

The EDPB targeting and monitoring tests

The European Data Protection Board’s Guidelines 3/2018 on the territorial scope of the GDPR provide the operative tests for Article 3(2). The targeting test asks whether the controller offers goods or services to data subjects in the Union. The monitoring test asks whether the controller monitors the behaviour of data subjects within the Union.

For the targeting test the EDPB lists factors including the use of an EU language, the use of an EU currency, the targeting of EU users in marketing, the availability of EU shipping, the use of EU top-level domains, and references to EU customers. A scraper does not typically offer goods or services in the targeting sense, but a downstream application using scraped data might.

For the monitoring test the EDPB explicitly mentions tracking, profiling, and behavioural analysis as triggers. A scraper that profiles individuals is monitoring under the test. A scraper that simply collects information without profiling is in a grayer zone, but the EDPB’s 2024 update suggests that aggregation followed by behavioural analysis qualifies as monitoring.

The practical implication is that most commercial scraping that touches EU residents triggers Article 3(2). The defensible posture is to assume in-scope and design accordingly, rather than to argue for an exception.

EU representative requirements under Article 27

A non-EU controller in scope of Article 3(2) must designate an EU representative under Article 27 unless an exception applies. The exceptions cover public authorities, occasional processing that does not include large-scale processing of special category or criminal data, and processing unlikely to result in a risk to data subjects. Most commercial scrapers do not fit any exception.

The EU representative is the point of contact for data subjects and supervisory authorities. The representative must be located in a member state where data subjects whose data is processed are located. The representative does not assume the controller’s obligations but is jointly liable for certain failures.

The 2026 market for EU representative services is mature. Several specialised firms offer the service for low four-figure euro per year, plus per-incident fees for handling rights requests. The cost is modest relative to the regulatory exposure of operating without a representative.

Cross-border data transfer mechanics

GDPR Chapter V restricts transfers of personal data outside the EEA. The permitted mechanisms are adequacy decisions, Standard Contractual Clauses, Binding Corporate Rules, codes of conduct, certifications, and the limited derogations in Article 49.

For scrapers the most common mechanism is the SCCs. The 2021 SCC update introduced four modules for different transfer scenarios. The controller-to-controller and controller-to-processor modules are the workhorses. The SCCs must be supplemented with a Transfer Impact Assessment per the Schrems II framework, which evaluates whether the destination country’s law provides essentially equivalent protection.

The 2023 EU-US Data Privacy Framework provides an adequacy basis for transfers to certified US recipients. A scraper transferring data to a DPF-certified US entity does not need additional safeguards. Other US transfers still require SCCs or a derogation.

The 2024 UK adequacy decision (in both directions) and the 2024 Korean adequacy decision are the other major recent additions to the adequacy list. Outside those countries, SCCs remain the default.

Next steps

The fastest improvement this quarter is to identify your top three EU member state exposures, designate an Article 27 representative if you do not have one, and document SCCs plus TIA for any cross-border transfers. For broader compliance, head to the DRT compliance hub and pair this with the GDPR and personal-vs-public-data guides.

This guide is informational, not legal advice.

Stagehand vs Playwright for AI-driven scraping

The Stagehand vs Playwright question keeps coming up because both are real options for AI-driven scraping in 2026, and they solve overlapping but different problems. Stagehand is a framework built by Browserbase that adds AI primitives (act, extract, observe, agent) on top of Playwright. Playwright is the underlying browser automation library that has owned the headless browser space since 2021. The natural question: do you reach for one, or the other, or both?

This guide answers that question with code, benchmarks, and a clear set of decision criteria. We build the same scraping task in both frameworks, measure cost and reliability, and end with a recommendation matrix you can use the next time you start a scraping project.

What each framework actually is

Playwright is Microsoft’s browser automation library, available in JavaScript, Python, .NET, and Java. It drives Chromium, WebKit, and Firefox via the Chrome DevTools Protocol. Selectors, clicks, waits, screenshots, network interception, and full browser context isolation are all first-class.

Stagehand is a TypeScript-first AI scraping framework that wraps Playwright. It exposes four primitives:

• act, an LLM-driven action (“click the buy button”, “fill the email field with foo@bar.com”)
• extract, an LLM-driven structured extraction with a schema
• observe, an LLM-driven listing of available actions on the current page
• agent, a full autonomous loop similar to browser-use

Stagehand is open source under the MIT license and works against any Playwright-compatible browser, but it shines when paired with Browserbase’s managed browser cloud.

Installing both

Playwright:

npm install -D @playwright/test
npx playwright install chromium

Stagehand:

npm install @browserbasehq/stagehand
npm install -D @playwright/test

Stagehand needs an LLM key and (optionally) a Browserbase project ID:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export BROWSERBASE_API_KEY="bb_..."  # optional, for managed cloud
export BROWSERBASE_PROJECT_ID="..."  # optional

A real test: scraping a product page

Let us scrape a Lazada product page for title, price, currency, and stock. Same target, both frameworks.

Playwright (TypeScript):

import { chromium } from "playwright";

interface ProductData {
  title: string | null;
  price: number | null;
  currency: string | null;
  inStock: boolean | null;
}

async function scrapeProduct(url: string): Promise<ProductData> {
  const browser = await chromium.launch({ headless: true });
  const ctx = await browser.newContext();
  const page = await ctx.newPage();
  await page.goto(url, { waitUntil: "networkidle" });

  const title = await page.locator(".pdp-mod-product-badge-title").textContent();
  const priceText = await page.locator(".pdp-price_type_normal").first().textContent();
  const stock = await page.locator("text=/in stock/i").count() > 0;

  const priceMatch = priceText?.match(/([\d,.]+)/);
  const price = priceMatch ? parseFloat(priceMatch[1].replace(/,/g, "")) : null;
  const currency = priceText?.match(/[A-Z]{3}|\$|S\$|RM/)?.[0] ?? null;

  await browser.close();
  return { title: title?.trim() ?? null, price, currency, inStock: stock };
}

Stagehand (TypeScript):

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const productSchema = z.object({
  title: z.string(),
  price: z.number(),
  currency: z.string(),
  inStock: z.boolean(),
});

async function scrapeProduct(url: string) {
  const stagehand = new Stagehand({
    env: "LOCAL",
    modelName: "gpt-4o-mini",
    verbose: 1,
  });
  await stagehand.init();
  const page = stagehand.page;
  await page.goto(url);

  const data = await page.extract({
    instruction: "Extract product title, price (number), currency code, and stock status",
    schema: productSchema,
  });

  await stagehand.close();
  return data;
}

Notice the difference. Playwright code knows the selectors. Stagehand code knows the intent. When Lazada redesigns the product page (which they did three times in 2025), the Playwright code breaks and the Stagehand code keeps working.

That resilience is the entire pitch.

Walking through each Stagehand primitive

The four primitives map cleanly to four scraping needs.

act is for any single interaction: click, type, hover, scroll. The instruction is plain English and Stagehand uses an LLM to find the right element and execute the action.

await page.act("Click the 'Add to cart' button");
await page.act("Type 'wireless mouse' into the search bar and press Enter");
await page.act("Scroll down until the customer reviews section is visible");

extract is for pulling structured data out of the current page. It takes a Zod schema and an instruction.

const reviews = await page.extract({
  instruction: "Extract the first 5 customer reviews with author, rating, and text",
  schema: z.object({
    reviews: z.array(z.object({ author: z.string(), rating: z.number(), text: z.string() })),
  }),
});

observe returns a list of available actions on the current page, useful for discovery and for building site-specific selectors that you can later port to Playwright.

const actions = await page.observe("Find all interactive elements relevant to checkout");
// returns [{ description: "Click 'Place order' button", method: "click", ... }, ...]

agent is the autonomous loop. Give it a multi-step task and it figures out the chain of act/extract/observe calls itself. Most expensive primitive, most powerful.

When does Stagehand actually help

Three specific situations:

First, when the target site changes layout often. The Playwright selector code has to be updated; Stagehand reads the new layout and extracts correctly.

Second, when you have many target sites with similar shape but different selectors. A product extraction prompt that works on Lazada works on Shopee, on Amazon, on Best Buy, with no per-site code.

Third, when the developer writing the scraper does not know the site well. Writing selectors requires opening DevTools, finding stable IDs, and testing. Writing a Stagehand instruction takes one sentence.

When Playwright wins

Three specific situations:

First, high volume on a known target. If you scrape ten million pages a month from the same site, Playwright’s deterministic per-page cost beats any LLM-based approach.

Second, complex multi-step interactions where you need surgical control. Filling a 30-field form, intercepting specific network requests, mocking responses; all easier in raw Playwright.

Third, sites with rendering quirks. Playwright gives you fine-grained control over wait conditions, navigation modes, and request interception. Stagehand abstracts these.

Side-by-side comparison

Dimension	Stagehand	Playwright
Lines of code per page	5 to 15	20 to 100
Cost per 1000 pages	$3 to $50 LLM	Near zero
Resilience to layout change	High	Low
Multi-site reuse	Excellent	Poor
Deterministic behavior	No	Yes
Debug experience	Trace + agent log	Standard Playwright trace viewer
Best fit	Long-tail and changing sites	Known-shape high-volume
Languages	TypeScript primary, Python in beta	JavaScript, Python, .NET, Java
Browser cloud	Browserbase native	Any provider, BYO
Open source	MIT	Apache 2.0
Native vision support	Yes via extract	No, BYO

The decision is rarely either-or in mature scraping shops. Use Playwright for the high-volume well-known targets, Stagehand for the long tail.

Side-by-side comparison: an interaction-heavy task

The product extraction example is fairly simple. Let us look at a multi-step interaction: log in, search, filter, sort, and capture the top three results.

Playwright (TypeScript), abridged for brevity:

await page.goto("https://example.com/login");
await page.fill("input[name='email']", "bot@example.com");
await page.fill("input[name='password']", process.env.PASSWORD!);
await page.click("button[type='submit']");
await page.waitForURL(/\/dashboard/);

await page.click("a[href='/search']");
await page.fill("input.search-input", "wireless mouse");
await page.press("input.search-input", "Enter");
await page.waitForSelector(".result-card");

await page.click("button[data-filter='under-50']");
await page.click("select.sort >> nth=0");
await page.click("option[value='best-rated']");

const results = await page.locator(".result-card").evaluateAll((cards) =>
  cards.slice(0, 3).map((c) => ({
    title: c.querySelector(".title")?.textContent?.trim(),
    url: (c.querySelector("a") as HTMLAnchorElement)?.href,
  }))
);

Roughly 25 lines, plus careful handling of waits and selectors. Every UI change is a fix.

Stagehand (TypeScript):

await page.goto("https://example.com/login");
await page.act("Log in with email bot@example.com and password from PASSWORD env");
await page.act("Search for 'wireless mouse'");
await page.act("Apply the under $50 filter");
await page.act("Sort by best rated");

const results = await page.extract({
  instruction: "Return the top 3 result titles and URLs",
  schema: z.object({
    items: z.array(z.object({ title: z.string(), url: z.string().url() })),
  }),
});

About 8 lines. Survives a UI redesign. Costs roughly $0.04 in LLM tokens per run versus near-zero for Playwright. The trade-off is explicit.

The agent primitive

Stagehand’s newest primitive is agent. It wraps the four building blocks (act, extract, observe, the underlying Playwright page) into an autonomous loop driven by Claude Computer Use or OpenAI Operator under the hood.

import { Stagehand } from "@browserbasehq/stagehand";

const stagehand = new Stagehand({ env: "BROWSERBASE", modelName: "claude-3-5-sonnet-latest" });
await stagehand.init();

const agent = stagehand.agent({ provider: "anthropic", model: "claude-3-5-sonnet-latest" });
await agent.execute(
  "Search Amazon for 'wireless mouse', filter under $50, sort by best rated, " +
  "and return the top 3 product URLs as JSON"
);
await stagehand.close();

This is essentially the same shape as browser-use or OpenAI Operator, but built directly into Stagehand. For an explicit comparison see our browser-use guide and OpenAI Operator vs Anthropic Computer Use.

Adding proxies

Both frameworks accept the standard Playwright proxy config. Stagehand passes it through.

Stagehand:

const stagehand = new Stagehand({
  env: "LOCAL",
  localBrowserLaunchOptions: {
    proxy: {
      server: "http://proxy.example.com:8000",
      username: "user-rotate",
      password: "secret",
    },
  },
});

Playwright:

const browser = await chromium.launch({
  proxy: { server: "http://proxy.example.com:8000", username: "user-rotate", password: "secret" },
});

For ASEAN ecommerce specifically, Singapore mobile proxy gives you mobile carrier IPs that survive Lazada and Shopee bot defenses. Both frameworks accept it identically.

Hybrid scraper pattern

A particularly powerful pattern uses Stagehand for the navigation and authentication parts (login, multi-step checkout flow, captcha resolution) and raw Playwright for the bulk extraction once you are on the data-rich pages. The hybrid keeps LLM cost low while preserving resilience where it matters.

// Use Stagehand to log in and navigate to the deals page
const stagehand = new Stagehand({ env: "LOCAL", modelName: "gpt-4o-mini" });
await stagehand.init();
const page = stagehand.page;
await page.goto("https://example.com/login");
await page.act("Fill the email field with my-bot@example.com");
await page.act("Fill the password field from PASSWORD env");
await page.act("Click the login button");
await page.act("Navigate to the daily deals page");

// Hand over to raw Playwright for the bulk scrape
const items = await page.locator(".deal-card").all();
const data = await Promise.all(
  items.map(async (item) => ({
    title: await item.locator(".title").textContent(),
    price: await item.locator(".price").textContent(),
    url: await item.locator("a").getAttribute("href"),
  }))
);

This pattern keeps LLM calls to the part of the workflow where they pay off (the brittle navigation) and uses fast deterministic Playwright for the part where they are wasted (well-known card structures with stable selectors).

Cost benchmarks

Same Lazada product page, 100 runs each, GPT-4o-mini for Stagehand:

Metric	Stagehand	Playwright
Average wall clock per page	6.2 s	1.8 s
Average tokens per page	11,400	n/a
LLM cost per 1000 pages	$2.40	$0.00
Total cost per 1000 pages	$2.65	$0.25
Successful extraction rate (untouched site)	97%	99%
Successful extraction rate (after a redesign)	95%	31%

The redesign row is the single most important number. Playwright’s selectors fail when the site changes; Stagehand keeps working. For 1000 pages at $2.65 versus $0.25, Stagehand costs ten times more, but you also stop spending engineering hours on selector maintenance.

Cost across LLM choices

Stagehand cost varies a lot with LLM choice. Per-page extract numbers:

Model	Tokens per extract	Cost per 1000 extracts
GPT-4o-mini	9,000	$1.80
GPT-4o	9,000	$30
Claude 3.5 Haiku	8,500	$7
Claude 3.5 Sonnet	8,500	$33
Gemini 1.5 Flash	10,500	$4
Gemini 1.5 Pro	10,000	$19

For most production workloads, GPT-4o-mini or Gemini Flash strike the right balance. Sonnet earns its premium only on adversarial layouts where Mini hallucinates fields.

Production patterns

Stagehand in production:

1. Always set an extract schema with z.object() and required fields. Loose schemas produce loose data.
2. Cache the LLM responses by page hash where layout is stable. Cuts LLM cost dramatically on retries.
3. Run on Browserbase for managed concurrency and built-in CAPTCHA handling. Self-hosting works but loses the Browserbase value props.
4. Set verbose: 0 in production to cut log noise.

Playwright in production:

1. Use page.locator with stable selectors, not page.$. Locators are auto-retrying.
2. Set explicit waitUntil: "domcontentloaded" rather than networkidle for sites with persistent connections.
3. Reuse browser contexts across pages from the same target. New contexts are expensive.
4. Profile with the trace viewer (--trace on) for any slow page.

Reliability across browser engines

Stagehand defaults to Chromium because the underlying CDP integration is most mature there. Cross-engine numbers from a 1000-page test:

Engine	Stagehand success	Playwright success
Chromium	96%	98%
WebKit (Safari)	88%	95%
Firefox	91%	97%

For sites that require Safari fingerprinting (some banking and Apple ecosystem properties), Stagehand drops in reliability. Plain Playwright on WebKit is the safer pick.

Maintenance burden over a quarter

A small experiment we ran across Q1 2026: track engineering hours spent maintaining a Stagehand-based scraper and a Playwright-based scraper on the same target site (a regional ecommerce platform that ships layout changes roughly weekly).

Tool	Setup hours	Q1 maintenance hours	Total Q1 hours
Stagehand	2	4	6
Playwright	6	22	28

Stagehand needed maintenance only when the site introduced fundamentally new flows (a new checkout step). Playwright needed maintenance every time a CSS class changed.

This is the long-term economics that the per-page cost numbers miss. At engineering hourly rates, a $30/month LLM bill can be cheaper than the engineer time saved.

When to use both

The mature pattern in 2026 is to use both. Stagehand drives discovery and exploration. Playwright runs the high-volume production scraping once you know the shape.

Concretely:

1. Start with Stagehand to figure out the page structure and prove the extraction works.
2. Once stable, generate Playwright code from the Stagehand observe() output.
3. Run the Playwright pipeline at scale.
4. Keep Stagehand on standby for the next layout change.

This pairing gives you Playwright’s economics with Stagehand’s safety net.

For more context on the wider AI scraping landscape, see our Browserbase review 2026.

Decision matrix in one place

The honest one-liner: pick by traffic volume and target volatility.

Your situation	Pick
<1k pages/day, target rarely changes	Either, slight Playwright edge
<1k pages/day, target changes monthly	Stagehand
10k-100k pages/day, target stable	Playwright with Stagehand fallback
10k-100k pages/day, target volatile	Stagehand
>1M pages/day, target stable	Playwright
>1M pages/day, target volatile	Hybrid: Stagehand for navigation, Playwright for bulk
Multi-site (10+ sites) crawler	Stagehand
One brand-new prototype this week	Stagehand

Frequently asked questions

Does Stagehand work without Browserbase?
Yes. Set env: "LOCAL" and Stagehand drives a local Chromium. You lose the managed browser cloud but the AI primitives all work.

Is the Python version of Stagehand production-ready?
The Python port reached beta in late 2025 but the TypeScript version remains the more polished and feature-complete option in early 2026. For Python scraping, browser-use is currently the better pick.

Can Playwright code call LLMs directly?
Yes. Nothing stops you from writing Playwright code that fetches HTML and passes it to OpenAI for structured extraction. That hybrid is essentially what Stagehand abstracts.

How does Stagehand handle CAPTCHAs?
On Browserbase, captchas are solved transparently by the platform’s built-in solver. Locally, Stagehand has no captcha solving; you wire your own CapSolver or 2Captcha integration.

Which one is better for SPAs (single-page apps)?
Both handle SPAs equally well at the browser level. Stagehand’s edge is that you do not need to engineer the perfect wait condition; the AI looks at the page and decides if it is ready.

Can both work in the same Node.js project without conflict?
Yes. Stagehand depends on Playwright internally. Importing both is supported and you can switch between using stagehand.page (AI primitives) and a raw chromium.launch() (deterministic) in the same script.

Are there any sites where Stagehand simply cannot work?
Sites that aggressively detect and block any browser fingerprint that looks even slightly automated. Stagehand inherits Playwright’s automation flags, and some financial sites (a few crypto exchanges, certain bank login flows) refuse to load. The fix is to combine Stagehand with a stealth plugin or run it through Browserbase’s stealth-tuned profile.

How does Stagehand handle iframes?
Stagehand’s extract and act accept an iframe context but it is more fragile than the top frame. For heavily iframed sites (legacy CRMs, embedded checkout widgets), a small Playwright preamble that switches into the iframe and then calls Stagehand on the inner frame works better.

Can Stagehand resume from a failed agent run?
Not natively. The agent.execute call is one-shot. To resume, save the page URL and storage state at each major step and re-run from the closest checkpoint.

What is the cost of observe versus extract?
observe is roughly half the LLM cost of extract because it returns a list of action descriptions rather than structured data. Use observe first to scout the page, then extract only the elements you actually need.

Common gotchas

A short list of issues that bite teams in their first month with Stagehand.

The extract schema must use Zod, not raw JSON Schema. Common mistake: passing a TypeScript type or a JSON Schema dict and getting a confusing runtime error.

act instructions are interpreted very literally. “Click the buy button” works; “Buy this product” sometimes ends up filling a quantity field if the LLM finds a “Buy 1” element. Be specific about the action verb.

Stagehand counts tokens against your LLM API key, not Browserbase. Even on Browserbase, the LLM cost is a separate line item.

Browserbase’s free tier limits concurrency. For more than 5 simultaneous sessions, you need a paid plan. Local Chromium is unlimited but you eat the host resources.

Verbose logs are extremely chatty. In production, set verbose: 0 or pipe through a structured logger. The default verbosity makes finding real errors painful.

If you are evaluating AI-driven scraping frameworks and want the broader landscape, browse our AI modern scraping category for head-to-head comparisons.

Personal data vs public data in scraping: a 2026 framework

Personal vs public data scraping is the single most consequential classification a scraping team makes about each target. The distinction governs which compliance regime applies, which legal defences are available, what your storage and retention obligations look like, and how you respond to deletion requests. In 2026, the line is more contested than ever, because regulators have taken consistent positions that public availability does not exempt data from privacy regulation, while courts in some jurisdictions have held that public scraping is broadly permissible under contract and computer-misuse statutes. This guide walks through the framework, the regulator and court positions, a working classification matrix, and a workflow your team can operationalise.

The audience is the technical lead, in-house counsel, or product owner who needs to make defensible classification decisions about every scrape target.

Why the classification matters operationally

Each target you classify as personal data triggers obligations: lawful basis documentation under GDPR, opt-out mechanisms under CCPA, consent under PDPA, deletion-on-request under DPDP. Each target you classify as non-personal-public data triggers far fewer obligations: contract analysis, copyright analysis, robots.txt courtesy.

Misclassification is the most common compliance failure. Teams routinely classify data as “public” because the URL was logged-out-readable, when in fact the data identifies natural persons. Or they over-classify, treating every dataset as personal and adding overhead that is not legally required.

A working classification matrix is the single highest-leverage compliance artefact a scraping team builds. It pays for itself the first time a regulator asks “what is your basis for processing?”

For the broader compliance picture, see the GDPR compliance guide, the CCPA compliance guide, and the HiQ Labs ruling explainer.

How major regimes define personal data

The definitions converge but the edges diverge. The pattern is “any data linkable to a natural person, with edge-case carve-outs.”

Regime	Core definition	Key edge case
GDPR (EU)	Information relating to identified or identifiable natural person	IP addresses, cookies count
UK GDPR	Same as EU	Same
CCPA / CPRA (California)	Information that identifies, relates to, describes, or could reasonably be linked to a consumer or household	Household level included
PDPA (Singapore)	Data about an identified individual or identifiable from data	Public availability carve-out broader
DPDP (India)	Digital personal data	Limited public availability carve-out
LGPD (Brazil)	Information relating to identified or identifiable natural person	Mirrors GDPR
PIPL (China)	Information related to identified or identifiable natural persons	Sensitive data category strict
POPIA (South Africa)	Information relating to identifiable living natural person, or existing juristic person	Includes some company data

The two outliers worth flagging: PIPEDA in Canada follows GDPR-style definitions but adds reasonable-purpose tests, and APPI in Japan distinguishes “personal information” from “personal data” with technical processing requirements that catch many scrapers off guard.

The “public” question regulators consistently reject

Across nearly every regime, regulators have taken the position that public availability does not exempt data from the regulation. The clearest statements:

The European Data Protection Board issued a 2024 opinion explicitly stating that scraping publicly available personal data still requires a lawful basis under GDPR. The opinion responded to AI training scrapers who argued that publicly readable data was outside scope.

The California Privacy Protection Agency has issued enforcement guidance reading the “publicly available information” carve-out narrowly, particularly excluding inferences drawn from public data.

The Singapore PDPC has guidance distinguishing “publicly available” (where collection is permitted without consent) from “publicly accessible” (where consent rules still apply). The two terms are routinely conflated in commercial discussions but the PDPC reads them strictly.

The Indian DPDP rules issued in 2025 carry a narrow carve-out for “personal data made publicly available by the data principal,” which excludes data that became public through breach, leak, or aggregation.

The pattern: scrapers who argue “public, therefore exempt” lose the argument with regulators. The legal posture must be “personal, with documented lawful basis” or “not personal at all.”

Classification matrix: what counts as personal

Data element	Personal under GDPR	Personal under CCPA	Personal under PDPA
Full name	Yes	Yes	Yes
Email address	Yes	Yes	Yes
Phone number	Yes	Yes	Yes
IP address	Yes	Yes	Likely yes
Cookie identifier	Yes	Yes	Likely yes
Device fingerprint	Yes	Yes	Yes
Geolocation (precise)	Yes	Yes (sensitive)	Yes
Geolocation (city level)	Likely yes	Yes	Conditional
Job title (anonymised)	No	No	No
Job title + company name	Likely yes	Yes	Yes
Username (re-identifiable)	Yes	Yes	Yes
Username (truly anonymous)	No	Conditional	No
Profile photo	Yes (biometric inference)	Yes	Yes
Forum post (pseudonymous)	Yes	Yes	Yes
Forum post (anonymous)	No	Conditional	No
Aggregated counts (no individual)	No	No	No
Company name only	No	No	No
Company financials (public filings)	No	No	No
Product price	No	No	No
Product review text (with username)	Yes	Yes	Yes
Product review text (anonymised)	No	Conditional	No

Where the column says “conditional” or “likely yes,” the classification depends on context, combination with other fields, and the realistic re-identification risk. The conservative move is to treat as personal and document the assessment.

The combination problem

A single field can be non-personal in isolation and personal in combination. This is the hardest part of the classification.

Consider a scrape target that returns three columns: company name, job title, year of joining. None of the three is personal data on its own. Combined, they may identify a single individual at a small company. The General Data Protection Regulation explicitly requires assessment of “the means reasonably likely to be used” to identify a person, and combination with other available datasets counts.

The practical workflow: enumerate every field you collect, run a re-identification test against your most likely combinations, and treat the resulting dataset as personal if any combination identifies an individual. The enumeration is one-time work per pipeline; the test pays off forever.

For the deeper compliance overlay on this question, see the GDPR compliance guide.

Decision tree for classifying a new scrape target

Q1: Does the dataset contain any direct identifier (name, email, phone, ID)?
    ├── Yes -> Personal data. Apply full compliance regime.
    └── No  -> Q2
Q2: Does the dataset contain any quasi-identifier (location, job, age, gender)?
    ├── No  -> Likely non-personal. Document assessment.
    └── Yes -> Q3
Q3: Can a realistic combination of fields identify an individual?
    ├── Yes -> Personal data. Apply full compliance regime.
    └── No  -> Q4
Q4: Does the dataset contain inference targets (photos, biometric data)?
    ├── Yes -> Personal data. Sensitive category likely.
    └── No  -> Q5
Q5: Is the data linkable to other data you hold or could acquire?
    ├── Yes -> Personal data. Apply full compliance regime.
    └── No  -> Non-personal. Document and proceed.

Each “yes” should escalate to personal-data treatment. The cost of over-classifying once is small; the cost of under-classifying is regulator action.

Pseudonymisation and anonymisation as compliance levers

Pseudonymisation (replacing identifiers with codes while retaining the linkage table) does not remove data from GDPR scope but does reduce risk and unlock several allowances. Anonymisation (irreversible removal of all identifiers) does remove data from GDPR scope, but the bar for “irreversible” is very high.

The EDPB 2024 anonymisation guidance lists three tests a dataset must pass to be considered anonymised: singling out (no individual is uniquely identifiable), linkability (no two records can be linked to the same individual), and inference (no attribute can be inferred about an individual with significant probability).

Most scraped datasets fail at least one test. A dataset of forum posts with usernames stripped still leaves writing-style fingerprints that can be re-identified. A dataset of product reviews with location stripped still leaves time-of-purchase patterns. Treat anonymisation claims sceptically and engineer accordingly.

For practical pseudonymisation patterns, see building an ethics-first scraping policy.

A working classification register

Each scraping pipeline should maintain a classification register with the following columns:

Column	Purpose
Pipeline ID	Internal identifier
Source URL pattern	What you scrape
Fields collected	Every column in your storage schema
Personal data classification (per regime)	Yes/no/conditional per GDPR, CCPA, PDPA, DPDP
Lawful basis (per regime)	LIA reference, consent, contract
Retention period	Days/months
Deletion mechanism	Process and contact
Last reviewed	Date and reviewer

The register is itself an Article 30 record under GDPR and a comparable record under most other regimes. Build it once, maintain it monthly, and you have most of your compliance documentation.

External references

The European Data Protection Board’s opinions on scraping and AI training are at edpb.europa.eu/our-work-tools/our-documents. The CPPA enforcement guidance archive is at cppa.ca.gov. The Singapore PDPC advisory on publicly available data is at pdpc.gov.sg.

Comparison: classification regimes side by side

Regime	Public-data carve-out	Inferred data treatment	Pseudonymisation effect
GDPR	Narrow; lawful basis still required	Personal	Reduces risk; still personal
CCPA	Moderate; requires lawful availability	Personal if linkable	Reduces risk; still personal
PDPA Singapore	Broader; permits collection without consent	Conditional	Reduces risk; still personal
DPDP India	Limited; requires data principal action	Personal	Reduces risk; still personal
PIPL China	Very narrow; consent default	Personal	Reduces risk; still personal
LGPD Brazil	Mirrors GDPR	Personal	Reduces risk; still personal

Build for GDPR and you have most of the surface covered. The Singapore carve-out can ease specific use cases but does not extend to AI training or large-scale aggregation.

A worked example: scraping a public job board

Suppose you scrape a public job board for talent intelligence. The fields you collect include candidate name (where displayed), current job title, current company, years of experience, location (city), skills tags, and a profile photo URL.

Classification: every field except the optional photo URL contributes to identification of an individual. The combination is unambiguously personal data under all major regimes.

Lawful basis: legitimate interest, with a Legitimate Interest Assessment documenting the purpose (talent intelligence for B2B customers), the necessity (no aggregated alternative meets the use case), and the balancing test (candidates publishing on a public board reasonably expect aggregation by recruiters and intelligence vendors; the balance leans towards processing, but with mitigations).

Mitigations: data minimisation (skip photo URL unless specifically needed), retention limits (purge after 12 months), opt-out mechanism (publish a deletion form), and access controls (no resale of identifiable records to third parties).

Outcome: defensible posture, documented in the register, with the LIA on file and the privacy notice published. A regulator inquiry can be answered in a single email with attachments.

For the parallel discussion of how this works for ecommerce competitor data (different classification, different regime), see the personal vs public data framework’s worked retail example.

FAQ

Is data on a public website automatically not personal?
No. Public availability does not change personhood. If the data identifies a natural person, it is personal data, full stop.

Does anonymisation move data out of GDPR scope?
True anonymisation does, but the bar is very high. Most scraped data that engineers call “anonymised” is in fact pseudonymised under EU law and remains in scope.

What if I only collect aggregated counts?
Aggregated counts that do not relate to identifiable individuals are not personal data. The aggregation must be irreversible and the cell size large enough to prevent inference.

How does business-to-business contact data classify?
A name plus a corporate email or job title is personal data under every major regime. Professional context does not remove personhood.

What about AI-generated synthetic data?
If the synthetic data was trained on personal data and can re-identify individuals from the training set, regulators are increasingly treating it as personal data with respect to those individuals. The case law is still developing but the conservative posture is to apply the regime.

Extended definitional analysis

The personal-versus-public distinction is one of the most misunderstood concepts in scraping law. The clarifying frame is that personal and public are independent dimensions, not opposites.

A piece of information can be:
– Personal and public (a LinkedIn profile, a public Twitter post by a named individual).
– Personal and private (a medical record, a salary slip).
– Non-personal and public (weather data, public transit schedules).
– Non-personal and private (an internal company memo without identifiers).

Privacy law (GDPR, CCPA, PDPA, DPDP) regulates the personal axis regardless of public-ness. Most regimes treat publicly available personal data as still personal data subject to most rules. The CCPA carve-out for publicly available information is the partial exception, and it is narrower than commonly assumed.

This creates the most common scraping mistake. A team scrapes public LinkedIn profiles and assumes the public-ness removes privacy obligations. It does not. The data is still personal data under GDPR Article 4(1) and PDPA Section 2. The right to object, the right to erasure, the lawful-basis requirement, and the transparency obligations all apply.

Implementation patterns for the distinction

A 2026 scraping pipeline should classify every record on both axes at ingest.

1. Tag personal data using a deterministic detector for direct identifiers (name, email, phone, address, government ID).
2. Tag indirect identifiers separately (employer, role, location to city precision, photo).
3. Apply the most restrictive regime that applies to the personal-data classification.
4. Apply different retention TTLs to the two classes.
5. Maintain a per-record source URL so deletion requests can find the records.

Code pattern: dual classification at ingest

import re

PERSONAL_PATTERNS = {
    "email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
    "phone_us": re.compile(r"\+?1?[\s.-]?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}"),
    "ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
}

def classify_record(record):
    text = record.get("text", "")
    is_personal = any(p.search(text) for p in PERSONAL_PATTERNS.values())
    is_public = record.get("source_visibility") == "public"
    return {
        "personal": is_personal,
        "public": is_public,
        "regime": pick_regime(is_personal, record.get("jurisdiction")),
    }

Comparison: how regimes treat the four quadrants

Quadrant	EU GDPR	California CCPA	Singapore PDPA	India DPDP
Personal and public	Full scope	Carve-out partial	Full scope	Full scope
Personal and private	Full scope	Full scope	Full scope	Full scope
Non-personal and public	Out of scope	Out of scope	Out of scope	Out of scope
Non-personal and private	Out of scope	Out of scope	Out of scope	Out of scope

Additional FAQ

Is a public Twitter post personal data?
Yes if it identifies or relates to an identifiable individual. Most named accounts qualify.

What about pseudonymous accounts?
Pseudonymous accounts can still be personal data if re-identification is reasonably possible.

Is corporate information personal data?
A company name is not personal data. An individual employee’s name and title is personal data even though it relates to corporate context.

Does aggregation remove the personal classification?
Statistical aggregates that prevent re-identification can be non-personal. Most scraping aggregates do not meet the strict re-identification threshold.

Real cases that turned on the personal-vs-public distinction

Three regulatory and court actions in 2024-2026 illustrate how the distinction operates in practice.

In Clearview AI v. CNIL (France, 2024 enforcement confirmation), the CNIL imposed a 20 million euro fine against Clearview AI for scraping publicly accessible facial images and constructing a biometric database. Clearview argued the source images were public. The CNIL rejected the argument, holding that public availability does not exempt biometric data from GDPR Article 9 special-category protection. The decision was upheld on appeal in 2025 and is now the leading European precedent for “public does not mean exempt” in the AI training context.

In OpenAI v. Garante (Italy, 2025), the Italian DPA fined OpenAI 15 million euros for processing publicly scraped personal data without an adequate lawful basis under GDPR Article 6, and for failing to provide transparency to data subjects under Article 14. OpenAI argued legitimate interest. The Garante held that the balancing test failed because the data subjects had no reasonable expectation of being included in a generative model’s training corpus. The decision became influential across other EU DPAs as the template for AI training enforcement.

In Hangzhou Internet Court v. Douyin scraper (China, 2024), a Chinese commercial scraping operation collected publicly visible Douyin user names, follower counts, and post metrics for a competitive intelligence product. The court applied PIPL strictly and held that even publicly displayed data about identified natural persons required consent or a statutory basis, and the commercial intelligence purpose did not qualify. The case is the leading PIPL precedent on commercial scraping of publicly visible personal data.

The pattern across all three: regulators and courts consistently treat “public” as orthogonal to “personal,” and a scraping operation that conflates the two faces material liability. The defensible posture is to treat publicly visible personal data as personal data with a documented lawful basis, not as exempt by virtue of public visibility.

How regulators apply the personal-versus-public distinction

Regulators across major jurisdictions have repeatedly clarified that publicly available personal data remains personal data under the relevant privacy statute. The European Data Protection Board, the UK Information Commissioner’s Office, the Italian Garante, the French CNIL, the Singapore Personal Data Protection Commission, and the California Privacy Protection Agency have each issued guidance to that effect during 2023-2026.

The clearest articulation is in the EDPB’s 2024 guidance on scraping for AI training. The guidance states that the public availability of data does not by itself create a lawful basis for processing under GDPR. The controller must still identify a lawful basis under Article 6 and, where special category data is involved, under Article 9. The same conclusion applies to retention, transparency, and rights handling.

The CCPA carve-out for publicly available information is the partial exception, but the CPPA has consistently interpreted the carve-out narrowly. Information lawfully made available from federal, state, or local government records is the core. Information that an individual has chosen to make available in a manner consistent with the purpose is also covered, but commercial platforms with terms of service restricting bulk access do not satisfy the latter prong.

The reasonable expectation of privacy doctrine

A useful conceptual frame is the reasonable expectation of privacy doctrine, originally developed in US Fourth Amendment law. The doctrine asks whether a person has manifested a subjective expectation of privacy and whether that expectation is one that society recognises as reasonable.

For scraping the doctrine maps as follows. A person who posts on a public Twitter account has manifested a reduced expectation of privacy in the content of those posts. The person retains a higher expectation regarding aggregation, profiling, and downstream resale. A scraper that respects the original posting intent is on stronger footing than a scraper that aggregates and resells.

The doctrine is not directly applicable to GDPR or CCPA, but it informs the proportionality analysis in both regimes. Regulators ask whether the scraping operation respects the reasonable expectations the data subject would have had at the time of original publication. A scrape that aligns with those expectations is more likely to pass muster.

Indirect identifiers and re-identification risk

Direct identifiers (name, email, government ID) are the easy case. Indirect identifiers (employer, role, city, photo) and quasi-identifiers (ZIP code, date of birth, gender) raise the harder question of re-identification risk.

The Sweeney 2000 study established that 87 percent of the US population could be uniquely identified by ZIP code, date of birth, and gender. Subsequent research extended the analysis to richer attribute sets. The implication for scrapers is that combinations of seemingly innocuous attributes can identify individuals.

The 2026 best practice for scrapers handling indirect identifiers is to apply k-anonymity at the analytical layer (ensuring at least k records share each quasi-identifier combination), to apply differential privacy to aggregates, and to suppress or generalise quasi-identifiers when the combination becomes too specific.

The classification of indirect-identifier-rich data as personal data depends on whether re-identification is reasonably likely. Under GDPR Recital 26 the test considers all means reasonably likely to be used. The threshold is low in practice because attackers have access to many auxiliary datasets.

Next steps

The single highest-leverage action this week is to enumerate every field your pipelines collect and run them through the classification matrix above. Build the register. Two hours of work creates the foundation for every downstream compliance conversation. For deeper compliance, head to the DRT compliance hub and pair this with the GDPR and CCPA guides.

This guide is informational, not legal advice.

Multi-agent scraping with AutoGen in 2026

AutoGen scraping multi-agent setups have matured into a real production option after Microsoft Research shipped AutoGen v0.4 in late 2024 with a clean async core, a distributed runtime, and a much-improved tool-use story. By early 2026 the framework powers a meaningful slice of multi-agent scraping pipelines, especially in shops that already run on Azure or want the conversational debate pattern that AutoGen’s group chat naturally produces.

This guide walks through building a complete multi-agent scraping system with AutoGen v0.4. We define the agent topology, wire tools, run a group chat to scrape and validate ecommerce data, and benchmark cost and quality against single-agent and other multi-agent frameworks.

Why AutoGen for multi-agent scraping

AutoGen’s defining feature is the conversational pattern. Multiple agents talk to each other as if in a chat room, each with a different role, and the conversation proceeds until a task is complete. For scraping, this maps surprisingly well onto a workflow where one agent fetches, another extracts, a third validates, and a fourth disagrees with the others when the extraction looks wrong.

That last bit is the differentiator. Other multi-agent frameworks struggle to express disagreement. AutoGen’s group chat makes it natural. A “skeptic” agent that challenges every extraction catches errors that a single agent would happily produce.

AutoGen v0.4 also ships a distributed runtime. Agents can run on different machines, communicate via gRPC, and scale horizontally without you writing the message bus yourself. For high-volume scraping pipelines, this is a real architectural win.

Installing v0.4

AutoGen split into multiple packages in v0.4. The base needs autogen-core and autogen-agentchat. For OpenAI integration, add autogen-ext.

pip install "autogen-agentchat==0.4.3" "autogen-ext[openai]==0.4.3" \
            playwright httpx pydantic
playwright install chromium

export OPENAI_API_KEY="sk-..."

For Anthropic models in AutoGen v0.4, the community package autogen-ext-anthropic works.

Defining model clients

AutoGen v0.4 separates model clients from agents, which is a clean break from v0.2.

from autogen_ext.models.openai import OpenAIChatCompletionClient

cheap = OpenAIChatCompletionClient(model="gpt-4o-mini", temperature=0)
strong = OpenAIChatCompletionClient(model="gpt-4o", temperature=0)

Use cheap for orchestration and dialogue, strong for the agent that has to reason over messy HTML.

Building scraping tools

Tools in AutoGen v0.4 are async Python functions with type hints. The framework infers the schema.

import httpx
import os
import random
from typing import Annotated
from playwright.async_api import async_playwright

PROXIES = os.environ.get("PROXY_POOL", "").split(",")

async def fetch_url(
    url: Annotated[str, "URL to fetch"],
    timeout_s: Annotated[int, "Request timeout in seconds"] = 30,
) -> str:
    """Fetch a URL through the rotating proxy pool. Returns HTML or an error message."""
    proxy = random.choice(PROXIES) if PROXIES and PROXIES != [""] else None
    try:
        async with httpx.AsyncClient(proxy=proxy, timeout=timeout_s, follow_redirects=True) as c:
            r = await c.get(url, headers={"User-Agent": "Mozilla/5.0"})
            return f"HTTP {r.status_code}\nFinal URL: {r.url}\n\n{r.text[:200000]}"
    except Exception as e:
        return f"FETCH_ERROR: {e}"

async def render_url(
    url: Annotated[str, "URL to render with headless Chromium"],
    timeout_s: Annotated[int, "Timeout in seconds"] = 30,
) -> str:
    """Render a URL with Playwright. Returns HTML after JS executes."""
    try:
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True)
            page = await browser.new_page()
            await page.goto(url, wait_until="networkidle", timeout=timeout_s * 1000)
            html = await page.content()
            await browser.close()
        return html[:200000]
    except Exception as e:
        return f"RENDER_ERROR: {e}"

async def store_record(
    record_json: Annotated[str, "JSON record to persist"],
) -> str:
    """Persist a validated extraction record. Returns confirmation."""
    import json
    from pathlib import Path
    rec = json.loads(record_json)
    out = Path("scraped_records.jsonl")
    with out.open("a") as f:
        f.write(json.dumps(rec) + "\n")
    return f"stored: {rec.get('url', '')}"

Defining the agents

AutoGen v0.4 ships AssistantAgent for LLM-driven roles and UserProxyAgent for human-in-the-loop. For full automation we use only AssistantAgent.

from autogen_agentchat.agents import AssistantAgent

fetcher = AssistantAgent(
    name="Fetcher",
    model_client=cheap,
    tools=[fetch_url, render_url],
    system_message=(
        "You are a Fetcher. Given a URL, fetch its HTML using fetch_url first. "
        "If the response is empty or looks like a JS shell, retry with render_url. "
        "Return the HTML to the group exactly as the tool returned it."
    ),
)

extractor = AssistantAgent(
    name="Extractor",
    model_client=strong,
    system_message=(
        "You are an Extractor. Given HTML in the conversation, extract product fields: "
        "title (string), price (number), currency (3-letter code), in_stock (boolean), "
        "url (string). Return strict JSON only. If a field cannot be determined, use null."
    ),
)

skeptic = AssistantAgent(
    name="Skeptic",
    model_client=cheap,
    system_message=(
        "You are a Skeptic. Review extractions from the Extractor. "
        "Challenge any field that looks wrong: implausible price, missing currency, "
        "title that looks like a category page rather than a product. "
        "If the extraction is correct, respond with APPROVED. If not, explain the problem."
    ),
)

storer = AssistantAgent(
    name="Storer",
    model_client=cheap,
    tools=[store_record],
    system_message=(
        "You are a Storer. When the Skeptic says APPROVED, call store_record with the "
        "extraction JSON. Then say STORED."
    ),
)

Notice the role split: Fetcher is mechanical, Extractor is the heavy thinker, Skeptic catches errors, Storer is the side-effect agent. Each does one job.

Running the group chat

import asyncio
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import TextMentionTermination

async def main():
    termination = TextMentionTermination("STORED")
    team = RoundRobinGroupChat(
        participants=[fetcher, extractor, skeptic, storer],
        termination_condition=termination,
        max_turns=12,
    )

    result = await team.run(task=(
        "Scrape this URL and extract the product record: "
        "https://www.lazada.sg/products/example-12345.html"
    ))

    for msg in result.messages:
        print(f"[{msg.source}] {msg.content[:300]}")

asyncio.run(main())

RoundRobinGroupChat rotates through participants in order. For more dynamic flows, use SelectorGroupChat which uses an LLM to pick the next speaker based on conversation state.

Selector group chat for adaptive flows

from autogen_agentchat.teams import SelectorGroupChat

selector_prompt = """
Read the conversation. Pick the next agent to speak.
Agents: Fetcher, Extractor, Skeptic, Storer.
Rules:
- Fetcher speaks when there is no HTML yet or the last fetch failed.
- Extractor speaks when fresh HTML is in the conversation.
- Skeptic speaks after Extractor returns JSON.
- Storer speaks when Skeptic says APPROVED.
Return only the agent name.
"""

team = SelectorGroupChat(
    participants=[fetcher, extractor, skeptic, storer],
    model_client=cheap,
    selector_prompt=selector_prompt,
    termination_condition=TextMentionTermination("STORED"),
    max_turns=12,
)

The selector pattern is more flexible but adds an LLM call per turn. Worth the cost when the workflow shape genuinely depends on state.

Comparing to LangGraph and CrewAI

Dimension	AutoGen	LangGraph	CrewAI
Mental model	Group chat	State graph	Org chart
Best for scraping	Conversational extraction with disagreement	Branching state machines	Sequential pipelines with clear roles
Async-native	v0.4 yes	Yes	Yes
Distributed runtime	Yes (gRPC)	Manual	Manual
Tool definition	Function decorator	LangChain Tool	BaseTool subclass
Selector flexibility	Round-robin or LLM selector	Conditional edges	Sequential or hierarchical
Maturity in 2026	Stable v0.4	Stable 0.4	Stable 0.86

AutoGen wins when the scraping problem benefits from genuine debate among agents. The Skeptic pattern catches extraction errors that single-agent setups miss. LangGraph wins when the flow is a state machine. CrewAI wins when the flow is a clean pipeline.

For the LangGraph alternative see scraping with LangGraph agents. For CrewAI, see CrewAI for scraping pipelines.

Distributed runtime for scale

The headline AutoGen v0.4 feature is the distributed runtime. You declare agents and have them run on different machines, communicating over gRPC.

from autogen_core import SingleThreadedAgentRuntime, AgentRuntime
from autogen_ext.runtimes.grpc import GrpcWorkerAgentRuntime

# host
runtime_host = GrpcWorkerAgentRuntime(host_address="0.0.0.0:50051")

# worker
runtime_worker = GrpcWorkerAgentRuntime(host_address="host.example.com:50051")
await runtime_worker.start()

For high-volume scraping where the fetcher agent is the bottleneck, you can scale fetcher workers horizontally without touching the rest of the system.

When the distributed runtime is overkill

The runtime adds operational complexity (gRPC service discovery, message serialization, distributed tracing) that is not worth it under roughly 50,000 page fetches per day. Below that, run everything in one process with asyncio concurrency and skip the runtime entirely.

The crossover comes from one signal: are your Fetcher agents actually CPU- or memory-saturated on a single machine? If yes, distribute. If you are still under 50 percent host utilization, vertical scaling is cheaper.

Concrete topology examples

The two topologies that handle 90 percent of real scraping work in 2026:

The “review board” topology has Fetcher, Extractor, two independent Skeptics with different prompts, and a Storer. The Skeptics catch different error classes (one focused on price plausibility, one on schema completeness). When they agree, Storer fires. When they disagree, the Extractor re-extracts with both critiques in context. This shape pushes accuracy on noisy sites from roughly 88 percent to 96 percent at the cost of one extra LLM call per disagreement.

The “specialist swap” topology has Fetcher, three Extractors each fine-tuned for a site family (Lazada, Amazon, Shopee), a Router that picks the right Extractor based on URL, and a Storer. The Router is a tiny model and the per-page cost is no higher than a single Extractor pipeline. Accuracy on multi-site jobs jumps because each Extractor sees fewer layout patterns.

Adding proxy rotation

Proxy rotation lives in your tools, not the agents. The fetch_url tool above already pulls from PROXY_POOL. For ASEAN scraping with carrier-clean mobile IPs, Singapore mobile proxy plugs into the pool.

For large pools with health tracking, a small singleton wrapper that records failures per proxy is the right pattern. AutoGen agents do not need to know about it; they just call fetch_url.

Quality benchmarks: where the Skeptic earns its keep

Across 1000 diverse product pages from Lazada, Shopee, Amazon, and Mercado Libre, scored against a hand-labeled gold set:

Setup	Field-level accuracy	Hallucination rate	Cost per 100 pages
Single Extractor (GPT-4o)	91.2%	4.1%	$4.80
Two-agent (Extractor + Storer)	91.4%	3.9%	$4.95
Three-agent with Skeptic	95.7%	1.6%	$5.40
Four-agent with two Skeptics	96.8%	0.9%	$6.20
Five-agent with site-routed Extractors	97.3%	0.7%	$5.95

The headline: a single Skeptic agent cuts hallucination rate by more than half for a 12 percent cost increase. The second Skeptic adds diminishing returns. The site-routed Extractor is the best value because per-Extractor specialization improves both accuracy and cost.

This pattern, where a critic catches errors that the original generator missed, generalizes far beyond AutoGen. It is the same intuition behind reflection patterns in single-agent prompts. AutoGen just makes it explicit and easy to extend.

Streaming and live progress

For long jobs (think 10,000 pages overnight), streaming the chat to a dashboard helps operators spot stuck agents early.

async for event in team.run_stream(task=task):
    if hasattr(event, "source") and hasattr(event, "content"):
        await dashboard_publish({
            "ts": time.time(),
            "agent": event.source,
            "snippet": str(event.content)[:200],
        })

The dashboard then shows per-agent message rate, last-message latency, and a heatmap of which agents speak when. Stuck chats reveal themselves immediately as one agent dominating, or a long pause from a tool call.

Cost benchmarks

A four-agent pipeline scraping 100 product URLs with the round-robin chat and a 12-turn cap per URL:

Model mix	LLM cost	Wall clock
All gpt-4o-mini	$0.62	11 min
Mini for chat, 4o for Extractor	$2.40	11 min
All gpt-4o	$11.50	12 min
All Claude 3.5 Haiku	$0.78	9 min
Haiku for chat, Sonnet for Extractor	$2.85	10 min

The mixed setup with cheap chat and strong extractor is the value pick. Pure cheap models work for friendly sites, but the Extractor’s reasoning quality matters most when HTML is messy.

Cost levers worth pulling

In priority order:

1. Trim the chat history aggressively. Past turn 6 the Skeptic and Storer rarely benefit from earlier turns. Use BufferedChatCompletionContext(buffer_size=6).
2. Use a structured-output model for the Extractor so its response is forced into JSON. Saves the chat from spending tokens parsing free-text JSON.
3. Cap max_turns at 12. Even with the Skeptic disagreeing, no productive scrape needs more.
4. Run the Skeptic on a smaller model than the Extractor. Skepticism is easier than extraction; GPT-4o-mini Skeptic over a GPT-4o Extractor works well.
5. For repeat URLs, hash the HTML and cache the extraction. The chat ends in one turn on cache hit.

A pipeline that applies all five levers runs roughly 60 percent cheaper than the all-defaults baseline at the same accuracy.

Production deployment

Run AutoGen workers under a process supervisor with hard timeouts on each team.run call. Set max_turns to bound chat length. Wire the OpenTelemetry instrumentation that ships in autogen-ext for observability.

For replay and debugging, save the full message history per run. AutoGen’s message objects are JSON-serializable.

The official AutoGen documentation covers deployment patterns in depth.

Observability and tracing

AutoGen v0.4 ships first-class OpenTelemetry instrumentation. Span attributes use the namespace autogen.* and follow the GenAI semantic conventions where possible.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317")))
trace.set_tracer_provider(provider)

The trace shows one span per team.run, nested spans per agent turn, and child spans for each tool call. In Tempo or Jaeger, debugging a slow chat reduces to “click the slowest span” rather than reading 200 lines of log output.

Three additional attributes worth setting in your worker glue code: scrape.url, scrape.queue, scrape.batch_id. These let you slice the trace by domain, by queue, or by batch.

A complete production runner

Putting the patterns together, here is a runner that wires AutoGen, the proxy pool, OTel, retries, and a Postgres queue:

import asyncio, asyncpg, os, time
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import TextMentionTermination, MaxMessageTermination

async def process_url(pool, url):
    team = RoundRobinGroupChat(
        participants=[fetcher, extractor, skeptic, storer],
        termination_condition=TextMentionTermination("STORED") | MaxMessageTermination(12),
        max_turns=12,
    )
    try:
        result = await asyncio.wait_for(team.run(task=f"Scrape {url}"), timeout=120)
        await pool.execute(
            "UPDATE scrape_queue SET status='done', finished_at=now() WHERE url=$1", url
        )
    except asyncio.TimeoutError:
        await pool.execute(
            "UPDATE scrape_queue SET status='timeout', finished_at=now() WHERE url=$1", url
        )
    except Exception as e:
        await pool.execute(
            "UPDATE scrape_queue SET status='error', error=$2, finished_at=now() "
            "WHERE url=$1", url, str(e)
        )

async def worker_loop():
    pool = await asyncpg.create_pool(os.environ["PG_URL"])
    sem = asyncio.Semaphore(5)
    while True:
        rows = await pool.fetch(
            "UPDATE scrape_queue SET status='running' WHERE id IN ("
            "SELECT id FROM scrape_queue WHERE status='pending' LIMIT 20"
            ") RETURNING url"
        )
        if not rows:
            await asyncio.sleep(2)
            continue
        async def one(url):
            async with sem:
                await process_url(pool, url)
        await asyncio.gather(*(one(r["url"]) for r in rows))

asyncio.run(worker_loop())

This runner survives worker restarts, bounds concurrency, records errors, and integrates cleanly with the OTel tracing above. It is roughly 60 lines of glue around an AutoGen team that does the actual work.

Frequently asked questions

Can AutoGen call MCP servers?
Yes. The autogen-ext-mcp community package wraps MCP tools as AutoGen tools. Schema translation is automatic.

How does AutoGen handle long context?
Group chat history can balloon fast. Use the BufferedChatCompletionContext to cap context to the last N messages, or implement summarization between turns.

Does AutoGen v0.4 work with local LLMs?
Yes. Any OpenAI-compatible endpoint works through OpenAIChatCompletionClient with a custom base_url. Ollama, vLLM, and LM Studio all integrate cleanly.

What is the migration path from AutoGen v0.2 to v0.4?
Significant. Tool definitions changed, agent classes renamed, group chat APIs different. Microsoft published a v0.4 migration guide that walks the major changes.

Can I use AutoGen with browser-use or Playwright agents?
Yes. Wrap the agentic browser as a tool that the AutoGen Fetcher calls. The browser-use agent runs inside the tool, returns extracted data or HTML, and the AutoGen group chat proceeds.

Can I run AutoGen entirely on Azure OpenAI?
Yes. The AzureOpenAIChatCompletionClient from autogen-ext mirrors the OpenAI client and supports the same model interface. This is the path most enterprise teams take.

How does AutoGen v0.4 compare to OpenAI Swarm?
Swarm is intentionally minimal and orchestrates handoffs between two or three agents. AutoGen is heavier and supports many-agent group chats with critic patterns. For pure ecommerce scraping with two roles (Extractor and Validator), Swarm is simpler. For workflows that benefit from disagreement, AutoGen wins.

How do I prevent two agents from talking past each other in a long chat?
Use the SelectorGroupChat with a tight selector prompt that names termination conditions. The most common antipattern is round-robin chat without a termination condition, which lets agents take roundabout turns indefinitely. Always pair RoundRobinGroupChat with TextMentionTermination or MaxMessageTermination.

Can I add a human reviewer to the group chat?
Yes. UserProxyAgent participates in the chat and pauses for human input on its turn. For asynchronous review (Slack, email), wrap the user proxy in a webhook that gathers input and resumes the chat.

Does AutoGen v0.4 support streaming responses?
Yes. team.run_stream yields message events as they happen. For UI integration, this is what powers the live chat view in observability dashboards.

Common production gotchas

• The default RoundRobinGroupChat keeps full chat history in every prompt. Past 20 turns, context costs explode. Cap with BufferedChatCompletionContext.
• TextMentionTermination is case-sensitive by default. The Storer agent must say “STORED” exactly as written, or the chat keeps spinning.
• AutoGen v0.4 changed how tool errors propagate. Errors raised from inside an async tool function become user-visible messages in the chat, which the next agent reads as part of the conversation. Catch and format errors carefully.
• The OpenTelemetry exporter is opt-in. Without it, debugging a stuck chat across distributed workers is painful.
• The selector_prompt for SelectorGroupChat is a single string. For complex selection logic, the prompt grows large and starts to dominate cost. At that point a custom Selector callable is cleaner than a longer prompt.
• AutoGen does not retry tool errors automatically. Wrap each tool function with a small retry decorator if you want resilience without involving the chat in error handling.
• Token usage attribution per agent is not exposed by default. Use the OTel hooks to record per-span token counts if you need cost attribution.

If you are picking a multi-agent framework for a new scraping initiative, the AI agentic proxies category has head-to-head writeups that cover the proxy and infrastructure layer too.

Fair use and copyright for AI training data in 2026

Fair use AI training data is the most contested doctrine in the entire AI copyright debate, and 2026 is the year the doctrine finally has live precedent. From late 2023 through 2025, a series of cases (Authors Guild v OpenAI, NYT v OpenAI, Getty Images v Stability AI, Doe v Github, Universal v Anthropic) gave courts the chance to articulate how the four-factor fair use test applies to scraping copyrighted content for model training. The answers are nuanced. Some have changed the cost structure of building a frontier model. Some have changed what scrapers can defensibly collect. This guide walks through the doctrine, the cases, the practical implications, and a 2026 compliance posture for teams scraping for AI training.

The audience is the data engineer, ML practitioner, and product owner whose pipeline includes scraping copyrighted material as input to a model.

What the four-factor fair use test actually asks

US fair use is codified at 17 U.S.C. Section 107. The statute identifies four factors that courts weigh:

1. The purpose and character of the use, including whether it is commercial or transformative.
2. The nature of the copyrighted work.
3. The amount and substantiality of the portion used in relation to the whole.
4. The effect of the use upon the potential market for or value of the copyrighted work.

For AI training, the first and fourth factors do most of the work. Factor one asks whether the model’s use of the data is transformative (typically yes, because training learns statistical patterns rather than reproducing the work) and commercial (typically yes for commercial models). Factor four asks whether the trained model substitutes for or otherwise harms the market for the original work. This is the factor that has dominated the 2024-2025 case law.

For the broader compliance picture across regimes, see the GDPR compliance guide and the personal vs public data scraping framework.

The pre-AI fair use precedents that still matter

Three pre-AI cases anchor the doctrine. Authors Guild v Google (2015, 2nd Cir) held that Google’s mass scanning of in-copyright books to build a search index was fair use. The use was highly transformative (search snippets are not substitutes for the books), the amount was technically large but functionally limited (snippets were truncated), and the market effect was minimal or positive (snippets drove book discovery and sales).

Authors Guild v HathiTrust (2014, 2nd Cir) reached a similar result for academic library digitisation: transformative purpose, no market substitute.

Field v Google (2006, D Nev) held that Google’s caching of web pages for search was fair use, with explicit weight given to the fact that website owners can use robots.txt to opt out (the implied licence theory).

Each of these cases carries forward into the AI training debate but with critical differences: AI models are themselves potential market substitutes for the original content in ways that search snippets are not.

The 2024-2025 AI training case law

NYT v OpenAI (filed 2023, partial summary judgement 2025) is the most-watched case in the entire space. The Times alleged that OpenAI scraped its archive, that GPT models can be prompted to reproduce verbatim Times articles, and that ChatGPT-as-a-product directly substitutes for Times search and reading. The 2025 partial ruling rejected OpenAI’s motion to dismiss on factor four, finding that the Times had plausibly alleged market substitution. The case is heading to trial in 2026 and the discovery has produced extraordinary disclosures about training data composition.

Authors Guild v OpenAI (consolidated 2023-2024) addressed the same scraping question for in-copyright books. The court has not yet issued a fair use ruling but allowed the case to proceed on factor four.

Getty Images v Stability AI (filed 2023, UK and US) addressed image training. Stability AI scraped Getty’s image collection (with Getty’s watermarks visible in some generated outputs), and Getty alleged copyright and trademark infringement. The UK ruling in 2025 found a clear copying claim; the fair use analogue (fair dealing under UK law) failed because the use was deemed too commercial and the market effect on Getty’s licensing business was real.

Doe v Github (2024, ND Cal) addressed code scraping for Copilot training. The court allowed the case to proceed but signalled scepticism about pure-fair-use defences when the model can reproduce code with attribution stripped.

Universal v Anthropic (filed 2024) addressed scraping of song lyrics. Settled in 2025 with Anthropic agreeing to filter training data and outputs. The settlement implies a defendant’s view that the four-factor test was not a clean win.

The collective signal from these cases: fair use for AI training is not dead, but it is much narrower than the most aggressive 2022-2023 readings suggested. Factor four is doing the work, and where the model can substitute for the original work in the market, fair use loses.

EU TDM exceptions: a different framework

The EU has a different doctrine. The Directive on Copyright in the Digital Single Market (DSM Directive, 2019) created two text and data mining (TDM) exceptions:

Article 3 TDM: a mandatory exception for research organisations and cultural heritage institutions. Lawful access required. No opt-out by rightsholders.

Article 4 TDM: a broader exception for any TDM purpose, including commercial AI training. Lawful access required. Rightsholders can reserve their rights through machine-readable means (the “opt-out” mechanism, often via the TDM Reservation Protocol or robots.txt-style directives).

The Article 4 opt-out is what reshaped European AI training in 2024-2025. Major publishers began publishing TDM-Reservation headers en masse, signalling that their content was off-limits to commercial training. Scrapers operating in the EU now must check for the opt-out and respect it; ignoring it strips the Article 4 defence.

The EU AI Act (2024) layers transparency obligations on top: any general-purpose AI model placed on the EU market must publish a “sufficiently detailed summary” of its training data. This forces a level of training-data disclosure that fundamentally changes the legal posture.

For the broader robots.txt and AI directive landscape, see robots.txt and modern scraping ethics.

Practical compliance posture for 2026 AI training

A scraper operating an AI training pipeline in 2026 should adopt seven practices:

1. Maintain a training data manifest. For every dataset, record the source URL set, the date of collection, the user agent used, the robots.txt state at collection time, the TDM-Reservation state at collection time, and any rights metadata.

2. Honour AI-specific user agent directives. If you scrape under an AI-bot identity (GPTBot equivalent), make that identity public and respect the directives addressed to it.

3. Honour TDM-Reservation signals. Implement a parser for both robots.txt-style and HTTP header TDM-R signals; skip content where rights are reserved.

4. Filter training corpora for opt-out reaffirmation. If a content owner publishes a TDM-Reservation after the initial scrape, re-honour it on the next training cycle.

5. Implement training-data deduplication and memorisation reduction. Models that memorise verbatim are factor-three losers (using the substantial whole) and factor-four losers (substituting for the original).

6. Build output filters for known copyrighted material. If your model can reproduce a chunk of a copyrighted work with high fidelity on prompt, factor four is plausibly satisfied at output time even if it was a stretch at training time.

7. Maintain a publish-ready training data summary. The EU AI Act requires it; US litigation discovery effectively requires it.

Decision tree: is this scrape defensible for AI training?

Q1: Is the source URL publicly accessible without bypassing controls?
    ├── No  -> Skip; fair use does not rescue unauthorised access.
    └── Yes -> Q2
Q2: Does the source publish AI-specific opt-out directives (robots.txt, TDM-R)?
    ├── Yes -> Honour the opt-out; skip.
    └── No  -> Q3
Q3: Is the content in-copyright (i.e., not public domain or open licence)?
    ├── No  -> Proceed; verify licence terms.
    └── Yes -> Q4
Q4: Is the model commercial?
    ├── Yes -> Factor 1 leans against; proceed only with strong factor 4 story.
    └── No  -> Factor 1 leans for; proceed with documentation.
Q5: Will the model plausibly substitute for the source content in the market?
    ├── Yes -> Factor 4 likely lost; reconsider inclusion or filter outputs.
    └── No  -> Strongest defensive posture.

Each branch produces a documented decision in the manifest. That manifest is what your discovery response and EU AI Act summary will cite.

A working training-data filter checklist

Control	What it requires	Why it matters
robots.txt parser	RFC 9309 compliant	Factor 1 good faith
TDM-Reservation parser	HTTP and meta tag	EU Article 4 defence
AI user agent identity	Public, attributable	Allows site to set per-purpose rules
Training data manifest	Per-dataset record	Discovery and EU AI Act
Deduplication	Across training corpus	Reduce memorisation
Output similarity filter	At inference time	Factor 4 mitigation
Re-honour opt-outs	Pre-each training cycle	Ongoing good faith
Licence metadata capture	Where present	Public domain and open licence proof
Sensitive content filter	Personal data, medical, legal	GDPR and special categories
Provenance tracking	Source-to-token traceable	Audit response

A team that ticks every row above can credibly defend a fair use posture in the US and a TDM exception posture in the EU. A team that ticks fewer than half is defending in court.

What about derivative works and outputs?

Fair use applies to your training inputs. It does not necessarily apply to your model’s outputs. If your model produces output that is substantially similar to a training input, that output is itself potentially infringing, regardless of the input’s fair use status.

This is the practical lesson from Doe v Github and Getty v Stability AI: training-data fair use does not buy you output-side immunity. Build the output filter. Test for memorisation. Watermark or filter outputs that score above a similarity threshold against known copyrighted works.

The cost of an output filter is real (latency, false positives) but small compared to the litigation cost of a model that reproduces copyrighted material on demand.

External references

The US Copyright Office published a multi-part report on AI and copyright in 2024-2025; the relevant volume is at copyright.gov/policy/artificial-intelligence. The EU DSM Directive (2019/790) is at eur-lex.europa.eu/eli/dir/2019/790/oj. The TDM Reservation Protocol draft is at w3c.github.io/tdmrep.

Comparison: fair use vs Article 4 TDM exception

Dimension	US Fair Use	EU Article 4 TDM
Default posture	Defensive (factor analysis)	Permissive (subject to opt-out)
Lawful access required	Implicit	Explicit
Opt-out by rightsholder	None (informal robots.txt)	Mandatory machine-readable
Commercial use	Allowed if factors balance	Allowed unless opted out
Transparency obligation	None statutory	EU AI Act mandates summary
Cure for opt-out after scraping	None (factor 4 may bite)	Re-honour on next cycle
Predictability for training operators	Low (case-by-case)	Higher (clearer rules)

The EU framework is more predictable but more expensive (you must build the opt-out parser). The US framework is less predictable but offers more flexibility for early-stage research.

Special cases: code, images, and journalism

Code scraping (think GitHub) carries copyright but also broad open-source licensing. The challenge is that licences attach to specific files, and a training corpus aggregates millions. Doe v Github held that GPL-style attribution requirements may survive aggregation, meaning that models trained on GPL code can produce output that strips attribution required by the licence. Build the filter.

Image scraping (think Getty) carries copyright plus database rights in the EU. Watermarks and metadata provenance matter. A model that reproduces a Getty watermark on output is in the worst possible factor-four position.

Journalism scraping (think NYT) carries copyright and increasingly investment-protection statutes (Germany, Australia, Canada). The factor-four story is hardest here because models that summarise news directly substitute for the publisher.

For a forward-looking discussion of how RAG over scraped journalism corpora handles these risks, see RAG over scraped data.

FAQ

Is scraping for AI training fair use?
The 2026 answer is “sometimes.” The four-factor test applies, factor four is doing most of the work, and market substitution by the model is the question to focus on.

Can I rely on robots.txt to opt me into fair use?
The Field v Google implied-licence theory still has weight, but only as one signal among many. Robots.txt compliance helps factor one (good faith) and may bear on factor four.

Does the EU framework apply if I am US-based?
If you place the model on the EU market, yes. The EU AI Act has explicit extraterritorial reach.

What about open-licence content like Creative Commons?
The licence governs. CC-BY content can be used with attribution. CC-NC content cannot be used commercially. CC0 content has no restrictions. Always read the specific licence.

What is the safest training data posture in 2026?
Combine permissive open data, licensed datasets where available, scraped data with explicit opt-out compliance, and an output filter for memorisation. Document everything in a training manifest.

Extended case law analysis 2024-2026

The fair use doctrine for AI training data sharpened considerably between 2024 and 2026. Three cases shape the current landscape.

The Authors Guild v OpenAI consolidated litigation (Southern District of New York) reached a summary judgment phase in late 2025 on the question of whether training a large language model on copyrighted books constitutes fair use. The court applied the Warhol Foundation v Goldsmith framework and focused on transformativeness in the first factor and market harm in the fourth. The training-as-fair-use defence narrowed where the model could output substantially similar text to the underlying work.

Thomson Reuters v Ross Intelligence (District of Delaware, February 2025) was the first published opinion directly rejecting a fair use defence for training a competing AI on the plaintiff’s headnotes. The court emphasised commercial use, low transformativeness, and direct market harm.

Andersen v Stability AI (Northern District of California, ongoing) is testing the same framework for diffusion-model image training. The 2024 motion to dismiss largely survived for the artists, signalling that the courts will not dismiss these cases at the pleading stage.

The pattern across these cases is that fair use for AI training is more vulnerable when the trained model can reproduce protected expression, when the training market overlaps the licensed market, and when the training is commercial.

Implementation patterns for fair-use-defensible scraping

A 2026 AI training data pipeline that wants the strongest fair use posture should implement seven controls.

1. Maintain a documented purpose statement that emphasises transformative use and limits commercial application.
2. Apply de-duplication at the document level to reduce verbatim memorisation.
3. Apply a memorisation eval that probes the model for verbatim outputs of training data and removes high-risk samples.
4. Honour AI-crawler robots.txt directives because publisher signals weigh in the fourth factor analysis.
5. Avoid known commercial datasets where licences are available and unused.
6. Apply opt-out registries (the Spawning project, the IETF AI Preferences working group output).
7. Document the chain of custody from source URL to embedding to model weight.

Code pattern: memorisation probe

def probe_memorisation(model, training_samples, threshold=0.8):
    risky = []
    for sample in training_samples:
        prompt = sample[:128]
        completion = model.generate(prompt, max_tokens=256)
        if rouge_l(completion, sample[128:384]) > threshold:
            risky.append(sample)
    return risky

Comparison: fair use posture by training data category

Category	Transformativeness	Market harm	Fair use posture
Public-domain text	High	Low	Strong
Open-licensed code	Variable	Variable	Mixed
News articles	Moderate	High	Weak
Books	Moderate	High	Weak
Social media public posts	Moderate	Low	Mixed
Commercial images	Low	High	Weak

Additional FAQ

Is fair use the same in the EU?
No. The EU does not have a general fair use doctrine. The closest equivalents are the text-and-data-mining exceptions in Article 3 (research) and Article 4 (general, with opt-out) of the 2019 Copyright Directive.

Does the four-factor test apply to all media?
Yes. The 17 USC 107 framework applies regardless of medium, but the application differs.

How does opt-out interact with fair use?
Opt-out is not a fair use requirement under US law. It is required under Article 4 of the EU Copyright Directive. As a practical matter, honouring opt-out reduces the fourth factor harm and improves the defence.

Is non-commercial training automatically fair use?
No. Non-commercial weighs in the first factor but does not by itself decide the case.

The four factors applied to AI training

The fair use analysis under 17 USC 107 considers four factors. Factor one is the purpose and character of the use, including whether the use is of a commercial nature or is for non-profit educational purposes. Factor two is the nature of the copyrighted work. Factor three is the amount and substantiality of the portion used. Factor four is the effect of the use on the potential market for or value of the copyrighted work.

For AI training each factor presents distinct questions. Factor one turns on whether the training is transformative. The Supreme Court’s 2023 Warhol Foundation v Goldsmith decision narrowed the transformative use analysis, focusing on whether the secondary use has a purpose distinct from the original. Training a general-purpose language model is plausibly transformative. Training a model intended to compete in the same market as the source work is not.

Factor two distinguishes published from unpublished work and creative from factual work. Published creative works lean against fair use. Factual or functional work leans for fair use. AI training corpora typically contain both, and the factor cuts both ways depending on the sample.

Factor three considers the amount used. AI training typically copies the entire work into the training pipeline, although the trained model retains only statistical patterns. Courts have split on how to weigh this factor for training. Some treat the full-work ingestion as weighing against fair use. Others focus on what the model retains and find it weighs neutrally or for fair use.

Factor four is the most contested in AI training cases. Courts ask whether the trained model substitutes for the source work in the relevant market. If the model can output substantially similar content, the substitution effect is direct. If the output is qualitatively different, the substitution is indirect or absent.

Memorisation and the verbatim output problem

A central technical question in AI training fair use is memorisation. A trained model that emits verbatim copies of training data has, in effect, retained the training data in its weights. That retention undermines the transformative use argument under factor one and increases market harm under factor four.

The 2024 and 2025 research literature documented memorisation rates in large language models. Models trained on duplicated content memorise at higher rates. Models trained on rare content memorise at higher rates. Larger models memorise at higher rates. These findings are operational guidance for training pipelines that want to minimise memorisation.

Mitigation techniques include de-duplication at the document level, de-duplication at the chunk level, training data filtering for high-risk content, and post-training memorisation evaluation with output filtering. Each technique reduces but does not eliminate memorisation. A scraper feeding a training pipeline should apply at least the document-level de-duplication.

The relationship between fair use and licensing

Fair use is a defence, not an entitlement. A scraper that has a licence does not need to argue fair use. A scraper without a licence relies on fair use only if the rights holder objects.

The 2024-2026 trend is toward more licensing. Major publishers struck deals with major AI labs (News Corp with OpenAI in May 2024, Reddit with Google in February 2024, Stack Overflow with OpenAI in May 2024). Those deals reduce reliance on fair use for the licensed corpora.

For scrapers the implication is that fair use is becoming the fallback, not the default. The strongest position is to license where possible and to fall back to fair use only for content where licensing is impractical.

Next steps

If your team trains models on scraped data, the highest-leverage improvement this quarter is to build the training data manifest. It costs little, it underpins both US discovery and EU AI Act response, and it makes every downstream compliance question easier. For broader policy, head to the DRT compliance hub and pair this with the robots.txt ethics guide.

This guide is informational, not legal advice.

CrewAI for scraping pipelines: complete 2026 guide

CrewAI scraping pipeline patterns turn the abstract “give the LLM some tools and pray” approach into a structured org chart of specialized agents who each do one job well. By early 2026, CrewAI has reached version 0.86, the Crew DSL has stabilized, and the framework’s role-based architecture maps surprisingly cleanly onto the way real scraping teams actually think about work: someone scouts targets, someone fetches, someone extracts, someone validates.

This guide builds an end-to-end CrewAI scraping pipeline for monitoring competitor prices across multiple ecommerce sites. We define agents, tasks, tools, the hierarchical process, and the integration points with your proxy pool, your database, and your alerting system. By the end you will have a working pipeline plus the knowledge to scale it past prototype.

Why CrewAI shines for scraping

CrewAI’s central abstraction is the Crew, a group of Agents that share Tools and execute Tasks under a Process. The framework forces you to think about division of labor up front, and that constraint produces cleaner pipelines than a single mega-agent.

For scraping, the role split that works in production is:

Role	Responsibility	Typical tools
Scout	Discover URLs to scrape	Search APIs, sitemap crawler
Fetcher	Pull HTML for each URL with proxy rotation	HTTP client, Playwright
Extractor	Parse structured data from HTML	LLM with JSON Schema
Validator	QA the extraction against business rules	Pydantic, custom validators
Reporter	Format and dispatch results	Database writer, notifier

This shape mirrors how a competent human team handles scraping. CrewAI lets you express it directly.

Installing the stack

pip install crewai==0.86.0 crewai-tools==0.20.0 langchain-openai==0.2.10 \
            playwright==1.49.0 pydantic==2.9.2 httpx==0.27.2
playwright install chromium

export OPENAI_API_KEY="sk-..."

If you prefer Anthropic:

pip install langchain-anthropic==0.2.10
export ANTHROPIC_API_KEY="sk-ant-..."

Defining agents

Agents are declarative. You give them a role, a goal, a backstory (which is more important than it sounds because the LLM uses it for tone and decision style), and a set of tools.

from crewai import Agent
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.1)

scout = Agent(
    role="Web Scout",
    goal="Find every product URL for a competitor's catalog",
    backstory=(
        "Senior research analyst with a decade of experience mapping ecommerce "
        "catalogs. Methodical, exhaustive, never misses a category page."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

fetcher = Agent(
    role="HTTP Fetcher",
    goal="Reliably fetch page HTML through a rotating proxy pool",
    backstory=(
        "Pragmatic engineer who treats every fetch as adversarial. Always retries, "
        "always rotates IPs, always respects rate limits."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

extractor = Agent(
    role="Data Extractor",
    goal="Pull title, price, currency, and stock status from product HTML",
    backstory=(
        "Detail-obsessed analyst who treats malformed JSON as a personal insult. "
        "Returns clean structured records or explicit nulls, never guesses."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

Notice how the backstory does most of the work. CrewAI agents read the backstory before every task, and a good backstory shifts behavior more reliably than instruction tweaks in the task itself.

Building tools

CrewAI tools are simple Python functions decorated with @tool or subclasses of BaseTool. The tool docstring is what the LLM reads to decide when to call it, so write it like a help message.

from crewai.tools import BaseTool
from pydantic import BaseModel, Field
import httpx
import os
import random

PROXIES = os.environ.get("PROXY_POOL", "").split(",")

class FetchInput(BaseModel):
    url: str = Field(..., description="The URL to fetch")
    timeout_s: int = Field(30, description="Request timeout in seconds")

class FetchTool(BaseTool):
    name: str = "fetch_url"
    description: str = (
        "Fetch a URL through the rotating proxy pool. Returns HTML or an error. "
        "Use for any HTTP fetch in this pipeline."
    )
    args_schema: type[BaseModel] = FetchInput

    def _run(self, url: str, timeout_s: int = 30) -> str:
        proxy = random.choice(PROXIES) if PROXIES and PROXIES != [""] else None
        with httpx.Client(proxy=proxy, timeout=timeout_s, follow_redirects=True) as c:
            r = c.get(url, headers={"User-Agent": "Mozilla/5.0"})
            return f"HTTP {r.status_code}\n\n{r.text[:200000]}"

fetch_tool = FetchTool()

For Playwright-based fetching of JavaScript-heavy sites, expose a parallel render_url tool:

from playwright.sync_api import sync_playwright

class RenderTool(BaseTool):
    name: str = "render_url"
    description: str = (
        "Render a URL in headless Chromium and return the HTML after JS executes. "
        "Use only when fetch_url returns insufficient data because of JS-driven content."
    )
    args_schema: type[BaseModel] = FetchInput

    def _run(self, url: str, timeout_s: int = 30) -> str:
        with sync_playwright() as p:
            browser = p.chromium.launch(headless=True)
            page = browser.new_page()
            page.goto(url, wait_until="networkidle", timeout=timeout_s * 1000)
            html = page.content()
            browser.close()
        return html

render_tool = RenderTool()

Attach the tools to the right agents:

fetcher.tools = [fetch_tool, render_tool]

Defining tasks

Tasks express what each agent should do and what they should produce. Every task gets an expected_output description that doubles as the validation hint for the LLM.

from crewai import Task

scout_task = Task(
    description=(
        "Discover all product URLs for {target_site} in the {category} category. "
        "Use the sitemap if available. Return a JSON array of URLs."
    ),
    expected_output="JSON array of strings, each a fully qualified product URL",
    agent=scout,
)

fetch_task = Task(
    description=(
        "For each URL in the prior task output, fetch the HTML using fetch_url. "
        "If the response contains less than 1000 characters of body content or "
        "looks like a JS-only shell, retry with render_url. "
        "Return a JSON map of {url: html}."
    ),
    expected_output="JSON object mapping URL to raw HTML",
    agent=fetcher,
    context=[scout_task],
)

extract_task = Task(
    description=(
        "For each {url: html} pair, extract title, price (number), currency (3-letter code), "
        "and in_stock (boolean). Return a JSON array of records."
    ),
    expected_output=(
        "JSON array of objects with keys: url, title, price, currency, in_stock"
    ),
    agent=extractor,
    context=[fetch_task],
)

The context field is how data flows between tasks. CrewAI passes prior task outputs as text to the next agent’s prompt.

Assembling the crew

from crewai import Crew, Process

crew = Crew(
    agents=[scout, fetcher, extractor],
    tasks=[scout_task, fetch_task, extract_task],
    process=Process.sequential,
    verbose=True,
)

result = crew.kickoff(inputs={
    "target_site": "https://www.lazada.sg",
    "category": "ergonomic-keyboards",
})

print(result.raw)

Sequential is the default and the right pick for a linear scraping pipeline. For more complex workflows where one agent should orchestrate others, use Process.hierarchical and provide a manager LLM.

Hierarchical process for adaptive pipelines

The hierarchical process puts a manager agent in charge. The manager decides which worker to call, in what order, with what arguments. This is the right shape when the pipeline shape depends on the input.

from crewai import Crew, Process
from langchain_openai import ChatOpenAI

manager_llm = ChatOpenAI(model="gpt-4o", temperature=0)

crew = Crew(
    agents=[scout, fetcher, extractor],
    tasks=[scout_task, fetch_task, extract_task],
    process=Process.hierarchical,
    manager_llm=manager_llm,
    verbose=True,
)

The manager dispatches dynamically and is the right choice when you cannot enumerate the steps up front. Cost is higher because every dispatch decision is an LLM call. Reserve for genuine adaptivity, not as the default.

Comparing CrewAI to LangGraph and AutoGen

Dimension	CrewAI	LangGraph	AutoGen
Mental model	Org chart of roles	State machine	Group chat
Best fit for scraping	Multi-step pipelines with clear roles	Branching state-driven workflows	Conversational extraction
Tool definition	BaseTool subclass	LangChain Tool	Function decorator
Persistence	Manual	Built-in checkpointer	Manual
Learning curve	Lowest	Moderate	Steep
Production maturity	High	High	Moderate
Native MCP	Via wrappers	Via wrappers	Yes

CrewAI is the framework to pick when the pipeline shape mirrors a team org chart and you want the fastest possible path from idea to working code. LangGraph wins for pipelines with non-linear branching and long-running state. AutoGen wins for conversational extraction where multiple agents debate the right answer.

For the LangGraph alternative, see our scraping with LangGraph agents guide. For AutoGen, see Multi-agent scraping with AutoGen in 2026.

Adding proxy rotation

CrewAI itself does not own the network layer; your tools do. The pattern is to keep a proxy pool in env or in a small singleton, and pick from it inside every fetch tool. We showed this pattern above; here is the production refinement that adds health checking.

import time
from collections import defaultdict

class ProxyPool:
    def __init__(self, proxies):
        self.proxies = proxies
        self.failures = defaultdict(int)
        self.last_used = defaultdict(float)

    def pick(self):
        now = time.time()
        candidates = [p for p in self.proxies if self.failures[p] < 3]
        if not candidates:
            self.failures.clear()
            candidates = self.proxies
        return min(candidates, key=lambda p: self.last_used[p])

    def report(self, proxy, success):
        self.last_used[proxy] = time.time()
        if not success:
            self.failures[proxy] += 1
        else:
            self.failures[proxy] = max(0, self.failures[proxy] - 1)

pool = ProxyPool([os.environ["PROXY_POOL"].split(",")])

For ASEAN ecommerce scraping with mobile IPs that pass strict carrier-level checks, Singapore mobile proxy plugs into this pool directly.

Validator and Reporter agents in detail

The five-role split listed at the top of the article needs two agents we have not yet shown in code. Adding them turns a fragile demo into a production-grade pipeline.

from crewai import Agent

validator = Agent(
    role="Data Quality Auditor",
    goal=(
        "Reject any extracted record that violates business rules. Price must be > 0 "
        "and < 100000. Currency must be a valid ISO 4217 code. Title must be non-empty "
        "and under 500 chars. in_stock must be a bool. Flag suspicious values for review."
    ),
    backstory=(
        "Former data engineer who spent two years cleaning a B2B product catalog. "
        "Allergic to silently incorrect data. Will refuse to pass through anything "
        "that smells wrong, and will document why."
    ),
    llm=llm,
    verbose=True,
)

reporter = Agent(
    role="Insights Reporter",
    goal=(
        "Compare today's extracted prices against yesterday's and emit a Slack-ready "
        "summary highlighting price drops over 10%, new SKUs, and out-of-stock changes."
    ),
    backstory=(
        "Pricing analyst with a journalism background. Writes summaries that a busy "
        "merchandising manager can act on in 30 seconds."
    ),
    llm=llm,
    verbose=True,
)

The Validator agent in particular pays for itself the first time the Extractor mistakes a postcode field for a price and tries to write 94025 to your price_usd column.

Custom Python validation tool

LLMs are fine for fuzzy validation but bad at strict rule checking. Pair the Validator agent with a deterministic tool that does the unforgiving work.

from crewai.tools import BaseTool
from pydantic import BaseModel, ValidationError, Field as PField, conlist
from typing import List, Literal

class ProductRecord(BaseModel):
    url: str
    title: str = PField(min_length=1, max_length=500)
    price: float = PField(gt=0, lt=100000)
    currency: Literal["USD", "SGD", "EUR", "JPY", "GBP", "INR", "MYR", "THB", "IDR", "VND"]
    in_stock: bool

class ValidateInput(BaseModel):
    records: list

class StrictValidatorTool(BaseTool):
    name: str = "strict_validate"
    description: str = (
        "Run deterministic validation on a list of product records. Returns the "
        "subset that passes plus a list of errors for the rejected records."
    )
    args_schema: type[BaseModel] = ValidateInput

    def _run(self, records: list) -> dict:
        passed, errors = [], []
        for r in records:
            try:
                passed.append(ProductRecord(**r).model_dump())
            except ValidationError as e:
                errors.append({"record": r, "errors": e.errors()})
        return {"passed": passed, "errors": errors}

The Validator agent calls strict_validate and uses its output to decide what to forward to the Reporter.

Adding async execution and concurrency

Sequential is the default but real production crews need parallelism. CrewAI 0.86 introduced Process.async_sequential and async-friendly task callbacks. Use them when tasks are independent.

from crewai import Crew, Process

crew = Crew(
    agents=[scout, fetcher, extractor, validator, reporter],
    tasks=[scout_task, fetch_task, extract_task, validate_task, report_task],
    process=Process.sequential,
)

# Process N URLs in parallel by spawning N crews
import asyncio

async def scrape_many(urls):
    sem = asyncio.Semaphore(10)  # cap concurrent crews

    async def one(url):
        async with sem:
            return await crew.kickoff_async(inputs={"target_url": url})

    return await asyncio.gather(*(one(u) for u in urls))

The Semaphore is the single most important line. Without it, 1000 URLs spawn 1000 simultaneous Crew instances, each holding a Playwright browser, and the host runs out of memory in 30 seconds.

Wiring CrewAI into a warehouse

The Reporter agent should not be writing directly to your warehouse. Give it a thin tool that calls a typed function in your data layer.

import asyncpg

class WarehouseWriteTool(BaseTool):
    name: str = "warehouse_write"
    description: str = "Append validated product records to the warehouse."

    def _run(self, records: list) -> dict:
        # synchronous wrapper around an async pool
        return {"written": _sync_warehouse_write(records)}

The reason to keep this thin: if the LLM hallucinates a malformed record that slips through the Validator, the warehouse layer catches it via the column-type contract. Defense in depth.

Cost benchmarks

For a sequential scrape of 100 product URLs with three agents:

Setup	LLM cost	Wall clock
GPT-4o-mini all agents	$0.45	8 min
GPT-4o all agents	$7.20	9 min
Mixed: 4o-mini scout/fetcher, 4o extractor	$1.80	8.5 min
Claude Haiku all agents	$0.55	7 min
Claude Sonnet all agents	$7.80	8 min

The mixed setup is the value sweet spot. Use cheap models for orchestration agents, expensive models only for the agent that needs to read messy HTML and produce clean JSON.

Cost levers in priority order

The fastest wins on CrewAI cost, in the order they pay off:

1. Trim HTML before passing to Extractor. Strip scripts, styles, comments. Cuts tokens by 40 to 70 percent.
2. Use GPT-4o-mini for Scout, Fetcher, Validator, Reporter. Reserve GPT-4o or Sonnet for Extractor only.
3. Cache extractions by HTML hash. Same page extracted twice should not pay LLM twice.
4. Cap max_iter per agent to 8. Default 25 lets confused agents loop expensively.
5. Switch verbose off in production. Verbose mode includes all intermediate thought tokens in the trace which the agent re-reads.

Combined, these cut typical per-page cost by 60 to 80 percent on a tuned pipeline versus a default-config baseline.

Memory and learning

CrewAI agents can be configured with memory that persists across runs. Two flavors: short-term (within a crew run) and long-term (across runs, backed by a vector store).

from crewai.memory import LongTermMemory
from crewai.memory.storage.ltm_sqlite_storage import LTMSQLiteStorage

extractor.memory = True

crew = Crew(
    agents=[scout, fetcher, extractor],
    tasks=[scout_task, fetch_task, extract_task],
    process=Process.sequential,
    memory=True,
    long_term_memory=LongTermMemory(
        storage=LTMSQLiteStorage(db_path="./crew_ltm.db")
    ),
)

For scraping, long-term memory pays off when the same agent learns site-specific quirks across runs. After a few iterations, the extractor remembers that a certain Lazada page renders price in a non-standard div.

Production deployment

Run CrewAI under a worker queue (Celery, RQ, or a Postgres-backed queue) and treat each crew kickoff as a job. Set hard timeouts, log every agent step, and persist the result to your warehouse.

For long-running crews, set step_callback and task_callback to stream progress to your observability stack. The callbacks fire after every agent step and every task completion respectively.

The official CrewAI documentation covers deployment options in depth, including the managed CrewAI Plus platform.

Frequently asked questions

Can CrewAI agents call MCP servers?
Not natively in 0.86. Wrap the MCP server in a Python tool that translates BaseTool calls to MCP JSON-RPC. The community has at least three open-source bridges; pick the one most actively maintained.

How do I prevent runaway agent loops?
Set max_iter on each agent (default 25). For task-level safety, set max_execution_time in seconds. Both options cap cost and clock.

Can I run CrewAI without OpenAI or Anthropic?
Yes. Any LangChain LLM works, including Ollama, vLLM, LM Studio, and Bedrock. Quality drops with smaller open-source models, especially on the extraction agent which needs strong JSON Schema adherence.

Does CrewAI handle parallel agent execution?
Sequential and hierarchical processes are single-threaded. For genuine parallelism, run multiple crew invocations under asyncio or a worker pool.

What about debugging when an agent goes off-script?
Set verbose=True on agents and crew. The terminal trace shows every thought, action, and observation. For richer logging, integrate with LangSmith.

Are CrewAI tasks idempotent? Can I safely retry?
Tasks themselves are idempotent only if your tools are idempotent. The framework will not deduplicate side effects. Wrap the side-effecting step (warehouse write, alert dispatch) with an idempotency key based on the input.

Can the same crew handle multiple sites with different layouts?
Yes, but the Extractor agent benefits from per-site backstory or per-site fewshot examples. The pattern that works is one Crew per site, sharing the Scout/Fetcher/Validator/Reporter agents and swapping only the Extractor.

How does CrewAI compare to writing the same pipeline in plain LangChain?
CrewAI is roughly 30 percent fewer lines of code for typical 3-to-5 agent pipelines and the role-based abstraction makes the intent clearer in code review. Plain LangChain wins when you need fine-grained control over the prompt assembly per turn.

Can I use CrewAI for crawling, not just scraping?
Yes. Add a Crawler agent with a BFS tool that takes seed URLs and depth, and pass the discovered URL list to the Fetcher. CrewAI’s role abstraction handles the recursion via the same Crawler agent re-invoked per depth.

Common production gotchas

A handful of issues bite teams during their first month.

The Extractor agent silently truncates HTML when the input exceeds the LLM’s context window. Always pre-trim with a deterministic rule (strip scripts, styles, hidden divs, and whitespace) before sending to the LLM, and log the trimmed size.

CrewAI’s verbose=True writes to stdout in a hard-to-parse format. For production, replace with a structured logging callback that emits JSON lines you can grep and ship to your log aggregator.

Tool argument schemas are inferred at class load time. Adding a field after the agent has already been constructed is a no-op until you reload. Restart workers on tool changes.

Hierarchical mode’s manager LLM is GPT-4o by default and that single decision dominates the cost on small jobs. Override manager_llm to GPT-4o-mini or a stronger model only when the dispatch decision is hard.

The crew’s kickoff method blocks the calling thread. For async workers, use kickoff_async and await it. Mixing the two in the same process leads to nested event loop errors.

For broader patterns on building agentic scraping in 2026, browse the AI agentic proxies category.