Personal data vs public data in scraping: a 2026 framework

Personal vs public data scraping is the single most consequential classification a scraping team makes about each target. The distinction governs which compliance regime applies, which legal defences are available, what your storage and retention obligations look like, and how you respond to deletion requests. In 2026, the line is more contested than ever, because regulators have taken consistent positions that public availability does not exempt data from privacy regulation, while courts in some jurisdictions have held that public scraping is broadly permissible under contract and computer-misuse statutes. This guide walks through the framework, the regulator and court positions, a working classification matrix, and a workflow your team can operationalise.

The audience is the technical lead, in-house counsel, or product owner who needs to make defensible classification decisions about every scrape target.

Why the classification matters operationally

Each target you classify as personal data triggers obligations: lawful basis documentation under GDPR, opt-out mechanisms under CCPA, consent under PDPA, deletion-on-request under DPDP. Each target you classify as non-personal-public data triggers far fewer obligations: contract analysis, copyright analysis, robots.txt courtesy.

Misclassification is the most common compliance failure. Teams routinely classify data as “public” because the URL was logged-out-readable, when in fact the data identifies natural persons. Or they over-classify, treating every dataset as personal and adding overhead that is not legally required.

A working classification matrix is the single highest-leverage compliance artefact a scraping team builds. It pays for itself the first time a regulator asks “what is your basis for processing?”

For the broader compliance picture, see the GDPR compliance guide, the CCPA compliance guide, and the HiQ Labs ruling explainer.

How major regimes define personal data

The definitions converge but the edges diverge. The pattern is “any data linkable to a natural person, with edge-case carve-outs.”

Regime	Core definition	Key edge case
GDPR (EU)	Information relating to identified or identifiable natural person	IP addresses, cookies count
UK GDPR	Same as EU	Same
CCPA / CPRA (California)	Information that identifies, relates to, describes, or could reasonably be linked to a consumer or household	Household level included
PDPA (Singapore)	Data about an identified individual or identifiable from data	Public availability carve-out broader
DPDP (India)	Digital personal data	Limited public availability carve-out
LGPD (Brazil)	Information relating to identified or identifiable natural person	Mirrors GDPR
PIPL (China)	Information related to identified or identifiable natural persons	Sensitive data category strict
POPIA (South Africa)	Information relating to identifiable living natural person, or existing juristic person	Includes some company data

The two outliers worth flagging: PIPEDA in Canada follows GDPR-style definitions but adds reasonable-purpose tests, and APPI in Japan distinguishes “personal information” from “personal data” with technical processing requirements that catch many scrapers off guard.

The “public” question regulators consistently reject

Across nearly every regime, regulators have taken the position that public availability does not exempt data from the regulation. The clearest statements:

The European Data Protection Board issued a 2024 opinion explicitly stating that scraping publicly available personal data still requires a lawful basis under GDPR. The opinion responded to AI training scrapers who argued that publicly readable data was outside scope.

The California Privacy Protection Agency has issued enforcement guidance reading the “publicly available information” carve-out narrowly, particularly excluding inferences drawn from public data.

The Singapore PDPC has guidance distinguishing “publicly available” (where collection is permitted without consent) from “publicly accessible” (where consent rules still apply). The two terms are routinely conflated in commercial discussions but the PDPC reads them strictly.

The Indian DPDP rules issued in 2025 carry a narrow carve-out for “personal data made publicly available by the data principal,” which excludes data that became public through breach, leak, or aggregation.

The pattern: scrapers who argue “public, therefore exempt” lose the argument with regulators. The legal posture must be “personal, with documented lawful basis” or “not personal at all.”

Classification matrix: what counts as personal

Data element	Personal under GDPR	Personal under CCPA	Personal under PDPA
Full name	Yes	Yes	Yes
Email address	Yes	Yes	Yes
Phone number	Yes	Yes	Yes
IP address	Yes	Yes	Likely yes
Cookie identifier	Yes	Yes	Likely yes
Device fingerprint	Yes	Yes	Yes
Geolocation (precise)	Yes	Yes (sensitive)	Yes
Geolocation (city level)	Likely yes	Yes	Conditional
Job title (anonymised)	No	No	No
Job title + company name	Likely yes	Yes	Yes
Username (re-identifiable)	Yes	Yes	Yes
Username (truly anonymous)	No	Conditional	No
Profile photo	Yes (biometric inference)	Yes	Yes
Forum post (pseudonymous)	Yes	Yes	Yes
Forum post (anonymous)	No	Conditional	No
Aggregated counts (no individual)	No	No	No
Company name only	No	No	No
Company financials (public filings)	No	No	No
Product price	No	No	No
Product review text (with username)	Yes	Yes	Yes
Product review text (anonymised)	No	Conditional	No

Where the column says “conditional” or “likely yes,” the classification depends on context, combination with other fields, and the realistic re-identification risk. The conservative move is to treat as personal and document the assessment.

The combination problem

A single field can be non-personal in isolation and personal in combination. This is the hardest part of the classification.

Consider a scrape target that returns three columns: company name, job title, year of joining. None of the three is personal data on its own. Combined, they may identify a single individual at a small company. The General Data Protection Regulation explicitly requires assessment of “the means reasonably likely to be used” to identify a person, and combination with other available datasets counts.

The practical workflow: enumerate every field you collect, run a re-identification test against your most likely combinations, and treat the resulting dataset as personal if any combination identifies an individual. The enumeration is one-time work per pipeline; the test pays off forever.

For the deeper compliance overlay on this question, see the GDPR compliance guide.

Decision tree for classifying a new scrape target

Q1: Does the dataset contain any direct identifier (name, email, phone, ID)?
    ├── Yes -> Personal data. Apply full compliance regime.
    └── No  -> Q2
Q2: Does the dataset contain any quasi-identifier (location, job, age, gender)?
    ├── No  -> Likely non-personal. Document assessment.
    └── Yes -> Q3
Q3: Can a realistic combination of fields identify an individual?
    ├── Yes -> Personal data. Apply full compliance regime.
    └── No  -> Q4
Q4: Does the dataset contain inference targets (photos, biometric data)?
    ├── Yes -> Personal data. Sensitive category likely.
    └── No  -> Q5
Q5: Is the data linkable to other data you hold or could acquire?
    ├── Yes -> Personal data. Apply full compliance regime.
    └── No  -> Non-personal. Document and proceed.

Each “yes” should escalate to personal-data treatment. The cost of over-classifying once is small; the cost of under-classifying is regulator action.

Pseudonymisation and anonymisation as compliance levers

Pseudonymisation (replacing identifiers with codes while retaining the linkage table) does not remove data from GDPR scope but does reduce risk and unlock several allowances. Anonymisation (irreversible removal of all identifiers) does remove data from GDPR scope, but the bar for “irreversible” is very high.

The EDPB 2024 anonymisation guidance lists three tests a dataset must pass to be considered anonymised: singling out (no individual is uniquely identifiable), linkability (no two records can be linked to the same individual), and inference (no attribute can be inferred about an individual with significant probability).

Most scraped datasets fail at least one test. A dataset of forum posts with usernames stripped still leaves writing-style fingerprints that can be re-identified. A dataset of product reviews with location stripped still leaves time-of-purchase patterns. Treat anonymisation claims sceptically and engineer accordingly.

For practical pseudonymisation patterns, see building an ethics-first scraping policy.

A working classification register

Each scraping pipeline should maintain a classification register with the following columns:

Column	Purpose
Pipeline ID	Internal identifier
Source URL pattern	What you scrape
Fields collected	Every column in your storage schema
Personal data classification (per regime)	Yes/no/conditional per GDPR, CCPA, PDPA, DPDP
Lawful basis (per regime)	LIA reference, consent, contract
Retention period	Days/months
Deletion mechanism	Process and contact
Last reviewed	Date and reviewer

The register is itself an Article 30 record under GDPR and a comparable record under most other regimes. Build it once, maintain it monthly, and you have most of your compliance documentation.

External references

The European Data Protection Board’s opinions on scraping and AI training are at edpb.europa.eu/our-work-tools/our-documents. The CPPA enforcement guidance archive is at cppa.ca.gov. The Singapore PDPC advisory on publicly available data is at pdpc.gov.sg.

Comparison: classification regimes side by side

Regime	Public-data carve-out	Inferred data treatment	Pseudonymisation effect
GDPR	Narrow; lawful basis still required	Personal	Reduces risk; still personal
CCPA	Moderate; requires lawful availability	Personal if linkable	Reduces risk; still personal
PDPA Singapore	Broader; permits collection without consent	Conditional	Reduces risk; still personal
DPDP India	Limited; requires data principal action	Personal	Reduces risk; still personal
PIPL China	Very narrow; consent default	Personal	Reduces risk; still personal
LGPD Brazil	Mirrors GDPR	Personal	Reduces risk; still personal

Build for GDPR and you have most of the surface covered. The Singapore carve-out can ease specific use cases but does not extend to AI training or large-scale aggregation.

A worked example: scraping a public job board

Suppose you scrape a public job board for talent intelligence. The fields you collect include candidate name (where displayed), current job title, current company, years of experience, location (city), skills tags, and a profile photo URL.

Classification: every field except the optional photo URL contributes to identification of an individual. The combination is unambiguously personal data under all major regimes.

Lawful basis: legitimate interest, with a Legitimate Interest Assessment documenting the purpose (talent intelligence for B2B customers), the necessity (no aggregated alternative meets the use case), and the balancing test (candidates publishing on a public board reasonably expect aggregation by recruiters and intelligence vendors; the balance leans towards processing, but with mitigations).

Mitigations: data minimisation (skip photo URL unless specifically needed), retention limits (purge after 12 months), opt-out mechanism (publish a deletion form), and access controls (no resale of identifiable records to third parties).

Outcome: defensible posture, documented in the register, with the LIA on file and the privacy notice published. A regulator inquiry can be answered in a single email with attachments.

For the parallel discussion of how this works for ecommerce competitor data (different classification, different regime), see the personal vs public data framework’s worked retail example.

FAQ

Is data on a public website automatically not personal?
No. Public availability does not change personhood. If the data identifies a natural person, it is personal data, full stop.

Does anonymisation move data out of GDPR scope?
True anonymisation does, but the bar is very high. Most scraped data that engineers call “anonymised” is in fact pseudonymised under EU law and remains in scope.

What if I only collect aggregated counts?
Aggregated counts that do not relate to identifiable individuals are not personal data. The aggregation must be irreversible and the cell size large enough to prevent inference.

How does business-to-business contact data classify?
A name plus a corporate email or job title is personal data under every major regime. Professional context does not remove personhood.

What about AI-generated synthetic data?
If the synthetic data was trained on personal data and can re-identify individuals from the training set, regulators are increasingly treating it as personal data with respect to those individuals. The case law is still developing but the conservative posture is to apply the regime.

Extended definitional analysis

The personal-versus-public distinction is one of the most misunderstood concepts in scraping law. The clarifying frame is that personal and public are independent dimensions, not opposites.

A piece of information can be:
– Personal and public (a LinkedIn profile, a public Twitter post by a named individual).
– Personal and private (a medical record, a salary slip).
– Non-personal and public (weather data, public transit schedules).
– Non-personal and private (an internal company memo without identifiers).

Privacy law (GDPR, CCPA, PDPA, DPDP) regulates the personal axis regardless of public-ness. Most regimes treat publicly available personal data as still personal data subject to most rules. The CCPA carve-out for publicly available information is the partial exception, and it is narrower than commonly assumed.

This creates the most common scraping mistake. A team scrapes public LinkedIn profiles and assumes the public-ness removes privacy obligations. It does not. The data is still personal data under GDPR Article 4(1) and PDPA Section 2. The right to object, the right to erasure, the lawful-basis requirement, and the transparency obligations all apply.

Implementation patterns for the distinction

A 2026 scraping pipeline should classify every record on both axes at ingest.

1. Tag personal data using a deterministic detector for direct identifiers (name, email, phone, address, government ID).
2. Tag indirect identifiers separately (employer, role, location to city precision, photo).
3. Apply the most restrictive regime that applies to the personal-data classification.
4. Apply different retention TTLs to the two classes.
5. Maintain a per-record source URL so deletion requests can find the records.

Code pattern: dual classification at ingest

import re

PERSONAL_PATTERNS = {
    "email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),
    "phone_us": re.compile(r"\+?1?[\s.-]?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}"),
    "ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
}

def classify_record(record):
    text = record.get("text", "")
    is_personal = any(p.search(text) for p in PERSONAL_PATTERNS.values())
    is_public = record.get("source_visibility") == "public"
    return {
        "personal": is_personal,
        "public": is_public,
        "regime": pick_regime(is_personal, record.get("jurisdiction")),
    }

Comparison: how regimes treat the four quadrants

Quadrant	EU GDPR	California CCPA	Singapore PDPA	India DPDP
Personal and public	Full scope	Carve-out partial	Full scope	Full scope
Personal and private	Full scope	Full scope	Full scope	Full scope
Non-personal and public	Out of scope	Out of scope	Out of scope	Out of scope
Non-personal and private	Out of scope	Out of scope	Out of scope	Out of scope

Additional FAQ

Is a public Twitter post personal data?
Yes if it identifies or relates to an identifiable individual. Most named accounts qualify.

What about pseudonymous accounts?
Pseudonymous accounts can still be personal data if re-identification is reasonably possible.

Is corporate information personal data?
A company name is not personal data. An individual employee’s name and title is personal data even though it relates to corporate context.

Does aggregation remove the personal classification?
Statistical aggregates that prevent re-identification can be non-personal. Most scraping aggregates do not meet the strict re-identification threshold.

Real cases that turned on the personal-vs-public distinction

Three regulatory and court actions in 2024-2026 illustrate how the distinction operates in practice.

In Clearview AI v. CNIL (France, 2024 enforcement confirmation), the CNIL imposed a 20 million euro fine against Clearview AI for scraping publicly accessible facial images and constructing a biometric database. Clearview argued the source images were public. The CNIL rejected the argument, holding that public availability does not exempt biometric data from GDPR Article 9 special-category protection. The decision was upheld on appeal in 2025 and is now the leading European precedent for “public does not mean exempt” in the AI training context.

In OpenAI v. Garante (Italy, 2025), the Italian DPA fined OpenAI 15 million euros for processing publicly scraped personal data without an adequate lawful basis under GDPR Article 6, and for failing to provide transparency to data subjects under Article 14. OpenAI argued legitimate interest. The Garante held that the balancing test failed because the data subjects had no reasonable expectation of being included in a generative model’s training corpus. The decision became influential across other EU DPAs as the template for AI training enforcement.

In Hangzhou Internet Court v. Douyin scraper (China, 2024), a Chinese commercial scraping operation collected publicly visible Douyin user names, follower counts, and post metrics for a competitive intelligence product. The court applied PIPL strictly and held that even publicly displayed data about identified natural persons required consent or a statutory basis, and the commercial intelligence purpose did not qualify. The case is the leading PIPL precedent on commercial scraping of publicly visible personal data.

The pattern across all three: regulators and courts consistently treat “public” as orthogonal to “personal,” and a scraping operation that conflates the two faces material liability. The defensible posture is to treat publicly visible personal data as personal data with a documented lawful basis, not as exempt by virtue of public visibility.

How regulators apply the personal-versus-public distinction

Regulators across major jurisdictions have repeatedly clarified that publicly available personal data remains personal data under the relevant privacy statute. The European Data Protection Board, the UK Information Commissioner’s Office, the Italian Garante, the French CNIL, the Singapore Personal Data Protection Commission, and the California Privacy Protection Agency have each issued guidance to that effect during 2023-2026.

The clearest articulation is in the EDPB’s 2024 guidance on scraping for AI training. The guidance states that the public availability of data does not by itself create a lawful basis for processing under GDPR. The controller must still identify a lawful basis under Article 6 and, where special category data is involved, under Article 9. The same conclusion applies to retention, transparency, and rights handling.

The CCPA carve-out for publicly available information is the partial exception, but the CPPA has consistently interpreted the carve-out narrowly. Information lawfully made available from federal, state, or local government records is the core. Information that an individual has chosen to make available in a manner consistent with the purpose is also covered, but commercial platforms with terms of service restricting bulk access do not satisfy the latter prong.

The reasonable expectation of privacy doctrine

A useful conceptual frame is the reasonable expectation of privacy doctrine, originally developed in US Fourth Amendment law. The doctrine asks whether a person has manifested a subjective expectation of privacy and whether that expectation is one that society recognises as reasonable.

For scraping the doctrine maps as follows. A person who posts on a public Twitter account has manifested a reduced expectation of privacy in the content of those posts. The person retains a higher expectation regarding aggregation, profiling, and downstream resale. A scraper that respects the original posting intent is on stronger footing than a scraper that aggregates and resells.

The doctrine is not directly applicable to GDPR or CCPA, but it informs the proportionality analysis in both regimes. Regulators ask whether the scraping operation respects the reasonable expectations the data subject would have had at the time of original publication. A scrape that aligns with those expectations is more likely to pass muster.

Indirect identifiers and re-identification risk

Direct identifiers (name, email, government ID) are the easy case. Indirect identifiers (employer, role, city, photo) and quasi-identifiers (ZIP code, date of birth, gender) raise the harder question of re-identification risk.

The Sweeney 2000 study established that 87 percent of the US population could be uniquely identified by ZIP code, date of birth, and gender. Subsequent research extended the analysis to richer attribute sets. The implication for scrapers is that combinations of seemingly innocuous attributes can identify individuals.

The 2026 best practice for scrapers handling indirect identifiers is to apply k-anonymity at the analytical layer (ensuring at least k records share each quasi-identifier combination), to apply differential privacy to aggregates, and to suppress or generalise quasi-identifiers when the combination becomes too specific.

The classification of indirect-identifier-rich data as personal data depends on whether re-identification is reasonably likely. Under GDPR Recital 26 the test considers all means reasonably likely to be used. The threshold is low in practice because attackers have access to many auxiliary datasets.

Next steps

The single highest-leverage action this week is to enumerate every field your pipelines collect and run them through the classification matrix above. Build the register. Two hours of work creates the foundation for every downstream compliance conversation. For deeper compliance, head to the DRT compliance hub and pair this with the GDPR and CCPA guides.

This guide is informational, not legal advice.

Multi-agent scraping with AutoGen in 2026

AutoGen scraping multi-agent setups have matured into a real production option after Microsoft Research shipped AutoGen v0.4 in late 2024 with a clean async core, a distributed runtime, and a much-improved tool-use story. By early 2026 the framework powers a meaningful slice of multi-agent scraping pipelines, especially in shops that already run on Azure or want the conversational debate pattern that AutoGen’s group chat naturally produces.

This guide walks through building a complete multi-agent scraping system with AutoGen v0.4. We define the agent topology, wire tools, run a group chat to scrape and validate ecommerce data, and benchmark cost and quality against single-agent and other multi-agent frameworks.

Why AutoGen for multi-agent scraping

AutoGen’s defining feature is the conversational pattern. Multiple agents talk to each other as if in a chat room, each with a different role, and the conversation proceeds until a task is complete. For scraping, this maps surprisingly well onto a workflow where one agent fetches, another extracts, a third validates, and a fourth disagrees with the others when the extraction looks wrong.

That last bit is the differentiator. Other multi-agent frameworks struggle to express disagreement. AutoGen’s group chat makes it natural. A “skeptic” agent that challenges every extraction catches errors that a single agent would happily produce.

AutoGen v0.4 also ships a distributed runtime. Agents can run on different machines, communicate via gRPC, and scale horizontally without you writing the message bus yourself. For high-volume scraping pipelines, this is a real architectural win.

Installing v0.4

AutoGen split into multiple packages in v0.4. The base needs autogen-core and autogen-agentchat. For OpenAI integration, add autogen-ext.

pip install "autogen-agentchat==0.4.3" "autogen-ext[openai]==0.4.3" \
            playwright httpx pydantic
playwright install chromium

export OPENAI_API_KEY="sk-..."

For Anthropic models in AutoGen v0.4, the community package autogen-ext-anthropic works.

Defining model clients

AutoGen v0.4 separates model clients from agents, which is a clean break from v0.2.

from autogen_ext.models.openai import OpenAIChatCompletionClient

cheap = OpenAIChatCompletionClient(model="gpt-4o-mini", temperature=0)
strong = OpenAIChatCompletionClient(model="gpt-4o", temperature=0)

Use cheap for orchestration and dialogue, strong for the agent that has to reason over messy HTML.

Building scraping tools

Tools in AutoGen v0.4 are async Python functions with type hints. The framework infers the schema.

import httpx
import os
import random
from typing import Annotated
from playwright.async_api import async_playwright

PROXIES = os.environ.get("PROXY_POOL", "").split(",")

async def fetch_url(
    url: Annotated[str, "URL to fetch"],
    timeout_s: Annotated[int, "Request timeout in seconds"] = 30,
) -> str:
    """Fetch a URL through the rotating proxy pool. Returns HTML or an error message."""
    proxy = random.choice(PROXIES) if PROXIES and PROXIES != [""] else None
    try:
        async with httpx.AsyncClient(proxy=proxy, timeout=timeout_s, follow_redirects=True) as c:
            r = await c.get(url, headers={"User-Agent": "Mozilla/5.0"})
            return f"HTTP {r.status_code}\nFinal URL: {r.url}\n\n{r.text[:200000]}"
    except Exception as e:
        return f"FETCH_ERROR: {e}"

async def render_url(
    url: Annotated[str, "URL to render with headless Chromium"],
    timeout_s: Annotated[int, "Timeout in seconds"] = 30,
) -> str:
    """Render a URL with Playwright. Returns HTML after JS executes."""
    try:
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True)
            page = await browser.new_page()
            await page.goto(url, wait_until="networkidle", timeout=timeout_s * 1000)
            html = await page.content()
            await browser.close()
        return html[:200000]
    except Exception as e:
        return f"RENDER_ERROR: {e}"

async def store_record(
    record_json: Annotated[str, "JSON record to persist"],
) -> str:
    """Persist a validated extraction record. Returns confirmation."""
    import json
    from pathlib import Path
    rec = json.loads(record_json)
    out = Path("scraped_records.jsonl")
    with out.open("a") as f:
        f.write(json.dumps(rec) + "\n")
    return f"stored: {rec.get('url', '')}"

Defining the agents

AutoGen v0.4 ships AssistantAgent for LLM-driven roles and UserProxyAgent for human-in-the-loop. For full automation we use only AssistantAgent.

from autogen_agentchat.agents import AssistantAgent

fetcher = AssistantAgent(
    name="Fetcher",
    model_client=cheap,
    tools=[fetch_url, render_url],
    system_message=(
        "You are a Fetcher. Given a URL, fetch its HTML using fetch_url first. "
        "If the response is empty or looks like a JS shell, retry with render_url. "
        "Return the HTML to the group exactly as the tool returned it."
    ),
)

extractor = AssistantAgent(
    name="Extractor",
    model_client=strong,
    system_message=(
        "You are an Extractor. Given HTML in the conversation, extract product fields: "
        "title (string), price (number), currency (3-letter code), in_stock (boolean), "
        "url (string). Return strict JSON only. If a field cannot be determined, use null."
    ),
)

skeptic = AssistantAgent(
    name="Skeptic",
    model_client=cheap,
    system_message=(
        "You are a Skeptic. Review extractions from the Extractor. "
        "Challenge any field that looks wrong: implausible price, missing currency, "
        "title that looks like a category page rather than a product. "
        "If the extraction is correct, respond with APPROVED. If not, explain the problem."
    ),
)

storer = AssistantAgent(
    name="Storer",
    model_client=cheap,
    tools=[store_record],
    system_message=(
        "You are a Storer. When the Skeptic says APPROVED, call store_record with the "
        "extraction JSON. Then say STORED."
    ),
)

Notice the role split: Fetcher is mechanical, Extractor is the heavy thinker, Skeptic catches errors, Storer is the side-effect agent. Each does one job.

Running the group chat

import asyncio
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import TextMentionTermination

async def main():
    termination = TextMentionTermination("STORED")
    team = RoundRobinGroupChat(
        participants=[fetcher, extractor, skeptic, storer],
        termination_condition=termination,
        max_turns=12,
    )

    result = await team.run(task=(
        "Scrape this URL and extract the product record: "
        "https://www.lazada.sg/products/example-12345.html"
    ))

    for msg in result.messages:
        print(f"[{msg.source}] {msg.content[:300]}")

asyncio.run(main())

RoundRobinGroupChat rotates through participants in order. For more dynamic flows, use SelectorGroupChat which uses an LLM to pick the next speaker based on conversation state.

Selector group chat for adaptive flows

from autogen_agentchat.teams import SelectorGroupChat

selector_prompt = """
Read the conversation. Pick the next agent to speak.
Agents: Fetcher, Extractor, Skeptic, Storer.
Rules:
- Fetcher speaks when there is no HTML yet or the last fetch failed.
- Extractor speaks when fresh HTML is in the conversation.
- Skeptic speaks after Extractor returns JSON.
- Storer speaks when Skeptic says APPROVED.
Return only the agent name.
"""

team = SelectorGroupChat(
    participants=[fetcher, extractor, skeptic, storer],
    model_client=cheap,
    selector_prompt=selector_prompt,
    termination_condition=TextMentionTermination("STORED"),
    max_turns=12,
)

The selector pattern is more flexible but adds an LLM call per turn. Worth the cost when the workflow shape genuinely depends on state.

Comparing to LangGraph and CrewAI

Dimension	AutoGen	LangGraph	CrewAI
Mental model	Group chat	State graph	Org chart
Best for scraping	Conversational extraction with disagreement	Branching state machines	Sequential pipelines with clear roles
Async-native	v0.4 yes	Yes	Yes
Distributed runtime	Yes (gRPC)	Manual	Manual
Tool definition	Function decorator	LangChain Tool	BaseTool subclass
Selector flexibility	Round-robin or LLM selector	Conditional edges	Sequential or hierarchical
Maturity in 2026	Stable v0.4	Stable 0.4	Stable 0.86

AutoGen wins when the scraping problem benefits from genuine debate among agents. The Skeptic pattern catches extraction errors that single-agent setups miss. LangGraph wins when the flow is a state machine. CrewAI wins when the flow is a clean pipeline.

For the LangGraph alternative see scraping with LangGraph agents. For CrewAI, see CrewAI for scraping pipelines.

Distributed runtime for scale

The headline AutoGen v0.4 feature is the distributed runtime. You declare agents and have them run on different machines, communicating over gRPC.

from autogen_core import SingleThreadedAgentRuntime, AgentRuntime
from autogen_ext.runtimes.grpc import GrpcWorkerAgentRuntime

# host
runtime_host = GrpcWorkerAgentRuntime(host_address="0.0.0.0:50051")

# worker
runtime_worker = GrpcWorkerAgentRuntime(host_address="host.example.com:50051")
await runtime_worker.start()

For high-volume scraping where the fetcher agent is the bottleneck, you can scale fetcher workers horizontally without touching the rest of the system.

When the distributed runtime is overkill

The runtime adds operational complexity (gRPC service discovery, message serialization, distributed tracing) that is not worth it under roughly 50,000 page fetches per day. Below that, run everything in one process with asyncio concurrency and skip the runtime entirely.

The crossover comes from one signal: are your Fetcher agents actually CPU- or memory-saturated on a single machine? If yes, distribute. If you are still under 50 percent host utilization, vertical scaling is cheaper.

Concrete topology examples

The two topologies that handle 90 percent of real scraping work in 2026:

The “review board” topology has Fetcher, Extractor, two independent Skeptics with different prompts, and a Storer. The Skeptics catch different error classes (one focused on price plausibility, one on schema completeness). When they agree, Storer fires. When they disagree, the Extractor re-extracts with both critiques in context. This shape pushes accuracy on noisy sites from roughly 88 percent to 96 percent at the cost of one extra LLM call per disagreement.

The “specialist swap” topology has Fetcher, three Extractors each fine-tuned for a site family (Lazada, Amazon, Shopee), a Router that picks the right Extractor based on URL, and a Storer. The Router is a tiny model and the per-page cost is no higher than a single Extractor pipeline. Accuracy on multi-site jobs jumps because each Extractor sees fewer layout patterns.

Adding proxy rotation

Proxy rotation lives in your tools, not the agents. The fetch_url tool above already pulls from PROXY_POOL. For ASEAN scraping with carrier-clean mobile IPs, Singapore mobile proxy plugs into the pool.

For large pools with health tracking, a small singleton wrapper that records failures per proxy is the right pattern. AutoGen agents do not need to know about it; they just call fetch_url.

Quality benchmarks: where the Skeptic earns its keep

Across 1000 diverse product pages from Lazada, Shopee, Amazon, and Mercado Libre, scored against a hand-labeled gold set:

Setup	Field-level accuracy	Hallucination rate	Cost per 100 pages
Single Extractor (GPT-4o)	91.2%	4.1%	$4.80
Two-agent (Extractor + Storer)	91.4%	3.9%	$4.95
Three-agent with Skeptic	95.7%	1.6%	$5.40
Four-agent with two Skeptics	96.8%	0.9%	$6.20
Five-agent with site-routed Extractors	97.3%	0.7%	$5.95

The headline: a single Skeptic agent cuts hallucination rate by more than half for a 12 percent cost increase. The second Skeptic adds diminishing returns. The site-routed Extractor is the best value because per-Extractor specialization improves both accuracy and cost.

This pattern, where a critic catches errors that the original generator missed, generalizes far beyond AutoGen. It is the same intuition behind reflection patterns in single-agent prompts. AutoGen just makes it explicit and easy to extend.

Streaming and live progress

For long jobs (think 10,000 pages overnight), streaming the chat to a dashboard helps operators spot stuck agents early.

async for event in team.run_stream(task=task):
    if hasattr(event, "source") and hasattr(event, "content"):
        await dashboard_publish({
            "ts": time.time(),
            "agent": event.source,
            "snippet": str(event.content)[:200],
        })

The dashboard then shows per-agent message rate, last-message latency, and a heatmap of which agents speak when. Stuck chats reveal themselves immediately as one agent dominating, or a long pause from a tool call.

Cost benchmarks

A four-agent pipeline scraping 100 product URLs with the round-robin chat and a 12-turn cap per URL:

Model mix	LLM cost	Wall clock
All gpt-4o-mini	$0.62	11 min
Mini for chat, 4o for Extractor	$2.40	11 min
All gpt-4o	$11.50	12 min
All Claude 3.5 Haiku	$0.78	9 min
Haiku for chat, Sonnet for Extractor	$2.85	10 min

The mixed setup with cheap chat and strong extractor is the value pick. Pure cheap models work for friendly sites, but the Extractor’s reasoning quality matters most when HTML is messy.

Cost levers worth pulling

In priority order:

1. Trim the chat history aggressively. Past turn 6 the Skeptic and Storer rarely benefit from earlier turns. Use BufferedChatCompletionContext(buffer_size=6).
2. Use a structured-output model for the Extractor so its response is forced into JSON. Saves the chat from spending tokens parsing free-text JSON.
3. Cap max_turns at 12. Even with the Skeptic disagreeing, no productive scrape needs more.
4. Run the Skeptic on a smaller model than the Extractor. Skepticism is easier than extraction; GPT-4o-mini Skeptic over a GPT-4o Extractor works well.
5. For repeat URLs, hash the HTML and cache the extraction. The chat ends in one turn on cache hit.

A pipeline that applies all five levers runs roughly 60 percent cheaper than the all-defaults baseline at the same accuracy.

Production deployment

Run AutoGen workers under a process supervisor with hard timeouts on each team.run call. Set max_turns to bound chat length. Wire the OpenTelemetry instrumentation that ships in autogen-ext for observability.

For replay and debugging, save the full message history per run. AutoGen’s message objects are JSON-serializable.

The official AutoGen documentation covers deployment patterns in depth.

Observability and tracing

AutoGen v0.4 ships first-class OpenTelemetry instrumentation. Span attributes use the namespace autogen.* and follow the GenAI semantic conventions where possible.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317")))
trace.set_tracer_provider(provider)

The trace shows one span per team.run, nested spans per agent turn, and child spans for each tool call. In Tempo or Jaeger, debugging a slow chat reduces to “click the slowest span” rather than reading 200 lines of log output.

Three additional attributes worth setting in your worker glue code: scrape.url, scrape.queue, scrape.batch_id. These let you slice the trace by domain, by queue, or by batch.

A complete production runner

Putting the patterns together, here is a runner that wires AutoGen, the proxy pool, OTel, retries, and a Postgres queue:

import asyncio, asyncpg, os, time
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import TextMentionTermination, MaxMessageTermination

async def process_url(pool, url):
    team = RoundRobinGroupChat(
        participants=[fetcher, extractor, skeptic, storer],
        termination_condition=TextMentionTermination("STORED") | MaxMessageTermination(12),
        max_turns=12,
    )
    try:
        result = await asyncio.wait_for(team.run(task=f"Scrape {url}"), timeout=120)
        await pool.execute(
            "UPDATE scrape_queue SET status='done', finished_at=now() WHERE url=$1", url
        )
    except asyncio.TimeoutError:
        await pool.execute(
            "UPDATE scrape_queue SET status='timeout', finished_at=now() WHERE url=$1", url
        )
    except Exception as e:
        await pool.execute(
            "UPDATE scrape_queue SET status='error', error=$2, finished_at=now() "
            "WHERE url=$1", url, str(e)
        )

async def worker_loop():
    pool = await asyncpg.create_pool(os.environ["PG_URL"])
    sem = asyncio.Semaphore(5)
    while True:
        rows = await pool.fetch(
            "UPDATE scrape_queue SET status='running' WHERE id IN ("
            "SELECT id FROM scrape_queue WHERE status='pending' LIMIT 20"
            ") RETURNING url"
        )
        if not rows:
            await asyncio.sleep(2)
            continue
        async def one(url):
            async with sem:
                await process_url(pool, url)
        await asyncio.gather(*(one(r["url"]) for r in rows))

asyncio.run(worker_loop())

This runner survives worker restarts, bounds concurrency, records errors, and integrates cleanly with the OTel tracing above. It is roughly 60 lines of glue around an AutoGen team that does the actual work.

Frequently asked questions

Can AutoGen call MCP servers?
Yes. The autogen-ext-mcp community package wraps MCP tools as AutoGen tools. Schema translation is automatic.

How does AutoGen handle long context?
Group chat history can balloon fast. Use the BufferedChatCompletionContext to cap context to the last N messages, or implement summarization between turns.

Does AutoGen v0.4 work with local LLMs?
Yes. Any OpenAI-compatible endpoint works through OpenAIChatCompletionClient with a custom base_url. Ollama, vLLM, and LM Studio all integrate cleanly.

What is the migration path from AutoGen v0.2 to v0.4?
Significant. Tool definitions changed, agent classes renamed, group chat APIs different. Microsoft published a v0.4 migration guide that walks the major changes.

Can I use AutoGen with browser-use or Playwright agents?
Yes. Wrap the agentic browser as a tool that the AutoGen Fetcher calls. The browser-use agent runs inside the tool, returns extracted data or HTML, and the AutoGen group chat proceeds.

Can I run AutoGen entirely on Azure OpenAI?
Yes. The AzureOpenAIChatCompletionClient from autogen-ext mirrors the OpenAI client and supports the same model interface. This is the path most enterprise teams take.

How does AutoGen v0.4 compare to OpenAI Swarm?
Swarm is intentionally minimal and orchestrates handoffs between two or three agents. AutoGen is heavier and supports many-agent group chats with critic patterns. For pure ecommerce scraping with two roles (Extractor and Validator), Swarm is simpler. For workflows that benefit from disagreement, AutoGen wins.

How do I prevent two agents from talking past each other in a long chat?
Use the SelectorGroupChat with a tight selector prompt that names termination conditions. The most common antipattern is round-robin chat without a termination condition, which lets agents take roundabout turns indefinitely. Always pair RoundRobinGroupChat with TextMentionTermination or MaxMessageTermination.

Can I add a human reviewer to the group chat?
Yes. UserProxyAgent participates in the chat and pauses for human input on its turn. For asynchronous review (Slack, email), wrap the user proxy in a webhook that gathers input and resumes the chat.

Does AutoGen v0.4 support streaming responses?
Yes. team.run_stream yields message events as they happen. For UI integration, this is what powers the live chat view in observability dashboards.

Common production gotchas

• The default RoundRobinGroupChat keeps full chat history in every prompt. Past 20 turns, context costs explode. Cap with BufferedChatCompletionContext.
• TextMentionTermination is case-sensitive by default. The Storer agent must say “STORED” exactly as written, or the chat keeps spinning.
• AutoGen v0.4 changed how tool errors propagate. Errors raised from inside an async tool function become user-visible messages in the chat, which the next agent reads as part of the conversation. Catch and format errors carefully.
• The OpenTelemetry exporter is opt-in. Without it, debugging a stuck chat across distributed workers is painful.
• The selector_prompt for SelectorGroupChat is a single string. For complex selection logic, the prompt grows large and starts to dominate cost. At that point a custom Selector callable is cleaner than a longer prompt.
• AutoGen does not retry tool errors automatically. Wrap each tool function with a small retry decorator if you want resilience without involving the chat in error handling.
• Token usage attribution per agent is not exposed by default. Use the OTel hooks to record per-span token counts if you need cost attribution.

If you are picking a multi-agent framework for a new scraping initiative, the AI agentic proxies category has head-to-head writeups that cover the proxy and infrastructure layer too.

Fair use and copyright for AI training data in 2026

Fair use AI training data is the most contested doctrine in the entire AI copyright debate, and 2026 is the year the doctrine finally has live precedent. From late 2023 through 2025, a series of cases (Authors Guild v OpenAI, NYT v OpenAI, Getty Images v Stability AI, Doe v Github, Universal v Anthropic) gave courts the chance to articulate how the four-factor fair use test applies to scraping copyrighted content for model training. The answers are nuanced. Some have changed the cost structure of building a frontier model. Some have changed what scrapers can defensibly collect. This guide walks through the doctrine, the cases, the practical implications, and a 2026 compliance posture for teams scraping for AI training.

The audience is the data engineer, ML practitioner, and product owner whose pipeline includes scraping copyrighted material as input to a model.

What the four-factor fair use test actually asks

US fair use is codified at 17 U.S.C. Section 107. The statute identifies four factors that courts weigh:

1. The purpose and character of the use, including whether it is commercial or transformative.
2. The nature of the copyrighted work.
3. The amount and substantiality of the portion used in relation to the whole.
4. The effect of the use upon the potential market for or value of the copyrighted work.

For AI training, the first and fourth factors do most of the work. Factor one asks whether the model’s use of the data is transformative (typically yes, because training learns statistical patterns rather than reproducing the work) and commercial (typically yes for commercial models). Factor four asks whether the trained model substitutes for or otherwise harms the market for the original work. This is the factor that has dominated the 2024-2025 case law.

For the broader compliance picture across regimes, see the GDPR compliance guide and the personal vs public data scraping framework.

The pre-AI fair use precedents that still matter

Three pre-AI cases anchor the doctrine. Authors Guild v Google (2015, 2nd Cir) held that Google’s mass scanning of in-copyright books to build a search index was fair use. The use was highly transformative (search snippets are not substitutes for the books), the amount was technically large but functionally limited (snippets were truncated), and the market effect was minimal or positive (snippets drove book discovery and sales).

Authors Guild v HathiTrust (2014, 2nd Cir) reached a similar result for academic library digitisation: transformative purpose, no market substitute.

Field v Google (2006, D Nev) held that Google’s caching of web pages for search was fair use, with explicit weight given to the fact that website owners can use robots.txt to opt out (the implied licence theory).

Each of these cases carries forward into the AI training debate but with critical differences: AI models are themselves potential market substitutes for the original content in ways that search snippets are not.

The 2024-2025 AI training case law

NYT v OpenAI (filed 2023, partial summary judgement 2025) is the most-watched case in the entire space. The Times alleged that OpenAI scraped its archive, that GPT models can be prompted to reproduce verbatim Times articles, and that ChatGPT-as-a-product directly substitutes for Times search and reading. The 2025 partial ruling rejected OpenAI’s motion to dismiss on factor four, finding that the Times had plausibly alleged market substitution. The case is heading to trial in 2026 and the discovery has produced extraordinary disclosures about training data composition.

Authors Guild v OpenAI (consolidated 2023-2024) addressed the same scraping question for in-copyright books. The court has not yet issued a fair use ruling but allowed the case to proceed on factor four.

Getty Images v Stability AI (filed 2023, UK and US) addressed image training. Stability AI scraped Getty’s image collection (with Getty’s watermarks visible in some generated outputs), and Getty alleged copyright and trademark infringement. The UK ruling in 2025 found a clear copying claim; the fair use analogue (fair dealing under UK law) failed because the use was deemed too commercial and the market effect on Getty’s licensing business was real.

Doe v Github (2024, ND Cal) addressed code scraping for Copilot training. The court allowed the case to proceed but signalled scepticism about pure-fair-use defences when the model can reproduce code with attribution stripped.

Universal v Anthropic (filed 2024) addressed scraping of song lyrics. Settled in 2025 with Anthropic agreeing to filter training data and outputs. The settlement implies a defendant’s view that the four-factor test was not a clean win.

The collective signal from these cases: fair use for AI training is not dead, but it is much narrower than the most aggressive 2022-2023 readings suggested. Factor four is doing the work, and where the model can substitute for the original work in the market, fair use loses.

EU TDM exceptions: a different framework

The EU has a different doctrine. The Directive on Copyright in the Digital Single Market (DSM Directive, 2019) created two text and data mining (TDM) exceptions:

Article 3 TDM: a mandatory exception for research organisations and cultural heritage institutions. Lawful access required. No opt-out by rightsholders.

Article 4 TDM: a broader exception for any TDM purpose, including commercial AI training. Lawful access required. Rightsholders can reserve their rights through machine-readable means (the “opt-out” mechanism, often via the TDM Reservation Protocol or robots.txt-style directives).

The Article 4 opt-out is what reshaped European AI training in 2024-2025. Major publishers began publishing TDM-Reservation headers en masse, signalling that their content was off-limits to commercial training. Scrapers operating in the EU now must check for the opt-out and respect it; ignoring it strips the Article 4 defence.

The EU AI Act (2024) layers transparency obligations on top: any general-purpose AI model placed on the EU market must publish a “sufficiently detailed summary” of its training data. This forces a level of training-data disclosure that fundamentally changes the legal posture.

For the broader robots.txt and AI directive landscape, see robots.txt and modern scraping ethics.

Practical compliance posture for 2026 AI training

A scraper operating an AI training pipeline in 2026 should adopt seven practices:

1. Maintain a training data manifest. For every dataset, record the source URL set, the date of collection, the user agent used, the robots.txt state at collection time, the TDM-Reservation state at collection time, and any rights metadata.

2. Honour AI-specific user agent directives. If you scrape under an AI-bot identity (GPTBot equivalent), make that identity public and respect the directives addressed to it.

3. Honour TDM-Reservation signals. Implement a parser for both robots.txt-style and HTTP header TDM-R signals; skip content where rights are reserved.

4. Filter training corpora for opt-out reaffirmation. If a content owner publishes a TDM-Reservation after the initial scrape, re-honour it on the next training cycle.

5. Implement training-data deduplication and memorisation reduction. Models that memorise verbatim are factor-three losers (using the substantial whole) and factor-four losers (substituting for the original).

6. Build output filters for known copyrighted material. If your model can reproduce a chunk of a copyrighted work with high fidelity on prompt, factor four is plausibly satisfied at output time even if it was a stretch at training time.

7. Maintain a publish-ready training data summary. The EU AI Act requires it; US litigation discovery effectively requires it.

Decision tree: is this scrape defensible for AI training?

Q1: Is the source URL publicly accessible without bypassing controls?
    ├── No  -> Skip; fair use does not rescue unauthorised access.
    └── Yes -> Q2
Q2: Does the source publish AI-specific opt-out directives (robots.txt, TDM-R)?
    ├── Yes -> Honour the opt-out; skip.
    └── No  -> Q3
Q3: Is the content in-copyright (i.e., not public domain or open licence)?
    ├── No  -> Proceed; verify licence terms.
    └── Yes -> Q4
Q4: Is the model commercial?
    ├── Yes -> Factor 1 leans against; proceed only with strong factor 4 story.
    └── No  -> Factor 1 leans for; proceed with documentation.
Q5: Will the model plausibly substitute for the source content in the market?
    ├── Yes -> Factor 4 likely lost; reconsider inclusion or filter outputs.
    └── No  -> Strongest defensive posture.

Each branch produces a documented decision in the manifest. That manifest is what your discovery response and EU AI Act summary will cite.

A working training-data filter checklist

Control	What it requires	Why it matters
robots.txt parser	RFC 9309 compliant	Factor 1 good faith
TDM-Reservation parser	HTTP and meta tag	EU Article 4 defence
AI user agent identity	Public, attributable	Allows site to set per-purpose rules
Training data manifest	Per-dataset record	Discovery and EU AI Act
Deduplication	Across training corpus	Reduce memorisation
Output similarity filter	At inference time	Factor 4 mitigation
Re-honour opt-outs	Pre-each training cycle	Ongoing good faith
Licence metadata capture	Where present	Public domain and open licence proof
Sensitive content filter	Personal data, medical, legal	GDPR and special categories
Provenance tracking	Source-to-token traceable	Audit response

A team that ticks every row above can credibly defend a fair use posture in the US and a TDM exception posture in the EU. A team that ticks fewer than half is defending in court.

What about derivative works and outputs?

Fair use applies to your training inputs. It does not necessarily apply to your model’s outputs. If your model produces output that is substantially similar to a training input, that output is itself potentially infringing, regardless of the input’s fair use status.

This is the practical lesson from Doe v Github and Getty v Stability AI: training-data fair use does not buy you output-side immunity. Build the output filter. Test for memorisation. Watermark or filter outputs that score above a similarity threshold against known copyrighted works.

The cost of an output filter is real (latency, false positives) but small compared to the litigation cost of a model that reproduces copyrighted material on demand.

External references

The US Copyright Office published a multi-part report on AI and copyright in 2024-2025; the relevant volume is at copyright.gov/policy/artificial-intelligence. The EU DSM Directive (2019/790) is at eur-lex.europa.eu/eli/dir/2019/790/oj. The TDM Reservation Protocol draft is at w3c.github.io/tdmrep.

Comparison: fair use vs Article 4 TDM exception

Dimension	US Fair Use	EU Article 4 TDM
Default posture	Defensive (factor analysis)	Permissive (subject to opt-out)
Lawful access required	Implicit	Explicit
Opt-out by rightsholder	None (informal robots.txt)	Mandatory machine-readable
Commercial use	Allowed if factors balance	Allowed unless opted out
Transparency obligation	None statutory	EU AI Act mandates summary
Cure for opt-out after scraping	None (factor 4 may bite)	Re-honour on next cycle
Predictability for training operators	Low (case-by-case)	Higher (clearer rules)

The EU framework is more predictable but more expensive (you must build the opt-out parser). The US framework is less predictable but offers more flexibility for early-stage research.

Special cases: code, images, and journalism

Code scraping (think GitHub) carries copyright but also broad open-source licensing. The challenge is that licences attach to specific files, and a training corpus aggregates millions. Doe v Github held that GPL-style attribution requirements may survive aggregation, meaning that models trained on GPL code can produce output that strips attribution required by the licence. Build the filter.

Image scraping (think Getty) carries copyright plus database rights in the EU. Watermarks and metadata provenance matter. A model that reproduces a Getty watermark on output is in the worst possible factor-four position.

Journalism scraping (think NYT) carries copyright and increasingly investment-protection statutes (Germany, Australia, Canada). The factor-four story is hardest here because models that summarise news directly substitute for the publisher.

For a forward-looking discussion of how RAG over scraped journalism corpora handles these risks, see RAG over scraped data.

FAQ

Is scraping for AI training fair use?
The 2026 answer is “sometimes.” The four-factor test applies, factor four is doing most of the work, and market substitution by the model is the question to focus on.

Can I rely on robots.txt to opt me into fair use?
The Field v Google implied-licence theory still has weight, but only as one signal among many. Robots.txt compliance helps factor one (good faith) and may bear on factor four.

Does the EU framework apply if I am US-based?
If you place the model on the EU market, yes. The EU AI Act has explicit extraterritorial reach.

What about open-licence content like Creative Commons?
The licence governs. CC-BY content can be used with attribution. CC-NC content cannot be used commercially. CC0 content has no restrictions. Always read the specific licence.

What is the safest training data posture in 2026?
Combine permissive open data, licensed datasets where available, scraped data with explicit opt-out compliance, and an output filter for memorisation. Document everything in a training manifest.

Extended case law analysis 2024-2026

The fair use doctrine for AI training data sharpened considerably between 2024 and 2026. Three cases shape the current landscape.

The Authors Guild v OpenAI consolidated litigation (Southern District of New York) reached a summary judgment phase in late 2025 on the question of whether training a large language model on copyrighted books constitutes fair use. The court applied the Warhol Foundation v Goldsmith framework and focused on transformativeness in the first factor and market harm in the fourth. The training-as-fair-use defence narrowed where the model could output substantially similar text to the underlying work.

Thomson Reuters v Ross Intelligence (District of Delaware, February 2025) was the first published opinion directly rejecting a fair use defence for training a competing AI on the plaintiff’s headnotes. The court emphasised commercial use, low transformativeness, and direct market harm.

Andersen v Stability AI (Northern District of California, ongoing) is testing the same framework for diffusion-model image training. The 2024 motion to dismiss largely survived for the artists, signalling that the courts will not dismiss these cases at the pleading stage.

The pattern across these cases is that fair use for AI training is more vulnerable when the trained model can reproduce protected expression, when the training market overlaps the licensed market, and when the training is commercial.

Implementation patterns for fair-use-defensible scraping

A 2026 AI training data pipeline that wants the strongest fair use posture should implement seven controls.

1. Maintain a documented purpose statement that emphasises transformative use and limits commercial application.
2. Apply de-duplication at the document level to reduce verbatim memorisation.
3. Apply a memorisation eval that probes the model for verbatim outputs of training data and removes high-risk samples.
4. Honour AI-crawler robots.txt directives because publisher signals weigh in the fourth factor analysis.
5. Avoid known commercial datasets where licences are available and unused.
6. Apply opt-out registries (the Spawning project, the IETF AI Preferences working group output).
7. Document the chain of custody from source URL to embedding to model weight.

Code pattern: memorisation probe

def probe_memorisation(model, training_samples, threshold=0.8):
    risky = []
    for sample in training_samples:
        prompt = sample[:128]
        completion = model.generate(prompt, max_tokens=256)
        if rouge_l(completion, sample[128:384]) > threshold:
            risky.append(sample)
    return risky

Comparison: fair use posture by training data category

Category	Transformativeness	Market harm	Fair use posture
Public-domain text	High	Low	Strong
Open-licensed code	Variable	Variable	Mixed
News articles	Moderate	High	Weak
Books	Moderate	High	Weak
Social media public posts	Moderate	Low	Mixed
Commercial images	Low	High	Weak

Additional FAQ

Is fair use the same in the EU?
No. The EU does not have a general fair use doctrine. The closest equivalents are the text-and-data-mining exceptions in Article 3 (research) and Article 4 (general, with opt-out) of the 2019 Copyright Directive.

Does the four-factor test apply to all media?
Yes. The 17 USC 107 framework applies regardless of medium, but the application differs.

How does opt-out interact with fair use?
Opt-out is not a fair use requirement under US law. It is required under Article 4 of the EU Copyright Directive. As a practical matter, honouring opt-out reduces the fourth factor harm and improves the defence.

Is non-commercial training automatically fair use?
No. Non-commercial weighs in the first factor but does not by itself decide the case.

The four factors applied to AI training

The fair use analysis under 17 USC 107 considers four factors. Factor one is the purpose and character of the use, including whether the use is of a commercial nature or is for non-profit educational purposes. Factor two is the nature of the copyrighted work. Factor three is the amount and substantiality of the portion used. Factor four is the effect of the use on the potential market for or value of the copyrighted work.

For AI training each factor presents distinct questions. Factor one turns on whether the training is transformative. The Supreme Court’s 2023 Warhol Foundation v Goldsmith decision narrowed the transformative use analysis, focusing on whether the secondary use has a purpose distinct from the original. Training a general-purpose language model is plausibly transformative. Training a model intended to compete in the same market as the source work is not.

Factor two distinguishes published from unpublished work and creative from factual work. Published creative works lean against fair use. Factual or functional work leans for fair use. AI training corpora typically contain both, and the factor cuts both ways depending on the sample.

Factor three considers the amount used. AI training typically copies the entire work into the training pipeline, although the trained model retains only statistical patterns. Courts have split on how to weigh this factor for training. Some treat the full-work ingestion as weighing against fair use. Others focus on what the model retains and find it weighs neutrally or for fair use.

Factor four is the most contested in AI training cases. Courts ask whether the trained model substitutes for the source work in the relevant market. If the model can output substantially similar content, the substitution effect is direct. If the output is qualitatively different, the substitution is indirect or absent.

Memorisation and the verbatim output problem

A central technical question in AI training fair use is memorisation. A trained model that emits verbatim copies of training data has, in effect, retained the training data in its weights. That retention undermines the transformative use argument under factor one and increases market harm under factor four.

The 2024 and 2025 research literature documented memorisation rates in large language models. Models trained on duplicated content memorise at higher rates. Models trained on rare content memorise at higher rates. Larger models memorise at higher rates. These findings are operational guidance for training pipelines that want to minimise memorisation.

Mitigation techniques include de-duplication at the document level, de-duplication at the chunk level, training data filtering for high-risk content, and post-training memorisation evaluation with output filtering. Each technique reduces but does not eliminate memorisation. A scraper feeding a training pipeline should apply at least the document-level de-duplication.

The relationship between fair use and licensing

Fair use is a defence, not an entitlement. A scraper that has a licence does not need to argue fair use. A scraper without a licence relies on fair use only if the rights holder objects.

The 2024-2026 trend is toward more licensing. Major publishers struck deals with major AI labs (News Corp with OpenAI in May 2024, Reddit with Google in February 2024, Stack Overflow with OpenAI in May 2024). Those deals reduce reliance on fair use for the licensed corpora.

For scrapers the implication is that fair use is becoming the fallback, not the default. The strongest position is to license where possible and to fall back to fair use only for content where licensing is impractical.

Next steps

If your team trains models on scraped data, the highest-leverage improvement this quarter is to build the training data manifest. It costs little, it underpins both US discovery and EU AI Act response, and it makes every downstream compliance question easier. For broader policy, head to the DRT compliance hub and pair this with the robots.txt ethics guide.

This guide is informational, not legal advice.

CrewAI for scraping pipelines: complete 2026 guide

CrewAI scraping pipeline patterns turn the abstract “give the LLM some tools and pray” approach into a structured org chart of specialized agents who each do one job well. By early 2026, CrewAI has reached version 0.86, the Crew DSL has stabilized, and the framework’s role-based architecture maps surprisingly cleanly onto the way real scraping teams actually think about work: someone scouts targets, someone fetches, someone extracts, someone validates.

This guide builds an end-to-end CrewAI scraping pipeline for monitoring competitor prices across multiple ecommerce sites. We define agents, tasks, tools, the hierarchical process, and the integration points with your proxy pool, your database, and your alerting system. By the end you will have a working pipeline plus the knowledge to scale it past prototype.

Why CrewAI shines for scraping

CrewAI’s central abstraction is the Crew, a group of Agents that share Tools and execute Tasks under a Process. The framework forces you to think about division of labor up front, and that constraint produces cleaner pipelines than a single mega-agent.

For scraping, the role split that works in production is:

Role	Responsibility	Typical tools
Scout	Discover URLs to scrape	Search APIs, sitemap crawler
Fetcher	Pull HTML for each URL with proxy rotation	HTTP client, Playwright
Extractor	Parse structured data from HTML	LLM with JSON Schema
Validator	QA the extraction against business rules	Pydantic, custom validators
Reporter	Format and dispatch results	Database writer, notifier

This shape mirrors how a competent human team handles scraping. CrewAI lets you express it directly.

Installing the stack

pip install crewai==0.86.0 crewai-tools==0.20.0 langchain-openai==0.2.10 \
            playwright==1.49.0 pydantic==2.9.2 httpx==0.27.2
playwright install chromium

export OPENAI_API_KEY="sk-..."

If you prefer Anthropic:

pip install langchain-anthropic==0.2.10
export ANTHROPIC_API_KEY="sk-ant-..."

Defining agents

Agents are declarative. You give them a role, a goal, a backstory (which is more important than it sounds because the LLM uses it for tone and decision style), and a set of tools.

from crewai import Agent
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.1)

scout = Agent(
    role="Web Scout",
    goal="Find every product URL for a competitor's catalog",
    backstory=(
        "Senior research analyst with a decade of experience mapping ecommerce "
        "catalogs. Methodical, exhaustive, never misses a category page."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

fetcher = Agent(
    role="HTTP Fetcher",
    goal="Reliably fetch page HTML through a rotating proxy pool",
    backstory=(
        "Pragmatic engineer who treats every fetch as adversarial. Always retries, "
        "always rotates IPs, always respects rate limits."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

extractor = Agent(
    role="Data Extractor",
    goal="Pull title, price, currency, and stock status from product HTML",
    backstory=(
        "Detail-obsessed analyst who treats malformed JSON as a personal insult. "
        "Returns clean structured records or explicit nulls, never guesses."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
)

Notice how the backstory does most of the work. CrewAI agents read the backstory before every task, and a good backstory shifts behavior more reliably than instruction tweaks in the task itself.

Building tools

CrewAI tools are simple Python functions decorated with @tool or subclasses of BaseTool. The tool docstring is what the LLM reads to decide when to call it, so write it like a help message.

from crewai.tools import BaseTool
from pydantic import BaseModel, Field
import httpx
import os
import random

PROXIES = os.environ.get("PROXY_POOL", "").split(",")

class FetchInput(BaseModel):
    url: str = Field(..., description="The URL to fetch")
    timeout_s: int = Field(30, description="Request timeout in seconds")

class FetchTool(BaseTool):
    name: str = "fetch_url"
    description: str = (
        "Fetch a URL through the rotating proxy pool. Returns HTML or an error. "
        "Use for any HTTP fetch in this pipeline."
    )
    args_schema: type[BaseModel] = FetchInput

    def _run(self, url: str, timeout_s: int = 30) -> str:
        proxy = random.choice(PROXIES) if PROXIES and PROXIES != [""] else None
        with httpx.Client(proxy=proxy, timeout=timeout_s, follow_redirects=True) as c:
            r = c.get(url, headers={"User-Agent": "Mozilla/5.0"})
            return f"HTTP {r.status_code}\n\n{r.text[:200000]}"

fetch_tool = FetchTool()

For Playwright-based fetching of JavaScript-heavy sites, expose a parallel render_url tool:

from playwright.sync_api import sync_playwright

class RenderTool(BaseTool):
    name: str = "render_url"
    description: str = (
        "Render a URL in headless Chromium and return the HTML after JS executes. "
        "Use only when fetch_url returns insufficient data because of JS-driven content."
    )
    args_schema: type[BaseModel] = FetchInput

    def _run(self, url: str, timeout_s: int = 30) -> str:
        with sync_playwright() as p:
            browser = p.chromium.launch(headless=True)
            page = browser.new_page()
            page.goto(url, wait_until="networkidle", timeout=timeout_s * 1000)
            html = page.content()
            browser.close()
        return html

render_tool = RenderTool()

Attach the tools to the right agents:

fetcher.tools = [fetch_tool, render_tool]

Defining tasks

Tasks express what each agent should do and what they should produce. Every task gets an expected_output description that doubles as the validation hint for the LLM.

from crewai import Task

scout_task = Task(
    description=(
        "Discover all product URLs for {target_site} in the {category} category. "
        "Use the sitemap if available. Return a JSON array of URLs."
    ),
    expected_output="JSON array of strings, each a fully qualified product URL",
    agent=scout,
)

fetch_task = Task(
    description=(
        "For each URL in the prior task output, fetch the HTML using fetch_url. "
        "If the response contains less than 1000 characters of body content or "
        "looks like a JS-only shell, retry with render_url. "
        "Return a JSON map of {url: html}."
    ),
    expected_output="JSON object mapping URL to raw HTML",
    agent=fetcher,
    context=[scout_task],
)

extract_task = Task(
    description=(
        "For each {url: html} pair, extract title, price (number), currency (3-letter code), "
        "and in_stock (boolean). Return a JSON array of records."
    ),
    expected_output=(
        "JSON array of objects with keys: url, title, price, currency, in_stock"
    ),
    agent=extractor,
    context=[fetch_task],
)

The context field is how data flows between tasks. CrewAI passes prior task outputs as text to the next agent’s prompt.

Assembling the crew

from crewai import Crew, Process

crew = Crew(
    agents=[scout, fetcher, extractor],
    tasks=[scout_task, fetch_task, extract_task],
    process=Process.sequential,
    verbose=True,
)

result = crew.kickoff(inputs={
    "target_site": "https://www.lazada.sg",
    "category": "ergonomic-keyboards",
})

print(result.raw)

Sequential is the default and the right pick for a linear scraping pipeline. For more complex workflows where one agent should orchestrate others, use Process.hierarchical and provide a manager LLM.

Hierarchical process for adaptive pipelines

The hierarchical process puts a manager agent in charge. The manager decides which worker to call, in what order, with what arguments. This is the right shape when the pipeline shape depends on the input.

from crewai import Crew, Process
from langchain_openai import ChatOpenAI

manager_llm = ChatOpenAI(model="gpt-4o", temperature=0)

crew = Crew(
    agents=[scout, fetcher, extractor],
    tasks=[scout_task, fetch_task, extract_task],
    process=Process.hierarchical,
    manager_llm=manager_llm,
    verbose=True,
)

The manager dispatches dynamically and is the right choice when you cannot enumerate the steps up front. Cost is higher because every dispatch decision is an LLM call. Reserve for genuine adaptivity, not as the default.

Comparing CrewAI to LangGraph and AutoGen

Dimension	CrewAI	LangGraph	AutoGen
Mental model	Org chart of roles	State machine	Group chat
Best fit for scraping	Multi-step pipelines with clear roles	Branching state-driven workflows	Conversational extraction
Tool definition	BaseTool subclass	LangChain Tool	Function decorator
Persistence	Manual	Built-in checkpointer	Manual
Learning curve	Lowest	Moderate	Steep
Production maturity	High	High	Moderate
Native MCP	Via wrappers	Via wrappers	Yes

CrewAI is the framework to pick when the pipeline shape mirrors a team org chart and you want the fastest possible path from idea to working code. LangGraph wins for pipelines with non-linear branching and long-running state. AutoGen wins for conversational extraction where multiple agents debate the right answer.

For the LangGraph alternative, see our scraping with LangGraph agents guide. For AutoGen, see Multi-agent scraping with AutoGen in 2026.

Adding proxy rotation

CrewAI itself does not own the network layer; your tools do. The pattern is to keep a proxy pool in env or in a small singleton, and pick from it inside every fetch tool. We showed this pattern above; here is the production refinement that adds health checking.

import time
from collections import defaultdict

class ProxyPool:
    def __init__(self, proxies):
        self.proxies = proxies
        self.failures = defaultdict(int)
        self.last_used = defaultdict(float)

    def pick(self):
        now = time.time()
        candidates = [p for p in self.proxies if self.failures[p] < 3]
        if not candidates:
            self.failures.clear()
            candidates = self.proxies
        return min(candidates, key=lambda p: self.last_used[p])

    def report(self, proxy, success):
        self.last_used[proxy] = time.time()
        if not success:
            self.failures[proxy] += 1
        else:
            self.failures[proxy] = max(0, self.failures[proxy] - 1)

pool = ProxyPool([os.environ["PROXY_POOL"].split(",")])

For ASEAN ecommerce scraping with mobile IPs that pass strict carrier-level checks, Singapore mobile proxy plugs into this pool directly.

Validator and Reporter agents in detail

The five-role split listed at the top of the article needs two agents we have not yet shown in code. Adding them turns a fragile demo into a production-grade pipeline.

from crewai import Agent

validator = Agent(
    role="Data Quality Auditor",
    goal=(
        "Reject any extracted record that violates business rules. Price must be > 0 "
        "and < 100000. Currency must be a valid ISO 4217 code. Title must be non-empty "
        "and under 500 chars. in_stock must be a bool. Flag suspicious values for review."
    ),
    backstory=(
        "Former data engineer who spent two years cleaning a B2B product catalog. "
        "Allergic to silently incorrect data. Will refuse to pass through anything "
        "that smells wrong, and will document why."
    ),
    llm=llm,
    verbose=True,
)

reporter = Agent(
    role="Insights Reporter",
    goal=(
        "Compare today's extracted prices against yesterday's and emit a Slack-ready "
        "summary highlighting price drops over 10%, new SKUs, and out-of-stock changes."
    ),
    backstory=(
        "Pricing analyst with a journalism background. Writes summaries that a busy "
        "merchandising manager can act on in 30 seconds."
    ),
    llm=llm,
    verbose=True,
)

The Validator agent in particular pays for itself the first time the Extractor mistakes a postcode field for a price and tries to write 94025 to your price_usd column.

Custom Python validation tool

LLMs are fine for fuzzy validation but bad at strict rule checking. Pair the Validator agent with a deterministic tool that does the unforgiving work.

from crewai.tools import BaseTool
from pydantic import BaseModel, ValidationError, Field as PField, conlist
from typing import List, Literal

class ProductRecord(BaseModel):
    url: str
    title: str = PField(min_length=1, max_length=500)
    price: float = PField(gt=0, lt=100000)
    currency: Literal["USD", "SGD", "EUR", "JPY", "GBP", "INR", "MYR", "THB", "IDR", "VND"]
    in_stock: bool

class ValidateInput(BaseModel):
    records: list

class StrictValidatorTool(BaseTool):
    name: str = "strict_validate"
    description: str = (
        "Run deterministic validation on a list of product records. Returns the "
        "subset that passes plus a list of errors for the rejected records."
    )
    args_schema: type[BaseModel] = ValidateInput

    def _run(self, records: list) -> dict:
        passed, errors = [], []
        for r in records:
            try:
                passed.append(ProductRecord(**r).model_dump())
            except ValidationError as e:
                errors.append({"record": r, "errors": e.errors()})
        return {"passed": passed, "errors": errors}

The Validator agent calls strict_validate and uses its output to decide what to forward to the Reporter.

Adding async execution and concurrency

Sequential is the default but real production crews need parallelism. CrewAI 0.86 introduced Process.async_sequential and async-friendly task callbacks. Use them when tasks are independent.

from crewai import Crew, Process

crew = Crew(
    agents=[scout, fetcher, extractor, validator, reporter],
    tasks=[scout_task, fetch_task, extract_task, validate_task, report_task],
    process=Process.sequential,
)

# Process N URLs in parallel by spawning N crews
import asyncio

async def scrape_many(urls):
    sem = asyncio.Semaphore(10)  # cap concurrent crews

    async def one(url):
        async with sem:
            return await crew.kickoff_async(inputs={"target_url": url})

    return await asyncio.gather(*(one(u) for u in urls))

The Semaphore is the single most important line. Without it, 1000 URLs spawn 1000 simultaneous Crew instances, each holding a Playwright browser, and the host runs out of memory in 30 seconds.

Wiring CrewAI into a warehouse

The Reporter agent should not be writing directly to your warehouse. Give it a thin tool that calls a typed function in your data layer.

import asyncpg

class WarehouseWriteTool(BaseTool):
    name: str = "warehouse_write"
    description: str = "Append validated product records to the warehouse."

    def _run(self, records: list) -> dict:
        # synchronous wrapper around an async pool
        return {"written": _sync_warehouse_write(records)}

The reason to keep this thin: if the LLM hallucinates a malformed record that slips through the Validator, the warehouse layer catches it via the column-type contract. Defense in depth.

Cost benchmarks

For a sequential scrape of 100 product URLs with three agents:

Setup	LLM cost	Wall clock
GPT-4o-mini all agents	$0.45	8 min
GPT-4o all agents	$7.20	9 min
Mixed: 4o-mini scout/fetcher, 4o extractor	$1.80	8.5 min
Claude Haiku all agents	$0.55	7 min
Claude Sonnet all agents	$7.80	8 min

The mixed setup is the value sweet spot. Use cheap models for orchestration agents, expensive models only for the agent that needs to read messy HTML and produce clean JSON.

Cost levers in priority order

The fastest wins on CrewAI cost, in the order they pay off:

1. Trim HTML before passing to Extractor. Strip scripts, styles, comments. Cuts tokens by 40 to 70 percent.
2. Use GPT-4o-mini for Scout, Fetcher, Validator, Reporter. Reserve GPT-4o or Sonnet for Extractor only.
3. Cache extractions by HTML hash. Same page extracted twice should not pay LLM twice.
4. Cap max_iter per agent to 8. Default 25 lets confused agents loop expensively.
5. Switch verbose off in production. Verbose mode includes all intermediate thought tokens in the trace which the agent re-reads.

Combined, these cut typical per-page cost by 60 to 80 percent on a tuned pipeline versus a default-config baseline.

Memory and learning

CrewAI agents can be configured with memory that persists across runs. Two flavors: short-term (within a crew run) and long-term (across runs, backed by a vector store).

from crewai.memory import LongTermMemory
from crewai.memory.storage.ltm_sqlite_storage import LTMSQLiteStorage

extractor.memory = True

crew = Crew(
    agents=[scout, fetcher, extractor],
    tasks=[scout_task, fetch_task, extract_task],
    process=Process.sequential,
    memory=True,
    long_term_memory=LongTermMemory(
        storage=LTMSQLiteStorage(db_path="./crew_ltm.db")
    ),
)

For scraping, long-term memory pays off when the same agent learns site-specific quirks across runs. After a few iterations, the extractor remembers that a certain Lazada page renders price in a non-standard div.

Production deployment

Run CrewAI under a worker queue (Celery, RQ, or a Postgres-backed queue) and treat each crew kickoff as a job. Set hard timeouts, log every agent step, and persist the result to your warehouse.

For long-running crews, set step_callback and task_callback to stream progress to your observability stack. The callbacks fire after every agent step and every task completion respectively.

The official CrewAI documentation covers deployment options in depth, including the managed CrewAI Plus platform.

Frequently asked questions

Can CrewAI agents call MCP servers?
Not natively in 0.86. Wrap the MCP server in a Python tool that translates BaseTool calls to MCP JSON-RPC. The community has at least three open-source bridges; pick the one most actively maintained.

How do I prevent runaway agent loops?
Set max_iter on each agent (default 25). For task-level safety, set max_execution_time in seconds. Both options cap cost and clock.

Can I run CrewAI without OpenAI or Anthropic?
Yes. Any LangChain LLM works, including Ollama, vLLM, LM Studio, and Bedrock. Quality drops with smaller open-source models, especially on the extraction agent which needs strong JSON Schema adherence.

Does CrewAI handle parallel agent execution?
Sequential and hierarchical processes are single-threaded. For genuine parallelism, run multiple crew invocations under asyncio or a worker pool.

What about debugging when an agent goes off-script?
Set verbose=True on agents and crew. The terminal trace shows every thought, action, and observation. For richer logging, integrate with LangSmith.

Are CrewAI tasks idempotent? Can I safely retry?
Tasks themselves are idempotent only if your tools are idempotent. The framework will not deduplicate side effects. Wrap the side-effecting step (warehouse write, alert dispatch) with an idempotency key based on the input.

Can the same crew handle multiple sites with different layouts?
Yes, but the Extractor agent benefits from per-site backstory or per-site fewshot examples. The pattern that works is one Crew per site, sharing the Scout/Fetcher/Validator/Reporter agents and swapping only the Extractor.

How does CrewAI compare to writing the same pipeline in plain LangChain?
CrewAI is roughly 30 percent fewer lines of code for typical 3-to-5 agent pipelines and the role-based abstraction makes the intent clearer in code review. Plain LangChain wins when you need fine-grained control over the prompt assembly per turn.

Can I use CrewAI for crawling, not just scraping?
Yes. Add a Crawler agent with a BFS tool that takes seed URLs and depth, and pass the discovered URL list to the Fetcher. CrewAI’s role abstraction handles the recursion via the same Crawler agent re-invoked per depth.

Common production gotchas

A handful of issues bite teams during their first month.

The Extractor agent silently truncates HTML when the input exceeds the LLM’s context window. Always pre-trim with a deterministic rule (strip scripts, styles, hidden divs, and whitespace) before sending to the LLM, and log the trimmed size.

CrewAI’s verbose=True writes to stdout in a hard-to-parse format. For production, replace with a structured logging callback that emits JSON lines you can grep and ship to your log aggregator.

Tool argument schemas are inferred at class load time. Adding a field after the agent has already been constructed is a no-op until you reload. Restart workers on tool changes.

Hierarchical mode’s manager LLM is GPT-4o by default and that single decision dominates the cost on small jobs. Override manager_llm to GPT-4o-mini or a stronger model only when the dispatch decision is hard.

The crew’s kickoff method blocks the calling thread. For async workers, use kickoff_async and await it. Mixing the two in the same process leads to nested event loop errors.

For broader patterns on building agentic scraping in 2026, browse the AI agentic proxies category.

The HiQ vs LinkedIn ruling: what scrapers should know in 2026

The HiQ LinkedIn scraping ruling is one of the most cited court decisions in the entire web data space, and one of the most misunderstood. Almost every commercial scraper today operates under a mental model of “scraping public data is legal because of HiQ.” That mental model is partially correct, partially wrong, and partially incomplete in 2026 in ways that matter for how you build your pipeline. This guide walks through what the case actually decided, how Van Buren v. United States changed the landscape in 2021, what happened after the case eventually settled in 2022, and what the practical takeaway is for scrapers operating in 2026.

The audience is the technical lead, in-house counsel, or product owner who has heard the case name dropped in vendor pitches and customer conversations and wants the actual story.

What the case was actually about

HiQ Labs was a small data analytics company that scraped publicly visible LinkedIn profiles to build a workforce analytics product. They sold predictions about employee flight risk to large enterprise customers. LinkedIn sent HiQ a cease-and-desist letter in 2017 demanding they stop. HiQ sued LinkedIn for a declaratory judgement that their scraping was lawful and for an injunction preventing LinkedIn from blocking them.

The dispute was framed primarily under the Computer Fraud and Abuse Act (CFAA), 18 U.S.C. Section 1030. LinkedIn argued that HiQ’s continued scraping after the cease-and-desist letter constituted “access without authorisation” under the CFAA. HiQ argued that publicly visible data, accessible without a login, could not be “without authorisation” because no authorisation was required in the first place.

The Northern District of California granted HiQ a preliminary injunction in 2017. The Ninth Circuit affirmed in 2019. The Supreme Court vacated and remanded in 2021 in light of Van Buren. The Ninth Circuit affirmed again on remand in 2022. The case eventually settled later that year, with HiQ agreeing to certain limits and LinkedIn dropping its claims.

For the broader compliance picture across jurisdictions, see the GDPR compliance guide and the CCPA compliance guide.

What the Ninth Circuit actually held

The Ninth Circuit in 2019 (and again in 2022) held that the CFAA’s “without authorisation” language likely does not cover access to publicly available websites, because for such sites no authorisation is needed in the first place. The court drew an analogy: a cease-and-desist letter does not transform a public sidewalk into private property. The walking is the access; the public availability is the authorisation.

The court emphasised three points:

1. The CFAA was originally enacted to address computer hacking, not civil disputes about access to public webpages.
2. Reading “without authorisation” broadly to include cease-and-desist-revocation would create a “criminal” regime in which any TOS violation became a federal crime.
3. The First Amendment and antitrust concerns weighed against allowing platforms to use the CFAA as a private weapon to control access to public information.

The decision did not say all scraping is legal. It said the CFAA does not criminalise scraping of publicly accessible data merely because the platform sent a cease-and-desist letter.

How Van Buren changed the landscape in 2021

In June 2021, the Supreme Court decided Van Buren v. United States. The case did not involve scraping, but it interpreted the CFAA’s “exceeds authorised access” language. A police officer had used his lawful access to a database to look up information for an improper purpose. The government argued he had “exceeded” his authorisation because his access was conditioned on use for proper purposes.

The Supreme Court rejected this reading. It held that “exceeds authorised access” means accessing files, folders, or databases that are off-limits, not accessing permitted data for impermissible purposes. The “gates-up-or-down” model: if the gate is up, your access is not unauthorised even if the use is.

This narrow reading of the CFAA aligned exactly with the Ninth Circuit’s HiQ analysis, and the Supreme Court remanded HiQ for reconsideration in light of Van Buren. The Ninth Circuit, on remand, reaffirmed its earlier decision.

The combined effect: in the Ninth Circuit (covering California and most of the western US), the CFAA does not reach scraping of publicly available websites, regardless of TOS violations or cease-and-desist letters.

What the eventual 2022 settlement actually said

Most coverage of HiQ stopped at the Ninth Circuit affirmance. Less covered: the case settled in late 2022. The settlement terms were partly confidential, but several public components were disclosed. HiQ agreed to a permanent injunction prohibiting it from scraping LinkedIn member data going forward. LinkedIn dropped its remaining claims.

Why did HiQ agree to a permanent injunction if they had won? Two reasons. First, after years of litigation, the company had effectively lost commercial momentum and was wound down. Second, while the CFAA claim had been knocked out, LinkedIn’s parallel state-law claims (breach of contract for TOS violations, trespass to chattels, tortious interference) were still live. The CFAA win did not resolve the state-law theories.

This is the part most scrapers miss. Winning the CFAA fight does not win the state-law fight. Public availability does not erase contract. A scraper that creates an account, agrees to terms, and then scrapes is in a fundamentally different posture than a scraper that hits public URLs without ever logging in.

The Meta v. Bright Data parallel

In January 2024, the US District Court for the Northern District of California decided Meta v. Bright Data. The fact pattern was deliberately similar to HiQ: Bright Data scraped public Facebook and Instagram pages and resold the data. Meta sued for breach of contract (TOS violation) and tortious interference.

The court held that Meta’s TOS only bound logged-in users. Bright Data’s scraping of publicly accessible logged-out pages did not breach the contract because no contract had ever formed. This was a very strong scraper-side ruling, but it was narrow: it applied only to logged-out scraping. The moment a scraper authenticates, the TOS attaches.

The Israeli court ruled similarly in a parallel proceeding the same year. EU regulators were quick to point out that the absence of a contract violation does not equal a lawful basis under GDPR, but the court rulings did establish a clear US-side rule: logged-out scraping of public data is legally robust against TOS-based claims.

For the broader public-vs-personal data analysis, see the personal vs public data scraping framework.

What the rulings collectively permit and forbid

Activity	Legal posture in 2026 (US)
Scraping public URLs, no login	Generally permitted; CFAA does not reach
Scraping behind a login you created	TOS applies; breach-of-contract risk
Scraping after a cease-and-desist letter (logged out)	Permitted under HiQ
Scraping behind a paywall you bypassed	High risk; CFAA may reach (gate is down)
Bulk personal data resale	TOS-independent risks: state privacy law
Scraping for AI training	Legal under HiQ; copyright fair use is separate
Scraping with fake accounts	TOS breach; state-law exposure

The asymmetry between logged-in and logged-out is the most important practical takeaway. Public, logged-out scraping has a strong legal floor in the US. Anything behind authentication exists under the platform’s terms.

Decision tree: is your scraping covered by HiQ?

Q1: Is the target URL accessible without any login?
    ├── No  -> HiQ does not protect you. Evaluate TOS and CFAA.
    └── Yes -> Q2
Q2: Are you in the Ninth Circuit's jurisdiction (or a court likely to follow)?
    ├── Yes -> CFAA risk is low.
    └── No  -> Other circuits have not all adopted; evaluate locally.
Q3: Did you create an account and accept terms?
    ├── Yes -> TOS attaches; CFAA may not, but contract claim does.
    └── No  -> Q4
Q4: Are you scraping personal data of identifiable individuals?
    ├── Yes -> CCPA, GDPR, PDPA, DPDP may apply independently.
    └── No  -> Q5
Q5: Are you bypassing technical access controls (CAPTCHA, IP block, rate limit)?
    ├── Yes -> CFAA "gate is down" risk increases.
    └── No  -> Strongest defensive posture.

The combination of “logged-out, public, identifiable-but-public, no controls bypassed” is the strongest position. Each “yes” to authentication, controls bypass, or personal data adds risk that HiQ does not resolve.

Practical implications for 2026 scraping pipelines

Three operational implications.

First, separate your logged-out and logged-in scraping infrastructure. Logged-out scraping enjoys HiQ-grade protection. Logged-in scraping operates under TOS and contract law. Mixing the two in one pipeline obscures your legal posture and weakens both.

Second, do not bypass technical access controls. The Van Buren “gates-up-or-down” model means that if a site puts up a gate (CAPTCHA, IP block, paywall) and you bypass it, you have moved from “authorisation not required” to “authorisation explicitly denied.” That is a different legal universe.

Third, document your access methodology. If you ever need to invoke HiQ in your defence, you will need to prove that your access was logged-out, that no controls were bypassed, and that the data was genuinely publicly accessible. A scrape that goes through a residential proxy mesh after using browser fingerprint spoofing to defeat a fingerprinting check is not “publicly accessible” in the HiQ sense.

For a deeper dive on the bot management and fingerprinting question, see the DataDome vs PerimeterX vs Akamai comparison.

External references

The Ninth Circuit opinion in HiQ Labs v LinkedIn (2022 remand) is at cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf. The Supreme Court opinion in Van Buren v United States is at supremecourt.gov/opinions/20pdf/19-783_k53l.pdf. The Meta v Bright Data summary judgement is in the public PACER record for case 3:23-cv-00077.

Comparison: HiQ doctrine vs state contract law vs GDPR

Issue	HiQ / Van Buren (CFAA)	State contract law	GDPR
Reaches public URLs	No	No (no contract)	Yes
Reaches logged-in scraping	Limited	Yes (TOS)	Yes
Requires lawful basis	No	No	Yes
Personal data carve-out	Irrelevant	Irrelevant	Public availability not a defence
Statutory damages	Up to USD 1k per access	Variable	Up to 4% revenue
Cease-and-desist effect	None on logged-out	Strengthens contract claim	Irrelevant
Settlement strategy lever	Weak (clean win)	Strong (TOS hook)	Strong (DPA leverage)

The takeaway: HiQ is a strong shield against CFAA claims for logged-out public scraping. It is not a shield against contract, GDPR, or other state privacy law claims. Build your legal posture for all four regimes simultaneously.

What changed in 2024-2025 case law

Three additional cases shaped the 2026 landscape.

X Corp v Bright Data (2024, ND Cal) reaffirmed Meta v Bright Data: logged-out scraping is not a TOS breach because no contract attaches. The court was explicit that platform terms cannot bind non-users.

Reddit v Anthropic (2025, ND Cal) is still pending as of mid-2026. Reddit alleges Anthropic scraped past explicit robots.txt directives blocking ClaudeBot. The case will test whether ignoring robots.txt for AI training constitutes any kind of cognisable claim distinct from CFAA. The outcome will reshape AI scraping practice.

Doe v Github (2024, ND Cal) addressed scraping of open-source code for AI training, holding that the public availability of code on GitHub did not waive copyright protections in derivative or memorised outputs. Public availability and copyright protection are separate inquiries.

For a forward-looking discussion of where the AI training case law is heading, see fair use for AI training data in 2026.

FAQ

Is all web scraping legal because of HiQ?
No. HiQ knocked out one specific federal claim (the CFAA) for one specific kind of scraping (logged-out public data). It did not legalise all scraping. Contract, copyright, privacy, and trade-secret claims survive independently.

Does HiQ apply outside the Ninth Circuit?
The Ninth Circuit covers California and the western US, where most tech litigation lands. Other circuits have not uniformly adopted the same reading, but the trend post-Van Buren is in that direction.

What about scraping social media platforms?
Logged-out, public-page scraping is reasonably defensible under HiQ and Meta v Bright Data. Logged-in scraping is a TOS issue and should be evaluated separately.

Can a cease-and-desist letter make my scraping illegal?
Under the CFAA in the Ninth Circuit, no. Under state contract or trespass law, it can strengthen the platform’s claim. Treat a C&D as a serious signal even if it does not change the federal analysis.

Did HiQ actually win in the end?
The CFAA fight, yes. The commercial fight, no. HiQ wound down operations and agreed to a permanent injunction in the 2022 settlement. The case is a legal win and a commercial cautionary tale.

Extended case law analysis

The hiQ Labs v LinkedIn litigation ran from 2017 to 2022 and produced four major opinions. The 2019 Ninth Circuit opinion held that scraping public data did not violate the CFAA’s without authorisation prong because public data is by definition authorised for any visitor. The Supreme Court’s Van Buren v United States decision (June 2021) reinforced the gates-up gates-down reading of the CFAA, which strongly supported the hiQ position. The 2022 Ninth Circuit opinion on remand reaffirmed the public-data holding and remanded the contract claims, which hiQ ultimately settled.

The post-hiQ landscape contains four important precedents that scrapers should know.

1. Meta Platforms v Bright Data (Northern District of California, January 2024). Meta’s CFAA and contract claims against Bright Data largely failed at summary judgment for public data scraping. The court relied on hiQ for the CFAA analysis and held that Bright Data had not formed a contract by browsing logged-out pages.

2. X Corp v Bright Data (Northern District of California, May 2024). The court reached a similar conclusion, dismissing X’s claims for scraping public data while users were not logged in.

3. Ryanair v PR Aviation (CJEU, 2015) and the 2024 follow-ups. The European pathway places more weight on database rights and contract than on CFAA-equivalent statutes.

4. The 2024-2025 wave of state-level scraping statutes, including bills introduced in California, Texas, and New York that propose explicit rules for AI training data scraping.

Implementation patterns post-hiQ

Operators acting on the hiQ ruling should follow a five-step posture.

1. Distinguish logged-out from logged-in scraping. The hiQ holding extends only to logged-out public scraping. Logged-in scraping involves account terms which hiQ does not protect.

2. Avoid technical bypass of authentication, rate limits, or bot detection beyond ordinary headless browser use. Bypass is what triggers the CFAA in post-hiQ cases.

3. Maintain a documented record of the scrape’s purpose, frequency, and destination. Litigation discovery will surface this and a clean record helps.

4. Honour cease and desist letters carefully. The hiQ ruling does not give scrapers a right to ignore valid C and D letters that allege contract or tort claims. The right response is legal review, not silence.

5. Apply privacy law independently. CFAA protection does not equal GDPR or CCPA protection. Personal data is regulated by separate statutes.

Code pattern: distinguishing logged-out scraping

def is_logged_out_only(session):
    if session.cookies:
        return False
    if "Authorization" in session.headers:
        return False
    if "X-Auth-Token" in session.headers:
        return False
    return True

Comparison: scraping legal posture before and after hiQ

Question	Pre-hiQ default	Post-hiQ default (US public data)	Caveat
CFAA risk for public scraping	Substantial	Low	Bypass changes the analysis
Contract risk	Moderate	Moderate to high	Browsewrap terms still litigated
Trespass to chattels	Low	Low	Resource exhaustion claims survive
Privacy law risk	High	High	Independent of CFAA
Copyright	Moderate	Moderate	Fair use defence is fact-specific

Additional FAQ

Does hiQ apply to data behind a login?
No. The holding is limited to public, unauthenticated data. Logged-in scraping involves account terms.

Does hiQ apply outside the United States?
No. Each jurisdiction has its own statutes. The European Union, the United Kingdom, Singapore, and India apply different frameworks.

Can a publisher block scrapers technically?
Yes. Technical blocks (rate limits, IP bans, bot detection) are lawful. Bypassing them weakens the hiQ defence.

Is the hiQ ruling settled law?
The Ninth Circuit holding stands. Other circuits have not directly contradicted it but they could. A scraping operation should not assume nationwide uniformity.

Cases that built on the hiQ framework

Two post-hiQ decisions have shaped how courts apply the doctrine in 2024-2026.

In Meta Platforms v. Bright Data (N.D. Cal., January 2024), Judge Edward Chen granted summary judgment to Bright Data on Meta’s contract claims for scraping logged-out Facebook and Instagram public profile data. The court explicitly relied on the hiQ framework, holding that Meta could not enforce its terms of service against Bright Data because Bright Data had no account and had not assented to the terms. The decision reinforced the bright-line significance of the logged-in versus logged-out distinction. Meta’s separate trespass and unjust enrichment theories were also dismissed.

In X Corp v. Bright Data (N.D. Cal., May 2024), Judge William Alsup followed the same logic for logged-out scraping of X (formerly Twitter) public posts. Judge Alsup’s opinion went further than Meta v. Bright Data in characterising the policy stakes, observing that giving social media platforms unilateral power to control public-facing data would create de facto information monopolies inconsistent with US antitrust and free-speech traditions. The decision is now the most quotable post-hiQ pro-scraping precedent for logged-out commercial use.

Both cases stop at logged-out scraping. Neither protects the bypass of authentication, the use of fake accounts, or the circumvention of technical blocks. The combined doctrine is narrow but settled in the Ninth Circuit for the use cases it covers.

The litigation history of hiQ

The hiQ Labs v LinkedIn dispute began in May 2017 when LinkedIn sent hiQ a cease and desist letter demanding that hiQ stop scraping public LinkedIn profiles. hiQ sued for declaratory relief, arguing that scraping public data did not violate the Computer Fraud and Abuse Act and that LinkedIn’s blocking efforts violated antitrust and tortious interference principles.

The Northern District of California granted hiQ a preliminary injunction in August 2017. The Ninth Circuit affirmed in September 2019, holding that scraping public data did not violate the CFAA’s without authorisation prong. The Supreme Court vacated and remanded in light of Van Buren v United States in June 2021. The Ninth Circuit affirmed again on remand in April 2022, reaffirming the public-data holding.

The case finally settled in late 2022 after the contract claims were remanded to the district court. The settlement terms were not made public, but hiQ agreed to stop scraping LinkedIn and acknowledged having breached LinkedIn’s terms of service. The company subsequently wound down operations.

The procedural history matters because the holding stands even though the company that brought the case did not survive. The legal precedent is what scrapers operate under in 2026, and that precedent is favourable to public-data scraping but does not provide a shield against contract claims.

How the Supreme Court’s Van Buren decision changed the analysis

Van Buren v United States, decided in June 2021, addressed the without authorisation and exceeds authorised access prongs of the CFAA. The Supreme Court adopted a gates-up gates-down reading, under which a person violates the CFAA only when they access a computer system that is closed to them. A person who is authorised to access certain files but uses that access for an improper purpose does not exceed authorised access.

Applied to scraping, Van Buren strengthened the hiQ holding. Public LinkedIn profiles are gates-up for any visitor. A scraper that accesses them is not exceeding authorised access. The CFAA does not reach the activity.

Van Buren did not reach the question of when a scraper is gates-down. Subsequent cases have explored that question. A scraper that bypasses authentication, ignores IP blocks, or rotates User-Agent strings to evade detection is plausibly gates-down. The line is fact-specific and remains unsettled.

Practical operational implications

A 2026 scraping operation taking hiQ as authoritative should adopt five operational practices.

First, document the public-facing nature of every target page. Maintain screenshots showing that the page is reachable without login. Maintain timestamps. The evidentiary record matters in litigation discovery.

Second, avoid technical bypass. The hiQ holding does not protect a scraper that bypasses CAPTCHA, defeats bot detection, or rotates IPs to evade rate limits. Each of those activities is a potential gates-down trigger.

Third, respond promptly to cease and desist letters. Ignoring a C and D letter does not improve the legal position. The right response is legal review followed by either a measured response or a tactical pause.

Fourth, separate logged-in scraping from logged-out scraping operationally. Use different infrastructure, different credentials, different audit trails. The legal analysis is fundamentally different.

Fifth, monitor circuit splits. The hiQ ruling is Ninth Circuit law. Other circuits may reach different conclusions. A scraper operating nationally should track relevant cases in the Second, Fourth, and Eleventh Circuits.

Next steps

If your team relies on HiQ in any pitch, customer conversation, or compliance memo, the fastest improvement is to make the logged-out vs logged-in distinction explicit in your documentation. The legal protections are very different. For a fuller compliance posture across regimes, head to the DRT compliance hub and pair this with the GDPR and CCPA guides.

This guide is informational, not legal advice.

Scraping with LangGraph agents in 2026

LangGraph scraping agents have become the standard pattern for any non-trivial LLM-driven scraping pipeline that needs branching, retries, and checkpointed state. By early 2026, LangGraph has reached version 0.4 with stable APIs, the StateGraph primitive is rock solid, and the Postgres checkpointer makes it easy to resume long-running scrapes after a crash. If your scraping job is more than fetch-extract-store, LangGraph is the right framework.

This guide builds a real production scraping agent step by step. We define the state, wire the tool nodes, add retry edges, plug in a checkpointer, and benchmark cost against alternatives. By the end you will have a working LangGraph scraper that handles a flaky target site, recovers from failures, and emits clean structured data.

Why LangGraph beats LangChain agents for scraping

LangGraph scraping agents express the workflow as an explicit graph instead of an implicit ReAct loop. That difference matters in three places.

First, branching. A scraping agent often needs to take different paths depending on what the page returns. Did the site return a captcha? Branch to the solver. Is the price hidden behind a login? Branch to authentication. LangChain’s ReAct agent makes branching implicit through prompt engineering. LangGraph makes it explicit through edges.

Second, observability. When a scraper fails at 3 AM, you want to know exactly which node failed and what state was in scope. LangGraph’s state object plus LangSmith integration gives you that. The classic LangChain agent gives you a chain of thought that you have to read.

Third, persistence. LangGraph ships a checkpointer system. After every node, the state is persisted to SQLite or Postgres. If the worker dies mid-scrape, you resume from the last checkpoint with one line of code.

Where LangGraph fits next to LangChain

LangChain remains the right framework for prompt templates, LLM clients, retrievers, and document loaders. LangGraph builds on top, adding the runtime that orchestrates them. The mental model is: LangChain is the parts bin, LangGraph is the assembly line. Almost every production LangGraph scraping agent imports LangChain primitives for the LLM call and the prompt template, and reserves LangGraph for the routing and state machine.

Installing the stack

pip install langgraph==0.4.0 langchain==0.3.20 langchain-openai==0.2.10 \
            langgraph-checkpoint-postgres==2.0.10 \
            playwright==1.49.0 pydantic==2.9.2 httpx==0.27.2
playwright install chromium

For LangSmith tracing (free for personal projects):

export LANGSMITH_API_KEY="ls__..."
export LANGSMITH_TRACING="true"
export LANGSMITH_PROJECT="lazada-scraper"

Defining the agent state

The state is a Pydantic-style TypedDict that flows through every node. For a scraping agent it typically holds the input URL, the fetched HTML, the extracted data, and an error log.

from typing import TypedDict, Optional, List
from langgraph.graph import StateGraph, END

class ScrapeState(TypedDict):
    url: str
    html: Optional[str]
    captcha_detected: bool
    extracted: Optional[dict]
    errors: List[str]
    attempt: int

LangGraph reduces state by merging dicts on every node return. Mutations to the state inside a node are not seen by other nodes; only the returned dict is.

Building the nodes

Each node is a function that takes the state and returns a partial state update.

import asyncio
from playwright.async_api import async_playwright
from openai import AsyncOpenAI
import json

client = AsyncOpenAI()

async def fetch_node(state: ScrapeState) -> dict:
    """Fetch the URL with Playwright."""
    try:
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True)
            page = await browser.new_page()
            await page.goto(state["url"], wait_until="networkidle", timeout=30000)
            html = await page.content()
            await browser.close()
        return {"html": html, "attempt": state["attempt"] + 1}
    except Exception as e:
        return {
            "errors": state["errors"] + [f"fetch failed: {e}"],
            "attempt": state["attempt"] + 1,
        }

async def captcha_check_node(state: ScrapeState) -> dict:
    """Quick heuristic for captcha detection."""
    html = state.get("html", "") or ""
    flags = ["cf-challenge", "captcha", "px-captcha", "datadome", "recaptcha"]
    detected = any(f in html.lower() for f in flags)
    return {"captcha_detected": detected}

async def extract_node(state: ScrapeState) -> dict:
    """LLM-driven extraction with strict JSON Schema."""
    schema = {
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "price": {"type": "number"},
            "currency": {"type": "string"},
            "in_stock": {"type": "boolean"},
        },
        "required": ["title", "price", "currency", "in_stock"],
        "additionalProperties": False,
    }

    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "product", "schema": schema, "strict": True},
        },
        messages=[
            {"role": "system", "content": "Extract product data from HTML."},
            {"role": "user", "content": (state["html"] or "")[:200000]},
        ],
    )
    return {"extracted": json.loads(resp.choices[0].message.content)}

async def captcha_solver_node(state: ScrapeState) -> dict:
    """Stub — wire to 2Captcha, CapSolver, or similar."""
    return {
        "errors": state["errors"] + ["captcha solver not implemented"],
        "captcha_detected": False,
    }

Wiring the graph

This is the part that beats every alternative framework on clarity. You list nodes, list edges, and the runtime is built.

def route_after_captcha_check(state: ScrapeState):
    if state["captcha_detected"]:
        return "captcha_solver"
    return "extract"

def route_after_fetch(state: ScrapeState):
    if state.get("html") is None:
        if state["attempt"] >= 3:
            return END
        return "fetch"
    return "captcha_check"

graph = StateGraph(ScrapeState)
graph.add_node("fetch", fetch_node)
graph.add_node("captcha_check", captcha_check_node)
graph.add_node("captcha_solver", captcha_solver_node)
graph.add_node("extract", extract_node)

graph.set_entry_point("fetch")
graph.add_conditional_edges("fetch", route_after_fetch)
graph.add_conditional_edges("captcha_check", route_after_captcha_check)
graph.add_edge("captcha_solver", "fetch")  # retry after solving
graph.add_edge("extract", END)

app = graph.compile()

That graph handles the basic happy path, captcha branch, and a 3-attempt retry on fetch failures. LangGraph compiles it into an executable that you invoke with the initial state.

async def main():
    final = await app.ainvoke({
        "url": "https://www.lazada.sg/products/xyz",
        "html": None,
        "captcha_detected": False,
        "extracted": None,
        "errors": [],
        "attempt": 0,
    })
    print(json.dumps(final["extracted"], indent=2))

asyncio.run(main())

Adding a checkpointer

For long-running scraping jobs (think: scrape ten thousand products with intermediate state at each one), the checkpointer is mandatory. SQLite for development, Postgres for production.

from langgraph.checkpoint.postgres import PostgresSaver

DB_URI = "postgresql://scraper:secret@localhost:5432/scrapes"

async with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
    await checkpointer.setup()
    app = graph.compile(checkpointer=checkpointer)

    config = {"configurable": {"thread_id": "lazada-product-12345"}}

    async for event in app.astream(initial_state, config):
        print(event)

If the worker dies mid-graph, you restart with the same thread_id and LangGraph resumes from the last successful node. Critical for any scrape that takes more than a minute.

Adding tool nodes for proxy rotation

For real production work, every fetch should go through a rotating proxy pool. Wire it into the fetch node:

import random
import os

PROXIES = os.environ.get("PROXY_POOL", "").split(",")

async def fetch_node(state: ScrapeState) -> dict:
    proxy = random.choice(PROXIES) if PROXIES and PROXIES != [""] else None
    proxy_config = None
    if proxy:
        u, _, rest = proxy.partition("://")
        if "@" in rest:
            creds, host_port = rest.split("@", 1)
            user, password = creds.split(":")
            proxy_config = {"server": f"{u}://{host_port}", "username": user, "password": password}
        else:
            proxy_config = {"server": proxy}

    try:
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True, proxy=proxy_config)
            page = await browser.new_page()
            await page.goto(state["url"], wait_until="networkidle", timeout=30000)
            html = await page.content()
            await browser.close()
        return {"html": html, "attempt": state["attempt"] + 1}
    except Exception as e:
        return {
            "errors": state["errors"] + [f"fetch failed: {e}"],
            "attempt": state["attempt"] + 1,
        }

For ASEAN scraping where mobile IPs work better than residential, Singapore mobile proxy integrates as the proxy pool source.

Sticky proxies via thread state

For multi-page flows where the same session must hold the same exit IP, store the proxy in the state and pin it across nodes:

class ScrapeState(TypedDict):
    url: str
    html: Optional[str]
    captcha_detected: bool
    extracted: Optional[dict]
    errors: List[str]
    attempt: int
    sticky_proxy: Optional[str]  # bound on first fetch, reused across retries

async def fetch_node(state: ScrapeState) -> dict:
    proxy = state.get("sticky_proxy") or random.choice(PROXIES)
    # ... fetch with proxy
    return {"html": html, "sticky_proxy": proxy, "attempt": state["attempt"] + 1}

This is essential for cart and checkout flows on retailers that fingerprint the session-to-IP binding.

Parallel fan-out with Send

For the common pattern of “scrape 50 URLs and aggregate the results,” LangGraph supports a Send primitive that fans out across N parallel branches and rejoins.

from langgraph.graph import Send

def fanout(state):
    return [Send("scrape_one", {"url": u, "html": None, "errors": [], "attempt": 0})
            for u in state["urls"]]

graph = StateGraph(BatchState)
graph.add_node("scrape_one", scrape_one_node)
graph.add_node("aggregate", aggregate_node)
graph.set_entry_point("dispatcher")
graph.add_conditional_edges("dispatcher", fanout, ["scrape_one"])
graph.add_edge("scrape_one", "aggregate")
graph.add_edge("aggregate", END)

This gets you 50-way parallel scraping with proper backpressure (limit concurrency in the runtime config) and a single aggregated result. The pattern is faster than spawning 50 separate graph invocations because the aggregate state lives in one place.

Retry strategies that actually work

The naive retry pattern is “if fetch fails, retry up to N times.” In production this is rarely sufficient because failures cluster: a bad IP fails 5 times in a row before you decide to rotate. A better pattern uses exponential backoff and IP rotation between attempts.

async def fetch_node(state: ScrapeState) -> dict:
    import asyncio
    backoff = min(2 ** state["attempt"], 30)
    if state["attempt"] > 0:
        await asyncio.sleep(backoff)
    proxy = random.choice(PROXIES)  # NEW proxy on every retry
    # ... fetch with proxy

Combined with a circuit breaker on the proxy pool (evict any IP that fails 3 times in 10 minutes), the success rate on real-world targets jumps from roughly 88 percent to over 97 percent.

Comparing LangGraph to alternatives

Framework	State model	Branching	Persistence	Observability	Best fit
LangGraph	Explicit TypedDict	First-class	Built-in checkpointer	LangSmith integration	Complex multi-step pipelines
LangChain ReAct	Implicit, chat history	Prompt-driven	Manual	LangSmith integration	Simple one-shot tasks
CrewAI	Per-crew shared memory	Role-based	Manual	LangSmith or self	Multi-agent role play
AutoGen	Group chat state	Free-form	Manual	OpenTelemetry	Conversational agents
Custom asyncio	Whatever you build	Whatever you write	Whatever you write	Whatever you wire	Maximum flexibility

LangGraph wins for scraping specifically because scraping flows are state machines with clear branches: fetch then maybe captcha then extract then maybe retry. That maps to LangGraph’s primitives one to one. ReAct loops can do the same job but every behavior change requires prompt rewriting.

For more on CrewAI as an alternative, see CrewAI for scraping pipelines. For AutoGen, see Multi-agent scraping with AutoGen in 2026.

Cost benchmarks

Single product page extraction, end to end, on the graph above:

LLM model	Avg LLM tokens per page	LLM cost per page	Compute per page	Total per 1k pages
GPT-4o-mini	14,000	$0.0028	$0.001	$3.80
GPT-4o	14,000	$0.05	$0.001	$51
Claude 3.5 Sonnet	13,500	$0.052	$0.001	$53
Claude 3.5 Haiku	13,500	$0.011	$0.001	$12

For high-volume scraping where you control the prompt and the schema is simple, GPT-4o-mini or Claude Haiku is the right pick. For tricky sites where extraction quality matters more than cost, Sonnet or 4o is worth it.

Production deployment

Run LangGraph workers under a process supervisor (systemd, PM2, or Kubernetes) with a health-check endpoint. Use Redis or Postgres for the checkpointer in production.

A minimal worker loop:

import asyncio
from redis.asyncio import Redis

redis = Redis.from_url("redis://localhost:6379")

async def worker():
    while True:
        url = await redis.brpop("scrape:queue", timeout=10)
        if url is None:
            continue
        url = url[1].decode()
        state = make_initial_state(url)
        config = {"configurable": {"thread_id": f"scrape-{url}"}}
        try:
            final = await app.ainvoke(state, config)
            await store_result(url, final)
        except Exception as e:
            await redis.lpush("scrape:dead", url)

asyncio.run(worker())

For LangGraph deployment on serverless, the LangGraph Platform docs cover the managed option in detail.

A complete production graph with all the layers

Putting the patterns together gives a graph that handles the long tail. The nodes:

1. validate_url rejects malformed input early, saving downstream work.
2. fetch with sticky proxy and exponential backoff.
3. captcha_check flags Cloudflare, DataDome, PerimeterX, and Akamai.
4. captcha_solver calls 2Captcha for Turnstile and CapSolver for the rest.
5. extract pulls structured fields with strict schema.
6. validate_extracted checks Pydantic-level invariants (price > 0, in_stock is boolean).
7. enrich adds derived fields (USD-converted price, normalized SKU).
8. persist writes to Postgres with an upsert.
9. notify posts to Slack on price changes greater than 10 percent.

The graph branches at captcha_check (solver vs extract), at validate_extracted (re-fetch vs persist on validation failure), and at notify (skip if no significant change). Total node count: 9. Total edges: 14. The graph compiles in milliseconds and runs each scrape in roughly 4 to 8 seconds depending on captcha presence.

Real teams underestimate how much of their scraper is the surrounding plumbing (validate, enrich, persist, notify) versus the core fetch and extract. Having those as named nodes in a graph instead of buried inside the fetch function makes the system far easier to reason about when something breaks at 2 AM.

Cost and latency under realistic load

Numbers from a March 2026 production deployment running 50,000 product scrapes per day on a 4 vCPU 8 GB worker pool:

Metric	Value
Median graph latency end-to-end	4.8 s
p99 graph latency	22 s
Throughput per worker	12 scrapes/min
Workers needed for 50k/day	3
LLM cost per scrape (GPT-4o-mini)	$0.0028
Proxy cost per scrape (residential)	$0.0006
Compute cost per scrape (Fargate)	$0.0008
Total per-scrape cost	$0.0042
Daily total	$210

Compared to a non-LangGraph baseline (raw asyncio plus prompt) that ran at $190/day for the same throughput, LangGraph adds about 10 percent overhead in exchange for resumability, observability, and retry correctness. Most teams find the tradeoff lopsidedly worth it.

Observability with LangSmith

LangSmith remains the easiest way to see what a LangGraph agent is doing in production. Every node execution is traced with inputs, outputs, latency, and any LLM calls inside the node. The trace tree mirrors the graph structure, so you can spot a slow extract node or a node that errored without grepping logs.

Three patterns worth adopting from the start:

Tag every run with the URL being scraped and the source queue name. This makes filtering on the LangSmith UI trivial. langsmith_extra={"tags": [url_domain, queue_name]} in the invoke call.

Add custom metadata for the scrape ID and the proxy used. When something fails on a specific IP, you can filter by proxy to see if the failure is sticky to one bad IP versus a target-side ban.

Use LangSmith’s evaluation suite to regress against a frozen set of known-good HTML pages. Whenever you change the extract prompt, run the eval and confirm structured output quality has not regressed. This catches subtle prompt drift that production logs would not surface for days.

For teams that prefer self-hosted observability, OpenTelemetry instrumentation is supported via the langsmith SDK with the OTel exporter. Span attributes match the LangGraph node names, so Jaeger or Tempo show the same trace tree.

Frequently asked questions

Can LangGraph handle parallel scraping of multiple URLs?
Yes. Spawn one app invocation per URL, each with its own thread_id, and let asyncio handle concurrency. LangGraph also supports parallel branches inside a single graph using Send for fan-out patterns.

How do I version a LangGraph workflow safely?
Treat the graph definition as code, version it in git, and include a graph version in your thread_id. Old checkpoints stay tied to the old graph version, new ones to the new.

Does LangGraph work with local LLMs?
Yes. Anything that exposes an OpenAI-compatible API (Ollama, vLLM, LM Studio) plugs in as the LLM. Quality depends on the model. Llama 3.3 70B and Qwen 2.5 72B are the strongest open-source picks for extraction tasks in 2026.

Can I use LangGraph from JavaScript?
Yes. LangGraph.js is feature-equivalent to the Python version as of late 2025 and is the right pick for Node.js scraping pipelines.

What about cycles? Can a graph loop forever?
Cycles are allowed and useful for retry patterns, but every graph has a recursion_limit config (default 25) that prevents infinite loops. Set it explicitly for your use case.

Can I share state between two unrelated graph invocations?
Yes, by writing to an external store (Redis, Postgres) from inside a node. LangGraph state is per-thread by design; cross-thread sharing is intentional code, not implicit behavior.

How do I add human-in-the-loop approval to a scraping graph?
Use the interrupt_before config to pause the graph before a sensitive node (say, the one that publishes to your warehouse). LangGraph blocks until you call update_state and resume. This is how teams add manual review gates without restructuring the graph.

What about streaming partial results to a client?
The astream_events API emits a stream of node-level events. For long-running scrapes that feed a UI, stream the events over a WebSocket and the UI shows progress as each node completes.

Does LangGraph have built-in support for batching multiple URLs into one LLM call?
No, but you can implement it as a node that buffers up to N URLs and emits a single multi-extraction request. The trade-off is latency for cost: batching cuts LLM tokens by 30 to 50 percent on small extractions but adds a buffering delay. Most teams skip it and accept the per-page cost.

Production gotchas

• The Postgres checkpointer needs an explicit setup() call on first use. Skipping it produces a confusing “relation does not exist” error.
• Conditional edges that return a list of node names cause parallel execution. Returning a single string is sequential routing. Mixing the two is the most common bug we see in code review.
• LangGraph state is merged with a shallow update. If a node returns {"errors": [new_error]}, it overwrites the prior errors list. Use a reducer to append: errors: Annotated[list, operator.add].
• recursion_limit counts node executions, not loop iterations. A complex graph with many parallel branches can hit the default 25 limit unexpectedly.
• The Sqlite checkpointer is fine for development but locks aggressively under concurrent writes. Switch to Postgres before going production.
• Returning a partial state from a node with no fields removed but without including unchanged fields is correct, and merging is automatic. New developers often re-emit the full state thinking they have to, which works but obscures intent.
• The compile step is cheap; recompile on every code change in dev. In prod, compile once at startup and reuse the compiled app across requests.

If you are building a scraping team in 2026, the AI agentic proxies category covers the proxy and infrastructure side that pairs with LangGraph for full production deployments.

Robots.txt and modern scraping ethics in 2026

Robots.txt scraping ethics has become one of the most contested topics in 2026, because the file that started as a polite courtesy in 1994 is now treated as a quasi-contract in some jurisdictions and as marketing copy in others. The AI training surge of 2023 to 2025 forced site operators to add new directives and forced scrapers to take a position on whether those directives bind them. This guide walks through what robots.txt actually is, the new AI-specific directives that emerged in 2024 and 2025, how courts treated robots.txt across jurisdictions, and a defensible team policy you can adopt this quarter.

The audience is technical leads and product owners who need a clear position on how their scraping pipeline handles robots.txt, both for defensibility and for downstream customer expectations.

What robots.txt actually is and is not

Robots.txt is a plain-text file at the root of a domain that signals to automated agents which paths the site operator would prefer they not access. The Robots Exclusion Protocol was first proposed in 1994 by Martijn Koster and was formalised as RFC 9309 in 2022 by Google, the IETF, and a coalition of crawler operators. The RFC made several things explicit that had been folklore: the file is advisory, the syntax is well-defined, and compliance is a choice the crawler operator makes.

What robots.txt is: a published preference. A courtesy protocol. A widely-respected convention that lets site operators communicate scope to bots. It is also, in some courts and contracts, evidence of the site operator’s intent regarding access.

What robots.txt is not: a technical access control. A robots.txt directive cannot stop a non-compliant crawler. The file does not block traffic, does not authenticate users, does not change HTTP behaviour. A scraper that ignores robots.txt is doing something visible and verifiable, but not technically prevented.

The distinction matters because legal arguments about scraping increasingly turn on what the site operator did to communicate scope. Robots.txt is the cheapest, broadest, most-respected way to do that.

For the broader compliance picture, see the GDPR scraping compliance guide and the ethics-first scraping policy.

The 2024-2025 AI directive surge

Until 2023, robots.txt was overwhelmingly used to manage indexing crawlers (Googlebot, Bingbot) and a small number of well-known commercial scrapers. The AI training boom changed that. By mid-2024, a long list of new user-agents had to be considered, and site operators had to decide which to allow.

The major AI-specific user agents in 2026:

User agent	Operator	Purpose
GPTBot	OpenAI	Training data collection
ChatGPT-User	OpenAI	User-initiated browsing in ChatGPT
Google-Extended	Google	Bard/Gemini training opt-out
ClaudeBot	Anthropic	Training data collection
anthropic-ai	Anthropic	Older identifier (legacy)
PerplexityBot	Perplexity AI	Search index for Perplexity
Perplexity-User	Perplexity AI	User-initiated browsing
CCBot	Common Crawl	Open archive used by many models
Bytespider	ByteDance	Used for ByteDance LLMs
FacebookBot	Meta	Llama training and indexing
Applebot-Extended	Apple	Apple Intelligence training opt-out
Diffbot	Diffbot	Knowledge graph extraction
Amazonbot	Amazon	Alexa and product crawling

By Q2 2025, a study of the top 10,000 web domains found that more than 35 percent had added at least one AI-specific Disallow directive, up from less than 5 percent in early 2023. The New York Times, Reuters, the BBC, Stack Overflow, Quora, and most large publishers explicitly disallow GPTBot, ClaudeBot, and PerplexityBot. The signal is unambiguous.

A scraper operating in 2026 that wants to argue good-faith respect for site operator preferences must do more than parse a single robots.txt for Googlebot. The file is a multi-agent instruction set, and ignoring AI-specific directives is increasingly seen as bad-faith conduct.

Court treatment of robots.txt in different jurisdictions

US courts have been consistent that robots.txt is not by itself a legal access control. The HiQ Labs v LinkedIn line of cases (covered separately in the HiQ Labs ruling explainer) confirmed that scraping public data does not automatically violate the CFAA, regardless of robots.txt. However, several lower courts in 2024 and 2025 treated explicit robots.txt directives as relevant evidence of the site operator’s intent in trespass-to-chattels and breach-of-contract claims.

EU courts have leaned more towards treating robots.txt as part of the implied contract of access, especially for AI training use cases. A 2025 Hamburg ruling held that scraping past an explicit AI-bot disallow was relevant in the legitimate interest balancing test under GDPR Article 6(1)(f), tilting the balance away from the scraper.

UK courts have largely followed the US line, emphasising public availability. Singapore courts have not yet ruled directly, but PDPC guidance in 2025 cited robots.txt compliance as evidence of fair processing under the Personal Data Protection Act.

The pattern is clear: robots.txt does not by itself create a legal duty in most jurisdictions, but ignoring it weakens almost every legal defence you might rely on later. Compliance is cheap. Non-compliance is expensive when something goes wrong.

A scraper-side compliance checklist

Control	What it requires	Why it matters
Fetch and parse robots.txt before each domain	RFC 9309 compliant parser	Legal evidence of good faith
Honour your declared user agent	Identify accurately	Trust signal for site operators
Respect Disallow paths	Skip disallowed URLs	Ethical baseline
Honour Crawl-delay	Throttle per directive	Reduces server load
Cache robots.txt for 24 hours max	Re-fetch frequently	Compliance with site changes
Differentiate by purpose	Use different UA for indexing vs training	Allows site to set per-purpose rules
Log compliance decisions	Per-URL allowed/denied audit trail	Defensible posture
Honour Sitemap directives positively	Use sitemap as canonical scope	Reduces wasted requests
Skip noindex meta tags	Combine robots.txt with HTML-level meta	Full coverage
Provide opt-out contact	Public email or web form	Site operators can reach you

The first six rows are the minimum. The last four are the difference between “we comply” and “we are a model citizen.”

Decision tree: should I scrape this URL?

Q1: Does the domain publish robots.txt?
    ├── No  -> Scrape conservatively; default to crawl-delay 5s.
    └── Yes -> Q2
Q2: Does robots.txt list your user agent?
    ├── Yes -> Honour the directives for your UA.
    └── No  -> Q3
Q3: Does robots.txt have a wildcard (*) section?
    ├── Yes -> Honour the wildcard directives.
    └── No  -> Default to allow with conservative crawl-delay.
Q4: Is the URL within a Disallow path?
    ├── Yes -> Skip; log as denied; do not retry.
    └── No  -> Q5
Q5: Is the page tagged with noindex/nofollow at HTML level?
    ├── Yes -> Defer to HTML directive.
    └── No  -> Proceed with respect to crawl-delay.

Each decision is logged. The audit trail is what gives you a defensible posture if a site operator complains.

Practical Python implementation

A minimal RFC 9309 compliant fetcher in Python looks like this. The standard library urllib.robotparser has gaps; protego from Scrapy is more compliant.

from protego import Protego
import requests
from urllib.parse import urlparse

class RobotsCache:
    def __init__(self, user_agent="DRTScraper/1.0"):
        self.user_agent = user_agent
        self.cache = {}

    def can_fetch(self, url: str) -> bool:
        parsed = urlparse(url)
        domain = f"{parsed.scheme}://{parsed.netloc}"
        if domain not in self.cache:
            self._load(domain)
        rp = self.cache[domain]
        if rp is None:
            return True
        return rp.can_fetch(url, self.user_agent)

    def crawl_delay(self, url: str) -> float:
        parsed = urlparse(url)
        domain = f"{parsed.scheme}://{parsed.netloc}"
        if domain not in self.cache:
            self._load(domain)
        rp = self.cache[domain]
        if rp is None:
            return 1.0
        delay = rp.crawl_delay(self.user_agent)
        return float(delay) if delay else 1.0

    def _load(self, domain: str):
        try:
            resp = requests.get(
                f"{domain}/robots.txt",
                headers={"User-Agent": self.user_agent},
                timeout=10,
            )
            if resp.status_code == 200:
                self.cache[domain] = Protego.parse(resp.text)
            else:
                self.cache[domain] = None
        except Exception:
            self.cache[domain] = None

Wire this in front of every request. Log every denial. The cost is one HTTP fetch per domain per session. The benefit is a complete audit trail.

What about Crawl-delay, Request-rate, and Visit-time?

Crawl-delay is supported by most major crawlers but is not part of RFC 9309. It is a de facto standard. Treat it as binding because most site operators expect compliance.

Request-rate and Visit-time are older directives that never reached wide adoption. You can ignore them in 2026 with little risk, but if they are present, the conservative move is to honour them. They cost nothing.

The Sitemap directive is positive: it tells you where the site operator wants you to start. Use it. A scraper that follows the sitemap is far less likely to hit edge-case URLs that the site operator did not anticipate exposing.

The AI training opt-out as a separate signal

Beyond robots.txt, several site operators in 2025 began publishing dedicated AI training opt-out signals. The two main mechanisms in 2026:

1. The TDM Reservation Protocol, an emerging W3C draft that uses HTTP headers and <meta> tags to signal text and data mining opt-out separately from crawler directives.
2. The C2PA content credentials with embedded usage policies, which carry rights metadata for both human and machine consumers.

Both are still maturing. A scraper that wants to take the most defensible 2026 posture honours both signals in addition to robots.txt. It is more work but it places you at the front of the compliance curve.

A defensible team policy

A working policy has six parts: stated principles, technical implementation, audit logging, vendor management, opt-out handling, and review cadence. The shape of each part:

Stated principles: a one-page document, signed by the engineering lead and product lead, declaring that the team respects robots.txt by default, honours AI-specific directives, and treats compliance as a non-negotiable.

Technical implementation: the protego-based fetcher above, deployed in the request middleware of every scraping pipeline. No exceptions.

Audit logging: every denied URL is logged with timestamp, user agent, and the relevant directive. Logs retained for 12 months minimum.

Vendor management: proxy providers, scraping APIs, and data resellers contractually attest to robots.txt compliance.

Opt-out handling: a public contact email (privacy@yourcompany.com) for site operators to request removal, escalation, or clarification.

Review cadence: quarterly review of the principles, the AI user-agent list, and the audit trail.

For a longer treatment of how to write the principles document and operationalise the audit, see the ethics-first scraping policy guide.

External references

The RFC 9309 specification is at datatracker.ietf.org/doc/rfc9309. Google’s robots.txt parser (open source) is at github.com/google/robotstxt. The TDM Reservation Protocol draft is at w3c.github.io/tdmrep. The C2PA content credentials specification is at c2pa.org.

Comparison: respecting robots.txt vs ignoring it

Dimension	Respect	Ignore
Legal exposure (US)	Low	Moderate (evidence in trespass claims)
Legal exposure (EU)	Low	High (impacts GDPR balancing)
Customer trust	High	Low (especially enterprise B2B)
Site operator goodwill	High	Negative
Server load impact	Lower	Higher
Block rate from target	Low	High over time
Cost to implement	Negligible	Negligible
Long-term sustainability	High	Low

The asymmetry is striking. Compliance costs almost nothing. Non-compliance costs a lot when it costs anything.

FAQ

Is robots.txt legally binding?
Not directly in most jurisdictions. It is a published preference. But ignoring it is increasingly treated as evidence of bad faith in court and in regulator investigations.

Should I honour Crawl-delay even if it slows my pipeline?
Yes. The cost is negligible compared to the legal and goodwill risk of ignoring it.

Can I scrape if the site has no robots.txt?
Yes, but default to a conservative crawl-delay (5 seconds) and respect HTML-level noindex/nofollow tags.

What about pages behind login?
Robots.txt only governs publicly reachable URLs. Authenticated pages are governed by the terms of service of the platform.

Does GPTBot Disallow apply to me if I am not OpenAI?
The directive is explicitly addressed to GPTBot. It does not apply to your user agent. But the spirit of the directive is anti-AI-training, and a scraper that ingests data for AI training should honour the intent.

Extended legal and operational analysis

The robots exclusion protocol became RFC 9309 in 2022, formally codifying behaviour that had been industry custom since 1994. RFC 9309 does not by itself create a legal obligation. It documents how compliant crawlers behave. The legal force of robots.txt comes from adjacent doctrines, namely contract (terms of service that incorporate robots.txt by reference), trespass to chattels in some United States jurisdictions, and the Computer Fraud and Abuse Act when access is unauthorised.

The 2024-2026 period saw three shifts. First, AI-specific user agents proliferated, including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Anthropic-AI. Second, publishers began publishing site policy on AI training distinct from search indexing, often by adding AI-specific Disallow rules. Third, courts began treating robots.txt compliance as evidence of good faith even where it was not strictly required.

The hiQ v LinkedIn line of cases established that scraping public data does not by itself violate the CFAA, but did not absolve scrapers of contract or tort exposure. Subsequent cases (Meta v Bright Data 2024, X Corp v Bright Data 2024) reinforced the contract pathway. Both ended in dismissal for the scraper, but only after years of litigation expense. Robots.txt compliance was cited in both as one factor courts weighed.

Implementation patterns for 2026 robots compliance

A robust scraper in 2026 should implement six behaviours.

1. Fetch robots.txt before the first request and cache for at most twenty-four hours.
2. Honour the most-specific User-agent block, falling back to the wildcard.
3. Respect Crawl-delay where supported, with a minimum default of one second per request when not specified.
4. Honour Disallow paths exactly, including trailing slash semantics.
5. Read site-wide AI policy headers including the X-Robots-Tag and any noai or noindex directives.
6. Log every robots decision per request so audits can prove the behaviour.

Code pattern for a compliant fetcher

import urllib.robotparser
from urllib.parse import urljoin, urlparse

class CompliantFetcher:
    def __init__(self, user_agent):
        self.ua = user_agent
        self.parsers = {}

    def can_fetch(self, url):
        host = urlparse(url).netloc
        if host not in self.parsers:
            rp = urllib.robotparser.RobotFileParser()
            rp.set_url(f"https://{host}/robots.txt")
            try:
                rp.read()
            except Exception:
                return False
            self.parsers[host] = rp
        return self.parsers[host].can_fetch(self.ua, url)

    def crawl_delay(self, url):
        host = urlparse(url).netloc
        if host in self.parsers:
            return self.parsers[host].crawl_delay(self.ua) or 1.0
        return 1.0

Comparison: AI crawler policies on top sites in 2026

Site	GPTBot	ClaudeBot	Google-Extended	CCBot
nytimes.com	Disallow	Disallow	Disallow	Disallow
reddit.com	Disallow	Disallow	Allow (paid)	Disallow
stackoverflow.com	Allow	Allow	Allow	Allow
github.com	Allow	Allow	Allow	Allow
medium.com	Disallow	Disallow	Allow	Disallow
wikipedia.org	Allow	Allow	Allow	Allow

The pattern is that publishers with content-licensing revenue tend to disallow AI crawlers, while platforms with developer or community content tend to allow them.

Additional FAQ

Is ignoring robots.txt illegal?
Not by itself in most jurisdictions, but it weakens defences in contract, tort, and statutory disputes. It is also evidence of bad faith in regulator inquiries.

What if there is no robots.txt?
Treat absence as no specific policy. Apply default ethical behaviour including conservative rate limits and identification of the user agent.

Should AI training crawlers honour robots.txt differently from search crawlers?
Yes. The AI-specific user agents exist precisely so publishers can express different policies. A compliant AI crawler reads the AI-specific block first, then the wildcard, then defaults.

Does honouring robots.txt remove all legal risk?
No. Honouring robots.txt is one factor. Terms of service, copyright, privacy law, and trade secret doctrine still apply.

Real cases where robots.txt mattered in court

Two recent decisions illustrate how courts treat robots.txt in 2024-2026.

In Thomson Reuters v. Ross Intelligence (D. Del., February 2025 summary judgment), the court found that Ross’s training of a competing legal research AI on Westlaw headnotes was not protected fair use. While the case turned primarily on copyright and the commercial-substitution analysis, the trial record included extensive evidence about how Ross obtained the headnotes through a third-party intermediary that ignored Westlaw’s terms and crawl restrictions. Judge Bibas referenced the access pattern in the bad-faith analysis. The decision is now the most-cited US precedent for the proposition that disregarding access controls weakens an AI training defence.

In The New York Times v. Microsoft and OpenAI (S.D.N.Y., 2024 ongoing), the Times’ complaint specifically pleads that OpenAI’s GPTBot ignored or post-dated the Times’ robots.txt Disallow directive for the AI-specific user agent. The pleading frames robots.txt compliance as a baseline good-faith expectation in the publishing industry. While the case has not yet reached merits judgment, the pleading strategy reflects how plaintiffs now use robots.txt non-compliance as a narrative anchor for bad-faith allegations.

Both cases reinforce the operational lesson: robots.txt is not legally binding on its own, but ignoring it is now treated as a meaningful evidentiary fact in almost every commercial scraping dispute. The cost of compliance is trivial; the cost of non-compliance compounds across litigation, regulator inquiries, and platform agreements. A scraper that honours robots.txt by default and logs every decision has a defence narrative ready before any dispute arises.

The history and standardisation of robots.txt

Robots.txt was proposed by Martijn Koster in 1994 as a voluntary protocol for crawlers to declare and discover crawl preferences. It remained an informal de-facto standard for nearly three decades. RFC 9309, published in September 2022, formally specified the protocol after Google led a working group to align implementations.

RFC 9309 nailed down several previously ambiguous behaviours. The matching rules for User-agent strings, the handling of multiple matching groups, the precedence of Allow and Disallow rules, the canonicalisation of paths, and the maximum file size (500 KiB by default) are now specified. The RFC does not specify rate limiting, the meaning of Crawl-delay, or AI-specific user agents. Those remain extensions on top of the base protocol.

The standardisation matters for scrapers because compliant behaviour is now testable. A scraper can be checked against RFC 9309 test vectors, and gaps can be identified and fixed. Pre-RFC implementations often differed in edge cases. Post-RFC the expectation is that compliant crawlers behave identically.

Beyond robots.txt: meta robots, x-robots-tag, and llms.txt

Robots.txt is the front door but not the only signal. Meta robots tags in HTML, the X-Robots-Tag HTTP response header, and the proposed llms.txt convention all carry crawler instructions.

Meta robots tags appear in HTML head and apply per-page. They support directives including index, noindex, follow, nofollow, noarchive, nosnippet, and AI-specific directives like noai and noimageai (proposed 2024). A scraper should parse these per page.

X-Robots-Tag is the response header equivalent, useful for non-HTML resources (PDFs, images, JSON APIs). The directive vocabulary mirrors meta robots. Scrapers fetching non-HTML content should check the header.

The llms.txt convention proposed in 2024 by Jeremy Howard provides a structured site map specifically for LLM consumers. It complements rather than replaces robots.txt. Some publishers ship both, with robots.txt declaring access policy and llms.txt declaring content structure for AI clients.

The ethical dimension beyond compliance

Compliance with robots.txt is the floor, not the ceiling. Ethical scraping in 2026 considers four additional factors that robots.txt does not capture.

First, server load. A scraper that respects robots.txt but hammers the server with concurrent requests still imposes externalities. Conservative concurrency and adaptive backoff are part of ethical operation.

Second, content type. Some content (personal social media posts, sensitive forum threads) deserves additional restraint regardless of what robots.txt says. The scraper should apply context-sensitive judgement.

Third, downstream use. A scrape that respects robots.txt but feeds the data into a system that the publisher would object to (for example training a competing AI on a paywalled publisher’s free pages) is technically compliant but ethically thin.

Fourth, transparency. A scraper identified by a unique User-Agent string, with operator contact information in the User-Agent or in a public crawler page, makes itself accountable. Anonymous crawlers are correlated with abuse and are increasingly blocked at the platform level.

Next steps

The fastest improvement is to drop a Protego-based middleware into your scraper this week, log every denial for 30 days, and review the log for surprises. If you find your scraper has been hitting Disallow paths, fix it before a site operator notices. For the broader policy and team rollout, head to the DRT compliance hub and start with the ethics-first policy guide.

This guide is informational, not legal advice.

Claude Code vs Cursor for web scraping projects

The Claude Code vs Cursor scraping decision matters because both tools collapse the loop between writing a scraper and running it, but they collapse it differently. Cursor lives in your editor and is optimized for in-file edits with AI assist. Claude Code runs as a CLI agent and is optimized for autonomous execution of multi-step tasks. For scraping work, that distinction shows up immediately. Cursor wants you to drive. Claude Code wants to drive itself.

This comparison is built from running both tools on the same scraping projects in early 2026. Identical targets, identical proxies, identical models where possible. We covered building a Lazada price monitor, a job board aggregator, and a lightweight news clipping pipeline. Below is the honest picture of where each tool wins, where they tie, and which one we would pick for a new scraping project today.

What each tool actually is

Claude Code is Anthropic’s command-line agent that runs in any terminal, has direct file system access, executes arbitrary bash, and operates in an autonomous loop until your task is done or it asks for input. The default model is Claude Sonnet 4.5 with optional Opus for harder tasks.

Cursor is a VSCode fork with deep AI integration. The agent mode (released 2024, refined heavily through 2025) is now closer to Claude Code in capability, but its center of gravity is still the editor. You drive selections, you accept diffs, you steer.

Both ship MCP support, both can use external scraping tools, both can read your codebase. The difference is the human-in-the-loop ratio.

Architectural philosophy in one sentence each

Claude Code believes the best dev loop is “describe the outcome, walk away, come back to a green build.” Cursor believes the best dev loop is “see every diff, approve the smart ones, reject the bad ones, ship.” Neither is wrong. The right pick depends on which loop fits your team’s tolerance for autonomy.

Setting up for a scraping project

For Claude Code, the install is one line:

npm install -g @anthropic-ai/claude-code

Then in your project directory:

cd ~/projects/lazada-monitor
claude

You are dropped into an interactive session that already knows your file tree. Add a CLAUDE.md at the project root with conventions and tool preferences, and the agent reads it every session.

For Cursor, install the editor and open the project. Configure model preferences in settings. Add a .cursorrules file with project guidance.

Neither tool ships scraping-specific helpers. You bring your own Playwright, your own proxy pool, your own database client.

Sample CLAUDE.md for a scraping project

A useful starter file lives at the project root and shapes every session. Here is a battle-tested template:

# Project: Lazada Price Watcher

## Stack
- Python 3.12, Playwright, SQLite, httpx, pydantic
- Proxies via Singapore mobile proxy (creds in .env)
- Telegram alerts via python-telegram-bot

## Conventions
- All scraping code under scrapers/
- Pytest tests under tests/, run with `make test`
- Lock requirements with pip-compile
- Never commit .env

## Hard rules
- Never store passwords in plain text in DB
- Never bypass robots.txt without an explicit go-ahead
- Always validate Pydantic models before DB writes

Cursor’s .cursorrules covers the same ground but is read more passively. Claude Code re-reads CLAUDE.md every session, so updates take effect immediately.

A real scraping task: building a Lazada watcher

The test task: build a Python script that monitors a list of Lazada Singapore product URLs, extracts price and stock daily, writes to SQLite, and sends a Telegram alert when price drops more than 10 percent.

With Claude Code, the prompt was:

Build a Lazada price watcher. Read URLs from data/products.txt, scrape title,
price, and stock for each, store in data/prices.db with a timestamp, and send
a Telegram message via the bot token in .env when any price drops 10% or more
since the last run. Use Playwright with stealth defaults. Include retries and
proxy support. Add a cron-friendly entrypoint.

Claude Code wrote ten files in eleven minutes, including a Playwright scraper, a SQLite migration, a Telegram client, a Makefile, a requirements.txt, a README.md, and a sample crontab line. It ran the scraper against three test URLs to verify. Total tokens billed: about 380k input, 24k output, $1.20 on the Sonnet 4.5 model.

With Cursor, the same prompt produced a single-file scaffold in about three minutes. The scaffold was good but missing the Telegram client, the migration script, and the proxy support. Each follow-up needed a new agent prompt or manual edits. Total time to functional parity: 28 minutes including six follow-up turns.

Claude Code wins on autonomous shipping of a complete scaffold. Cursor wins on speed of any single edit and on quality of in-file refactor suggestions.

Second task: a job board aggregator

We ran a second test where the requirement was looser: aggregate jobs from Indeed Singapore, JobStreet, and LinkedIn into a single Postgres table, with deduplication by company plus title plus posted date.

Claude Code asked one clarifying question (whether to honor LinkedIn’s Terms of Service or just scrape with login) and then shipped the rest. Cursor produced a working Indeed scraper quickly but never volunteered to think about deduplication or schema, treating each ask as discrete.

The pattern repeated. Claude Code reasons across the whole project, Cursor reasons across the visible buffer.

Tool use and MCP integration

Both tools speak MCP. Configuration is similar.

Claude Code reads ~/.claude/mcp.json:

{
  "mcpServers": {
    "scraping": {
      "command": "python",
      "args": ["/Users/me/scraping-mcp/server.py"]
    },
    "playwright": {
      "command": "npx",
      "args": ["-y", "@executeautomation/playwright-mcp-server"]
    }
  }
}

Cursor reads ~/.cursor/mcp.json with the same shape.

In practice, Claude Code uses MCP tools more aggressively. The agent will reach for a screenshot tool if you mention you cannot tell what is rendering. Cursor, in agent mode, prefers to write code that calls the tool directly. Both work, both are correct, the styles differ.

For wiring up an MCP scraping server, see our scraping with MCP servers guide.

Tool selection accuracy

In a 50-task audit where both tools had access to the same five MCP tools (fetch, screenshot, extract, search, crawl), Claude Code picked the correct first tool 88 percent of the time. Cursor picked the correct first tool 71 percent of the time. The gap mostly came from Cursor’s preference to write fresh Python rather than reach for a tool, which is fine when the tool is overkill but wastes time on bread-and-butter scraping.

Debugging a broken scraper

This is where the styles diverge most.

Claude Code, when a scraper breaks, will run the script, read the traceback, edit the file, run again, and keep iterating until the test passes or it hits its task budget. You can step away.

Cursor agent mode will propose a fix, wait for you to accept, run the script if you ask it to, and bring you the next traceback. The loop is faster per iteration but slower per debugging session because every step needs your attention.

For shallow bugs (typo, missing import, wrong selector), Cursor’s faster loop wins. For deep bugs (race condition between Playwright launches, weird Cloudflare interaction, database lock), Claude Code’s autonomous iteration wins because it will try ten things in the time you would still be reading the third Cursor diff.

A real debugging vignette

A flaky Playwright test that failed once every five runs took Claude Code 22 minutes and three exploratory iterations to diagnose: a wait_until="networkidle" that was triggering before a delayed XHR, fixed with an explicit selector wait. Cursor took roughly the same time but the engineer had to babysit each step. The wall clock was identical, the engineer hours were not.

Side-by-side comparison

Dimension	Claude Code	Cursor
Default model	Claude Sonnet 4.5	Claude Sonnet 4.5 or GPT-5
Native interface	Terminal	VSCode fork
Best at	Multi-step autonomous tasks	In-editor refactors, line-by-line edits
Worst at	Real-time UI work, design feedback	Long unattended jobs
MCP support	Yes, native	Yes, native
Codebase awareness	Reads on demand, follows symlinks	Always-on indexed search
Cost per scraping pipeline scaffold	$0.50 to $2.00 per session	Subscription + variable model cost
Steepest learning curve	Bash and Unix fluency expected	None, IDE-native
Wins on Lazada monitor task	Faster end-to-end	Faster per-edit
Wins on debugging deep issues	Yes	No
Wins on quick selector fix	No	Yes
Plays well with sub-agents	Yes (Task tool)	Limited
Inline screenshot viewing	Via MCP only	Native
Multi-window/multi-cursor	No	Yes
Background mode (run in CI)	Yes	Limited
Autonomous test run loop	Yes	No (asks)

Cost analysis

Claude Code charges per-token through your Anthropic API key, or you can use a Claude Pro/Max subscription for fixed monthly cost with quota.

Cursor charges a flat $20/month for Pro with 500 fast model requests, then variable cost per request beyond. The Cursor model selection includes Claude Sonnet, Claude Opus, GPT-5, and Gemini.

For a small team scraping a few sites a day, Cursor Pro is the cheaper bill. For a heavy scraping shop where engineers run multi-hour autonomous jobs, Claude Code on API billing is more predictable because you only pay for what you use.

Real numbers from one week of mixed scraping work on a single engineer’s machine:

Tool	Sessions	Hours	Cost
Claude Code (API)	14	22	$34
Cursor (Pro + overage)	31	18	$24

Cursor came out cheaper for the same engineer doing the same projects, mostly because the editor-driven loop encouraged smaller, cheaper requests. Claude Code’s autonomous loop racks up tokens faster.

When the cost picture flips

Cost flips in favor of Claude Code as soon as the engineer steps away. A four-hour autonomous session that builds and tests three new scrapers might run $6 to $10 on Claude Code, but it freed the engineer for other work. The same outcome in Cursor would take the engineer four hours of attention. At any reasonable engineer hourly rate, the autonomous time wins.

The pattern we see in 2026 mid-size scraping teams: Cursor for the morning standup-to-lunch surgical work, Claude Code as a co-worker assigned long-running greenfield projects.

Working with proxies

Both tools handle proxy code identically because the proxy logic lives in your scraper, not the agent. The difference is in how easily the agent can debug a proxy issue.

Claude Code can curl a proxy directly to verify it works:

> Run: curl -x http://user:pass@proxy.example.com:8000 https://httpbin.org/ip

It reads the response and adjusts the scraper. Cursor can run the same curl through the integrated terminal but the result lives in a panel you have to focus.

For a deeper guide on proxy choices, see our best residential proxy providers 2026 writeup.

Secret handling

Both tools respect a .env file and neither will read or transmit it without a deliberate prompt. The risk surface is the same: a careless paste of a key into the chat is the most common leak vector. Set up .gitignore and pre-commit hooks regardless of which tool you use.

Headless browser handling

Both tools can drive Playwright. The interesting question is what they do when the scraper opens a browser window.

Claude Code does not have a UI, so headed Chromium opens on your local display. The agent can take screenshots if you give it a screenshot MCP tool. Otherwise it is blind to UI state.

Cursor in agent mode can ask Playwright for a screenshot and view the resulting PNG inline. This is a real advantage when you are debugging why a click is not landing.

For purely headless pipelines where the agent never needs to see the browser, this is a wash.

Multi-agent coordination

Claude Code supports sub-agents through the Task tool. You can spin up a specialist sub-agent for one part of the pipeline (say, captcha solving) and have it work in isolation. Cursor does not have a clean equivalent in 2026.

For scraping projects that need parallel work (say, scrape ten sites in parallel and aggregate), Claude Code’s sub-agent pattern is a real differentiator. You write a parent agent that dispatches one sub-agent per site, and the parent aggregates results.

For more on multi-agent scraping, see Multi-agent scraping with AutoGen in 2026.

A simple parallel pattern

A pattern that works well in production: a parent Claude Code session reads a list of 50 URLs, spawns 5 sub-agent tasks each handling 10 URLs, and aggregates the JSON outputs. The parent agent enforces a per-sub-agent timeout and retries failed batches. Total wall-clock time on 50 mixed URLs: roughly 8 minutes versus 35 minutes for sequential. Cost per sub-agent stays predictable because each one operates with a small task.

Documentation and community

Anthropic’s Claude Code docs are the canonical reference. The community on the official Discord and the agent-construction subreddit is active and ships custom skills daily.

Cursor’s docs are clean. The community is enormous (it is the most popular AI editor in 2026) but most discussion is general coding, not scraping-specific.

Workflow patterns we have seen succeed

A few patterns recur across teams that ship scraping work fast.

The “pair programmer” pattern uses Cursor for the scaffold and Claude Code for the harden-and-deploy. The engineer sketches in Cursor, then closes the editor and lets Claude Code add tests, error handling, retries, observability, and a Dockerfile.

The “specialist agent” pattern uses Cursor for daily UI editing and a dedicated long-running Claude Code instance per scraper. Each Claude Code instance owns its scraper directory, runs hourly cron via a wrapper, and posts diffs and incident summaries to Slack.

The “hands-off rebuild” pattern, when an old scraper fails, prompts Claude Code with “this scraper is broken in tests/test_x.py; figure out why and fix it” and walks away. Comes back to a passing build or a clear write-up of why the target site changed in a way that needs a product decision.

Which one to pick

If your team writes scraping code daily and you want the AI to handle multi-step shipping (build, test, deploy a scraper from a one-paragraph prompt), pick Claude Code. The autonomous loop saves real time.

If your team writes scraping code occasionally and most of your work is editing existing pipelines, pick Cursor. The in-editor experience is better for the surgical edit workflow that dominates maintenance.

The honest answer for many shops in 2026 is to use both. Cursor for daily editing, Claude Code for the heavy autonomous tasks. They cost together about what one engineer’s coffee budget runs in a month.

Frequently asked questions

Can I run Claude Code inside Cursor’s terminal?
Yes. Cursor’s integrated terminal runs claude like any other shell. You get Cursor’s editor experience plus Claude Code’s autonomy. This is the setup we recommend for engineers who like both.

Does Cursor’s MCP support match Claude Code’s?
Effectively yes in early 2026. Cursor was slower to ship MCP but the implementation now covers tools, resources, and prompts. Stdio and HTTP transports both supported.

Which one handles long context better?
Both default to Claude Sonnet 4.5 with 1M context. The actual context-handling quality is identical because the model is the same. The differentiator is how each tool prunes context across long sessions.

Can either tool drive Selenium for legacy targets?
Yes. Both can write and run Selenium code. Selenium is the right pick when you must support an ancient browser stack. For everything in 2026, Playwright is the better default.

What about Continue, Zed, Aider, or Cline?
Cline is the closest free competitor to Claude Code. Aider is excellent for git-aware in-place edits. Zed has shipping AI assist that is improving fast. None of them ship the autonomous loop with the polish Claude Code has, in our testing.

Can Claude Code run in CI to repair flaky scrapers automatically?
Yes. Pipe a failure log into claude --resume <session-id> from a GitHub Action and the agent will attempt a fix and open a PR. Set a budget cap to avoid runaway runs.

Which tool is better for a non-engineer running a one-off scrape?
Neither, honestly. Both expect baseline command-line and Python familiarity. For a true non-coder, look at no-code tools like Apify or browser extensions like Instant Data Scraper.

Does Cursor’s agent mode work without a Cursor subscription?
The free tier is severely limited (50 slow requests per month). For any serious scraping work, you need at least Pro.

How do both tools handle very long files like a 2000-line scraper?
Both default to chunked reads, but Claude Code is more conservative about loading the whole file into context. Cursor will sometimes load and re-emit the whole file in a single edit, which costs more tokens but produces a cleaner diff. For files over 1500 lines, Claude Code is the safer pick because partial edits are less likely to corrupt indentation or imports.

Can either tool ship to production directly?
Both can run git push, gh pr create, and CD pipelines via shell. Neither has a native deploy concept. The pattern that works is to wire your existing CD pipeline (GitHub Actions, Vercel, Fly) and let the agent push commits that trigger deploy.

Common pitfalls and gotchas

A short list of things teams trip over in their first month with either tool.

Letting the agent edit the lockfile silently. Both tools will helpfully update requirements.txt or package.json, but they sometimes pin to versions that break elsewhere in your stack. Make CI run pip install -r requirements.txt on a clean cache and fail loudly if it does not resolve.

Forgetting to budget the agent. Claude Code without a --max-turns cap can loop on a confused task and burn $10 in 20 minutes. Always set a budget for autonomous sessions.

Trusting the agent’s claim that “tests pass” without checking. Both tools occasionally report a green build when in fact they ran a subset. Make make test the only acceptance criterion in your CLAUDE.md, and verify by re-running yourself for important changes.

Using Cursor agent mode for tasks where the right answer is a one-line shell command. Cursor will write a Python script when awk would do. Recognize when a task does not need an editor at all.

Mixing both tools on the same file in the same minute. Both write to disk; both watch the file system. Race conditions on saves are real. Use one tool per task at a time.

For more comparisons across the agentic coding tool space and how each pairs with scraping infrastructure, browse our AI modern scraping category.

CCPA compliance for scrapers handling US consumer data

CCPA scraping compliance has grown into the second-most-cited blocker for B2C data pipelines, right behind GDPR. The California Consumer Privacy Act, as amended by the California Privacy Rights Act (CPRA) and now enforced by the California Privacy Protection Agency (CPPA), reshaped what US-touching scrapers can safely do. Many engineering teams still operate under the older 2018 CCPA mental model, and that gap is exactly where 2025 and 2026 enforcement actions landed. This guide walks through the actual rules as enforced today, the public-record carve-out that scraping operators rely on (and frequently misread), the consumer rights you must honour, and a checklist your team can implement this quarter.

The audience here is the data engineer or product lead who already runs a scraping pipeline that touches California residents and needs a defensible compliance posture in 2026.

What CCPA actually covers in scraping context

CCPA applies to any business that collects personal information of California residents and meets one of three thresholds: more than USD 25 million in annual revenue, buys or sells personal information of 100,000 or more consumers or households, or derives 50 percent or more of annual revenue from selling or sharing personal information. Scrapers hit the second and third thresholds easily.

Personal information under Cal. Civ. Code Section 1798.140(v) is defined extremely broadly: any information that identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked, directly or indirectly, with a particular consumer or household. The list of examples runs from the obvious (name, email, address, phone) to the operationally relevant (IP addresses, cookie identifiers, browsing history, geolocation, inferences drawn to create a consumer profile). If you scrape it and it relates to a person, it is personal information.

CPRA added a new category in 2023, sensitive personal information (SPI), which includes Social Security numbers, precise geolocation, race, ethnicity, religious or philosophical beliefs, union membership, contents of mail/email/text messages, genetic data, biometric data, health information, sex life, and sexual orientation. SPI carries additional restrictions and is the highest-risk class for scrapers.

For the broader US context and how state-level privacy laws are converging, see the personal vs public data scraping framework. For the EU equivalent, the GDPR compliance guide for scrapers is the right next read.

The publicly available information carve-out (and its limits)

CCPA explicitly excludes “publicly available information” from the definition of personal information. Section 1798.140(v)(2) defines publicly available as information that is lawfully made available from federal, state, or local government records, or information that a business has a reasonable basis to believe is lawfully made available to the general public by the consumer or from widely distributed media; or information made available by a person to whom the consumer has disclosed the information if the consumer has not restricted the information to a specific audience.

This is a real carve-out, but it is narrower than scrapers often assume. Three pitfalls.

First, the “lawfully made available” qualifier means information leaked, hacked, or scraped in violation of terms of service does not become publicly available just because it ended up online. A doxxing forum dump is not publicly available information under CCPA, even if you can read it.

Second, the “consumer has not restricted” carve-out means a profile a user marked private but you accessed via a workaround does not qualify. The user’s restriction state at the time of collection matters.

Third, inferences drawn from publicly available information are not themselves publicly available. If you scrape a profile photo from a public LinkedIn page and then run a face-recognition model against it to infer ethnicity, the inferred ethnicity is personal information (and likely SPI), even though the source was public.

The CPPA has signalled in 2024 and 2025 enforcement guidance that it reads the carve-out narrowly. Treat it as a defence you may invoke, not a shield you assume.

Compliance checklist for scrapers handling California data

Control	What it requires	Why it matters
Privacy policy with CCPA disclosures	Categories of PI collected, sources, purposes, third parties	Section 1798.130
“Do Not Sell or Share My Personal Information” link	Homepage link if you sell or share	Section 1798.135
Opt-out mechanism	Functional within 15 business days	Section 1798.135
Right to know request handling	Verifiable response within 45 days	Section 1798.130
Right to delete request handling	Verifiable deletion within 45 days	Section 1798.105
Right to correct request handling	Honour correction requests	Section 1798.106
Limit use of SPI	Honour the limit-the-use-of-SPI right	Section 1798.121
Service provider contracts	CCPA-compliant DPAs with vendors	Section 1798.140(ag)
Data minimisation	Only collect what is necessary and proportionate	CPRA Section 1798.100(c)
Retention schedules	Disclose and enforce retention periods	Section 1798.100(a)(3)
Annual cybersecurity audit (if high risk)	CPPA forthcoming regulations	CPRA
Risk assessment for high-risk processing	CPPA forthcoming regulations	CPRA

A scraper that ticks every row above operates inside the safe harbour. One that ticks half is exposed.

Consumer rights and the request workflow

CCPA grants California residents seven core rights: right to know, right to delete, right to correct, right to opt out of sale or sharing, right to limit use of SPI, right to non-discrimination, and right to data portability. For a scraper, the operationally heavy rights are right to know, right to delete, and right to opt out of sale.

Right to know means a consumer can request the categories and specific pieces of personal information you collected about them, the sources, the business or commercial purpose, and the third parties you shared with. You have 45 days to respond. The CPPA expects you to be able to identify the consumer in your dataset, which means your storage schema needs to be queryable by identifier types you collected (email, name plus zip, device ID).

Right to delete means once a verifiable request is received, you must delete the consumer’s personal information from your records and instruct service providers and contractors to do the same. There are exceptions (legal compliance, security, free speech, internal analytics consistent with consumer expectations), but the default is delete.

Right to opt out of sale or sharing is broader than many teams realise. “Sale” includes any disclosure for monetary or other valuable consideration. If you scrape data and license it to customers, that is a sale. You must honour the Global Privacy Control (GPC) signal as a valid opt-out, automatically and without requiring further action. The CPPA confirmed this in 2024 enforcement actions.

For a worked decision tree on how to triage rights requests, see the ethics-first scraping policy guide.

How CCPA enforcement shifted in 2024 and 2025

The CPPA, which took over administrative enforcement in 2023, brought a rulemaking and audit-driven approach that the original Attorney General enforcement lacked. Three trends.

First, the CPPA targeted data brokers explicitly. The Delete Act (SB 362), in force since 2026, requires data brokers to register annually and to honour a single deletion mechanism that consumers can use across all brokers at once. Scrapers that resell personal information meet the data broker definition under California law, full stop. Registration is not optional.

Second, enforcement action shifted from notice-and-cure to direct fine. The 30-day cure period that the original CCPA included was eliminated by CPRA. A scraper that fails to honour an opt-out request can face civil penalties of USD 2,500 per violation or USD 7,500 per intentional violation, with each individual consumer counted separately. A breach affecting 10,000 California residents can produce a USD 75 million liability ceiling.

Third, the CPPA has aggressively enforced the GPC requirement. A 2025 settlement with a major data broker centred on the broker’s failure to recognise GPC signals automatically. The fine was significant, the public-shaming letter was widely read, and the message was unmistakable: GPC is mandatory.

For the parallel UK and EU enforcement environment, see the GDPR compliance guide.

Decision tree for a US-touching scrape

Q1: Does the target site host personal info of California residents?
    ├── No  -> CCPA likely not in scope. Document the assessment.
    └── Yes -> Q2
Q2: Is the data clearly within the publicly available carve-out?
    ├── Yes -> Document why; still recommended to honour deletion requests.
    └── No  -> Q3
Q3: Does your business meet a CCPA threshold?
    ├── No  -> CCPA does not apply directly; state laws may.
    └── Yes -> Q4
Q4: Have you published a CCPA-compliant privacy policy?
    ├── No  -> Publish before launching.
    └── Yes -> Q5
Q5: Do you sell or share the scraped data?
    ├── Yes -> Add "Do Not Sell or Share" link; honour GPC; register if data broker.
    └── No  -> Q6
Q6: Will you process sensitive personal information?
    ├── Yes -> Honour limit-use right; consider risk assessment.
    └── No  -> Proceed; log the assessment in your records.

Service provider, contractor, and third party

CCPA distinguishes between three downstream relationships. A service provider processes personal information on your behalf under a written contract that restricts further use. A contractor is similar but typically engaged on a one-off basis. A third party receives personal information for its own purposes; this is where “sale” attaches.

Scrapers commonly sit in two roles: as a service provider when they scrape on behalf of a customer under a DPA, and as a third party when they license the dataset for the customer’s independent use. The DPA you sign with a proxy provider is a service provider agreement. The DPA you sign with a customer who buys your dataset is potentially a third-party arrangement, depending on how restrictive the contract is. Get this categorisation wrong and you have either misclassified a sale (CPPA fine territory) or imposed restrictions you cannot enforce (commercial conflict).

Comparison: CCPA vs GDPR for scrapers

Dimension	CCPA / CPRA	GDPR
Personal data definition	Broad, includes household	Broad, individual only
Lawful basis required	No, but right to opt out of sale	Yes, six bases
Public data carve-out	Yes (publicly available)	None
Right to delete	Yes (with exceptions)	Yes (Article 17)
Right to opt out of sale	Yes (mandatory GPC)	Implicit in lawful basis
Sensitive data category	Yes (SPI, CPRA addition)	Yes (special categories)
Extraterritorial reach	Yes if doing business in CA	Yes if processing EU data
Statutory damages	Yes, per-violation civil penalty	Administrative fines up to 4% revenue
Cure period	None (after CPRA)	Limited
Private right of action	Limited (data breach only)	Yes (Article 82)

The two regimes overlap heavily but diverge on lawful basis and the public data carve-out. Build for both and you have most US and EU coverage.

External references

The canonical statute is the California Civil Code, Title 1.81.5, hosted at oag.ca.gov/privacy/ccpa. The CPPA publishes its regulations and enforcement actions at cppa.ca.gov. The Global Privacy Control specification is at globalprivacycontrol.org.

Operationalising opt-out signals

The Global Privacy Control is a browser-emitted signal in the request headers (Sec-GPC: 1) that indicates the user has opted out of the sale or sharing of their personal information. The CPPA requires you to honour it automatically. Implementation for a scraping operator is two-part: detect the GPC signal at any user-facing surface (your website, your customer portal, your data preview pages) and treat any consumer whose original collection context included GPC as opted out by default.

For a scraped dataset, this is harder, because you typically do not have GPC headers from the scraping target. The practical workaround: when you receive a deletion or opt-out request, do not require the requester to re-authenticate from a GPC-enabled browser. Treat the request as valid based on identifier match alone, and document the verification path.

Special cases: data brokers, AI training, and hiring

The Delete Act (SB 362) made California the first US state with a single-source deletion mechanism for data brokers. Once the deletion portal is fully live (2026 phased rollout), any consumer can submit a single request that deletes their data across every registered broker. Scrapers who meet the data broker definition must register, must honour the central deletion list, and must not re-collect deleted consumers’ data within an enforcement window.

AI training is now subject to additional CPPA risk assessment requirements when the training set includes California residents’ personal information at scale. The risk assessment must address the necessity of the training data, the safeguards against re-identification, and the consumer rights surface for opt-out and deletion. Several large model providers were quietly fined in 2025 for failing to file the risk assessment.

Hiring and employee data was carved out of CCPA from 2018 to 2023 but became fully covered in 2023. A scraper that pulls professional profile data of California residents now operates under full CCPA, with no employment-context exemption.

FAQ

Is publicly available data exempt from CCPA?
Partially. The carve-out only covers data that was lawfully made publicly available and that the consumer has not restricted. Inferences drawn from public data are not themselves public.

Do I need to honour Global Privacy Control signals?
Yes. The CPPA confirmed in 2024 that GPC is a valid opt-out signal that must be honoured automatically.

What is the fine range under CCPA in 2026?
Civil penalties are USD 2,500 per violation or USD 7,500 per intentional violation, with each individual consumer counted separately.

Am I a data broker if I scrape and sell?
If you knowingly collect and sell personal information of consumers with whom you do not have a direct relationship, yes, and you must register annually under the Delete Act.

Does CCPA apply to B2B data?
Yes since 2023. The B2B carve-out expired and professional contact data of California residents is now fully covered.

Extended enforcement analysis 2024-2026

The California Privacy Protection Agency moved from rulemaking to active enforcement during 2024 and 2025. The DoorDash settlement (February 2024, USD 375,000) was the first to specifically cite cross-context scraping of consumer data without a working opt-out signal. The CPPA’s enforcement advisories in 2025 covered three patterns relevant to scrapers, namely failure to honour the Global Privacy Control header, failure to recognise the Sec-GPC header on automated traffic, and failure to surface a Do Not Sell or Share My Personal Information link in the privacy notice that links the scraping operation to the consumer-facing brand.

The Sephora case (August 2022, USD 1.2 million) remains the touchstone for California enforcement on opt-out signals. Sephora was found in violation for failing to process opt-out signals as valid CCPA requests. Every scraper that touches California residents should treat that case as authoritative and design GPC handling into the ingest layer, not a downstream marketing tool.

A pattern emerged in 2025 that scrapers should plan for. The CPPA increasingly views scraping followed by enrichment, segmentation, and resale as a sale or sharing event under CCPA, even if the scraping operator does not directly transfer data. The triggering test is whether the consumer would reasonably understand that their public information would be combined with non-public signals and sold downstream. For B2B people-data vendors this is now the central compliance question.

Implementation patterns for a CCPA-clean pipeline

The minimum control set for a US-touching 2026 scraping pipeline includes nine items.

1. A GPC and Sec-GPC header check at every fetch with the result logged per request.
2. A privacy notice link surfaced on every consumer-facing surface that touches scraped data.
3. A right-to-know workflow that responds within forty-five days with extension up to ninety.
4. A right-to-delete workflow with verification that does not over-collect identity proof.
5. A right-to-correct workflow added in 2023 amendments and now actively enforced.
6. A right-to-limit-use-of-sensitive-personal-information workflow.
7. A service provider contract with every downstream processor.
8. A data inventory that distinguishes personal information from sensitive personal information.
9. A retention schedule documented per category and enforced.

Worked example: GPC handling at fetch time

def should_index(response, headers):
    gpc = headers.get("Sec-GPC", "0")
    if gpc == "1":
        log.info("gpc_signal_present", url=response.url)
        return False  # treat as opt-out for downstream sale or share
    return True

The check belongs at the ingest layer because removing data downstream after vectorisation is harder than skipping it at fetch.

Additional FAQ

Do I need a CCPA notice if I never sell data?
Yes if you process personal information of California residents above the thresholds. The notice obligation is independent of sale.

Does the publicly available carve-out cover LinkedIn profiles?
Generally no. The carve-out applies to information lawfully made available from federal, state, or local government records, plus information the consumer or their authorised agent has made available. Commercial platforms with terms of service restricting bulk access do not satisfy the carve-out by themselves.

What is the difference between sale and share under CCPA?
Sale is exchange for monetary or other valuable consideration. Share is disclosure for cross-context behavioural advertising. Both trigger opt-out rights and the Do Not Sell or Share link.

How do I verify a deletion request without over-collecting?
Match against information you already hold. A consumer should not have to provide more identity than the minimum needed to confirm the match. Documentation of the verification logic is part of compliance.

Practical scope determination for CCPA

Determining whether the CCPA applies to a scraping operation requires analysis on three axes. First, does the scraping operation process personal information of California residents. Second, does the operating entity meet the size threshold (USD 25 million annual revenue, or 100,000 California consumers, or 50 percent of revenue from selling personal information). Third, does the activity fit within the CCPA’s exempted categories.

For most commercial scrapers the first axis is yes by default, the second axis is met for any team above small startup size, and the third axis offers little relief. The narrow exemptions for medical information governed by HIPAA, financial information governed by GLBA, and certain business-to-business communications during the transition period in earlier amendments are of limited use to a generic scraper.

The 2024 amendments and CPPA regulations clarified that aggregators, brokers, and AI training data vendors fall squarely within scope when they touch California-resident data. The CPPA’s enforcement priorities published in 2025 listed data brokers and AI training data as the top two areas of focus. Scrapers in those categories should plan for a CCPA registration where applicable and a higher level of regulatory attention.

Sensitive personal information and the right to limit

The CCPA’s 2023 amendments introduced a new category of sensitive personal information (SPI) and a new right to limit its use and disclosure. SPI includes Social Security numbers, driver’s licence numbers, financial account information, precise geolocation, racial or ethnic origin, religious beliefs, mail and email content, genetic data, biometric data, health data, and sex life or sexual orientation.

For scrapers the SPI category is operationally similar to GDPR Article 9 special category data. The scraper should detect SPI at ingest, route it to a separate handling pathway with stricter access controls, and surface a right-to-limit-use mechanism on the consumer-facing surface.

The right to limit is narrower than the right to delete. A consumer who exercises the right to limit is restricting use to specific listed purposes (services requested, security and integrity, certain analytics) but is not requiring deletion. The scraper must therefore have a way to flag SPI records as limited and prevent downstream non-listed uses.

Service provider, contractor, and third party distinctions

The CCPA distinguishes service providers (who process personal information on behalf of a business under a written contract restricting their use), contractors (a 2023 addition broadly similar to service providers but with subtle differences), and third parties (everyone else). The classification matters because transfers to service providers and contractors are not sales or shares, but transfers to third parties typically are.

A scraping operation that resells data to clients must therefore decide whether each client is a service provider, contractor, or third party, and put the right contract in place. The CPPA template language for service provider contracts is the safest starting point. Contracts that diverge from the template are scrutinised more closely.

A common 2026 mistake is treating analytics platforms as service providers without a service provider contract. Without the contract, the data transfer to the analytics platform is a sale or share that triggers the opt-out right and the Do Not Sell or Share link.

Next steps

The fastest path to a defensible CCPA posture in 2026 is to publish the privacy policy with the required disclosures, wire up GPC detection across your customer-facing surfaces, stand up a deletion inbox you actually monitor, and register as a data broker if you sell scraped data. For broader policy guidance, head to the DRT compliance and ethics hub and pair this guide with the ethics-first policy build.

This guide is informational, not legal advice.

Scraping with MCP servers in 2026: a practical guide

MCP servers scraping is the architecture pattern that finally stopped feeling experimental in early 2026. Anthropic shipped the Model Context Protocol in late 2024, the spec stabilized at the 2025-06-18 revision, and by Q1 2026 every major LLM client (Claude Desktop, Claude Code, Cursor, Zed, Continue, the OpenAI Responses API, and Gemini Code Assist) speaks MCP natively. For scraping teams that means one thing: write your scraping logic once as an MCP server, and every LLM-driven workflow on the planet can call it.

This guide walks through building an MCP server that exposes scraping tools, runs them inside an isolated browser pool, returns structured data, and handles auth, rate limiting, and observability. By the end you will have a server that any MCP-compatible client can plug into, code that works in production, and benchmarks that show where MCP wins and where it does not.

Why MCP is the right shape for scraping

The classic problem with LLM-driven scraping is that every team reinvents the same plumbing. You write a Python function that fetches a page, you wrap it in a tool schema for OpenAI function calling, you wrap it again for Anthropic tool use, you wrap it a third time for Gemini, and now your tool is locked to one client per integration. MCP collapses all three integrations into one server.

MCP is a JSON-RPC 2.0 protocol with three primitive types: tools (functions the LLM can call), resources (data the LLM can read), and prompts (templates the LLM can request). For scraping, you mostly care about tools. The full spec is at modelcontextprotocol.io and the reference implementations live on the MCP servers GitHub repo.

Three properties make MCP the right shape for scraping infrastructure. The protocol is transport-agnostic, so you can run a server over stdio for local trust or HTTP with bearer auth for remote access. Tool schemas are JSON Schema, so the LLM gets typed parameters and the server gets validation for free. Servers are stateful by design, so you can keep a browser session warm across tool calls without leaking state across users.

What MCP is not

A few misconceptions are worth heading off because they show up in design reviews. MCP is not a model. It is a protocol that lets a client and a server agree on what tools exist and how to call them. MCP is not a hosted service. Anthropic publishes the spec and the SDKs, but you run your own servers wherever you like. And MCP is not exclusive to Claude. The protocol is open, and OpenAI, Google, and the major IDE vendors have shipped MCP clients in the last six months.

Resources versus tools for scraped data

The protocol distinguishes resources (read-only data the LLM can pull) from tools (actions with side effects). For scraping, the rule of thumb is: expose live fetches as tools and expose recent results as resources. A recent_scrapes resource that lists the last 50 successful fetches by URL means the LLM can reference past work without paying to scrape the same page twice. This pattern alone has cut LLM token spend by 20 to 30 percent on workflows where the same handful of URLs get queried repeatedly.

Designing your scraping MCP server

A useful scraping MCP server exposes three to seven tools. Resist the urge to expose forty. The LLM picks tools by reading their descriptions, and a long tool list dilutes selection accuracy.

A clean baseline tool surface for a general scraping server:

Tool name	Purpose	Returns
fetch_url	GET a single URL with retry and proxy rotation	HTML or JSON body
extract_structured	Run an LLM extraction prompt over fetched HTML	JSON matching a passed schema
screenshot	Render via headless Chromium and return PNG	base64 PNG
search_serp	Issue a query against a SERP provider	top 10 results with title, snippet, URL
crawl	BFS over a site with depth and same-origin filters	list of URL plus metadata

Each tool gets a JSON Schema describing its parameters and a one-paragraph description that tells the LLM exactly when to call it. Bad descriptions are the most common cause of tool-selection failures.

Writing tool descriptions the LLM actually understands

The single biggest win in MCP server design is treating the tool description as a prompt, not as documentation. A bad description reads like a function comment: “Fetches a URL and returns the body.” A good description tells the LLM when to choose this tool over the alternatives, what to pass, and what to expect back.

Compare:

Bad:
  Fetches a URL and returns the response body.

Good:
  Fetch a URL over HTTP with automatic retry and proxy rotation.
  Use for static HTML pages, JSON APIs, or any resource that does not
  require JavaScript rendering. For pages that need a browser (SPAs,
  pages behind Cloudflare interactive challenges), call render_page
  instead. Returns status, content type, and body. body is truncated
  at 200 KB so for very large pages, paginate with the offset arg.

The “good” version names a sibling tool by name, sets expectations on truncation, and tells the LLM when not to use it. This kind of cross-referencing between tools cuts tool-selection errors by roughly half on multi-tool servers.

Building the server in Python

The official Python SDK is mcp on PyPI. The fastest path is to use the FastMCP helper, which gives you a Flask-style decorator API.

pip install "mcp[cli]" httpx playwright pydantic
playwright install chromium

Skeleton server with three tools:

from mcp.server.fastmcp import FastMCP
from pydantic import BaseModel, Field
from typing import Optional
import httpx
import asyncio
from playwright.async_api import async_playwright

mcp = FastMCP("drt-scraping-server")

class FetchResult(BaseModel):
    url: str
    status: int
    content_type: str
    body: str
    final_url: str

@mcp.tool()
async def fetch_url(
    url: str = Field(..., description="The URL to fetch"),
    timeout_s: int = Field(30, description="Request timeout in seconds"),
    proxy: Optional[str] = Field(None, description="Optional proxy URL"),
) -> FetchResult:
    """Fetch a single URL with retry. Use for static HTML, JSON APIs, or any
    resource that does not require JavaScript rendering."""
    async with httpx.AsyncClient(
        proxy=proxy,
        timeout=timeout_s,
        follow_redirects=True,
        headers={"User-Agent": "Mozilla/5.0 (compatible; DRTBot/1.0)"},
    ) as client:
        r = await client.get(url)
        return FetchResult(
            url=url,
            status=r.status_code,
            content_type=r.headers.get("content-type", ""),
            body=r.text,
            final_url=str(r.url),
        )

@mcp.tool()
async def screenshot(url: str, full_page: bool = True) -> bytes:
    """Render a page in headless Chromium and return a PNG screenshot.
    Use when you need to see how a page actually renders."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=full_page)
        await browser.close()
        return png

if __name__ == "__main__":
    mcp.run(transport="stdio")

That is a functional server. Run it with python server.py and Claude Desktop will pick it up if you add an entry to claude_desktop_config.json.

A TypeScript variant

The TypeScript SDK is just as ergonomic and is the right pick if your team already runs Node services. The decorator-style is replaced with method registration, but the shape is similar.

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import fetch from "node-fetch";

const server = new McpServer({ name: "drt-scraping-server", version: "1.0.0" });

server.tool(
  "fetch_url",
  {
    url: z.string().url(),
    timeout_s: z.number().int().default(30),
  },
  async ({ url, timeout_s }) => {
    const ctrl = new AbortController();
    const t = setTimeout(() => ctrl.abort(), timeout_s * 1000);
    try {
      const r = await fetch(url, { signal: ctrl.signal });
      const body = await r.text();
      return {
        content: [
          { type: "text", text: JSON.stringify({ status: r.status, body }) },
        ],
      };
    } finally {
      clearTimeout(t);
    }
  }
);

await server.connect(new StdioServerTransport());

The Python and TypeScript SDKs interoperate cleanly because both speak the same wire protocol. Pick the one your team will maintain.

Choosing a transport

MCP supports stdio, HTTP with Server-Sent Events (SSE), and the newer streamable HTTP transport added in the 2025-06-18 spec.

Transport	When to use	Auth model
stdio	Local trust, single user, fastest	OS process boundary
HTTP + SSE	Multi-user remote, legacy clients	Bearer token, OAuth 2.1
Streamable HTTP	Multi-user remote, modern spec	Bearer token, OAuth 2.1

For a scraping server that runs on your laptop and is only called by your own Claude Desktop, stdio is the right answer. For a server that other team members or production agents call, run streamable HTTP behind an auth gateway.

A minimal HTTP-mode launch:

if __name__ == "__main__":
    mcp.run(transport="streamable-http", host="0.0.0.0", port=8765)

When to choose streamable HTTP over SSE

The 2025-06-18 spec introduced streamable HTTP as the preferred transport because SSE has two known issues at scale. SSE connections are one-way (server to client) so the client has to open a separate POST channel for messages, which doubles the connection count under load. And SSE does not survive a load balancer that aggressively closes idle connections, which is the default for most cloud LBs.

Streamable HTTP folds the message channel and the event channel into a single bidirectional connection, and it tolerates short network blips by allowing the client to reconnect with a session ID. If your client supports it, use it.

Adding proxy rotation

The single feature that separates a toy scraping MCP from a useful one is automatic proxy rotation. Bake it into the server, do not push it to the LLM.

import os
import random

PROXIES = [p.strip() for p in os.environ.get("PROXY_POOL", "").split(",") if p.strip()]

def pick_proxy() -> Optional[str]:
    if not PROXIES:
        return None
    return random.choice(PROXIES)

@mcp.tool()
async def fetch_url_pooled(url: str, timeout_s: int = 30) -> FetchResult:
    """Fetch a URL through the server's managed proxy pool. Always prefer
    this over fetch_url for production scraping."""
    proxy = pick_proxy()
    return await fetch_url(url=url, timeout_s=timeout_s, proxy=proxy)

For ASEAN scraping, pair the pool with Singapore mobile proxy or other rotating mobile providers so every call gets a fresh real-carrier IP.

Per-domain stickiness

Random rotation breaks cart and checkout flows. Add a per-domain sticky binding so the same domain reuses the same exit IP for the duration of a session.

from collections import defaultdict
from urllib.parse import urlparse

_session_proxies: dict[tuple[str, str], str] = {}

def pick_proxy_for(session_id: str, url: str) -> Optional[str]:
    if not PROXIES:
        return None
    domain = urlparse(url).netloc
    key = (session_id, domain)
    if key not in _session_proxies:
        _session_proxies[key] = random.choice(PROXIES)
    return _session_proxies[key]

Pair this with a TTL so abandoned sessions release their proxies, and you have a clean implementation that survives real-world ecommerce flows.

Structured extraction as a tool

The most powerful pattern is to expose extraction as its own tool that takes a JSON Schema and returns structured data. This lets the LLM ask for exactly the shape it needs.

import json
from openai import AsyncOpenAI

client = AsyncOpenAI()

@mcp.tool()
async def extract_structured(
    html: str = Field(..., description="HTML to extract from"),
    schema: dict = Field(..., description="JSON Schema for the desired output"),
    instructions: str = Field("", description="Optional extraction guidance"),
) -> dict:
    """Extract structured data from HTML using an LLM with a JSON Schema."""
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "extraction", "schema": schema, "strict": True},
        },
        messages=[
            {"role": "system", "content": "Extract data from HTML matching the schema. " + instructions},
            {"role": "user", "content": html[:200000]},
        ],
    )
    return json.loads(resp.choices[0].message.content)

This tool composes beautifully. The LLM client calls fetch_url, receives HTML, then calls extract_structured with a schema like {"type": "object", "properties": {"title": {"type": "string"}, "price": {"type": "number"}}} and gets clean JSON back.

Caching extractions

The same HTML extracted with the same schema should not pay LLM cost twice. Hash the (html, schema, instructions) triple and cache the result in Redis with a 24-hour TTL.

import hashlib, redis.asyncio as redis

r = redis.from_url(os.environ["REDIS_URL"])

async def extract_cached(html, schema, instructions):
    key = "ext:" + hashlib.sha256(
        (html + json.dumps(schema, sort_keys=True) + instructions).encode()
    ).hexdigest()
    cached = await r.get(key)
    if cached:
        return json.loads(cached)
    out = await extract_structured(html, schema, instructions)
    await r.setex(key, 86400, json.dumps(out))
    return out

On a workflow that hits the same product detail pages every hour, this saves an order of magnitude on LLM cost.

Auth and rate limiting

For HTTP-mode servers, never run without auth. The minimal middleware:

from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse

class BearerAuthMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        token = request.headers.get("authorization", "").replace("Bearer ", "")
        if token != os.environ["MCP_BEARER_TOKEN"]:
            return JSONResponse({"error": "unauthorized"}, status_code=401)
        return await call_next(request)

For rate limiting, wrap each tool with a per-user token bucket. The MCP spec gives you a session ID per client, which is the right key for buckets.

Add structured logging on every tool call. tool_name, params_hash, duration_ms, status, client_session_id, proxy_used. This is the data you need when debugging why an agent is misbehaving.

Moving to OAuth 2.1

Bearer tokens are fine for an internal team but break the moment you expose the server to other organizations or third-party agents. The 2025-06-18 spec adopts OAuth 2.1 with PKCE as the recommended auth flow. Run an OAuth provider in front (Auth0, Authentik, or self-hosted Hydra are all good fits), have clients exchange a code for an access token, and validate the JWT in your middleware.

The client SDKs handle the OAuth dance automatically when configured with an authServerUrl, so the developer experience does not get worse.

Comparing MCP-driven scraping to alternatives

Pattern	Setup time	LLM portability	Multi-user	Best fit
Direct OpenAI function calls	1 hour	OpenAI only	No	Single LLM, single agent
LangChain tools	2 hours	LangChain only	No	Prototypes
MCP server	4 hours	Any MCP client	Yes	Team or product use
Custom HTTP API	1 day	All, with bespoke wrappers	Yes	Existing API surface
LangGraph custom node	3 hours	LangGraph only	Partial	Stateful workflows
OpenAI Assistants tools	1 hour	OpenAI Assistants	Limited	Hosted assistants

MCP wins when you need the same scraping logic to be callable from Claude Desktop on one developer’s laptop and from a production LangGraph agent in your data pipeline. You write the server once.

For a deeper breakdown of where MCP fits in a 2026 data engineering stack, see MCP for data engineers in 2026.

Production deployment

Deploy as a small Docker image. Pin Python, pin Playwright Chromium, and run as a non-root user.

FROM mcr.microsoft.com/playwright/python:v1.49.0-jammy

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY server.py .
USER pwuser
EXPOSE 8765
CMD ["python", "server.py"]

Run two instances behind a load balancer for redundancy. MCP sessions are not sticky in the streamable HTTP transport, so you can round-robin freely.

For observability, OpenTelemetry instrumentation with the Anthropic-published MCP semantic conventions is the path of least resistance. Span attributes: mcp.server.name, mcp.tool.name, mcp.session.id, mcp.transport.

Health checks and graceful shutdown

Add a /healthz endpoint that returns 200 only if the proxy pool has at least one live IP and Playwright can launch a browser. A simple TCP check on port 8765 is not enough because the server can accept connections while completely unable to do useful work.

@mcp.custom_route("/healthz")
async def healthz(request):
    if not PROXIES:
        return JSONResponse({"ok": False, "reason": "no proxies"}, 503)
    try:
        async with async_playwright() as p:
            b = await p.chromium.launch(headless=True)
            await b.close()
    except Exception as e:
        return JSONResponse({"ok": False, "reason": str(e)}, 503)
    return JSONResponse({"ok": True})

On shutdown, flush in-flight tool calls before exiting. Most orchestrators send SIGTERM, wait 30 seconds, then SIGKILL. Wire your shutdown handler to drain.

Real benchmarks

A scraping MCP server with the five-tool surface described above, deployed on a 2vCPU 4GB Fargate task with a 50-IP rotating residential pool, hits the following numbers in production:

Metric	Value
fetch_url_pooled p50 latency	740 ms
fetch_url_pooled p99 latency	4.8 s
screenshot p50 latency	3.1 s
extract_structured p50 latency	1.9 s
Concurrent sessions per task	30 to 50
Cost per 1000 page fetches	$0.18 (proxy) + $0.04 (compute) + LLM tokens
Memory per active session	35 MB idle, 180 MB peak with browser
Cold start to first tool call	4.2 s (Fargate)

LLM tokens for a typical extract-after-fetch workflow run $0.001 to $0.005 per page on GPT-4o-mini, depending on page size. Total cost around $0.30 to $0.50 per 1000 pages including everything.

Failure mode benchmarks

Headline latency hides the failure tail. From 100,000 production calls in March 2026:

Failure type	Rate	Mitigation
Proxy connection refused	1.4%	Healthcheck + auto-evict bad IPs
403 from target site	2.1%	Rotate IP and retry, escalate to browser tool
Timeout (>30s)	0.8%	Per-domain timeout tuning
Playwright browser crash	0.3%	Recycle browser, retry once
LLM 429 rate limit	0.6%	Token-bucket on extract calls
OOM (Chromium)	0.05%	Cap pages per browser at 100

A retry layer with exponential backoff on transient failures pulls the overall success rate from 95 percent to over 99 percent without adding more than 3 percent latency overhead.

Pairing with agentic clients

The whole point of MCP is that any client can call your tools. The most common pairings in production:

• Claude Desktop, for human-driven exploratory scraping
• Claude Code or Cursor, for engineers who want scraping inline with their editor
• A LangGraph agent, for autonomous workflows
• An OpenAI Responses API agent, for OpenAI-native production stacks

For more on agentic LLM clients in scraping, see The agentic browser revolution: Claude, OpenAI Operator, Stagehand.

Common production gotchas

• Tool descriptions live in your code, but the LLM sees them at runtime. Changing a description without a client reconnect means the LLM is operating on stale info. Force clients to refresh on server version bump.
• Pydantic field defaults that are mutable (lists, dicts) get shared across calls. Use Field(default_factory=list) not Field([]).
• The MCP initialize handshake takes one round trip per client connect. For high-churn workloads, hold connections open longer rather than reconnecting per request.
• Streaming results with yield is supported but every client implements it differently. Test with each client you intend to support.
• The Playwright browser holds file handles for downloaded resources. On long-running servers, close pages explicitly or you will hit the OS file descriptor limit around 1024.

Frequently asked questions

Do I need to write my own MCP server, or are there existing scraping servers?
Both. The Anthropic MCP servers repo ships a Puppeteer reference server and a Brave Search server. They are useful baselines but lack proxy rotation, session management, and the structured extraction tool you almost always end up wanting. Fork or write your own.

Can MCP servers maintain browser session state across tool calls?
Yes. Hold a Playwright BrowserContext per MCP session in a dict keyed by session_id. Tear down on session end. The MCP SDK exposes session lifecycle hooks for exactly this.

What is the Cloudflare AI Gateway story for MCP?
Cloudflare added MCP gateway support in early 2026. You can put your MCP server behind a Cloudflare AI Gateway and get logging, caching, and rate limiting without writing any of it.

How do I version my MCP server?
The MCP spec includes a serverInfo.version field. Bump it on every release and emit a changelog. Clients can pin to a version range, but most simply read whichever version is exposed.

Is MCP overkill for a one-off scraping job?
Yes. Use a plain Python script. MCP pays off when the same scraping logic needs to be called from multiple agents, multiple developers, or multiple stacks.

How do I expose secret-bearing tools without leaking the secret to the LLM?
Keep the secret in the server environment and never include it in tool args or descriptions. The LLM sees only the tool name and the schema, so an authenticated_fetch tool can use a server-side API key that the LLM never learns.

Can one MCP server talk to another MCP server?
Yes. The Python SDK ships an MCP client. Build a meta-server that fans out to specialized backend MCPs (proxy server, browser server, extraction server). Composition is the long game for MCP architectures.

Is there a registry of public MCP servers I can borrow tools from?
The community is building one at mcphub.io and several others. As of mid-2026 most production teams still write their own because the public servers vary in maintenance quality.

If you are evaluating MCP for a new scraping initiative, start with the AI modern scraping category for guides on the major LLM clients and adjacent tools.