Robots.txt and modern scraping ethics in 2026

Robots.txt scraping ethics has become one of the most contested topics in 2026, because the file that started as a polite courtesy in 1994 is now treated as a quasi-contract in some jurisdictions and as marketing copy in others. The AI training surge of 2023 to 2025 forced site operators to add new directives and forced scrapers to take a position on whether those directives bind them. This guide walks through what robots.txt actually is, the new AI-specific directives that emerged in 2024 and 2025, how courts treated robots.txt across jurisdictions, and a defensible team policy you can adopt this quarter.

The audience is technical leads and product owners who need a clear position on how their scraping pipeline handles robots.txt, both for defensibility and for downstream customer expectations.

What robots.txt actually is and is not

Robots.txt is a plain-text file at the root of a domain that signals to automated agents which paths the site operator would prefer they not access. The Robots Exclusion Protocol was first proposed in 1994 by Martijn Koster and was formalised as RFC 9309 in 2022 by Google, the IETF, and a coalition of crawler operators. The RFC made several things explicit that had been folklore: the file is advisory, the syntax is well-defined, and compliance is a choice the crawler operator makes.

What robots.txt is: a published preference. A courtesy protocol. A widely-respected convention that lets site operators communicate scope to bots. It is also, in some courts and contracts, evidence of the site operator’s intent regarding access.

What robots.txt is not: a technical access control. A robots.txt directive cannot stop a non-compliant crawler. The file does not block traffic, does not authenticate users, does not change HTTP behaviour. A scraper that ignores robots.txt is doing something visible and verifiable, but not technically prevented.

The distinction matters because legal arguments about scraping increasingly turn on what the site operator did to communicate scope. Robots.txt is the cheapest, broadest, most-respected way to do that.

For the broader compliance picture, see the GDPR scraping compliance guide and the ethics-first scraping policy.

The 2024-2025 AI directive surge

Until 2023, robots.txt was overwhelmingly used to manage indexing crawlers (Googlebot, Bingbot) and a small number of well-known commercial scrapers. The AI training boom changed that. By mid-2024, a long list of new user-agents had to be considered, and site operators had to decide which to allow.

The major AI-specific user agents in 2026:

User agent	Operator	Purpose
GPTBot	OpenAI	Training data collection
ChatGPT-User	OpenAI	User-initiated browsing in ChatGPT
Google-Extended	Google	Bard/Gemini training opt-out
ClaudeBot	Anthropic	Training data collection
anthropic-ai	Anthropic	Older identifier (legacy)
PerplexityBot	Perplexity AI	Search index for Perplexity
Perplexity-User	Perplexity AI	User-initiated browsing
CCBot	Common Crawl	Open archive used by many models
Bytespider	ByteDance	Used for ByteDance LLMs
FacebookBot	Meta	Llama training and indexing
Applebot-Extended	Apple	Apple Intelligence training opt-out
Diffbot	Diffbot	Knowledge graph extraction
Amazonbot	Amazon	Alexa and product crawling

By Q2 2025, a study of the top 10,000 web domains found that more than 35 percent had added at least one AI-specific Disallow directive, up from less than 5 percent in early 2023. The New York Times, Reuters, the BBC, Stack Overflow, Quora, and most large publishers explicitly disallow GPTBot, ClaudeBot, and PerplexityBot. The signal is unambiguous.

A scraper operating in 2026 that wants to argue good-faith respect for site operator preferences must do more than parse a single robots.txt for Googlebot. The file is a multi-agent instruction set, and ignoring AI-specific directives is increasingly seen as bad-faith conduct.

Court treatment of robots.txt in different jurisdictions

US courts have been consistent that robots.txt is not by itself a legal access control. The HiQ Labs v LinkedIn line of cases (covered separately in the HiQ Labs ruling explainer) confirmed that scraping public data does not automatically violate the CFAA, regardless of robots.txt. However, several lower courts in 2024 and 2025 treated explicit robots.txt directives as relevant evidence of the site operator’s intent in trespass-to-chattels and breach-of-contract claims.

EU courts have leaned more towards treating robots.txt as part of the implied contract of access, especially for AI training use cases. A 2025 Hamburg ruling held that scraping past an explicit AI-bot disallow was relevant in the legitimate interest balancing test under GDPR Article 6(1)(f), tilting the balance away from the scraper.

UK courts have largely followed the US line, emphasising public availability. Singapore courts have not yet ruled directly, but PDPC guidance in 2025 cited robots.txt compliance as evidence of fair processing under the Personal Data Protection Act.

The pattern is clear: robots.txt does not by itself create a legal duty in most jurisdictions, but ignoring it weakens almost every legal defence you might rely on later. Compliance is cheap. Non-compliance is expensive when something goes wrong.

A scraper-side compliance checklist

Control	What it requires	Why it matters
Fetch and parse robots.txt before each domain	RFC 9309 compliant parser	Legal evidence of good faith
Honour your declared user agent	Identify accurately	Trust signal for site operators
Respect Disallow paths	Skip disallowed URLs	Ethical baseline
Honour Crawl-delay	Throttle per directive	Reduces server load
Cache robots.txt for 24 hours max	Re-fetch frequently	Compliance with site changes
Differentiate by purpose	Use different UA for indexing vs training	Allows site to set per-purpose rules
Log compliance decisions	Per-URL allowed/denied audit trail	Defensible posture
Honour Sitemap directives positively	Use sitemap as canonical scope	Reduces wasted requests
Skip noindex meta tags	Combine robots.txt with HTML-level meta	Full coverage
Provide opt-out contact	Public email or web form	Site operators can reach you

The first six rows are the minimum. The last four are the difference between “we comply” and “we are a model citizen.”

Decision tree: should I scrape this URL?

Q1: Does the domain publish robots.txt?
    ├── No  -> Scrape conservatively; default to crawl-delay 5s.
    └── Yes -> Q2
Q2: Does robots.txt list your user agent?
    ├── Yes -> Honour the directives for your UA.
    └── No  -> Q3
Q3: Does robots.txt have a wildcard (*) section?
    ├── Yes -> Honour the wildcard directives.
    └── No  -> Default to allow with conservative crawl-delay.
Q4: Is the URL within a Disallow path?
    ├── Yes -> Skip; log as denied; do not retry.
    └── No  -> Q5
Q5: Is the page tagged with noindex/nofollow at HTML level?
    ├── Yes -> Defer to HTML directive.
    └── No  -> Proceed with respect to crawl-delay.

Each decision is logged. The audit trail is what gives you a defensible posture if a site operator complains.

Practical Python implementation

A minimal RFC 9309 compliant fetcher in Python looks like this. The standard library urllib.robotparser has gaps; protego from Scrapy is more compliant.

from protego import Protego
import requests
from urllib.parse import urlparse

class RobotsCache:
    def __init__(self, user_agent="DRTScraper/1.0"):
        self.user_agent = user_agent
        self.cache = {}

    def can_fetch(self, url: str) -> bool:
        parsed = urlparse(url)
        domain = f"{parsed.scheme}://{parsed.netloc}"
        if domain not in self.cache:
            self._load(domain)
        rp = self.cache[domain]
        if rp is None:
            return True
        return rp.can_fetch(url, self.user_agent)

    def crawl_delay(self, url: str) -> float:
        parsed = urlparse(url)
        domain = f"{parsed.scheme}://{parsed.netloc}"
        if domain not in self.cache:
            self._load(domain)
        rp = self.cache[domain]
        if rp is None:
            return 1.0
        delay = rp.crawl_delay(self.user_agent)
        return float(delay) if delay else 1.0

    def _load(self, domain: str):
        try:
            resp = requests.get(
                f"{domain}/robots.txt",
                headers={"User-Agent": self.user_agent},
                timeout=10,
            )
            if resp.status_code == 200:
                self.cache[domain] = Protego.parse(resp.text)
            else:
                self.cache[domain] = None
        except Exception:
            self.cache[domain] = None

Wire this in front of every request. Log every denial. The cost is one HTTP fetch per domain per session. The benefit is a complete audit trail.

What about Crawl-delay, Request-rate, and Visit-time?

Crawl-delay is supported by most major crawlers but is not part of RFC 9309. It is a de facto standard. Treat it as binding because most site operators expect compliance.

Request-rate and Visit-time are older directives that never reached wide adoption. You can ignore them in 2026 with little risk, but if they are present, the conservative move is to honour them. They cost nothing.

The Sitemap directive is positive: it tells you where the site operator wants you to start. Use it. A scraper that follows the sitemap is far less likely to hit edge-case URLs that the site operator did not anticipate exposing.

The AI training opt-out as a separate signal

Beyond robots.txt, several site operators in 2025 began publishing dedicated AI training opt-out signals. The two main mechanisms in 2026:

1. The TDM Reservation Protocol, an emerging W3C draft that uses HTTP headers and <meta> tags to signal text and data mining opt-out separately from crawler directives.
2. The C2PA content credentials with embedded usage policies, which carry rights metadata for both human and machine consumers.

Both are still maturing. A scraper that wants to take the most defensible 2026 posture honours both signals in addition to robots.txt. It is more work but it places you at the front of the compliance curve.

A defensible team policy

A working policy has six parts: stated principles, technical implementation, audit logging, vendor management, opt-out handling, and review cadence. The shape of each part:

Stated principles: a one-page document, signed by the engineering lead and product lead, declaring that the team respects robots.txt by default, honours AI-specific directives, and treats compliance as a non-negotiable.

Technical implementation: the protego-based fetcher above, deployed in the request middleware of every scraping pipeline. No exceptions.

Audit logging: every denied URL is logged with timestamp, user agent, and the relevant directive. Logs retained for 12 months minimum.

Vendor management: proxy providers, scraping APIs, and data resellers contractually attest to robots.txt compliance.

Opt-out handling: a public contact email (privacy@yourcompany.com) for site operators to request removal, escalation, or clarification.

Review cadence: quarterly review of the principles, the AI user-agent list, and the audit trail.

For a longer treatment of how to write the principles document and operationalise the audit, see the ethics-first scraping policy guide.

External references

The RFC 9309 specification is at datatracker.ietf.org/doc/rfc9309. Google’s robots.txt parser (open source) is at github.com/google/robotstxt. The TDM Reservation Protocol draft is at w3c.github.io/tdmrep. The C2PA content credentials specification is at c2pa.org.

Comparison: respecting robots.txt vs ignoring it

Dimension	Respect	Ignore
Legal exposure (US)	Low	Moderate (evidence in trespass claims)
Legal exposure (EU)	Low	High (impacts GDPR balancing)
Customer trust	High	Low (especially enterprise B2B)
Site operator goodwill	High	Negative
Server load impact	Lower	Higher
Block rate from target	Low	High over time
Cost to implement	Negligible	Negligible
Long-term sustainability	High	Low

The asymmetry is striking. Compliance costs almost nothing. Non-compliance costs a lot when it costs anything.

FAQ

Is robots.txt legally binding?
Not directly in most jurisdictions. It is a published preference. But ignoring it is increasingly treated as evidence of bad faith in court and in regulator investigations.

Should I honour Crawl-delay even if it slows my pipeline?
Yes. The cost is negligible compared to the legal and goodwill risk of ignoring it.

Can I scrape if the site has no robots.txt?
Yes, but default to a conservative crawl-delay (5 seconds) and respect HTML-level noindex/nofollow tags.

What about pages behind login?
Robots.txt only governs publicly reachable URLs. Authenticated pages are governed by the terms of service of the platform.

Does GPTBot Disallow apply to me if I am not OpenAI?
The directive is explicitly addressed to GPTBot. It does not apply to your user agent. But the spirit of the directive is anti-AI-training, and a scraper that ingests data for AI training should honour the intent.

Extended legal and operational analysis

The robots exclusion protocol became RFC 9309 in 2022, formally codifying behaviour that had been industry custom since 1994. RFC 9309 does not by itself create a legal obligation. It documents how compliant crawlers behave. The legal force of robots.txt comes from adjacent doctrines, namely contract (terms of service that incorporate robots.txt by reference), trespass to chattels in some United States jurisdictions, and the Computer Fraud and Abuse Act when access is unauthorised.

The 2024-2026 period saw three shifts. First, AI-specific user agents proliferated, including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Anthropic-AI. Second, publishers began publishing site policy on AI training distinct from search indexing, often by adding AI-specific Disallow rules. Third, courts began treating robots.txt compliance as evidence of good faith even where it was not strictly required.

The hiQ v LinkedIn line of cases established that scraping public data does not by itself violate the CFAA, but did not absolve scrapers of contract or tort exposure. Subsequent cases (Meta v Bright Data 2024, X Corp v Bright Data 2024) reinforced the contract pathway. Both ended in dismissal for the scraper, but only after years of litigation expense. Robots.txt compliance was cited in both as one factor courts weighed.

Implementation patterns for 2026 robots compliance

A robust scraper in 2026 should implement six behaviours.

1. Fetch robots.txt before the first request and cache for at most twenty-four hours.
2. Honour the most-specific User-agent block, falling back to the wildcard.
3. Respect Crawl-delay where supported, with a minimum default of one second per request when not specified.
4. Honour Disallow paths exactly, including trailing slash semantics.
5. Read site-wide AI policy headers including the X-Robots-Tag and any noai or noindex directives.
6. Log every robots decision per request so audits can prove the behaviour.

Code pattern for a compliant fetcher

import urllib.robotparser
from urllib.parse import urljoin, urlparse

class CompliantFetcher:
    def __init__(self, user_agent):
        self.ua = user_agent
        self.parsers = {}

    def can_fetch(self, url):
        host = urlparse(url).netloc
        if host not in self.parsers:
            rp = urllib.robotparser.RobotFileParser()
            rp.set_url(f"https://{host}/robots.txt")
            try:
                rp.read()
            except Exception:
                return False
            self.parsers[host] = rp
        return self.parsers[host].can_fetch(self.ua, url)

    def crawl_delay(self, url):
        host = urlparse(url).netloc
        if host in self.parsers:
            return self.parsers[host].crawl_delay(self.ua) or 1.0
        return 1.0

Comparison: AI crawler policies on top sites in 2026

Site	GPTBot	ClaudeBot	Google-Extended	CCBot
nytimes.com	Disallow	Disallow	Disallow	Disallow
reddit.com	Disallow	Disallow	Allow (paid)	Disallow
stackoverflow.com	Allow	Allow	Allow	Allow
github.com	Allow	Allow	Allow	Allow
medium.com	Disallow	Disallow	Allow	Disallow
wikipedia.org	Allow	Allow	Allow	Allow

The pattern is that publishers with content-licensing revenue tend to disallow AI crawlers, while platforms with developer or community content tend to allow them.

Additional FAQ

Is ignoring robots.txt illegal?
Not by itself in most jurisdictions, but it weakens defences in contract, tort, and statutory disputes. It is also evidence of bad faith in regulator inquiries.

What if there is no robots.txt?
Treat absence as no specific policy. Apply default ethical behaviour including conservative rate limits and identification of the user agent.

Should AI training crawlers honour robots.txt differently from search crawlers?
Yes. The AI-specific user agents exist precisely so publishers can express different policies. A compliant AI crawler reads the AI-specific block first, then the wildcard, then defaults.

Does honouring robots.txt remove all legal risk?
No. Honouring robots.txt is one factor. Terms of service, copyright, privacy law, and trade secret doctrine still apply.

Real cases where robots.txt mattered in court

Two recent decisions illustrate how courts treat robots.txt in 2024-2026.

In Thomson Reuters v. Ross Intelligence (D. Del., February 2025 summary judgment), the court found that Ross’s training of a competing legal research AI on Westlaw headnotes was not protected fair use. While the case turned primarily on copyright and the commercial-substitution analysis, the trial record included extensive evidence about how Ross obtained the headnotes through a third-party intermediary that ignored Westlaw’s terms and crawl restrictions. Judge Bibas referenced the access pattern in the bad-faith analysis. The decision is now the most-cited US precedent for the proposition that disregarding access controls weakens an AI training defence.

In The New York Times v. Microsoft and OpenAI (S.D.N.Y., 2024 ongoing), the Times’ complaint specifically pleads that OpenAI’s GPTBot ignored or post-dated the Times’ robots.txt Disallow directive for the AI-specific user agent. The pleading frames robots.txt compliance as a baseline good-faith expectation in the publishing industry. While the case has not yet reached merits judgment, the pleading strategy reflects how plaintiffs now use robots.txt non-compliance as a narrative anchor for bad-faith allegations.

Both cases reinforce the operational lesson: robots.txt is not legally binding on its own, but ignoring it is now treated as a meaningful evidentiary fact in almost every commercial scraping dispute. The cost of compliance is trivial; the cost of non-compliance compounds across litigation, regulator inquiries, and platform agreements. A scraper that honours robots.txt by default and logs every decision has a defence narrative ready before any dispute arises.

The history and standardisation of robots.txt

Robots.txt was proposed by Martijn Koster in 1994 as a voluntary protocol for crawlers to declare and discover crawl preferences. It remained an informal de-facto standard for nearly three decades. RFC 9309, published in September 2022, formally specified the protocol after Google led a working group to align implementations.

RFC 9309 nailed down several previously ambiguous behaviours. The matching rules for User-agent strings, the handling of multiple matching groups, the precedence of Allow and Disallow rules, the canonicalisation of paths, and the maximum file size (500 KiB by default) are now specified. The RFC does not specify rate limiting, the meaning of Crawl-delay, or AI-specific user agents. Those remain extensions on top of the base protocol.

The standardisation matters for scrapers because compliant behaviour is now testable. A scraper can be checked against RFC 9309 test vectors, and gaps can be identified and fixed. Pre-RFC implementations often differed in edge cases. Post-RFC the expectation is that compliant crawlers behave identically.

Beyond robots.txt: meta robots, x-robots-tag, and llms.txt

Robots.txt is the front door but not the only signal. Meta robots tags in HTML, the X-Robots-Tag HTTP response header, and the proposed llms.txt convention all carry crawler instructions.

Meta robots tags appear in HTML head and apply per-page. They support directives including index, noindex, follow, nofollow, noarchive, nosnippet, and AI-specific directives like noai and noimageai (proposed 2024). A scraper should parse these per page.

X-Robots-Tag is the response header equivalent, useful for non-HTML resources (PDFs, images, JSON APIs). The directive vocabulary mirrors meta robots. Scrapers fetching non-HTML content should check the header.

The llms.txt convention proposed in 2024 by Jeremy Howard provides a structured site map specifically for LLM consumers. It complements rather than replaces robots.txt. Some publishers ship both, with robots.txt declaring access policy and llms.txt declaring content structure for AI clients.

The ethical dimension beyond compliance

Compliance with robots.txt is the floor, not the ceiling. Ethical scraping in 2026 considers four additional factors that robots.txt does not capture.

First, server load. A scraper that respects robots.txt but hammers the server with concurrent requests still imposes externalities. Conservative concurrency and adaptive backoff are part of ethical operation.

Second, content type. Some content (personal social media posts, sensitive forum threads) deserves additional restraint regardless of what robots.txt says. The scraper should apply context-sensitive judgement.

Third, downstream use. A scrape that respects robots.txt but feeds the data into a system that the publisher would object to (for example training a competing AI on a paywalled publisher’s free pages) is technically compliant but ethically thin.

Fourth, transparency. A scraper identified by a unique User-Agent string, with operator contact information in the User-Agent or in a public crawler page, makes itself accountable. Anonymous crawlers are correlated with abuse and are increasingly blocked at the platform level.

Next steps

The fastest improvement is to drop a Protego-based middleware into your scraper this week, log every denial for 30 days, and review the log for surprises. If you find your scraper has been hitting Disallow paths, fix it before a site operator notices. For the broader policy and team rollout, head to the DRT compliance hub and start with the ethics-first policy guide.

This guide is informational, not legal advice.

Claude Code vs Cursor for web scraping projects

The Claude Code vs Cursor scraping decision matters because both tools collapse the loop between writing a scraper and running it, but they collapse it differently. Cursor lives in your editor and is optimized for in-file edits with AI assist. Claude Code runs as a CLI agent and is optimized for autonomous execution of multi-step tasks. For scraping work, that distinction shows up immediately. Cursor wants you to drive. Claude Code wants to drive itself.

This comparison is built from running both tools on the same scraping projects in early 2026. Identical targets, identical proxies, identical models where possible. We covered building a Lazada price monitor, a job board aggregator, and a lightweight news clipping pipeline. Below is the honest picture of where each tool wins, where they tie, and which one we would pick for a new scraping project today.

What each tool actually is

Claude Code is Anthropic’s command-line agent that runs in any terminal, has direct file system access, executes arbitrary bash, and operates in an autonomous loop until your task is done or it asks for input. The default model is Claude Sonnet 4.5 with optional Opus for harder tasks.

Cursor is a VSCode fork with deep AI integration. The agent mode (released 2024, refined heavily through 2025) is now closer to Claude Code in capability, but its center of gravity is still the editor. You drive selections, you accept diffs, you steer.

Both ship MCP support, both can use external scraping tools, both can read your codebase. The difference is the human-in-the-loop ratio.

Architectural philosophy in one sentence each

Claude Code believes the best dev loop is “describe the outcome, walk away, come back to a green build.” Cursor believes the best dev loop is “see every diff, approve the smart ones, reject the bad ones, ship.” Neither is wrong. The right pick depends on which loop fits your team’s tolerance for autonomy.

Setting up for a scraping project

For Claude Code, the install is one line:

npm install -g @anthropic-ai/claude-code

Then in your project directory:

cd ~/projects/lazada-monitor
claude

You are dropped into an interactive session that already knows your file tree. Add a CLAUDE.md at the project root with conventions and tool preferences, and the agent reads it every session.

For Cursor, install the editor and open the project. Configure model preferences in settings. Add a .cursorrules file with project guidance.

Neither tool ships scraping-specific helpers. You bring your own Playwright, your own proxy pool, your own database client.

Sample CLAUDE.md for a scraping project

A useful starter file lives at the project root and shapes every session. Here is a battle-tested template:

# Project: Lazada Price Watcher

## Stack
- Python 3.12, Playwright, SQLite, httpx, pydantic
- Proxies via Singapore mobile proxy (creds in .env)
- Telegram alerts via python-telegram-bot

## Conventions
- All scraping code under scrapers/
- Pytest tests under tests/, run with `make test`
- Lock requirements with pip-compile
- Never commit .env

## Hard rules
- Never store passwords in plain text in DB
- Never bypass robots.txt without an explicit go-ahead
- Always validate Pydantic models before DB writes

Cursor’s .cursorrules covers the same ground but is read more passively. Claude Code re-reads CLAUDE.md every session, so updates take effect immediately.

A real scraping task: building a Lazada watcher

The test task: build a Python script that monitors a list of Lazada Singapore product URLs, extracts price and stock daily, writes to SQLite, and sends a Telegram alert when price drops more than 10 percent.

With Claude Code, the prompt was:

Build a Lazada price watcher. Read URLs from data/products.txt, scrape title,
price, and stock for each, store in data/prices.db with a timestamp, and send
a Telegram message via the bot token in .env when any price drops 10% or more
since the last run. Use Playwright with stealth defaults. Include retries and
proxy support. Add a cron-friendly entrypoint.

Claude Code wrote ten files in eleven minutes, including a Playwright scraper, a SQLite migration, a Telegram client, a Makefile, a requirements.txt, a README.md, and a sample crontab line. It ran the scraper against three test URLs to verify. Total tokens billed: about 380k input, 24k output, $1.20 on the Sonnet 4.5 model.

With Cursor, the same prompt produced a single-file scaffold in about three minutes. The scaffold was good but missing the Telegram client, the migration script, and the proxy support. Each follow-up needed a new agent prompt or manual edits. Total time to functional parity: 28 minutes including six follow-up turns.

Claude Code wins on autonomous shipping of a complete scaffold. Cursor wins on speed of any single edit and on quality of in-file refactor suggestions.

Second task: a job board aggregator

We ran a second test where the requirement was looser: aggregate jobs from Indeed Singapore, JobStreet, and LinkedIn into a single Postgres table, with deduplication by company plus title plus posted date.

Claude Code asked one clarifying question (whether to honor LinkedIn’s Terms of Service or just scrape with login) and then shipped the rest. Cursor produced a working Indeed scraper quickly but never volunteered to think about deduplication or schema, treating each ask as discrete.

The pattern repeated. Claude Code reasons across the whole project, Cursor reasons across the visible buffer.

Tool use and MCP integration

Both tools speak MCP. Configuration is similar.

Claude Code reads ~/.claude/mcp.json:

{
  "mcpServers": {
    "scraping": {
      "command": "python",
      "args": ["/Users/me/scraping-mcp/server.py"]
    },
    "playwright": {
      "command": "npx",
      "args": ["-y", "@executeautomation/playwright-mcp-server"]
    }
  }
}

Cursor reads ~/.cursor/mcp.json with the same shape.

In practice, Claude Code uses MCP tools more aggressively. The agent will reach for a screenshot tool if you mention you cannot tell what is rendering. Cursor, in agent mode, prefers to write code that calls the tool directly. Both work, both are correct, the styles differ.

For wiring up an MCP scraping server, see our scraping with MCP servers guide.

Tool selection accuracy

In a 50-task audit where both tools had access to the same five MCP tools (fetch, screenshot, extract, search, crawl), Claude Code picked the correct first tool 88 percent of the time. Cursor picked the correct first tool 71 percent of the time. The gap mostly came from Cursor’s preference to write fresh Python rather than reach for a tool, which is fine when the tool is overkill but wastes time on bread-and-butter scraping.

Debugging a broken scraper

This is where the styles diverge most.

Claude Code, when a scraper breaks, will run the script, read the traceback, edit the file, run again, and keep iterating until the test passes or it hits its task budget. You can step away.

Cursor agent mode will propose a fix, wait for you to accept, run the script if you ask it to, and bring you the next traceback. The loop is faster per iteration but slower per debugging session because every step needs your attention.

For shallow bugs (typo, missing import, wrong selector), Cursor’s faster loop wins. For deep bugs (race condition between Playwright launches, weird Cloudflare interaction, database lock), Claude Code’s autonomous iteration wins because it will try ten things in the time you would still be reading the third Cursor diff.

A real debugging vignette

A flaky Playwright test that failed once every five runs took Claude Code 22 minutes and three exploratory iterations to diagnose: a wait_until="networkidle" that was triggering before a delayed XHR, fixed with an explicit selector wait. Cursor took roughly the same time but the engineer had to babysit each step. The wall clock was identical, the engineer hours were not.

Side-by-side comparison

Dimension	Claude Code	Cursor
Default model	Claude Sonnet 4.5	Claude Sonnet 4.5 or GPT-5
Native interface	Terminal	VSCode fork
Best at	Multi-step autonomous tasks	In-editor refactors, line-by-line edits
Worst at	Real-time UI work, design feedback	Long unattended jobs
MCP support	Yes, native	Yes, native
Codebase awareness	Reads on demand, follows symlinks	Always-on indexed search
Cost per scraping pipeline scaffold	$0.50 to $2.00 per session	Subscription + variable model cost
Steepest learning curve	Bash and Unix fluency expected	None, IDE-native
Wins on Lazada monitor task	Faster end-to-end	Faster per-edit
Wins on debugging deep issues	Yes	No
Wins on quick selector fix	No	Yes
Plays well with sub-agents	Yes (Task tool)	Limited
Inline screenshot viewing	Via MCP only	Native
Multi-window/multi-cursor	No	Yes
Background mode (run in CI)	Yes	Limited
Autonomous test run loop	Yes	No (asks)

Cost analysis

Claude Code charges per-token through your Anthropic API key, or you can use a Claude Pro/Max subscription for fixed monthly cost with quota.

Cursor charges a flat $20/month for Pro with 500 fast model requests, then variable cost per request beyond. The Cursor model selection includes Claude Sonnet, Claude Opus, GPT-5, and Gemini.

For a small team scraping a few sites a day, Cursor Pro is the cheaper bill. For a heavy scraping shop where engineers run multi-hour autonomous jobs, Claude Code on API billing is more predictable because you only pay for what you use.

Real numbers from one week of mixed scraping work on a single engineer’s machine:

Tool	Sessions	Hours	Cost
Claude Code (API)	14	22	$34
Cursor (Pro + overage)	31	18	$24

Cursor came out cheaper for the same engineer doing the same projects, mostly because the editor-driven loop encouraged smaller, cheaper requests. Claude Code’s autonomous loop racks up tokens faster.

When the cost picture flips

Cost flips in favor of Claude Code as soon as the engineer steps away. A four-hour autonomous session that builds and tests three new scrapers might run $6 to $10 on Claude Code, but it freed the engineer for other work. The same outcome in Cursor would take the engineer four hours of attention. At any reasonable engineer hourly rate, the autonomous time wins.

The pattern we see in 2026 mid-size scraping teams: Cursor for the morning standup-to-lunch surgical work, Claude Code as a co-worker assigned long-running greenfield projects.

Working with proxies

Both tools handle proxy code identically because the proxy logic lives in your scraper, not the agent. The difference is in how easily the agent can debug a proxy issue.

Claude Code can curl a proxy directly to verify it works:

> Run: curl -x http://user:pass@proxy.example.com:8000 https://httpbin.org/ip

It reads the response and adjusts the scraper. Cursor can run the same curl through the integrated terminal but the result lives in a panel you have to focus.

For a deeper guide on proxy choices, see our best residential proxy providers 2026 writeup.

Secret handling

Both tools respect a .env file and neither will read or transmit it without a deliberate prompt. The risk surface is the same: a careless paste of a key into the chat is the most common leak vector. Set up .gitignore and pre-commit hooks regardless of which tool you use.

Headless browser handling

Both tools can drive Playwright. The interesting question is what they do when the scraper opens a browser window.

Claude Code does not have a UI, so headed Chromium opens on your local display. The agent can take screenshots if you give it a screenshot MCP tool. Otherwise it is blind to UI state.

Cursor in agent mode can ask Playwright for a screenshot and view the resulting PNG inline. This is a real advantage when you are debugging why a click is not landing.

For purely headless pipelines where the agent never needs to see the browser, this is a wash.

Multi-agent coordination

Claude Code supports sub-agents through the Task tool. You can spin up a specialist sub-agent for one part of the pipeline (say, captcha solving) and have it work in isolation. Cursor does not have a clean equivalent in 2026.

For scraping projects that need parallel work (say, scrape ten sites in parallel and aggregate), Claude Code’s sub-agent pattern is a real differentiator. You write a parent agent that dispatches one sub-agent per site, and the parent aggregates results.

For more on multi-agent scraping, see Multi-agent scraping with AutoGen in 2026.

A simple parallel pattern

A pattern that works well in production: a parent Claude Code session reads a list of 50 URLs, spawns 5 sub-agent tasks each handling 10 URLs, and aggregates the JSON outputs. The parent agent enforces a per-sub-agent timeout and retries failed batches. Total wall-clock time on 50 mixed URLs: roughly 8 minutes versus 35 minutes for sequential. Cost per sub-agent stays predictable because each one operates with a small task.

Documentation and community

Anthropic’s Claude Code docs are the canonical reference. The community on the official Discord and the agent-construction subreddit is active and ships custom skills daily.

Cursor’s docs are clean. The community is enormous (it is the most popular AI editor in 2026) but most discussion is general coding, not scraping-specific.

Workflow patterns we have seen succeed

A few patterns recur across teams that ship scraping work fast.

The “pair programmer” pattern uses Cursor for the scaffold and Claude Code for the harden-and-deploy. The engineer sketches in Cursor, then closes the editor and lets Claude Code add tests, error handling, retries, observability, and a Dockerfile.

The “specialist agent” pattern uses Cursor for daily UI editing and a dedicated long-running Claude Code instance per scraper. Each Claude Code instance owns its scraper directory, runs hourly cron via a wrapper, and posts diffs and incident summaries to Slack.

The “hands-off rebuild” pattern, when an old scraper fails, prompts Claude Code with “this scraper is broken in tests/test_x.py; figure out why and fix it” and walks away. Comes back to a passing build or a clear write-up of why the target site changed in a way that needs a product decision.

Which one to pick

If your team writes scraping code daily and you want the AI to handle multi-step shipping (build, test, deploy a scraper from a one-paragraph prompt), pick Claude Code. The autonomous loop saves real time.

If your team writes scraping code occasionally and most of your work is editing existing pipelines, pick Cursor. The in-editor experience is better for the surgical edit workflow that dominates maintenance.

The honest answer for many shops in 2026 is to use both. Cursor for daily editing, Claude Code for the heavy autonomous tasks. They cost together about what one engineer’s coffee budget runs in a month.

Frequently asked questions

Can I run Claude Code inside Cursor’s terminal?
Yes. Cursor’s integrated terminal runs claude like any other shell. You get Cursor’s editor experience plus Claude Code’s autonomy. This is the setup we recommend for engineers who like both.

Does Cursor’s MCP support match Claude Code’s?
Effectively yes in early 2026. Cursor was slower to ship MCP but the implementation now covers tools, resources, and prompts. Stdio and HTTP transports both supported.

Which one handles long context better?
Both default to Claude Sonnet 4.5 with 1M context. The actual context-handling quality is identical because the model is the same. The differentiator is how each tool prunes context across long sessions.

Can either tool drive Selenium for legacy targets?
Yes. Both can write and run Selenium code. Selenium is the right pick when you must support an ancient browser stack. For everything in 2026, Playwright is the better default.

What about Continue, Zed, Aider, or Cline?
Cline is the closest free competitor to Claude Code. Aider is excellent for git-aware in-place edits. Zed has shipping AI assist that is improving fast. None of them ship the autonomous loop with the polish Claude Code has, in our testing.

Can Claude Code run in CI to repair flaky scrapers automatically?
Yes. Pipe a failure log into claude --resume <session-id> from a GitHub Action and the agent will attempt a fix and open a PR. Set a budget cap to avoid runaway runs.

Which tool is better for a non-engineer running a one-off scrape?
Neither, honestly. Both expect baseline command-line and Python familiarity. For a true non-coder, look at no-code tools like Apify or browser extensions like Instant Data Scraper.

Does Cursor’s agent mode work without a Cursor subscription?
The free tier is severely limited (50 slow requests per month). For any serious scraping work, you need at least Pro.

How do both tools handle very long files like a 2000-line scraper?
Both default to chunked reads, but Claude Code is more conservative about loading the whole file into context. Cursor will sometimes load and re-emit the whole file in a single edit, which costs more tokens but produces a cleaner diff. For files over 1500 lines, Claude Code is the safer pick because partial edits are less likely to corrupt indentation or imports.

Can either tool ship to production directly?
Both can run git push, gh pr create, and CD pipelines via shell. Neither has a native deploy concept. The pattern that works is to wire your existing CD pipeline (GitHub Actions, Vercel, Fly) and let the agent push commits that trigger deploy.

Common pitfalls and gotchas

A short list of things teams trip over in their first month with either tool.

Letting the agent edit the lockfile silently. Both tools will helpfully update requirements.txt or package.json, but they sometimes pin to versions that break elsewhere in your stack. Make CI run pip install -r requirements.txt on a clean cache and fail loudly if it does not resolve.

Forgetting to budget the agent. Claude Code without a --max-turns cap can loop on a confused task and burn $10 in 20 minutes. Always set a budget for autonomous sessions.

Trusting the agent’s claim that “tests pass” without checking. Both tools occasionally report a green build when in fact they ran a subset. Make make test the only acceptance criterion in your CLAUDE.md, and verify by re-running yourself for important changes.

Using Cursor agent mode for tasks where the right answer is a one-line shell command. Cursor will write a Python script when awk would do. Recognize when a task does not need an editor at all.

Mixing both tools on the same file in the same minute. Both write to disk; both watch the file system. Race conditions on saves are real. Use one tool per task at a time.

For more comparisons across the agentic coding tool space and how each pairs with scraping infrastructure, browse our AI modern scraping category.

CCPA compliance for scrapers handling US consumer data

CCPA scraping compliance has grown into the second-most-cited blocker for B2C data pipelines, right behind GDPR. The California Consumer Privacy Act, as amended by the California Privacy Rights Act (CPRA) and now enforced by the California Privacy Protection Agency (CPPA), reshaped what US-touching scrapers can safely do. Many engineering teams still operate under the older 2018 CCPA mental model, and that gap is exactly where 2025 and 2026 enforcement actions landed. This guide walks through the actual rules as enforced today, the public-record carve-out that scraping operators rely on (and frequently misread), the consumer rights you must honour, and a checklist your team can implement this quarter.

The audience here is the data engineer or product lead who already runs a scraping pipeline that touches California residents and needs a defensible compliance posture in 2026.

What CCPA actually covers in scraping context

CCPA applies to any business that collects personal information of California residents and meets one of three thresholds: more than USD 25 million in annual revenue, buys or sells personal information of 100,000 or more consumers or households, or derives 50 percent or more of annual revenue from selling or sharing personal information. Scrapers hit the second and third thresholds easily.

Personal information under Cal. Civ. Code Section 1798.140(v) is defined extremely broadly: any information that identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked, directly or indirectly, with a particular consumer or household. The list of examples runs from the obvious (name, email, address, phone) to the operationally relevant (IP addresses, cookie identifiers, browsing history, geolocation, inferences drawn to create a consumer profile). If you scrape it and it relates to a person, it is personal information.

CPRA added a new category in 2023, sensitive personal information (SPI), which includes Social Security numbers, precise geolocation, race, ethnicity, religious or philosophical beliefs, union membership, contents of mail/email/text messages, genetic data, biometric data, health information, sex life, and sexual orientation. SPI carries additional restrictions and is the highest-risk class for scrapers.

For the broader US context and how state-level privacy laws are converging, see the personal vs public data scraping framework. For the EU equivalent, the GDPR compliance guide for scrapers is the right next read.

The publicly available information carve-out (and its limits)

CCPA explicitly excludes “publicly available information” from the definition of personal information. Section 1798.140(v)(2) defines publicly available as information that is lawfully made available from federal, state, or local government records, or information that a business has a reasonable basis to believe is lawfully made available to the general public by the consumer or from widely distributed media; or information made available by a person to whom the consumer has disclosed the information if the consumer has not restricted the information to a specific audience.

This is a real carve-out, but it is narrower than scrapers often assume. Three pitfalls.

First, the “lawfully made available” qualifier means information leaked, hacked, or scraped in violation of terms of service does not become publicly available just because it ended up online. A doxxing forum dump is not publicly available information under CCPA, even if you can read it.

Second, the “consumer has not restricted” carve-out means a profile a user marked private but you accessed via a workaround does not qualify. The user’s restriction state at the time of collection matters.

Third, inferences drawn from publicly available information are not themselves publicly available. If you scrape a profile photo from a public LinkedIn page and then run a face-recognition model against it to infer ethnicity, the inferred ethnicity is personal information (and likely SPI), even though the source was public.

The CPPA has signalled in 2024 and 2025 enforcement guidance that it reads the carve-out narrowly. Treat it as a defence you may invoke, not a shield you assume.

Compliance checklist for scrapers handling California data

Control	What it requires	Why it matters
Privacy policy with CCPA disclosures	Categories of PI collected, sources, purposes, third parties	Section 1798.130
“Do Not Sell or Share My Personal Information” link	Homepage link if you sell or share	Section 1798.135
Opt-out mechanism	Functional within 15 business days	Section 1798.135
Right to know request handling	Verifiable response within 45 days	Section 1798.130
Right to delete request handling	Verifiable deletion within 45 days	Section 1798.105
Right to correct request handling	Honour correction requests	Section 1798.106
Limit use of SPI	Honour the limit-the-use-of-SPI right	Section 1798.121
Service provider contracts	CCPA-compliant DPAs with vendors	Section 1798.140(ag)
Data minimisation	Only collect what is necessary and proportionate	CPRA Section 1798.100(c)
Retention schedules	Disclose and enforce retention periods	Section 1798.100(a)(3)
Annual cybersecurity audit (if high risk)	CPPA forthcoming regulations	CPRA
Risk assessment for high-risk processing	CPPA forthcoming regulations	CPRA

A scraper that ticks every row above operates inside the safe harbour. One that ticks half is exposed.

Consumer rights and the request workflow

CCPA grants California residents seven core rights: right to know, right to delete, right to correct, right to opt out of sale or sharing, right to limit use of SPI, right to non-discrimination, and right to data portability. For a scraper, the operationally heavy rights are right to know, right to delete, and right to opt out of sale.

Right to know means a consumer can request the categories and specific pieces of personal information you collected about them, the sources, the business or commercial purpose, and the third parties you shared with. You have 45 days to respond. The CPPA expects you to be able to identify the consumer in your dataset, which means your storage schema needs to be queryable by identifier types you collected (email, name plus zip, device ID).

Right to delete means once a verifiable request is received, you must delete the consumer’s personal information from your records and instruct service providers and contractors to do the same. There are exceptions (legal compliance, security, free speech, internal analytics consistent with consumer expectations), but the default is delete.

Right to opt out of sale or sharing is broader than many teams realise. “Sale” includes any disclosure for monetary or other valuable consideration. If you scrape data and license it to customers, that is a sale. You must honour the Global Privacy Control (GPC) signal as a valid opt-out, automatically and without requiring further action. The CPPA confirmed this in 2024 enforcement actions.

For a worked decision tree on how to triage rights requests, see the ethics-first scraping policy guide.

How CCPA enforcement shifted in 2024 and 2025

The CPPA, which took over administrative enforcement in 2023, brought a rulemaking and audit-driven approach that the original Attorney General enforcement lacked. Three trends.

First, the CPPA targeted data brokers explicitly. The Delete Act (SB 362), in force since 2026, requires data brokers to register annually and to honour a single deletion mechanism that consumers can use across all brokers at once. Scrapers that resell personal information meet the data broker definition under California law, full stop. Registration is not optional.

Second, enforcement action shifted from notice-and-cure to direct fine. The 30-day cure period that the original CCPA included was eliminated by CPRA. A scraper that fails to honour an opt-out request can face civil penalties of USD 2,500 per violation or USD 7,500 per intentional violation, with each individual consumer counted separately. A breach affecting 10,000 California residents can produce a USD 75 million liability ceiling.

Third, the CPPA has aggressively enforced the GPC requirement. A 2025 settlement with a major data broker centred on the broker’s failure to recognise GPC signals automatically. The fine was significant, the public-shaming letter was widely read, and the message was unmistakable: GPC is mandatory.

For the parallel UK and EU enforcement environment, see the GDPR compliance guide.

Decision tree for a US-touching scrape

Q1: Does the target site host personal info of California residents?
    ├── No  -> CCPA likely not in scope. Document the assessment.
    └── Yes -> Q2
Q2: Is the data clearly within the publicly available carve-out?
    ├── Yes -> Document why; still recommended to honour deletion requests.
    └── No  -> Q3
Q3: Does your business meet a CCPA threshold?
    ├── No  -> CCPA does not apply directly; state laws may.
    └── Yes -> Q4
Q4: Have you published a CCPA-compliant privacy policy?
    ├── No  -> Publish before launching.
    └── Yes -> Q5
Q5: Do you sell or share the scraped data?
    ├── Yes -> Add "Do Not Sell or Share" link; honour GPC; register if data broker.
    └── No  -> Q6
Q6: Will you process sensitive personal information?
    ├── Yes -> Honour limit-use right; consider risk assessment.
    └── No  -> Proceed; log the assessment in your records.

Service provider, contractor, and third party

CCPA distinguishes between three downstream relationships. A service provider processes personal information on your behalf under a written contract that restricts further use. A contractor is similar but typically engaged on a one-off basis. A third party receives personal information for its own purposes; this is where “sale” attaches.

Scrapers commonly sit in two roles: as a service provider when they scrape on behalf of a customer under a DPA, and as a third party when they license the dataset for the customer’s independent use. The DPA you sign with a proxy provider is a service provider agreement. The DPA you sign with a customer who buys your dataset is potentially a third-party arrangement, depending on how restrictive the contract is. Get this categorisation wrong and you have either misclassified a sale (CPPA fine territory) or imposed restrictions you cannot enforce (commercial conflict).

Comparison: CCPA vs GDPR for scrapers

Dimension	CCPA / CPRA	GDPR
Personal data definition	Broad, includes household	Broad, individual only
Lawful basis required	No, but right to opt out of sale	Yes, six bases
Public data carve-out	Yes (publicly available)	None
Right to delete	Yes (with exceptions)	Yes (Article 17)
Right to opt out of sale	Yes (mandatory GPC)	Implicit in lawful basis
Sensitive data category	Yes (SPI, CPRA addition)	Yes (special categories)
Extraterritorial reach	Yes if doing business in CA	Yes if processing EU data
Statutory damages	Yes, per-violation civil penalty	Administrative fines up to 4% revenue
Cure period	None (after CPRA)	Limited
Private right of action	Limited (data breach only)	Yes (Article 82)

The two regimes overlap heavily but diverge on lawful basis and the public data carve-out. Build for both and you have most US and EU coverage.

External references

The canonical statute is the California Civil Code, Title 1.81.5, hosted at oag.ca.gov/privacy/ccpa. The CPPA publishes its regulations and enforcement actions at cppa.ca.gov. The Global Privacy Control specification is at globalprivacycontrol.org.

Operationalising opt-out signals

The Global Privacy Control is a browser-emitted signal in the request headers (Sec-GPC: 1) that indicates the user has opted out of the sale or sharing of their personal information. The CPPA requires you to honour it automatically. Implementation for a scraping operator is two-part: detect the GPC signal at any user-facing surface (your website, your customer portal, your data preview pages) and treat any consumer whose original collection context included GPC as opted out by default.

For a scraped dataset, this is harder, because you typically do not have GPC headers from the scraping target. The practical workaround: when you receive a deletion or opt-out request, do not require the requester to re-authenticate from a GPC-enabled browser. Treat the request as valid based on identifier match alone, and document the verification path.

Special cases: data brokers, AI training, and hiring

The Delete Act (SB 362) made California the first US state with a single-source deletion mechanism for data brokers. Once the deletion portal is fully live (2026 phased rollout), any consumer can submit a single request that deletes their data across every registered broker. Scrapers who meet the data broker definition must register, must honour the central deletion list, and must not re-collect deleted consumers’ data within an enforcement window.

AI training is now subject to additional CPPA risk assessment requirements when the training set includes California residents’ personal information at scale. The risk assessment must address the necessity of the training data, the safeguards against re-identification, and the consumer rights surface for opt-out and deletion. Several large model providers were quietly fined in 2025 for failing to file the risk assessment.

Hiring and employee data was carved out of CCPA from 2018 to 2023 but became fully covered in 2023. A scraper that pulls professional profile data of California residents now operates under full CCPA, with no employment-context exemption.

FAQ

Is publicly available data exempt from CCPA?
Partially. The carve-out only covers data that was lawfully made publicly available and that the consumer has not restricted. Inferences drawn from public data are not themselves public.

Do I need to honour Global Privacy Control signals?
Yes. The CPPA confirmed in 2024 that GPC is a valid opt-out signal that must be honoured automatically.

What is the fine range under CCPA in 2026?
Civil penalties are USD 2,500 per violation or USD 7,500 per intentional violation, with each individual consumer counted separately.

Am I a data broker if I scrape and sell?
If you knowingly collect and sell personal information of consumers with whom you do not have a direct relationship, yes, and you must register annually under the Delete Act.

Does CCPA apply to B2B data?
Yes since 2023. The B2B carve-out expired and professional contact data of California residents is now fully covered.

Extended enforcement analysis 2024-2026

The California Privacy Protection Agency moved from rulemaking to active enforcement during 2024 and 2025. The DoorDash settlement (February 2024, USD 375,000) was the first to specifically cite cross-context scraping of consumer data without a working opt-out signal. The CPPA’s enforcement advisories in 2025 covered three patterns relevant to scrapers, namely failure to honour the Global Privacy Control header, failure to recognise the Sec-GPC header on automated traffic, and failure to surface a Do Not Sell or Share My Personal Information link in the privacy notice that links the scraping operation to the consumer-facing brand.

The Sephora case (August 2022, USD 1.2 million) remains the touchstone for California enforcement on opt-out signals. Sephora was found in violation for failing to process opt-out signals as valid CCPA requests. Every scraper that touches California residents should treat that case as authoritative and design GPC handling into the ingest layer, not a downstream marketing tool.

A pattern emerged in 2025 that scrapers should plan for. The CPPA increasingly views scraping followed by enrichment, segmentation, and resale as a sale or sharing event under CCPA, even if the scraping operator does not directly transfer data. The triggering test is whether the consumer would reasonably understand that their public information would be combined with non-public signals and sold downstream. For B2B people-data vendors this is now the central compliance question.

Implementation patterns for a CCPA-clean pipeline

The minimum control set for a US-touching 2026 scraping pipeline includes nine items.

1. A GPC and Sec-GPC header check at every fetch with the result logged per request.
2. A privacy notice link surfaced on every consumer-facing surface that touches scraped data.
3. A right-to-know workflow that responds within forty-five days with extension up to ninety.
4. A right-to-delete workflow with verification that does not over-collect identity proof.
5. A right-to-correct workflow added in 2023 amendments and now actively enforced.
6. A right-to-limit-use-of-sensitive-personal-information workflow.
7. A service provider contract with every downstream processor.
8. A data inventory that distinguishes personal information from sensitive personal information.
9. A retention schedule documented per category and enforced.

Worked example: GPC handling at fetch time

def should_index(response, headers):
    gpc = headers.get("Sec-GPC", "0")
    if gpc == "1":
        log.info("gpc_signal_present", url=response.url)
        return False  # treat as opt-out for downstream sale or share
    return True

The check belongs at the ingest layer because removing data downstream after vectorisation is harder than skipping it at fetch.

Additional FAQ

Do I need a CCPA notice if I never sell data?
Yes if you process personal information of California residents above the thresholds. The notice obligation is independent of sale.

Does the publicly available carve-out cover LinkedIn profiles?
Generally no. The carve-out applies to information lawfully made available from federal, state, or local government records, plus information the consumer or their authorised agent has made available. Commercial platforms with terms of service restricting bulk access do not satisfy the carve-out by themselves.

What is the difference between sale and share under CCPA?
Sale is exchange for monetary or other valuable consideration. Share is disclosure for cross-context behavioural advertising. Both trigger opt-out rights and the Do Not Sell or Share link.

How do I verify a deletion request without over-collecting?
Match against information you already hold. A consumer should not have to provide more identity than the minimum needed to confirm the match. Documentation of the verification logic is part of compliance.

Practical scope determination for CCPA

Determining whether the CCPA applies to a scraping operation requires analysis on three axes. First, does the scraping operation process personal information of California residents. Second, does the operating entity meet the size threshold (USD 25 million annual revenue, or 100,000 California consumers, or 50 percent of revenue from selling personal information). Third, does the activity fit within the CCPA’s exempted categories.

For most commercial scrapers the first axis is yes by default, the second axis is met for any team above small startup size, and the third axis offers little relief. The narrow exemptions for medical information governed by HIPAA, financial information governed by GLBA, and certain business-to-business communications during the transition period in earlier amendments are of limited use to a generic scraper.

The 2024 amendments and CPPA regulations clarified that aggregators, brokers, and AI training data vendors fall squarely within scope when they touch California-resident data. The CPPA’s enforcement priorities published in 2025 listed data brokers and AI training data as the top two areas of focus. Scrapers in those categories should plan for a CCPA registration where applicable and a higher level of regulatory attention.

Sensitive personal information and the right to limit

The CCPA’s 2023 amendments introduced a new category of sensitive personal information (SPI) and a new right to limit its use and disclosure. SPI includes Social Security numbers, driver’s licence numbers, financial account information, precise geolocation, racial or ethnic origin, religious beliefs, mail and email content, genetic data, biometric data, health data, and sex life or sexual orientation.

For scrapers the SPI category is operationally similar to GDPR Article 9 special category data. The scraper should detect SPI at ingest, route it to a separate handling pathway with stricter access controls, and surface a right-to-limit-use mechanism on the consumer-facing surface.

The right to limit is narrower than the right to delete. A consumer who exercises the right to limit is restricting use to specific listed purposes (services requested, security and integrity, certain analytics) but is not requiring deletion. The scraper must therefore have a way to flag SPI records as limited and prevent downstream non-listed uses.

Service provider, contractor, and third party distinctions

The CCPA distinguishes service providers (who process personal information on behalf of a business under a written contract restricting their use), contractors (a 2023 addition broadly similar to service providers but with subtle differences), and third parties (everyone else). The classification matters because transfers to service providers and contractors are not sales or shares, but transfers to third parties typically are.

A scraping operation that resells data to clients must therefore decide whether each client is a service provider, contractor, or third party, and put the right contract in place. The CPPA template language for service provider contracts is the safest starting point. Contracts that diverge from the template are scrutinised more closely.

A common 2026 mistake is treating analytics platforms as service providers without a service provider contract. Without the contract, the data transfer to the analytics platform is a sale or share that triggers the opt-out right and the Do Not Sell or Share link.

Next steps

The fastest path to a defensible CCPA posture in 2026 is to publish the privacy policy with the required disclosures, wire up GPC detection across your customer-facing surfaces, stand up a deletion inbox you actually monitor, and register as a data broker if you sell scraped data. For broader policy guidance, head to the DRT compliance and ethics hub and pair this guide with the ethics-first policy build.

This guide is informational, not legal advice.

Scraping with MCP servers in 2026: a practical guide

MCP servers scraping is the architecture pattern that finally stopped feeling experimental in early 2026. Anthropic shipped the Model Context Protocol in late 2024, the spec stabilized at the 2025-06-18 revision, and by Q1 2026 every major LLM client (Claude Desktop, Claude Code, Cursor, Zed, Continue, the OpenAI Responses API, and Gemini Code Assist) speaks MCP natively. For scraping teams that means one thing: write your scraping logic once as an MCP server, and every LLM-driven workflow on the planet can call it.

This guide walks through building an MCP server that exposes scraping tools, runs them inside an isolated browser pool, returns structured data, and handles auth, rate limiting, and observability. By the end you will have a server that any MCP-compatible client can plug into, code that works in production, and benchmarks that show where MCP wins and where it does not.

Why MCP is the right shape for scraping

The classic problem with LLM-driven scraping is that every team reinvents the same plumbing. You write a Python function that fetches a page, you wrap it in a tool schema for OpenAI function calling, you wrap it again for Anthropic tool use, you wrap it a third time for Gemini, and now your tool is locked to one client per integration. MCP collapses all three integrations into one server.

MCP is a JSON-RPC 2.0 protocol with three primitive types: tools (functions the LLM can call), resources (data the LLM can read), and prompts (templates the LLM can request). For scraping, you mostly care about tools. The full spec is at modelcontextprotocol.io and the reference implementations live on the MCP servers GitHub repo.

Three properties make MCP the right shape for scraping infrastructure. The protocol is transport-agnostic, so you can run a server over stdio for local trust or HTTP with bearer auth for remote access. Tool schemas are JSON Schema, so the LLM gets typed parameters and the server gets validation for free. Servers are stateful by design, so you can keep a browser session warm across tool calls without leaking state across users.

What MCP is not

A few misconceptions are worth heading off because they show up in design reviews. MCP is not a model. It is a protocol that lets a client and a server agree on what tools exist and how to call them. MCP is not a hosted service. Anthropic publishes the spec and the SDKs, but you run your own servers wherever you like. And MCP is not exclusive to Claude. The protocol is open, and OpenAI, Google, and the major IDE vendors have shipped MCP clients in the last six months.

Resources versus tools for scraped data

The protocol distinguishes resources (read-only data the LLM can pull) from tools (actions with side effects). For scraping, the rule of thumb is: expose live fetches as tools and expose recent results as resources. A recent_scrapes resource that lists the last 50 successful fetches by URL means the LLM can reference past work without paying to scrape the same page twice. This pattern alone has cut LLM token spend by 20 to 30 percent on workflows where the same handful of URLs get queried repeatedly.

Designing your scraping MCP server

A useful scraping MCP server exposes three to seven tools. Resist the urge to expose forty. The LLM picks tools by reading their descriptions, and a long tool list dilutes selection accuracy.

A clean baseline tool surface for a general scraping server:

Tool name	Purpose	Returns
fetch_url	GET a single URL with retry and proxy rotation	HTML or JSON body
extract_structured	Run an LLM extraction prompt over fetched HTML	JSON matching a passed schema
screenshot	Render via headless Chromium and return PNG	base64 PNG
search_serp	Issue a query against a SERP provider	top 10 results with title, snippet, URL
crawl	BFS over a site with depth and same-origin filters	list of URL plus metadata

Each tool gets a JSON Schema describing its parameters and a one-paragraph description that tells the LLM exactly when to call it. Bad descriptions are the most common cause of tool-selection failures.

Writing tool descriptions the LLM actually understands

The single biggest win in MCP server design is treating the tool description as a prompt, not as documentation. A bad description reads like a function comment: “Fetches a URL and returns the body.” A good description tells the LLM when to choose this tool over the alternatives, what to pass, and what to expect back.

Compare:

Bad:
  Fetches a URL and returns the response body.

Good:
  Fetch a URL over HTTP with automatic retry and proxy rotation.
  Use for static HTML pages, JSON APIs, or any resource that does not
  require JavaScript rendering. For pages that need a browser (SPAs,
  pages behind Cloudflare interactive challenges), call render_page
  instead. Returns status, content type, and body. body is truncated
  at 200 KB so for very large pages, paginate with the offset arg.

The “good” version names a sibling tool by name, sets expectations on truncation, and tells the LLM when not to use it. This kind of cross-referencing between tools cuts tool-selection errors by roughly half on multi-tool servers.

Building the server in Python

The official Python SDK is mcp on PyPI. The fastest path is to use the FastMCP helper, which gives you a Flask-style decorator API.

pip install "mcp[cli]" httpx playwright pydantic
playwright install chromium

Skeleton server with three tools:

from mcp.server.fastmcp import FastMCP
from pydantic import BaseModel, Field
from typing import Optional
import httpx
import asyncio
from playwright.async_api import async_playwright

mcp = FastMCP("drt-scraping-server")

class FetchResult(BaseModel):
    url: str
    status: int
    content_type: str
    body: str
    final_url: str

@mcp.tool()
async def fetch_url(
    url: str = Field(..., description="The URL to fetch"),
    timeout_s: int = Field(30, description="Request timeout in seconds"),
    proxy: Optional[str] = Field(None, description="Optional proxy URL"),
) -> FetchResult:
    """Fetch a single URL with retry. Use for static HTML, JSON APIs, or any
    resource that does not require JavaScript rendering."""
    async with httpx.AsyncClient(
        proxy=proxy,
        timeout=timeout_s,
        follow_redirects=True,
        headers={"User-Agent": "Mozilla/5.0 (compatible; DRTBot/1.0)"},
    ) as client:
        r = await client.get(url)
        return FetchResult(
            url=url,
            status=r.status_code,
            content_type=r.headers.get("content-type", ""),
            body=r.text,
            final_url=str(r.url),
        )

@mcp.tool()
async def screenshot(url: str, full_page: bool = True) -> bytes:
    """Render a page in headless Chromium and return a PNG screenshot.
    Use when you need to see how a page actually renders."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="networkidle")
        png = await page.screenshot(full_page=full_page)
        await browser.close()
        return png

if __name__ == "__main__":
    mcp.run(transport="stdio")

That is a functional server. Run it with python server.py and Claude Desktop will pick it up if you add an entry to claude_desktop_config.json.

A TypeScript variant

The TypeScript SDK is just as ergonomic and is the right pick if your team already runs Node services. The decorator-style is replaced with method registration, but the shape is similar.

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import fetch from "node-fetch";

const server = new McpServer({ name: "drt-scraping-server", version: "1.0.0" });

server.tool(
  "fetch_url",
  {
    url: z.string().url(),
    timeout_s: z.number().int().default(30),
  },
  async ({ url, timeout_s }) => {
    const ctrl = new AbortController();
    const t = setTimeout(() => ctrl.abort(), timeout_s * 1000);
    try {
      const r = await fetch(url, { signal: ctrl.signal });
      const body = await r.text();
      return {
        content: [
          { type: "text", text: JSON.stringify({ status: r.status, body }) },
        ],
      };
    } finally {
      clearTimeout(t);
    }
  }
);

await server.connect(new StdioServerTransport());

The Python and TypeScript SDKs interoperate cleanly because both speak the same wire protocol. Pick the one your team will maintain.

Choosing a transport

MCP supports stdio, HTTP with Server-Sent Events (SSE), and the newer streamable HTTP transport added in the 2025-06-18 spec.

Transport	When to use	Auth model
stdio	Local trust, single user, fastest	OS process boundary
HTTP + SSE	Multi-user remote, legacy clients	Bearer token, OAuth 2.1
Streamable HTTP	Multi-user remote, modern spec	Bearer token, OAuth 2.1

For a scraping server that runs on your laptop and is only called by your own Claude Desktop, stdio is the right answer. For a server that other team members or production agents call, run streamable HTTP behind an auth gateway.

A minimal HTTP-mode launch:

if __name__ == "__main__":
    mcp.run(transport="streamable-http", host="0.0.0.0", port=8765)

When to choose streamable HTTP over SSE

The 2025-06-18 spec introduced streamable HTTP as the preferred transport because SSE has two known issues at scale. SSE connections are one-way (server to client) so the client has to open a separate POST channel for messages, which doubles the connection count under load. And SSE does not survive a load balancer that aggressively closes idle connections, which is the default for most cloud LBs.

Streamable HTTP folds the message channel and the event channel into a single bidirectional connection, and it tolerates short network blips by allowing the client to reconnect with a session ID. If your client supports it, use it.

Adding proxy rotation

The single feature that separates a toy scraping MCP from a useful one is automatic proxy rotation. Bake it into the server, do not push it to the LLM.

import os
import random

PROXIES = [p.strip() for p in os.environ.get("PROXY_POOL", "").split(",") if p.strip()]

def pick_proxy() -> Optional[str]:
    if not PROXIES:
        return None
    return random.choice(PROXIES)

@mcp.tool()
async def fetch_url_pooled(url: str, timeout_s: int = 30) -> FetchResult:
    """Fetch a URL through the server's managed proxy pool. Always prefer
    this over fetch_url for production scraping."""
    proxy = pick_proxy()
    return await fetch_url(url=url, timeout_s=timeout_s, proxy=proxy)

For ASEAN scraping, pair the pool with Singapore mobile proxy or other rotating mobile providers so every call gets a fresh real-carrier IP.

Per-domain stickiness

Random rotation breaks cart and checkout flows. Add a per-domain sticky binding so the same domain reuses the same exit IP for the duration of a session.

from collections import defaultdict
from urllib.parse import urlparse

_session_proxies: dict[tuple[str, str], str] = {}

def pick_proxy_for(session_id: str, url: str) -> Optional[str]:
    if not PROXIES:
        return None
    domain = urlparse(url).netloc
    key = (session_id, domain)
    if key not in _session_proxies:
        _session_proxies[key] = random.choice(PROXIES)
    return _session_proxies[key]

Pair this with a TTL so abandoned sessions release their proxies, and you have a clean implementation that survives real-world ecommerce flows.

Structured extraction as a tool

The most powerful pattern is to expose extraction as its own tool that takes a JSON Schema and returns structured data. This lets the LLM ask for exactly the shape it needs.

import json
from openai import AsyncOpenAI

client = AsyncOpenAI()

@mcp.tool()
async def extract_structured(
    html: str = Field(..., description="HTML to extract from"),
    schema: dict = Field(..., description="JSON Schema for the desired output"),
    instructions: str = Field("", description="Optional extraction guidance"),
) -> dict:
    """Extract structured data from HTML using an LLM with a JSON Schema."""
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "extraction", "schema": schema, "strict": True},
        },
        messages=[
            {"role": "system", "content": "Extract data from HTML matching the schema. " + instructions},
            {"role": "user", "content": html[:200000]},
        ],
    )
    return json.loads(resp.choices[0].message.content)

This tool composes beautifully. The LLM client calls fetch_url, receives HTML, then calls extract_structured with a schema like {"type": "object", "properties": {"title": {"type": "string"}, "price": {"type": "number"}}} and gets clean JSON back.

Caching extractions

The same HTML extracted with the same schema should not pay LLM cost twice. Hash the (html, schema, instructions) triple and cache the result in Redis with a 24-hour TTL.

import hashlib, redis.asyncio as redis

r = redis.from_url(os.environ["REDIS_URL"])

async def extract_cached(html, schema, instructions):
    key = "ext:" + hashlib.sha256(
        (html + json.dumps(schema, sort_keys=True) + instructions).encode()
    ).hexdigest()
    cached = await r.get(key)
    if cached:
        return json.loads(cached)
    out = await extract_structured(html, schema, instructions)
    await r.setex(key, 86400, json.dumps(out))
    return out

On a workflow that hits the same product detail pages every hour, this saves an order of magnitude on LLM cost.

Auth and rate limiting

For HTTP-mode servers, never run without auth. The minimal middleware:

from starlette.middleware import Middleware
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse

class BearerAuthMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        token = request.headers.get("authorization", "").replace("Bearer ", "")
        if token != os.environ["MCP_BEARER_TOKEN"]:
            return JSONResponse({"error": "unauthorized"}, status_code=401)
        return await call_next(request)

For rate limiting, wrap each tool with a per-user token bucket. The MCP spec gives you a session ID per client, which is the right key for buckets.

Add structured logging on every tool call. tool_name, params_hash, duration_ms, status, client_session_id, proxy_used. This is the data you need when debugging why an agent is misbehaving.

Moving to OAuth 2.1

Bearer tokens are fine for an internal team but break the moment you expose the server to other organizations or third-party agents. The 2025-06-18 spec adopts OAuth 2.1 with PKCE as the recommended auth flow. Run an OAuth provider in front (Auth0, Authentik, or self-hosted Hydra are all good fits), have clients exchange a code for an access token, and validate the JWT in your middleware.

The client SDKs handle the OAuth dance automatically when configured with an authServerUrl, so the developer experience does not get worse.

Comparing MCP-driven scraping to alternatives

Pattern	Setup time	LLM portability	Multi-user	Best fit
Direct OpenAI function calls	1 hour	OpenAI only	No	Single LLM, single agent
LangChain tools	2 hours	LangChain only	No	Prototypes
MCP server	4 hours	Any MCP client	Yes	Team or product use
Custom HTTP API	1 day	All, with bespoke wrappers	Yes	Existing API surface
LangGraph custom node	3 hours	LangGraph only	Partial	Stateful workflows
OpenAI Assistants tools	1 hour	OpenAI Assistants	Limited	Hosted assistants

MCP wins when you need the same scraping logic to be callable from Claude Desktop on one developer’s laptop and from a production LangGraph agent in your data pipeline. You write the server once.

For a deeper breakdown of where MCP fits in a 2026 data engineering stack, see MCP for data engineers in 2026.

Production deployment

Deploy as a small Docker image. Pin Python, pin Playwright Chromium, and run as a non-root user.

FROM mcr.microsoft.com/playwright/python:v1.49.0-jammy

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY server.py .
USER pwuser
EXPOSE 8765
CMD ["python", "server.py"]

Run two instances behind a load balancer for redundancy. MCP sessions are not sticky in the streamable HTTP transport, so you can round-robin freely.

For observability, OpenTelemetry instrumentation with the Anthropic-published MCP semantic conventions is the path of least resistance. Span attributes: mcp.server.name, mcp.tool.name, mcp.session.id, mcp.transport.

Health checks and graceful shutdown

Add a /healthz endpoint that returns 200 only if the proxy pool has at least one live IP and Playwright can launch a browser. A simple TCP check on port 8765 is not enough because the server can accept connections while completely unable to do useful work.

@mcp.custom_route("/healthz")
async def healthz(request):
    if not PROXIES:
        return JSONResponse({"ok": False, "reason": "no proxies"}, 503)
    try:
        async with async_playwright() as p:
            b = await p.chromium.launch(headless=True)
            await b.close()
    except Exception as e:
        return JSONResponse({"ok": False, "reason": str(e)}, 503)
    return JSONResponse({"ok": True})

On shutdown, flush in-flight tool calls before exiting. Most orchestrators send SIGTERM, wait 30 seconds, then SIGKILL. Wire your shutdown handler to drain.

Real benchmarks

A scraping MCP server with the five-tool surface described above, deployed on a 2vCPU 4GB Fargate task with a 50-IP rotating residential pool, hits the following numbers in production:

Metric	Value
fetch_url_pooled p50 latency	740 ms
fetch_url_pooled p99 latency	4.8 s
screenshot p50 latency	3.1 s
extract_structured p50 latency	1.9 s
Concurrent sessions per task	30 to 50
Cost per 1000 page fetches	$0.18 (proxy) + $0.04 (compute) + LLM tokens
Memory per active session	35 MB idle, 180 MB peak with browser
Cold start to first tool call	4.2 s (Fargate)

LLM tokens for a typical extract-after-fetch workflow run $0.001 to $0.005 per page on GPT-4o-mini, depending on page size. Total cost around $0.30 to $0.50 per 1000 pages including everything.

Failure mode benchmarks

Headline latency hides the failure tail. From 100,000 production calls in March 2026:

Failure type	Rate	Mitigation
Proxy connection refused	1.4%	Healthcheck + auto-evict bad IPs
403 from target site	2.1%	Rotate IP and retry, escalate to browser tool
Timeout (>30s)	0.8%	Per-domain timeout tuning
Playwright browser crash	0.3%	Recycle browser, retry once
LLM 429 rate limit	0.6%	Token-bucket on extract calls
OOM (Chromium)	0.05%	Cap pages per browser at 100

A retry layer with exponential backoff on transient failures pulls the overall success rate from 95 percent to over 99 percent without adding more than 3 percent latency overhead.

Pairing with agentic clients

The whole point of MCP is that any client can call your tools. The most common pairings in production:

• Claude Desktop, for human-driven exploratory scraping
• Claude Code or Cursor, for engineers who want scraping inline with their editor
• A LangGraph agent, for autonomous workflows
• An OpenAI Responses API agent, for OpenAI-native production stacks

For more on agentic LLM clients in scraping, see The agentic browser revolution: Claude, OpenAI Operator, Stagehand.

Common production gotchas

• Tool descriptions live in your code, but the LLM sees them at runtime. Changing a description without a client reconnect means the LLM is operating on stale info. Force clients to refresh on server version bump.
• Pydantic field defaults that are mutable (lists, dicts) get shared across calls. Use Field(default_factory=list) not Field([]).
• The MCP initialize handshake takes one round trip per client connect. For high-churn workloads, hold connections open longer rather than reconnecting per request.
• Streaming results with yield is supported but every client implements it differently. Test with each client you intend to support.
• The Playwright browser holds file handles for downloaded resources. On long-running servers, close pages explicitly or you will hit the OS file descriptor limit around 1024.

Frequently asked questions

Do I need to write my own MCP server, or are there existing scraping servers?
Both. The Anthropic MCP servers repo ships a Puppeteer reference server and a Brave Search server. They are useful baselines but lack proxy rotation, session management, and the structured extraction tool you almost always end up wanting. Fork or write your own.

Can MCP servers maintain browser session state across tool calls?
Yes. Hold a Playwright BrowserContext per MCP session in a dict keyed by session_id. Tear down on session end. The MCP SDK exposes session lifecycle hooks for exactly this.

What is the Cloudflare AI Gateway story for MCP?
Cloudflare added MCP gateway support in early 2026. You can put your MCP server behind a Cloudflare AI Gateway and get logging, caching, and rate limiting without writing any of it.

How do I version my MCP server?
The MCP spec includes a serverInfo.version field. Bump it on every release and emit a changelog. Clients can pin to a version range, but most simply read whichever version is exposed.

Is MCP overkill for a one-off scraping job?
Yes. Use a plain Python script. MCP pays off when the same scraping logic needs to be called from multiple agents, multiple developers, or multiple stacks.

How do I expose secret-bearing tools without leaking the secret to the LLM?
Keep the secret in the server environment and never include it in tool args or descriptions. The LLM sees only the tool name and the schema, so an authenticated_fetch tool can use a server-side API key that the LLM never learns.

Can one MCP server talk to another MCP server?
Yes. The Python SDK ships an MCP client. Build a meta-server that fans out to specialized backend MCPs (proxy server, browser server, extraction server). Composition is the long game for MCP architectures.

Is there a registry of public MCP servers I can borrow tools from?
The community is building one at mcphub.io and several others. As of mid-2026 most production teams still write their own because the public servers vary in maintenance quality.

If you are evaluating MCP for a new scraping initiative, start with the AI modern scraping category for guides on the major LLM clients and adjacent tools.

Etsy surfaces its best-seller badges and trending tag labels on public product pages, and scraping them at scale is genuinely useful for competitive research, niche validation, and dropshipping product discovery. The catch is that Etsy runs aggressive bot detection, rate-limits unauthenticated crawlers hard, and returns different HTML depending on whether your request looks like a browser or a script. Here is a practical 2026 guide to getting the data reliably.

What Data You Can Actually Pull

Etsy does not expose a public API for best-seller or trending data. Everything you care about lives in rendered HTML or embedded JSON-LD on product and search pages.

Useful fields per listing:

• Listing title, price, sale price
• “Bestseller” badge (a with class wt-badge--small)
• Star rating and review count
• Shop name and sales count
• Tags (visible on listing pages, not search results)
• Estimated monthly sales (inferred from review velocity, not served directly)

Trending tags appear in Etsy’s search autocomplete (/api/v3/ajax/typeahead/etsy/term) and in the “Shop by popular tags” carousels on category pages. Both endpoints are accessible without login but require consistent headers.

How Etsy Detects Bots

Before writing a single line of code, understand the detection stack you are up against:

Layer	Method	Notes
TLS fingerprinting	JA3/JA4 hash check	Requests/httpx fail without spoofing
Header validation	User-Agent, Accept, Sec-Fetch-*	Missing Sec-Fetch headers = instant block
IP reputation	DataDome (embedded on most pages)	Datacenter IPs blocked by default
Behavioral analysis	Mouse events, scroll timing	Only triggers on JS-heavy category pages
CAPTCHA	hCaptcha	Triggered on rapid listing traversal

The TLS fingerprint check is the highest-priority hurdle. Plain requests with a spoofed User-Agent still fails because the TLS handshake looks like Python. Use curl_cffi with impersonate="chrome120" or route through a residential proxy with its own TLS termination.

DataDome is the persistent layer. It tracks request cadence across sessions and will silently serve degraded HTML (no badge data, no review count) long before it serves a hard block. This is similar to the detection stack you encounter when doing more general marketplace work like scraping Walmart Marketplace seller data.

Scraping Best-Seller Listings: Working Approach

For listing-level data, the most reliable path in 2026 is:

1. Build a seed URL list from Etsy search (/search?q=&explicit=1&ship_to=US)
2. Paginate through results pages (up to page 250, ~6000 results per query)
3. For each listing URL, fetch the product page and parse the embedded