How to Scrape DoorDash Restaurants and Menus (2026)

DoorDash serves over 700,000 restaurants across the US, Canada, and Australia — and if you’re doing competitive pricing research, building a food delivery aggregator, or tracking menu trends across cities, scraping DoorDash restaurant menus is a very real engineering task. this article covers how DoorDash’s frontend and API actually work in 2026, what tools cut through their bot defenses, and how to extract structured menu and pricing data at scale without burning your IP pool.

how DoorDash serves its menu data

DoorDash is a Next.js app. the menu page you see in a browser is server-side rendered, but the actual item data comes from a graphQL endpoint: consumer-mobile-bff.doordash.com/consumer/graphql. same endpoint their mobile apps use.

the good news: that API returns clean JSON with item names, prices, descriptions, calories, and modifiers (size, add-ons, etc). the bad news: it requires authenticated session cookies and a valid x-channel-id header that rotates. unauthenticated requests get a 401 or a silent empty response. which is worse than a 401, honestly.

the HTML rendered pages also include a __NEXT_DATA__ JSON blob embedded in a <script> tag. for light scraping this is often easier to parse than reverse-engineering the graphQL schema, because it has most of the same data and doesn’t require auth headers.

import httpx
from bs4 import BeautifulSoup
import json

def extract_next_data(url: str, headers: dict) -> dict:
    r = httpx.get(url, headers=headers, follow_redirects=True)
    soup = BeautifulSoup(r.text, "html.parser")
    tag = soup.find("script", id="__NEXT_DATA__")
    if not tag:
        return {}
    return json.loads(tag.string)

headers = {
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
    "accept-language": "en-US,en;q=0.9",
}
data = extract_next_data("https://www.doordash.com/store/mcdonalds-new-york-12345/", headers)

the __NEXT_DATA__ path to menu items is props.pageProps.storeMenuProps.menuBook.categories[]. items sit inside each category with price (in cents), name, description, and imageUrl.

anti-bot defenses you’ll actually hit

DoorDash runs Cloudflare plus an internal bot management layer that fingerprints TLS JA3/JA4, HTTP/2 settings, and browser behavior. at moderate scale (200+ requests/hour from one IP) you’ll see:

• 403 Forbidden with a Cloudflare challenge page
• 429 Too Many Requests on the graphQL endpoint
• silent 200 responses returning empty menu data (the sneaky one)
• captcha interstitials on restaurant listing pages

rotating proxies help, but the fingerprinting layer means bare requests from httpx or requests get flagged even with valid cookies. you need TLS fingerprinting to match a real browser’s Client Hello. curl-cffi is the standard fix in 2026 — it mimics the TLS handshake of Chrome or Safari.

from curl_cffi import requests as cffi_requests

r = cffi_requests.get(
    "https://www.doordash.com/store/some-restaurant-99999/",
    impersonate="chrome120",
    headers=headers,
)

for serious scale, headless browsers (Playwright with stealth) or commercial scraping APIs handle the fingerprinting for you. similar defenses show up across the food delivery space: if you’re also pulling from Grubhub, the approach in How to Scrape Grubhub Menu Data Across Cities (2026) covers that platform’s specific quirks in more detail.

scraping at scale: proxies, rate limits, and infrastructure

a single residential IP running 50 requests/hour stays under the radar for light single-restaurant monitoring. anything broader needs IP rotation.

approach	cost	scale	detection risk
datacenter proxies	~$1-3/GB	high throughput	high (easy fingerprint)
residential proxies	~$5-15/GB	medium-high	medium
mobile proxies	~$15-40/GB	medium	low (best for auth’d flows)
scraping API (Apify, ScraperAPI, Brightdata)	~$1-5/1000 req	elastic	low (managed for you)

for menu data without login, residential proxies at 1 request per 3-5 seconds per IP are stable. if you need to scrape authenticated cart and pricing flows (DoorDash shows different prices with DashPass vs without), mobile proxies are more reliable because they share the same IP type the DoorDash mobile app actually uses.

session management matters too. DoorDash sets dd_access_token and dd_refresh_token cookies on login. if you’re scraping at scale with real accounts, rotate accounts and cookies together, not just IPs. a valid account with a stale cookie from a different IP triggers a re-auth challenge.

Uber Eats has a similar account-cookie coupling but is generally more tolerant of IP switches. there’s a full breakdown in How to Scrape Uber Eats Restaurant Listings at Scale (2026) if you’re running a multi-platform pipeline.

structuring the output: menus, modifiers, and pricing

raw DoorDash menu data is nested. a basic burger listing can have 4 levels of modifier groups (size, toppings, sauces, add-ons), and each modifier has its own price delta. flattening this into a usable schema takes some thought upfront or you’ll regret it later.

a clean schema for analysis looks like this:

• restaurant_id (DoorDash internal store ID)
• restaurant_name
• scraped_at (unix timestamp)
• item_id, item_name, category
• base_price_cents
• modifier_group_name, modifier_name, modifier_price_delta_cents
• item_calories, item_image_url

steps to build a clean pipeline:

1. fetch restaurant listing pages or search results to collect store IDs and slugs
2. for each store ID, fetch the menu page or graphQL endpoint
3. extract __NEXT_DATA__ or parse the graphQL response
4. flatten the nested modifier structure into rows (one row per item-modifier combo)
5. write to postgres or parquet with scraped_at for time-series tracking
6. schedule via Airflow or a simple cron every 6-24 hours depending on refresh needs

DoorDash does update prices dynamically — surge pricing on busy nights is documented — so if you’re doing price intelligence the timestamp matters more than most fields.

if your scope extends beyond the US market, How to Scrape Deliveroo Restaurant Menus UK + EU (2026) and How to Scrape Foodpanda Menu Data Asia + EU (2026) cover the regional equivalents with their own anti-bot quirks.

legal and rate-limit considerations

DoorDash’s terms of service prohibit scraping. enforcement has been inconsistent, but the hiQ vs LinkedIn precedent and its follow-on cases suggest that scraping publicly visiable data is defensible for research under the CFAA in the US. that said, this isn’t legal advice, and commercial use cases should run the ToS language by a lawyer first.

practically: DoorDash doesn’t gate its menus behind a login (you can see prices without an account). scraping publicly visible pages for research, price monitoring, or competitive analysis sits in a different risk bucket than scraping behind auth — similar to how How to Scrape Temu Product Data and Pricing in 2026 (Anti-Bot Guide) frames public product pricing.

and slow down. there’s no reason to hit 100 requests per second when 2-5 is fine for most use cases. fast and aggressive gets your proxy pool flagged and your costs up. slow and steady keeps you out of their radar entirely.

Bottom line

for most menu scraping needs, the __NEXT_DATA__ approach with curl-cffi and residential proxy rotation is the fastest path to production-ready data. if you need DashPass pricing or modifier-level accuracy at high volume, the graphQL endpoint with proper session management is worth the extra setup. DRT covers the full stack of food delivery scraping across platforms — check the sibling guides above if your pipeline spans more than one market.

How to Scrape FanDuel Odds and Lines in 2026

FanDuel is one of the hardest sportsbooks to scrape programmatically in 2026. Their odds pages load dynamically via a private REST API, they rotate Cloudflare JS challenges, and their IP detection is aggressive — geo-restricted by state on top of that. But the data is valuable: real-time lines on NFL, NBA, MLB, NHL, PGA, and dozens of other markets, often updating every few seconds during live games. This guide covers what actually works, what will get you blocked fast, and where the engineering tradeoffs land.

How FanDuel’s odds delivery actually works

FanDuel doesn’t serve odds as static HTML. The page shell loads, then the browser fires a series of fetch calls to https://sbapi.fanduel.com/api/.... Most of the useful data comes from two undocumented endpoints:

• GET /api/event-page?eventId=... returns full market and runner data for a single game
• GET /api/content-managed-page?page=SPORT_HOME&... returns lobby odds for a sport’s main page

Both return JSON. No API schema is published, but the response shape has been stable since late 2024.

The complication: every request requires a valid X-Auth-Token header. FanDuel generates this client-side via a fingerprinting flow that varies by state. You can’t replicate it with a static string — you either harvest it from a live browser session or reverse the generation logic, which is obfuscated and changes periodically.

Extracting the auth token and session cookies

The most reliable method is headless browser token extraction. Launch Playwright, navigate to the target state URL (e.g., fanduel.com/pennsylvania), wait for network idle, and intercept outgoing XHR calls to capture the token.

from playwright.sync_api import sync_playwright

def get_fanduel_token(state_url: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        ctx = browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
        )
        page = ctx.new_page()
        captured = {}

        def handle_request(req):
            if "sbapi.fanduel.com" in req.url:
                token = req.headers.get("x-auth-token")
                if token:
                    captured["token"] = token
                    captured["cookies"] = ctx.cookies()

        page.on("request", handle_request)
        page.goto(state_url, wait_until="networkidle")
        browser.close()
        return captured

Token lifetime is roughly 30 to 60 minutes. Rotate fresh sessions before expiry rather than refreshing mid-scrape — the silent failure mode is a 200 response with empty markets[], not a clean 401.

One thing to watch: FanDuel geo-gates their API by actual IP location, not just the URL slug. Hitting fanduel.com/new-jersey from a California IP gets a 403 or a redirect to a blocked-state page. You need a residential or mobile proxy in the target state. This is the same geo-blocking mechanic that drives up proxy costs in scraping BetMGM lines across states — budget for it before you start.

Parsing the odds response

Once you have a valid token and matching cookies, the API calls are straightforward. The event-page endpoint returns a nested structure: markets[] containing runners[], each with a winRunnerOdds.americanDisplayOdds.americanOdds field (deeply nested, yes).

Key fields per runner:

• runnerId — stable identifier for a team or player outcome
• runnerName — human-readable label
• americanOdds — moneyline or spread price
• handicap — the spread value (null for moneylines)
• marketType — e.g., MATCH_ODDS, ASIAN_HANDICAP, TOTAL_SCORE
• inPlay — boolean, critical if you’re tracking live odds

For spread and totals markets, handicap carries the line number. Cross-check marketType before assuming structure — FanDuel sometimes nests alternate lines inside the same market object, which will corrupt your data if you parse it naively.

If you’re building a cross-book model, you’ll want to pull from sharper books too. The approach in how to scrape Pinnacle sports lines for sharp models is a solid reference: Pinnacle’s API is more permissive and serves as a useful no-vig baseline for line movement analysis.

Anti-bot detection and what actually bypasses it

FanDuel runs Cloudflare with JS challenge mode on most state subdomains. Headless Chromium without stealth patches fails the challenge in roughly 80% of attempts as of early 2026.

Minimum viable bypass stack, in order:

1. Use playwright-stealth or puppeteer-extra-plugin-stealth to patch navigator properties
2. Disable WebDriver flags (--disable-blink-features=AutomationControlled)
3. Run a real Chrome binary rather than bundled Chromium where possible
4. Route through a residential proxy with a matching state IP
5. Add realistic mouse movement and short page interaction before the token extraction

FanDuel’s detection is noticeably more aggresive than DraftKings. If you’re already scraping that book, the guide on how to scrape DraftKings odds and lines covers similar stealth patterns — but FanDuel requires stricter fingerprint consistency across the full session (same IP, same user agent, same viewport from page load through API calls).

For scale above 20 to 30 concurrent sessions, you’ll burn through residential proxies fast. Mobile proxies on a rotating pool help with detection rates but cost more. That tradeoff is real and there’s no clean answer.

Bet365’s session architecture has similar complexity. The rotation and geo-routing patterns in how to scrape Bet365 odds around the world apply here too, particularly if you’re running across multiple state jurisdictions simultaneously.

Provider and approach comparison

Method	Token required	Geo-restriction	Scale	Relative cost
Headless browser (Playwright)	Auto-captured	Yes, state IP	Low-medium	Medium
Reverse-engineered direct API	Manual extraction	Yes, state IP	High	Low (after setup)
Third-party odds API (OddsAPI, Sportradar)	No	No	High	High
Managed scraping service (Bright Data, ScrapeHero)	No	Handled	High	Very high

For one-off research or model inputs, a third-party odds API is worth it. OddsAPI covers FanDuel lines and costs under $100/month at moderate volume. It doesn’t expose market depth or alternate lines, so if you need that granularity you’re back to direct access. Sportradar has full coverage but pricing is enterprise-grade.

For teams building internal tooling that needs raw market data without an intermediary, the headless extraction approach scales reasonably to around 5 to 10 state markets before operational complexity gets painful. Beyond that, the time cost of maintaining session rotation and stealth patches usually justifies a managed service.

If your scraping pipeline spans data categories beyond sports, it’s worth studying how G2 and Capterra scraping pipelines handle rate limiting and token rotation at scale — the underlying queue and retry architecture transfers well to high-frequency odds collection.

Common error codes and what they mean

Error	Likely cause	Fix
403 Forbidden	IP geo-mismatch or Cloudflare block	Rotate proxy, re-establish session
401 Unauthorized	Token expired or missing header	Re-extract from fresh browser session
429 Too Many Requests	Rate limit hit (~20 req/min per IP)	Back off 60s, slow cadence
503 Service Unavailable	High-traffic game event surge	Retry with exponential backoff
200 with empty markets[]	Stale session (silent failure)	Log market count, alert on zero

The silent 200 with empty markets is the one that burns you in production. Add a check on len(response["markets"]) per response and treat zero as a session-expired signal.

Bottom line

FanDuel is scrappable in 2026 but it takes real infrastructure: headless stealth browsers, state-matched residential proxies, and solid session rotation. If you just need odds for model inputs or comparison tools, OddsAPI covers FanDuel and saves weeks of engineering overhead. If you need raw market depth and alternate lines, the direct approach outlined here is the path. DRT covers the full sportsbook scraping stack — from FanDuel and DraftKings through international books like Bet365 and Pinnacle — so check back as these APIs and detection layers evolve.

How to Scrape Wix and Squarespace Stores in 2026

Scraping Wix and Squarespace stores is harder than most ecommerce targets, not because they have better bot protection, but because neither platform exposes a clean public API for product data. what you get instead is heavily client-rendered HTML, proprietary JSON blobs buried inside <script> tags, and JavaScript-dependent pagination that breaks naive scrapers on the first request. this guide covers what actually works in 2026, with platform-specific patterns, tool picks, and honest tradeoffs.

how Wix and Squarespace serve product data

both platforms render product catalogs via JavaScript frameworks, not server-side HTML. that single fact drives every decision you’ll make downstream.

Wix (now Wix Studio for most new stores) loads product data through its internal _api/wix-ecommerce-storefront-web/api endpoint. the JSON response is structured and predictable once you find it. Wix also injects a window.__VIEWER_MODEL__ object into the page source on many storefronts — catalog state, no browser render needed.

Squarespace uses its own Commerce API (/api/2/commerce/) internally. product collections are served as JSON at /api/2/commerce/products with pagination via cursor. the HTML source usually includes a Static.SQUARESPACE_CONTEXT JSON block with store metadata, active collection IDs, and sometimes a partial product list.

this is a different situation from How to Scrape Shopify Stores at Scale 2026 (Without Getting Blocked), where /products.json is a public, documented endpoint anyone can hit. Wix and Squarespace require reverse-engineering internal APIs. there are no docs.

fingerprinting the platform before you write a single line

check which platform you’re dealing with first. they look similar from the outside and the scraper for one won’t work on the other.

signal	Wix	Squarespace
HTTP response header	X-Wix-Request-Id present	X-ServedBy: squarespace
HTML source	window.rendererModel or __VIEWER_MODEL__	Static.SQUARESPACE_CONTEXT block
asset CDN	static.parastorage.com	static1.squarespace.com
robots.txt	disallows on /_api/	usually /api/ blocked

a curl request to the root URL and a quick grep takes under two seconds. more reliable than URL guessing, more reliable than favicon matching. if you’re running a multi-platform pipeline that also handles Magento or BigCommerce, the detection approaches in How to Scrape Magento Stores in 2026: API and HTML Patterns and How to Scrape BigCommerce Stores Programmatically (2026) follow the same logic.

scraping Wix stores

the viewer model path

fastest option: parse window.__VIEWER_MODEL__ directly from the HTML. no browser, no JS execution.

import httpx, re, json

def fetch_wix_viewer_model(url: str) -> dict:
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }
    r = httpx.get(url, headers=headers, follow_redirects=True, timeout=15)
    match = re.search(
        r'window\.__VIEWER_MODEL__\s*=\s*(\{.+?\})(?=;\s*</script>)',
        r.text, re.DOTALL
    )
    if match:
        return json.loads(match.group(1))
    return {}

this works on roughly 60-70% of Wix storefronts. for the rest — newer Wix Studio builds mostly — you POST to _api/wix-ecommerce-storefront-web/api directly using the store’s metaSiteId. find it in the viewer model or the page source. responses are paginated JSON with products under a predictable key path.

things that will trip you up:

• Wix rate-limits by IP around 1 req/2s. rotate residential proxies after every 50-80 requests
• _api/ endpoints return 403 without X-Wix-Brand or with an unexpected Origin header. mirror these from a real browser session on the target store
• product variants are nested under productItems inside each product object, not at the top level. easy to miss

Wix anti-bot

higher-traffic stores run Imperva (Incapsula). you’ll hit a JS challenge page. Playwright with rebrowser-patches plus rotating residential proxies clears it reliably. pure httpx won’t — don’t try to solve Imperva challenges in a pure HTTP client, it’s a time sink.

scraping Squarespace stores

Squarespace’s internal Commerce API is the easier of the two. products paginate at /api/2/commerce/products with a cursor param. the sequence:

1. GET /api/2/commerce/products?per_page=200 — first page plus pagination.nextPageCursor
2. GET /api/2/commerce/products?per_page=200&cursor=<nextPageCursor> — repeat until pagination.hasNextPage is false
3. parse items[] from each response — each has variants, pricing, images, categories

the per_page cap is 200. most stores have under 2,000 products, so you’re looking at maybe 1-10 requests per store. no auth required on public storefronts. it’s almost too easy.

password-protected stores redirect /api/2/ calls to the password page. check Content-Type on the response. JSON means open, HTML means gated. skip and move on.

but unlike the pattern-matching grunt work needed for How to Scrape WooCommerce Stores 2026: Pattern Recognition Approach, Squarespace gives you a consistent API surface regardless of theme. you write the scraper once and it works on every store.

proxy and rate-limit strategy

both platforms use Cloudflare CDN for static assets, but API traffic runs through different stacks. from testing in 2026:

• Wix: soft rate-limit around 1 req/2s per IP, 429 with retry-after
• Squarespace: more lenient, roughly 5-10 req/s per IP before a temporary block

for Wix, residential proxies are not optional for any serious run. datacenter IPs get blocked at the Imperva layer before they ever reach product data. for Squarespace, datacenter proxies from Bright Data or Oxylabs work fine on most stores — which cuts cost a lot if you’re doing bulk collection.

rules that hold up in practice:

• rotate IP every 50 Wix _api/ requests, or immediately on any 403
• rotate every 200 Squarespace /api/2/commerce/ requests, or on any 429
• keep User-Agent and Accept-Language consistent within a session. rotating headers independently from IPs creates a fingerprint mismatch and triggers blocks faster than the rate limit would

common failure modes

• empty items[] on Squarespace: store uses a non-default catalog structure. check Static.SQUARESPACE_CONTEXT for activePageCollections, swap in the right collection ID
• Wix viewer model missing products: the store runs on Wix Blocks. fall back to Playwright and intercept XHR to _api/wix-ecommerce-storefront-web/
• 403 on Wix _api/: missing X-Wix-Brand or wrong Referer. copy headers from a live browser session on that exact store, not from a different Wix site
• Squarespace returns HTML not JSON: password-protected or in maintenence mode. skip it
• Wix pagination stops early: metaSiteId mismatch. extract the ID from each target URL independently, never reuse across stores

bottom line

Squarespace is the easier target — consistent API, no browser required, minimal anti-bot. Wix takes more setup: viewer model extraction, Playwright fallback for JS-heavy stores, residential proxies if Imperva shows up. if you’re building a multi-platform ecommerce scraper, validate on Squarespace first, then add Wix. dataresearchtools.com covers the full ecommerce scraping stack — the same reverse-engineering approach here applies to any headless-first storefront you’ll run into in 2026.

Privacy-preserving scraping: differential privacy and federated learning

Privacy preserving scraping is no longer a research-only topic. The combination of regulatory pressure (GDPR, CCPA, PDPA, DPDP), customer expectations, and the maturation of privacy-engineering tooling made privacy-preserving techniques into production patterns through 2024-2026. For scraping operators, three techniques matter: differential privacy (release aggregate statistics with mathematical privacy guarantees), federated learning (train models without centralising raw data), and secure aggregation (combine inputs from multiple parties without any single party seeing raw individual data). Each has matured into accessible tooling, each has real production use cases for scraped data, and each shifts the compliance and competitive picture in important ways. This guide walks through what each technique actually does, where they fit in scraping pipelines, the production tooling in 2026, and a practical adoption roadmap.

The audience is the data engineer, ML lead, or compliance partner whose scraping pipeline produces sensitive aggregates and who wants to know which privacy-engineering techniques actually fit the use case.

Why privacy-preserving techniques matter for scraping

Three reasons.

First, regulatory pressure. GDPR Article 25 (privacy by design), CCPA’s data minimisation principle, PDPA’s protection obligation, and DPDP’s purpose limitation all reward techniques that reduce the personal-data exposure surface. A scraper using differential privacy to publish aggregates faces materially less regulatory risk than one publishing raw records.

Second, customer trust. Enterprise customers in regulated industries (finance, healthcare, government) increasingly require privacy-preserving outputs. A vendor that can prove differential privacy, federated learning, or secure aggregation gets the deal a vendor that cannot does not.

Third, competitive differentiation. The 2025-2026 wave of AI-training fines (covered in fair use and copyright for AI training data) put pressure on raw-data resellers. Vendors who pivot to privacy-preserving outputs are better positioned.

For the broader compliance picture, see the GDPR scraping compliance guide and the personal vs public data scraping framework.

Differential privacy explained

Differential privacy (DP) is a mathematical guarantee about the privacy of an individual within an aggregate result. Formally: a query mechanism is epsilon-DP if changing or removing one individual’s data from the input changes the output’s probability distribution by at most a factor of e^epsilon. Smaller epsilon means stronger privacy; epsilon=0 would be perfect privacy but useless results; typical production values are 0.1 to 5.

In practice, DP is implemented by adding calibrated noise to query results. The noise is calibrated to the query’s sensitivity (how much one individual can change the result) and the privacy budget epsilon.

For scraping pipelines, DP is most useful when releasing aggregate statistics: counts, averages, distributions over a scraped corpus that contains personal data. A scraper that publishes “number of products in category X by region” can use DP to release the number with provable per-individual privacy.

A minimal DP implementation in Python using OpenDP:

from opendp.measurements import make_laplace
from opendp.transformations import make_count_distinct
from opendp.combinators import make_chain

def dp_count(values, epsilon: float = 1.0):
    transform = make_count_distinct(input_domain=...)
    measure = make_laplace(scale=1.0 / epsilon)
    mechanism = make_chain(transform, measure)
    return mechanism(values)

The key engineering work is sensitivity analysis (how much one record can change the output) and budget management (how to spend epsilon across multiple queries on the same data).

Federated learning explained

Federated learning (FL) trains a model across multiple data holders without centralising raw data. Each holder trains a local model on their data, sends model updates (gradients or weights) to a central aggregator, and the aggregator combines updates into a global model.

For scraping, FL becomes interesting in two scenarios:

1. Multiple scraping operations cooperate to train a shared model without sharing raw scraped data.
2. A scraping operator trains a model with edge clients (browsers, devices) without uploading raw data to central servers.

The 2026 production FL frameworks: Flower (open source, language-agnostic), TensorFlow Federated, PySyft (PyTorch-aligned). Production deployments still skew small (under 100 participants typically) but the tooling is mature.

A minimal Flower-based federated training loop:

import flwr as fl
from typing import List, Tuple

def evaluate(parameters):
    # Run global eval on held-out test set
    return loss, num_examples, {}

class FedAvgWithEval(fl.server.strategy.FedAvg):
    def evaluate(self, server_round: int, parameters):
        return evaluate(parameters)

fl.server.start_server(
    server_address="0.0.0.0:8080",
    config=fl.server.ServerConfig(num_rounds=10),
    strategy=FedAvgWithEval(min_fit_clients=3, min_evaluate_clients=3),
)

Each scraping client runs a parallel client process that fits the model on local data and reports updates.

Secure aggregation explained

Secure aggregation lets multiple parties combine numerical inputs (e.g., model gradients, statistics) such that the aggregator only sees the sum, not the individual inputs. The cryptographic technique is multi-party computation (MPC) or homomorphic encryption (HE).

For scraping, secure aggregation matters when multiple operators want to compute joint statistics (industry-wide aggregates, joint AI training) without revealing raw data to each other.

The 2026 production tooling: Google’s TFF Secure Aggregation, Meta’s CrypTen, Microsoft’s SEAL, OpenMined’s TenSEAL.

In practice, secure aggregation is operationally heavy and is usually paired with FL rather than used standalone.

Where each technique fits in scraping pipelines

Technique	Best fit	Production maturity	Compliance benefit
Differential privacy	Releasing aggregate stats from scraped personal data	High	GDPR Art 25, “publicly available” reframing
Federated learning	Multi-operator model training	Medium	Reduces raw data centralisation
Secure aggregation	Multi-operator statistics	Medium	Hides individual operator inputs
Zero-knowledge proofs	Proving properties of data without revealing data	Medium-low	Strong privacy claims
Synthetic data	Releasing useful approximations	High	Often paired with DP
K-anonymity / l-diversity	Anonymising individual records	Mature but limited	Older approach; weaker than DP

For most scraping operators, differential privacy is the highest-leverage starting point. It is mature, well-tooled, and fits the most common use case (publishing aggregates).

For the parallel discussion of how this overlays with verifiable credentials, see verifiable credentials and scraping.

Decision tree: which technique fits this scraping use case?

Q1: Are you publishing aggregate statistics from a personal-data corpus?
    ├── Yes -> Differential privacy.
    └── No  -> Q2
Q2: Are multiple parties contributing data to a shared model?
    ├── Yes -> Federated learning, optionally with secure aggregation.
    └── No  -> Q3
Q3: Do you need to prove a property of data without revealing the data?
    ├── Yes -> Zero-knowledge proofs.
    └── No  -> Q4
Q4: Do you need to release a useful dataset that approximates real data?
    ├── Yes -> Synthetic data generation, ideally with DP guarantees.
    └── No  -> Standard pipeline.

Comparison: DP vs FL vs SA

Dimension	Differential Privacy	Federated Learning	Secure Aggregation
Privacy unit	Individual record	Individual data holder	Individual contribution
Centralisation needed	Aggregator sees noisy result	Aggregator sees model updates	Aggregator sees only sum
Compute overhead	Low	High (multiple training rounds)	Very high (MPC)
Network overhead	Low	Medium	High
Production maturity	High	Medium	Medium-low
Tooling	OpenDP, Tumult Labs, Diffprivlib	Flower, TFF, PySyft	TFF SecAgg, CrypTen, SEAL
Best paired with	Aggregate publishing	Multi-party training	Federated learning

Differential privacy at scale: practical guidance

Production DP requires four practical disciplines.

First, sensitivity analysis. Calculate the maximum change one record can cause in your output. Bounded sums need clipping; unbounded queries (top-k, percentiles) need careful handling.

Second, privacy budget management. Each query consumes part of the budget. Track total epsilon spent per dataset; do not exceed the budget the privacy posture commits to.

Third, query composition. Multiple queries compose: epsilon_total ≤ sum of epsilon_per_query (basic composition) or with tighter bounds via advanced composition (Renyi DP, zCDP). Use libraries that handle composition automatically.

Fourth, public release versus internal use. Public DP releases need stronger budgets (small epsilon, generous noise). Internal DP releases (analyst-facing dashboards) can run higher epsilon.

The 2026 mature DP libraries:

Library	Maintainer	Strength
OpenDP	OpenDP project	Most rigorous; verified building blocks
Tumult Labs Analytics	Tumult Labs	Production-grade SQL-style interface
Google Differential Privacy	Google	Wide language support
Diffprivlib	IBM	scikit-learn-style API
OpenMined PyDP	OpenMined	Friendly Python API

For most scraping operators, OpenDP or Tumult Analytics are the production picks.

Federated learning at scale: practical guidance

Production FL works best when:

1. The data lives where it should not be centralised (edge devices, partner organisations, sovereign data).
2. The computation pattern is compatible with model-update aggregation (most modern ML training is).
3. The participants are stable enough to complete multi-round training.

The 2026 mature FL frameworks:

Framework	Maintainer	Strength
Flower	Adap	Language-agnostic, production deployments
TensorFlow Federated	Google	TF-aligned, strong simulation
PySyft	OpenMined	PyTorch-aligned, research-friendly
FedML	FedML, Inc.	Strong cross-platform
OpenFL	Intel	Cross-vendor, strong governance

For scraping use cases involving multiple operators, Flower has the most production-grade reference deployments.

Worked example: DP aggregate release of scraped product data

A scraping operator publishes weekly statistics about product availability across major retailers. The dataset contains seller identities, product details, and per-seller stockout events. Sellers are personal data when they are individuals (sole proprietors).

Without DP: the publication is a flat table of seller-level aggregates. Each row reveals one seller’s stockout rate, exposing potentially sensitive operational information.

With DP: the operator releases category-level aggregates with calibrated Laplace noise. The aggregate “Category X had Y stockouts last week” is published with epsilon = 1, providing a meaningful privacy guarantee while preserving the headline value of the data product.

from opendp.measurements import make_laplace
from opendp.transformations import make_count_by_categories

def category_stockouts(events, categories, epsilon: float = 1.0):
    counts = {c: 0 for c in categories}
    for e in events:
        if e["stockout"] and e["category"] in counts:
            counts[e["category"]] += 1
    sensitivity = 1
    measure = make_laplace(scale=sensitivity / epsilon)
    return {c: max(0, n + measure()) for c, n in counts.items()}

The result: a category-level publication with provable privacy at epsilon=1, defensible against regulator inquiry.

External references

The OpenDP framework is at opendp.org. The Flower federated learning framework is at flower.ai. The Google Differential Privacy library is at github.com/google/differential-privacy. Tumult Labs Analytics documentation is at docs.tmlt.dev. The IETF privacy preservation working group drafts are at datatracker.ietf.org.

Synthetic data: the adjacent technique

Synthetic data generation is sometimes paired with DP to produce releasable datasets. The pattern: train a generative model on real data with DP guarantees, then release samples from the generative model.

Production synthetic data tools in 2026: SDV (Synthetic Data Vault), Mostly AI, Gretel.ai, Tonic.ai. The maturity is high for tabular data, lower for unstructured (text, image) data.

For scraping operators, synthetic data is most useful when sharing scraped datasets with downstream customers who cannot directly handle raw personal data. The synthetic version preserves statistical properties while removing individual records.

Adoption roadmap

A 12-month roadmap for a scraping operator adopting privacy-preserving techniques:

Quarter	Deliverable
Q1	Sensitivity analysis on existing aggregate releases; budget framework
Q2	First DP release on internal dashboards; epsilon budget assigned
Q3	Public DP release for one product category; customer feedback
Q4	Evaluate FL or secure aggregation for multi-operator pilots

A team that completes Q1-Q3 has a defensible DP-based product. Q4 is the optionality for going further.

For the broader policy build, see building an ethics-first scraping policy.

Comparison: privacy-preserving outputs vs raw-data outputs

Output type	Compliance risk	Customer value	Engineering cost
Raw scraped data	High	Highest	Lowest
Pseudonymised data	High (still personal)	High	Low
K-anonymised data	Medium	Medium	Low (but limited)
DP aggregates	Low	Medium-high	Medium
FL-trained model	Low	Medium	High
Synthetic data	Low (with DP)	Medium	Medium-high

The risk-value tradeoff favours privacy-preserving outputs as compliance pressure rises. The 2026 trend is unmistakable: vendors moving up this table win deals.

FAQ

Is differential privacy production-ready in 2026?
Yes. Multiple mature libraries (OpenDP, Tumult, Google DP) have substantial production deployments.

What epsilon should I use?
Common values: 0.1 (very strong) for high-risk releases, 1-5 for typical releases, higher for internal-only. The right value depends on the threat model and sensitivity.

Can I use federated learning instead of centralising scraped data?
Sometimes, when the data sources are organisations willing to participate. For unilateral scraping, FL does not apply.

Does DP help with GDPR compliance?
Yes. DP-released aggregates that satisfy the EDPB’s anonymisation tests can move outside GDPR scope. Verify per release with counsel.

What about zero-knowledge proofs for scraping?
Niche but growing. Use cases include proving compliance properties to auditors without revealing the underlying data.

Extended privacy-preserving scraping analysis

Privacy-preserving scraping is the discipline of collecting only what is needed, in a form that minimises personal data exposure, with measurable safeguards. The 2026 toolkit consists of six techniques.

1. Differential privacy at aggregation. Add calibrated noise so individual records cannot be reconstructed.
2. Pseudonymisation at ingest. Replace direct identifiers with stable tokens.
3. K-anonymity at publication. Suppress or generalise records below the k threshold.
4. Federated processing. Compute on the source rather than centralising raw data.
5. Secure multi-party computation. Combine inputs from multiple parties without exposing them to each other.
6. Homomorphic encryption. Compute on encrypted data with the result decrypted at the end.

Each has costs (latency, accuracy, complexity) and benefits (compliance posture, breach minimisation).

Implementation pattern: differential privacy aggregation

import numpy as np

def laplace_mechanism(true_value, sensitivity, epsilon):
    scale = sensitivity / epsilon
    noise = np.random.laplace(0, scale)
    return true_value + noise

def dp_count(records, epsilon=1.0):
    sensitivity = 1.0
    true_count = len(records)
    return laplace_mechanism(true_count, sensitivity, epsilon)

def dp_mean(values, lower, upper, epsilon=1.0):
    clipped = np.clip(values, lower, upper)
    sensitivity = (upper - lower) / len(clipped)
    true_mean = np.mean(clipped)
    return laplace_mechanism(true_mean, sensitivity, epsilon)

Implementation pattern: pseudonymisation with key separation

import hmac
import hashlib

class Pseudonymiser:
    def __init__(self, key):
        self.key = key

    def tokenise(self, identifier):
        return hmac.new(self.key, identifier.encode(), hashlib.sha256).hexdigest()

    def rotate_key(self, new_key):
        self.key = new_key

Implementation pattern: k-anonymity check

from collections import Counter

def k_anonymous(records, quasi_identifiers, k=5):
    keys = [tuple(r.get(q) for q in quasi_identifiers) for r in records]
    counts = Counter(keys)
    return all(c >= k for c in counts.values())

def suppress_below_k(records, quasi_identifiers, k=5):
    keys = [tuple(r.get(q) for q in quasi_identifiers) for r in records]
    counts = Counter(keys)
    return [r for r, key in zip(records, keys) if counts[key] >= k]

Federated processing pattern

def federated_aggregate(participants, query):
    partials = []
    for p in participants:
        partial = p.compute_local(query)
        partials.append(partial)
    return aggregate(partials)

def compute_local(query):
    result = run_query(query, local_data)
    return laplace_mechanism(result, sensitivity=1.0, epsilon=1.0)

Comparison: privacy techniques tradeoffs

Technique	Privacy strength	Accuracy cost	Compute cost	Best for
Pseudonymisation	Moderate	None	Low	All ingest
K-anonymity	Moderate	Suppression	Low	Publication
Differential privacy	Strong	Noise	Low to moderate	Aggregates
Federated	Strong	None	High coordination	Multi-party
Secure MPC	Strongest	None	Very high	Sensitive joins
Homomorphic encryption	Strongest	None	Highest	Computed-on-encrypted

Operational pattern: privacy budget tracking

For DP systems, track the cumulative epsilon spent per data subject across queries. When the budget is exhausted, stop answering queries about that subject.

class PrivacyBudget:
    def __init__(self, total_epsilon):
        self.total = total_epsilon
        self.spent = {}

    def spend(self, subject_id, epsilon):
        current = self.spent.get(subject_id, 0)
        if current + epsilon > self.total:
            return False
        self.spent[subject_id] = current + epsilon
        return True

Additional FAQ

Does pseudonymisation remove GDPR scope?
No. Pseudonymous data remains personal data. Only true anonymisation removes scope.

What epsilon is acceptable for DP?
Common guidance is 0.1-1.0 for strong privacy, 1-10 for moderate. The choice is workload-specific.

Is federated learning the same as federated processing?
Federated learning is a special case for ML training. Federated processing is the broader umbrella for any compute-on-source workflow.

How does this interact with AI training?
DP-SGD and PATE are training-time techniques that bound the privacy leakage of the trained model. They complement but do not replace data-collection privacy controls.

The data minimisation principle in practice

Data minimisation is a foundational privacy principle that appears in GDPR Article 5(1)(c), CCPA’s purpose limitation provisions, PDPA’s necessity test, and DPDP’s purpose specification requirements. The principle is straightforward in theory and demanding in practice.

A scraper applying data minimisation collects only the fields necessary for the stated purpose. If the purpose is competitor pricing analysis, the scraper collects product names and prices, not customer reviews. If the purpose is sentiment analysis, the scraper collects review text but not reviewer identifiers. The minimisation is per-field, per-record, and per-purpose.

The 2026 implementation pattern starts at the schema level. The scraper defines a target schema containing only the necessary fields. The fetch and parse logic populates only those fields. Additional content available on the page is discarded.

A common failure mode is over-collection at the fetch step followed by post-fetch filtering. The over-collected data may be retained in logs, caches, or backups even if it is not loaded into the primary store. The 2026 best practice is to filter as early in the pipeline as possible, ideally at the fetcher.

The de-identification spectrum

De-identification is not a binary state. The spectrum runs from raw identified data through pseudonymisation, masking, generalisation, suppression, and finally to true anonymisation. Each step strengthens privacy and weakens utility.

Pseudonymisation replaces direct identifiers with stable tokens. The tokens enable record linkage without revealing the original identifiers. GDPR Recital 26 explicitly notes that pseudonymous data remains personal data because re-identification is possible.

Masking replaces parts of identifiers with placeholders. An email address might be masked to j***@example.com. Masking reduces direct identification while preserving some utility for analysis.

Generalisation replaces specific values with broader categories. A specific age (34) becomes an age range (30-39). A specific city becomes a region. Generalisation is a primary tool in k-anonymity.

Suppression removes records or fields entirely. Records that cannot be sufficiently anonymised are dropped. Suppression is the most conservative option and often the most defensible.

True anonymisation removes the personal data classification. Under GDPR’s strict reading, true anonymisation requires that re-identification be impossible by any means reasonably likely. Most scraping pipelines do not achieve true anonymisation. The pragmatic operational target is strong pseudonymisation plus minimisation plus aggregation.

Differential privacy in practice

Differential privacy is the mathematically rigorous framework for releasing statistics about a population without revealing individuals. The framework introduces calibrated noise to query results, with the noise calibrated by an epsilon parameter that bounds the privacy loss.

For scrapers DP applies most naturally to aggregate releases. A scraper that publishes a count of records, an average, or a distribution can apply DP noise to the published statistic. Individual records remain protected.

The epsilon parameter is the central tuning knob. Lower epsilon means stronger privacy and noisier results. Higher epsilon means weaker privacy and more accurate results. The 2026 best practice is to choose epsilon per use case, typically in the 0.1-1.0 range for strong privacy and 1-10 range for moderate privacy.

A practical complication is the privacy budget. Each query consumes some epsilon. Repeated queries on the same dataset accumulate epsilon, and the cumulative epsilon bounds the total privacy loss. A privacy-aware system tracks the cumulative epsilon and stops responding when the budget is exhausted.

Federated learning and federated processing

Federated approaches keep raw data at the source and centralise only derived signals. The patterns differ in what is centralised.

Federated learning trains a model across many participants, with each participant computing gradient updates locally and the central coordinator aggregating the updates. The raw data never leaves the participant. The model captures the aggregate signal.

Federated processing is broader. It covers any computation where the input is at the source and only the result moves to the coordinator. Federated SQL queries, federated analytics, and federated search all qualify.

For scrapers federated approaches are useful when the data sources are willing to compute locally but unwilling to share raw data. A consortium of publishers might agree to a federated analytics arrangement that yields industry statistics without exposing individual subscriber data.

The 2026 federated toolkit includes Google’s Federated Learning of Cohorts (FLoC, deprecated and replaced by Topics), Mozilla’s Distributed Aggregation Protocol, the OpenMined frameworks, and several research-grade systems. Production deployments are growing in healthcare and finance, where the privacy stakes are highest.

Next steps

The fastest first step is to identify one aggregate release in your current pipeline and pilot a DP version using OpenDP or Tumult. The engineering effort is bounded; the compliance and customer-trust upside is real. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the personal vs public data framework.

This guide is informational, not engineering or legal advice.

How to scrape Flipkart India product data

Scrape Flipkart India effectively in 2026 and you have access to one of the two dominant ecommerce platforms in the largest English-speaking online market in the world. Flipkart serves over 200 million registered users in India, indexes hundreds of millions of SKUs across electronics, fashion, grocery, and home goods, and runs the Big Billion Days sales that move tens of millions of units in a single week. For brand managers tracking pricing, agencies running competitive intelligence, or product teams sizing demand in India, Flipkart and Amazon India together cover the market.

This guide walks the full Flipkart India scraping stack: which endpoints to hit, how to handle Walmart-owned Flipkart’s bot defenses, how to manage Indian mobile carrier proxies and INR pricing, and how to keep extraction quality high across the site’s wide category structure. Working Python and Playwright code throughout.

What Flipkart India exposes

Surface	URL pattern	Best for
Product detail page	flipkart.com/{slug}/p/{pid}	Full extraction with reviews and Q&A
Search results	flipkart.com/search?q={query}	Discovery
Internal API	flipkart.com/api/3/page/fetch	High-throughput product extraction
Category landing	flipkart.com/{category}	Category sweeps

The internal /api/3/page/fetch endpoint returns clean JSON used by the React frontend. It requires a CSRF token and a session cookie, which a browser session provides for free.

Anti-bot defenses

Flipkart uses a custom bot defense stack:

1. PerimeterX (now HUMAN) on the public web pages
2. Aggressive IP reputation; data center IPs blocked or heavily challenged
3. Custom request signing on internal API endpoints
4. Header-based fingerprinting (specific Accept-Language and User-Agent expected)

The recommended path: Indian mobile carrier IPs (Jio, Airtel, Vi), real Chromium with a mobile user agent and Indian locale, and patient throttling.

Working browser-based scraper

import asyncio
import json
import re
from playwright.async_api import async_playwright

async def scrape_flipkart_in(product_url: str, proxy: dict | None = None) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy=proxy,
            args=["--disable-blink-features=AutomationControlled"],
        )
        ctx = await browser.new_context(
            user_agent="Mozilla/5.0 (Linux; Android 13; CPH2483) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36",
            locale="en-IN",
            timezone_id="Asia/Kolkata",
            extra_http_headers={"Accept-Language": "en-IN,en;q=0.9,hi;q=0.8"},
            viewport={"width": 412, "height": 915},
        )
        page = await ctx.new_page()

        api_payload = {}
        async def on_response(resp):
            if "/api/3/page/fetch" in resp.url:
                try:
                    api_payload["data"] = await resp.json()
                except Exception:
                    pass
        page.on("response", on_response)

        await page.goto(product_url, wait_until="networkidle", timeout=45000)
        html = await page.content()
        await browser.close()

    if api_payload.get("data"):
        return _normalize_flipkart_api(api_payload["data"])
    return _parse_flipkart_html(html, product_url)

def _normalize_flipkart_api(data: dict) -> dict:
    slots = data.get("RESPONSE", {}).get("slots", [])
    product = next((s for s in slots if s.get("widget", {}).get("type") == "PRODUCT_SUMMARY"), {})
    if not product:
        return {"error": "no_product_summary"}
    info = product.get("widget", {}).get("data", {}).get("productSummary", {})
    return {
        "title": info.get("title"),
        "brand": info.get("brand"),
        "price_inr": info.get("pricing", {}).get("finalPrice", {}).get("value"),
        "original_price_inr": info.get("pricing", {}).get("mrp", {}).get("value"),
        "discount_percent": info.get("pricing", {}).get("totalDiscount"),
        "rating": info.get("ratingsAndReviews", {}).get("rating", {}).get("average"),
        "review_count": info.get("ratingsAndReviews", {}).get("rating", {}).get("count"),
        "in_stock": info.get("availability", {}).get("status") == "IN_STOCK",
    }

def _parse_flipkart_html(html: str, url: str) -> dict:
    # fallback to BeautifulSoup-based parsing if API intercept fails
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, "html.parser")
    title = soup.select_one("span.B_NuCI") or soup.select_one("h1 span")
    price = soup.select_one("div._30jeq3._16Jk6d") or soup.select_one("div._30jeq3")
    return {
        "title": title.text.strip() if title else None,
        "price_inr": _parse_inr(price.text) if price else None,
        "url": url,
    }

def _parse_inr(s: str) -> float:
    return float(re.sub(r"[^\d]", "", s) or 0)

asyncio.run(scrape_flipkart_in("https://www.flipkart.com/example-product/p/itm123456"))

Mobile user agent and the Indian locale matter. Flipkart serves a mobile-optimized API path with cleaner JSON to mobile clients.

Indian Rupee price handling

INR uses the symbol ₹ and uses comma in the Indian numbering system: ₹1,23,456.78 (lakh-crore grouping, not Western thousands). The API returns plain numbers, so this is only a display concern. For parsing scraped UI text:

import re

def parse_inr(s: str) -> float:
    # handle both Western (1,234,567) and Indian (12,34,567) grouping
    return float(re.sub(r"[^\d.]", "", s) or 0)

USD conversion in 2026 hovers around 84-87 INR per USD. Always store the raw INR; convert only for display.

Multi-language considerations

Flipkart serves predominantly English content nationwide. Some product titles include Hindi or regional script (Tamil, Telugu, Bengali, Malayalam) for grocery and traditional goods. UTF-8 storage handles all of them.

Search queries respect English. Hindi search queries work in Devanagari script but coverage is sparse outside the largest categories.

Mobile proxy rotation

Indian mobile carrier IPs (Jio, Airtel, Vi/Vodafone Idea) are the cleanest source. Indian residential IPs work for low volume; mobile is required for sustained throughput.

import random

IN_MOBILE_PROXIES = [
    {"server": "socks5://us:pw@in-jio-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@in-airtel-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@in-vi-1.proxy.example.com:1080"},
]

async def scrape_with_proxy(url: str):
    proxy = random.choice(IN_MOBILE_PROXIES)
    return await scrape_flipkart_in(url, proxy=proxy)

For mobile proxy strategy, see best mobile proxy providers 2026.

Discovering product URLs

Flipkart sitemaps are split by category:

import httpx
import xml.etree.ElementTree as ET

async def list_flipkart_sitemaps() -> list[str]:
    sitemap_index = "https://www.flipkart.com/sitemap.xml"
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.get(sitemap_index, headers={"User-Agent": "Mozilla/5.0"})
        root = ET.fromstring(r.text)
        ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
        return [s.find("sm:loc", ns).text for s in root.findall("sm:sitemap", ns)]

For category-driven discovery, browse search results with paginated queries:

async def search_flipkart(query: str, page: int = 1) -> list[dict]:
    url = f"https://www.flipkart.com/search?q={query}&page={page}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="en-IN")
        pg = await ctx.new_page()
        await pg.goto(url, wait_until="networkidle")
        items = await pg.locator("a[href*='/p/']").all()
        results = []
        for item in items[:60]:
            href = await item.get_attribute("href")
            title = await item.text_content()
            if href:
                results.append({"url": f"https://www.flipkart.com{href}", "title": (title or "").strip()})
        await browser.close()
    return results

Comparison to other Indian markets

Market	Bot defense	Volume	Mobile proxy required
Flipkart India	High	Largest with Amazon	Yes
Amazon India	High	Largest with Flipkart	Yes
Myntra (fashion)	High	Largest fashion	Yes
Meesho	Medium	Large	Recommended
Snapdeal	Medium	Smaller	Optional
JioMart	High	Growing	Recommended
BigBasket (grocery)	Medium	Medium	Recommended

For broader India coverage, Flipkart and Amazon India together cover roughly 80 percent of Indian online retail GMV.

Geographic IP pinning

Flipkart serves slightly different content based on detected IP geolocation. Delivery options, COD availability, and even some pricing tiers vary by city. For consistent scraping:

CITY_PROXY_POOLS = {
    "delhi": ["socks5://us:pw@in-jio-delhi-1...", "socks5://us:pw@in-jio-delhi-2..."],
    "mumbai": ["socks5://us:pw@in-jio-mumbai-1...", ...],
    "bangalore": ["socks5://us:pw@in-jio-bangalore-1...", ...],
}

async def scrape_for_city(url: str, city: str = "delhi"):
    proxy = {"server": random.choice(CITY_PROXY_POOLS[city])}
    return await scrape_flipkart_in(url, proxy=proxy)

For city-level price intelligence, sample the same product across multiple metro pools weekly.

Reviewer-level data and sentiment

For sentiment analysis, the review payload exposes:

Field	Use
rating (1-5)	Numeric sentiment
text	Long-form review text
helpful_count	Community endorsement
verified_buyer	Trust signal
images	Photo evidence (counterfeit detection)
date	Review recency

For brand intelligence, the verified_buyer flag is the most important. Reviews from non-verified buyers are roughly 4x more likely to be fake.

Cross-marketplace deduplication

For brand intelligence projects covering both Flipkart and Amazon India, deduplicating SKUs is non-trivial because each platform uses its own product ID system. The right approach:

1. Match by EAN/UPC barcode where present (often missing on Flipkart)
2. Fall back to fuzzy match on title + brand + key attributes
3. Use LLM-based similarity for ambiguous cases

async def fuzzy_match_skus(flipkart_item: dict, amazon_items: list[dict]) -> dict | None:
    # cheap embedding-based similarity, then validate top match with LLM
    candidates = embedding_search(flipkart_item["title"], [a["title"] for a in amazon_items], top_k=3)
    for c in candidates:
        if await llm_verify_match(flipkart_item, c):
            return c
    return None

Production patterns

Three patterns matter.

First, throttle conservatively. 1-2 requests per second per IP, longer pauses on first session. Flipkart’s challenge mechanism escalates fast under sustained traffic.

Second, intercept the page-fetch API. The API path returns cleaner JSON than parsing the React-rendered HTML. The interception pattern shown above is the production-grade approach.

Third, cache CSRF and session cookies. Pull them from a warm session at the start of a worker run, reuse for the worker lifetime, refresh on auth failure.

async def get_session_cookies():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="en-IN")
        pg = await ctx.new_page()
        await pg.goto("https://www.flipkart.com", wait_until="networkidle")
        cookies = await ctx.cookies()
        await browser.close()
        return cookies

Big Billion Days strategy

Flipkart’s flagship sales event runs early October and triples or quadruples site traffic. For brand intelligence during BBD:

Pre-BBD (1 week prior): snapshot all watched SKUs at high frequency to capture the baseline.

During BBD: switch to lighter polling on watched SKUs. Avoid scraping unrelated catalog data because the bot defense tightens.

Post-BBD (1 week after): resume normal scraping. Compare price trajectories to identify items that retained discounts versus those that snapped back to MRP.

Logging the entire BBD pricing graph for important SKUs is gold for next year’s pricing strategy work.

Storage schema

CREATE TABLE flipkart_in_products (
    id BIGSERIAL PRIMARY KEY,
    pid TEXT NOT NULL,
    url TEXT NOT NULL,
    title TEXT NOT NULL,
    brand TEXT,
    price_inr NUMERIC(12,2) NOT NULL,
    original_price_inr NUMERIC(12,2),
    discount_percent INTEGER,
    rating NUMERIC(3,2),
    review_count INTEGER,
    in_stock BOOLEAN NOT NULL,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    raw_jsonb JSONB,
    UNIQUE(pid)
);
CREATE INDEX idx_flipkart_extracted_at ON flipkart_in_products(extracted_at);
CREATE INDEX idx_flipkart_brand ON flipkart_in_products(brand);

Real benchmark numbers

A March 2026 production run, 10,000 Flipkart India products with the API capture pattern:

Metric	Value
Success rate	92%
Median latency per item	6.4 s
p99 latency	21 s
Cost per 1000 items	$19
PerimeterX challenge rate	6.1%
429 throttle rate	1.7%

PerimeterX challenges are the leading failure cause. With proper stealth and IP rotation, you can keep the rate under 7 percent.

Stealth fingerprint hardening

Out-of-the-box headless Chromium fails on Flipkart within roughly 60 requests per IP. Combine the AutomationControlled patch with realistic Indian mobile fingerprints:

context_init = """
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, 'languages', {get: () => ['en-IN', 'en', 'hi']});
Object.defineProperty(navigator, 'platform', {get: () => 'Linux armv8l'});
Object.defineProperty(screen, 'colorDepth', {get: () => 32});
window.chrome = {runtime: {}, app: {}};
"""

await ctx.add_init_script(context_init)

Combined with mobile IPs and warm sessions, success rates climb from roughly 60 percent to over 90 percent.

Cost expectations

10,000 Flipkart India products per month with Indian mobile proxies:

Component	Cost
Indian mobile proxy traffic (~3MB/page)	$90-$150
Browser compute	$40
LLM extraction (optional)	$30
Total	$160-$220

Indian mobile IPs are competitive in cost with Indonesian and Thai mobile pools.

Legal considerations

India’s Digital Personal Data Protection Act (DPDP Act, enforced from 2024) regulates personal data. Public commercial data (product listings, prices, seller-level data at city granularity) is not personal data.

The Flipkart terms of use prohibit automated access. Civil enforcement only; no criminal exposure for scraping public commercial data in India.

For deeper compliance reading, see our India DPDP Act for scrapers.

Variants and SKUs

Flipkart products often have variants (size, color, model). The API exposes them in the variantOptions and swatchOptions widgets. Variant-level pricing matters for competitive intelligence:

def parse_variants(api_data: dict) -> list[dict]:
    slots = api_data.get("RESPONSE", {}).get("slots", [])
    variant_widget = next((s for s in slots if s.get("widget", {}).get("type") == "VARIANT_OPTIONS"), None)
    if not variant_widget:
        return []
    options = variant_widget.get("widget", {}).get("data", {}).get("variantOptions", [])
    return [
        {"value": o.get("value"), "pid": o.get("productId"), "available": o.get("available")}
        for o in options
    ]

Reviews

Reviews are paginated client-side. Each page loads roughly 10 reviews:

async def scrape_reviews(product_url: str, max_pages: int = 5):
    reviews = []
    for page in range(1, max_pages + 1):
        url = f"{product_url}/product-reviews?page={page}"
        # browser-based fetch and parse review divs
        pass
    return reviews

Review text is personal commentary; reviewer names are personal data under DPDP. Strip both for any pipeline beyond aggregate ratings unless you have legal basis for retention.

Flipkart-specific data points

A few fields specific to Flipkart that other platforms do not expose:

fAssured flag: Flipkart’s quality and fast-shipping certification. Strong predictor of conversion and the Indian equivalent of Amazon’s Prime badge.

bankOffers: array of bank-specific discounts (HDFC, ICICI, SBI cashback offers). These can shave 10 to 15 percent off the headline price for cardholders.

exchangeOffer: trade-in pricing for old devices, common on phones, laptops, and televisions. Captures the effective post-trade price.

emiOptions: EMI (installment) terms, including no-cost EMI flag. EMI dominates large-ticket purchases in India.

def extract_flipkart_specific(api_data: dict) -> dict:
    info = _get_product_info(api_data)
    pricing = info.get("pricing", {})
    return {
        "f_assured": info.get("fAssured", False),
        "bank_offers_count": len(pricing.get("bankOffers", [])),
        "best_bank_discount_inr": max(
            (o.get("discount", {}).get("value", 0) for o in pricing.get("bankOffers", [])),
            default=0,
        ),
        "emi_starting_inr": pricing.get("emi", {}).get("startingValue"),
        "no_cost_emi": pricing.get("emi", {}).get("noCost", False),
    }

Q&A and Q&A sentiment

Flipkart has a buyer Q&A section that often contains questions other shoppers ask. The endpoint:

async def get_qna(pid: str, session_cookies: dict) -> list[dict]:
    url = f"https://www.flipkart.com/api/3/product/{pid}/questions"
    async with httpx.AsyncClient(cookies=session_cookies) as c:
        r = await c.get(url, headers={"User-Agent": "Mozilla/5.0 ..."})
        return r.json().get("questions", [])

For brand monitoring, scraping competitor Q&A reveals customer concerns that the brand could address in their own listings.

Indian ecommerce calendar

Indian ecommerce has unique peak periods that affect scraping load:

• Republic Day Sales (late January)
• Independence Day Sales (mid-August)
• Big Billion Days (early October, Flipkart’s flagship event)
• Diwali season (late October to mid-November)
• New Year sales (December to early January)

During peak windows, expect 3 to 5x normal load on Flipkart infrastructure plus more aggressive bot defense. Scale your IP pool by 2x and increase pacing margins. Big Billion Days specifically: pause non-critical scraping for the week.

AI-driven extraction fallback

For pages where API interception fails, fall through to LLM extraction:

async def scrape_with_fallback(url: str) -> dict:
    try:
        return await scrape_flipkart_in(url)
    except (NoAPIPayloadError, KeyError):
        html = await fetch_html(url)
        return await llm_extract_product(html, schema=PRODUCT_SCHEMA)

The LLM fallback runs at roughly 4x the cost per page but catches the cases where the deterministic path breaks.

Frequently asked questions

Can I use Flipkart’s official Affiliate API?
The Flipkart Affiliate program offers an API for affiliates with rate limits and category restrictions. Useful for affiliate marketers; less useful for general competitive intelligence.

Why does my scraper start failing during Big Billion Days?
Flipkart traffic spikes during sales events. The bot defense team also tightens during these periods. Pause aggressive scraping during the BBD week and resume after.

How do I scrape Flipkart Camera and similar specialty categories?
Same patterns. Specialty categories often have richer attribute data; capture the full attributes block as JSONB.

Does Flipkart support multiple sellers per product like Amazon?
Yes. The sellers block lists all sellers offering the same product, with their respective prices, ratings, and shipping options. Critical for brand intelligence on grey-market sellers.

Will residential IPs work?
For low volume, yes. For sustained scraping (tens of thousands of pages per day), Indian mobile IPs are required.

How do I scrape Flipkart Plus exclusive offers?
Flipkart Plus pricing requires a logged-in session with a Plus subscription. Saved storage state from a manual login enables this. Treat the credentials as sensitive.

Can I scrape Myntra (Flipkart-owned) with the same tooling?
Largely yes. Myntra uses similar PerimeterX defense and a similar React frontend. The API shapes differ but the patterns transfer.

Can I scrape Flipkart Wholesale (B2B)?
The Wholesale platform requires business registration. Public-facing pricing is limited; full catalogs are gated behind login.

How do I track price drops on a watch list of products?
Snapshot daily for the watch list, store in a price_history table, run a query for items where the latest price is at least 10 percent below a 7-day rolling average. Send alerts via Telegram or email.

Does Flipkart have variant-level reviews?
Yes. Reviews are tagged with the variant they were written about (size and color). Capture the variant tag for accurate variant-specific sentiment.

How do I detect out-of-stock pulse for inventory intelligence?
Track availability.status over time. Repeated OUT_OF_STOCK transitions correlate with sales velocity and demand intelligence.

Common production gotchas

A few patterns that cause issues in Flipkart scraping:

The PerimeterX cookie expires after 30 minutes of inactivity. Sessions need refresh more often than the cookie lifetime suggests.

Indian carrier IPs have higher latency (200 to 500 ms) than residential. Plan for slower per-page timing.

The Indian numbering system (lakh, crore) only appears in display, not in API. The API returns plain integers.

Flipkart occasionally rolls out region-specific UI experiments based on IP geolocation (Delhi vs Mumbai vs Bengaluru can see slightly different layouts). Pin the IP region for consistent scraping.

Affiliate URLs include tracking parameters that change. Strip them before storing canonical URLs.

The mobile site (m.flipkart.com) and desktop site return different DOMs and slightly different API shapes. Pick mobile for cleaner data and stick with it.

Cost optimization for Flipkart specifically

Three patterns specifically valuable for Flipkart:

Block image and font requests via Playwright route interception. Flipkart product pages load 5 to 7 MB of imagery by default. Blocking cuts proxy bandwidth by 75 percent.

Cache API responses by product_id. The same PID rarely has changing data within a 4-hour window outside of flash sales.

Use the API capture pattern over HTML parsing. The intercepted JSON contains structured data; HTML parsing is brittle as Flipkart frequently ships frontend updates.

Combined, these cut typical per-page cost from $0.040 to $0.018, more than half.

Compliance specifics for India

Beyond DPDP, Indian ecommerce data has a few specific regulations to consider:

The Consumer Protection (E-Commerce) Rules 2020 require seller information transparency. The data is public on Flipkart and not regulated for scraping.

The Information Technology Rules 2021 govern cybersecurity but exempt scraping of public commercial data.

For scraping personal reviewer data (names, profile photos), DPDP requires explicit consent which you do not have. Strip personal identifiers from any review data you persist.

For broader Asian ecommerce coverage, browse the ecommerce category.

Browser DevTools Protocol changes in 2026: scrapers’ impact

DevTools Protocol 2026 is undergoing the most significant set of changes in the protocol’s decade-long history. Chrome DevTools Protocol (CDP), the underlying transport for Puppeteer, Playwright, and most modern browser automation, is being progressively reshaped by three concurrent forces: tighter fingerprint surface restrictions, the convergence with the WebDriver BiDi standard, and the security hardening required to defend against the emerging class of agentic browsers. For scrapers, the implications are direct: the patterns that worked in 2023 are partially broken, the patterns that emerged in 2024 are being formalised, and the patterns that will work in 2027 are still being shaped. This guide walks through the protocol changes, the impact on common scraping libraries, the migration patterns that work, and a forward-looking posture for operators.

The audience is the scraping engineer or platform owner whose pipeline depends on browser automation via CDP, WebDriver, or higher-level frameworks built on them.

What CDP and WebDriver BiDi actually are

The Chrome DevTools Protocol is the JSON-RPC over WebSocket protocol that Chrome exposes for browser inspection and control. Originally designed for the Chrome DevTools UI, CDP became the foundation for headless and automated browsing because it gives complete programmatic control over the browser: navigation, page evaluation, network interception, screenshot capture, fingerprint manipulation.

WebDriver, defined by W3C, is the older standardised browser automation protocol. Selenium uses WebDriver. WebDriver BiDi is the next-generation WebDriver, designed to provide bidirectional event-driven control similar to CDP while maintaining the W3C standard’s portability across browsers.

The 2024-2026 trajectory is convergence: BiDi capabilities catching up with CDP, browsers exposing both, and automation frameworks supporting both with BiDi as the default for cross-browser work.

Protocol	Current state (mid-2026)	Browser support	Scraping use
CDP (Chrome)	Mature, evolving	Chrome, Edge, Brave (Chromium-based)	Most existing scrapers
WebDriver Classic	Stable	All major browsers	Selenium-based scrapers
WebDriver BiDi	Production with caveats	Chrome, Firefox, Safari (partial)	New cross-browser scrapers

The 2024-2026 changes that bite scrapers

Five concrete changes shape the scraping landscape.

Change one: Chrome’s fingerprint surface tightening. Chrome’s Privacy Sandbox initiative removed or restricted several fingerprintable APIs (User-Agent client hints reduction, narrowed font enumeration, randomised screen size in incognito). These changes affect both genuine privacy users and bot management, with the net effect of narrowing the fingerprint differentiation between bots and humans.

Change two: CDP runtime detection. Bot management vendors (DataDome, PerimeterX, Akamai, Cloudflare) have invested heavily in detecting CDP-driven browsers via subtle runtime artefacts. The 2024-2025 detection waves caught most plain Puppeteer and Playwright deployments; current production scrapers require careful configuration or specialised stealth libraries.

Change three: Network. domain restrictions. Several CDP commands in the Network. domain (request modification, header injection) gained restrictions in Chrome 120-130 to limit abuse. Scrapers that intercepted and modified responses must adapt.

Change four: Headless mode unification. Chrome 109 introduced “new headless” (Headless Shell), and Chrome 132 began deprecating the old headless. The new headless is closer to a real Chrome but with different performance and detection characteristics.

Change five: WebDriver BiDi rolling out. New BiDi-only features (network interception, log capture) make BiDi a credible CDP alternative for many use cases, and frameworks (Playwright, Selenium) are increasingly defaulting to BiDi where possible.

For the broader anti-bot context, see DataDome vs PerimeterX vs Akamai. For the agentic browser angle, see the agentic browser revolution.

Impact on scraping libraries

Library	Protocol	2026 status	Migration concern
Puppeteer	CDP	Active, Chromium-only	Headless detection rising
Playwright	CDP + BiDi	Active, multi-browser	BiDi default for FF and WebKit
Selenium	WebDriver Classic + BiDi	Active	BiDi rollout in progress
Cypress	CDP	Active	Test-focused, less scraping use
Stagehand	Playwright underneath	Active	Inherits Playwright transitions
browser-use	Playwright underneath	Active	Inherits Playwright transitions
Selenium-Wire	WebDriver Classic	Maintenance	Network interception harder

Playwright and Selenium with BiDi are the forward-compatible bets. Puppeteer remains strong for Chrome-only work. Older libraries built on CDP-only assumptions are increasingly fragile.

Detection arms race: what bot management sees

CDP-driven browsers leave subtle traces that bot management can detect:

Trace	Source	Detection
navigator.webdriver true	Old WebDriver default	Trivially detectable
Missing chrome.runtime	Old headless	Detectable
Inconsistent UA strings	Manual UA override	Easy to flag
Permissions API anomalies	CDP affects permissions	Detectable
Cdc_* properties (ChromeDriver)	Selenium-specific	Specific signature
Runtime.evaluate timings	CDP injection	Subtle but detectable
iframe contentDocument access	CDP-mediated access	Detectable
WebGL fingerprint anomalies	Headless renderer	Detectable

The 2026 patterns that defeat most of these:

1. Use Stagehand or Browserbase, which absorb stealth concerns at the platform level.
2. Use undetected-chromedriver or similar stealth-modified drivers (effective but requires maintenance).
3. Use the new Chrome headless mode rather than old headless.
4. Apply runtime patches (puppeteer-extra-plugin-stealth equivalents).
5. Run real Chrome (not headless) when detection budget allows.

For the deeper fingerprinting question, see TLS fingerprinting for scrapers and canvas fingerprinting bypass techniques.

A migration pattern from CDP-direct to BiDi

For scrapers writing to raw CDP, the BiDi migration path is well-defined. The high-level mapping:

CDP domain	BiDi equivalent	Notes
Page.navigate	browsingContext.navigate	Direct mapping
Page.captureScreenshot	browsingContext.captureScreenshot	Direct mapping
Network.enable	network.addIntercept	Different API style
Runtime.evaluate	script.evaluate	Direct mapping
Input.dispatchMouseEvent	input.performActions	Different action model
Target.*	session.*	Different lifecycle model

A scraper using Playwright sees almost none of this; the framework abstracts the protocol choice. A scraper using raw CDP via puppeteer-core or chrome-remote-interface needs to migrate manually.

Decision tree: which protocol to build on in 2026

Q1: Are you Chromium-only or cross-browser?
    ├── Chromium-only -> Q2
    └── Cross-browser -> Use Playwright or Selenium (with BiDi).
Q2: Is your work Puppeteer-aligned (single-browser, low-level)?
    ├── Yes -> Stick with Puppeteer; track BiDi for future migration.
    └── No  -> Q3
Q3: Do you need stealth above all?
    ├── Yes -> Stagehand or Browserbase; or stealth-patched Playwright.
    └── No  -> Playwright with default settings.

What scraping operators should plan for

Three concrete planning items.

First, plan for new headless. The old headless is being retired. New headless behaves more like real Chrome, with different performance and detection characteristics. Migrate now; do not wait for forced removal.

Second, plan for BiDi. New scraping infrastructure should be BiDi-first or framework-agnostic (Playwright handles both). CDP-only investments should be evaluated for migration.

Third, plan for tighter privacy controls. Chrome’s privacy sandbox, Firefox’s Total Cookie Protection, and Safari’s ITP all affect what a browser exposes. Scrapers that depend on specific browser behaviours should test in current and beta Chrome regularly.

For the broader infrastructure planning, see self-hosted proxy infrastructure.

Network interception in the new world

Network interception is one of the most affected areas. CDP’s Network.requestIntercepted and Fetch.* domains evolved with restrictions. BiDi’s network.addIntercept added a more standardised interface.

For scrapers that intercept requests (commonly to skip image/font loading for performance, or to capture specific responses for parsing), the migration:

# Playwright (works with CDP and BiDi automatically)
async def intercept_setup(page):
    async def block_resources(route, request):
        if request.resource_type in ("image", "font", "media"):
            await route.abort()
        else:
            await route.continue_()
    await page.route("**/*", block_resources)

This pattern is portable across the protocol transition because Playwright abstracts the underlying protocol.

Comparison: CDP vs WebDriver Classic vs WebDriver BiDi

Dimension	CDP	WebDriver Classic	WebDriver BiDi
Standard body	Google (Chromium)	W3C	W3C
Browser support	Chromium	All	Chrome, Firefox, partial Safari
Bidirectional events	Yes	No (polling)	Yes
Network interception	Yes (with restrictions)	No	Yes
Console capture	Yes	Limited	Yes
Performance	Excellent	Adequate	Good
Detectability by bot management	Higher	Higher	Equivalent (early)
Future trajectory	Coexists with BiDi	Maintenance	Default cross-browser
Best for in 2026	Chromium-only deep work	Selenium-existing	New cross-browser

External references

The Chrome DevTools Protocol viewer is at chromedevtools.github.io/devtools-protocol. The W3C WebDriver BiDi specification is at w3.org/TR/webdriver-bidi. Playwright’s protocol notes are at playwright.dev/docs/api/class-browser. Chrome’s release notes for headless changes are at developer.chrome.com/blog.

A forward-looking posture

For a scraping operation building a new pipeline in mid-2026:

1. Use Playwright for browser automation. Default to BiDi where Playwright supports it; CDP fallback is automatic.
2. Use Stagehand or Browserbase for managed Chrome with stealth built in, when the cost is justified.
3. Use new headless mode, never old headless.
4. Maintain a stealth-test corpus (a small set of bot-management-fronted pages) to validate detection state on every Chrome version.
5. Track Chrome and Firefox release notes for protocol changes. Subscribe to the chromium-discuss mailing list and the WebDriver BiDi GitHub.

For the deeper pattern of testing scraping pipelines against bot detection, see DataDome vs PerimeterX vs Akamai.

Code-level patterns that survive the transition

Three patterns that work in both CDP and BiDi worlds:

# 1. Resource blocking via Playwright
await page.route("**/*", lambda route, request: (
    route.abort() if request.resource_type in ("image", "font", "media")
    else route.continue_()
))

# 2. Response capture via Playwright
async def on_response(response):
    if "api/products" in response.url:
        await save(await response.json())
page.on("response", on_response)

# 3. Cookie set via Playwright (works with both protocols)
await context.add_cookies([{
    "name": "session", "value": "x", "domain": ".example.com", "path": "/"
}])

Frameworks insulate scrapers from most protocol churn. Direct CDP calls do not.

What about WebRTC, WebTransport, and emerging APIs

The 2026 web includes APIs that did not exist when Puppeteer was designed. WebRTC, WebTransport, the Reporting API, the Permissions Policy framework, and more. CDP and BiDi are evolving to expose these, but coverage is uneven.

For scraping operators working on sites that use these APIs (real-time apps, video conferencing, modern push messaging), the protocol layer matters more than for traditional sites. Expect continued churn through 2027 as the protocols extend coverage.

FAQ

Should I migrate from Puppeteer to Playwright?
For new projects, Playwright is the better default (cross-browser, BiDi-ready). For existing Puppeteer projects, migration is optional unless you need cross-browser.

Is Selenium still relevant in 2026?
Yes. Selenium with BiDi is competitive for cross-browser scraping. Selenium 4+ supports BiDi.

What is the impact of Chrome’s privacy sandbox on scraping?
It tightens the fingerprintable surface. Some bot detection signals weaken; some new ones emerge. Continuous testing required.

Should I use new headless or real Chrome?
New headless is closer to real Chrome and is the supported path. Use real Chrome only when detection budget demands it.

What is the future of CDP after BiDi matures?
CDP coexists with BiDi indefinitely; Chromium will continue to expose both. BiDi is the cross-browser standard; CDP is the Chromium-deep option.

Extended CDP and BiDi analysis

The Chrome DevTools Protocol (CDP) and WebDriver BiDi diverged sharply between 2024 and 2026. CDP remains Chrome-specific. BiDi is the cross-browser standard backed by W3C with implementations in Chrome, Firefox, and WebKit (in progress).

The 2024-2026 changes that affected scrapers are.

1. Chrome 124 (April 2024) deprecated several CDP domains in favour of BiDi-equivalent commands.
2. Chrome 130 (October 2024) added BiDi support for network interception that was previously CDP-only.
3. Firefox 128 (July 2024) reached BiDi parity for the most common scraping operations.
4. The W3C BiDi specification reached Candidate Recommendation in late 2025.

For scrapers the practical implication is that new code should target BiDi for cross-browser portability. Existing CDP code should migrate over a 12-24 month window.

Migration pattern: CDP to BiDi

// Before (CDP)
const cdpSession = await page.context().newCDPSession(page);
await cdpSession.send("Network.enable");
cdpSession.on("Network.requestWillBeSent", event => {
  console.log("CDP request:", event.request.url);
});

// After (BiDi)
const browser = await playwright.chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();
context.on("request", request => {
  console.log("BiDi request:", request.url());
});

Network interception in BiDi

await page.route("**/*api*", async route => {
  const request = route.request();
  const body = request.postData();
  const modified = body.replace("limit=10", "limit=100");
  await route.continue({ postData: modified });
});

await page.route("**/api/data", async route => {
  await route.fulfill({
    status: 200,
    contentType: "application/json",
    body: JSON.stringify({ items: [] }),
  });
});

Detection signals from CDP and BiDi

Bot detectors look for several signals that distinguish automated browsers.

• navigator.webdriver returns true under both CDP and BiDi unless patched.
• The presence of cdc_ properties on the document (older Selenium signature).
• The Runtime.evaluate domain enabled via CDP.
• TLS fingerprints distinct from production browser builds.
• Mouse and keyboard event timing distributions.

The 2026 counter-detection toolkit includes patched Chromium builds (puppeteer-extra-plugin-stealth maintained, undetected-chromedriver, rebrowser-puppeteer), residential proxies, and human-jitter event timing.

Pattern: jitter for action timing

async function humanType(page, selector, text) {
  await page.click(selector);
  for (const char of text) {
    await page.keyboard.type(char);
    await page.waitForTimeout(50 + Math.random() * 150);
  }
}

async function humanScroll(page, totalDelta) {
  let scrolled = 0;
  while (scrolled < totalDelta) {
    const step = 100 + Math.random() * 200;
    await page.mouse.wheel(0, step);
    scrolled += step;
    await page.waitForTimeout(200 + Math.random() * 500);
  }
}

Comparison: CDP vs WebDriver Classic vs WebDriver BiDi 2026

Feature	CDP	WD Classic	WD BiDi
Cross-browser	Chrome only	Cross-browser	Cross-browser
Bidirectional events	Yes	No	Yes
Network interception	Mature	Limited	Mature 2025
Performance	Highest	Lower	Comparable to CDP
Standardised	No	W3C	W3C
Future direction	Maintenance	Maintenance	Active development

What about WebRTC and WebTransport

WebRTC and WebTransport are emerging as scraping targets for real-time data (chat, sports scores, finance). The 2026 toolkit includes.

• node-webrtc and aiortc for programmatic WebRTC peer connections.
• The WebTransport API in Chromium for HTTP/3 stream-based scraping.
• Direct QUIC libraries for lower-level access.

Scraping these protocols is meaningfully harder than HTTPS scraping because the connection state is richer and signalling is more complex.

Additional FAQ

Should I migrate from CDP to BiDi now?
Yes for new projects. For existing CDP code migrate as the BiDi equivalent reaches parity for your use case.

Does BiDi defeat bot detection?
No by itself. Detectors look at the same browser-level signals regardless of protocol.

What about Playwright vs Puppeteer in 2026?
Both ship BiDi support. Playwright has stronger cross-browser story. Puppeteer has tighter Chrome integration.

How do I test for protocol regressions?
Pin the browser version in CI. Run a smoke test suite per browser per channel (stable, beta, dev) to catch upstream breakage early.

Common pitfalls when migrating CDP scrapers to BiDi

The migration looks clean on paper, but the failure modes in production cluster around a handful of patterns that engineers consistently underestimate. Understanding them up front saves weeks of intermittent breakage.

The first pitfall is assuming feature parity exists when it does not. As of mid-2026, BiDi reached parity with CDP for navigation, screenshots, basic network interception, and script evaluation, but it lags CDP for low-level operations like Target.attachToTarget for service workers, fine-grained Performance.getMetrics access, and several Page.* events around lifecycle that mature scrapers depend on for retry logic. Scrapers that orchestrate multiple tabs, frames, or workers at the protocol layer will hit gaps. The mitigation is to build a thin compatibility shim that falls back to CDP for the gap operations while BiDi handles the bulk of the workload.

The second pitfall is event ordering. CDP delivers Network.* events in a documented sequence (requestWillBeSent, then responseReceived, then loadingFinished). BiDi’s network module emits a similar but not identical sequence, and the ordering guarantees are weaker for chunked responses. Scrapers that match request and response pairs by sequence position will see misaligned data. Match by request ID instead.

The third pitfall is target lifecycle. CDP’s Target.* domain provides explicit attach and detach semantics that scrapers use to follow popups and new windows. BiDi’s session and browsingContext model is different, and Playwright’s higher-level abstractions hide the difference but do not eliminate it. If the scraper opens many windows or follows OAuth-style redirects across browsing contexts, expect to rewrite the orchestration loop.

The fourth pitfall is error reporting. CDP errors are typed JSON-RPC errors with stable error codes. BiDi errors are W3C-defined and use different codes for similar conditions. Scraper retry logic that branches on error codes needs an error code translation layer for the migration period.

The fifth pitfall is browser version coupling. BiDi support varies by Chrome and Firefox version. A scraper that works on Chrome 130 BiDi may break on Chrome 128 BiDi. Pin the browser version in containers, validate on the next two stable channels in CI, and refuse to run on unverified or untested versions in production.

The Chrome DevTools Protocol architecture

The Chrome DevTools Protocol exposes Chromium’s internals to external clients via WebSocket-based RPC. The protocol is organised into domains (Network, Page, Runtime, DOM, etc.), each containing methods and events. A client sends method calls and receives results plus events.

CDP was originally designed for the Chrome DevTools UI but became the de facto automation protocol for headless Chromium. Puppeteer (introduced 2017) and Playwright (introduced 2020) both build on CDP. Selenium added CDP support alongside its WebDriver-based automation.

The protocol’s tight coupling to Chromium internals is its strength and its weakness. Clients have access to fine-grained control that other protocols lack. The trade-off is that the protocol is Chrome-only and changes with each Chromium release.

The 2024-2026 changes that affected scrapers were largely driven by Chromium’s internal evolution. Domains that wrapped specific implementation details were deprecated when those details changed. Replacement domains exposed the new internal structures with similar but not identical interfaces.

WebDriver BiDi as the cross-browser successor

WebDriver BiDi is the W3C standard that intends to be the cross-browser successor to CDP. BiDi adds bidirectional communication to the existing WebDriver Classic protocol, enabling event-driven automation that CDP supports but WebDriver Classic does not.

BiDi reached Candidate Recommendation in late 2025. Implementations exist in Chrome (Chromium-based), Firefox (Geckodriver-based), and WebKit (in progress). The implementations vary in completeness, with Chrome and Firefox closest to parity.

For scrapers BiDi solves the cross-browser fragmentation problem. A scraper written against BiDi runs against Chrome, Firefox, and eventually Safari without code changes. The portability is meaningful for teams that test against multiple browsers or that want to switch browsers as detection landscapes evolve.

BiDi has performance comparable to CDP in 2026. Early implementations had higher latency, but optimisation has narrowed the gap. The remaining performance gap is task-specific, generally favouring CDP for high-frequency event scenarios and BiDi for control-flow-heavy scenarios.

Network interception parity

Network interception (intercepting requests, modifying headers, mocking responses) is the most-used CDP feature in scraping. CDP supported it from the beginning. WebDriver BiDi added comparable functionality through a 2025 update.

The BiDi network interception API uses the addIntercept and continueRequest patterns. A client adds an intercept matching a URL pattern, receives requestPaused events when matching requests fire, and continues or fulfils each request. The API maps cleanly onto Playwright’s existing route function.

The 2026 best practice for new scraping code is to use Playwright’s BiDi-backed network interception rather than CDP. Existing code can stay on CDP for the lifetime of the project. Migration is recommended when other reasons drive a refactor.

Detection and the protocol-level signal

Bot detection vendors look at protocol-level signals to identify automated browsers. CDP usage leaves traces that detectors can find. BiDi usage leaves different traces. Both are detectable.

The 2026 detection patterns include checking for the presence of specific runtime features that automation enables, looking for timing patterns characteristic of automation libraries, and probing for behaviours that humans rarely exhibit. None of the detection methods is foolproof, but together they create a strong signal.

Counter-detection in 2026 typically includes patched browsers (puppeteer-extra-plugin-stealth, undetected-chromedriver, rebrowser-puppeteer), residential proxies for IP-level evasion, and human-like timing for behavioural evasion. The arms race continues with neither side decisively winning.

A 2026 trend that affects the calculus is the increasing acceptance of agent traffic by large platforms. A site that recognises a verified agent presenting a signed identity may grant access without the heavy-handed bot detection that anonymous scrapers face. The shift moves the detection question from how to evade detection to how to participate in the verified-agent ecosystem.

Next steps

If your pipeline is Puppeteer-based and you have not evaluated Playwright in two years, this quarter is the right time. The cross-browser ergonomics and BiDi-readiness pay off through the 2026-2027 protocol transitions. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the agentic browser revolution guide.

This guide is informational, not engineering or legal advice.

How to scrape Mercado Libre Mexico in 2026

Scrape Mercado Libre Mexico effectively in 2026 and you have access to the largest ecommerce marketplace in Latin America. Mercado Libre Mexico (MLM) serves over 60 million Mexican shoppers, processes billions of pesos in monthly GMV, and indexes more SKUs than Amazon Mexico. Brand managers, agencies, and price intelligence teams cannot get a complete LATAM picture without it.

Unlike most ecommerce scraping targets, Mercado Libre offers a public Items API that returns clean structured data without authentication. This single fact makes MLM the friendliest large ecommerce target in the world to scrape. This guide covers the API path, the browser fallback for cases the API does not cover, anti-bot considerations for sustained scraping, and production patterns for Mexico-specific data quality.

What Mercado Libre Mexico exposes

Surface	URL pattern	Best for
Public Items API	api.mercadolibre.com/items/{item_id}	High-throughput product extraction
Public Sites API	api.mercadolibre.com/sites/MLM/search?q={query}	Discovery, search, category browse
Product detail page	mercadolibre.com.mx/p/MLM{product_id}	Full UI extraction (if API misses fields)
Seller API	api.mercadolibre.com/users/{user_id}	Seller profile data

The MLM site code in the API is “MLM” (Mexico). Other LATAM markets use MLA (Argentina), MLB (Brazil), MCO (Colombia), MLC (Chile), MLU (Uruguay), MPE (Peru). The patterns below work across all sites with site code substitution.

Public API access

The Items API requires no authentication for read access:

import asyncio
import httpx

async def get_item(item_id: str) -> dict:
    url = f"https://api.mercadolibre.com/items/{item_id}"
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.get(url, headers={"User-Agent": "DRTBot/1.0 (research@example.com)"})
        r.raise_for_status()
        return r.json()

item = asyncio.run(get_item("MLM3000123456"))
print(item["title"], item["price"], item["currency_id"])

That is the entire scraping pipeline for product detail data. No browser, no proxies for low volume, no captcha. The API returns a complete product record:

{
    "id": "MLM3000123456",
    "site_id": "MLM",
    "title": "Apple Iphone 15 Pro 256gb Titanio Natural",
    "seller_id": 12345678,
    "category_id": "MLM1055",
    "price": 28999,
    "base_price": 28999,
    "original_price": 35999,
    "initial_quantity": 50,
    "available_quantity": 38,
    "sold_quantity": 12,
    "currency_id": "MXN",
    "condition": "new",
    "permalink": "https://articulo.mercadolibre.com.mx/...",
    "thumbnail": "https://http2.mlstatic.com/...",
    "shipping": {"free_shipping": True},
    "attributes": [...],
}

For high-volume extraction, the API alone gets you most of the way.

Field reference for the Items API

The Items API returns a rich object. Worth knowing the most useful fields:

Field	Type	Use
id	string	Unique item identifier (MLM-prefixed)
title	string	Product title
seller_id	int	Seller user ID, link to Users API
price	number	Current price in MXN
original_price	number	Pre-promotion price (null if no promotion)
available_quantity	int	Stock count (capped at 50)
sold_quantity	int	Lifetime sold count
condition	string	“new”, “used”, “refurbished”
permalink	string	Public product URL
attributes	array	Product attributes (brand, model, etc)
variations	array	Variant details if applicable
shipping	object	Shipping options including free_shipping flag
location	object	Geographical location of the item
pictures	array	Image URLs
category_id	string	MLM category identifier
domain_id	string	Higher-level product domain (e.g. MLM-CELLPHONES)
listing_type_id	string	Listing tier (“gold_special”, “gold_pro”, etc)
health	number	Listing quality score (0 to 1)
catalog_listing	bool	Whether the item is part of the official catalog
catalog_product_id	string	Catalog reference if catalog_listing

For a full schema, the Mercado Libre developer docs maintain the canonical reference.

Discovery via Sites API

async def search_mlm(query: str, offset: int = 0, limit: int = 50) -> dict:
    url = f"https://api.mercadolibre.com/sites/MLM/search?q={query}&offset={offset}&limit={limit}"
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.get(url)
        return r.json()

results = asyncio.run(search_mlm("auriculares bluetooth"))
for item in results["results"]:
    print(item["id"], item["title"], item["price"])

The search response includes pagination, facets, available filters, and the top 50 results per page. Walk pagination with the offset parameter to harvest a category.

Rate limits and authentication

Public API endpoints work without auth but are rate-limited per IP. For serious volume, register an application at Mercado Libre developers and use the OAuth-issued token; rate limits jump significantly.

async def authenticated_get(item_id: str, access_token: str) -> dict:
    url = f"https://api.mercadolibre.com/items/{item_id}"
    headers = {"Authorization": f"Bearer {access_token}"}
    async with httpx.AsyncClient() as client:
        r = await client.get(url, headers=headers)
        return r.json()

OAuth flow is standard. Application registration is free.

When the API is not enough

The API exposes most product fields cleanly. Two cases require browser fallback:

1. Reviews. Item review text is rendered client-side and not in the API response.
2. Q&A. Buyer questions and seller answers are paginated client-side.

Browser-based fallback for reviews:

from playwright.async_api import async_playwright

async def scrape_reviews(permalink: str) -> list[dict]:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="es-MX")
        pg = await ctx.new_page()
        await pg.goto(permalink, wait_until="networkidle")
        # scroll to load reviews
        await pg.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        await pg.wait_for_timeout(2000)
        reviews = []
        for el in await pg.locator(".ui-review-capability__rating").all():
            reviews.append({
                "rating": int(await el.get_attribute("data-rating") or 0),
                "text": (await el.text_content() or "").strip(),
            })
        await browser.close()
    return reviews

Mexican Peso price handling

Mexican Peso uses the symbol $ (same as USD), which causes confusion. The currency ISO code is MXN. Always store the currency_id from the API alongside the price.

The peso uses comma as decimal separator and period as thousands: $1.234.567,89 MXN. The API returns plain numbers, so this is only a display concern.

Spanish language handling

MLM listings are in Spanish (Mexican variant). Standard UTF-8 handling is sufficient. Two specific gotchas:

First, accents and tildes (á, é, í, ó, ú, ñ) appear in titles. URL-encoding for search queries is essential:

from urllib.parse import quote
query = quote("computación")
url = f"https://api.mercadolibre.com/sites/MLM/search?q={query}"

Second, regional Spanish vocabulary differs. “Auriculares” (Mexico) vs “audífonos” (Spain) for headphones. Search for both terms when building cross-LATAM queries.

Adding proxies for sustained scale

For high-volume scraping, route through residential or mobile proxies. Mexican ISP IPs work well; data center IPs get challenged faster.

import random

PROXIES = [
    "http://us:pw@mx-residential-1.proxy.example.com:8000",
    "http://us:pw@mx-residential-2.proxy.example.com:8000",
]

async def get_item_with_proxy(item_id: str) -> dict:
    proxy = random.choice(PROXIES)
    url = f"https://api.mercadolibre.com/items/{item_id}"
    async with httpx.AsyncClient(proxy=proxy, timeout=30) as client:
        r = await client.get(url)
        return r.json()

For LATAM proxy strategy, see our best residential proxy providers 2026.

OAuth and rate limit tiers

Public API limits run roughly 1000 requests per minute per IP. With OAuth authentication, the limit jumps to 5000 to 20000 requests per minute depending on application tier.

For high-volume teams, registering an application is well worth the 30-minute setup. The OAuth flow is standard:

async def get_access_token(client_id: str, client_secret: str, refresh_token: str) -> str:
    url = "https://api.mercadolibre.com/oauth/token"
    data = {
        "grant_type": "refresh_token",
        "client_id": client_id,
        "client_secret": client_secret,
        "refresh_token": refresh_token,
    }
    async with httpx.AsyncClient() as c:
        r = await c.post(url, data=data)
        return r.json()["access_token"]

Tokens expire after 6 hours; refresh tokens are long-lived. Cache tokens to avoid hitting the OAuth endpoint per request.

Catalog vs marketplace listings

Mercado Libre has two parallel concepts:

Catalog listings: a single canonical product page that aggregates many seller offers (similar to Amazon’s product pages). Catalog listings are at mercadolibre.com.mx/p/MLM{catalog_id}.

Marketplace listings: individual seller listings, each with their own item_id, even if they sell the same product.

For brand intelligence, catalog listings give you the easy “who sells X” view but marketplace listings give you the long-tail seller activity. Capture both.

async def get_catalog_listings(catalog_product_id: str) -> list[dict]:
    url = f"https://api.mercadolibre.com/products/{catalog_product_id}/items"
    async with httpx.AsyncClient() as c:
        r = await c.get(url)
        return r.json()

The endpoint returns all marketplace listings for a single catalog product, which is the basis for cross-seller price comparison on the same SKU.

Comparison to other LATAM markets

Market	Public API	Volume	Bot defense
Mercado Libre Mexico	Yes (clean)	Largest in MX	Low (API path)
Mercado Libre Brazil	Yes (clean)	Largest in BR	Low (API path)
Mercado Libre Argentina	Yes (clean)	Largest in AR	Low (API path)
Amazon Mexico	No	Medium	High
Walmart Mexico	No	Medium	High
Liverpool Mexico	No	Smaller	Medium
OLX Mexico	Limited	Medium	Medium

For a deeper LATAM treatment, see scrape OLX Brazil and LATAM marketplaces.

Production patterns

Three patterns matter.

First, batch with multi-get. The API supports up to 20 item IDs per request:

async def get_items_batch(item_ids: list[str]) -> list[dict]:
    ids_csv = ",".join(item_ids[:20])
    url = f"https://api.mercadolibre.com/items?ids={ids_csv}"
    async with httpx.AsyncClient() as client:
        r = await client.get(url)
        results = r.json()
        return [r["body"] for r in results if r.get("code") == 200]

This cuts request count by 20x for catalog-wide scrapes.

Second, capture seller-level data. Many MLM sellers list across multiple item IDs for the same product. Joining at seller-level helps deduplicate.

Third, monitor for category remapping. Mercado Libre periodically restructures categories. Cache the category tree weekly:

async def get_category_tree(site_id: str = "MLM") -> dict:
    url = f"https://api.mercadolibre.com/sites/{site_id}/categories"
    async with httpx.AsyncClient() as client:
        r = await client.get(url)
        return r.json()

Mexico-specific consumer behavior insights

A few patterns specific to the Mexican ecommerce market that affect what data matters:

Installments dominate. More than 60 percent of MLM transactions over MXN $5,000 use meses sin intereses. Track installment availability separately from headline price.

Hot Sale (May) and El Buen Fin (November) are the two biggest sales events. Inventory and pricing dynamics during these weeks are dramatically different from steady state. Plan for higher poll frequency.

Cash on delivery is still common. The payment_methods block lists supported methods; cash availability correlates with price tier and seller reputation.

OXXO payment (a convenience-store-based offline payment) is the largest non-card payment method. Listings supporting OXXO have a flag in the payment_methods array.

Mexican consumers cluster around major metro areas (Mexico City, Guadalajara, Monterrey). Seller location data is useful for shipping-time-based price intelligence.

Storage schema

CREATE TABLE mlm_items (
    id TEXT PRIMARY KEY,
    site_id TEXT NOT NULL,
    title TEXT NOT NULL,
    seller_id BIGINT,
    category_id TEXT,
    price NUMERIC(12,2) NOT NULL,
    original_price NUMERIC(12,2),
    currency_id CHAR(3) NOT NULL,
    available_quantity INTEGER,
    sold_quantity INTEGER,
    condition TEXT,
    permalink TEXT NOT NULL,
    free_shipping BOOLEAN,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    raw_jsonb JSONB
);
CREATE INDEX idx_mlm_extracted_at ON mlm_items(extracted_at);
CREATE INDEX idx_mlm_seller_id ON mlm_items(seller_id);
CREATE INDEX idx_mlm_category_id ON mlm_items(category_id);

Real benchmark numbers

A March 2026 production run, 10,000 MLM items via the public Items API with rotating Mexican residential proxies:

Metric	Value
Success rate	99.4%
Median latency per item	0.3 s
p99 latency	1.8 s
Cost per 1000 items	$4
429 throttle rate	0.4%
Failed item lookups	0.6% (mostly 404 on removed items)

Compared to browser-based scraping of Lazada or Shopee at $80 to $200 per 1000 items, MLM is dramatically cheaper. The public API is a real differentiator.

Capturing the seller and shop side

Mercado Libre’s seller data is rich. The Users API returns seller reputation, registration date, location, and feedback statistics:

async def get_seller(user_id: int) -> dict:
    url = f"https://api.mercadolibre.com/users/{user_id}"
    async with httpx.AsyncClient() as c:
        r = await c.get(url)
        return r.json()

# Seller payload includes:
#   nickname, registration_date, country_id, address (city, state),
#   user_type ("normal", "official_store", "brand"),
#   seller_reputation (level, transactions count, ratings)

For brand intelligence, seller-level data lets you spot unauthorized resellers, track grey market activity, and identify counterfeit hot spots. The Mexican market specifically has heavy unauthorized reselling of imported electronics.

Cost expectations

10,000 MLM products per month with API access only:

Component	Cost
API requests (proxied)	$20-$40
Compute	$10
Total	$30-$50

Mercado Libre is the cheapest large ecommerce target to scrape because of the public API. For comparison, Lazada or Shopee at the same volume runs $150-$280.

Legal considerations

Mexico’s Federal Law on the Protection of Personal Data Held by Private Parties (LFPDPPP) regulates personal data. Public commercial data (product listings, prices, seller-level data at city granularity) is not personal data.

Mercado Libre’s terms of service explicitly allow programmatic access through the public API for non-commercial-impersonation use cases. Scraping the public website at high volume can technically violate the terms but the API path is contractually clean.

For broader LATAM compliance, see scraping EU sites: jurisdictional realities, which covers similar principles applied to Mexican LFPDPPP.

Mercado Libre-specific data points

A few MLM-only fields worth capturing:

mercado_envios flag: indicates Mercado Libre handles fulfillment. Strong predictor of customer satisfaction and conversion.

gold_special and gold_pro listing types: paid promotional tiers that affect ranking. Capture as a quality signal.

installments block: Mexican consumers heavily use installment plans (meses sin intereses). The number of available installments and whether interest-free are major purchase drivers.

reputation on the seller: a 5-tier color score (verde to rojo) that summarizes seller quality. Critical for brand intelligence to flag low-reputation sellers carrying brand SKUs.

def extract_mlm_specific(item: dict) -> dict:
    return {
        "mercado_envios": item.get("shipping", {}).get("mode") == "me2",
        "listing_type_id": item.get("listing_type_id"),
        "free_shipping": item.get("shipping", {}).get("free_shipping", False),
        "installments": item.get("installments", {}).get("quantity", 1),
        "interest_free": item.get("installments", {}).get("rate", 1) == 0,
    }

Question and answer scraping

Mercado Libre has a buyer Q&A system that often contains useful product information not present in the official listing. The Q&A API:

async def get_questions(item_id: str) -> list[dict]:
    url = f"https://api.mercadolibre.com/questions/search?item={item_id}"
    async with httpx.AsyncClient() as c:
        r = await c.get(url)
        return r.json().get("questions", [])

Each question includes the question text, the seller’s answer, and timestamps. For brand monitoring, this catches competitor-versus-product comparisons that appear in buyer questions.

Cross-LATAM expansion

Once you have a working MLM pipeline, expanding to other Mercado Libre sites is essentially a configuration change:

SITE_CONFIG = {
    "MLM": {"country": "Mexico", "currency": "MXN", "language": "es-MX"},
    "MLB": {"country": "Brazil", "currency": "BRL", "language": "pt-BR"},
    "MLA": {"country": "Argentina", "currency": "ARS", "language": "es-AR"},
    "MCO": {"country": "Colombia", "currency": "COP", "language": "es-CO"},
    "MLC": {"country": "Chile", "currency": "CLP", "language": "es-CL"},
    "MLU": {"country": "Uruguay", "currency": "UYU", "language": "es-UY"},
    "MPE": {"country": "Peru", "currency": "PEN", "language": "es-PE"},
}

async def get_item_anywhere(site_id: str, item_id: str) -> dict:
    return await get_item(item_id)  # API is global, item_ids are site-prefixed

For full LATAM coverage, run the same scraper against each site and store with site_id as a partition key.

Frequently asked questions

Why is the API the recommended path here when other guides recommend browser-based scraping?
Because Mercado Libre is unique among large ecommerce sites in offering a clean, well-documented, no-auth public API. Most sites do not. When the official API works, use it.

What about web scraping the Mercado Libre site directly?
The site is heavily defended (Cloudflare, custom challenges) and rate-limited harder than the API. There is essentially no reason to web-scrape MLM when the API works.

Are there fields in the website that the API does not expose?
Reviews and Q&A are not in the Items API and need browser scraping. Everything else (pricing, stock, attributes, shipping, seller info) is in the API.

Can I write to the MLM API (e.g. update listings)?
Yes if you are a registered seller and authenticate with OAuth. Read-only public endpoints work without auth.

How do I track price history?
The API returns the current price. For historical pricing, snapshot the API response daily and store in a time-series table.

Can I monitor specific catalog products instead of scraping by item_id?
Yes. Use the catalog product endpoint to get all current listings for a single canonical product, then track the listings over time. This is more efficient for brand monitoring than item-by-item scraping.

Can I scrape other LATAM Mercado Libre sites with the same code?
Yes. Substitute the site_id in URLs (MLA for Argentina, MLB for Brazil, etc.). Currency and language change accordingly.

What about Mercado Pago payment data?
Not exposed publicly. Payment information is restricted to seller-side reports through authenticated APIs.

How do I track promotional events like Hot Sale Mexico?
Mercado Libre runs Hot Sale (May), El Buen Fin (November), and Cyber Monday Mexico. During these events, prices change hourly. Increase poll frequency on flagged SKUs and capture the original_price to detect promotion vs sale dynamics.

Does Mercado Libre have variant-level data like Shopee?
Yes. The variations field on the item response lists each variant with its own price, stock, and attributes. Treat variants as separate rows for accurate inventory tracking.

Can I scrape Mercado Libre Classifieds (vehicles, real estate)?
Yes. The Classifieds API uses the same shape with category-specific extra fields. Vehicles include brand, model, year, mileage; real estate includes property type, bedrooms, location.

How do I handle item_id changes after a relisting?
When a seller relists an item, it gets a new item_id. The previous item_id returns 404. Track by SKU plus seller for stable identification across relisting events.

Common production gotchas

A few patterns that cause issues in MLM scraping:

The API returns prices as integers when the value is whole-peso, floats when fractional. Cast consistently to Decimal to avoid type drift.

Some categories have site-specific quirks. Real estate on MLM lists prices in MXN by default but in USD for high-end properties. Always check currency_id.

The available_quantity field caps at 50 even for higher-stock items. For accurate inventory, use the seller-side reports if you have access; otherwise treat 50 as “in stock plenty”.

Removed items return 404 for several months, then start returning 410 Gone. Handle both.

Search API pagination caps at offset 1000. To enumerate beyond, use the scroll_id returned in the search response.

Does Mercado Libre have an SDK?
Official SDKs exist for PHP, Python, Java, and JavaScript. The Python SDK (mercadolibre) is reasonable for prototypes but most production teams use direct httpx calls for finer control.

Can I get historical sales data?
The sold_quantity field is current cumulative. For sales velocity over time, snapshot daily and compute deltas.

For more LATAM ecommerce coverage, browse the ecommerce category.

Edge AI scraping: running models at the network edge

Edge AI scraping has moved from research into mainstream production through 2024-2026. The combination of edge compute platforms (Cloudflare Workers AI, Vercel Edge Functions, Fastly Compute@Edge, AWS Lambda@Edge) and increasingly capable small models (Llama 3.2 3B, Phi-3 Mini, Mistral 7B, Gemma 2B) made it economical to run inference close to the network rather than in centralised model APIs. For scraping operators, this matters because edge AI changes the cost structure, the latency characteristics, the privacy posture, and the operational model of AI-augmented scraping. This guide walks through what edge AI actually is for scraping, the platforms that matter in 2026, the model choices that work, the patterns that fit edge constraints, and a practical playbook for moving inference closer to the data.

The audience is the data engineer or platform owner running AI-augmented scraping who wants to understand where edge fits.

What edge AI means for scraping

Three things at once.

First, model inference runs at edge locations rather than in central regions. Instead of round-tripping every request to a US-east OpenAI endpoint, inference happens at one of dozens (Cloudflare 300+, Vercel 25+, Fastly 90+) of edge locations close to the requester or the data source.

Second, the model is typically smaller. Edge platforms support small-to-mid-sized models (under 10B parameters typically) due to memory and cold-start constraints. Frontier models still run centrally; edge runs supporting models.

Third, the edge platform absorbs operational complexity. The edge runtime handles routing, scaling, cold starts, and deployment. The developer writes a function; the platform runs it close to the user.

For scraping, the implication is that AI tasks adjacent to the scrape (classification, extraction, summarisation, language detection, content moderation, deduplication) can move to the edge while heavyweight reasoning stays central.

For the broader emerging tech context, see the agentic browser revolution and RAG over scraped data.

The 2026 edge AI platforms

Four platforms in production scraping use:

Platform	Runtime	Native AI	Model catalogue
Cloudflare Workers AI	V8 isolates, Wasm	Yes (Workers AI)	50+ pre-deployed (Llama, Mistral, Whisper, embedding models)
Vercel Edge Functions	V8 isolates	Through partners	OpenAI, Anthropic, fal.ai integrations
Fastly Compute@Edge	Wasm	Limited	Custom WASM models possible
AWS Lambda@Edge	Node.js, Python	Limited	Bedrock-adjacent integrations

Cloudflare Workers AI is the most scraping-relevant in 2026 because it includes a substantial model catalogue running natively at the edge with no cold-start tax. The pricing model (per neurons-per-month) makes inference economical at scale.

A worked example: edge classification before central LLM

A common pattern: a scraper ingests millions of pages per day. Most pages need only basic classification (language, content type, freshness signal). A small fraction (say 5 percent) require deep LLM analysis. Running the LLM on every page is wasteful; running classification on every page is necessary.

The edge solution: deploy a lightweight classifier at the edge that runs on every scraped page and forwards only the relevant pages to the central LLM.

// Cloudflare Worker: edge classification gate
export default {
  async fetch(request, env) {
    const { url, html } = await request.json();
    const text = extractMainContent(html).slice(0, 2000);

    const classifyResult = await env.AI.run(
      "@cf/meta/llama-3.2-3b-instruct",
      {
        prompt: `Classify the following page. Return JSON with fields:
                 {category: news|product|profile|other, language: ISO code,
                  freshness_signal: stale|fresh|unknown, requires_deep_analysis: boolean}.
                 Content: ${text}`,
        max_tokens: 100,
      }
    );

    const classification = JSON.parse(classifyResult.response);

    if (classification.requires_deep_analysis) {
      return Response.json({
        forward: true,
        classification,
        url,
      });
    }
    return Response.json({
      forward: false,
      classification,
      url,
    });
  },
};

The economic outcome: for a 1M-page-per-day pipeline, edge classification at fractions of a cent per page filters down to 50K pages per day requiring central LLM analysis, with the central LLM bill dropping by 95 percent.

Where edge AI fits in scraping pipelines

Six concrete patterns:

Pattern	Edge model	Saves
Page classification	3B model	Central LLM tokens for irrelevant pages
Language detection	Tiny model (FastText, Lingua)	Routing logic complexity
Extraction (structured)	Small instruct model	Central LLM for routine extraction
Embedding generation	bge-small, e5-small	Centralised embedding API costs
Deduplication (semantic)	Embedding + similarity	Central pipeline duplicate work
Content moderation	Small classifier	Manual review queue

Each pattern moves work that does not need frontier-model intelligence to the edge, where it runs cheaper and faster.

For the broader pipeline pattern, see building scraping pipelines with Prefect 3.

Model choices for edge inference

The 2026 small-model landscape has matured significantly. The models that perform well at the edge:

Model	Parameters	Strengths	Notes
Llama 3.2 3B Instruct	3B	General instruct, good multilingual	Cloudflare native
Llama 3.2 1B	1B	Tiny, fast, basic tasks	Cloudflare native
Phi-3 Mini	3.8B	Strong reasoning for size	Multiple platforms
Mistral 7B	7B	Balanced; production-tested	Most platforms
Gemma 2 2B	2B	Strong instruction-following	Multiple platforms
BGE-Small	Embedding	Multilingual	Cloudflare native
E5-Small	Embedding	English-strong	Cloudflare native
Whisper Tiny	ASR	Audio transcription	Cloudflare native

Picking the right model is the central engineering decision. The pattern: pick the smallest model that meets your quality bar, validate against your evaluation set, deploy.

Decision tree: should this AI task run at the edge?

Q1: Does the task happen on every scraped page?
    ├── Yes -> Edge candidate (volume justifies edge optimisation).
    └── No  -> Q2
Q2: Is the task latency-sensitive (sub-100ms)?
    ├── Yes -> Edge candidate (round-trip to central API too slow).
    └── No  -> Q3
Q3: Does the task require frontier model reasoning?
    ├── Yes -> Stay central. Edge cannot match frontier capability.
    └── No  -> Q4
Q4: Does the task need to run close to data (privacy, residency)?
    ├── Yes -> Edge candidate.
    └── No  -> Q5
Q5: Is the model size under 10B parameters and the prompt under 4K tokens?
    ├── Yes -> Edge candidate.
    └── No  -> Stay central or hybrid.

The decision tree captures the typical fit. Volume, latency, capability ceiling, residency, and size constraints all push toward or away from edge.

Cost economics at the edge

Rough cost benchmarks for 1M classifications of 1000-token inputs in mid-2026:

Approach	Cost (USD)	Latency p50	Latency p99
OpenAI GPT-4o-mini (central)	150	800ms	3000ms
Anthropic Haiku (central)	250	600ms	2500ms
Cloudflare Workers AI Llama 3B	30	200ms	800ms
Self-hosted Llama 3B (on-prem)	50 (compute)	300ms	1200ms
Self-hosted Llama 70B	600 (compute)	1000ms	4000ms

The pattern: edge AI on small models is the cost leader for high-volume routing-style tasks. Frontier models at central locations are the right choice for nuanced reasoning. The architecture combines both.

For the deeper cost discussion, see AI scraping cost benchmark.

Privacy and residency

Edge AI improves privacy in two ways.

First, data does not have to leave the region. A page scraped from an EU site can be classified at an EU edge location without the content reaching US-based central inference. For GDPR compliance (covered in the GDPR scraping compliance guide), this matters.

Second, the data lifecycle is shorter. Edge functions are stateless by default; the page content is processed and forgotten. Central inference often involves logging and retention.

The privacy improvement is real but not absolute. Most edge platforms still log requests for billing and observability. A scraping operator with strict residency requirements should verify the platform’s data processing terms.

Operational patterns: deployment and observability

Three patterns that work in production.

Pattern one: managed edge with platform AI. Cloudflare Workers AI or Vercel Edge with provider AI. Lowest operational overhead. Use when the platform’s model catalogue meets your needs.

Pattern two: managed edge with custom model. Deploy your own small model to the edge via the platform’s WASM/binary support. Higher complexity, but unlocks proprietary or fine-tuned models. Cloudflare WASM and Fastly Compute support this.

Pattern three: hybrid edge plus central. The most common production pattern. Edge handles classification, embedding, simple extraction. Central handles reasoning, summarisation, complex extraction. The edge function makes the routing decision.

For deployment specifics on running scrapers at the edge themselves (not just AI), see running scrapers on Cloudflare Workers.

Edge embeddings and semantic search

A specific high-leverage pattern: generate embeddings at the edge as part of the scrape, before the data ever reaches central infrastructure.

export default {
  async fetch(request, env) {
    const { url, text } = await request.json();
    const embedding = await env.AI.run(
      "@cf/baai/bge-base-en-v1.5",
      { text }
    );
    await env.VECTOR_INDEX.upsert([
      { id: hash(url), values: embedding.data[0],
        metadata: { url, scraped_at: new Date().toISOString() } },
    ]);
    return new Response("OK");
  },
};

The embedding generation, which would historically have run in a central worker after the scrape completed, now runs at the edge as part of the scrape. The latency saving is real (no round-trip to central embedding API) and the cost saving is substantial.

For the broader vector database integration, see vector databases for scraping pipelines.

Comparison: edge AI platforms for scraping

Platform	Native AI catalogue	Cold start	Egress cost	Best for
Cloudflare Workers AI	50+ models	None (V8 isolates)	Free	Most scraping AI use cases
Vercel Edge Functions	Provider integrations	Minimal	Per request	Vercel-stack scraping
Fastly Compute@Edge	Custom WASM	Minimal	Per request	Custom-model needs
AWS Lambda@Edge	Bedrock adjacency	Cold start risk	Per request + AWS-typical	AWS-stack scraping
Self-hosted edge	Anything	None (warm)	Variable	Maximum control

Cloudflare Workers AI dominates the 2026 scraping use case because of the native model catalogue, the cold-start-free runtime, and the pricing model. Vercel and Fastly are competitive for specific stacks.

Limitations and where edge AI does not fit

Three classes of task remain central-only:

1. Frontier-model reasoning. Claude Opus, GPT-4o, Gemini Ultra do not run at the edge in 2026. Tasks that need their capabilities stay central.

2. Long-context tasks. Edge runtimes typically have memory caps that limit context to 8K-32K tokens. Long-document analysis stays central.

3. Stateful workflows. Edge functions are stateless; multi-step agentic workflows that require memory across steps need central orchestration even if individual steps run at the edge.

The pragmatic 2026 architecture splits the work: edge for high-volume simple tasks, central for low-volume complex tasks, with the edge making the routing decision.

For the broader agentic context, see the agentic browser revolution.

External references

Cloudflare Workers AI documentation is at developers.cloudflare.com/workers-ai. Vercel Edge Functions docs are at vercel.com/docs/functions/edge-functions. Fastly Compute@Edge is at docs.fastly.com/products/compute. The Hugging Face small-model leaderboard is at huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.

Operational checklist

Item	Owner	Done when
Identify edge-eligible AI tasks in pipeline	Engineering	Inventory complete
Select edge platform	Platform	Decision documented
Pick edge model per task	ML lead	Eval results signed off
Implement edge function with logging	Engineering	Deployed in staging
Run quality eval against central baseline	ML lead	Quality within tolerance
Implement central fallback for edge failures	Engineering	Fallback tested
Wire monitoring (latency, error rate)	Platform	Dashboards live
Document privacy posture	Compliance	Privacy assessment complete
Cutover with shadow mode	Engineering	Old path retired after stable

FAQ

What is the smallest model that performs well at the edge?
For routing-style classification, Llama 3.2 1B or Gemma 2 2B work well. For extraction, Llama 3.2 3B or Phi-3 Mini. Validate against your eval set.

Can frontier models run at the edge?
Not in 2026. Frontier models exceed edge memory and runtime constraints. Edge handles small/medium models; frontier stays central.

Is edge AI cheaper than central API?
For high-volume tasks (embeddings, classification, simple extraction), yes by 5-10x. For low-volume nuanced tasks, the difference is marginal.

What happens during edge AI outages?
Most platforms have multi-region failover. Build central fallback for the same task to maintain pipeline operation during outages.

Can I run my own fine-tuned model at the edge?
On Cloudflare WASM and Fastly Compute, yes if you can compile your model to WASM. On Workers AI, only models in the platform catalogue.

Extended edge AI scraping analysis

Edge AI scraping moves the model inference closer to the data, reducing round-trip latency and enabling on-device privacy. By 2026 three deployment patterns dominate.

1. Browser-side inference using WebGPU plus ONNX Runtime Web or transformers.js.
2. Edge-worker inference using Cloudflare Workers AI, Vercel Edge, or Fastly Compute.
3. Device-side inference using llama.cpp, MLX, or Apple Neural Engine.

For scraping the use cases include in-page extraction without round-tripping HTML to a server, content classification at the edge before storage, and PII redaction before centralised aggregation.

Pattern: WebGPU classification of scraped pages

import { pipeline, env } from "@xenova/transformers";

env.backends.onnx.wasm.proxy = true;

const classifier = await pipeline(
  "text-classification",
  "Xenova/distilbert-base-uncased-finetuned-sst-2-english",
  { device: "webgpu" }
);

async function classifyPage(html) {
  const text = stripHtml(html).slice(0, 2000);
  const result = await classifier(text);
  return result;
}

Pattern: Cloudflare Workers AI for edge extraction

export default {
  async fetch(request, env) {
    const url = new URL(request.url).searchParams.get("u");
    const page = await fetch(url).then(r => r.text());
    const text = stripHtml(page).slice(0, 4000);
    const completion = await env.AI.run(
      "@cf/meta/llama-3.1-8b-instruct",
      {
        messages: [
          { role: "system", content: "Extract product name, price, and availability from the text. Return JSON only." },
          { role: "user", content: text },
        ],
      }
    );
    return new Response(completion.response, {
      headers: { "Content-Type": "application/json" },
    });
  },
};

Pattern: on-device inference with llama.cpp

from llama_cpp import Llama

llm = Llama(
    model_path="./models/Phi-3-mini-4k-instruct-q4.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,
)

def extract(text, schema):
    prompt = f"Extract per schema: {schema}\n\nText: {text}\n\nJSON:"
    output = llm(prompt, max_tokens=512, stop=["\n\n"], temperature=0.0)
    return output["choices"][0]["text"].strip()

Privacy and compliance benefits

Edge inference provides three compliance benefits.

1. Personal data can be redacted before leaving the user’s device.
2. Cross-border transfer obligations can be reduced because data never leaves the jurisdiction.
3. Aggregation can be done on derived signals rather than raw personal data.

Comparison: edge AI deployment options 2026

Option	Latency to first token	Cost model	Privacy posture
Browser WebGPU	100-300ms	Free (user device)	Strongest
Cloudflare Workers AI	50-200ms	Per-request	Moderate
Vercel Edge	100-300ms	Per-request	Moderate
AWS Lambda + Bedrock	200-500ms	Per-token	Moderate
On-device (mobile)	50-200ms	Free (user device)	Strongest
Centralised GPU server	50-100ms	Per-token plus infra	Weakest

Model size and quality tradeoffs

Edge deployment forces smaller models. The 2026 sweet spots are.

• 1-3B parameters for browser WebGPU (Phi-3, Llama 3.2 1B/3B).
• 7-13B for edge workers with hosted GPU (Mistral, Llama 3.1 8B).
• 70B+ remains centralised for complex tasks.

A pattern is to route by task complexity. Simple extraction goes to the 1-3B edge model. Complex synthesis goes to a 70B centralised model. The router decides per request.

Additional FAQ

Is edge AI mature enough for production scraping?
Yes for classification, redaction, and simple extraction. Complex multi-step reasoning still benefits from larger centralised models.

How do I update edge models?
For browser WebGPU, version the model file and use service worker caching. For edge workers, use the platform’s deployment pipeline. For on-device, follow the platform’s app-update mechanism.

What about quality?
Quantised small models (4-bit, 8-bit) achieve 90-95 percent of full-precision quality on extraction tasks. Validate per use case.

How does this interact with cost?
Edge AI shifts cost from inference per-token to development complexity. The break-even depends on volume. Above one million requests per month edge often wins.

Common pitfalls in edge AI scraping deployments

Three failure modes show up consistently when teams move edge AI from prototype to production.

The first pitfall is silent quality regression after a model update. Cloudflare and similar platforms periodically refresh hosted model weights, and a model identifier like @cf/meta/llama-3.1-8b-instruct can point to different underlying weights over time. Pin specific model revisions where the platform allows, and run a daily eval against a fixed regression set so quality drops are caught within hours rather than weeks.

The second pitfall is treating the edge as stateful. Edge workers spin up and down across regions, and any state held in worker memory disappears between invocations. Scrapers that try to dedupe URLs in worker-local memory will see duplicates because two simultaneous workers in different regions hold different state. Push deduplication and rate-limit state to a shared store like Workers KV, Durable Objects, or a regional Redis.

The third pitfall is assuming WebGPU works everywhere. WebGPU shipped to most browsers by 2026 but coverage on older Android, locked-down enterprise browsers, and some mobile Safari versions remains spotty. A scraper that depends on browser-side WebGPU inference must implement a server-side fallback path and detect WebGPU availability at runtime, otherwise the pipeline silently produces no output for a segment of users.

The economics of edge versus centralised inference

The decision to run inference at the edge versus in a centralised GPU cluster is increasingly an economic decision rather than a technical one. The break-even point depends on volume, latency requirements, and privacy requirements.

For low-volume workloads (under 1 million inferences per month) centralised inference via API is typically cheapest. The fixed costs of edge deployment (model packaging, deployment pipeline, monitoring) outweigh the per-inference savings.

For medium-volume workloads (1-100 million inferences per month) edge becomes competitive. Cloudflare Workers AI, AWS Lambda with smaller models, and Vercel Edge offer per-request pricing that compares favourably to centralised API pricing. The decision typically rests on latency and privacy preferences.

For high-volume workloads (over 100 million inferences per month) edge typically wins materially. The marginal cost per inference at the edge approaches zero (the user device or the platform’s already-allocated resources), while centralised costs scale linearly.

The 2026 inflection has moved many real workloads into the edge-favouring zone. Classification, extraction, redaction, and short-form generation are increasingly profitable at the edge.

The model size frontier for edge

Edge deployment is constrained by model size. Browser WebGPU realistically supports 1-3 billion parameter models. Edge workers with hosted GPU support 7-13 billion parameter models. On-device with modern mobile silicon supports 1-7 billion parameter models depending on the device.

The 2024-2026 wave of small high-quality models (Phi-3, Llama 3.2, Mistral, Gemma) raised the quality floor at every size tier. A 3B-parameter model in 2026 outperforms a 13B model from 2023 on many extraction and classification tasks. The trend means edge-deployable models are increasingly capable of production-quality work.

Quantisation extends the frontier further. A 7B-parameter model quantised to 4-bit fits in roughly 4 GB of memory, which is achievable on modern phones and on Cloudflare Workers AI. Quantisation costs 1-3 percent quality on most tasks, which is usually acceptable for production extraction workloads.

Browser WebGPU as a deployment target

Browser WebGPU is the most exotic edge deployment target but also the most privacy-friendly. Inference happens entirely on the user’s device. No data leaves the browser. The site cost is the model file size (typically 1-3 GB for useful models).

The 2026 toolkit for WebGPU inference includes transformers.js (the JavaScript port of Hugging Face transformers), ONNX Runtime Web, and several specialised libraries. Each ships pre-quantised models that load quickly and run on consumer GPUs.

The user experience considerations for WebGPU inference include the model download (long on first visit, cached afterwards), the GPU memory consumption (must be considered alongside the page’s other GPU usage), and the inference latency (typically 100-500 ms per generation step, slow compared to centralised GPU but fast enough for many use cases).

A 2026 pattern that is gaining adoption is hybrid inference. The page first attempts WebGPU inference. If unavailable or unacceptably slow, the page falls back to an edge worker or centralised API. The fallback is invisible to the user but provides graceful degradation.

On-device inference for mobile and desktop

Mobile and desktop applications can ship inference models directly. Apple’s Core ML, Android’s Neural Networks API, and the cross-platform llama.cpp and MLC-LLM libraries provide the deployment pipelines.

The 2026 best practice for mobile on-device inference is to ship a quantised 1-3B parameter model with the app. The model handles common tasks (classification, summarisation, simple extraction) without network round-trips. Larger or more complex tasks fall back to a server.

Desktop deployment is less constrained by memory and battery. A desktop app can ship a 7-13B model and use the host GPU. The capability available is closer to centralised inference, with the privacy and latency advantages of local execution.

The 2026 release of high-quality 1-3B models that fit easily on consumer hardware made on-device inference economically attractive for the first time. Many applications that previously required server inference can now run locally with comparable quality.

Next steps

The fastest first move is to identify one high-volume AI task in your pipeline (classification, embedding, language detection) and prototype an edge implementation in Cloudflare Workers AI. The cost saving and latency improvement will speak for themselves. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the RAG over scraped data guide.

This guide is informational, not engineering or legal advice.

How to scrape Coupang Korea: a practical 2026 guide

Scrape Coupang Korea correctly in 2026 and you have access to the dominant ecommerce platform in one of the most digitally mature markets in the world. Coupang serves over 22 million active customers in South Korea, runs the country’s largest same-day delivery network (Rocket Delivery), and indexes hundreds of millions of SKUs ranging from groceries to electronics to fashion. For brand managers tracking pricing, agencies running competitive intelligence, or product teams sizing demand, Coupang is non-negotiable.

This guide covers Coupang Korea scraping end-to-end: which endpoints to hit, how to handle the bot defenses Coupang has stacked since their NYSE listing, how to deal with Korean character encoding and KRW pricing nuances, and how to manage Korean mobile carrier proxies. Working Python code throughout.

What Coupang Korea exposes

Three surfaces matter:

Surface	URL pattern	Best for
Product detail page	coupang.com/vp/products/{product_id}	Full extraction with reviews
Internal API	coupang.com/vp/products/{product_id}/items/{item_id}/vendor-items	Variant-level data
Search results	coupang.com/np/search?q={query}	Discovery

Coupang’s structure is more nested than Lazada or Shopee. A “product” can have multiple “items” (variants), each with multiple “vendor items” (different sellers offering the same item). For complete competitive intelligence, you need vendor-item-level data, not just product-level.

Anti-bot defenses

Coupang uses a custom bot defense stack assembled by the Coupang security team:

1. Cloudflare protection on the public web pages
2. Aggressive IP reputation scoring; data center IPs are heavily challenged
3. Custom JavaScript challenges that defeat headless Chromium with default settings
4. Header-based fingerprinting (specific Accept-Language and User-Agent combinations expected)

The recommended path in 2026: Korean mobile carrier IPs (KT, SK Telecom, LG U+), real Chromium driven through CDP with a Korean locale, and patient throttling.

Working browser-based scraper

import asyncio
import json
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup

async def scrape_coupang_kr(product_url: str, proxy: dict | None = None) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy=proxy,
            args=["--disable-blink-features=AutomationControlled"],
        )
        ctx = await browser.new_context(
            user_agent="Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1",
            locale="ko-KR",
            timezone_id="Asia/Seoul",
            extra_http_headers={"Accept-Language": "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7"},
            viewport={"width": 390, "height": 844},
        )
        page = await ctx.new_page()
        await page.goto(product_url, wait_until="networkidle", timeout=45000)
        html = await page.content()
        await browser.close()

    return _parse_coupang_html(html)

def _parse_coupang_html(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    title = soup.select_one("h2.prod-buy-header__title")
    price_el = soup.select_one(".total-price strong")
    original_price = soup.select_one(".price-amount.origin-price")
    rating_el = soup.select_one(".rating-star-num")
    review_count_el = soup.select_one(".count")
    stock_el = soup.select_one(".out-of-stock")

    return {
        "title": title.text.strip() if title else None,
        "price_krw": _parse_krw(price_el.text) if price_el else None,
        "original_price_krw": _parse_krw(original_price.text) if original_price else None,
        "rating": float(rating_el.get("style", "").replace("width:", "").replace("%;", "")) / 20 if rating_el else None,
        "review_count": int(review_count_el.text.strip("()").replace(",", "")) if review_count_el else None,
        "in_stock": stock_el is None,
    }

def _parse_krw(text: str) -> float:
    import re
    digits = re.sub(r"[^\d]", "", text or "")
    return float(digits) if digits else 0.0

asyncio.run(scrape_coupang_kr("https://www.coupang.com/vp/products/1234567890"))

Mobile user agent matters more here than on most sites. Coupang serves a more API-friendly (smaller, JSON-heavy) version to mobile clients.

Capturing the internal product API

Coupang’s product detail page makes several internal API calls. Intercepting them gives cleaner JSON than parsing HTML.

async def scrape_with_api_capture(url: str, proxy: dict | None = None) -> dict:
    api_payloads = {}
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True, proxy=proxy)
        ctx = await browser.new_context(locale="ko-KR")
        page = await ctx.new_page()

        async def handler(resp):
            if "/vp/products/" in resp.url and resp.status == 200:
                try:
                    if "application/json" in resp.headers.get("content-type", ""):
                        api_payloads[resp.url] = await resp.json()
                except Exception:
                    pass

        page.on("response", handler)
        await page.goto(url, wait_until="networkidle", timeout=45000)
        await asyncio.sleep(2)
        await browser.close()
    return api_payloads

The intercepted payloads include the structured price, stock, vendor, and review data without you having to parse HTML.

Korean Won price handling

Korean Won uses the symbol ₩ and is whole-number (no fractional units). Prices appear as “1,234,500원” or “₩1,234,500”. Strip everything that is not a digit to parse:

import re

def parse_krw(s: str) -> float:
    return float(re.sub(r"[^\d]", "", s) or 0)

Conversion rates fluctuate but rough USD ratio in 2026 is around 1,400 KRW per USD. Always store the raw KRW value; convert only for display.

Korean character handling

Korean uses Hangul (한글) which is well-supported by UTF-8. Two specific gotchas:

First, Hangul has both completed syllable blocks (가, 나) and decomposed forms (Jamo). Coupang uses completed forms. Make sure your storage layer normalizes via NFC.

import unicodedata

def normalize_korean(s: str) -> str:
    return unicodedata.normalize("NFC", s)

Second, product titles often mix Hangul, Latin (brand names like Samsung, LG, Apple), and CJK ideographs (some traditional terms). Storage and indexing should support all three.

Mobile proxy rotation

Korean mobile carrier IPs (KT, SK Telecom, LG U+) are the cleanest source for Coupang scraping. Korean residential IPs work for low volume; mobile is required for sustained throughput.

import random

KR_MOBILE_PROXIES = [
    {"server": "socks5://us:pw@kr-kt-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@kr-skt-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@kr-lgu-1.proxy.example.com:1080"},
]

async def scrape_with_proxy(url: str):
    proxy = random.choice(KR_MOBILE_PROXIES)
    return await scrape_coupang_kr(url, proxy=proxy)

For broader proxy strategy in Asia, see best mobile proxy providers 2026.

Discovering product URLs

Coupang’s category structure is deeply nested. Sitemap discovery works:

import httpx
import xml.etree.ElementTree as ET

async def list_coupang_sitemap_urls(limit: int = 5) -> list[str]:
    sitemap_index = "https://www.coupang.com/sitemap.xml"
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.get(sitemap_index)
        root = ET.fromstring(r.text)
        ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
        sitemaps = [s.find("sm:loc", ns).text for s in root.findall("sm:sitemap", ns)][:limit]

        urls = []
        for sm_url in sitemaps:
            r = await client.get(sm_url)
            sm_root = ET.fromstring(r.text)
            urls.extend(u.find("sm:loc", ns).text for u in sm_root.findall("sm:url", ns))
        return urls

Coupang category landing pages also expose paginated listings:

async def search_coupang(query: str, page: int = 1) -> list[dict]:
    url = f"https://www.coupang.com/np/search?q={query}&page={page}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="ko-KR")
        pg = await ctx.new_page()
        await pg.goto(url, wait_until="networkidle")
        items = await pg.locator(".search-product").all()
        results = []
        for item in items:
            href = await item.locator("a").first.get_attribute("href")
            title = await item.locator(".name").text_content()
            results.append({"url": f"https://www.coupang.com{href}", "title": title.strip() if title else ""})
        await browser.close()
    return results

Korean address and seller data

Korean ecommerce uses a unique address structure (시 / 도 / 군 / 구 / 동 hierarchy). Vendor location data on Coupang typically appears at the city or district level. For brand intelligence, normalize to a hierarchical structure:

KOREAN_REGIONS = {
    "Seoul": "서울특별시",
    "Busan": "부산광역시",
    "Gyeonggi": "경기도",
    # ...
}

def normalize_korean_region(text: str) -> str | None:
    for english, korean in KOREAN_REGIONS.items():
        if korean in text or english in text:
            return english
    return None

For PIPA compliance, store at city level, not specific addresses.

Comparison to other Asian markets

Market	Bot defense	Volume	Mobile proxy required
Coupang Korea	High	Largest in Korea	Yes
Naver Smart Store	High	Very large	Yes
Gmarket Korea	Medium	Large	Recommended
11Street Korea	Medium	Medium	Optional
Rakuten Japan	High	Largest in Japan	Yes
Amazon Japan	Medium	Largest in Japan	Optional

For Japan specifically, see our Rakuten Japan scraping guide.

Stealth fingerprint hardening for Coupang

Coupang’s Cloudflare integration trips on the standard headless Chromium fingerprint. Combine the AutomationControlled patch with realistic Korean mobile fingerprints:

context_init = """
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, 'languages', {get: () => ['ko-KR', 'ko', 'en']});
Object.defineProperty(navigator, 'platform', {get: () => 'iPhone'});
Object.defineProperty(screen, 'colorDepth', {get: () => 32});
"""

await ctx.add_init_script(context_init)

Additionally, Coupang weighs the order and casing of HTTP headers. Use extra_http_headers to send a Korean-realistic header set in the right order:

ctx = await browser.new_context(
    extra_http_headers={
        "Accept-Language": "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Sec-Ch-Ua-Platform": '"iOS"',
        "Sec-Ch-Ua-Mobile": "?1",
    },
)

These details push the bot score from “high” to “medium” on Coupang’s internal scoring, which is enough to keep the session alive.

Coupang Rocket vs Marketplace

Coupang sells in two modes. Coupang-fulfilled (Rocket Delivery) products are sold by Coupang directly. Marketplace products are sold by third parties through Coupang. Both appear on the same product page, often with multiple vendor offers.

For competitive intelligence, vendor-level data matters. A single product might have 20 different vendors offering it at 20 different prices. The product-level price is meaningless without the vendor breakdown.

async def scrape_coupang_vendors(product_id: int, item_id: int):
    url = f"https://www.coupang.com/vp/products/{product_id}/items/{item_id}/vendor-items"
    # fetch with browser session, parse vendor list
    pass

Cost optimization tactics

Three patterns specifically valuable for Coupang scraping:

Block image and font requests. Coupang product pages load 4 to 6 MB of imagery by default. Blocking via Playwright route interception cuts proxy bandwidth by 75 percent.

Cache vendor data per item. The vendor list rarely changes hourly. Refresh vendor data once per day for most items, more often only for hot SKUs.

Use the API capture pattern over HTML parsing. The intercepted JSON contains structured data; HTML parsing is brittle as Coupang ships frontend updates.

Combined, these cut typical per-page cost from $0.038 to $0.019, roughly half.

Korean ecommerce calendar awareness

Korean ecommerce has different peak periods than Western or ASEAN markets. Plan capacity around:

• Lunar New Year (Seollal): late January to mid-February. Surge in gift purchases.
• Pepero Day (November 11): minor spike (different from China’s Singles Day but on the same date).
• Coupang’s own anniversary sales: irregular schedule, usually late summer.
• Christmas and New Year: standard global peak.

During peak windows, expect 3x normal load on Coupang infrastructure plus more aggressive bot defense. Scale your IP pool by 2x and increase pacing margins.

Production patterns

Three patterns matter.

First, throttle conservatively. 1-2 requests per second per IP. Coupang challenges aggressive scrapers within minutes.

Second, capture warm sessions. Sessions that have visited the homepage, browsed a category, and visited an item have a much lower challenge rate than cold sessions.

Third, monitor for the Cloudflare interstitial. If your scraper starts hitting “Just a moment…” pages, your IP pool is being challenged. Pause and rotate.

Vendor-level data extraction

Coupang’s vendor-items endpoint is the only way to see all sellers offering a single SKU. The shape:

async def fetch_vendor_items(product_id: int, item_id: int, session_cookies: dict) -> list[dict]:
    url = (f"https://www.coupang.com/vp/products/{product_id}/items/{item_id}"
           f"/vendor-items")
    async with httpx.AsyncClient(cookies=session_cookies) as c:
        r = await c.get(url, headers={
            "Accept": "application/json",
            "User-Agent": "Mozilla/5.0 ...",
            "Referer": f"https://www.coupang.com/vp/products/{product_id}",
        })
        return r.json().get("vendorItems", [])

Each vendor item includes price, stock, shipping cost, vendor name, vendor rating, and delivery type. For brand intelligence (catching unauthorized resellers, monitoring grey-market pricing), this data is gold.

Real benchmarks

A March 2026 production run, 10,000 Coupang products with the API capture pattern:

Metric	Value
Success rate	91%
Median latency per item	5.8 s
p99 latency	18 s
Cost per 1000 items	$19
Cloudflare challenge rate	5.3%
429 throttle rate	1.4%

Cloudflare challenges are the leading failure cause. With proper stealth and IP rotation, you can keep the rate under 6 percent.

Storage schema

CREATE TABLE coupang_products (
    id BIGSERIAL PRIMARY KEY,
    product_id BIGINT NOT NULL,
    item_id BIGINT,
    vendor_item_id BIGINT,
    url TEXT NOT NULL,
    title TEXT NOT NULL,
    price_krw NUMERIC(12,0) NOT NULL,
    original_price_krw NUMERIC(12,0),
    rating NUMERIC(3,2),
    review_count INTEGER,
    in_stock BOOLEAN NOT NULL,
    is_rocket BOOLEAN DEFAULT FALSE,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    raw_jsonb JSONB,
    UNIQUE(product_id, item_id, vendor_item_id)
);
CREATE INDEX idx_coupang_extracted_at ON coupang_products(extracted_at);

AI-driven extraction fallback

For pages where the deterministic JSON interception fails (Coupang ships UI updates frequently), fall through to LLM extraction:

async def scrape_with_fallback(url: str) -> dict:
    try:
        return await scrape_with_api_capture(url)
    except (NoPayloadError, KeyError):
        html = await fetch_html(url)
        return await llm_extract_product(html, schema=PRODUCT_SCHEMA)

The LLM fallback runs at roughly 4x the cost per page but catches the cases where the deterministic path breaks. This hybrid keeps the happy path fast and cheap while staying resilient to frontend changes.

Cost expectations

10,000 Coupang Korea products per month with Korean mobile proxies:

Component	Cost
Korean mobile proxy traffic (~2.5MB/page)	$80-$130
Browser compute	$40
LLM extraction (optional)	$30
Total	$150-$200

Korean mobile IPs are slightly cheaper than Indonesian mobile, partly because Korean carrier infrastructure has more capacity.

Legal considerations

Korea’s Personal Information Protection Act (PIPA) is strict. Public commercial data (product listings, prices, vendor names) is not personal data. Customer reviews that include real names are personal data and require care; the typical compliance pattern is to extract only ratings and review counts, not review text or reviewer names.

The Coupang terms of service prohibit automated access. Civil enforcement only; no criminal exposure for scraping public commercial data.

For broader compliance reading, see GDPR compliance for web scraping, which covers many of the same principles applied to Korean PIPA.

Coupang-specific data quirks

Several Coupang-only data points that other ecommerce platforms do not expose:

Rocket Wow membership pricing. Members get different prices on many SKUs. The page renders both prices and Wow-only prices appear with a Wow badge. Capture both.

Coupang Card discount. Coupang’s branded credit card offers an automatic discount that appears on the product page. Capture as a separate field; it affects price comparison logic.

Same-day delivery flag. The “Rocket Delivery” badge indicates next-day or same-day delivery. For demand intelligence, this flag is correlated with sales velocity.

Origin country. Coupang labels imported products with origin country (China, Korea, USA, etc). For brand and trade intelligence, this is essential.

def extract_coupang_specific(page_data: dict) -> dict:
    return {
        "wow_price_krw": page_data.get("wowPrice"),
        "card_discount_krw": page_data.get("cardDiscountAmount"),
        "is_rocket_delivery": page_data.get("rocketDelivery", False),
        "origin_country": page_data.get("originCountry"),
    }

Review and rating extraction

Coupang reviews are paginated and load lazily. Each review includes star rating, text, photos, and a buyer-helpful counter. The endpoint:

async def fetch_reviews(product_id: int, page: int = 1, size: int = 30) -> dict:
    url = (f"https://www.coupang.com/vp/product/reviews"
           f"?productId={product_id}&page={page}&size={size}")
    # uses the same session cookies as product fetches
    ...

For sentiment analysis, capture the text plus rating. For authenticity (counterfeit detection), photos are a strong signal because genuine buyers post product photos and fake reviews rarely do.

Frequently asked questions

Can I use Coupang’s Partner API?
Coupang has an Affiliate Partner API for sellers and an Open API for partners. If you qualify, official APIs are the safe path. For competitive intelligence (you are not a seller), scraping is the practical option.

Why does my scraper work for an hour then start failing?
IP reputation degradation. Mobile IPs survive longer than residential, but every IP eventually gets flagged with sustained traffic. Rotate aggressively.

How does Coupang’s anti-bot compare to Naver Smart Store?
Naver is harder. Coupang relies on Cloudflare plus custom challenges; Naver has its own homegrown defense plus deep integration with Korean identity verification. For Naver scraping, expect 2x the cost and 30 percent lower success rate.

Can I scrape Coupang affiliate links?
The affiliate program API gives you tracked product URLs you can include in content. The scraping pattern for product data is the same; only the URL structure adds a tracking parameter.

Can I scrape Coupang Eats (food delivery)?
Yes with similar patterns. Coupang Eats has a mobile-first interface that works best with mobile user agents and Korean mobile IPs.

How do I detect when a Coupang product moves between Rocket and Marketplace?
Track the is_rocket flag over time. A change from true to false often signals supply chain or pricing changes that brand managers care about.

What about Coupang Play (streaming) metadata?
Title and synopsis data are scrapable. View counts and engagement data are not exposed publicly.

How do I handle the seller location data?
Vendor profiles include city-level location for marketplace sellers. Store at city granularity; scraping shop-level address details ventures into PIPA territory.

What about Coupang Fresh (groceries)?
Same scraping pattern with a slightly different URL structure (coupang.com/vp/products/{id} is universal but Fresh items have additional perishability and chilled-delivery flags).

Can I scrape Coupang from outside Korea?
Yes for the public web pages, but mobile carrier IPs from Korea perform dramatically better. From a US IP, expect a 3x challenge rate.

How do I track price changes accurately on Coupang?
Snapshot daily for stable products, hourly for hot deals. Coupang prices can change multiple times per day during 11.11-style sales.

Does Coupang have a search-suggest API I can use for keyword discovery?
Yes, at coupang.com/np/search/suggestion?q={prefix}. Useful for brand monitoring and trend tracking.

Common production gotchas

• The Cloudflare challenge cookie expires after 30 minutes. Sessions need refresh more often than on Lazada or Shopee.
• Korean character encoding in URLs uses %EC%-style percent-encoding. URL parsing libraries usually handle it but logging may show garbled text.
• Some Coupang pages require login for full pricing visibility (loyalty pricing). Scraping anonymously gets you the public price tier only.
• The mobile site (m.coupang.com) returns slightly different DOM than the desktop site. Pick one and stick with it.
• Vendor data updates more frequently than product data. Re-scrape vendors weekly even if products are stable.

Storing variant data

Coupang’s nested product/item/vendor-item structure deserves a normalized schema. A relational design that scales:

CREATE TABLE coupang_product_master (
    product_id BIGINT PRIMARY KEY,
    title TEXT NOT NULL,
    brand TEXT,
    category_id INTEGER,
    first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE coupang_items (
    item_id BIGINT PRIMARY KEY,
    product_id BIGINT REFERENCES coupang_product_master(product_id),
    variant_attributes JSONB NOT NULL,
    first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE coupang_vendor_items (
    vendor_item_id BIGINT PRIMARY KEY,
    item_id BIGINT REFERENCES coupang_items(item_id),
    vendor_id BIGINT NOT NULL,
    vendor_name TEXT,
    is_rocket BOOLEAN DEFAULT FALSE,
    first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE coupang_price_history (
    id BIGSERIAL PRIMARY KEY,
    vendor_item_id BIGINT REFERENCES coupang_vendor_items(vendor_item_id),
    price_krw NUMERIC(12,0) NOT NULL,
    in_stock BOOLEAN NOT NULL,
    captured_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_coupang_price_history_vendor_time
    ON coupang_price_history(vendor_item_id, captured_at);

This shape supports the most common queries (price over time per vendor, which vendors offer SKU X, average price across vendors) without requiring expensive joins.

For more Asian ecommerce coverage, browse the ecommerce category.

Vector databases for scraping pipelines in 2026

Vector databases scraping pipelines are inseparable in 2026. Almost every meaningful scraping operation that powers RAG, semantic search, recommendation, or AI-assisted analysis ends up writing embeddings to a vector store. The choice of vector database matters more than most teams initially recognise: it shapes ingestion throughput, query latency, hybrid retrieval support, operational overhead, cost economics, and the migration path when the system grows. The market consolidated around five serious options in 2024-2025, and the mid-2026 picture is clearer than ever. This guide walks through the production-grade vector databases, the comparison criteria that matter for scraping workloads, the deployment patterns that work, and a selection framework your team can apply.

The audience is the data engineer or platform owner choosing a vector database for a scraping-driven AI pipeline.

Why vector databases matter for scraping

Three reasons.

First, embedding storage and similarity search at scale require purpose-built infrastructure. A scraping operation that ingests 10 million documents produces tens of millions of vectors (one per chunk, often more). Storing and querying these in a general-purpose database does not scale.

Second, the retrieval pattern is different from traditional databases. Vector queries are nearest-neighbour searches over high-dimensional vectors, with hybrid sparse-plus-dense often required. The right database makes hybrid trivial; the wrong database makes it custom code.

Third, the operational characteristics matter: ingestion throughput, query latency at p99, memory footprint, replication, and cost per million vectors all shape the production experience.

For the broader RAG context, see RAG over scraped data production patterns. For the MCP integration, see MCP for data engineers.

The 2026 vector database landscape

Five production-grade options:

Database	Type	Open source	Hosted	Strength
Qdrant	Purpose-built	Yes (Apache 2.0)	Yes (Qdrant Cloud)	Performance + filters
Weaviate	Purpose-built	Yes (BSD)	Yes (Weaviate Cloud)	Modules + multi-modal
Pinecone	Purpose-built	No	Yes only	Operational simplicity
pgvector (Postgres extension)	Embedded	Yes	Yes (Supabase, Neon, RDS)	Postgres-native
Milvus	Purpose-built	Yes (Apache 2.0)	Yes (Zilliz Cloud)	Massive scale

The choice between them is rarely about raw performance. All five can serve millions of queries per day. The choice is about operational fit, ecosystem, and the rest of your stack.

Qdrant: the performance and filtering favourite

Qdrant is a purpose-built vector database written in Rust. It launched in 2021 and matured through 2023-2025 to become the production favourite for performance-sensitive workloads.

Strengths:
– Excellent query performance at scale (millions of vectors).
– Strong payload filtering: combine vector search with metadata filters efficiently.
– Open source with a permissive licence (Apache 2.0).
– Mature client libraries (Python, TypeScript, Go, Rust).
– Hybrid search (dense + sparse) supported natively as of 2024.

Weaknesses:
– Self-hosting requires more operational sophistication than pgvector.
– Hosted Qdrant Cloud is reasonably priced but not the cheapest.
– Less ecosystem integration than Weaviate (modules) or pgvector (Postgres).

Best for: production scraping pipelines where filter combination and query performance matter; teams comfortable with self-hosted infrastructure.

A minimal Qdrant ingestion in Python:

from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

client = QdrantClient(host="qdrant.internal", port=6333)
client.recreate_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
)
client.upsert(
    collection_name="docs",
    points=[
        PointStruct(id=i, vector=embedding,
                    payload={"url": url, "scraped_at": ts, "text": text[:200]})
        for i, (embedding, url, ts, text) in enumerate(rows)
    ],
)

Weaviate: the modules and multi-modal favourite

Weaviate is purpose-built, written in Go, and launched in 2019. It has a stronger orientation around modular pipelines (built-in embedding generation, reranking, summarisation) and multi-modal data (images, audio, video alongside text).

Strengths:
– Modules: built-in connectors for OpenAI, Cohere, Hugging Face, ColBERT, and many more.
– Multi-modal native: cleaner support for cross-modal queries.
– GraphQL query language: convenient for complex retrieval.
– Open source (BSD licence) with managed hosting.

Weaknesses:
– More opinionated; the modular design adds complexity to simple use cases.
– Performance similar to Qdrant but the operational characteristics differ.
– Smaller community than pgvector or Pinecone.

Best for: multi-modal pipelines, teams that benefit from built-in embedding/reranking modules, GraphQL-friendly stacks.

Pinecone: the operational-simplicity choice

Pinecone is the original commercial vector database, launched 2019. It is hosted-only and proprietary. Its value proposition is operational simplicity: a managed service with predictable pricing, zero infrastructure ownership.

Strengths:
– Zero-ops: no self-hosting required.
– Predictable pricing model.
– Strong production reliability.
– Clean Python SDK, well-documented.

Weaknesses:
– Hosted-only; no self-host option.
– More expensive at scale than self-hosted alternatives.
– Closed source; no inspection of internals.
– Filter performance has historically lagged Qdrant.

Best for: teams that want to outsource vector database operations entirely; pre-production, smaller teams; situations where vendor lock-in is acceptable.

pgvector: the Postgres-native option

pgvector is a Postgres extension that adds vector data types and similarity search. It launched in 2021 and matured significantly in 2024-2025 with HNSW index support and improved performance.

Strengths:
– Postgres native: reuse existing Postgres expertise, tooling, backups, observability.
– Cost-effective when Postgres is already in the stack.
– Transactional consistency with relational data.
– Hosted everywhere (RDS, Supabase, Neon, CloudSQL).
– Open source, free.

Weaknesses:
– Performance lower than purpose-built databases at large scale (50M+ vectors).
– Index types (IVFFlat, HNSW) and parameter tuning require expertise.
– No native sparse-plus-dense hybrid search; requires combining with full-text search manually.

Best for: teams already running Postgres, smaller corpora (under 50M vectors), use cases where transactional consistency with relational data is valuable.

A pgvector setup with HNSW:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
  id BIGSERIAL PRIMARY KEY,
  embedding vector(1024),
  url TEXT,
  scraped_at TIMESTAMPTZ,
  text TEXT
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Milvus: the massive-scale option

Milvus is open source, written in C++ and Go, and designed for extreme scale (billions of vectors). It launched in 2019 and matured through 2024-2025 into the standard for the largest deployments.

Strengths:
– Scale: production deployments at billions of vectors and 10K+ QPS.
– Distributed architecture: separate compute and storage scale independently.
– Multiple index types (IVF_FLAT, IVF_SQ8, HNSW, DiskANN).
– Strong China-region adoption with mature Mandarin documentation.

Weaknesses:
– Operational complexity: distributed Milvus requires real DevOps investment.
– For corpora under 100M vectors, the complexity is overkill.
– The hosted version (Zilliz Cloud) is mature but less ecosystem-adopted.

Best for: extreme-scale deployments, teams with mature DevOps, organisations with massive scraping operations producing billions of vectors.

For the deeper proxy infrastructure question, see self-hosted proxy infrastructure.

Comparison matrix

Dimension	Qdrant	Weaviate	Pinecone	pgvector	Milvus
Open source	Yes	Yes	No	Yes	Yes
Self-hosting	Yes	Yes	No	Yes (via Postgres)	Yes
Managed hosting	Yes	Yes	Yes only	Yes (Supabase, Neon, RDS)	Yes (Zilliz)
Hybrid search native	Yes	Yes	Limited	Manual	Yes
Multi-modal	Limited	Strong	Limited	Limited	Strong
Filter performance	Excellent	Good	Moderate	Good	Excellent
Scale ceiling	100M+ vectors	100M+ vectors	100M+ vectors	50M vectors	10B+ vectors
Ecosystem maturity	High	High	High	High (Postgres)	High
Best client lang	Python, TS, Go	Python, GraphQL	Python	SQL	Python, Java, Go
Cost at 10M vectors	Low (self) / Medium (cloud)	Low (self) / Medium (cloud)	Medium-High	Low (self)	Low (self)
Cost at 1B vectors	High (self)	High (self)	Highest (cloud only)	Not recommended	Best for scale

Decision tree: pick a vector database

Q1: Is your team running Postgres already?
    ├── Yes -> Q2
    └── No  -> Q3
Q2: Will the corpus stay under 50M vectors for the next 18 months?
    ├── Yes -> pgvector. Reuse the stack.
    └── No  -> Q3
Q3: Is operational simplicity (no self-host) the priority?
    ├── Yes -> Pinecone (commercial); Qdrant Cloud or Weaviate Cloud (open).
    └── No  -> Q4
Q4: Is the corpus at 1B+ vectors or expected to be?
    ├── Yes -> Milvus.
    └── No  -> Q5
Q5: Are filter combinations central to your queries?
    ├── Yes -> Qdrant.
    └── No  -> Q6
Q6: Do you need built-in modules or strong multi-modal?
    ├── Yes -> Weaviate.
    └── No  -> Qdrant (sensible default).

The decision tree handles 80 percent of cases. Edge cases (regulatory data residency, specific cloud provider lock-in, language-specific embedding tooling) override.

Production deployment patterns

Three patterns for production deployment.

Pattern one: managed cloud, single region. Pinecone, Qdrant Cloud, Weaviate Cloud, Zilliz Cloud, or Supabase pgvector. Lowest operational overhead. Suitable for most teams.

Pattern two: self-hosted on Kubernetes. Qdrant, Weaviate, Milvus, or pgvector deployed via Helm charts on EKS/GKE/AKS. Medium operational overhead. Required for data-residency or cost-sensitive deployments.

Pattern three: self-hosted on bare metal. Same databases as pattern two, deployed directly on dedicated hardware. Highest operational overhead, lowest cost per vector. Required for the largest deployments where cloud egress and managed-service margins dominate.

For the broader infrastructure question, see building scraping pipelines with Prefect 3.

Operational characteristics

Beyond the headline benchmark numbers, the operational characteristics that matter:

Characteristic	Why it matters
Ingestion throughput	Determines time to backfill a large corpus
Query p99 latency	Determines user-facing response time
Memory footprint	Determines hosting cost
Index build time	Determines time-to-first-query after data load
Replication and HA	Determines uptime SLO
Backup and restore	Determines disaster recovery RTO/RPO
Schema evolution	Determines pain when payload schema changes
Multi-tenancy	Determines whether you can run shared collections

A team that picks a database without evaluating these often discovers them painfully in month two of production.

Hybrid search implementation

Hybrid retrieval (dense embedding + sparse keyword) is the production default in 2026. The implementation differs across databases:

Database	Hybrid mechanism
Qdrant	Native sparse vectors; combine with dense via fusion
Weaviate	Native hybrid query mode
Pinecone	Sparse-dense indexes (sparse vectors with dense)
pgvector	Combine with Postgres full-text search; manual fusion
Milvus	Native hybrid via multi-field queries

A Qdrant hybrid query:

from qdrant_client.models import Prefetch, Fusion, FusionQuery

results = client.query_points(
    collection_name="docs",
    prefetch=[
        Prefetch(query=dense_vector, using="dense", limit=20),
        Prefetch(query=sparse_vector, using="sparse", limit=20),
    ],
    query=FusionQuery(fusion=Fusion.RRF),
    limit=5,
)

Reciprocal Rank Fusion (RRF) is the standard merger.

For the deeper retrieval discussion, see RAG over scraped data.

Cost economics at scale

Rough cost benchmarks for storing and querying 100M vectors of 1024 dimensions in mid-2026:

Option	Storage cost (monthly)	Query cost per 1M	Notes
Qdrant Cloud	USD 800-1500	Included up to volume	Predictable
Weaviate Cloud	USD 900-1800	Included up to volume	Module surcharges
Pinecone	USD 1200-2500	USD 0.40	Tier-based
pgvector on RDS	USD 600-1200	Included	Suboptimal at this scale
Milvus self-hosted	USD 400-800	Included	Plus DevOps overhead
Qdrant self-hosted	USD 300-600	Included	Plus DevOps overhead

Self-hosting wins on raw cost. Managed wins on total cost of ownership when DevOps capacity is constrained.

External references

The Qdrant documentation is at qdrant.tech/documentation. Weaviate’s documentation is at weaviate.io/developers/weaviate. Pinecone’s docs are at docs.pinecone.io. pgvector’s repository is at github.com/pgvector/pgvector. Milvus is at milvus.io. Vector database benchmarks are tracked at vectordbbench.com.

Migration patterns

Teams frequently migrate vector databases as scale grows. The pattern that works:

1. Start with pgvector if Postgres exists, or Qdrant if not.
2. Migrate when corpus or QPS exceeds the comfortable operating range (50M vectors for pgvector; 100M+ for purpose-built single-node).
3. Plan migration as: dual-write during transition; read-cutover after validation; old database retired after 2 weeks.
4. Embedding models do not need to change unless the migration coincides with a model upgrade. Vector dimensions must match.

A typical migration is one engineer-month of effort. The cost is real but predictable.

FAQ

Which vector database is best for scraping pipelines?
Qdrant is the sensible default for most. pgvector if you already run Postgres at sub-50M scale. Pinecone if zero-ops is paramount.

Do I need a vector database if I use OpenAI embeddings?
Yes. The embeddings need to be stored and searched somewhere. OpenAI provides embeddings; vector databases provide retrieval.

Is pgvector good enough for production?
Yes, up to about 50M vectors. Beyond that, the index build times and query latencies push toward purpose-built options.

What about hybrid sparse-plus-dense search?
Native in Qdrant, Weaviate, and Milvus. Manual in pgvector. Limited in Pinecone unless using sparse-dense indexes.

Can I run multi-tenant collections?
All five support some form of multi-tenancy via collection separation or payload filtering. Implementation differs.

Extended vector database analysis

The vector database market consolidated around several production-ready options in 2026. The choice depends on scale, latency, and operational preference.

• pgvector on PostgreSQL. Best for teams already running Postgres. Hits 100M vector scale comfortably with HNSW indexing. Strong filter performance via standard SQL.
• Qdrant. Rust-based, excellent filter performance, strong for hybrid search. Self-hosted or managed.
• Weaviate. Schema-driven, GraphQL surface, strong for multi-tenancy.
• Pinecone. Managed only, simplest operations, highest cost per vector.
• Milvus. High scale (billions of vectors), more operational complexity.
• LanceDB. Embedded, columnar, best for analytics-style workloads.

Production ingestion pattern with deduplication

import hashlib
from typing import List, Dict

class IngestionPipeline:
    def __init__(self, embedder, vector_store, dedupe_table):
        self.embedder = embedder
        self.vector_store = vector_store
        self.dedupe = dedupe_table

    def doc_hash(self, doc: Dict) -> str:
        content = f"{doc['url']}|{doc['text']}"
        return hashlib.sha256(content.encode()).hexdigest()

    async def ingest(self, docs: List[Dict]) -> int:
        new_docs = []
        for doc in docs:
            h = self.doc_hash(doc)
            if not await self.dedupe.exists(h):
                doc["_hash"] = h
                new_docs.append(doc)
        if not new_docs:
            return 0
        embeddings = await self.embedder.embed_batch(
            [d["text"] for d in new_docs], batch_size=64
        )
        records = [{
            "id": d["_hash"],
            "vector": e,
            "metadata": {
                "url": d["url"],
                "scraped_at": d.get("scraped_at"),
                "source": d.get("source"),
                "purpose": d.get("purpose"),
            },
        } for d, e in zip(new_docs, embeddings)]
        await self.vector_store.upsert(records)
        for d in new_docs:
            await self.dedupe.add(d["_hash"])
        return len(new_docs)

Index choice matters

The HNSW vs IVF-PQ vs DiskANN choice affects latency, recall, and memory.

• HNSW. In-memory graph index. Best recall and latency. RAM-bound.
• IVF-PQ. Quantised inverted file. Memory-efficient at moderate recall cost.
• DiskANN. Disk-based graph index. Scales beyond RAM. Slightly higher latency.
• ScaNN. Google’s hybrid. Strong recall and speed in benchmarks.

A 2026 production choice often pairs HNSW for the hot tier (last 30 days, in-memory) with IVF-PQ or DiskANN for the cold tier.

Filter performance: pre-filter vs post-filter

Pre-filtering applies the metadata filter before the vector search. Post-filtering applies it after. Pre-filter is correct but slower when filter selectivity is low. Post-filter is faster but may return fewer than k results.

A 2026 pattern is adaptive filtering. The query planner estimates filter selectivity and chooses pre or post per query.

def adaptive_search(query_vec, filter_dict, k=10, selectivity_threshold=0.05):
    estimated = estimate_selectivity(filter_dict)
    if estimated < selectivity_threshold:
        return vector_store.search(query_vec, k=k, filter=filter_dict, mode="pre")
    else:
        return vector_store.search(query_vec, k=k*3, filter=filter_dict, mode="post")[:k]

Comparison: vector databases 2026

DB	Max scale	Latency p95	Filter perf	Best for
pgvector	100M+	20-50ms	Excellent (SQL)	Postgres shops
Qdrant	1B+	10-30ms	Excellent	Hybrid search
Weaviate	500M+	15-40ms	Good	Multi-tenant
Pinecone	Multi-billion	30-100ms	Moderate	Managed simplicity
Milvus	10B+	10-50ms	Good	Massive scale
LanceDB	100M+	20-60ms	Good	Embedded analytics

Cost optimisation patterns

Vector database costs follow three drivers.

1. Storage (per million vectors per month).
2. Compute for index building and query.
3. Network egress for hosted services.

The 2026 cost-cutting patterns are.

• Quantisation (PQ, OPQ, scalar quantisation) to reduce vector size 4x to 32x.
• Dimensionality reduction via Matryoshka embeddings to truncate vectors at query time.
• Tiered storage with hot, warm, cold partitions.
• Embedding model swap to a smaller model for cost-sensitive workloads.

Additional FAQ

Should I use a vector DB or a SQL extension?
For most teams pgvector is enough up to 100M vectors. Beyond that consider a dedicated vector DB.

How do I handle incremental updates?
HNSW supports insert efficiently. Delete is harder; many production systems use tombstones plus periodic reindex.

What about hybrid search?
Most modern vector DBs support BM25-plus-vector fusion via reciprocal rank fusion or weighted scores. Use it.

How do I version embeddings?
When the embedding model changes you must re-embed. Maintain two indexes during the transition and dual-write.

The choice between embedded and dedicated vector databases

A team building a scraping-plus-RAG pipeline faces an early decision: embedded vector database or dedicated. Embedded options (LanceDB, Chroma in single-node mode, FAISS) live in the application process. Dedicated options (Qdrant, Weaviate, Milvus, Pinecone) live in their own service.

Embedded wins for prototyping, single-node deployments, and analytics-style workloads where the embedding store is read more than written. Dedicated wins for multi-service deployments, multi-team usage, high-write workloads, and operational requirements like backup and replication.

The 2026 pattern is to start embedded for the first 90 days of a project and migrate to dedicated when the project graduates to production. The migration is non-trivial but well-traveled. The embedded prototype validates the data model and the schema before the dedicated commitment.

A specific case worth calling out is pgvector. pgvector lives in PostgreSQL and benefits from the operational maturity of Postgres. For teams that already run Postgres, pgvector is often the right answer up to 100 million vectors. The savings on operating an additional database service often outweigh the marginal performance benefits of a dedicated vector DB at moderate scale.

The embedding model decision

The embedding model is the foundation of every vector store. A change of embedding model requires re-embedding the entire corpus, which is expensive at scale. The model choice should therefore be considered carefully.

The 2026 leaderboard (MTEB) lists models by retrieval quality on standard benchmarks. The current state of the art for English is around 70+ on the average MTEB score. The popular open-weight choices include the BGE family, the Jina embeddings v3, and the Cohere multilingual embeddings (proprietary but well-regarded).

Dimensionality matters for storage and latency. A 1536-dimensional embedding takes 6 KB per vector at float32. A 768-dimensional embedding takes 3 KB. Smaller dimensions also support faster nearest neighbour search. The 2024 Matryoshka representation learning approach lets a single model produce embeddings that can be truncated to smaller dimensions with graceful quality degradation.

For multilingual scraping the choice narrows. Models trained on diverse language data perform better on cross-lingual queries. The Cohere multilingual embeddings, the BGE-M3 model, and the Jina multilingual variants are the strong open choices.

The chunking strategy decision

Chunking is the most underrated decision in vector pipelines. The chunk size, the chunking strategy, and the overlap all materially affect retrieval quality.

Fixed-size chunking (split every 512 tokens) is simple but breaks semantic boundaries. Document-structure-aware chunking respects headings, paragraphs, code blocks, and tables. Semantic chunking uses a small model to find sentence boundaries that preserve meaning.

The 2026 pattern is to use document-structure-aware chunking as the default, with semantic chunking for prose-heavy content where structure is weak. Fixed-size chunking remains useful as a fallback for content with no exploitable structure.

Overlap improves retrieval quality at the cost of storage. A typical 2026 default is 10-20 percent overlap. Below 10 percent the boundary effects degrade retrieval. Above 20 percent the marginal benefit is small.

Cost optimisation in production

A production vector store at scale costs real money. Three optimisation patterns drive material savings.

The first is quantisation. Standard float32 vectors can be compressed to int8 (4x reduction) or int4 (8x reduction) with small recall impact. Product quantisation goes further (32x reduction) with more recall impact. The 2026 best practice is to ship int8 quantised indexes for hot data.

The second is tiered storage. Hot data (last 30 days) lives in HNSW in-memory. Warm data (last year) lives in a quantised disk-backed index. Cold data (older) lives in compressed object storage with on-demand re-indexing. The tier boundaries are workload-specific.

The third is matryoshka truncation. A model trained for matryoshka representation produces embeddings that work at multiple dimensionalities. The vector store can store the full dimension and serve at a smaller dimension when latency matters, or vice versa.

Each pattern stacks. A vector store using all three can be 50-100x cheaper than a naive deployment of the same workload, with modest recall impact. The 2026 best practice is to layer the techniques rather than choose one.

Next steps

If you have not picked a vector database yet, evaluate Qdrant and pgvector for your specific workload. An afternoon of prototyping with both will tell you more than weeks of comparison reading. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the RAG over scraped data guide.

This guide is informational, not engineering or legal advice.