How to scrape Rakuten Japan in 2026

Scrape Rakuten Japan and you encounter a marketplace shape that does not exist anywhere else in the world. Rakuten Ichiba is a federation of more than 50,000 individual seller storefronts hosted under the Rakuten umbrella, each with its own brand, layout, and pricing logic. Unlike Amazon Japan where every listing is normalized into a single product page, Rakuten lets every merchant publish their own product page for the same item, which means the same EAN can have hundreds of listings across hundreds of sellers. The scraping landscape is shaped by three things: the official Rakuten Ichiba Item API which is generous but rate-limited, the per-merchant storefronts which require HTML scraping for the long tail, and a Japanese-language catalogue that requires careful encoding handling.

This guide focuses on Rakuten Ichiba (the marketplace), which is the dominant property under rakuten.co.jp. Rakuten Travel, Rakuten Books, and Rakuten Mobile use related but distinct APIs.

The official Rakuten Ichiba Item API

Rakuten publishes a free public API at https://app.rakuten.co.jp/services/api/IchibaItem/Search/20220601 that searches the entire Ichiba catalogue. Registration is required to obtain an applicationId, which serves as the rate-limit token. The free tier allows roughly 1 request per second per applicationId, which sounds restrictive but is enough for substantial workloads if you parallelize across multiple registered application IDs.

import httpx
import asyncio

API_URL = "https://app.rakuten.co.jp/services/api/IchibaItem/Search/20220601"

async def search_ichiba(keyword: str, app_id: str, page: int = 1, hits: int = 30):
    params = {
        "applicationId": app_id,
        "keyword": keyword,
        "page": page,
        "hits": hits,
        "sort": "-updateTimestamp",
        "format": "json",
    }
    async with httpx.AsyncClient(timeout=20) as c:
        r = await c.get(API_URL, params=params)
        if r.status_code == 200:
            return r.json().get("Items", [])
        return []

The API returns up to 100 hits per request, paginated to 100 pages, for a maximum of 10,000 hits per query. To exceed that, decompose the query by genre, price band, or shop. The genreId parameter accepts the Rakuten genre tree which is published separately at the IchibaGenre Search API.

Each item response includes itemCode (the canonical Rakuten SKU as <shopcode>:<itemnumber>), itemName, itemPrice in JPY, shopCode, shopName, reviewCount, reviewAverage, imageFlag, and a list of small/medium/large image URLs. For most analytical use cases, the API alone is sufficient and you do not need to scrape HTML.

When you still need HTML scraping

The official API does not expose seller storefront banners, custom seller categorizations, or the freeform descriptions that Rakuten merchants write on their own pages. For brand monitoring use cases that need to verify how a specific seller is presenting a product, you have to scrape the merchant storefront HTML.

Storefront URLs look like https://item.rakuten.co.jp/<shopcode>/<itemnumber>/. The HTML is encoded as Shift-JIS or UTF-8 depending on the merchant template, so always read the Content-Type header and decode accordingly. Encoding bugs are the most common silent failure in Rakuten scraping projects.

import httpx
from bs4 import BeautifulSoup

async def fetch_storefront(shop_code: str, item_number: str, proxy: str):
    url = f"https://item.rakuten.co.jp/{shop_code}/{item_number}/"
    async with httpx.AsyncClient(proxy=proxy, timeout=20) as c:
        r = await c.get(url)
        if r.status_code != 200:
            return None
        ctype = r.headers.get("content-type", "").lower()
        if "shift_jis" in ctype or "shift-jis" in ctype:
            text = r.content.decode("shift_jis", errors="replace")
        else:
            text = r.text
        soup = BeautifulSoup(text, "lxml")
        return {
            "title": soup.select_one("title").get_text(strip=True) if soup.select_one("title") else None,
            "html_size": len(text),
        }

Storefront scraping requires Japanese residential proxies. Rakuten’s CDN serves significantly slower paths to non-Japanese visitors, and merchant storefronts often refuse non-JP traffic outright if they have enabled the country lock setting in their seller dashboard.

Genre-based catalogue discovery

Rakuten’s genre tree is the most reliable way to enumerate the catalogue. The IchibaGenre Search API returns the genre hierarchy starting from a root node. By recursively walking the tree, you can map every leaf genre and then issue Item Search queries against each leaf to collect SKUs.

GENRE_API = "https://app.rakuten.co.jp/services/api/IchibaGenre/Search/20140222"

async def fetch_children(genre_id: int, app_id: str):
    params = {"applicationId": app_id, "genreId": genre_id, "format": "json"}
    async with httpx.AsyncClient(timeout=20) as c:
        r = await c.get(GENRE_API, params=params)
        if r.status_code == 200:
            return r.json().get("children", [])
        return []

The genre tree has roughly 50,000 leaf nodes. Walking it once takes about 14 hours at 1 request per second. Cache the result in a database and refresh quarterly. The structure changes slowly enough that quarterly is sufficient for most use cases.

Multi-tenant pricing and seller analytics

Because the same EAN can be sold by multiple Rakuten merchants, the same physical product often appears at very different prices across the marketplace. For brand owners, this matters because it surfaces parallel imports, gray market listings, and unauthorized resellers. For competitive intelligence, it surfaces the price floor that the most aggressive seller is willing to defend.

The pattern is to group items by EAN (when published in the API response) or by a fuzzy match on itemName and merchant brand. Grouping by EAN is cleaner but only works for SKUs where the merchant published it. For the long tail, fuzzy match by normalized title and brand is the practical fallback.

Field	Source	Reliability
itemCode	API	Canonical SKU per merchant
EAN/JAN	API attributes	About 60% coverage
itemName	API	Always present, often noisy
itemPrice	API	Authoritative
shopCode	API	Canonical merchant ID

For grouping across merchants, build a normalization pass that strips merchant tags from the title (sellers add [Free shipping] and [Same day ship] markers liberally), normalizes character widths, and removes common Japanese boilerplate.

Detecting and routing around CAPTCHA challenges on Rakuten

When Rakuten flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Japan residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Working with JPY pricing and FX normalization

Pricing in Japan is denominated in JPY, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local JPY and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons.

Comparing Rakuten to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Rakuten Ichiba	Japan	Large	High
Amazon Japan	Adjacent markets	Medium	Medium
Yahoo Shopping	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Rakuten Ichiba but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for Japan

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Japan has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="rakuten-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Rakuten datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks.

The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing. Each pass merges groups produced by the previous pass.

import hashlib
from PIL import Image
import imagehash

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

def normalize_title(title: str) -> str:
    title = title.lower()
    for token in ["[free shipping]", "[same day]", "(new)", "*sale*"]:
        title = title.replace(token, "")
    return " ".join(title.split())

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups). Both failure modes break downstream analytics.

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. New attribute fields appear, old fields are deprecated, category trees are reorganized, and pricing display logic changes. Your scraper code has to evolve with these changes, and a versioning pattern that keeps old data interpretable is critical.

The pattern that works best is to stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

ALTER TABLE snapshot ADD COLUMN scraper_version VARCHAR(16);
CREATE INDEX scraper_version_idx ON snapshot(scraper_version);

Pair this with a small registry table that documents what each scraper version did differently. When a downstream user asks why a particular metric jumped on a specific date, the version registry usually has the answer.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle.

The first signal is freshness deadline. Every SKU has a maximum staleness budget (say 24 hours), and any SKU older than its budget gets refreshed.

The second signal is volatility. SKUs that have changed price recently get higher refresh priority because they are more likely to change again. SKUs that have been stable for weeks can drop to a longer refresh interval.

The third signal is business priority. SKUs that downstream users actually query (tracked by query logs) get higher refresh priority than dormant SKUs that nobody has looked at in months.

def schedule_refresh(sku_id: int, last_changed_at, last_queried_at, last_fetched_at) -> int:
    """returns priority score; higher = refresh sooner"""
    age = (now - last_fetched_at).total_seconds() / 3600
    volatility = 10 if (now - last_changed_at).days < 7 else 1
    relevance = 5 if (now - last_queried_at).days < 1 else 1
    return age * volatility * relevance

This kind of priority-driven scheduler reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

Common pitfalls when scraping Rakuten Ichiba

Three issues dominate Rakuten Ichiba scraping. The first is shop-vs-product attribution. Rakuten is fundamentally a marketplace of independent shops, each with its own URL pattern, stock policy, and shipping terms. The same JAN code can appear in 50+ shops at different prices. Aggregating to the JAN level without preserving shop_id loses the price-dispersion signal that makes Rakuten data interesting.

The second is point-multiplier inflation. Rakuten campaigns layer point bonuses (5x, 10x, even 20x on Super Sale days). The visible price is in JPY but the effective price after points can be 10-15% lower for SPU members. A scraper that ignores the point multiplier field misreports the realized cost to the buyer. Capture both price and point_multiplier and compute the effective price downstream.

The third is character encoding edge cases. Some legacy shops still serve Shift-JIS or EUC-JP rather than UTF-8. Modern Rakuten infrastructure normalizes most pages to UTF-8 but shop-hosted product descriptions can leak through with the original encoding. Detect encoding per response rather than assuming UTF-8 globally.

FAQ

Do I need a Japanese IP to use the official API?
No. The IchibaItem Search API is accessible from anywhere in the world with a valid applicationId. Only the merchant storefront HTML scraping requires Japanese IP addresses for reliable access.

How many applicationIds can I register?
Rakuten allows multiple applicationIds per developer account. The terms of service require each applicationId to correspond to a real application, and aggressive parallelization across many IDs can trigger account-level throttling. For most production use, 3 to 5 IDs is sufficient and avoids the account-level scrutiny.

Does the API include stock counts?
No. The IchibaItem response includes an availability flag but not numerical stock. For stock-level data you have to scrape the storefront HTML, where merchants sometimes publish real-time stock indicators.

What about Rakuten Books and Rakuten Travel?
Rakuten publishes separate APIs for each major property: BooksTotal Search, TravelHotel Search, and so on. They follow the same registration model and similar rate limits. The applicationId is shared across all Rakuten APIs, so a single registration unlocks the whole family.

Can I scrape Rakuten reviews via the API?
The official API does not expose review text, only aggregate scores and counts. To collect review text you have to scrape the storefront HTML, which raises personal data considerations because reviewer display names are visible. Limit your collection to anonymized aggregates or work with explicit consent if reviewer identity matters.

Is the official Rakuten Ichiba API still useful in 2026?
Yes for catalog discovery and basic price data, but rate limits make it impractical for minute-level monitoring. Most production scrapers blend the official API for breadth with targeted scraping for depth.

How do I track Rakuten Super Sale price drops accurately?
Take baseline snapshots 14 days before the event and compare against intra-event hourly snapshots. Many sellers raise prices in the week before to inflate the apparent discount.

To pair this with a broader Asia ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape Jumia Africa product listings

Scrape Jumia Africa and you are scraping the largest pan-African ecommerce platform, with separate country instances for Nigeria, Kenya, Egypt, Morocco, Ivory Coast, Senegal, Ghana, Uganda, and a handful of other markets. Jumia was founded in 2012 and went public on the NYSE in 2019, and by 2026 it remains the dominant horizontal marketplace across Sub-Saharan Africa with category mixes that lean heavily into mobile phones, home appliances, and fashion. The scraping landscape is shaped by three things: per-country subdomains with different catalogues, a pricing system that mixes Jumia direct, Jumia Mall verified sellers, and a long tail of independent merchants, and an anti-bot layer that becomes more aggressive on the larger Nigeria and Egypt domains.

This guide focuses on Jumia Nigeria as the canonical example, with notes on cross-country differences. The patterns transfer to every Jumia country instance with minor adjustments to the domain and currency.

Mapping the Jumia URL and listing structure

Each Jumia country lives at its own subdomain: www.jumia.com.ng for Nigeria, www.jumia.co.ke for Kenya, www.jumia.com.eg for Egypt, www.jumia.ci for Ivory Coast, and so on. Within each country, the URL structure is consistent: https://www.jumia.com.ng/<product-slug>.html for product detail pages and https://www.jumia.com.ng/<category-slug>/ for category listings. Product slugs include a SKU identifier embedded near the end, which is the canonical key you should store.

Jumia exposes a server-rendered HTML front end with a JavaScript-hydrated layer for filters and recommendations. There is no fully public JSON API, but the product detail pages embed a <script type="application/ld+json"> block that carries Schema.org Product data including price, availability, brand, and SKU. Parsing this JSON-LD block is dramatically more reliable than scraping the visible HTML because the schema is stable while the visible markup changes regularly.

import httpx
from bs4 import BeautifulSoup
import json

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept-Language": "en-NG,en;q=0.9",
}

def parse_product(html: str) -> dict:
    soup = BeautifulSoup(html, "lxml")
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string)
        except (json.JSONDecodeError, TypeError):
            continue
        if isinstance(data, dict) and data.get("@type") == "Product":
            return {
                "sku": data.get("sku"),
                "name": data.get("name"),
                "brand": (data.get("brand") or {}).get("name"),
                "price": (data.get("offers") or {}).get("price"),
                "currency": (data.get("offers") or {}).get("priceCurrency"),
                "availability": (data.get("offers") or {}).get("availability"),
                "rating": (data.get("aggregateRating") or {}).get("ratingValue"),
            }
    return {}

async def fetch_product(url: str, proxy: str) -> dict:
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code != 200:
            return {}
        return parse_product(r.text)

Jumia’s JSON-LD includes the offer price but not the seller breakdown. To get the per-seller information, you have to parse the seller block from the HTML directly. The seller block is structured around a data-merchant-name attribute that is consistent across countries.

Country-specific proxy strategy for Jumia

Jumia’s bot detection becomes meaningfully stricter on the larger country instances. Nigeria and Egypt see the highest scrutiny, both because of the volume of legitimate scraping that already happens (price intelligence vendors, brand monitoring tools, affiliate networks) and because of the volume of fraud activity that uses the same patterns. Kenya and Ivory Coast see lighter enforcement.

Your proxy strategy should match the country you are scraping. For Nigeria and Egypt, use residential or mobile IPs in-country. MTN, Airtel, and 9mobile mobile pools work well for Nigeria. Vodafone and Etisalat residential pools work for Egypt. For the smaller markets, pan-African residential pools or even GCC-region pools sometimes work because Jumia’s geo-classification is less granular for low-volume countries.

Country	Recommended proxy origin	Tolerance per IP
Nigeria	Nigerian mobile or residential	100 req/hr per IP
Egypt	Egyptian residential	100 req/hr per IP
Kenya	Kenyan or pan-African residential	200 req/hr per IP
Morocco	Moroccan or French residential	200 req/hr per IP
Ivory Coast	Pan-African or French residential	250 req/hr per IP
Senegal	Pan-African or French residential	250 req/hr per IP

The cost differential between in-country residential and pan-African residential pools is significant for Nigerian inventory specifically. For most production workloads, the cost is justified by the success rate uplift. For lighter monitoring workloads under 1,000 SKUs per day, pan-African pools sometimes work even for Nigeria with careful rate limiting.

Crawling category trees and pagination

Category listings on Jumia are paginated with a ?page=N query parameter. The maximum reachable page depends on the category but typically caps at 50 pages of 40 products each. To go deeper, decompose by sub-category, brand facet, or price band. The category pages also embed a JSON-LD ItemList block that gives you structured access to the listing.

async def crawl_category(base_url: str, max_pages: int, proxy_pool):
    items = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = f"{base_url}?page={page}"
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url)
            if r.status_code != 200:
                continue
            soup = BeautifulSoup(r.text, "lxml")
            cards = soup.select("article.prd")
            if not cards:
                break
            for card in cards:
                items.append({
                    "sku": card.get("data-sku"),
                    "name": card.select_one("h3.name").get_text(strip=True) if card.select_one("h3.name") else None,
                    "price": card.select_one("div.prc").get_text(strip=True) if card.select_one("div.prc") else None,
                    "url": "https://www.jumia.com.ng" + card.select_one("a.core")["href"] if card.select_one("a.core") else None,
                })
    return items

For very broad categories like Phones and Tablets in Nigeria, the visible pagination only covers the first 2,000 SKUs. The remaining tail requires faceted decomposition. Build a recursive crawler that subdivides any category exceeding 2,000 results into brand and price-band buckets until each bucket fits within the pagination cap.

Tracking Jumia Mall vs. third-party merchant signal

Jumia distinguishes between Jumia Mall (verified sellers with quality guarantees) and ordinary third-party merchants. The distinction matters for analytics because Jumia Mall pricing is often the more stable signal while third-party pricing is more volatile. The seller card on the product page exposes a Jumia Mall badge that you can detect by looking for the mall-badge CSS class.

For brand monitoring use cases, separate the dataset into Mall and non-Mall slices. The Mall slice gives you the canonical price point that the brand wants to maintain. The non-Mall slice gives you the gray-market and parallel-import price activity that often signals supply chain shifts or unauthorized resellers.

Rate limits, retries, and session management

Jumia does not publish rate limits, but observed behavior is consistent across countries. A single IP can sustain about 1 request per 2 seconds for an hour before triggering a soft block that returns either a 429 or a Cloudflare challenge. After a 10-30 minute cooldown the IP is usable again. The cooldown extends if you continue retrying through the block.

import asyncio
async def safe_request(url: str, proxy_pool, max_retries: int = 3):
    for attempt in range(max_retries):
        proxy = proxy_pool.next()
        try:
            async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
                r = await c.get(url)
                if r.status_code == 200:
                    return r
                if r.status_code in (429, 503):
                    await asyncio.sleep(60)
                    continue
        except httpx.HTTPError:
            pass
        await asyncio.sleep(5 * (attempt + 1))
    return None

The exponential backoff matters more on Jumia than on some other African ecommerce sites because the block escalation is relatively slow but the cooldown extends quickly under retries. A patient retry pattern outperforms an aggressive one.

Working with multi-currency Jumia datasets

Each Jumia country uses its own local currency: NGN for Nigeria, KES for Kenya, EGP for Egypt, MAD for Morocco, XOF for Ivory Coast and Senegal. For pan-African analyses, normalize to USD or EUR using daily FX rates rather than scrape-time conversions. NGN in particular has had significant devaluation events in recent years, and any cross-time analysis needs to account for that.

Detecting and routing around CAPTCHA challenges on Jumia

When Jumia flags your traffic, the response is usually a Cloudflare interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched (a specific SKU your client cares about), have a fallback path that uses a headless browser with a real Nigeria residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split: 95% of requests go through the lightweight HTTP and JSON path, 5% fall through to the browser path on challenge.

Working with NGN pricing and FX normalization

Pricing in Nigeria is denominated in NGN, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. The correct pattern is to store the price in local NGN and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing Jumia to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Jumia	Nigeria	Large	High
Konga	Adjacent markets	Medium	Medium
Jiji	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Jumia but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against Jumia, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for Nigeria

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Nigeria has its own consumer protection and personal data frameworks that overlay any general analysis. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment of a scraper that targets Jumia, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="jumia-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Jumia datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. New attribute fields appear, old fields are deprecated, category trees are reorganized, and pricing display logic changes. Your scraper code has to evolve with these changes, and a versioning pattern that keeps old data interpretable is critical. Stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

ALTER TABLE snapshot ADD COLUMN scraper_version VARCHAR(16);
CREATE INDEX scraper_version_idx ON snapshot(scraper_version);

Pair this with a small registry table that documents what each scraper version did differently. When a downstream user asks why a particular metric jumped on a specific date, the version registry usually has the answer.

Common pitfalls when scraping Jumia

Three issues catch most teams off guard. The first is country-domain fragmentation. Jumia operates 11 country sites (jumia.com.ng, jumia.co.ke, jumia.ci, etc), each with its own currency, language, and seller pool. The same SKU can sit on multiple country sites with different prices and different stock. A scraper that treats jumia.com.ng prices as representative of West Africa understates Ivorian prices by 10-25% on average.

The second is JumiaPay vs cash-on-delivery price drift. Some sellers offer a JumiaPay discount that is rendered only when the user signs in. Anonymous scraping captures the list price; authenticated scraping captures the discounted price. Decide which population matters for your analysis and stay consistent.

The third is flash-sale staleness. The flash_sales API serves a curated subset of SKUs with countdown timers. The countdown is computed client-side from a server-issued epoch. If you cache the response for more than 60 seconds during an active flash sale, the timer drifts and downstream consumers see incorrect end times. Bypass cache for flash-sale endpoints and accept the higher request cost during campaign hours.

FAQ

Can I scrape all Jumia countries from a single Nigerian residential IP pool?
Technically yes, but Jumia geo-classifies the visitor IP and serves different catalogues per country. A Nigerian IP requesting jumia.co.ke works but raises the bot score because real Kenyan traffic does not originate from Nigerian residential networks. For each country you intend to scrape, source proxies from that country or from a regional adjacent market.

How does Jumia handle the difference between Mall and non-Mall sellers in the API?
There is no public API. In the HTML, Jumia Mall items are tagged with a CSS badge and a separate data-mall attribute on the seller card. Capture this signal at parse time and persist it in your snapshot table so downstream analyses can filter by Mall status.

Does Jumia’s price include shipping?
The price shown on the listing card is the product price exclusive of shipping. Shipping is calculated at checkout based on delivery zone and seller fulfillment method. For total cost analyses you would need to simulate add-to-basket flows, which is more complex and less reliable. Most price intelligence projects work with the listed price as the canonical signal.

Are Jumia reviews scrapable for sentiment analysis?
Reviews are visible on product pages and accessible through HTML scraping. They include the reviewer’s display name (usually a first name and initial) and the review text. From a privacy perspective, treat the display name as personal data and avoid storing it in long-term datasets. The review text and rating are the analytically useful fields.

Does Jumia run a sitemap I can use for SKU discovery?
Jumia exposes sitemaps at /sitemap.xml for each country. They list categories and product URLs but the per-product entries are typically only a sample of the full catalogue. For exhaustive SKU discovery, combine sitemap parsing with category and seller crawls.

Can one scraper cover all 11 Jumia country sites?
Architecturally yes, but you need per-country proxy pools. A Lagos residential IP that hits jumia.co.ke succeeds at low volume but is throttled aggressively above ~200 requests per hour.

How do I reconcile Jumia data with Konga or other competitors?
Use brand+model+capacity as the join key for electronics. EAN/UPC coverage is patchy, so fall back to fuzzy title matching with a confidence score before merging analytics.

To build a broader Africa ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape Noon UAE ecommerce in 2026

Scrape Noon UAE and you are scraping the dominant home-grown ecommerce platform in the Gulf, with separate marketplaces for the United Arab Emirates, Saudi Arabia, and Egypt. Noon was launched in 2017 as a Mohammed bin Rashid initiative to compete with Amazon’s regional expansion, and by 2026 it has captured a meaningful share of GCC ecommerce GMV through Noon Daily for grocery, Noon Food for delivery, and the core noon.com marketplace for general merchandise. The scraping landscape is shaped by three things: dual-language content (Arabic and English), country-specific catalogues with overlapping but not identical SKUs, and a Cloudflare-front-end that aggressively profiles non-GCC traffic.

This guide covers Noon UAE specifically. Most patterns transfer to noon.com.sa (Saudi) and noon.com.eg (Egypt) with minor adjustments to the country code parameter and currency parsing. The target keyword for this guide is scrape Noon UAE.

How Noon’s URL and language structure works

Noon URLs include a country segment, a language segment, and the product slug. A UAE Arabic URL looks like https://www.noon.com/uae-ar/<product-slug>/<sku>/p/, and the English equivalent is https://www.noon.com/uae-en/.... The two URLs serve different language pages but resolve to the same SKU. The trailing /p/ is what tells Noon’s routing layer that this is a product detail page.

The SKU at the end of the URL is the canonical identifier. A given product is sold in UAE, Saudi, and Egypt with the same SKU prefix but different listing IDs per country. If you are building a cross-country catalogue, capture both the SKU and the country segment as a composite key.

Noon’s product API lives at https://www.noon.com/_svc/catalog/api/v3/u/<sku>/p. The endpoint accepts an X-Locale header that controls language and country. Set it to en-ae for UAE English, ar-ae for UAE Arabic, en-sa for Saudi English, and so on. The JSON shape is consistent across locales.

import httpx
import asyncio

NOON_API = "https://www.noon.com/_svc/catalog/api/v3/u"

def headers(locale: str):
    return {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15",
        "Accept": "application/json",
        "X-Locale": locale,
        "X-Platform": "web",
        "Referer": f"https://www.noon.com/{locale.split('-')[1]}-{locale.split('-')[0]}/",
    }

async def fetch_product(sku: str, locale: str, proxy: str):
    url = f"{NOON_API}/{sku}/p"
    async with httpx.AsyncClient(proxy=proxy, headers=headers(locale), timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return None

The response includes the canonical product object, an offers array with merchant pricing, an attributes array with structured specs, and a crossSellRecommendations block that you can use to discover related SKUs without crawling categories.

Why UAE residential proxies matter for Noon

Noon’s bot detection is built on Cloudflare Bot Management plus an internal scoring service that considers the visitor IP geography, the language requested, and the device fingerprint. A request from an AWS Frankfurt IP asking for X-Locale: ar-ae is immediately suspicious because real UAE Arabic-language traffic almost never originates from European data centers.

The cleanest signal you can give Noon is a UAE residential IP requesting the locale that matches its country. UAE residential pools are smaller than Turkish or Vietnamese pools because the population is smaller, but several major proxy providers list inventory specifically in UAE through Etisalat and du. Expect to pay a premium versus general residential pools.

For workloads under 5,000 SKUs per day, you can sometimes get away with rotating GCC residential IPs (any Saudi or Kuwait IP usually works as long as the locale matches). For higher volumes, dedicated UAE inventory matters because the pool size shrinks and Noon’s behavioral scoring catches the pattern of foreign IPs rapid-fire requesting Arabic content.

Pulling category and search data

Noon exposes a search API at https://www.noon.com/_svc/search/api/v3/u/search. The endpoint takes a query string, a country segment, sort, and pagination params. For category sweeps, you pass the category slug as a filter facet rather than a path param.

async def search_noon(query: str, locale: str, page: int, proxy: str):
    url = "https://www.noon.com/_svc/search/api/v3/u/search"
    params = {
        "q": query,
        "page": page,
        "limit": 50,
        "sort": "popularity",
    }
    async with httpx.AsyncClient(proxy=proxy, headers=headers(locale), timeout=20) as c:
        r = await c.get(url, params=params)
        if r.status_code == 200:
            return r.json().get("hits", [])
        return []

The search API caps result depth at roughly 1,000 hits per query, which is the typical CDN safeguard against scraping. To go deeper into a category, decompose the query by brand, price band, or attribute facets. The search response includes the available facets and counts, which gives you a recipe for subdivision.

Handling Arabic text and RTL parsing

If your downstream pipeline is going to treat product titles as English-only, you will lose half of Noon’s catalogue. Many sellers list products in both languages, but a meaningful long tail of grocery, beauty, and fashion is Arabic-only or has more detailed Arabic descriptions than English ones. Capture both languages from day one.

The simplest pattern is to make two requests per SKU, one with ar-ae and one with en-ae, and merge the results. The title, description, attributes, and seller.name fields differ between the two responses. Everything numeric (price, stock, rating) is identical. For storage, use a JSONB or JSON column that holds both language variants and let downstream consumers pick whichever they need.

For full-text search across the dataset, Postgres with the arabic text search configuration works well, as does Elasticsearch with the Arabic analyzer. Avoid stripping diacritics during ingestion because they carry semantic meaning in Arabic product names, especially for branded items.

Comparing Noon’s three country marketplaces

Country	Currency	Catalogue size estimate	Bot scrutiny	Typical proxy cost
UAE (uae)	AED	8M+ SKUs	High	$$$
Saudi (sa)	SAR	12M+ SKUs	Highest	$$$$
Egypt (eg)	EGP	4M+ SKUs	Medium	$$

Saudi gets the most attention from Noon’s bot defenses because it generates the largest GMV. Egypt has the lightest defenses but the lowest catalogue depth. If you are building a Gulf-wide price intelligence product, plan for separate proxy budgets per country and don’t try to scrape all three from the same IP pool.

Rate limits and retry patterns

Noon’s rate limit thresholds are not published, but observed behavior is consistent: a single residential IP can sustain about 1 product detail request per second for 30-60 minutes before triggering a soft block that returns 403 with a Cloudflare challenge page. After a 5-15 minute cooldown the IP is usable again. Sticky-session residential proxies handle this gracefully with rotation on 403.

async def safe_fetch(sku: str, locale: str, proxy_pool, max_retries: int = 3):
    for attempt in range(max_retries):
        proxy = proxy_pool.next()
        try:
            data = await fetch_product(sku, locale, proxy)
            if data:
                return data
        except httpx.HTTPError:
            pass
        await asyncio.sleep(2 ** attempt + 5)
    return None

For large category sweeps, distribute work across IPs so no single IP exceeds the throughput threshold. A scheduling layer that tracks per-IP request counts and back-off windows pays for itself quickly.

Storing Noon snapshots

Schema for a per-country product snapshot:

CREATE TABLE noon_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    country VARCHAR(2) NOT NULL,
    sku VARCHAR(64) NOT NULL,
    seller_id VARCHAR(64),
    price_aed DECIMAL(12,2),
    sale_price_aed DECIMAL(12,2),
    in_stock BOOLEAN,
    rating DECIMAL(3,2),
    review_count INT,
    title_en TEXT,
    title_ar TEXT,
    PRIMARY KEY (snapshot_at, country, sku)
);

For Saudi and Egypt, swap the price column to local currency. If you are normalizing across countries, store both local currency and a derived AED-equivalent column updated daily from a central FX table. Don’t try to convert at scrape time because exchange-rate noise will pollute your trend lines.

Linking Noon scraping to broader GCC strategy

Noon is the largest single property in GCC ecommerce, but it is not the only one. Amazon UAE and amazon.sa are still major players, and category-specific sites like Sharaf DG for electronics matter for some verticals. Build your scraping stack with a multi-source mindset from the start, even if you are only launching with Noon. Our GCC ecommerce scraping overview collects related guides as we publish them.

For broader proxy strategy in MENA markets, see our residential proxy provider ranking, which now includes vendor-by-vendor UAE and Saudi inventory counts.

Detecting and routing around CAPTCHA challenges

When Noon flags your traffic, the response is usually a Cloudflare interrogation page rather than an HTTP 4xx. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched (a specific SKU your client cares about), have a fallback path that uses a headless browser with real UAE residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split: 95% of requests go through the lightweight HTTP+JSON path, 5% fall through to the browser path on challenge.

Working with AED pricing and FX normalization

Pricing in Noon is denominated in AED, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. The correct pattern is to store the price in local AED and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing Noon to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Noon	UAE	Large	High
Amazon UAE	Adjacent markets	Medium	Medium
Sharaf DG	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Noon but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against Noon, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for UAE

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but UAE has its own consumer protection and personal data frameworks that overlay any general analysis. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment of a scraper that targets Noon, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="Noon-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Noon datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Common pitfalls when scraping noon

Three failure patterns recur across noon scrapers. The first is country code confusion. noon runs separate storefronts for UAE, Saudi Arabia, and Egypt, each with its own pricing, currency, and stock. The country selector is set via cookie and URL prefix (/uae-en, /saudi-en, /egypt-en). A scraper that drops the country prefix gets redirected by IP geolocation, which corrupts the dataset when proxies move between PoPs. Always pin the country in both the URL and the x-locale header.

The second is double-currency rendering. Saudi pages show prices in SAR but include a USD conversion in the markup for some SKUs. Naive selectors pick whichever appears first in the DOM, which flips between SAR and USD across page versions. Read the price from the structured JSON-LD block, not from visible HTML.

The third is mall-vs-marketplace confusion. noon Mall (curated brand storefronts) and noon Marketplace (third-party sellers) coexist on the same product page. The seller_type and is_noon_mall flags distinguish them. Analytics that treat all listings as one population miss the fact that Mall pricing is more stable while Marketplace pricing reprices weekly. Segment your dataset by seller type before computing trend metrics.

FAQ

Is scraping Noon legal in the UAE?
Public product data is generally considered fair to scrape in most jurisdictions, but UAE law has a strong personal data and consumer protection framework. Restrict your scraping to product, price, and seller data. Avoid pulling buyer reviews with personally identifying details, avoid scraping logged-in pages, and respect any explicit terms of service. For commercial use, consult legal counsel familiar with UAE Federal Decree-Law No. 45 of 2021.

Why do prices differ between Noon UAE and Noon Saudi for the same product?
Each country marketplace is operated as a separate entity with separate seller relationships, separate logistics, and separate pricing decisions. The same SKU can be listed by completely different sellers in the two markets at different prices. Treat each country as an independent dataset for analytical purposes.

Does Noon block VPN traffic?
Noon does not specifically block VPN traffic. It blocks any IP that fingerprints as a data center or hosting provider, which is what most consumer VPN exits look like. Residential and mobile IPs from genuine consumer ISPs in the GCC are the only reliable way to scrape at volume.

How does Noon handle the Friday-Saturday weekend?
Noon catalogue updates and pricing changes continue through the GCC weekend, often with promotional bursts on Friday afternoons. If you are tracking pricing dynamics, your snapshot cadence should not skip weekends.

Can I use Noon’s affiliate API instead of scraping?
Noon runs an affiliate program through Tradedoubler and other networks. The affiliate APIs provide product data for approved affiliates but the catalogue coverage and refresh frequency are limited compared to scraping the public site. For competitive intelligence use cases, scraping remains the higher-fidelity option.

Does noon block residential proxies from non-GCC countries?
Yes, with increasing severity. Casual product lookups from EU or US residential IPs usually succeed, but sustained scraping is throttled within 24 hours. UAE, Saudi, or Egyptian residential or mobile IPs are required for production volume.

How does noon handle price changes during White Friday?
Prices update in waves every 1-3 hours during the campaign. Pre-campaign baselines should be captured at least 7 days before the event to expose true discount depth.

If you are scoping a scraping infrastructure for this market, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape Trendyol Turkey in 2026

Scrape Trendyol Turkey at scale and you immediately discover that the marketplace behaves differently from Western counterparts. Trendyol is the dominant ecommerce platform in Turkey, owned by Alibaba Group since 2018, and serves more than 30 million active buyers across categories from fashion to electronics to grocery delivery via Trendyol Go. The site enforces Turkey-specific pricing, lira denomination, KDV (VAT) inclusive display, and a recommendation engine that geo-personalizes feeds based on the visitor IP. If you fetch Trendyol from a US data center IP, you get a stripped-down catalogue with no inventory data and frequent CAPTCHA challenges. If you fetch from a Turkish residential or mobile IP, you get the same payload a real Istanbul shopper sees.

This guide walks through everything you need to scrape Trendyol Turkey product, seller, and price data reliably in 2026. The patterns apply whether you are running price intelligence for a retail brand, building a competitive monitor for a Turkish seller, or feeding a category-level dataset into a machine learning pipeline.

Why Trendyol needs Turkey-resident proxies

Trendyol uses a CDN configuration that classifies the visitor IP into one of three buckets before serving content: domestic Turkish residential, domestic mobile, or international. International visitors get a slow path with aggressive rate limiting, frequent Cloudflare interstitials, and a noticeable degradation in the JSON payloads exposed to the browser. The most obvious symptom is missing seller information and missing stock counts when you scrape from the wrong country.

Use a Turkish residential or mobile proxy and the JSON endpoints behind the product card return full payloads that include the merchant ID, fulfillment warehouse, regional inventory, and KDV-inclusive prices. The cost difference between a Turkish residential pool and a US data center pool is real, but the data quality difference is larger. For most operations the math works out in favor of paying for clean Turkish IPs.

Mapping the Trendyol URL and JSON structure

Trendyol product URLs follow a predictable pattern that includes the brand slug, product slug, and a numeric product content ID. A typical URL looks like https://www.trendyol.com/<brand>/<product-slug>-p-<contentId>. The contentId is the stable identifier you want to capture in your database because the slug portion changes when sellers rename products.

Behind the scenes, Trendyol product pages hydrate from a JSON endpoint at https://public.trendyol.com/discovery-web-productgw-service/api/productDetail/<contentId>. This endpoint returns price, seller list, variants, ratings, and stock per variant in a single response. Hitting this endpoint directly is dramatically faster than parsing the HTML, and it is much less brittle to layout changes.

import httpx
import asyncio
from typing import Optional

TRENDYOL_API = "https://public.trendyol.com/discovery-web-productgw-service/api/productDetail"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "Accept-Language": "tr-TR,tr;q=0.9,en;q=0.8",
    "Referer": "https://www.trendyol.com/",
}

async def fetch_product(content_id: int, proxy: str) -> Optional[dict]:
    url = f"{TRENDYOL_API}/{content_id}"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as client:
        r = await client.get(url)
        if r.status_code == 200:
            return r.json()
        if r.status_code == 429:
            await asyncio.sleep(30)
            return None
        return None

The JSON returns a result object with the canonical product description, a variants array (size, color, stock, sellerId), and a merchantListings array containing every seller offering that product, the price, the cargo cost, and the campaign discount. For competitive intelligence on a single SKU across multiple sellers, this single endpoint is everything you need.

Building a category crawler that respects pagination

For category sweeps, Trendyol exposes a separate search and listing endpoint at https://public.trendyol.com/discovery-web-searchgw-service/v2/api/infinite-scroll. This endpoint accepts a category code, page number, sort option, and filter facets. The infinite-scroll naming reflects the front-end pattern, but the API is plain paginated JSON.

import httpx, asyncio, json

LISTING_API = "https://public.trendyol.com/discovery-web-searchgw-service/v2/api/infinite-scroll"

async def fetch_category_page(category_id: int, page: int, proxy: str):
    params = {
        "wc": category_id,
        "pi": page,
        "culture": "tr-TR",
        "sst": "BEST_SELLER",
        "userGenderId": "",
    }
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as client:
        r = await client.get(LISTING_API, params=params)
        if r.status_code != 200:
            return []
        data = r.json()
        return data.get("result", {}).get("products", [])

async def crawl_category(category_id: int, proxies: list[str], max_pages: int = 50):
    all_rows = []
    for page in range(1, max_pages + 1):
        proxy = proxies[page % len(proxies)]
        rows = await fetch_category_page(category_id, page, proxy)
        if not rows:
            break
        all_rows.extend(rows)
        await asyncio.sleep(2)
    return all_rows

Trendyol caps a single category sweep at roughly 200 pages of 24 products each. For very broad categories you need to subdivide by facet (price band, brand, color) to recover the long tail. The aggregations field in the response tells you which facets are available and the count of products behind each facet.

Handling pricing, KDV, and campaign discounts

Trendyol pricing is messy in the way Turkish ecommerce is messy. Every price is presented as KDV-inclusive (VAT-included), but campaign discounts, basket discounts, and seller-level promo codes mean the headline price almost never matches what the buyer actually pays at checkout. If you are building a competitive intelligence dashboard, decide upfront which price you mean by price.

The product detail JSON exposes four useful fields:

Field	Meaning
originalPrice	Sticker price before any discount, KDV inclusive
sellingPrice	Current display price after seller discount
discountedPrice	Price after Trendyol campaign overlay
basketPrice	Price visible to buyer when added to basket (sometimes lower)

For most monitoring use cases, log all four every time. Models that try to compare to competitor sites need discountedPrice because that is the visible price on the listing card. Brand teams enforcing MAP (minimum advertised price) policies need originalPrice and sellingPrice because those are the prices the seller is publishing.

Proxy strategy for Trendyol at scale

Trendyol’s bot detection is layered. The first layer is Cloudflare bot management, which fingerprints TLS, HTTP/2 frames, and header order. The second layer is application-level behavioral analysis that watches for unrealistic page navigation patterns. The third is IP reputation scoring against a Turkish baseline.

For sub-10,000 product per day workloads, a small Turkish residential pool with rotating IPs per request is enough. For 100,000+ products per day, the math shifts toward sticky sessions on mobile IPs. The mobile IP costs more per port, but a single mobile IP can usually sustain a request rate of 5-10 product detail calls per second for hours without being flagged, and the per-product cost works out lower at high volume.

Reasonable starting allocation:

• 1 mobile port on Türk Telekom or Turkcell: handles 50,000 product details per day
• Backup of 50 rotating residential IPs in Turkey: handles category sweeps and seller crawls
• Single fallback datacenter pool in Frankfurt: useful only for non-personalized public endpoints like sitemap discovery

For a deeper look at how different proxy categories behave under ecommerce scraping loads, see our residential vs mobile proxy comparison for ecommerce and our proxy provider ranking for 2026.

Avoiding common Trendyol scraping mistakes

The first mistake is treating the productCode in the URL as the product identifier. Trendyol uses three different identifiers in different parts of the system: the SEO slug, the contentId in the URL, and the merchant SKU inside the JSON. Always store the contentId as your primary key. The slug changes, the merchant SKU changes per seller, only the contentId is stable across the catalogue.

The second mistake is parsing the rendered HTML rather than calling the JSON endpoints. Trendyol re-renders the product card layout regularly. The HTML breaks every few months and your scraper needs maintenance. The JSON endpoints are the contract used by Trendyol’s own front end, and they are far more stable.

The third mistake is ignoring the seller dimension. A product page can list 30+ sellers offering the same SKU at different prices and shipping conditions. If you only capture the buy-box winner, you miss the entire competitive landscape on the listing. The merchantListings array is the source of truth for seller-level price intelligence.

Storing Trendyol data for analytics

For most workloads, a wide table per product snapshot works well in DuckDB or PostgreSQL. The schema should track the four price fields above, plus seller, stock per variant, ratings, review count, and the campaign banner if any. Take snapshots at a frequency aligned to your decision cadence. For dynamic-pricing competitors, every 4-6 hours captures meaningful change. For weekly category reports, a daily snapshot is enough.

CREATE TABLE trendyol_product_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    content_id BIGINT NOT NULL,
    seller_id BIGINT NOT NULL,
    original_price DECIMAL(12,2),
    selling_price DECIMAL(12,2),
    discounted_price DECIMAL(12,2),
    basket_price DECIMAL(12,2),
    in_stock INT,
    rating DECIMAL(3,2),
    review_count INT,
    campaign_text TEXT,
    PRIMARY KEY (snapshot_at, content_id, seller_id)
);
CREATE INDEX trendyol_content_idx ON trendyol_product_snapshot(content_id);

A 100k-product daily snapshot table will grow to roughly 30M rows per year. DuckDB handles that comfortably on a laptop. PostgreSQL handles it comfortably on a single node. Either way, partition by snapshot_at weekly or monthly to keep query plans tight.

Detecting and routing around CAPTCHA challenges

When Trendyol flags your traffic, the response is usually a Cloudflare interrogation page rather than an HTTP 4xx. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched (a specific SKU your client cares about), have a fallback path that uses a headless browser with real Turkey residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split: 95% of requests go through the lightweight HTTP+JSON path, 5% fall through to the browser path on challenge.

Working with TRY pricing and FX normalization

Pricing in Trendyol is denominated in TRY, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. The correct pattern is to store the price in local TRY and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing Trendyol to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Trendyol	Turkey	Large	High
Hepsiburada	Adjacent markets	Medium	Medium
GittiGidiyor	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Trendyol but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against Trendyol, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for Turkey

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Turkey has its own consumer protection and personal data frameworks that overlay any general analysis. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment of a scraper that targets Trendyol, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="Trendyol-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Trendyol datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Common pitfalls when scraping Trendyol

Three failure modes account for most production incidents on Trendyol scrapers. The first is silent variant collapse. The product detail endpoint nests variant arrays inside allVariants and slicingAttributes. Naive flatteners pick the first variant and drop the rest, which means size and color price differences vanish from the dataset. Always iterate the full variant array and emit one row per content_id plus listing_id pair.

The second is timezone drift on price-change events. Trendyol’s backend timestamps are in Europe/Istanbul (UTC+3, no DST), but many cloud functions default to UTC. If you compare today’s price snapshot taken at 09:00 UTC against yesterday’s snapshot taken at 22:00 UTC you are comparing two windows separated by 11 hours, not 24. Pin the snapshot timestamp to the local Trendyol day and store both timestamps explicitly.

The third is campaign-price contamination. The originalPrice and sellingPrice fields carry the headline price, but campaigns like flashDiscount and crossDiscount apply at checkout and only appear inside the promotions array. A scraper that stores sellingPrice as the realized price will overstate revenue by 8-15% during major campaigns like Legendary Friday and Birthday Week. Compute the realized price by walking the promotions array and applying each rule in order.

FAQ

Do I need to log in to scrape Trendyol product data?
No. Product details, category listings, and seller data are all available without authentication. Login is only required if you want to scrape order history, wallet balance, or personalized recommendations. For 99% of competitive intelligence and price monitoring use cases, anonymous scraping is sufficient.

Will the HTML scrape work without proxies if I rate limit aggressively?
You can pull a few hundred product pages per day from a single non-Turkish IP without immediate bans. Beyond that you hit either Cloudflare interstitials or a soft block where the JSON endpoints start returning 403. For any sustained operation, Turkish residential or mobile IPs are required.

How fresh is the price data on the public JSON endpoint?
The productDetail endpoint reflects current selling state with a CDN cache lifetime of about 60-180 seconds. For most monitoring workloads that is real-time enough. If you need true real-time pricing, the legacy productgw-service endpoint occasionally bypasses cache, but it is undocumented and can change without notice.

Does Trendyol expose stock counts or just availability?
The product detail JSON includes a stock integer for each variant. For top-selling SKUs, sellers often inflate stock counts to keep the buy box. For mid-tier and long-tail SKUs, the stock value is usually accurate to within 10-20%. Use it for trend signals rather than absolute inventory truth.

Can I scrape Trendyol Go (grocery) using the same approach?
Trendyol Go uses a different subdomain and a different API surface focused on hyper-local fulfillment. The proxy and rate-limit principles transfer, but the endpoints and JSON shape are different. Plan for separate code paths if your project covers both retail and grocery.

How do I detect when Trendyol rotates its anti-bot challenge variant?
Watch for a sudden jump in the share of responses returning HTML rather than JSON for the same endpoint. A rotation typically lifts the HTML share above 5% within an hour and stabilizes after 24-48 hours as your fingerprint pool adapts.

What is the right cadence for category-level snapshots vs SKU-level snapshots?
Category listings refresh every 6-12 hours for most analytical use cases. SKU-level price and stock snapshots run hourly for top 1000 SKUs and every 4-6 hours for the long tail.

If you are scoping a scraping infrastructure for this market, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

Cookie Jar Persistence Patterns for Logged-In Scrapers (2026)

Cookie Jar Persistence Patterns for Logged-In Scrapers (2026)

Most logged-in scrapers do not fail because the parser is weak, they fail because session state gets treated like a disposable detail. cookie jar persistence is the difference between a scraper that stays warm for 21 days and one that burns a fresh login every hour, trips fraud rules, and drags your proxy bill upward. in 2026, with more sites binding sessions to device hints, IP clusters, and risk scores, the winning pattern is not just “save cookies”, it is “persist the right state with explicit invalidation rules”.

Why Persistence Matters More in 2026

Five years ago, a flat Netscape cookie file was enough for many targets. now, major retailers, SaaS dashboards, marketplaces, and B2B portals often combine cookies with local storage tokens, CSRF state, signed session metadata, and lightweight browser fingerprints. if your job restarts without restoring that bundle coherently, you get soft logged out or challenged.

The economics are blunt. a clean re-login flow through Playwright with proxy warm-up, a CAPTCHA solve, and a post-login checkpoint often costs 8 to 45 seconds. multiplied across 500 accounts, that becomes hours of dead time and a spike in anti-bot exposure.

There is also an operational angle. the more often you hit login and recovery flows, the more often you encounter MFA, device verification, or OTP detours. if you already have a stable jar strategy, you reduce how often you need the heavier recovery playbooks discussed in How to Handle 2FA / OTP Walls in Scrapers: Patterns for 2026.

What To Persist, And Where Teams Usually Get It Wrong

The common mistake is persisting only HTTP cookies from requests or httpx, while the actual logged-in state also depends on browser-side storage and request context. for browser-driven targets, persist these pieces together:

• cookies, including expiry, domain, path, secure, and httpOnly flags
• local storage keys used for access tokens, feature gates, or device IDs
• session storage, only when the target actually reads it after restore
• CSRF or anti-forgery tokens, if they are long-lived enough to reuse
• account-to-proxy affinity metadata
• user agent and key browser version details

If you ignore affinity metadata, the jar restores fine, but the next request exits from a different ASN or country and triggers a risk review. a cookie jar is not just a blob, it is a binding between identity, network posture, and client profile.

Here is the practical comparison most teams end up making:

Pattern	Good for	Breaks when	Typical cost profile
flat file cookie jar	simple requests jobs, low-value sessions	multi-worker concurrency, token drift, host crashes	cheapest, but brittle
SQLite-backed jar	single host fleets, moderate concurrency	cross-region scaling, lock contention	low cost, solid baseline
Redis session store	distributed workers, account pools, fast invalidation	poor TTL policy, missing encryption	excellent operationally, moderate complexity
browser context snapshots	Playwright-heavy targets, JS auth flows	browser version mismatch, oversized blobs	high storage, best fidelity

For Python-only HTTP clients, httpx plus a serialized Cookies object can work if the target is straightforward. for browser-led flows, Playwright storage state is usually the right primitive because it captures cookies and local storage together. requests and LWP::UserAgent can still be effective for thin authenticated endpoints, but they are weaker options once the site expects real browser continuity.

Storage Patterns That Actually Hold Up

The most reliable pattern for 2026 is tiered persistence. do not put every account session into one global store with no structure. split by account, target, and environment, then attach policy to each bucket.

1. use a stable account key such as target:account_id.
2. store the session payload plus proxy_pool_id, user_agent, created_at, last_seen_at, and risk_score.
3. track a short heartbeat on successful authenticated requests.
4. expire aggressively when the site rotates auth aggressively, otherwise keep warm and refresh opportunistically.

A realistic setup is Redis for hot sessions and SQLite for local fallback or forensic replay. Redis gives you fast invalidation and distributed access. SQLite gives you an inspectable local artifact during incident response.

Short-lived sessions should not be refreshed on every request. that is wasteful and can increase write amplification by 10x. instead, refresh on meaningful events:

• after successful login
• after token rotation detected in response headers or storage state
• after completing a high-risk checkpoint flow
• every N successful authenticated page loads, typically 10 to 25

If you are also recycling CAPTCHA solves or challenge bypass artifacts, keep those stores logically separate from the cookie jar. mixing them tends to create bad invalidation logic. the reuse economics are related, but the lifecycle is different, which is why the operational pattern in Captcha-Token Recycling: Solving Once, Reusing 50 Times (2026 Patterns) should remain its own subsystem.

Rotation, Expiry, And Invalidation Rules

A persistent jar is only useful if you are willing to kill it at the right time. too many teams let expired or poisoned sessions bounce around the queue for hours. that creates request storms and account locks.

Use three states, not two: healthy, suspect, and dead. suspect is the important one. move a jar there after one hard 401, one redirect to login, or one anti-bot interstitial that was not present on the previous request. only retry from suspect once. if it fails again, mark dead and trigger re-auth.

This is where simple metrics pay off. for one retail-monitoring fleet I have seen, adding suspect-state handling reduced wasted authenticated retries by 42 percent in the first week.

Practical invalidation rules:

• kill immediately on password reset, forced logout, or explicit session revocation
• downgrade to suspect on one anomalous geo mismatch
• cap session age even if still working, usually 7 to 30 days depending on target
• rotate browser major versions carefully, because version jumps can poison otherwise valid state

Do not ignore clock drift. signed cookies and CSRF bundles often fail when containers drift by more than a minute or two.

A Concrete Implementation Pattern

For browser-first targets, Playwright plus Redis is the current sweet spot. store Playwright storage_state, encrypt it at rest, and restore only with the same browser family and a proxy from the same pool. for simpler HTTP-only targets, httpx with a Redis-backed cookie store is lighter and cheaper.

Example, a compact Python pattern that restores a session, uses it, and writes back only after authenticated success:

import json
import redis
import httpx

r = redis.Redis(host="127.0.0.1", port=6379, decode_responses=True)
key = "session:example:acct_1837"

jar_json = r.get(key)
cookies = httpx.Cookies()
if jar_json:
    for c in json.loads(jar_json):
        cookies.set(
            c["name"],
            c["value"],
            domain=c.get("domain"),
            path=c.get("path", "/"),
        )

with httpx.Client(cookies=cookies, timeout=20.0) as client:
    resp = client.get("https://target.example.com/dashboard")
    if resp.status_code == 200 and "account overview" in resp.text.lower():
        snapshot = []
        for c in client.cookies.jar:
            snapshot.append({
                "name": c.name,
                "value": c.value,
                "domain": c.domain,
                "path": c.path,
            })
        r.setex(key, 86400, json.dumps(snapshot))

This is intentionally plain. in production, add AES-GCM encryption, version your payload schema, and attach metadata outside the raw cookie array. if you are handling 5,000 to 50,000 active sessions, add a janitor that prunes dead keys and reports anomaly rates per target.

One more blunt recommendation, do not share one persisted jar across workers without ownership rules. assign a lease for a short window, 60 to 300 seconds is typical, so two workers do not mutate the same session concurrently.

Bottom line

Treat cookie persistence as a first-class auth subsystem, not a convenience feature. for most logged-in scrapers in 2026, the best default is Playwright storage state or httpx cookies backed by Redis, with proxy affinity, explicit suspect-state handling, and hard invalidation rules. DRT-style coverage tends to focus on bypass tricks, but in practice, disciplined session storage is what keeps authenticated fleets fast, quiet, and cheap.

How to Scrape DoorDash Restaurants and Menus (2026)

DoorDash serves over 700,000 restaurants across the US, Canada, and Australia — and if you’re doing competitive pricing research, building a food delivery aggregator, or tracking menu trends across cities, scraping DoorDash restaurant menus is a very real engineering task. this article covers how DoorDash’s frontend and API actually work in 2026, what tools cut through their bot defenses, and how to extract structured menu and pricing data at scale without burning your IP pool.

how DoorDash serves its menu data

DoorDash is a Next.js app. the menu page you see in a browser is server-side rendered, but the actual item data comes from a graphQL endpoint: consumer-mobile-bff.doordash.com/consumer/graphql. same endpoint their mobile apps use.

the good news: that API returns clean JSON with item names, prices, descriptions, calories, and modifiers (size, add-ons, etc). the bad news: it requires authenticated session cookies and a valid x-channel-id header that rotates. unauthenticated requests get a 401 or a silent empty response. which is worse than a 401, honestly.

the HTML rendered pages also include a __NEXT_DATA__ JSON blob embedded in a <script> tag. for light scraping this is often easier to parse than reverse-engineering the graphQL schema, because it has most of the same data and doesn’t require auth headers.

import httpx
from bs4 import BeautifulSoup
import json

def extract_next_data(url: str, headers: dict) -> dict:
    r = httpx.get(url, headers=headers, follow_redirects=True)
    soup = BeautifulSoup(r.text, "html.parser")
    tag = soup.find("script", id="__NEXT_DATA__")
    if not tag:
        return {}
    return json.loads(tag.string)

headers = {
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
    "accept-language": "en-US,en;q=0.9",
}
data = extract_next_data("https://www.doordash.com/store/mcdonalds-new-york-12345/", headers)

the __NEXT_DATA__ path to menu items is props.pageProps.storeMenuProps.menuBook.categories[]. items sit inside each category with price (in cents), name, description, and imageUrl.

anti-bot defenses you’ll actually hit

DoorDash runs Cloudflare plus an internal bot management layer that fingerprints TLS JA3/JA4, HTTP/2 settings, and browser behavior. at moderate scale (200+ requests/hour from one IP) you’ll see:

• 403 Forbidden with a Cloudflare challenge page
• 429 Too Many Requests on the graphQL endpoint
• silent 200 responses returning empty menu data (the sneaky one)
• captcha interstitials on restaurant listing pages

rotating proxies help, but the fingerprinting layer means bare requests from httpx or requests get flagged even with valid cookies. you need TLS fingerprinting to match a real browser’s Client Hello. curl-cffi is the standard fix in 2026 — it mimics the TLS handshake of Chrome or Safari.

from curl_cffi import requests as cffi_requests

r = cffi_requests.get(
    "https://www.doordash.com/store/some-restaurant-99999/",
    impersonate="chrome120",
    headers=headers,
)

for serious scale, headless browsers (Playwright with stealth) or commercial scraping APIs handle the fingerprinting for you. similar defenses show up across the food delivery space: if you’re also pulling from Grubhub, the approach in How to Scrape Grubhub Menu Data Across Cities (2026) covers that platform’s specific quirks in more detail.

scraping at scale: proxies, rate limits, and infrastructure

a single residential IP running 50 requests/hour stays under the radar for light single-restaurant monitoring. anything broader needs IP rotation.

approach	cost	scale	detection risk
datacenter proxies	~$1-3/GB	high throughput	high (easy fingerprint)
residential proxies	~$5-15/GB	medium-high	medium
mobile proxies	~$15-40/GB	medium	low (best for auth’d flows)
scraping API (Apify, ScraperAPI, Brightdata)	~$1-5/1000 req	elastic	low (managed for you)

for menu data without login, residential proxies at 1 request per 3-5 seconds per IP are stable. if you need to scrape authenticated cart and pricing flows (DoorDash shows different prices with DashPass vs without), mobile proxies are more reliable because they share the same IP type the DoorDash mobile app actually uses.

session management matters too. DoorDash sets dd_access_token and dd_refresh_token cookies on login. if you’re scraping at scale with real accounts, rotate accounts and cookies together, not just IPs. a valid account with a stale cookie from a different IP triggers a re-auth challenge.

Uber Eats has a similar account-cookie coupling but is generally more tolerant of IP switches. there’s a full breakdown in How to Scrape Uber Eats Restaurant Listings at Scale (2026) if you’re running a multi-platform pipeline.

structuring the output: menus, modifiers, and pricing

raw DoorDash menu data is nested. a basic burger listing can have 4 levels of modifier groups (size, toppings, sauces, add-ons), and each modifier has its own price delta. flattening this into a usable schema takes some thought upfront or you’ll regret it later.

a clean schema for analysis looks like this:

• restaurant_id (DoorDash internal store ID)
• restaurant_name
• scraped_at (unix timestamp)
• item_id, item_name, category
• base_price_cents
• modifier_group_name, modifier_name, modifier_price_delta_cents
• item_calories, item_image_url

steps to build a clean pipeline:

1. fetch restaurant listing pages or search results to collect store IDs and slugs
2. for each store ID, fetch the menu page or graphQL endpoint
3. extract __NEXT_DATA__ or parse the graphQL response
4. flatten the nested modifier structure into rows (one row per item-modifier combo)
5. write to postgres or parquet with scraped_at for time-series tracking
6. schedule via Airflow or a simple cron every 6-24 hours depending on refresh needs

DoorDash does update prices dynamically — surge pricing on busy nights is documented — so if you’re doing price intelligence the timestamp matters more than most fields.

if your scope extends beyond the US market, How to Scrape Deliveroo Restaurant Menus UK + EU (2026) and How to Scrape Foodpanda Menu Data Asia + EU (2026) cover the regional equivalents with their own anti-bot quirks.

legal and rate-limit considerations

DoorDash’s terms of service prohibit scraping. enforcement has been inconsistent, but the hiQ vs LinkedIn precedent and its follow-on cases suggest that scraping publicly visiable data is defensible for research under the CFAA in the US. that said, this isn’t legal advice, and commercial use cases should run the ToS language by a lawyer first.

practically: DoorDash doesn’t gate its menus behind a login (you can see prices without an account). scraping publicly visible pages for research, price monitoring, or competitive analysis sits in a different risk bucket than scraping behind auth — similar to how How to Scrape Temu Product Data and Pricing in 2026 (Anti-Bot Guide) frames public product pricing.

and slow down. there’s no reason to hit 100 requests per second when 2-5 is fine for most use cases. fast and aggressive gets your proxy pool flagged and your costs up. slow and steady keeps you out of their radar entirely.

Bottom line

for most menu scraping needs, the __NEXT_DATA__ approach with curl-cffi and residential proxy rotation is the fastest path to production-ready data. if you need DashPass pricing or modifier-level accuracy at high volume, the graphQL endpoint with proper session management is worth the extra setup. DRT covers the full stack of food delivery scraping across platforms — check the sibling guides above if your pipeline spans more than one market.

How to Scrape FanDuel Odds and Lines in 2026

FanDuel is one of the hardest sportsbooks to scrape programmatically in 2026. Their odds pages load dynamically via a private REST API, they rotate Cloudflare JS challenges, and their IP detection is aggressive — geo-restricted by state on top of that. But the data is valuable: real-time lines on NFL, NBA, MLB, NHL, PGA, and dozens of other markets, often updating every few seconds during live games. This guide covers what actually works, what will get you blocked fast, and where the engineering tradeoffs land.

How FanDuel’s odds delivery actually works

FanDuel doesn’t serve odds as static HTML. The page shell loads, then the browser fires a series of fetch calls to https://sbapi.fanduel.com/api/.... Most of the useful data comes from two undocumented endpoints:

• GET /api/event-page?eventId=... returns full market and runner data for a single game
• GET /api/content-managed-page?page=SPORT_HOME&... returns lobby odds for a sport’s main page

Both return JSON. No API schema is published, but the response shape has been stable since late 2024.

The complication: every request requires a valid X-Auth-Token header. FanDuel generates this client-side via a fingerprinting flow that varies by state. You can’t replicate it with a static string — you either harvest it from a live browser session or reverse the generation logic, which is obfuscated and changes periodically.

Extracting the auth token and session cookies

The most reliable method is headless browser token extraction. Launch Playwright, navigate to the target state URL (e.g., fanduel.com/pennsylvania), wait for network idle, and intercept outgoing XHR calls to capture the token.

from playwright.sync_api import sync_playwright

def get_fanduel_token(state_url: str) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        ctx = browser.new_context(
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
        )
        page = ctx.new_page()
        captured = {}

        def handle_request(req):
            if "sbapi.fanduel.com" in req.url:
                token = req.headers.get("x-auth-token")
                if token:
                    captured["token"] = token
                    captured["cookies"] = ctx.cookies()

        page.on("request", handle_request)
        page.goto(state_url, wait_until="networkidle")
        browser.close()
        return captured

Token lifetime is roughly 30 to 60 minutes. Rotate fresh sessions before expiry rather than refreshing mid-scrape — the silent failure mode is a 200 response with empty markets[], not a clean 401.

One thing to watch: FanDuel geo-gates their API by actual IP location, not just the URL slug. Hitting fanduel.com/new-jersey from a California IP gets a 403 or a redirect to a blocked-state page. You need a residential or mobile proxy in the target state. This is the same geo-blocking mechanic that drives up proxy costs in scraping BetMGM lines across states — budget for it before you start.

Parsing the odds response

Once you have a valid token and matching cookies, the API calls are straightforward. The event-page endpoint returns a nested structure: markets[] containing runners[], each with a winRunnerOdds.americanDisplayOdds.americanOdds field (deeply nested, yes).

Key fields per runner:

• runnerId — stable identifier for a team or player outcome
• runnerName — human-readable label
• americanOdds — moneyline or spread price
• handicap — the spread value (null for moneylines)
• marketType — e.g., MATCH_ODDS, ASIAN_HANDICAP, TOTAL_SCORE
• inPlay — boolean, critical if you’re tracking live odds

For spread and totals markets, handicap carries the line number. Cross-check marketType before assuming structure — FanDuel sometimes nests alternate lines inside the same market object, which will corrupt your data if you parse it naively.

If you’re building a cross-book model, you’ll want to pull from sharper books too. The approach in how to scrape Pinnacle sports lines for sharp models is a solid reference: Pinnacle’s API is more permissive and serves as a useful no-vig baseline for line movement analysis.

Anti-bot detection and what actually bypasses it

FanDuel runs Cloudflare with JS challenge mode on most state subdomains. Headless Chromium without stealth patches fails the challenge in roughly 80% of attempts as of early 2026.

Minimum viable bypass stack, in order:

1. Use playwright-stealth or puppeteer-extra-plugin-stealth to patch navigator properties
2. Disable WebDriver flags (--disable-blink-features=AutomationControlled)
3. Run a real Chrome binary rather than bundled Chromium where possible
4. Route through a residential proxy with a matching state IP
5. Add realistic mouse movement and short page interaction before the token extraction

FanDuel’s detection is noticeably more aggresive than DraftKings. If you’re already scraping that book, the guide on how to scrape DraftKings odds and lines covers similar stealth patterns — but FanDuel requires stricter fingerprint consistency across the full session (same IP, same user agent, same viewport from page load through API calls).

For scale above 20 to 30 concurrent sessions, you’ll burn through residential proxies fast. Mobile proxies on a rotating pool help with detection rates but cost more. That tradeoff is real and there’s no clean answer.

Bet365’s session architecture has similar complexity. The rotation and geo-routing patterns in how to scrape Bet365 odds around the world apply here too, particularly if you’re running across multiple state jurisdictions simultaneously.

Provider and approach comparison

Method	Token required	Geo-restriction	Scale	Relative cost
Headless browser (Playwright)	Auto-captured	Yes, state IP	Low-medium	Medium
Reverse-engineered direct API	Manual extraction	Yes, state IP	High	Low (after setup)
Third-party odds API (OddsAPI, Sportradar)	No	No	High	High
Managed scraping service (Bright Data, ScrapeHero)	No	Handled	High	Very high

For one-off research or model inputs, a third-party odds API is worth it. OddsAPI covers FanDuel lines and costs under $100/month at moderate volume. It doesn’t expose market depth or alternate lines, so if you need that granularity you’re back to direct access. Sportradar has full coverage but pricing is enterprise-grade.

For teams building internal tooling that needs raw market data without an intermediary, the headless extraction approach scales reasonably to around 5 to 10 state markets before operational complexity gets painful. Beyond that, the time cost of maintaining session rotation and stealth patches usually justifies a managed service.

If your scraping pipeline spans data categories beyond sports, it’s worth studying how G2 and Capterra scraping pipelines handle rate limiting and token rotation at scale — the underlying queue and retry architecture transfers well to high-frequency odds collection.

Common error codes and what they mean

Error	Likely cause	Fix
403 Forbidden	IP geo-mismatch or Cloudflare block	Rotate proxy, re-establish session
401 Unauthorized	Token expired or missing header	Re-extract from fresh browser session
429 Too Many Requests	Rate limit hit (~20 req/min per IP)	Back off 60s, slow cadence
503 Service Unavailable	High-traffic game event surge	Retry with exponential backoff
200 with empty markets[]	Stale session (silent failure)	Log market count, alert on zero

The silent 200 with empty markets is the one that burns you in production. Add a check on len(response["markets"]) per response and treat zero as a session-expired signal.

Bottom line

FanDuel is scrappable in 2026 but it takes real infrastructure: headless stealth browsers, state-matched residential proxies, and solid session rotation. If you just need odds for model inputs or comparison tools, OddsAPI covers FanDuel and saves weeks of engineering overhead. If you need raw market depth and alternate lines, the direct approach outlined here is the path. DRT covers the full sportsbook scraping stack — from FanDuel and DraftKings through international books like Bet365 and Pinnacle — so check back as these APIs and detection layers evolve.

How to Scrape Wix and Squarespace Stores in 2026

Scraping Wix and Squarespace stores is harder than most ecommerce targets, not because they have better bot protection, but because neither platform exposes a clean public API for product data. what you get instead is heavily client-rendered HTML, proprietary JSON blobs buried inside <script> tags, and JavaScript-dependent pagination that breaks naive scrapers on the first request. this guide covers what actually works in 2026, with platform-specific patterns, tool picks, and honest tradeoffs.

how Wix and Squarespace serve product data

both platforms render product catalogs via JavaScript frameworks, not server-side HTML. that single fact drives every decision you’ll make downstream.

Wix (now Wix Studio for most new stores) loads product data through its internal _api/wix-ecommerce-storefront-web/api endpoint. the JSON response is structured and predictable once you find it. Wix also injects a window.__VIEWER_MODEL__ object into the page source on many storefronts — catalog state, no browser render needed.

Squarespace uses its own Commerce API (/api/2/commerce/) internally. product collections are served as JSON at /api/2/commerce/products with pagination via cursor. the HTML source usually includes a Static.SQUARESPACE_CONTEXT JSON block with store metadata, active collection IDs, and sometimes a partial product list.

this is a different situation from How to Scrape Shopify Stores at Scale 2026 (Without Getting Blocked), where /products.json is a public, documented endpoint anyone can hit. Wix and Squarespace require reverse-engineering internal APIs. there are no docs.

fingerprinting the platform before you write a single line

check which platform you’re dealing with first. they look similar from the outside and the scraper for one won’t work on the other.

signal	Wix	Squarespace
HTTP response header	X-Wix-Request-Id present	X-ServedBy: squarespace
HTML source	window.rendererModel or __VIEWER_MODEL__	Static.SQUARESPACE_CONTEXT block
asset CDN	static.parastorage.com	static1.squarespace.com
robots.txt	disallows on /_api/	usually /api/ blocked

a curl request to the root URL and a quick grep takes under two seconds. more reliable than URL guessing, more reliable than favicon matching. if you’re running a multi-platform pipeline that also handles Magento or BigCommerce, the detection approaches in How to Scrape Magento Stores in 2026: API and HTML Patterns and How to Scrape BigCommerce Stores Programmatically (2026) follow the same logic.

scraping Wix stores

the viewer model path

fastest option: parse window.__VIEWER_MODEL__ directly from the HTML. no browser, no JS execution.

import httpx, re, json

def fetch_wix_viewer_model(url: str) -> dict:
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    }
    r = httpx.get(url, headers=headers, follow_redirects=True, timeout=15)
    match = re.search(
        r'window\.__VIEWER_MODEL__\s*=\s*(\{.+?\})(?=;\s*</script>)',
        r.text, re.DOTALL
    )
    if match:
        return json.loads(match.group(1))
    return {}

this works on roughly 60-70% of Wix storefronts. for the rest — newer Wix Studio builds mostly — you POST to _api/wix-ecommerce-storefront-web/api directly using the store’s metaSiteId. find it in the viewer model or the page source. responses are paginated JSON with products under a predictable key path.

things that will trip you up:

• Wix rate-limits by IP around 1 req/2s. rotate residential proxies after every 50-80 requests
• _api/ endpoints return 403 without X-Wix-Brand or with an unexpected Origin header. mirror these from a real browser session on the target store
• product variants are nested under productItems inside each product object, not at the top level. easy to miss

Wix anti-bot

higher-traffic stores run Imperva (Incapsula). you’ll hit a JS challenge page. Playwright with rebrowser-patches plus rotating residential proxies clears it reliably. pure httpx won’t — don’t try to solve Imperva challenges in a pure HTTP client, it’s a time sink.

scraping Squarespace stores

Squarespace’s internal Commerce API is the easier of the two. products paginate at /api/2/commerce/products with a cursor param. the sequence:

1. GET /api/2/commerce/products?per_page=200 — first page plus pagination.nextPageCursor
2. GET /api/2/commerce/products?per_page=200&cursor=<nextPageCursor> — repeat until pagination.hasNextPage is false
3. parse items[] from each response — each has variants, pricing, images, categories

the per_page cap is 200. most stores have under 2,000 products, so you’re looking at maybe 1-10 requests per store. no auth required on public storefronts. it’s almost too easy.

password-protected stores redirect /api/2/ calls to the password page. check Content-Type on the response. JSON means open, HTML means gated. skip and move on.

but unlike the pattern-matching grunt work needed for How to Scrape WooCommerce Stores 2026: Pattern Recognition Approach, Squarespace gives you a consistent API surface regardless of theme. you write the scraper once and it works on every store.

proxy and rate-limit strategy

both platforms use Cloudflare CDN for static assets, but API traffic runs through different stacks. from testing in 2026:

• Wix: soft rate-limit around 1 req/2s per IP, 429 with retry-after
• Squarespace: more lenient, roughly 5-10 req/s per IP before a temporary block

for Wix, residential proxies are not optional for any serious run. datacenter IPs get blocked at the Imperva layer before they ever reach product data. for Squarespace, datacenter proxies from Bright Data or Oxylabs work fine on most stores — which cuts cost a lot if you’re doing bulk collection.

rules that hold up in practice:

• rotate IP every 50 Wix _api/ requests, or immediately on any 403
• rotate every 200 Squarespace /api/2/commerce/ requests, or on any 429
• keep User-Agent and Accept-Language consistent within a session. rotating headers independently from IPs creates a fingerprint mismatch and triggers blocks faster than the rate limit would

common failure modes

• empty items[] on Squarespace: store uses a non-default catalog structure. check Static.SQUARESPACE_CONTEXT for activePageCollections, swap in the right collection ID
• Wix viewer model missing products: the store runs on Wix Blocks. fall back to Playwright and intercept XHR to _api/wix-ecommerce-storefront-web/
• 403 on Wix _api/: missing X-Wix-Brand or wrong Referer. copy headers from a live browser session on that exact store, not from a different Wix site
• Squarespace returns HTML not JSON: password-protected or in maintenence mode. skip it
• Wix pagination stops early: metaSiteId mismatch. extract the ID from each target URL independently, never reuse across stores

bottom line

Squarespace is the easier target — consistent API, no browser required, minimal anti-bot. Wix takes more setup: viewer model extraction, Playwright fallback for JS-heavy stores, residential proxies if Imperva shows up. if you’re building a multi-platform ecommerce scraper, validate on Squarespace first, then add Wix. dataresearchtools.com covers the full ecommerce scraping stack — the same reverse-engineering approach here applies to any headless-first storefront you’ll run into in 2026.

Privacy-preserving scraping: differential privacy and federated learning

Privacy preserving scraping is no longer a research-only topic. The combination of regulatory pressure (GDPR, CCPA, PDPA, DPDP), customer expectations, and the maturation of privacy-engineering tooling made privacy-preserving techniques into production patterns through 2024-2026. For scraping operators, three techniques matter: differential privacy (release aggregate statistics with mathematical privacy guarantees), federated learning (train models without centralising raw data), and secure aggregation (combine inputs from multiple parties without any single party seeing raw individual data). Each has matured into accessible tooling, each has real production use cases for scraped data, and each shifts the compliance and competitive picture in important ways. This guide walks through what each technique actually does, where they fit in scraping pipelines, the production tooling in 2026, and a practical adoption roadmap.

The audience is the data engineer, ML lead, or compliance partner whose scraping pipeline produces sensitive aggregates and who wants to know which privacy-engineering techniques actually fit the use case.

Why privacy-preserving techniques matter for scraping

Three reasons.

First, regulatory pressure. GDPR Article 25 (privacy by design), CCPA’s data minimisation principle, PDPA’s protection obligation, and DPDP’s purpose limitation all reward techniques that reduce the personal-data exposure surface. A scraper using differential privacy to publish aggregates faces materially less regulatory risk than one publishing raw records.

Second, customer trust. Enterprise customers in regulated industries (finance, healthcare, government) increasingly require privacy-preserving outputs. A vendor that can prove differential privacy, federated learning, or secure aggregation gets the deal a vendor that cannot does not.

Third, competitive differentiation. The 2025-2026 wave of AI-training fines (covered in fair use and copyright for AI training data) put pressure on raw-data resellers. Vendors who pivot to privacy-preserving outputs are better positioned.

For the broader compliance picture, see the GDPR scraping compliance guide and the personal vs public data scraping framework.

Differential privacy explained

Differential privacy (DP) is a mathematical guarantee about the privacy of an individual within an aggregate result. Formally: a query mechanism is epsilon-DP if changing or removing one individual’s data from the input changes the output’s probability distribution by at most a factor of e^epsilon. Smaller epsilon means stronger privacy; epsilon=0 would be perfect privacy but useless results; typical production values are 0.1 to 5.

In practice, DP is implemented by adding calibrated noise to query results. The noise is calibrated to the query’s sensitivity (how much one individual can change the result) and the privacy budget epsilon.

For scraping pipelines, DP is most useful when releasing aggregate statistics: counts, averages, distributions over a scraped corpus that contains personal data. A scraper that publishes “number of products in category X by region” can use DP to release the number with provable per-individual privacy.

A minimal DP implementation in Python using OpenDP:

from opendp.measurements import make_laplace
from opendp.transformations import make_count_distinct
from opendp.combinators import make_chain

def dp_count(values, epsilon: float = 1.0):
    transform = make_count_distinct(input_domain=...)
    measure = make_laplace(scale=1.0 / epsilon)
    mechanism = make_chain(transform, measure)
    return mechanism(values)

The key engineering work is sensitivity analysis (how much one record can change the output) and budget management (how to spend epsilon across multiple queries on the same data).

Federated learning explained

Federated learning (FL) trains a model across multiple data holders without centralising raw data. Each holder trains a local model on their data, sends model updates (gradients or weights) to a central aggregator, and the aggregator combines updates into a global model.

For scraping, FL becomes interesting in two scenarios:

1. Multiple scraping operations cooperate to train a shared model without sharing raw scraped data.
2. A scraping operator trains a model with edge clients (browsers, devices) without uploading raw data to central servers.

The 2026 production FL frameworks: Flower (open source, language-agnostic), TensorFlow Federated, PySyft (PyTorch-aligned). Production deployments still skew small (under 100 participants typically) but the tooling is mature.

A minimal Flower-based federated training loop:

import flwr as fl
from typing import List, Tuple

def evaluate(parameters):
    # Run global eval on held-out test set
    return loss, num_examples, {}

class FedAvgWithEval(fl.server.strategy.FedAvg):
    def evaluate(self, server_round: int, parameters):
        return evaluate(parameters)

fl.server.start_server(
    server_address="0.0.0.0:8080",
    config=fl.server.ServerConfig(num_rounds=10),
    strategy=FedAvgWithEval(min_fit_clients=3, min_evaluate_clients=3),
)

Each scraping client runs a parallel client process that fits the model on local data and reports updates.

Secure aggregation explained

Secure aggregation lets multiple parties combine numerical inputs (e.g., model gradients, statistics) such that the aggregator only sees the sum, not the individual inputs. The cryptographic technique is multi-party computation (MPC) or homomorphic encryption (HE).

For scraping, secure aggregation matters when multiple operators want to compute joint statistics (industry-wide aggregates, joint AI training) without revealing raw data to each other.

The 2026 production tooling: Google’s TFF Secure Aggregation, Meta’s CrypTen, Microsoft’s SEAL, OpenMined’s TenSEAL.

In practice, secure aggregation is operationally heavy and is usually paired with FL rather than used standalone.

Where each technique fits in scraping pipelines

Technique	Best fit	Production maturity	Compliance benefit
Differential privacy	Releasing aggregate stats from scraped personal data	High	GDPR Art 25, “publicly available” reframing
Federated learning	Multi-operator model training	Medium	Reduces raw data centralisation
Secure aggregation	Multi-operator statistics	Medium	Hides individual operator inputs
Zero-knowledge proofs	Proving properties of data without revealing data	Medium-low	Strong privacy claims
Synthetic data	Releasing useful approximations	High	Often paired with DP
K-anonymity / l-diversity	Anonymising individual records	Mature but limited	Older approach; weaker than DP

For most scraping operators, differential privacy is the highest-leverage starting point. It is mature, well-tooled, and fits the most common use case (publishing aggregates).

For the parallel discussion of how this overlays with verifiable credentials, see verifiable credentials and scraping.

Decision tree: which technique fits this scraping use case?

Q1: Are you publishing aggregate statistics from a personal-data corpus?
    ├── Yes -> Differential privacy.
    └── No  -> Q2
Q2: Are multiple parties contributing data to a shared model?
    ├── Yes -> Federated learning, optionally with secure aggregation.
    └── No  -> Q3
Q3: Do you need to prove a property of data without revealing the data?
    ├── Yes -> Zero-knowledge proofs.
    └── No  -> Q4
Q4: Do you need to release a useful dataset that approximates real data?
    ├── Yes -> Synthetic data generation, ideally with DP guarantees.
    └── No  -> Standard pipeline.

Comparison: DP vs FL vs SA

Dimension	Differential Privacy	Federated Learning	Secure Aggregation
Privacy unit	Individual record	Individual data holder	Individual contribution
Centralisation needed	Aggregator sees noisy result	Aggregator sees model updates	Aggregator sees only sum
Compute overhead	Low	High (multiple training rounds)	Very high (MPC)
Network overhead	Low	Medium	High
Production maturity	High	Medium	Medium-low
Tooling	OpenDP, Tumult Labs, Diffprivlib	Flower, TFF, PySyft	TFF SecAgg, CrypTen, SEAL
Best paired with	Aggregate publishing	Multi-party training	Federated learning

Differential privacy at scale: practical guidance

Production DP requires four practical disciplines.

First, sensitivity analysis. Calculate the maximum change one record can cause in your output. Bounded sums need clipping; unbounded queries (top-k, percentiles) need careful handling.

Second, privacy budget management. Each query consumes part of the budget. Track total epsilon spent per dataset; do not exceed the budget the privacy posture commits to.

Third, query composition. Multiple queries compose: epsilon_total ≤ sum of epsilon_per_query (basic composition) or with tighter bounds via advanced composition (Renyi DP, zCDP). Use libraries that handle composition automatically.

Fourth, public release versus internal use. Public DP releases need stronger budgets (small epsilon, generous noise). Internal DP releases (analyst-facing dashboards) can run higher epsilon.

The 2026 mature DP libraries:

Library	Maintainer	Strength
OpenDP	OpenDP project	Most rigorous; verified building blocks
Tumult Labs Analytics	Tumult Labs	Production-grade SQL-style interface
Google Differential Privacy	Google	Wide language support
Diffprivlib	IBM	scikit-learn-style API
OpenMined PyDP	OpenMined	Friendly Python API

For most scraping operators, OpenDP or Tumult Analytics are the production picks.

Federated learning at scale: practical guidance

Production FL works best when:

1. The data lives where it should not be centralised (edge devices, partner organisations, sovereign data).
2. The computation pattern is compatible with model-update aggregation (most modern ML training is).
3. The participants are stable enough to complete multi-round training.

The 2026 mature FL frameworks:

Framework	Maintainer	Strength
Flower	Adap	Language-agnostic, production deployments
TensorFlow Federated	Google	TF-aligned, strong simulation
PySyft	OpenMined	PyTorch-aligned, research-friendly
FedML	FedML, Inc.	Strong cross-platform
OpenFL	Intel	Cross-vendor, strong governance

For scraping use cases involving multiple operators, Flower has the most production-grade reference deployments.

Worked example: DP aggregate release of scraped product data

A scraping operator publishes weekly statistics about product availability across major retailers. The dataset contains seller identities, product details, and per-seller stockout events. Sellers are personal data when they are individuals (sole proprietors).

Without DP: the publication is a flat table of seller-level aggregates. Each row reveals one seller’s stockout rate, exposing potentially sensitive operational information.

With DP: the operator releases category-level aggregates with calibrated Laplace noise. The aggregate “Category X had Y stockouts last week” is published with epsilon = 1, providing a meaningful privacy guarantee while preserving the headline value of the data product.

from opendp.measurements import make_laplace
from opendp.transformations import make_count_by_categories

def category_stockouts(events, categories, epsilon: float = 1.0):
    counts = {c: 0 for c in categories}
    for e in events:
        if e["stockout"] and e["category"] in counts:
            counts[e["category"]] += 1
    sensitivity = 1
    measure = make_laplace(scale=sensitivity / epsilon)
    return {c: max(0, n + measure()) for c, n in counts.items()}

The result: a category-level publication with provable privacy at epsilon=1, defensible against regulator inquiry.

External references

The OpenDP framework is at opendp.org. The Flower federated learning framework is at flower.ai. The Google Differential Privacy library is at github.com/google/differential-privacy. Tumult Labs Analytics documentation is at docs.tmlt.dev. The IETF privacy preservation working group drafts are at datatracker.ietf.org.

Synthetic data: the adjacent technique

Synthetic data generation is sometimes paired with DP to produce releasable datasets. The pattern: train a generative model on real data with DP guarantees, then release samples from the generative model.

Production synthetic data tools in 2026: SDV (Synthetic Data Vault), Mostly AI, Gretel.ai, Tonic.ai. The maturity is high for tabular data, lower for unstructured (text, image) data.

For scraping operators, synthetic data is most useful when sharing scraped datasets with downstream customers who cannot directly handle raw personal data. The synthetic version preserves statistical properties while removing individual records.

Adoption roadmap

A 12-month roadmap for a scraping operator adopting privacy-preserving techniques:

Quarter	Deliverable
Q1	Sensitivity analysis on existing aggregate releases; budget framework
Q2	First DP release on internal dashboards; epsilon budget assigned
Q3	Public DP release for one product category; customer feedback
Q4	Evaluate FL or secure aggregation for multi-operator pilots

A team that completes Q1-Q3 has a defensible DP-based product. Q4 is the optionality for going further.

For the broader policy build, see building an ethics-first scraping policy.

Comparison: privacy-preserving outputs vs raw-data outputs

Output type	Compliance risk	Customer value	Engineering cost
Raw scraped data	High	Highest	Lowest
Pseudonymised data	High (still personal)	High	Low
K-anonymised data	Medium	Medium	Low (but limited)
DP aggregates	Low	Medium-high	Medium
FL-trained model	Low	Medium	High
Synthetic data	Low (with DP)	Medium	Medium-high

The risk-value tradeoff favours privacy-preserving outputs as compliance pressure rises. The 2026 trend is unmistakable: vendors moving up this table win deals.

FAQ

Is differential privacy production-ready in 2026?
Yes. Multiple mature libraries (OpenDP, Tumult, Google DP) have substantial production deployments.

What epsilon should I use?
Common values: 0.1 (very strong) for high-risk releases, 1-5 for typical releases, higher for internal-only. The right value depends on the threat model and sensitivity.

Can I use federated learning instead of centralising scraped data?
Sometimes, when the data sources are organisations willing to participate. For unilateral scraping, FL does not apply.

Does DP help with GDPR compliance?
Yes. DP-released aggregates that satisfy the EDPB’s anonymisation tests can move outside GDPR scope. Verify per release with counsel.

What about zero-knowledge proofs for scraping?
Niche but growing. Use cases include proving compliance properties to auditors without revealing the underlying data.

Extended privacy-preserving scraping analysis

Privacy-preserving scraping is the discipline of collecting only what is needed, in a form that minimises personal data exposure, with measurable safeguards. The 2026 toolkit consists of six techniques.

1. Differential privacy at aggregation. Add calibrated noise so individual records cannot be reconstructed.
2. Pseudonymisation at ingest. Replace direct identifiers with stable tokens.
3. K-anonymity at publication. Suppress or generalise records below the k threshold.
4. Federated processing. Compute on the source rather than centralising raw data.
5. Secure multi-party computation. Combine inputs from multiple parties without exposing them to each other.
6. Homomorphic encryption. Compute on encrypted data with the result decrypted at the end.

Each has costs (latency, accuracy, complexity) and benefits (compliance posture, breach minimisation).

Implementation pattern: differential privacy aggregation

import numpy as np

def laplace_mechanism(true_value, sensitivity, epsilon):
    scale = sensitivity / epsilon
    noise = np.random.laplace(0, scale)
    return true_value + noise

def dp_count(records, epsilon=1.0):
    sensitivity = 1.0
    true_count = len(records)
    return laplace_mechanism(true_count, sensitivity, epsilon)

def dp_mean(values, lower, upper, epsilon=1.0):
    clipped = np.clip(values, lower, upper)
    sensitivity = (upper - lower) / len(clipped)
    true_mean = np.mean(clipped)
    return laplace_mechanism(true_mean, sensitivity, epsilon)

Implementation pattern: pseudonymisation with key separation

import hmac
import hashlib

class Pseudonymiser:
    def __init__(self, key):
        self.key = key

    def tokenise(self, identifier):
        return hmac.new(self.key, identifier.encode(), hashlib.sha256).hexdigest()

    def rotate_key(self, new_key):
        self.key = new_key

Implementation pattern: k-anonymity check

from collections import Counter

def k_anonymous(records, quasi_identifiers, k=5):
    keys = [tuple(r.get(q) for q in quasi_identifiers) for r in records]
    counts = Counter(keys)
    return all(c >= k for c in counts.values())

def suppress_below_k(records, quasi_identifiers, k=5):
    keys = [tuple(r.get(q) for q in quasi_identifiers) for r in records]
    counts = Counter(keys)
    return [r for r, key in zip(records, keys) if counts[key] >= k]

Federated processing pattern

def federated_aggregate(participants, query):
    partials = []
    for p in participants:
        partial = p.compute_local(query)
        partials.append(partial)
    return aggregate(partials)

def compute_local(query):
    result = run_query(query, local_data)
    return laplace_mechanism(result, sensitivity=1.0, epsilon=1.0)

Comparison: privacy techniques tradeoffs

Technique	Privacy strength	Accuracy cost	Compute cost	Best for
Pseudonymisation	Moderate	None	Low	All ingest
K-anonymity	Moderate	Suppression	Low	Publication
Differential privacy	Strong	Noise	Low to moderate	Aggregates
Federated	Strong	None	High coordination	Multi-party
Secure MPC	Strongest	None	Very high	Sensitive joins
Homomorphic encryption	Strongest	None	Highest	Computed-on-encrypted

Operational pattern: privacy budget tracking

For DP systems, track the cumulative epsilon spent per data subject across queries. When the budget is exhausted, stop answering queries about that subject.

class PrivacyBudget:
    def __init__(self, total_epsilon):
        self.total = total_epsilon
        self.spent = {}

    def spend(self, subject_id, epsilon):
        current = self.spent.get(subject_id, 0)
        if current + epsilon > self.total:
            return False
        self.spent[subject_id] = current + epsilon
        return True

Additional FAQ

Does pseudonymisation remove GDPR scope?
No. Pseudonymous data remains personal data. Only true anonymisation removes scope.

What epsilon is acceptable for DP?
Common guidance is 0.1-1.0 for strong privacy, 1-10 for moderate. The choice is workload-specific.

Is federated learning the same as federated processing?
Federated learning is a special case for ML training. Federated processing is the broader umbrella for any compute-on-source workflow.

How does this interact with AI training?
DP-SGD and PATE are training-time techniques that bound the privacy leakage of the trained model. They complement but do not replace data-collection privacy controls.

The data minimisation principle in practice

Data minimisation is a foundational privacy principle that appears in GDPR Article 5(1)(c), CCPA’s purpose limitation provisions, PDPA’s necessity test, and DPDP’s purpose specification requirements. The principle is straightforward in theory and demanding in practice.

A scraper applying data minimisation collects only the fields necessary for the stated purpose. If the purpose is competitor pricing analysis, the scraper collects product names and prices, not customer reviews. If the purpose is sentiment analysis, the scraper collects review text but not reviewer identifiers. The minimisation is per-field, per-record, and per-purpose.

The 2026 implementation pattern starts at the schema level. The scraper defines a target schema containing only the necessary fields. The fetch and parse logic populates only those fields. Additional content available on the page is discarded.

A common failure mode is over-collection at the fetch step followed by post-fetch filtering. The over-collected data may be retained in logs, caches, or backups even if it is not loaded into the primary store. The 2026 best practice is to filter as early in the pipeline as possible, ideally at the fetcher.

The de-identification spectrum

De-identification is not a binary state. The spectrum runs from raw identified data through pseudonymisation, masking, generalisation, suppression, and finally to true anonymisation. Each step strengthens privacy and weakens utility.

Pseudonymisation replaces direct identifiers with stable tokens. The tokens enable record linkage without revealing the original identifiers. GDPR Recital 26 explicitly notes that pseudonymous data remains personal data because re-identification is possible.

Masking replaces parts of identifiers with placeholders. An email address might be masked to j***@example.com. Masking reduces direct identification while preserving some utility for analysis.

Generalisation replaces specific values with broader categories. A specific age (34) becomes an age range (30-39). A specific city becomes a region. Generalisation is a primary tool in k-anonymity.

Suppression removes records or fields entirely. Records that cannot be sufficiently anonymised are dropped. Suppression is the most conservative option and often the most defensible.

True anonymisation removes the personal data classification. Under GDPR’s strict reading, true anonymisation requires that re-identification be impossible by any means reasonably likely. Most scraping pipelines do not achieve true anonymisation. The pragmatic operational target is strong pseudonymisation plus minimisation plus aggregation.

Differential privacy in practice

Differential privacy is the mathematically rigorous framework for releasing statistics about a population without revealing individuals. The framework introduces calibrated noise to query results, with the noise calibrated by an epsilon parameter that bounds the privacy loss.

For scrapers DP applies most naturally to aggregate releases. A scraper that publishes a count of records, an average, or a distribution can apply DP noise to the published statistic. Individual records remain protected.

The epsilon parameter is the central tuning knob. Lower epsilon means stronger privacy and noisier results. Higher epsilon means weaker privacy and more accurate results. The 2026 best practice is to choose epsilon per use case, typically in the 0.1-1.0 range for strong privacy and 1-10 range for moderate privacy.

A practical complication is the privacy budget. Each query consumes some epsilon. Repeated queries on the same dataset accumulate epsilon, and the cumulative epsilon bounds the total privacy loss. A privacy-aware system tracks the cumulative epsilon and stops responding when the budget is exhausted.

Federated learning and federated processing

Federated approaches keep raw data at the source and centralise only derived signals. The patterns differ in what is centralised.

Federated learning trains a model across many participants, with each participant computing gradient updates locally and the central coordinator aggregating the updates. The raw data never leaves the participant. The model captures the aggregate signal.

Federated processing is broader. It covers any computation where the input is at the source and only the result moves to the coordinator. Federated SQL queries, federated analytics, and federated search all qualify.

For scrapers federated approaches are useful when the data sources are willing to compute locally but unwilling to share raw data. A consortium of publishers might agree to a federated analytics arrangement that yields industry statistics without exposing individual subscriber data.

The 2026 federated toolkit includes Google’s Federated Learning of Cohorts (FLoC, deprecated and replaced by Topics), Mozilla’s Distributed Aggregation Protocol, the OpenMined frameworks, and several research-grade systems. Production deployments are growing in healthcare and finance, where the privacy stakes are highest.

Next steps

The fastest first step is to identify one aggregate release in your current pipeline and pilot a DP version using OpenDP or Tumult. The engineering effort is bounded; the compliance and customer-trust upside is real. For broader emerging-tech context, head to the DRT emerging-tech hub and pair this with the personal vs public data framework.

This guide is informational, not engineering or legal advice.

How to scrape Flipkart India product data

Scrape Flipkart India effectively in 2026 and you have access to one of the two dominant ecommerce platforms in the largest English-speaking online market in the world. Flipkart serves over 200 million registered users in India, indexes hundreds of millions of SKUs across electronics, fashion, grocery, and home goods, and runs the Big Billion Days sales that move tens of millions of units in a single week. For brand managers tracking pricing, agencies running competitive intelligence, or product teams sizing demand in India, Flipkart and Amazon India together cover the market.

This guide walks the full Flipkart India scraping stack: which endpoints to hit, how to handle Walmart-owned Flipkart’s bot defenses, how to manage Indian mobile carrier proxies and INR pricing, and how to keep extraction quality high across the site’s wide category structure. Working Python and Playwright code throughout.

What Flipkart India exposes

Surface	URL pattern	Best for
Product detail page	flipkart.com/{slug}/p/{pid}	Full extraction with reviews and Q&A
Search results	flipkart.com/search?q={query}	Discovery
Internal API	flipkart.com/api/3/page/fetch	High-throughput product extraction
Category landing	flipkart.com/{category}	Category sweeps

The internal /api/3/page/fetch endpoint returns clean JSON used by the React frontend. It requires a CSRF token and a session cookie, which a browser session provides for free.

Anti-bot defenses

Flipkart uses a custom bot defense stack:

1. PerimeterX (now HUMAN) on the public web pages
2. Aggressive IP reputation; data center IPs blocked or heavily challenged
3. Custom request signing on internal API endpoints
4. Header-based fingerprinting (specific Accept-Language and User-Agent expected)

The recommended path: Indian mobile carrier IPs (Jio, Airtel, Vi), real Chromium with a mobile user agent and Indian locale, and patient throttling.

Working browser-based scraper

import asyncio
import json
import re
from playwright.async_api import async_playwright

async def scrape_flipkart_in(product_url: str, proxy: dict | None = None) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy=proxy,
            args=["--disable-blink-features=AutomationControlled"],
        )
        ctx = await browser.new_context(
            user_agent="Mozilla/5.0 (Linux; Android 13; CPH2483) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Mobile Safari/537.36",
            locale="en-IN",
            timezone_id="Asia/Kolkata",
            extra_http_headers={"Accept-Language": "en-IN,en;q=0.9,hi;q=0.8"},
            viewport={"width": 412, "height": 915},
        )
        page = await ctx.new_page()

        api_payload = {}
        async def on_response(resp):
            if "/api/3/page/fetch" in resp.url:
                try:
                    api_payload["data"] = await resp.json()
                except Exception:
                    pass
        page.on("response", on_response)

        await page.goto(product_url, wait_until="networkidle", timeout=45000)
        html = await page.content()
        await browser.close()

    if api_payload.get("data"):
        return _normalize_flipkart_api(api_payload["data"])
    return _parse_flipkart_html(html, product_url)

def _normalize_flipkart_api(data: dict) -> dict:
    slots = data.get("RESPONSE", {}).get("slots", [])
    product = next((s for s in slots if s.get("widget", {}).get("type") == "PRODUCT_SUMMARY"), {})
    if not product:
        return {"error": "no_product_summary"}
    info = product.get("widget", {}).get("data", {}).get("productSummary", {})
    return {
        "title": info.get("title"),
        "brand": info.get("brand"),
        "price_inr": info.get("pricing", {}).get("finalPrice", {}).get("value"),
        "original_price_inr": info.get("pricing", {}).get("mrp", {}).get("value"),
        "discount_percent": info.get("pricing", {}).get("totalDiscount"),
        "rating": info.get("ratingsAndReviews", {}).get("rating", {}).get("average"),
        "review_count": info.get("ratingsAndReviews", {}).get("rating", {}).get("count"),
        "in_stock": info.get("availability", {}).get("status") == "IN_STOCK",
    }

def _parse_flipkart_html(html: str, url: str) -> dict:
    # fallback to BeautifulSoup-based parsing if API intercept fails
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(html, "html.parser")
    title = soup.select_one("span.B_NuCI") or soup.select_one("h1 span")
    price = soup.select_one("div._30jeq3._16Jk6d") or soup.select_one("div._30jeq3")
    return {
        "title": title.text.strip() if title else None,
        "price_inr": _parse_inr(price.text) if price else None,
        "url": url,
    }

def _parse_inr(s: str) -> float:
    return float(re.sub(r"[^\d]", "", s) or 0)

asyncio.run(scrape_flipkart_in("https://www.flipkart.com/example-product/p/itm123456"))

Mobile user agent and the Indian locale matter. Flipkart serves a mobile-optimized API path with cleaner JSON to mobile clients.

Indian Rupee price handling

INR uses the symbol ₹ and uses comma in the Indian numbering system: ₹1,23,456.78 (lakh-crore grouping, not Western thousands). The API returns plain numbers, so this is only a display concern. For parsing scraped UI text:

import re

def parse_inr(s: str) -> float:
    # handle both Western (1,234,567) and Indian (12,34,567) grouping
    return float(re.sub(r"[^\d.]", "", s) or 0)

USD conversion in 2026 hovers around 84-87 INR per USD. Always store the raw INR; convert only for display.

Multi-language considerations

Flipkart serves predominantly English content nationwide. Some product titles include Hindi or regional script (Tamil, Telugu, Bengali, Malayalam) for grocery and traditional goods. UTF-8 storage handles all of them.

Search queries respect English. Hindi search queries work in Devanagari script but coverage is sparse outside the largest categories.

Mobile proxy rotation

Indian mobile carrier IPs (Jio, Airtel, Vi/Vodafone Idea) are the cleanest source. Indian residential IPs work for low volume; mobile is required for sustained throughput.

import random

IN_MOBILE_PROXIES = [
    {"server": "socks5://us:pw@in-jio-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@in-airtel-1.proxy.example.com:1080"},
    {"server": "socks5://us:pw@in-vi-1.proxy.example.com:1080"},
]

async def scrape_with_proxy(url: str):
    proxy = random.choice(IN_MOBILE_PROXIES)
    return await scrape_flipkart_in(url, proxy=proxy)

For mobile proxy strategy, see best mobile proxy providers 2026.

Discovering product URLs

Flipkart sitemaps are split by category:

import httpx
import xml.etree.ElementTree as ET

async def list_flipkart_sitemaps() -> list[str]:
    sitemap_index = "https://www.flipkart.com/sitemap.xml"
    async with httpx.AsyncClient(timeout=30) as client:
        r = await client.get(sitemap_index, headers={"User-Agent": "Mozilla/5.0"})
        root = ET.fromstring(r.text)
        ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
        return [s.find("sm:loc", ns).text for s in root.findall("sm:sitemap", ns)]

For category-driven discovery, browse search results with paginated queries:

async def search_flipkart(query: str, page: int = 1) -> list[dict]:
    url = f"https://www.flipkart.com/search?q={query}&page={page}"
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="en-IN")
        pg = await ctx.new_page()
        await pg.goto(url, wait_until="networkidle")
        items = await pg.locator("a[href*='/p/']").all()
        results = []
        for item in items[:60]:
            href = await item.get_attribute("href")
            title = await item.text_content()
            if href:
                results.append({"url": f"https://www.flipkart.com{href}", "title": (title or "").strip()})
        await browser.close()
    return results

Comparison to other Indian markets

Market	Bot defense	Volume	Mobile proxy required
Flipkart India	High	Largest with Amazon	Yes
Amazon India	High	Largest with Flipkart	Yes
Myntra (fashion)	High	Largest fashion	Yes
Meesho	Medium	Large	Recommended
Snapdeal	Medium	Smaller	Optional
JioMart	High	Growing	Recommended
BigBasket (grocery)	Medium	Medium	Recommended

For broader India coverage, Flipkart and Amazon India together cover roughly 80 percent of Indian online retail GMV.

Geographic IP pinning

Flipkart serves slightly different content based on detected IP geolocation. Delivery options, COD availability, and even some pricing tiers vary by city. For consistent scraping:

CITY_PROXY_POOLS = {
    "delhi": ["socks5://us:pw@in-jio-delhi-1...", "socks5://us:pw@in-jio-delhi-2..."],
    "mumbai": ["socks5://us:pw@in-jio-mumbai-1...", ...],
    "bangalore": ["socks5://us:pw@in-jio-bangalore-1...", ...],
}

async def scrape_for_city(url: str, city: str = "delhi"):
    proxy = {"server": random.choice(CITY_PROXY_POOLS[city])}
    return await scrape_flipkart_in(url, proxy=proxy)

For city-level price intelligence, sample the same product across multiple metro pools weekly.

Reviewer-level data and sentiment

For sentiment analysis, the review payload exposes:

Field	Use
rating (1-5)	Numeric sentiment
text	Long-form review text
helpful_count	Community endorsement
verified_buyer	Trust signal
images	Photo evidence (counterfeit detection)
date	Review recency

For brand intelligence, the verified_buyer flag is the most important. Reviews from non-verified buyers are roughly 4x more likely to be fake.

Cross-marketplace deduplication

For brand intelligence projects covering both Flipkart and Amazon India, deduplicating SKUs is non-trivial because each platform uses its own product ID system. The right approach:

1. Match by EAN/UPC barcode where present (often missing on Flipkart)
2. Fall back to fuzzy match on title + brand + key attributes
3. Use LLM-based similarity for ambiguous cases

async def fuzzy_match_skus(flipkart_item: dict, amazon_items: list[dict]) -> dict | None:
    # cheap embedding-based similarity, then validate top match with LLM
    candidates = embedding_search(flipkart_item["title"], [a["title"] for a in amazon_items], top_k=3)
    for c in candidates:
        if await llm_verify_match(flipkart_item, c):
            return c
    return None

Production patterns

Three patterns matter.

First, throttle conservatively. 1-2 requests per second per IP, longer pauses on first session. Flipkart’s challenge mechanism escalates fast under sustained traffic.

Second, intercept the page-fetch API. The API path returns cleaner JSON than parsing the React-rendered HTML. The interception pattern shown above is the production-grade approach.

Third, cache CSRF and session cookies. Pull them from a warm session at the start of a worker run, reuse for the worker lifetime, refresh on auth failure.

async def get_session_cookies():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        ctx = await browser.new_context(locale="en-IN")
        pg = await ctx.new_page()
        await pg.goto("https://www.flipkart.com", wait_until="networkidle")
        cookies = await ctx.cookies()
        await browser.close()
        return cookies

Big Billion Days strategy

Flipkart’s flagship sales event runs early October and triples or quadruples site traffic. For brand intelligence during BBD:

Pre-BBD (1 week prior): snapshot all watched SKUs at high frequency to capture the baseline.

During BBD: switch to lighter polling on watched SKUs. Avoid scraping unrelated catalog data because the bot defense tightens.

Post-BBD (1 week after): resume normal scraping. Compare price trajectories to identify items that retained discounts versus those that snapped back to MRP.

Logging the entire BBD pricing graph for important SKUs is gold for next year’s pricing strategy work.

Storage schema

CREATE TABLE flipkart_in_products (
    id BIGSERIAL PRIMARY KEY,
    pid TEXT NOT NULL,
    url TEXT NOT NULL,
    title TEXT NOT NULL,
    brand TEXT,
    price_inr NUMERIC(12,2) NOT NULL,
    original_price_inr NUMERIC(12,2),
    discount_percent INTEGER,
    rating NUMERIC(3,2),
    review_count INTEGER,
    in_stock BOOLEAN NOT NULL,
    extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    raw_jsonb JSONB,
    UNIQUE(pid)
);
CREATE INDEX idx_flipkart_extracted_at ON flipkart_in_products(extracted_at);
CREATE INDEX idx_flipkart_brand ON flipkart_in_products(brand);

Real benchmark numbers

A March 2026 production run, 10,000 Flipkart India products with the API capture pattern:

Metric	Value
Success rate	92%
Median latency per item	6.4 s
p99 latency	21 s
Cost per 1000 items	$19
PerimeterX challenge rate	6.1%
429 throttle rate	1.7%

PerimeterX challenges are the leading failure cause. With proper stealth and IP rotation, you can keep the rate under 7 percent.

Stealth fingerprint hardening

Out-of-the-box headless Chromium fails on Flipkart within roughly 60 requests per IP. Combine the AutomationControlled patch with realistic Indian mobile fingerprints:

context_init = """
Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, 'languages', {get: () => ['en-IN', 'en', 'hi']});
Object.defineProperty(navigator, 'platform', {get: () => 'Linux armv8l'});
Object.defineProperty(screen, 'colorDepth', {get: () => 32});
window.chrome = {runtime: {}, app: {}};
"""

await ctx.add_init_script(context_init)

Combined with mobile IPs and warm sessions, success rates climb from roughly 60 percent to over 90 percent.

Cost expectations

10,000 Flipkart India products per month with Indian mobile proxies:

Component	Cost
Indian mobile proxy traffic (~3MB/page)	$90-$150
Browser compute	$40
LLM extraction (optional)	$30
Total	$160-$220

Indian mobile IPs are competitive in cost with Indonesian and Thai mobile pools.

Legal considerations

India’s Digital Personal Data Protection Act (DPDP Act, enforced from 2024) regulates personal data. Public commercial data (product listings, prices, seller-level data at city granularity) is not personal data.

The Flipkart terms of use prohibit automated access. Civil enforcement only; no criminal exposure for scraping public commercial data in India.

For deeper compliance reading, see our India DPDP Act for scrapers.

Variants and SKUs

Flipkart products often have variants (size, color, model). The API exposes them in the variantOptions and swatchOptions widgets. Variant-level pricing matters for competitive intelligence:

def parse_variants(api_data: dict) -> list[dict]:
    slots = api_data.get("RESPONSE", {}).get("slots", [])
    variant_widget = next((s for s in slots if s.get("widget", {}).get("type") == "VARIANT_OPTIONS"), None)
    if not variant_widget:
        return []
    options = variant_widget.get("widget", {}).get("data", {}).get("variantOptions", [])
    return [
        {"value": o.get("value"), "pid": o.get("productId"), "available": o.get("available")}
        for o in options
    ]

Reviews

Reviews are paginated client-side. Each page loads roughly 10 reviews:

async def scrape_reviews(product_url: str, max_pages: int = 5):
    reviews = []
    for page in range(1, max_pages + 1):
        url = f"{product_url}/product-reviews?page={page}"
        # browser-based fetch and parse review divs
        pass
    return reviews

Review text is personal commentary; reviewer names are personal data under DPDP. Strip both for any pipeline beyond aggregate ratings unless you have legal basis for retention.

Flipkart-specific data points

A few fields specific to Flipkart that other platforms do not expose:

fAssured flag: Flipkart’s quality and fast-shipping certification. Strong predictor of conversion and the Indian equivalent of Amazon’s Prime badge.

bankOffers: array of bank-specific discounts (HDFC, ICICI, SBI cashback offers). These can shave 10 to 15 percent off the headline price for cardholders.

exchangeOffer: trade-in pricing for old devices, common on phones, laptops, and televisions. Captures the effective post-trade price.

emiOptions: EMI (installment) terms, including no-cost EMI flag. EMI dominates large-ticket purchases in India.

def extract_flipkart_specific(api_data: dict) -> dict:
    info = _get_product_info(api_data)
    pricing = info.get("pricing", {})
    return {
        "f_assured": info.get("fAssured", False),
        "bank_offers_count": len(pricing.get("bankOffers", [])),
        "best_bank_discount_inr": max(
            (o.get("discount", {}).get("value", 0) for o in pricing.get("bankOffers", [])),
            default=0,
        ),
        "emi_starting_inr": pricing.get("emi", {}).get("startingValue"),
        "no_cost_emi": pricing.get("emi", {}).get("noCost", False),
    }

Q&A and Q&A sentiment

Flipkart has a buyer Q&A section that often contains questions other shoppers ask. The endpoint:

async def get_qna(pid: str, session_cookies: dict) -> list[dict]:
    url = f"https://www.flipkart.com/api/3/product/{pid}/questions"
    async with httpx.AsyncClient(cookies=session_cookies) as c:
        r = await c.get(url, headers={"User-Agent": "Mozilla/5.0 ..."})
        return r.json().get("questions", [])

For brand monitoring, scraping competitor Q&A reveals customer concerns that the brand could address in their own listings.

Indian ecommerce calendar

Indian ecommerce has unique peak periods that affect scraping load:

• Republic Day Sales (late January)
• Independence Day Sales (mid-August)
• Big Billion Days (early October, Flipkart’s flagship event)
• Diwali season (late October to mid-November)
• New Year sales (December to early January)

During peak windows, expect 3 to 5x normal load on Flipkart infrastructure plus more aggressive bot defense. Scale your IP pool by 2x and increase pacing margins. Big Billion Days specifically: pause non-critical scraping for the week.

AI-driven extraction fallback

For pages where API interception fails, fall through to LLM extraction:

async def scrape_with_fallback(url: str) -> dict:
    try:
        return await scrape_flipkart_in(url)
    except (NoAPIPayloadError, KeyError):
        html = await fetch_html(url)
        return await llm_extract_product(html, schema=PRODUCT_SCHEMA)

The LLM fallback runs at roughly 4x the cost per page but catches the cases where the deterministic path breaks.

Frequently asked questions

Can I use Flipkart’s official Affiliate API?
The Flipkart Affiliate program offers an API for affiliates with rate limits and category restrictions. Useful for affiliate marketers; less useful for general competitive intelligence.

Why does my scraper start failing during Big Billion Days?
Flipkart traffic spikes during sales events. The bot defense team also tightens during these periods. Pause aggressive scraping during the BBD week and resume after.

How do I scrape Flipkart Camera and similar specialty categories?
Same patterns. Specialty categories often have richer attribute data; capture the full attributes block as JSONB.

Does Flipkart support multiple sellers per product like Amazon?
Yes. The sellers block lists all sellers offering the same product, with their respective prices, ratings, and shipping options. Critical for brand intelligence on grey-market sellers.

Will residential IPs work?
For low volume, yes. For sustained scraping (tens of thousands of pages per day), Indian mobile IPs are required.

How do I scrape Flipkart Plus exclusive offers?
Flipkart Plus pricing requires a logged-in session with a Plus subscription. Saved storage state from a manual login enables this. Treat the credentials as sensitive.

Can I scrape Myntra (Flipkart-owned) with the same tooling?
Largely yes. Myntra uses similar PerimeterX defense and a similar React frontend. The API shapes differ but the patterns transfer.

Can I scrape Flipkart Wholesale (B2B)?
The Wholesale platform requires business registration. Public-facing pricing is limited; full catalogs are gated behind login.

How do I track price drops on a watch list of products?
Snapshot daily for the watch list, store in a price_history table, run a query for items where the latest price is at least 10 percent below a 7-day rolling average. Send alerts via Telegram or email.

Does Flipkart have variant-level reviews?
Yes. Reviews are tagged with the variant they were written about (size and color). Capture the variant tag for accurate variant-specific sentiment.

How do I detect out-of-stock pulse for inventory intelligence?
Track availability.status over time. Repeated OUT_OF_STOCK transitions correlate with sales velocity and demand intelligence.

Common production gotchas

A few patterns that cause issues in Flipkart scraping:

The PerimeterX cookie expires after 30 minutes of inactivity. Sessions need refresh more often than the cookie lifetime suggests.

Indian carrier IPs have higher latency (200 to 500 ms) than residential. Plan for slower per-page timing.

The Indian numbering system (lakh, crore) only appears in display, not in API. The API returns plain integers.

Flipkart occasionally rolls out region-specific UI experiments based on IP geolocation (Delhi vs Mumbai vs Bengaluru can see slightly different layouts). Pin the IP region for consistent scraping.

Affiliate URLs include tracking parameters that change. Strip them before storing canonical URLs.

The mobile site (m.flipkart.com) and desktop site return different DOMs and slightly different API shapes. Pick mobile for cleaner data and stick with it.

Cost optimization for Flipkart specifically

Three patterns specifically valuable for Flipkart:

Block image and font requests via Playwright route interception. Flipkart product pages load 5 to 7 MB of imagery by default. Blocking cuts proxy bandwidth by 75 percent.

Cache API responses by product_id. The same PID rarely has changing data within a 4-hour window outside of flash sales.

Use the API capture pattern over HTML parsing. The intercepted JSON contains structured data; HTML parsing is brittle as Flipkart frequently ships frontend updates.

Combined, these cut typical per-page cost from $0.040 to $0.018, more than half.

Compliance specifics for India

Beyond DPDP, Indian ecommerce data has a few specific regulations to consider:

The Consumer Protection (E-Commerce) Rules 2020 require seller information transparency. The data is public on Flipkart and not regulated for scraping.

The Information Technology Rules 2021 govern cybersecurity but exempt scraping of public commercial data.

For scraping personal reviewer data (names, profile photos), DPDP requires explicit consent which you do not have. Strip personal identifiers from any review data you persist.

For broader Asian ecommerce coverage, browse the ecommerce category.