Scraping car dealership inventory data in 2026

Scrape dealership inventory and you tap into one of the largest single-vertical datasets in commercial scraping. The U.S. has roughly 18,000 franchised dealerships and another 35,000 independent used-car dealers, each maintaining a public inventory feed. The aggregator sites (AutoTrader, Cars.com, CarGurus, TrueCar) sit on top, and OEM dealer locators (BMW, Toyota, Ford) sit alongside. The combined dataset gives you near-real-time visibility into pricing, model mix, days-on-lot, and regional inventory distribution. The scraping landscape is shaped by three things: an aggressive layer of bot detection on the major aggregator sites, a long tail of dealer-by-dealer scraping required for full coverage, and an inventory schema that varies meaningfully across sources.

This guide focuses on the U.S. market because it is the largest and best-documented. The patterns transfer to UK and EU dealership scraping with minor adjustments.

Source taxonomy and data shapes

The dealership inventory data ecosystem has three distinct source types, each with its own scraping characteristics.

Aggregator sites consolidate inventory from thousands of dealers into a single browseable catalogue. AutoTrader, Cars.com, and CarGurus are the dominant U.S. aggregators. They expose listing search APIs (most undocumented) that return VIN, make, model, year, mileage, asking price, dealer name, and location. The advantage is breadth in a single source. The disadvantage is aggressive bot defenses because the aggregators are themselves businesses that monetize the data.

OEM dealer locators (BMW, Mercedes, Toyota, Ford, GM brand sites) expose new-vehicle inventory across the manufacturer’s authorized dealer network. These tend to be less aggressively defended than aggregators because they are designed for consumers shopping for a specific brand. The schema is brand-specific and includes manufacturer-specific options like build configurations.

Direct dealership websites are the long-tail source. Most dealerships use one of a handful of website platforms (DealerOn, Dealer.com, DealerInspire, AutoTrader’s own platform). Each platform has its own URL structure and inventory feed format, but within a platform the structure is consistent.

import httpx
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15",
    "Accept": "text/html,application/json",
    "Accept-Language": "en-US,en;q=0.9",
}

async def scrape_dealer_inventory(dealer_url: str, proxy: str):
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(f"{dealer_url}/inventory")
        if r.status_code != 200:
            return []
        soup = BeautifulSoup(r.text, "lxml")
        # Most dealer platforms embed an inventory JSON in script tags
        for script in soup.find_all("script"):
            if script.string and "vehicleInventory" in (script.string or ""):
                return parse_dealer_json(script.string)
        return []

For comprehensive coverage, scrape both the aggregators and a sample of direct dealer sites. The aggregators give you breadth fast; direct dealer sites give you the most current pricing because aggregators cache for 12-24 hours.

VIN as the canonical identifier

The Vehicle Identification Number is the universal canonical identifier for any specific vehicle. Every car has a unique 17-character VIN. Aggregator listings, dealer feeds, and OEM dealer locators all expose VINs, which makes cross-source deduplication straightforward.

CREATE TABLE vehicle_listing_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    vin VARCHAR(17) NOT NULL,
    source VARCHAR(32) NOT NULL,
    dealer_id VARCHAR(64),
    dealer_zip VARCHAR(10),
    asking_price_usd INT,
    mileage INT,
    days_on_lot INT,
    PRIMARY KEY (snapshot_at, vin, source)
);
CREATE INDEX vin_idx ON vehicle_listing_snapshot(vin);

Tracking the same VIN across sources reveals interesting patterns. The same vehicle often sits at different asking prices on different aggregators because dealers list at different price points across channels. The dealer’s own website often has the freshest price; aggregators lag by 12-24 hours.

Pricing normalization across sources

Aggregator sites display “asking price” but the meaning varies. Some include freight and dealer prep fees; others exclude them. Some show MSRP minus advertised incentives; others show the dealer’s actual posted price. Build a normalization step that captures both the raw advertised price and a normalized “out-the-door estimate” that adds estimated taxes and fees.

Source	Price field	Includes destination fee	Includes estimated tax
AutoTrader	listingPrice	Sometimes	No
Cars.com	priceWithFees	Usually	No
CarGurus	dealerPrice	Sometimes	No
OEM site	msrpPlusFees	Yes	No
Direct dealer	varies	Varies	No

For brand monitoring use cases, store the raw fields from each source and compute the normalized comparison at query time. Hard-coding normalization at scrape time loses the underlying signal and makes downstream debugging harder when the upstream definitions change.

Days-on-lot and price-history derivation

The two most analytically valuable derived metrics are days-on-lot (how long a vehicle has been listed) and price-history (the sequence of price changes during the listing). Neither is exposed directly by most sources, but both can be derived from snapshot diffs.

For days-on-lot, track the first appearance of each VIN in your snapshots and compute the difference in days against the current snapshot. For price-history, compare the asking_price field across consecutive snapshots and emit a price-change event whenever it differs.

def derive_price_changes(prev_snap: dict, curr_snap: dict) -> list:
    changes = []
    for vin, curr_row in curr_snap.items():
        prev_row = prev_snap.get(vin)
        if not prev_row:
            continue
        if prev_row["asking_price"] != curr_row["asking_price"]:
            changes.append({
                "vin": vin,
                "old_price": prev_row["asking_price"],
                "new_price": curr_row["asking_price"],
                "delta": curr_row["asking_price"] - prev_row["asking_price"],
                "changed_at": now(),
            })
    return changes

The price-change event stream is the foundation for the dealer-pricing-strategy reports that a finance team or a brand team actually wants to consume.

Proxy strategy for dealership scraping

Aggregator sites enforce aggressive bot detection and require U.S. residential or mobile IPs for sustained scraping. Direct dealer sites are much less aggressive and often work from datacenter IPs as long as you respect basic rate limits. OEM dealer locators sit in between.

For workloads under 5,000 listings per day, a small U.S. residential pool is sufficient. For comprehensive daily snapshots covering 500,000+ active listings, a dedicated mobile proxy pool with 50+ ports is the production-grade approach. The math works out at $300-500 per month for the proxy infrastructure, which is small relative to the analytical value of the dataset.

For deeper proxy strategy guidance, see our residential vs mobile proxy comparison and our best web scraping APIs ranking.

Detecting and routing around bot challenges

When automotive inventory sources flag your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment....

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. For pages that absolutely must be fetched, have a fallback path that uses a headless browser. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of vertical. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Pipeline orchestration and scheduling

For any non-trivial automotive inventory scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and sources.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_source(source_id: str, page: int):
    return crawl_one_page(source_id, page)

@flow(name="automotive-inventory-daily-sweep")
def daily_sweep(source_ids: list):
    futures = []
    for sid in source_ids:
        for page in range(1, 30):
            futures.append(fetch_source.submit(sid, page))
    return [f.result() for f in futures]

Run the flow on a cadence aligned to how dynamic the underlying data is. For automotive inventory where pricing or availability changes intraday, a 4-6 hour cadence catches meaningful movements without driving up proxy costs.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. Structural checks verify that every required field is present and of the expected type. Distributional checks compare the current snapshot against recent history. Semantic checks compare related fields for consistency.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same record within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads. The collection layer handles the network conversation. The parsing layer transforms raw bytes into structured records. The storage layer holds the canonical snapshots in a query-optimized format. The serving layer exposes the data to consumers.

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Legal and compliance considerations

Public data across automotive inventory sources is generally treated as fair to scrape in most jurisdictions, but always confine your collection to non-personal data: identifiers, prices, structured attributes, and aggregates. Avoid collecting personally identifying details, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Sample analytics queries

Once your snapshots are landing reliably, the analytics layer is where the value materializes:

-- Trend over the last 30 days
SELECT date_trunc('day', snapshot_at) AS day,
       COUNT(*) AS records,
       AVG(price) AS avg_price
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY 1
ORDER BY 1;

-- New entities first seen in the last 14 days
SELECT entity_id, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY entity_id
HAVING MIN(snapshot_at) > now() - interval '14 days';

Add a category share view, a source concentration view, and a price-volatility view and you have a solid foundation for a automotive inventory intelligence product.

Versioning your scraper for source evolution

Every dealership inventory source evolves its schema regularly. New fields appear, old fields are deprecated, and pricing display logic changes. Your scraper code has to evolve with these changes, and a versioning pattern that keeps old data interpretable is critical. Stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads. The collection layer handles the network conversation. The parsing layer transforms raw bytes into structured records. The storage layer holds the canonical snapshots in a query-optimized format. The serving layer exposes the data to consumers.

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same VIN within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output.

Building a regional dealer panel for ongoing intelligence

The most defensible analytical product on top of dealership data is a regional dealer panel: a curated set of 500-1,000 dealers across geographic and brand segments that you snapshot consistently every day. The panel approach has three advantages over scraping the entire universe.

First, panel-based intelligence has cleaner longitudinal continuity. The same set of dealers appearing in every snapshot lets you compute month-over-month and year-over-year changes without having to deduplicate against the broader population.

Second, the panel reduces proxy cost dramatically. Instead of scraping 50,000 dealer sites daily, you scrape 1,000 with high reliability. The cost reduction is 50x and the analytical signal is often stronger because the panel is balanced by region and brand.

Third, the panel approach lets you weight the results to match the underlying U.S. population of dealers. With known weights, you can produce population-level estimates from the panel that are more credible to downstream consumers than ad-hoc full-population scrapes that may have coverage gaps.

def weighted_panel_metric(panel_df, weights_df, metric_col):
    merged = panel_df.merge(weights_df, on=["region", "brand"])
    return (merged[metric_col] * merged["weight"]).sum() / merged["weight"].sum()

The panel design itself is the analytical asset. Document the inclusion criteria, the weighting scheme, and the refresh cadence in a methodology doc that you publish alongside the data product.

Common pitfalls when scraping dealership inventory

Three failure modes show up across nearly every dealership scraping project. The first is VIN duplication across dealer groups. A vehicle that moves between dealers in the same auto group keeps its VIN but appears under a new listing id. A scraper that deduplicates by listing id alone double-counts inventory. Always include VIN as a secondary deduplication key and reconcile at the VIN level downstream.

The second is incentive vs sticker price confusion. Manufacturer incentives, dealer cash, and trade-in bonuses are layered separately on the listing page. The MSRP, dealer_price, and out_the_door_price are three different numbers that can differ by $3,000-$8,000. Capture all three and let the analytics layer decide which is canonical for the question being answered.

The third is days-on-lot misattribution. Most platforms compute days-on-lot from the first scrape, not from the original listing date. A vehicle that was on the lot for 60 days before your scraper started shows up as a fresh listing. Backfill the original listing date from VIN history APIs (Carfax, AutoCheck) for vehicles you care about, or accept that the first 90 days of your dataset will undercount aging inventory.

FAQ

Are dealership inventory pages legal to scrape?
Vehicle inventory data is generally considered public commercial information. Aggregator sites have terms of service that often prohibit automated access; their enforcement focuses on commercial competitors. Confine your collection to non-personal data and document your lawful basis for processing.

What’s the typical refresh rate I should target?
Once daily catches the major price-change signals. Twice daily (early morning and late evening) catches dealer-set pricing changes that happen at the start and end of the business day. For real-time alerting on price changes, hourly is feasible but increases proxy costs significantly.

How should I handle private-seller listings on Craigslist or Facebook Marketplace?
Private-seller data has stronger personal-data implications. Limit collection to the listing description and price; avoid storing seller contact details. For most analytical use cases, dealer inventory alone provides sufficient signal without the privacy complexity of private-seller data.

Does CARFAX or AutoCheck data overlap with what I can scrape?
CARFAX and AutoCheck sell vehicle history reports per VIN and are not realistically scrapable. They are licensed data products. For analytics that need vehicle history, license the data; for analytics that focus on listing dynamics, the public listings are the relevant dataset.

Can I use the data to compute a fair-market-value model?
Yes. With 30+ days of daily snapshots across 100,000+ listings, you have enough data to fit a regression model for price as a function of make, model, year, mileage, region, and days-on-lot. KBB and Edmunds publish similar models commercially; building your own is a substantial undertaking but feasible.

How do I track price drops on a specific VIN over time?
Hash on VIN as the primary key and store every price observation as a row, not a column. This makes time-series queries trivial in any SQL backend.

Do dealer websites use dynamic pricing engines?
Increasingly yes. vAuto, Dealer.com, and several others reprice inventory automatically based on market data. Expect price changes within minutes of competitor movements on hot SKUs.

To build broader vertical scraping pipelines, browse the ecommerce scraping category for tooling reviews and framework deep dives.

How to scrape Takealot South Africa in 2026

Scrape Takealot South Africa and you tap into the dominant ecommerce platform in South Africa, owned by Naspers and the regional incumbent against which all other South African online retailers benchmark. Takealot operates a hybrid first-party and marketplace model with its own logistics network and the Mr D Food delivery sub-brand. The scraping landscape is shaped by three things: a JSON product API that powers the front end, a moderate Cloudflare layer that profiles non-South African traffic, and a relatively scrape-friendly architecture compared to Western marketplaces of similar scale.

This guide focuses on Takealot at takealot.com as the canonical example.

Mapping Takealot URL and JSON structure

Takealot product URLs follow the pattern https://www.takealot.com/<product-slug>/PLID<plid>. The trailing PLID (Product Listing ID) is the canonical SKU identifier. Behind every product page sits a JSON endpoint at https://api.takealot.com/rest/v-1-12-0/product-details/PLID<plid>. The endpoint returns price, stock, full description, images, seller information, and the offer stack.

import httpx

def api(plid: str) -> str:
    return f"https://api.takealot.com/rest/v-1-12-0/product-details/PLID{plid}"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-ZA,en;q=0.9",
}

async def fetch_takealot(plid: str, proxy: str):
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(api(plid))
        if r.status_code == 200:
            return r.json()
        return None

The response includes core (canonical product), buybox (the winning seller offer with price, stock, fulfillment), gallery (image set), attributes, variants, and reviews_summary. For most analytical use cases the API alone is sufficient and you do not need to fetch the rendered HTML.

South African proxy strategy

Takealot’s bot detection profiles visitor IP geography. South African residential or mobile IPs through Vodacom, MTN South Africa, or Cell C are strongly preferred for sustained scraping. Pan-African residential pools work for light loads but degrade at higher volumes. European residential pools work surprisingly well for short bursts because of historical CDN routing patterns, but the success rate degrades quickly under sustained load.

For workloads under 5,000 product reads per day, a small South African residential pool with sticky 15-minute sessions is sufficient. For higher volumes, dedicated South African mobile ports through Vodacom are the cleaner path.

Crawling the category tree

Takealot exposes a category tree at https://api.takealot.com/rest/v-1-12-0/category/all. Each category has a url_key and an id. The listing endpoint at https://api.takealot.com/rest/v-1-12-0/searches/products accepts category, sort, and pagination parameters, with practical limits of 100 pages of 36 products each.

async def crawl_category(category_slug: str, proxy_pool, max_pages: int = 100):
    results = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = "https://api.takealot.com/rest/v-1-12-0/searches/products"
        params = {
            "filter": f"Category:{category_slug}",
            "sort": "Relevance",
            "rows": 36,
            "page": page,
        }
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url, params=params)
            if r.status_code != 200:
                break
            results.extend(r.json().get("results", []))
    return results

For broader categories, decompose by brand or price-band facets exposed in the search response.

Buybox vs. all offers

Takealot follows the Amazon-style buybox model where one seller wins the default offer position on a product page. The product detail JSON exposes both the buybox winner and the full offer stack. For brand monitoring, the buybox tracking is the primary signal because it determines what most buyers see, but the full offer stack reveals the gray-market and parallel-import landscape.

Field	Source	Analytical use
buybox.price	API	Default visible price most buyers see
offers[].price	API	Full price ladder across all sellers
buybox.seller	API	Current buybox winner
offers[].seller	API	All sellers offering the SKU

Schema for Takealot snapshots

CREATE TABLE takealot_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    plid VARCHAR(16) NOT NULL,
    seller_id VARCHAR(64) NOT NULL,
    is_buybox BOOLEAN,
    price_zar DECIMAL(12,2),
    list_price_zar DECIMAL(12,2),
    in_stock BOOLEAN,
    fulfillment VARCHAR(32),
    PRIMARY KEY (snapshot_at, plid, seller_id)
);

For dynamic-pricing competitors, snapshot every 4-6 hours captures meaningful changes. For weekly category reports, daily is sufficient. Take care to preserve the buybox winner per snapshot so you can compute buybox-flip frequency, which is one of the most useful signals for sellers competing for placement.

Detecting and routing around CAPTCHA challenges on Takealot

When Takealot flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real South Africa residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle.

Working with ZAR pricing and FX normalization

Pricing on Takealot is denominated in ZAR, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local ZAR and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread.

Comparing Takealot to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Takealot	South Africa	Large	High
Bidorbuy	Adjacent markets	Medium	Medium
Loot	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Takealot but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal.

Operational monitoring and alerting

Every production scraper needs three monitoring layers. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for South Africa

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but South Africa has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@flow(name="takealot-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    return [f.result() for f in futures]

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is.

Sample analytics queries

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days';

These queries power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product.

Building robust deduplication across noisy listings

The long-tail catalogue is full of near-duplicate listings. The standard deduplication approach uses a three-pass funnel: exact match on EAN, normalized title plus brand TF-IDF similarity, then perceptual image hash similarity.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge or under-merge.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp using freshness deadline, volatility, and business priority signals to decide what to refetch on each cycle. Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads.

The collection layer handles the network conversation: HTTP requests, proxy assignment, retry logic, and rate limit enforcement. It should know nothing about the data shape and nothing about how the data will eventually be queried. Its only job is to fetch raw bytes reliably and hand them off to the next layer with metadata about which IP, which user agent, and which timestamp produced them.

The parsing layer transforms raw bytes into structured records. It owns the schema, the field normalization, and the validation rules. When the upstream HTML or JSON structure changes, only the parsing layer needs to adapt. Keep parsers idempotent and version them aggressively so old raw bytes can be re-parsed when you discover bugs.

The storage layer holds the canonical snapshots in a query-optimized format. For most ecommerce datasets, a column-oriented store like DuckDB, ClickHouse, or BigQuery outperforms row-oriented Postgres at analytical scale. The trade-off is write latency and update support; column stores prefer append-only and bulk loads, which fits the snapshot model naturally.

The serving layer exposes the data to consumers, whether that is a BI dashboard, an API for downstream systems, or an alerting pipeline. Keep the serving layer denormalized and pre-aggregated where possible. Recomputing complex analytics on every dashboard load wastes resources and hurts responsiveness.

# Pseudo-code for the four-layer split
async def collect(url: str, proxy_pool) -> RawFetch:
    proxy = proxy_pool.next()
    response = await http_get(url, proxy)
    return RawFetch(url=url, body=response.text, fetched_at=now(), ip=proxy.ip)

def parse(raw: RawFetch) -> Snapshot:
    data = json.loads(raw.body)
    return Snapshot(sku=data["id"], price=data["price"], ...)

def store(snapshot: Snapshot, db) -> None:
    db.append("snapshots", snapshot)

def serve(query: str, db) -> list:
    return db.query(query)

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. The checks fall into three categories: structural, distributional, and semantic.

Structural checks verify that every required field is present and of the expected type. A snapshot row missing the price field is not a real snapshot. A row with a negative price is not a real price.

Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer SKUs than yesterday, something broke either in collection or in the upstream catalogue. Either way, the on-call engineer needs to investigate before downstream consumers see broken data.

Semantic checks compare related fields for consistency. If a SKU shows in_stock = true but stock_quantity = 0, one of the fields is wrong. If the discount percentage is computed from list_price and selling_price, the computed value should match the stated discount field.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    invalid = [r for r in snapshot if r.get("price", -1) < 0]
    if invalid:
        errors.append(f"{len(invalid)} rows have invalid price")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same SKU within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

Common pitfalls when scraping Takealot

Three issues catch most teams. The first is plid vs tsin confusion. Takealot uses plid (product listing id) and tsin (Takealot stock-keeping number) interchangeably in URLs and APIs. The plid identifies the product page; the tsin identifies a specific variant. Joining datasets on the wrong key collapses variants into the parent product and loses color/size pricing.

The second is Daily Deals vs Blue Dot Sale staleness. Takealot’s headline promotions expire on a fixed cadence but the cached product detail JSON can lag by 5-15 minutes after expiry. A snapshot taken at the boundary captures a price that is no longer purchasable. Validate active promotions by cross-checking the promotion_end_time epoch against the scrape timestamp.

The third is third-party seller marketplace dilution. Takealot’s marketplace lets third-party sellers list against the same parent listing. The Buy Box price can flip between Takealot first-party and a marketplace seller within minutes. Capture buy_box_seller on every snapshot or your time series will look noisier than the underlying market is.

FAQ

Is the Takealot API officially documented?
The api.takealot.com endpoints are the same endpoints used by the public web site. They have been stable for several years but are not contractually supported.

Can I scrape Takealot from European or US IPs?
For light occasional reads, yes. For sustained scraping, Takealot blocks non-South African IPs after a few hours of activity. South African residential or mobile IPs are strongly preferred.

Does Takealot expose stock counts in the API?
The API returns availability boolean and a low-stock indicator but not exact stock counts for most SKUs. For SKUs with very low stock (under 5 units), Takealot sometimes shows the exact count in the buybox response.

How does Takealot handle the Mr D Food sub-brand?
Mr D Food uses a separate API surface focused on hyperlocal restaurant delivery. The patterns here apply to the main Takealot retail catalogue. Plan for a separate code path if your project covers Mr D.

What about Takealot’s marketplace seller restrictions?
Takealot vets marketplace sellers and has different fulfillment options (FBT for Fulfilled by Takealot, FBM for Fulfilled by Merchant). The fulfillment field in the API exposes which option each seller uses, which matters for delivery promise analytics.

Does Takealot block non-South African IPs?
Casual lookups succeed from most regions. Sustained scraping at production volume requires South African residential or mobile IPs. JNB and CPT proxies perform best in our testing.

How do I separate Takealot’s first-party stock from marketplace stock?
The merchant_id field identifies Takealot’s house merchant (typically id 1) versus third-party sellers. Filter on merchant_id == 1 to isolate first-party stock for retail analytics.

To build a broader South Africa ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives.

How to scrape Konga Nigeria product listings

Scrape Konga Nigeria and you tap into the second-largest general-merchandise marketplace in Nigeria, alongside Jumia. Konga has shifted ownership multiple times since its founding in 2012 and has rebuilt its catalogue and logistics under each phase. By 2026 the platform serves a strong base in electronics, fashion, and groceries with a hybrid first-party and third-party seller model. The scraping landscape is shaped by three things: a JSON catalogue API that powers the front end, an aggressive Cloudflare front end that profiles non-Nigerian traffic strictly, and a SKU schema that overlaps partially with Jumia for cross-marketplace analytics.

This guide focuses on Konga at konga.com as the canonical example. The patterns also apply to KongaPay merchant subsystems with minor adjustments.

Mapping Konga URL and JSON structure

Konga product URLs follow the pattern https://www.konga.com/product/<product-slug>-<productId>. The trailing productId is the canonical SKU. Behind every product page sits a JSON endpoint at https://api.konga.com/v1/catalog/products/sku/<productId>. The endpoint returns price, stock, full description, images, and seller information.

import httpx

API = "https://api.konga.com/v1/catalog/products/sku"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-NG,en;q=0.9",
}

async def fetch_konga(sku: str, proxy: str):
    url = f"{API}/{sku}"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return None

The response includes the canonical product object with sku, name, brand, original_price, special_price, stock_status, seller_name, seller_id, categories, and rating_summary. For most analytical use cases the API is sufficient.

Nigerian proxy strategy

Konga’s Cloudflare front end is aggressive against non-Nigerian IPs. Nigerian residential or mobile IPs through MTN, Airtel, 9mobile, or Glo are required for sustained scraping. Pan-African residential pools work for light loads but degrade at volume.

For workloads under 2,000 product reads per day, a small Nigerian residential pool with sticky 15-minute sessions is sufficient. For higher volumes, dedicated Nigerian mobile ports are the cleaner path because they sustain higher request rates without challenges.

Crawling the category tree

Konga exposes a category tree at https://api.konga.com/v1/catalog/categories. Each category has a url_key and an id. The listing endpoint at https://api.konga.com/v1/catalog/products accepts category, sort, and pagination parameters, with practical limits of 25 pages of 40 products each.

async def crawl_category(category_id: int, proxy_pool, max_pages: int = 25):
    results = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = "https://api.konga.com/v1/catalog/products"
        params = {
            "category_id": category_id,
            "page": page,
            "limit": 40,
            "sort": "relevance",
        }
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url, params=params)
            if r.status_code != 200:
                break
            items = r.json().get("products", [])
            if not items:
                break
            results.extend(items)
    return results

For broader categories like Phones and Tablets, decompose by brand and price band facets exposed in the search response.

Cross-checking Konga pricing against Jumia

The most analytically interesting Nigerian ecommerce signal is the price differential between Konga and Jumia for the same SKU. They overlap heavily on consumer electronics, household goods, and fashion. For brand monitoring, the price gap often signals which platform is running a promotion or which platform a particular seller is using as the price-leader channel.

Field	Konga	Jumia	Notes
Canonical ID	sku (string)	SKU embedded in URL	Different schemes
Mall equivalent	KongaCare	Jumia Mall	Both flag verified sellers
Price scheme	original + special	regular + sale	Both expose pre/post-discount
Update cadence	Hourly	Hourly	Both refresh throughout the day

For SKU matching across the two platforms, group by EAN where available, then by normalized title plus brand for the long tail. Match rates on Nigerian ecommerce specifically tend to be lower than other markets because EAN coverage is voluntary and many sellers use product names that emphasize Nigerian-specific marketing claims.

Detecting and routing around CAPTCHA challenges on Konga

When Konga flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Nigeria residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle.

Working with NGN pricing and FX normalization

Pricing on Konga is denominated in NGN, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local NGN and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread.

Comparing Konga to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Konga	Nigeria	Large	High
Jumia	Adjacent markets	Medium	Medium
Jiji	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Konga but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal.

Operational monitoring and alerting

Every production scraper needs three monitoring layers. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for Nigeria

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Nigeria has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@flow(name="konga-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    return [f.result() for f in futures]

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is.

Sample analytics queries

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days';

These queries power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product.

Building robust deduplication across noisy listings

The long-tail catalogue is full of near-duplicate listings. The standard deduplication approach uses a three-pass funnel: exact match on EAN, normalized title plus brand TF-IDF similarity, then perceptual image hash similarity.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge or under-merge.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp using freshness deadline, volatility, and business priority signals to decide what to refetch on each cycle. Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads.

The collection layer handles the network conversation: HTTP requests, proxy assignment, retry logic, and rate limit enforcement. It should know nothing about the data shape and nothing about how the data will eventually be queried. Its only job is to fetch raw bytes reliably and hand them off to the next layer with metadata about which IP, which user agent, and which timestamp produced them.

The parsing layer transforms raw bytes into structured records. It owns the schema, the field normalization, and the validation rules. When the upstream HTML or JSON structure changes, only the parsing layer needs to adapt. Keep parsers idempotent and version them aggressively so old raw bytes can be re-parsed when you discover bugs.

The storage layer holds the canonical snapshots in a query-optimized format. For most ecommerce datasets, a column-oriented store like DuckDB, ClickHouse, or BigQuery outperforms row-oriented Postgres at analytical scale. The trade-off is write latency and update support; column stores prefer append-only and bulk loads, which fits the snapshot model naturally.

The serving layer exposes the data to consumers, whether that is a BI dashboard, an API for downstream systems, or an alerting pipeline. Keep the serving layer denormalized and pre-aggregated where possible. Recomputing complex analytics on every dashboard load wastes resources and hurts responsiveness.

# Pseudo-code for the four-layer split
async def collect(url: str, proxy_pool) -> RawFetch:
    proxy = proxy_pool.next()
    response = await http_get(url, proxy)
    return RawFetch(url=url, body=response.text, fetched_at=now(), ip=proxy.ip)

def parse(raw: RawFetch) -> Snapshot:
    data = json.loads(raw.body)
    return Snapshot(sku=data["id"], price=data["price"], ...)

def store(snapshot: Snapshot, db) -> None:
    db.append("snapshots", snapshot)

def serve(query: str, db) -> list:
    return db.query(query)

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. The checks fall into three categories: structural, distributional, and semantic.

Structural checks verify that every required field is present and of the expected type. A snapshot row missing the price field is not a real snapshot. A row with a negative price is not a real price.

Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer SKUs than yesterday, something broke either in collection or in the upstream catalogue. Either way, the on-call engineer needs to investigate before downstream consumers see broken data.

Semantic checks compare related fields for consistency. If a SKU shows in_stock = true but stock_quantity = 0, one of the fields is wrong. If the discount percentage is computed from list_price and selling_price, the computed value should match the stated discount field.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    invalid = [r for r in snapshot if r.get("price", -1) < 0]
    if invalid:
        errors.append(f"{len(invalid)} rows have invalid price")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same SKU within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

Common pitfalls when scraping Konga

Three issues recur. The first is KongaPay-financing price masking. Konga shows the headline price on the product card and a financed monthly figure on the detail page. Some scrapers extract the monthly figure and treat it as the SKU price, which understates the cash price by an order of magnitude. Always pull the selling_price field from the product JSON, not from rendered HTML.

The second is seller-type confusion. Konga operates a first-party (Konga Retail) and a third-party seller pool on the same SKU. Pricing dynamics differ: first-party prices are stable for weeks, third-party prices reprice daily. Aggregating both into one population smears the trend signal. Segment by seller_type before computing any time-series metric.

The third is FX-driven price stale state. The Naira moves frequently against the USD. Sellers reprice imported electronics on a 1-3 day lag after FX shocks. A scraper that stores only the local-currency price loses the FX-attribution signal. Capture the scrape-time FX rate from a stable reference (CBN or Bloomberg) and compute USD-equivalent prices in a derived column.

FAQ

Is the Konga API officially documented?
The api.konga.com endpoints are the same endpoints used by the public web site. They have been stable for several years but are not contractually supported.

Can I scrape Konga from a UK or US residential IP?
For occasional product lookups, yes. For sustained scraping, Konga blocks non-Nigerian IPs aggressively. Nigerian residential or mobile IPs are strongly preferred.

Does Konga distinguish between first-party and third-party sellers?
Yes. The seller_id and seller_name fields identify the seller. KongaCare-flagged sellers receive verification badges. For analytics, separating first-party Konga listings from third-party sellers is essential because the price dynamics differ.

How does Konga handle the Naira’s frequent revaluation?
Konga prices update with the underlying Naira movements, but lag behind FX shocks by 24-72 hours as sellers reprice. For longitudinal analyses, normalize prices to a stable currency using daily reference FX rates.

What about Konga’s logistics promise vs. actual delivery?
The product page shows the logistics promise but the API does not expose delivery success rates. For analytics that need to assess seller delivery reliability, you have to scrape buyer reviews and aggregate the delivery-related sentiment, which requires more text processing.

Are there days when Konga rate-limits scrapers more aggressively?
Yes. Black Friday week (late November) and December gift season (Dec 15-24) carry the tightest rate limits. Pre-stage baseline snapshots in October to avoid being throttled during the actual event.

How do I match Konga SKUs to Jumia for cross-marketplace analytics?
Use brand + model + storage capacity as the canonical join. Konga’s GTIN coverage is sparse, so fuzzy title matching with a similarity threshold of 0.85 is the practical fallback.

Does Konga’s mobile app expose a different API surface than the web site?
Yes. The Android app uses protobuf endpoints under mobileapi.konga.com for personalization, while the public web JSON layer under api.konga.com covers catalog and pricing. For catalog and pricing intelligence the web endpoints are sufficient and easier to maintain. Reverse-engineering the protobuf surface only pays back when the analytics design requires the personalization signal.

To build a broader Nigeria ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives.

How to scrape Tiki Vietnam ecommerce

Scrape Tiki Vietnam and you tap into one of the three dominant ecommerce platforms in Vietnam, alongside Shopee Vietnam and Lazada Vietnam. Tiki was founded in 2010 and has built a strong reputation around fast 2-hour delivery (TikiNow) in major cities and a curated catalogue that leans into electronics, books, and household goods. The scraping landscape is shaped by three things: a publicly accessible JSON API that powers most of the site, Vietnamese-language content with Latin characters but heavy diacritics, and a moderate Cloudflare front end that profiles non-Vietnamese traffic.

This guide focuses on Tiki at tiki.vn as the canonical example. The patterns transfer to TikiNow city-specific catalogues with minor adjustments.

Mapping Tiki URL and JSON structure

Tiki product URLs follow the pattern https://tiki.vn/<product-slug>-p<productId>.html. The trailing productId (an integer) is the canonical SKU identifier. Behind every product page sits a JSON endpoint at https://tiki.vn/api/v2/products/<productId>. The endpoint accepts query parameters for store_id and platform and returns price, stock, full description, images, and seller information.

import httpx

API = "https://tiki.vn/api/v2/products"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15",
    "Accept": "application/json",
    "Accept-Language": "vi-VN,vi;q=0.9,en;q=0.8",
}

async def fetch_tiki(product_id: int, proxy: str):
    url = f"{API}/{product_id}"
    params = {"platform": "web", "spid": ""}
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url, params=params)
        if r.status_code == 200:
            return r.json()
        return None

The response includes id, name, price, original_price, discount_rate, stock_item.qty, inventory_status, seller.id, seller.name, brand.name, categories, and rating_average. For most analytical use cases the API alone is sufficient.

Vietnamese proxy strategy

Tiki bot detection profiles visitor IP at the country level. For light scraping under 5,000 product reads per day, clean Asian datacenter IPs from Singapore or Tokyo work. For higher volumes, Vietnamese residential IPs through Viettel, VNPT, or FPT Telecom dramatically improve success rates and avoid Cloudflare interstitials.

For full catalogue sweeps, dedicated Vietnamese mobile inventory pays for itself. The cost differential against pan-Asian residential is meaningful but the success rate at scale is significantly higher.

Crawling the category tree

Tiki exposes a category tree at https://api.tiki.vn/raiden/v2/menu-config. Each category has a url_key and an id that you can use to query the listing API at https://tiki.vn/api/personalish/v1/blocks/listings. Pagination uses page and limit query parameters, with practical limits of 50 pages of 50 products each.

async def crawl_category(category_id: int, proxy_pool, max_pages: int = 50):
    results = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = "https://tiki.vn/api/personalish/v1/blocks/listings"
        params = {
            "limit": 50,
            "page": page,
            "category": category_id,
            "aggregations": 2,
        }
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url, params=params)
            if r.status_code != 200:
                break
            items = r.json().get("data", [])
            if not items:
                break
            results.extend(items)
    return results

For deeper coverage of large categories, decompose by brand or price band facets. The aggregations response includes the available facets and counts per facet.

Handling Vietnamese diacritics

Vietnamese product names use Latin characters with extensive diacritics. Always store the original UTF-8 text without normalizing diacritics, because diacritic stripping changes the meaning of many words and breaks brand matches. For full-text search, use Postgres with the unaccent extension or Elasticsearch with the Vietnamese analyzer, which both handle diacritic-aware matching correctly.

Cross-checking Tiki pricing against Shopee Vietnam

The most analytically interesting Vietnamese ecommerce signal is the price differential between Tiki and Shopee for the same SKU. They overlap heavily on electronics, beauty, and household goods. For brand monitoring, the price gap on a given SKU often signals which platform is running a flash promotion at any given time.

Field	Tiki	Shopee VN	Notes
Canonical ID	productId (int)	itemid (int)	Different schemes
EAN coverage	~40%	~30%	Voluntary by seller
Update cadence	Hourly	Sub-hourly	Both update frequently
Promotion model	Flash sale + voucher	Flash sale + coin + voucher	Shopee promotion stack is more complex

For SKU matching across platforms, group by EAN where available and by normalized title plus brand for the long tail.

Detecting and routing around CAPTCHA challenges on Tiki

When Tiki flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Vietnam residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle.

Working with VND pricing and FX normalization

Pricing on Tiki is denominated in VND, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local VND and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread.

Comparing Tiki to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Tiki	Vietnam	Large	High
Shopee Vietnam	Adjacent markets	Medium	Medium
Lazada Vietnam	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Tiki but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal.

Operational monitoring and alerting

Every production scraper needs three monitoring layers. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for Vietnam

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Vietnam has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@flow(name="tiki-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    return [f.result() for f in futures]

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is.

Sample analytics queries

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days';

These queries power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product.

Building robust deduplication across noisy listings

The long-tail catalogue is full of near-duplicate listings. The standard deduplication approach uses a three-pass funnel: exact match on EAN, normalized title plus brand TF-IDF similarity, then perceptual image hash similarity.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge or under-merge.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp using freshness deadline, volatility, and business priority signals to decide what to refetch on each cycle. Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads.

The collection layer handles the network conversation: HTTP requests, proxy assignment, retry logic, and rate limit enforcement. It should know nothing about the data shape and nothing about how the data will eventually be queried. Its only job is to fetch raw bytes reliably and hand them off to the next layer with metadata about which IP, which user agent, and which timestamp produced them.

The parsing layer transforms raw bytes into structured records. It owns the schema, the field normalization, and the validation rules. When the upstream HTML or JSON structure changes, only the parsing layer needs to adapt. Keep parsers idempotent and version them aggressively so old raw bytes can be re-parsed when you discover bugs.

The storage layer holds the canonical snapshots in a query-optimized format. For most ecommerce datasets, a column-oriented store like DuckDB, ClickHouse, or BigQuery outperforms row-oriented Postgres at analytical scale. The trade-off is write latency and update support; column stores prefer append-only and bulk loads, which fits the snapshot model naturally.

The serving layer exposes the data to consumers, whether that is a BI dashboard, an API for downstream systems, or an alerting pipeline. Keep the serving layer denormalized and pre-aggregated where possible. Recomputing complex analytics on every dashboard load wastes resources and hurts responsiveness.

# Pseudo-code for the four-layer split
async def collect(url: str, proxy_pool) -> RawFetch:
    proxy = proxy_pool.next()
    response = await http_get(url, proxy)
    return RawFetch(url=url, body=response.text, fetched_at=now(), ip=proxy.ip)

def parse(raw: RawFetch) -> Snapshot:
    data = json.loads(raw.body)
    return Snapshot(sku=data["id"], price=data["price"], ...)

def store(snapshot: Snapshot, db) -> None:
    db.append("snapshots", snapshot)

def serve(query: str, db) -> list:
    return db.query(query)

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. The checks fall into three categories: structural, distributional, and semantic.

Structural checks verify that every required field is present and of the expected type. A snapshot row missing the price field is not a real snapshot. A row with a negative price is not a real price.

Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer SKUs than yesterday, something broke either in collection or in the upstream catalogue. Either way, the on-call engineer needs to investigate before downstream consumers see broken data.

Semantic checks compare related fields for consistency. If a SKU shows in_stock = true but stock_quantity = 0, one of the fields is wrong. If the discount percentage is computed from list_price and selling_price, the computed value should match the stated discount field.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    invalid = [r for r in snapshot if r.get("price", -1) < 0]
    if invalid:
        errors.append(f"{len(invalid)} rows have invalid price")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same SKU within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

Common pitfalls when scraping Tiki

Three issues recur on Tiki scrapers. The first is TikiNOW vs Marketplace separation. TikiNOW (2-hour delivery in HCMC and Hanoi) and Tiki Marketplace (multi-day fulfillment elsewhere) coexist on the same product page. The is_tikinow flag and seller_id distinguish them. For delivery-speed analytics, segment by both fields.

The second is Vietnamese diacritic normalization. Product titles include Vietnamese diacritics (e.g., điện thoại). Some pipelines strip diacritics for ASCII compatibility, which collapses distinct Vietnamese words into the same key and corrupts brand-level aggregates. Preserve diacritics end-to-end and only strip them at the final analytics layer if your downstream BI tool cannot handle UTF-8.

The third is VAT inclusion drift. Tiki shows VAT-inclusive prices on most SKUs but VAT-exclusive prices for some B2B-flagged SKUs. The vat_included field on the API response tells you which is which. Naive scrapers store the raw price and downstream analytics compare apples to oranges.

FAQ

Is the Tiki API officially documented?
The endpoints described here are the unauthenticated APIs that the public web site uses. They have been stable for several years but they are not contractually supported. Build defensively with schema drift alerts.

Can I scrape Tiki from Singapore or Hong Kong residential IPs?
Yes for light loads under 5,000 product reads per day. For sustained scraping at higher volumes, Vietnamese IPs are strongly preferred because they sustain higher request rates without challenges.

Does Tiki distinguish between Tiki Trading and third-party sellers?
Yes. The seller.id field identifies the seller; sellers with id 1 are Tiki’s own first-party Trading. Third-party sellers have their own IDs. For analytics, separating first-party from third-party listings is essential because the price dynamics and turnover characteristics differ significantly.

How does TikiNow city-specific delivery affect the data?
TikiNow availability depends on the destination postcode. The API returns a generic stock value but the actual same-day delivery promise is computed against the buyer location. For most analytical use cases, treat the stock value as the canonical inventory signal.

What about Vietnamese tax and VAT in pricing?
Tiki prices are displayed VAT-inclusive at 10%. The displayed price is what the buyer pays. For brand teams comparing against MAP policies set in net-of-tax terms, you need to back out the VAT to align with the brand reference price.

Does Tiki publish an official API?
Tiki Open Platform serves sellers, not analysts. For market-intelligence work, public-page scraping is the operational path. The endpoints under tiki.vn/api/v2/ are stable in practice.

How do I track Tiki’s flash sales accurately?
The flash_sale block carries start_time and end_time epochs. Sample at 5-15 minute intervals during the active window to capture stock burndown.

To build a broader Vietnam ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives.

How to scrape PChome Taiwan product data

Scrape PChome Taiwan and you are scraping the dominant general-merchandise marketplace in Taiwan, famous for its 24-hour delivery service across the island and operated by PChome Online Inc. The platform is the largest player in Taiwanese B2C ecommerce by GMV, ahead of Momoshop and Yahoo Shopping Taiwan, and it has a relatively scrape-friendly architecture for a marketplace of its size. The scraping landscape is shaped by three things: a publicly available JSON API that powers the front end, a category structure that maps cleanly to URL paths, and a Cloudflare front end that profiles non-Taiwanese traffic less aggressively than most Asian marketplaces.

This guide covers PChome 24h (the express-delivery property at 24h.pchome.com.tw) and notes where the patterns extend to PChome Shopping Mall (shopping.pchome.com.tw).

Mapping PChome URL and JSON structure

PChome 24h product URLs follow the pattern https://24h.pchome.com.tw/prod/<productCode>. The productCode is a 12-character alphanumeric identifier that is the canonical SKU. Behind every product page sits a JSON endpoint at https://ecapi.pchome.com.tw/ecshop/prodapi/v2/prod/<productCode>. The endpoint returns price, stock, full description, images, and category path in a single response.

import httpx

API = "https://ecapi.pchome.com.tw/ecshop/prodapi/v2/prod"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "zh-TW,zh;q=0.9",
}

async def fetch_pchome(product_code: str, proxy: str):
    url = f"{API}/{product_code}"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return None

The response includes Id (productCode), Name, Brand, Price.P (current price in TWD), Price.M (member price), Stock, Slogan, Description, Pic (image filenames), and Cate (category path as a list). For most analytical use cases the API is sufficient and you do not need to render the HTML.

Taiwanese proxy strategy

PChome’s bot detection is moderate compared to other regional marketplaces. The site profiles visitor IP at the country level but does not aggressively block non-Taiwan traffic for the API endpoints. For light scraping (under 5,000 product reads per day), a clean datacenter IP from a Tokyo or Singapore region works. For higher volumes, Taiwanese residential or mobile IPs through Chunghwa Telecom or Far EasTone produce dramatically better success rates and avoid the occasional Cloudflare interstitial.

For full catalogue sweeps that touch hundreds of thousands of SKUs per day, dedicated Taiwan inventory pays for itself. The cost differential against generic Asian residential is meaningful but the success rate differential at scale is larger.

Crawling the category tree

PChome exposes the category tree at https://ecapi.pchome.com.tw/cdn/ecshop/prodapi/v2/cateinfo/<rootCateId>/category. Categories are organized as a six-level deep hierarchy. The API returns child nodes with their own categoryId values that you can recursively walk to enumerate the full tree.

async def fetch_category(cate_id: str, proxy: str):
    url = f"https://ecapi.pchome.com.tw/cdn/ecshop/prodapi/v2/cateinfo/{cate_id}/category"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return []

For each leaf category, the listing endpoint at https://ecapi.pchome.com.tw/ecshop/prodapi/v2/cateprod/<cateId>/prod returns the products in that category. Pagination uses a start and rows parameter, with practical limits of 1,000 products per category. For broader categories, decompose by brand or price band facet.

Working with Traditional Chinese text

PChome catalogue text is in Traditional Chinese. If your downstream pipeline expects Simplified Chinese, plan a conversion step using the OpenCC library. Conversion is generally lossless from Traditional to Simplified for product catalogue text, though edge cases (region-specific brand names, technical jargon) sometimes need a dictionary override.

from opencc import OpenCC

cc = OpenCC("t2s")  # traditional to simplified
simplified_title = cc.convert(traditional_title)

For full-text search across the dataset, both Postgres and Elasticsearch handle Chinese text well with the right analyzer. Use the chinese text search configuration in Postgres or the IK analyzer in Elasticsearch. Avoid space-tokenized full-text search because Chinese has no word boundaries and naive tokenization fails badly.

Cross-checking PChome pricing against Momoshop

The most analytically interesting Taiwan ecommerce signal is the price differential between PChome and Momoshop, the two dominant marketplaces, for the same SKU. They overlap heavily on electronics, home appliances, and beauty, and the price gap on a given SKU often signals which platform is running a promotion at any given time.

To match SKUs across platforms, group by EAN or by normalized title plus brand. The match rate is roughly 60% on EAN and an additional 20% on title-plus-brand, leaving a long tail of platform-exclusive SKUs.

Field	PChome 24h	Momoshop	Notes
Canonical ID	productCode (12 char)	i_code (numeric)	Different schemes
EAN coverage	~50%	~45%	Voluntary by seller
Update cadence	Hourly	Hourly	Both update prices throughout the day
Promotion model	Site-wide and member	Site-wide and category	Different promotion engines

For brand monitoring use cases, build a master SKU table that joins PChome productCode and Momoshop i_code via EAN where available. For SKUs without EAN matches, run a daily fuzzy-matching job and surface the matches for human review.

Detecting and routing around CAPTCHA challenges on PChome

When PChome flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Taiwan residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Working with TWD pricing and FX normalization

Pricing on PChome is denominated in TWD, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local TWD and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing PChome to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
PChome 24h	Taiwan	Large	High
Momoshop	Adjacent markets	Medium	Medium
Yahoo Shopping TW	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on PChome 24h but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against PChome, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for Taiwan

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Taiwan has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="pchome-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across PChome datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks.

The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups).

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. Stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks. The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups). Both failure modes break downstream analytics in subtle ways that take weeks to detect.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle. The first signal is freshness deadline. Every SKU has a maximum staleness budget, and any SKU older than its budget gets refreshed. The second signal is volatility. SKUs that have changed price recently get higher refresh priority. The third signal is business priority. SKUs that downstream users actually query get higher refresh priority than dormant SKUs that nobody has looked at in months.

This kind of priority-driven scheduler reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

Common pitfalls when scraping PChome 24h

Three issues dominate PChome scraping. The first is the prod ID vs SKU ID confusion. PChome uses a prod_id for the product page and a separate sku_id for variants like color and capacity. The same prod_id can have 3-10 sku_ids with different prices. Naive scrapers join on prod_id and report the cheapest variant as the product price, which understates the average sale price by 8-20% for electronics.

The second is 24-hour delivery vs marketplace mixing. PChome 24h’s flagship promise is Taiwan-wide same-day or next-day delivery, but the marketplace (PChome Mall) sits on the same domain with longer fulfillment. The is_24h flag separates them. Aggregating both into the same dataset distorts delivery-speed analytics.

The third is Traditional vs Simplified Chinese normalization. PChome stores product titles in Traditional Chinese. Cross-referencing with Mainland data sources requires a TC->SC conversion via OpenCC or similar. Encoding-only conversion (UTF-8 reading) does not produce a Mainland-readable string.

FAQ

Is the PChome API officially documented?
The ecapi.pchome.com.tw endpoints are the same endpoints used by the PChome web front end. They have been stable for several years but are not contractually supported as a public API. Build defensively with schema drift alerts.

Does PChome offer an affiliate API I should use instead?
PChome operates an affiliate program but the affiliate feed lacks real-time pricing and does not include the full catalogue. For competitive intelligence purposes, scraping the public API remains the higher-fidelity path.

Can I scrape PChome from China mainland IPs?
PChome does not specifically block China mainland IPs but the success rate is meaningfully lower because the cross-strait routing introduces high latency and connection instability. Taiwan or Hong Kong IPs are strongly preferred.

How does PChome handle the difference between PChome 24h and PChome Shopping Mall?
24h is the express-delivery property with a smaller curated catalogue and PChome’s own logistics. Shopping Mall is the larger marketplace with third-party sellers and longer delivery times. The APIs share patterns but use different endpoint paths and product code schemes. Plan for separate code paths if your project covers both.

Are Taiwan-specific holidays a meaningful factor for snapshot scheduling?
Taiwan’s major shopping events (Double 11, Double 12, Lunar New Year, 618) drive significant pricing and promotion activity. During these windows, increase your snapshot frequency to capture the rapid price changes. The default 24-hour cadence misses important intraday movements during peak promotional periods.

Does PChome block non-Taiwan IPs aggressively?
Casual lookups succeed from most regions. Sustained scraping above 100 requests per hour from non-TW IPs triggers rate limits. Taiwan residential IPs are the safe path for production.

How do I track PChome Double 11 promotions?
Capture daily baselines from October 1 onwards. PChome staggers promotions across two weeks leading into 11/11 with daily reveals, so a single pre-event baseline misses the early discount waves.

To build a broader Taiwan ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape OLX Brazil and LATAM marketplaces

Scrape OLX Brazil and you tap into the largest classifieds platform in Latin America, covering used cars, real estate, jobs, and general merchandise across Brazil, Argentina, Colombia, Peru, and Ecuador. OLX operates as a horizontal classifieds platform where individual sellers and small businesses publish ads, which makes the data shape fundamentally different from a marketplace like Mercado Libre. The scraping landscape is shaped by three things: a state-level geographic hierarchy that gates listings, an undocumented but stable JSON API behind the search experience, and Cloudflare bot protection that classifies non-Brazilian traffic aggressively.

This guide focuses on OLX Brazil at olx.com.br as the canonical example, with notes on how the patterns transfer to OLX Argentina, OLX Colombia, and the smaller LATAM properties.

Mapping OLX URL and listing structure

OLX Brazil URLs follow a regional hierarchy. National listings live under the root domain, but most search activity is filtered by state. URLs include the state code in the path: https://www.olx.com.br/sp/regiao-de-sao-paulo for Sao Paulo state, https://www.olx.com.br/rj for Rio de Janeiro, and so on for the 27 federative units. Within a state, listings are further filtered by category, sub-category, and city.

Individual ad URLs follow the pattern https://<region>.olx.com.br/<category>/<subcategory>/<title-slug>-<adId>. The trailing adId is the canonical identifier you should store as the primary key. Slugs change when sellers edit ads; adIds are stable for the life of the listing.

The OLX search experience is powered by a JSON API at https://www.olx.com.br/api/relevance/search. The endpoint accepts a category, subcategory, region, and pagination parameters and returns a structured listing payload. This is dramatically more reliable than HTML scraping because the API contract is stable while the visible markup changes regularly.

import httpx

SEARCH_API = "https://www.olx.com.br/api/relevance/search"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "pt-BR,pt;q=0.9",
}

async def search_olx(category_id: int, region_code: str, page: int, proxy: str):
    params = {
        "category": category_id,
        "region": region_code,
        "o": page,
        "lim": 50,
    }
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(SEARCH_API, params=params)
        if r.status_code == 200:
            return r.json().get("ad_list", [])
        return []

The response includes list_id (the canonical ad id), subject (title), body (description), price in BRL, category_id, category_name, state, city, seller_name, seller_phone_status, images, and created_at. For most analytical use cases this is sufficient and you do not need to fetch the individual ad detail page.

Brazilian proxy strategy

OLX Brazil’s bot detection is aggressive against non-Brazilian IPs. Cloudflare profiles the visitor IP geography, and any IP outside Brazil receives elevated scrutiny. For sustained scraping, Brazilian residential or mobile IPs are required. Vivo, Claro, and TIM mobile pools work well. Brazilian residential pools through major providers also work but cost more per gigabyte than the mobile equivalent at scale.

For workloads under 10,000 ad reads per day, a small Brazilian residential pool with sticky 15-minute sessions is sufficient. For higher volumes, dedicated mobile ports are the cleaner path because they can sustain higher request rates without challenges.

Region	Recommended proxy origin	Tolerance per IP
Sao Paulo state	Brazilian mobile or residential in SP	200 req/hr per IP
Rio de Janeiro	Brazilian residential anywhere	200 req/hr per IP
Northeast states	Brazilian residential	250 req/hr per IP
South states	Brazilian residential	250 req/hr per IP

Cross-state IPs work fine for any state target because the OLX bot logic does not enforce intra-Brazil geographic consistency. The geographic check is at the country level only.

Crawling categories and pagination

OLX exposes roughly 80 top-level categories and 400 sub-categories across the Brazilian site. The full category tree is published at https://www.olx.com.br/api/categories and changes slowly enough that you can cache it weekly. For each category, the search API allows pagination up to roughly 100 pages of 50 ads each, for a maximum of 5,000 ads per query.

For deeper coverage of large categories like used cars or real estate, decompose by state and then by city. The cities of Sao Paulo, Rio de Janeiro, Belo Horizonte, Salvador, Brasilia, Curitiba, and Recife collectively cover roughly 40% of total OLX listings; the remaining listings spread across hundreds of smaller cities.

async def crawl_state_category(state: str, category_id: int, proxy_pool, max_pages: int = 100):
    all_ads = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        ads = await search_olx(category_id, state, page, proxy)
        if not ads:
            break
        all_ads.extend(ads)
    return all_ads

A national daily snapshot across all states and categories is roughly 10-15 million ad reads. That is achievable on a moderately-sized mobile proxy pool with parallelism across 20-50 concurrent sessions. Plan for 8-12 hours of wall-clock time for the full sweep.

Phone number redaction and contact extraction

OLX masks seller phone numbers behind a click-to-reveal flow. The seller_phone_status field in the search response indicates whether a phone is published, but the actual number is only revealed by hitting the contact API with an authenticated session. From a privacy perspective, treat the phone number as personal data and avoid collecting it unless you have a clear lawful basis and a documented use case.

For analytics that need to dedupe sellers across many ads, hash a stable seller identifier (the OLX user_id is published in the ad detail) rather than collecting raw phone numbers. The hashed user_id is sufficient for seller concentration analysis without creating personal-data exposure.

Schema for OLX classifieds snapshots

CREATE TABLE olx_ad_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    list_id BIGINT NOT NULL,
    state VARCHAR(2) NOT NULL,
    city VARCHAR(64),
    category_id INT,
    subject TEXT,
    price_brl DECIMAL(12,2),
    seller_user_id_hash VARCHAR(64),
    image_count INT,
    created_at TIMESTAMP,
    is_active BOOLEAN,
    PRIMARY KEY (snapshot_at, list_id)
);
CREATE INDEX olx_state_cat_idx ON olx_ad_snapshot(state, category_id);

For longitudinal classifieds analytics, the most valuable derived metric is time-on-market: the number of days an ad stays active before being delisted (because the item sold or the seller withdrew). Compute this from the diff between consecutive snapshots. Time-on-market by category, by price band, and by state is the headline insight in any classifieds intelligence product.

For broader pattern guidance on classifieds-style scrapers, see our residential proxy provider ranking and our Mercado Libre Mexico scraping guide.

Detecting and routing around CAPTCHA challenges on OLX

When OLX flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Brazil residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Working with BRL pricing and FX normalization

Pricing on OLX is denominated in BRL, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local BRL and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing OLX to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
OLX	Brazil	Large	High
Mercado Libre	Adjacent markets	Medium	Medium
Webmotors	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on OLX but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against OLX, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for Brazil

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Brazil has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="olx-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across OLX datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks.

The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups).

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. Stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks. The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups). Both failure modes break downstream analytics in subtle ways that take weeks to detect.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle. The first signal is freshness deadline. Every SKU has a maximum staleness budget, and any SKU older than its budget gets refreshed. The second signal is volatility. SKUs that have changed price recently get higher refresh priority. The third signal is business priority. SKUs that downstream users actually query get higher refresh priority than dormant SKUs that nobody has looked at in months.

This kind of priority-driven scheduler reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

FAQ

Can I scrape all LATAM OLX countries with the same code?
The API contract is mostly consistent across OLX Argentina, OLX Colombia, and OLX Peru, but the parameter names and the category trees differ. Build a per-country adapter rather than assuming one code path covers everything. Brazilian-specific date parsing (DD/MM/YYYY) and currency formatting (R$ prefix) need country-specific handlers.

Is the OLX API officially documented?
No. The endpoints described here are the unauthenticated APIs that the public web site uses. They have been stable for several years but they are not contractually supported. Build defensively: log unknown response fields, alert on schema drift, and keep the parser version-stamped.

How does OLX handle ads from professional sellers vs. private sellers?
The ad object includes a professional_ad boolean flag that distinguishes dealer or business listings from individual private sellers. For analytics, separating professional from private listings is essential because the price dynamics and turnover characteristics are very different.

Can I track an ad across multiple snapshots to see if the price changed?
Yes. The list_id is stable for the life of the ad. Comparing the price field across consecutive snapshots gives you the price-change history. About 15-20% of ads see at least one price change before the ad is delisted.

What about OLX motors vs. general merchandise?
OLX Brazil operates a sub-property at autos.olx.com.br for vehicles with additional automotive-specific filters and an inventory model that overlaps with Webmotors. The same API patterns work but with vehicle-specific category IDs and additional fields like make, model, year, and mileage.

To build a broader LATAM classifieds intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape Auchan Drive France grocery data

Scrape Auchan Drive France and you tap into one of the largest hypermarket grocery catalogues in Europe, served through a click-and-collect model that varies prices by store. Auchan operates a federation of physical hypermarkets and Drive pickup points across France, with each Drive store maintaining its own catalogue and its own price list within the corporate pricing framework. The scraping landscape is shaped by three things: postcode-driven store selection that gates every product page, a JSON catalogue API that returns store-specific availability, and a Cloudflare front end that profiles non-French traffic aggressively.

This guide focuses on Auchan Drive specifically, which is the click-and-collect property at www.auchan.fr/drive. Auchan also operates a separate home delivery property and a hypermarket banner catalogue with different APIs.

The store-selection prerequisite

Every Auchan product page requires a store context before it returns prices. The store context is set by selecting a postal code or a specific Drive location, which sets a mag cookie that all subsequent requests must include. Without the store cookie, you get either redirects to the homepage or a stripped-down view with no pricing.

import httpx

async def select_store(postcode: str, proxy: str):
    async with httpx.AsyncClient(proxy=proxy, follow_redirects=True, timeout=20) as c:
        # Hit the store selector endpoint to get a session
        r = await c.get(f"https://www.auchan.fr/store-selector?postalCode={postcode}")
        # The mag cookie should now be set
        return c.cookies

Different postcodes resolve to different Drive stores. If you want catalogue coverage across all of France, you need to enumerate the Drive store list and rotate through stores in your scraping pipeline. The store list is published as a static JSON at https://www.auchan.fr/api/stores that returns roughly 200 active Drive locations.

The Auchan product API

Once a store is selected, product details come from the catalogue API at https://www.auchan.fr/api/v1/products/<gtin>. The endpoint returns the canonical product object along with the store-specific price, promotion, and stock state.

async def fetch_product(gtin: str, store_cookies, proxy: str):
    url = f"https://www.auchan.fr/api/v1/products/{gtin}"
    async with httpx.AsyncClient(proxy=proxy, cookies=store_cookies, timeout=20) as c:
        r = await c.get(url, headers={"Accept": "application/json"})
        if r.status_code == 200:
            return r.json()
        return None

The response includes gtin, name, brand, price, pricePerUnit, unit, inStock, promotion, nutritionalInfo, and categoryPath. The pricePerUnit field is the most analytically useful for grocery work because grocery products are sold in highly variable pack sizes and the per-unit price is the only fair comparison across formats.

Crawling the category tree

Auchan’s category tree is exposed at https://www.auchan.fr/api/v1/categories. It returns a nested JSON of department, sub-department, and category nodes. Each terminal category has a slug that is used in listing URLs.

Listing endpoints look like https://www.auchan.fr/api/v1/products?categoryId=<id>&page=<n>&size=50. The pagination cap is 1,000 products per category, which is enough for most categories but requires faceted decomposition for the largest ones (epicerie sucree, boissons).

async def crawl_category(category_id: int, store_cookies, proxy_pool, max_pages: int = 20):
    all_items = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = "https://www.auchan.fr/api/v1/products"
        params = {"categoryId": category_id, "page": page, "size": 50}
        async with httpx.AsyncClient(proxy=proxy, cookies=store_cookies, timeout=20) as c:
            r = await c.get(url, params=params, headers={"Accept": "application/json"})
            if r.status_code != 200:
                break
            data = r.json()
            items = data.get("products", [])
            if not items:
                break
            all_items.extend(items)
    return all_items

For a daily catalogue snapshot across all stores, the work scales linearly with the number of stores you cover. A national snapshot covering 200 stores at 50,000 SKUs each is 10M product reads per day. That is achievable on a small mobile proxy pool with reasonable parallelism, but it requires careful work scheduling.

Multi-store price comparison

The most analytically interesting Auchan dataset is the price variation across stores for the same SKU. Even within Auchan, prices for identical products differ by store based on local competitive pressure, promotional schedules, and regional sourcing. For brand owners, this surfaces where Auchan is using a SKU as a loss leader versus where they are pricing at full margin.

CREATE TABLE auchan_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    store_id VARCHAR(32) NOT NULL,
    gtin VARCHAR(14) NOT NULL,
    price_eur DECIMAL(10,2),
    price_per_unit_eur DECIMAL(12,4),
    unit VARCHAR(8),
    in_stock BOOLEAN,
    promotion_text TEXT,
    PRIMARY KEY (snapshot_at, store_id, gtin)
);

A price-variance view across stores reveals interesting patterns. Stores in dense urban Paris consistently price higher than stores in suburban regions. Stores near a major Lidl or Aldi competitor consistently price 5-10% lower on the categories where the competitor is strongest.

Proxy strategy for Auchan

Auchan’s bot detection sits on Cloudflare with additional behavioral analysis. French residential or mobile IPs are required for stable scraping. EU residential pools sometimes work but the success rate is meaningfully lower than France-specific inventory.

For workloads under 5,000 product reads per day, a small French residential pool with sticky sessions of 10-20 minutes is sufficient. For higher volumes, a dedicated mobile port on Orange or SFR is the cleaner path. The mobile IP costs more but its trust score is high enough to sustain 10+ requests per second for hours without challenges.

Rate limits and request shaping

Auchan does not publish rate limits but the observed behavior is consistent: roughly 2 requests per second per IP is the safe sustained rate. Bursts up to 10 requests per second work for short periods. Beyond that you hit Cloudflare challenges that take 10-30 minutes to clear per IP.

The request shaping that performs best on Auchan is store-by-store sequential scraping rather than SKU-by-SKU parallel scraping. Each store has its own session cookie, and rotating store cookies on every request adds overhead without improving throughput. Stick with one store cookie per IP for the duration of that store’s catalogue sweep, then rotate.

Detecting and routing around CAPTCHA challenges on Auchan

When Auchan flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real France residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Working with EUR pricing and FX normalization

Pricing in France is denominated in EUR, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local EUR and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons.

Comparing Auchan to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Auchan Drive	France	Large	High
Carrefour Drive	Adjacent markets	Medium	Medium
Leclerc Drive	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Auchan Drive but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for France

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but France has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="auchan-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Auchan datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks.

The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing. Each pass merges groups produced by the previous pass.

import hashlib
from PIL import Image
import imagehash

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

def normalize_title(title: str) -> str:
    title = title.lower()
    for token in ["[free shipping]", "[same day]", "(new)", "*sale*"]:
        title = title.replace(token, "")
    return " ".join(title.split())

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups). Both failure modes break downstream analytics.

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. New attribute fields appear, old fields are deprecated, category trees are reorganized, and pricing display logic changes. Your scraper code has to evolve with these changes, and a versioning pattern that keeps old data interpretable is critical.

The pattern that works best is to stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

ALTER TABLE snapshot ADD COLUMN scraper_version VARCHAR(16);
CREATE INDEX scraper_version_idx ON snapshot(scraper_version);

Pair this with a small registry table that documents what each scraper version did differently. When a downstream user asks why a particular metric jumped on a specific date, the version registry usually has the answer.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle.

The first signal is freshness deadline. Every SKU has a maximum staleness budget (say 24 hours), and any SKU older than its budget gets refreshed.

The second signal is volatility. SKUs that have changed price recently get higher refresh priority because they are more likely to change again. SKUs that have been stable for weeks can drop to a longer refresh interval.

The third signal is business priority. SKUs that downstream users actually query (tracked by query logs) get higher refresh priority than dormant SKUs that nobody has looked at in months.

def schedule_refresh(sku_id: int, last_changed_at, last_queried_at, last_fetched_at) -> int:
    """returns priority score; higher = refresh sooner"""
    age = (now - last_fetched_at).total_seconds() / 3600
    volatility = 10 if (now - last_changed_at).days < 7 else 1
    relevance = 5 if (now - last_queried_at).days < 1 else 1
    return age * volatility * relevance

This kind of priority-driven scheduler reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

Common pitfalls when scraping Auchan Drive

Three issues recur. The first is store-level price variance. Auchan Drive prices are set per fulfillment store, not nationally. The same SKU can vary by 5-15% between Lille and Marseille. A scraper that does not pin a store_id averages across stores and loses the geographic price signal that makes Drive data analytically interesting. Always store the drive_id alongside every price.

The second is unit-of-sale ambiguity. Fresh produce is sold by weight (per kilo) but packaged in approximate units. The product page shows both prix au kilo and prix par piece. The realized cost depends on the actual weight at checkout. For trend analysis, pin to prix au kilo and ignore the per-piece estimate.

The third is loyalty-program price drift. Auchan Waaoh card holders see different prices for some promotional SKUs. Anonymous scraping captures the public price. Cardholder scraping requires session cookies that expire every few hours. Decide which population your analysis serves and stay consistent.

FAQ

Why does the same product show different prices on different Auchan Drive stores?
Auchan Drive stores set their own pricing within a corporate framework. Local competition (proximity to a Leclerc, Lidl, or Carrefour) and regional sourcing costs both influence the per-store price. Multi-store snapshots are the only way to see the full pricing landscape.

Does Auchan offer an affiliate or partner API?
Auchan participates in affiliate networks that expose a subset of the catalogue with commission tracking, but the affiliate feeds are not designed for competitive intelligence. They lag the live catalogue by 24-48 hours and exclude promotions. For real-time data, scraping the public catalogue is the only path.

Can I scrape Auchan from a Belgian or Spanish residential IP?
Auchan operates in multiple European countries with separate sites. Auchan Belgium uses a different domain (auchan.be) and a different catalogue. For French data specifically, French IPs are strongly preferred. Belgian or Spanish IPs work for short bursts but degrade quickly.

How fresh are the prices in the API response?
Auchan refreshes prices in batches. The catalogue API reflects the current published price with a CDN cache lifetime of 60-120 seconds. For most monitoring use cases that is real-time enough. For high-frequency competitor tracking, hitting the API every 5-10 minutes catches all meaningful changes.

Are there legal considerations specific to scraping French ecommerce sites?
France enforces GDPR strictly and has additional consumer protection rules through the DGCCRF. Limit your collection to non-personal product, price, and availability data. Avoid scraping any personal data, any logged-in pages, and any data that includes individual customer reviews with identifiable information.

How often do Drive prices change?
Most SKUs reprice weekly on Sunday night for the new promotional period. Fresh produce and fish reprice daily. Snapshots every Monday morning capture the canonical weekly state.

Does Auchan Drive expose stock counts?
Only as a binary in_stock flag and an occasional ‘low stock’ badge. True quantity is hidden. For demand-signal analytics, treat the in_stock flag as the only reliable inventory feature.

To build a broader European grocery intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape Rakuten Japan in 2026

Scrape Rakuten Japan and you encounter a marketplace shape that does not exist anywhere else in the world. Rakuten Ichiba is a federation of more than 50,000 individual seller storefronts hosted under the Rakuten umbrella, each with its own brand, layout, and pricing logic. Unlike Amazon Japan where every listing is normalized into a single product page, Rakuten lets every merchant publish their own product page for the same item, which means the same EAN can have hundreds of listings across hundreds of sellers. The scraping landscape is shaped by three things: the official Rakuten Ichiba Item API which is generous but rate-limited, the per-merchant storefronts which require HTML scraping for the long tail, and a Japanese-language catalogue that requires careful encoding handling.

This guide focuses on Rakuten Ichiba (the marketplace), which is the dominant property under rakuten.co.jp. Rakuten Travel, Rakuten Books, and Rakuten Mobile use related but distinct APIs.

The official Rakuten Ichiba Item API

Rakuten publishes a free public API at https://app.rakuten.co.jp/services/api/IchibaItem/Search/20220601 that searches the entire Ichiba catalogue. Registration is required to obtain an applicationId, which serves as the rate-limit token. The free tier allows roughly 1 request per second per applicationId, which sounds restrictive but is enough for substantial workloads if you parallelize across multiple registered application IDs.

import httpx
import asyncio

API_URL = "https://app.rakuten.co.jp/services/api/IchibaItem/Search/20220601"

async def search_ichiba(keyword: str, app_id: str, page: int = 1, hits: int = 30):
    params = {
        "applicationId": app_id,
        "keyword": keyword,
        "page": page,
        "hits": hits,
        "sort": "-updateTimestamp",
        "format": "json",
    }
    async with httpx.AsyncClient(timeout=20) as c:
        r = await c.get(API_URL, params=params)
        if r.status_code == 200:
            return r.json().get("Items", [])
        return []

The API returns up to 100 hits per request, paginated to 100 pages, for a maximum of 10,000 hits per query. To exceed that, decompose the query by genre, price band, or shop. The genreId parameter accepts the Rakuten genre tree which is published separately at the IchibaGenre Search API.

Each item response includes itemCode (the canonical Rakuten SKU as <shopcode>:<itemnumber>), itemName, itemPrice in JPY, shopCode, shopName, reviewCount, reviewAverage, imageFlag, and a list of small/medium/large image URLs. For most analytical use cases, the API alone is sufficient and you do not need to scrape HTML.

When you still need HTML scraping

The official API does not expose seller storefront banners, custom seller categorizations, or the freeform descriptions that Rakuten merchants write on their own pages. For brand monitoring use cases that need to verify how a specific seller is presenting a product, you have to scrape the merchant storefront HTML.

Storefront URLs look like https://item.rakuten.co.jp/<shopcode>/<itemnumber>/. The HTML is encoded as Shift-JIS or UTF-8 depending on the merchant template, so always read the Content-Type header and decode accordingly. Encoding bugs are the most common silent failure in Rakuten scraping projects.

import httpx
from bs4 import BeautifulSoup

async def fetch_storefront(shop_code: str, item_number: str, proxy: str):
    url = f"https://item.rakuten.co.jp/{shop_code}/{item_number}/"
    async with httpx.AsyncClient(proxy=proxy, timeout=20) as c:
        r = await c.get(url)
        if r.status_code != 200:
            return None
        ctype = r.headers.get("content-type", "").lower()
        if "shift_jis" in ctype or "shift-jis" in ctype:
            text = r.content.decode("shift_jis", errors="replace")
        else:
            text = r.text
        soup = BeautifulSoup(text, "lxml")
        return {
            "title": soup.select_one("title").get_text(strip=True) if soup.select_one("title") else None,
            "html_size": len(text),
        }

Storefront scraping requires Japanese residential proxies. Rakuten’s CDN serves significantly slower paths to non-Japanese visitors, and merchant storefronts often refuse non-JP traffic outright if they have enabled the country lock setting in their seller dashboard.

Genre-based catalogue discovery

Rakuten’s genre tree is the most reliable way to enumerate the catalogue. The IchibaGenre Search API returns the genre hierarchy starting from a root node. By recursively walking the tree, you can map every leaf genre and then issue Item Search queries against each leaf to collect SKUs.

GENRE_API = "https://app.rakuten.co.jp/services/api/IchibaGenre/Search/20140222"

async def fetch_children(genre_id: int, app_id: str):
    params = {"applicationId": app_id, "genreId": genre_id, "format": "json"}
    async with httpx.AsyncClient(timeout=20) as c:
        r = await c.get(GENRE_API, params=params)
        if r.status_code == 200:
            return r.json().get("children", [])
        return []

The genre tree has roughly 50,000 leaf nodes. Walking it once takes about 14 hours at 1 request per second. Cache the result in a database and refresh quarterly. The structure changes slowly enough that quarterly is sufficient for most use cases.

Multi-tenant pricing and seller analytics

Because the same EAN can be sold by multiple Rakuten merchants, the same physical product often appears at very different prices across the marketplace. For brand owners, this matters because it surfaces parallel imports, gray market listings, and unauthorized resellers. For competitive intelligence, it surfaces the price floor that the most aggressive seller is willing to defend.

The pattern is to group items by EAN (when published in the API response) or by a fuzzy match on itemName and merchant brand. Grouping by EAN is cleaner but only works for SKUs where the merchant published it. For the long tail, fuzzy match by normalized title and brand is the practical fallback.

Field	Source	Reliability
itemCode	API	Canonical SKU per merchant
EAN/JAN	API attributes	About 60% coverage
itemName	API	Always present, often noisy
itemPrice	API	Authoritative
shopCode	API	Canonical merchant ID

For grouping across merchants, build a normalization pass that strips merchant tags from the title (sellers add [Free shipping] and [Same day ship] markers liberally), normalizes character widths, and removes common Japanese boilerplate.

Detecting and routing around CAPTCHA challenges on Rakuten

When Rakuten flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Japan residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Working with JPY pricing and FX normalization

Pricing in Japan is denominated in JPY, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local JPY and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons.

Comparing Rakuten to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Rakuten Ichiba	Japan	Large	High
Amazon Japan	Adjacent markets	Medium	Medium
Yahoo Shopping	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Rakuten Ichiba but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for Japan

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Japan has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="rakuten-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Rakuten datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks.

The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing. Each pass merges groups produced by the previous pass.

import hashlib
from PIL import Image
import imagehash

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

def normalize_title(title: str) -> str:
    title = title.lower()
    for token in ["[free shipping]", "[same day]", "(new)", "*sale*"]:
        title = title.replace(token, "")
    return " ".join(title.split())

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups). Both failure modes break downstream analytics.

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. New attribute fields appear, old fields are deprecated, category trees are reorganized, and pricing display logic changes. Your scraper code has to evolve with these changes, and a versioning pattern that keeps old data interpretable is critical.

The pattern that works best is to stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

ALTER TABLE snapshot ADD COLUMN scraper_version VARCHAR(16);
CREATE INDEX scraper_version_idx ON snapshot(scraper_version);

Pair this with a small registry table that documents what each scraper version did differently. When a downstream user asks why a particular metric jumped on a specific date, the version registry usually has the answer.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle.

The first signal is freshness deadline. Every SKU has a maximum staleness budget (say 24 hours), and any SKU older than its budget gets refreshed.

The second signal is volatility. SKUs that have changed price recently get higher refresh priority because they are more likely to change again. SKUs that have been stable for weeks can drop to a longer refresh interval.

The third signal is business priority. SKUs that downstream users actually query (tracked by query logs) get higher refresh priority than dormant SKUs that nobody has looked at in months.

def schedule_refresh(sku_id: int, last_changed_at, last_queried_at, last_fetched_at) -> int:
    """returns priority score; higher = refresh sooner"""
    age = (now - last_fetched_at).total_seconds() / 3600
    volatility = 10 if (now - last_changed_at).days < 7 else 1
    relevance = 5 if (now - last_queried_at).days < 1 else 1
    return age * volatility * relevance

This kind of priority-driven scheduler reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

Common pitfalls when scraping Rakuten Ichiba

Three issues dominate Rakuten Ichiba scraping. The first is shop-vs-product attribution. Rakuten is fundamentally a marketplace of independent shops, each with its own URL pattern, stock policy, and shipping terms. The same JAN code can appear in 50+ shops at different prices. Aggregating to the JAN level without preserving shop_id loses the price-dispersion signal that makes Rakuten data interesting.

The second is point-multiplier inflation. Rakuten campaigns layer point bonuses (5x, 10x, even 20x on Super Sale days). The visible price is in JPY but the effective price after points can be 10-15% lower for SPU members. A scraper that ignores the point multiplier field misreports the realized cost to the buyer. Capture both price and point_multiplier and compute the effective price downstream.

The third is character encoding edge cases. Some legacy shops still serve Shift-JIS or EUC-JP rather than UTF-8. Modern Rakuten infrastructure normalizes most pages to UTF-8 but shop-hosted product descriptions can leak through with the original encoding. Detect encoding per response rather than assuming UTF-8 globally.

FAQ

Do I need a Japanese IP to use the official API?
No. The IchibaItem Search API is accessible from anywhere in the world with a valid applicationId. Only the merchant storefront HTML scraping requires Japanese IP addresses for reliable access.

How many applicationIds can I register?
Rakuten allows multiple applicationIds per developer account. The terms of service require each applicationId to correspond to a real application, and aggressive parallelization across many IDs can trigger account-level throttling. For most production use, 3 to 5 IDs is sufficient and avoids the account-level scrutiny.

Does the API include stock counts?
No. The IchibaItem response includes an availability flag but not numerical stock. For stock-level data you have to scrape the storefront HTML, where merchants sometimes publish real-time stock indicators.

What about Rakuten Books and Rakuten Travel?
Rakuten publishes separate APIs for each major property: BooksTotal Search, TravelHotel Search, and so on. They follow the same registration model and similar rate limits. The applicationId is shared across all Rakuten APIs, so a single registration unlocks the whole family.

Can I scrape Rakuten reviews via the API?
The official API does not expose review text, only aggregate scores and counts. To collect review text you have to scrape the storefront HTML, which raises personal data considerations because reviewer display names are visible. Limit your collection to anonymized aggregates or work with explicit consent if reviewer identity matters.

Is the official Rakuten Ichiba API still useful in 2026?
Yes for catalog discovery and basic price data, but rate limits make it impractical for minute-level monitoring. Most production scrapers blend the official API for breadth with targeted scraping for depth.

How do I track Rakuten Super Sale price drops accurately?
Take baseline snapshots 14 days before the event and compare against intra-event hourly snapshots. Many sellers raise prices in the week before to inflate the apparent discount.

To pair this with a broader Asia ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape Jumia Africa product listings

Scrape Jumia Africa and you are scraping the largest pan-African ecommerce platform, with separate country instances for Nigeria, Kenya, Egypt, Morocco, Ivory Coast, Senegal, Ghana, Uganda, and a handful of other markets. Jumia was founded in 2012 and went public on the NYSE in 2019, and by 2026 it remains the dominant horizontal marketplace across Sub-Saharan Africa with category mixes that lean heavily into mobile phones, home appliances, and fashion. The scraping landscape is shaped by three things: per-country subdomains with different catalogues, a pricing system that mixes Jumia direct, Jumia Mall verified sellers, and a long tail of independent merchants, and an anti-bot layer that becomes more aggressive on the larger Nigeria and Egypt domains.

This guide focuses on Jumia Nigeria as the canonical example, with notes on cross-country differences. The patterns transfer to every Jumia country instance with minor adjustments to the domain and currency.

Mapping the Jumia URL and listing structure

Each Jumia country lives at its own subdomain: www.jumia.com.ng for Nigeria, www.jumia.co.ke for Kenya, www.jumia.com.eg for Egypt, www.jumia.ci for Ivory Coast, and so on. Within each country, the URL structure is consistent: https://www.jumia.com.ng/<product-slug>.html for product detail pages and https://www.jumia.com.ng/<category-slug>/ for category listings. Product slugs include a SKU identifier embedded near the end, which is the canonical key you should store.

Jumia exposes a server-rendered HTML front end with a JavaScript-hydrated layer for filters and recommendations. There is no fully public JSON API, but the product detail pages embed a <script type="application/ld+json"> block that carries Schema.org Product data including price, availability, brand, and SKU. Parsing this JSON-LD block is dramatically more reliable than scraping the visible HTML because the schema is stable while the visible markup changes regularly.

import httpx
from bs4 import BeautifulSoup
import json

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept-Language": "en-NG,en;q=0.9",
}

def parse_product(html: str) -> dict:
    soup = BeautifulSoup(html, "lxml")
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string)
        except (json.JSONDecodeError, TypeError):
            continue
        if isinstance(data, dict) and data.get("@type") == "Product":
            return {
                "sku": data.get("sku"),
                "name": data.get("name"),
                "brand": (data.get("brand") or {}).get("name"),
                "price": (data.get("offers") or {}).get("price"),
                "currency": (data.get("offers") or {}).get("priceCurrency"),
                "availability": (data.get("offers") or {}).get("availability"),
                "rating": (data.get("aggregateRating") or {}).get("ratingValue"),
            }
    return {}

async def fetch_product(url: str, proxy: str) -> dict:
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code != 200:
            return {}
        return parse_product(r.text)

Jumia’s JSON-LD includes the offer price but not the seller breakdown. To get the per-seller information, you have to parse the seller block from the HTML directly. The seller block is structured around a data-merchant-name attribute that is consistent across countries.

Country-specific proxy strategy for Jumia

Jumia’s bot detection becomes meaningfully stricter on the larger country instances. Nigeria and Egypt see the highest scrutiny, both because of the volume of legitimate scraping that already happens (price intelligence vendors, brand monitoring tools, affiliate networks) and because of the volume of fraud activity that uses the same patterns. Kenya and Ivory Coast see lighter enforcement.

Your proxy strategy should match the country you are scraping. For Nigeria and Egypt, use residential or mobile IPs in-country. MTN, Airtel, and 9mobile mobile pools work well for Nigeria. Vodafone and Etisalat residential pools work for Egypt. For the smaller markets, pan-African residential pools or even GCC-region pools sometimes work because Jumia’s geo-classification is less granular for low-volume countries.

Country	Recommended proxy origin	Tolerance per IP
Nigeria	Nigerian mobile or residential	100 req/hr per IP
Egypt	Egyptian residential	100 req/hr per IP
Kenya	Kenyan or pan-African residential	200 req/hr per IP
Morocco	Moroccan or French residential	200 req/hr per IP
Ivory Coast	Pan-African or French residential	250 req/hr per IP
Senegal	Pan-African or French residential	250 req/hr per IP

The cost differential between in-country residential and pan-African residential pools is significant for Nigerian inventory specifically. For most production workloads, the cost is justified by the success rate uplift. For lighter monitoring workloads under 1,000 SKUs per day, pan-African pools sometimes work even for Nigeria with careful rate limiting.

Crawling category trees and pagination

Category listings on Jumia are paginated with a ?page=N query parameter. The maximum reachable page depends on the category but typically caps at 50 pages of 40 products each. To go deeper, decompose by sub-category, brand facet, or price band. The category pages also embed a JSON-LD ItemList block that gives you structured access to the listing.

async def crawl_category(base_url: str, max_pages: int, proxy_pool):
    items = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = f"{base_url}?page={page}"
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url)
            if r.status_code != 200:
                continue
            soup = BeautifulSoup(r.text, "lxml")
            cards = soup.select("article.prd")
            if not cards:
                break
            for card in cards:
                items.append({
                    "sku": card.get("data-sku"),
                    "name": card.select_one("h3.name").get_text(strip=True) if card.select_one("h3.name") else None,
                    "price": card.select_one("div.prc").get_text(strip=True) if card.select_one("div.prc") else None,
                    "url": "https://www.jumia.com.ng" + card.select_one("a.core")["href"] if card.select_one("a.core") else None,
                })
    return items

For very broad categories like Phones and Tablets in Nigeria, the visible pagination only covers the first 2,000 SKUs. The remaining tail requires faceted decomposition. Build a recursive crawler that subdivides any category exceeding 2,000 results into brand and price-band buckets until each bucket fits within the pagination cap.

Tracking Jumia Mall vs. third-party merchant signal

Jumia distinguishes between Jumia Mall (verified sellers with quality guarantees) and ordinary third-party merchants. The distinction matters for analytics because Jumia Mall pricing is often the more stable signal while third-party pricing is more volatile. The seller card on the product page exposes a Jumia Mall badge that you can detect by looking for the mall-badge CSS class.

For brand monitoring use cases, separate the dataset into Mall and non-Mall slices. The Mall slice gives you the canonical price point that the brand wants to maintain. The non-Mall slice gives you the gray-market and parallel-import price activity that often signals supply chain shifts or unauthorized resellers.

Rate limits, retries, and session management

Jumia does not publish rate limits, but observed behavior is consistent across countries. A single IP can sustain about 1 request per 2 seconds for an hour before triggering a soft block that returns either a 429 or a Cloudflare challenge. After a 10-30 minute cooldown the IP is usable again. The cooldown extends if you continue retrying through the block.

import asyncio
async def safe_request(url: str, proxy_pool, max_retries: int = 3):
    for attempt in range(max_retries):
        proxy = proxy_pool.next()
        try:
            async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
                r = await c.get(url)
                if r.status_code == 200:
                    return r
                if r.status_code in (429, 503):
                    await asyncio.sleep(60)
                    continue
        except httpx.HTTPError:
            pass
        await asyncio.sleep(5 * (attempt + 1))
    return None

The exponential backoff matters more on Jumia than on some other African ecommerce sites because the block escalation is relatively slow but the cooldown extends quickly under retries. A patient retry pattern outperforms an aggressive one.

Working with multi-currency Jumia datasets

Each Jumia country uses its own local currency: NGN for Nigeria, KES for Kenya, EGP for Egypt, MAD for Morocco, XOF for Ivory Coast and Senegal. For pan-African analyses, normalize to USD or EUR using daily FX rates rather than scrape-time conversions. NGN in particular has had significant devaluation events in recent years, and any cross-time analysis needs to account for that.

Detecting and routing around CAPTCHA challenges on Jumia

When Jumia flags your traffic, the response is usually a Cloudflare interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched (a specific SKU your client cares about), have a fallback path that uses a headless browser with a real Nigeria residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split: 95% of requests go through the lightweight HTTP and JSON path, 5% fall through to the browser path on challenge.

Working with NGN pricing and FX normalization

Pricing in Nigeria is denominated in NGN, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. The correct pattern is to store the price in local NGN and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing Jumia to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Jumia	Nigeria	Large	High
Konga	Adjacent markets	Medium	Medium
Jiji	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Jumia but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against Jumia, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for Nigeria

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Nigeria has its own consumer protection and personal data frameworks that overlay any general analysis. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment of a scraper that targets Jumia, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="jumia-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Jumia datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. New attribute fields appear, old fields are deprecated, category trees are reorganized, and pricing display logic changes. Your scraper code has to evolve with these changes, and a versioning pattern that keeps old data interpretable is critical. Stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

ALTER TABLE snapshot ADD COLUMN scraper_version VARCHAR(16);
CREATE INDEX scraper_version_idx ON snapshot(scraper_version);

Pair this with a small registry table that documents what each scraper version did differently. When a downstream user asks why a particular metric jumped on a specific date, the version registry usually has the answer.

Common pitfalls when scraping Jumia

Three issues catch most teams off guard. The first is country-domain fragmentation. Jumia operates 11 country sites (jumia.com.ng, jumia.co.ke, jumia.ci, etc), each with its own currency, language, and seller pool. The same SKU can sit on multiple country sites with different prices and different stock. A scraper that treats jumia.com.ng prices as representative of West Africa understates Ivorian prices by 10-25% on average.

The second is JumiaPay vs cash-on-delivery price drift. Some sellers offer a JumiaPay discount that is rendered only when the user signs in. Anonymous scraping captures the list price; authenticated scraping captures the discounted price. Decide which population matters for your analysis and stay consistent.

The third is flash-sale staleness. The flash_sales API serves a curated subset of SKUs with countdown timers. The countdown is computed client-side from a server-issued epoch. If you cache the response for more than 60 seconds during an active flash sale, the timer drifts and downstream consumers see incorrect end times. Bypass cache for flash-sale endpoints and accept the higher request cost during campaign hours.

FAQ

Can I scrape all Jumia countries from a single Nigerian residential IP pool?
Technically yes, but Jumia geo-classifies the visitor IP and serves different catalogues per country. A Nigerian IP requesting jumia.co.ke works but raises the bot score because real Kenyan traffic does not originate from Nigerian residential networks. For each country you intend to scrape, source proxies from that country or from a regional adjacent market.

How does Jumia handle the difference between Mall and non-Mall sellers in the API?
There is no public API. In the HTML, Jumia Mall items are tagged with a CSS badge and a separate data-mall attribute on the seller card. Capture this signal at parse time and persist it in your snapshot table so downstream analyses can filter by Mall status.

Does Jumia’s price include shipping?
The price shown on the listing card is the product price exclusive of shipping. Shipping is calculated at checkout based on delivery zone and seller fulfillment method. For total cost analyses you would need to simulate add-to-basket flows, which is more complex and less reliable. Most price intelligence projects work with the listed price as the canonical signal.

Are Jumia reviews scrapable for sentiment analysis?
Reviews are visible on product pages and accessible through HTML scraping. They include the reviewer’s display name (usually a first name and initial) and the review text. From a privacy perspective, treat the display name as personal data and avoid storing it in long-term datasets. The review text and rating are the analytically useful fields.

Does Jumia run a sitemap I can use for SKU discovery?
Jumia exposes sitemaps at /sitemap.xml for each country. They list categories and product URLs but the per-product entries are typically only a sample of the full catalogue. For exhaustive SKU discovery, combine sitemap parsing with category and seller crawls.

Can one scraper cover all 11 Jumia country sites?
Architecturally yes, but you need per-country proxy pools. A Lagos residential IP that hits jumia.co.ke succeeds at low volume but is throttled aggressively above ~200 requests per hour.

How do I reconcile Jumia data with Konga or other competitors?
Use brand+model+capacity as the join key for electronics. EAN/UPC coverage is patchy, so fall back to fuzzy title matching with a confidence score before merging analytics.

To build a broader Africa ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.

How to scrape Noon UAE ecommerce in 2026

Scrape Noon UAE and you are scraping the dominant home-grown ecommerce platform in the Gulf, with separate marketplaces for the United Arab Emirates, Saudi Arabia, and Egypt. Noon was launched in 2017 as a Mohammed bin Rashid initiative to compete with Amazon’s regional expansion, and by 2026 it has captured a meaningful share of GCC ecommerce GMV through Noon Daily for grocery, Noon Food for delivery, and the core noon.com marketplace for general merchandise. The scraping landscape is shaped by three things: dual-language content (Arabic and English), country-specific catalogues with overlapping but not identical SKUs, and a Cloudflare-front-end that aggressively profiles non-GCC traffic.

This guide covers Noon UAE specifically. Most patterns transfer to noon.com.sa (Saudi) and noon.com.eg (Egypt) with minor adjustments to the country code parameter and currency parsing. The target keyword for this guide is scrape Noon UAE.

How Noon’s URL and language structure works

Noon URLs include a country segment, a language segment, and the product slug. A UAE Arabic URL looks like https://www.noon.com/uae-ar/<product-slug>/<sku>/p/, and the English equivalent is https://www.noon.com/uae-en/.... The two URLs serve different language pages but resolve to the same SKU. The trailing /p/ is what tells Noon’s routing layer that this is a product detail page.

The SKU at the end of the URL is the canonical identifier. A given product is sold in UAE, Saudi, and Egypt with the same SKU prefix but different listing IDs per country. If you are building a cross-country catalogue, capture both the SKU and the country segment as a composite key.

Noon’s product API lives at https://www.noon.com/_svc/catalog/api/v3/u/<sku>/p. The endpoint accepts an X-Locale header that controls language and country. Set it to en-ae for UAE English, ar-ae for UAE Arabic, en-sa for Saudi English, and so on. The JSON shape is consistent across locales.

import httpx
import asyncio

NOON_API = "https://www.noon.com/_svc/catalog/api/v3/u"

def headers(locale: str):
    return {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15",
        "Accept": "application/json",
        "X-Locale": locale,
        "X-Platform": "web",
        "Referer": f"https://www.noon.com/{locale.split('-')[1]}-{locale.split('-')[0]}/",
    }

async def fetch_product(sku: str, locale: str, proxy: str):
    url = f"{NOON_API}/{sku}/p"
    async with httpx.AsyncClient(proxy=proxy, headers=headers(locale), timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return None

The response includes the canonical product object, an offers array with merchant pricing, an attributes array with structured specs, and a crossSellRecommendations block that you can use to discover related SKUs without crawling categories.

Why UAE residential proxies matter for Noon

Noon’s bot detection is built on Cloudflare Bot Management plus an internal scoring service that considers the visitor IP geography, the language requested, and the device fingerprint. A request from an AWS Frankfurt IP asking for X-Locale: ar-ae is immediately suspicious because real UAE Arabic-language traffic almost never originates from European data centers.

The cleanest signal you can give Noon is a UAE residential IP requesting the locale that matches its country. UAE residential pools are smaller than Turkish or Vietnamese pools because the population is smaller, but several major proxy providers list inventory specifically in UAE through Etisalat and du. Expect to pay a premium versus general residential pools.

For workloads under 5,000 SKUs per day, you can sometimes get away with rotating GCC residential IPs (any Saudi or Kuwait IP usually works as long as the locale matches). For higher volumes, dedicated UAE inventory matters because the pool size shrinks and Noon’s behavioral scoring catches the pattern of foreign IPs rapid-fire requesting Arabic content.

Pulling category and search data

Noon exposes a search API at https://www.noon.com/_svc/search/api/v3/u/search. The endpoint takes a query string, a country segment, sort, and pagination params. For category sweeps, you pass the category slug as a filter facet rather than a path param.

async def search_noon(query: str, locale: str, page: int, proxy: str):
    url = "https://www.noon.com/_svc/search/api/v3/u/search"
    params = {
        "q": query,
        "page": page,
        "limit": 50,
        "sort": "popularity",
    }
    async with httpx.AsyncClient(proxy=proxy, headers=headers(locale), timeout=20) as c:
        r = await c.get(url, params=params)
        if r.status_code == 200:
            return r.json().get("hits", [])
        return []

The search API caps result depth at roughly 1,000 hits per query, which is the typical CDN safeguard against scraping. To go deeper into a category, decompose the query by brand, price band, or attribute facets. The search response includes the available facets and counts, which gives you a recipe for subdivision.

Handling Arabic text and RTL parsing

If your downstream pipeline is going to treat product titles as English-only, you will lose half of Noon’s catalogue. Many sellers list products in both languages, but a meaningful long tail of grocery, beauty, and fashion is Arabic-only or has more detailed Arabic descriptions than English ones. Capture both languages from day one.

The simplest pattern is to make two requests per SKU, one with ar-ae and one with en-ae, and merge the results. The title, description, attributes, and seller.name fields differ between the two responses. Everything numeric (price, stock, rating) is identical. For storage, use a JSONB or JSON column that holds both language variants and let downstream consumers pick whichever they need.

For full-text search across the dataset, Postgres with the arabic text search configuration works well, as does Elasticsearch with the Arabic analyzer. Avoid stripping diacritics during ingestion because they carry semantic meaning in Arabic product names, especially for branded items.

Comparing Noon’s three country marketplaces

Country	Currency	Catalogue size estimate	Bot scrutiny	Typical proxy cost
UAE (uae)	AED	8M+ SKUs	High	$$$
Saudi (sa)	SAR	12M+ SKUs	Highest	$$$$
Egypt (eg)	EGP	4M+ SKUs	Medium	$$

Saudi gets the most attention from Noon’s bot defenses because it generates the largest GMV. Egypt has the lightest defenses but the lowest catalogue depth. If you are building a Gulf-wide price intelligence product, plan for separate proxy budgets per country and don’t try to scrape all three from the same IP pool.

Rate limits and retry patterns

Noon’s rate limit thresholds are not published, but observed behavior is consistent: a single residential IP can sustain about 1 product detail request per second for 30-60 minutes before triggering a soft block that returns 403 with a Cloudflare challenge page. After a 5-15 minute cooldown the IP is usable again. Sticky-session residential proxies handle this gracefully with rotation on 403.

async def safe_fetch(sku: str, locale: str, proxy_pool, max_retries: int = 3):
    for attempt in range(max_retries):
        proxy = proxy_pool.next()
        try:
            data = await fetch_product(sku, locale, proxy)
            if data:
                return data
        except httpx.HTTPError:
            pass
        await asyncio.sleep(2 ** attempt + 5)
    return None

For large category sweeps, distribute work across IPs so no single IP exceeds the throughput threshold. A scheduling layer that tracks per-IP request counts and back-off windows pays for itself quickly.

Storing Noon snapshots

Schema for a per-country product snapshot:

CREATE TABLE noon_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    country VARCHAR(2) NOT NULL,
    sku VARCHAR(64) NOT NULL,
    seller_id VARCHAR(64),
    price_aed DECIMAL(12,2),
    sale_price_aed DECIMAL(12,2),
    in_stock BOOLEAN,
    rating DECIMAL(3,2),
    review_count INT,
    title_en TEXT,
    title_ar TEXT,
    PRIMARY KEY (snapshot_at, country, sku)
);

For Saudi and Egypt, swap the price column to local currency. If you are normalizing across countries, store both local currency and a derived AED-equivalent column updated daily from a central FX table. Don’t try to convert at scrape time because exchange-rate noise will pollute your trend lines.

Linking Noon scraping to broader GCC strategy

Noon is the largest single property in GCC ecommerce, but it is not the only one. Amazon UAE and amazon.sa are still major players, and category-specific sites like Sharaf DG for electronics matter for some verticals. Build your scraping stack with a multi-source mindset from the start, even if you are only launching with Noon. Our GCC ecommerce scraping overview collects related guides as we publish them.

For broader proxy strategy in MENA markets, see our residential proxy provider ranking, which now includes vendor-by-vendor UAE and Saudi inventory counts.

Detecting and routing around CAPTCHA challenges

When Noon flags your traffic, the response is usually a Cloudflare interrogation page rather than an HTTP 4xx. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet.

For pages that absolutely must be fetched (a specific SKU your client cares about), have a fallback path that uses a headless browser with real UAE residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split: 95% of requests go through the lightweight HTTP+JSON path, 5% fall through to the browser path on challenge.

Working with AED pricing and FX normalization

Pricing in Noon is denominated in AED, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. The correct pattern is to store the price in local AED and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank’s wholesale feed. Avoid scraping retail FX rates because they include the bank’s spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing Noon to other regional marketplaces

Marketplace	Country focus	Catalogue scale	Bot strictness
Noon	UAE	Large	High
Amazon UAE	Adjacent markets	Medium	Medium
Sharaf DG	Adjacent markets	Smaller	Lower

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Noon but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against Noon, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for UAE

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but UAE has its own consumer protection and personal data frameworks that overlay any general analysis. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment of a scraper that targets Noon, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="Noon-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across Noon datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Common pitfalls when scraping noon

Three failure patterns recur across noon scrapers. The first is country code confusion. noon runs separate storefronts for UAE, Saudi Arabia, and Egypt, each with its own pricing, currency, and stock. The country selector is set via cookie and URL prefix (/uae-en, /saudi-en, /egypt-en). A scraper that drops the country prefix gets redirected by IP geolocation, which corrupts the dataset when proxies move between PoPs. Always pin the country in both the URL and the x-locale header.

The second is double-currency rendering. Saudi pages show prices in SAR but include a USD conversion in the markup for some SKUs. Naive selectors pick whichever appears first in the DOM, which flips between SAR and USD across page versions. Read the price from the structured JSON-LD block, not from visible HTML.

The third is mall-vs-marketplace confusion. noon Mall (curated brand storefronts) and noon Marketplace (third-party sellers) coexist on the same product page. The seller_type and is_noon_mall flags distinguish them. Analytics that treat all listings as one population miss the fact that Mall pricing is more stable while Marketplace pricing reprices weekly. Segment your dataset by seller type before computing trend metrics.

FAQ

Is scraping Noon legal in the UAE?
Public product data is generally considered fair to scrape in most jurisdictions, but UAE law has a strong personal data and consumer protection framework. Restrict your scraping to product, price, and seller data. Avoid pulling buyer reviews with personally identifying details, avoid scraping logged-in pages, and respect any explicit terms of service. For commercial use, consult legal counsel familiar with UAE Federal Decree-Law No. 45 of 2021.

Why do prices differ between Noon UAE and Noon Saudi for the same product?
Each country marketplace is operated as a separate entity with separate seller relationships, separate logistics, and separate pricing decisions. The same SKU can be listed by completely different sellers in the two markets at different prices. Treat each country as an independent dataset for analytical purposes.

Does Noon block VPN traffic?
Noon does not specifically block VPN traffic. It blocks any IP that fingerprints as a data center or hosting provider, which is what most consumer VPN exits look like. Residential and mobile IPs from genuine consumer ISPs in the GCC are the only reliable way to scrape at volume.

How does Noon handle the Friday-Saturday weekend?
Noon catalogue updates and pricing changes continue through the GCC weekend, often with promotional bursts on Friday afternoons. If you are tracking pricing dynamics, your snapshot cadence should not skip weekends.

Can I use Noon’s affiliate API instead of scraping?
Noon runs an affiliate program through Tradedoubler and other networks. The affiliate APIs provide product data for approved affiliates but the catalogue coverage and refresh frequency are limited compared to scraping the public site. For competitive intelligence use cases, scraping remains the higher-fidelity option.

Does noon block residential proxies from non-GCC countries?
Yes, with increasing severity. Casual product lookups from EU or US residential IPs usually succeed, but sustained scraping is throttled within 24 hours. UAE, Saudi, or Egyptian residential or mobile IPs are required for production volume.

How does noon handle price changes during White Friday?
Prices update in waves every 1-3 hours during the campaign. Pre-campaign baselines should be captured at least 7 days before the event to expose true discount depth.

If you are scoping a scraping infrastructure for this market, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.