Scraping legal records: court dockets, case databases

Scrape court dockets and you tap into one of the most analytically valuable but operationally challenging public-information ecosystems on the web. Federal court records sit behind PACER (the Public Access to Court Electronic Records system), state court records spread across roughly 80 distinct state and county systems with no shared schema, and the public-good aggregators (CourtListener, Justia) try to consolidate everything into a single searchable layer. The scraping landscape is shaped by three things: PACER’s per-page billing model that constrains the economics of comprehensive federal scraping, state court diversity that requires per-jurisdiction adapters, and a strong public-good ecosystem (RECAP, Free Law Project) that handles much of the heavy lifting through federated contributions.

This guide focuses on practical patterns for federal court data via PACER and CourtListener, and on the high-volume state systems that matter most for commercial legal intelligence applications.

PACER and the RECAP archive

PACER is the canonical source for U.S. federal court records covering 94 district courts, 13 circuit courts, and 90+ bankruptcy courts. Access is per-page-billed at $0.10 per page with a $30 quarterly cap if you don’t exceed it. Free read-only access is available for opinions but document downloads are billed.

The RECAP project (Reliable Electronic Court Access for Public) is a Free Law Project initiative that builds a free archive of PACER documents through a browser extension that uploads pages users have already paid for. The RECAP archive (accessible through CourtListener) contains tens of millions of federal documents that are freely searchable and downloadable. For most federal docket research, RECAP is the cheapest and easiest entry point.

import httpx

CL_API = "https://www.courtlistener.com/api/rest/v3"

async def search_courtlistener(query: str, court: str = None):
    url = f"{CL_API}/search/"
    params = {"q": query, "type": "r"}  # r = RECAP documents
    if court:
        params["court"] = court
    async with httpx.AsyncClient(timeout=30) as c:
        r = await c.get(url, params=params)
        if r.status_code == 200:
            return r.json().get("results", [])
        return []

CourtListener exposes a free REST API with reasonable rate limits for non-commercial use. For commercial use, paid tiers unlock higher rate limits and bulk export.

State court systems and their diversity

State court systems are the harder problem. There is no centralized state court records system; each of the 50 states (plus DC) operates its own architecture. Some states (California, Florida, New York) have unified state-wide systems. Others delegate to county-level systems with no unified portal.

The high-volume states for commercial legal intelligence are roughly:

For a national legal intelligence product, building per-jurisdiction adapters for the top 20-30 states covers roughly 80% of commercially relevant cases. The remaining long tail requires either heroic scraping effort or partnering with a commercial aggregator like UniCourt or Trellis.

Docket parsing and entity extraction

A docket entry includes a sequence number, a filed date, the document title, the filer (attorney or pro se party), and a link to the underlying document if filed electronically. Parsing the structured docket sequence is straightforward; extracting the legal entities (parties, attorneys, law firms, judges) requires named entity recognition tuned to legal text.

import spacy
nlp = spacy.load("en_core_web_lg")

def extract_legal_entities(docket_text: str) -> dict:
    doc = nlp(docket_text)
    return {
        "persons": [ent.text for ent in doc.ents if ent.label_ == "PERSON"],
        "organizations": [ent.text for ent in doc.ents if ent.label_ == "ORG"],
        "dates": [ent.text for ent in doc.ents if ent.label_ == "DATE"],
    }

For specialized legal NER, the Free Law Project publishes models trained on legal text. These models recognize judges, attorney names, and law firm names with substantially higher accuracy than general-purpose NER models on the same text.

Schema for legal docket snapshots

CREATE TABLE docket_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    court_id VARCHAR(32) NOT NULL,
    case_number VARCHAR(64) NOT NULL,
    docket_entry_seq INT NOT NULL,
    filed_at TIMESTAMP,
    document_title TEXT,
    filer_name TEXT,
    document_url TEXT,
    PRIMARY KEY (snapshot_at, court_id, case_number, docket_entry_seq)
);
CREATE INDEX docket_case_idx ON docket_snapshot(court_id, case_number);

For analytics on attorney activity or law firm activity, build a derived table that aggregates docket entries per attorney and per firm per week. Filing velocity per attorney reveals workload patterns; filing velocity per firm reveals competitive positioning.

For broader pattern guidance, see our residential proxy provider ranking and our GDPR compliance guide for scraping.

Detecting and routing around bot challenges

When court systems and PACER flag your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment....

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of vertical. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        return sum(1 for _, ok in bucket if ok) / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Pipeline orchestration and scheduling

For any non-trivial legal records scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and sources.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_source(source_id: str, page: int):
    return crawl_one_page(source_id, page)

@flow(name="legal-records-daily-sweep")
def daily_sweep(source_ids: list):
    futures = []
    for sid in source_ids:
        for page in range(1, 30):
            futures.append(fetch_source.submit(sid, page))
    return [f.result() for f in futures]

Run the flow on a cadence aligned to how dynamic the underlying data is. For legal records where records change intraday, a 4-6 hour cadence catches meaningful movements. For longer-cycle data, daily is sufficient and the cost saving is meaningful.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. Structural checks verify that every required field is present and of the expected type. A snapshot row missing the canonical identifier is not a real snapshot. Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer records than yesterday, something broke either in collection or in the upstream source. Semantic checks compare related fields for consistency.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size below threshold")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: serve cached responses when consumers ask for the same record within the same hour. The second is conditional GET using ETag or If-Modified-Since headers when supported. The third is selective field hydration when the upstream API supports field selection.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers: collection, parsing, storage, and serving. The collection layer handles the network conversation and knows nothing about data shape. The parsing layer transforms raw bytes into structured records and owns the schema. The storage layer holds the canonical snapshots in a query-optimized format like DuckDB, ClickHouse, or BigQuery. The serving layer exposes the data to consumers and should be denormalized and pre-aggregated where possible.

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Legal and compliance considerations

Public legal records data is generally treated as fair to scrape in most jurisdictions, but always confine your collection to non-personal data: identifiers, structured attributes, and aggregates. Avoid collecting personally identifying details, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and the OECD guidance on AI training data sourcing remain useful starting points for documenting your approach.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across legal records datasets:

-- Volume trend over the last 30 days
SELECT date_trunc('day', snapshot_at) AS day, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY 1 ORDER BY 1;

-- New entities first seen in the last 14 days
SELECT entity_id, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY entity_id
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

-- Source distribution
SELECT source, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY source
ORDER BY records DESC;

Add a category share view, a source concentration view, and a price-volatility view (where applicable) and you have a solid foundation for a legal records intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Versioning your scraper for source evolution

Every legal records source evolves its schema regularly. New fields appear, old fields are deprecated, and display logic changes. Stamp every snapshot row with the scraper version that produced it. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

Pair this with a small registry table that documents what each scraper version did differently. When a downstream user asks why a particular metric jumped on a specific date, the version registry usually has the answer. This habit pays for itself dramatically the first time a parser change introduces a subtle metric drift.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with source size, which becomes expensive at multi-million record scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle: freshness deadline, volatility, and business priority. Records that downstream users actually query get higher refresh priority than dormant records that nobody has looked at in months.

Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the records that actually matter to the business.

Building a litigation analytics dashboard

The most common analytical product on top of legal docket scraping is a litigation analytics dashboard that tracks new filings per court per practice area per week, attorney and firm activity rankings, and judge assignment patterns. The dashboard layer sits on top of the snapshot store and pre-computes the most common views for fast serving.

def attorney_activity(start_date, end_date):
    return db.query("""
        SELECT filer_name, COUNT(*) AS filing_count
        FROM docket_snapshot
        WHERE filed_at BETWEEN %s AND %s
        GROUP BY filer_name
        ORDER BY filing_count DESC
        LIMIT 100
    """, [start_date, end_date])

For commercial legal intelligence products, the headline metrics are practice-area-specific filing trends (patent litigation in the Eastern District of Texas, product liability in California, antitrust in the Southern District of New York). Each of these views informs both law firm business development and litigation finance investment decisions.

Layer in win-rate analytics by attorney, by firm, and by judge for the deepest commercial intelligence. Win rates require parsing case outcomes from disposition orders, which is itself a non-trivial NLP problem. The Free Law Project publishes models that classify dispositions with reasonable accuracy.

International legal scraping notes

Outside the U.S., court access varies dramatically. The UK courts publish judgments through BAILII (free and comprehensive). The European Court of Justice publishes through CURIA (free, structured). Common-law countries like Australia and Canada have similar judgment-publication frameworks. Civil-law countries vary widely; some publish judgments comprehensively, others restrict access to parties.

For multi-jurisdictional legal research, the dataset shape is judgment-and-opinion-centric rather than docket-centric because the docket-tracking culture is much stronger in the U.S. than elsewhere. Build separate adapters for the docket-tracking jurisdictions and the judgment-publishing jurisdictions, with a shared canonical schema for cross-jurisdictional queries.

The personal data dimension is more constrained outside the U.S. EU GDPR specifically considers court records and the names of private individuals appearing in them. The CJEU’s right-to-be-forgotten decisions interact with court-record scraping in nuanced ways. For commercial deployment of EU-facing legal scraping products, specialized counsel is essential.

Common pitfalls when scraping court dockets

Three issues recur across PACER and state court scrapers. The first is sealed-document leakage. PACER serves sealed documents with a ‘sealed’ flag in the metadata but the underlying file is sometimes still downloadable due to clerk error. Hard-code a metadata check before any download and treat any sealed-flag positive as a non-fetch, even if the URL resolves.

The second is docket-entry numbering inconsistency. Federal courts number docket entries sequentially within a case but renumber after consolidation or transfer. State courts use their own conventions. A scraper that joins on docket entry number across consolidations loses the chronology. Use the docket entry timestamp as the secondary sort key.

The third is OCR-quality drift on scanned filings. Older filings (pre-2010 in most districts) are scanned PDFs with variable OCR quality. A keyword search for case-relevant terms misses 5-15% of older documents because the OCR layer dropped or misread the term. Run a second-pass OCR (Textract, Google Vision) on critical historical documents and store both OCR layers for cross-validation.

FAQ

Is scraping court dockets legal?
Court records are public records and are explicitly intended to be accessible. PACER access requires registration and billing; state court access varies. The legal scraping question is generally not whether you can access the data but whether your access pattern complies with the system’s terms and rate limits.

What about sealed cases and protective orders?
Sealed cases and documents under protective orders are not public and should never be scraped. The access systems generally enforce this through their access controls; if you somehow encounter sealed material, do not store or process it.

Can I scrape attorney names and use them for marketing?
Attorney names are public information published in court dockets. Using them for general legal-vertical analytics is fair. Using them for direct marketing to attorneys is regulated by state bar rules and CAN-SPAM equivalents; consult specialized counsel before building marketing lists from docket data.

How do I handle the PACER per-page billing?
For comprehensive federal coverage, the RECAP archive via CourtListener is dramatically cheaper than direct PACER access. For real-time coverage of specific cases, direct PACER access is needed. Budget roughly $1,000-3,000 per month per analyst-equivalent for active federal docket monitoring.

What about international court systems?
UK court records have varying public-access levels through the Royal Courts of Justice and county courts. EU court records also vary widely. The patterns transfer with per-jurisdiction adapters but the legal landscape (especially around personal data in court records) is more restrictive than in the U.S.

Are PACER fees still applicable in 2026?
Yes, with the standard quarterly waiver under $30. RECAP Project mirrors public dockets to reduce repeat downloads of the same document.

How do I monitor a specific case for new filings?
Poll the docket index endpoint hourly during business hours and emit a row for each new docket entry detected. Diff against the previous snapshot rather than re-scraping the full case.

To build broader OSINT and legal intelligence pipelines, browse the cybersecurity-osint category for tooling reviews and framework deep dives.

FERPA-Compliant Education Data Scraping in 2026

Scraping educational data in 2026 without understanding FERPA is how companies end up with cease-and-desist letters from university general counsel. the Family Educational Rights and Privacy Act was written in 1974, but its reach now extends into API endpoints, LMS platforms, and AI training pipelines pulling from .edu domains. enforcement has gotten sharper as edtech data became commercially valuable — and universities have gotten better at noticing.

What FERPA actually covers (and what it doesn’t)

FERPA protects “education records” — any record directly related to a student and maintained by an educational institution or a party acting on its behalf. sounds narrow. it isn’t.

Protected data includes:

• student names linked to course enrollment
• GPA, grades, and academic standing
• disciplinary records
• financial aid data
• student IDs and email addresses issued by the institution

what FERPA doesn’t protect: directory information a school has designated public (typically name, major, dates of attendance), publicly published research, faculty data, and alumni content individuals have made public themselves. this distinction is where most scraping projects live or die.

the practical rule: if data requires institutional authentication to access, treat it as protected. if it’s rendered on a public .edu page with no login wall, you’re probably in directory-information territory — but check the school’s specific policy before building a pipeline around it. they vary more than you’d expect.

The four FERPA scenarios scrapers actually hit

Scenario 1: public faculty and research pages

scraping faculty profiles, lab pages, and published papers is generally fine. Google Scholar, Semantic Scholar, and OpenAlex do it at scale. the data is intentionally public and doesn’t map to student records. but still respect robots.txt and rate limits — the Reddit Lawsuit and Web Scraping: Legal Implications for Data Collectors case showed that ToS violations can escalate into CFAA exposure even when the underlying data looks obviously public.

Scenario 2: LMS and student portal scraping

Blackboard, Canvas, Moodle — all authentication-gated. scraping them with harvested credentials or session tokens is a FERPA violation (third-party unauthorized access to education records) plus a probable CFAA violation. don’t do it. this applies whether you’re building a grade aggregator, an academic benchmarking product, or anything else that requires acting as a logged-in student. there’s no version of this that’s fine.

Scenario 3: third-party edtech APIs

if you’re building on top of Clever, Classlink, or Google Classroom, you’re a “school official” under FERPA the moment you receive student PII through their APIs. your data use has to align with the “legitimate educational interest” the school authorized. selling downstream, using it for ad targeting, or training commercial AI models without explicit consent is a violation — and the institution carries liability. they will come after the vendor.

Scenario 4: LinkedIn / GitHub for student recruitment data

scraping LinkedIn for students who list their university is a gray zone. the student disclosed that information voluntarily on a non-institutional platform. FERPA doesn’t govern LinkedIn. but if you’re cross-referencing those profiles with an institutional directory feed to build a student contact list, you’ve created a derived education record — and that pulls the whole dataset into FERPA’s orbit. it’s the combination that creates the problem.

Risk comparison: data sources by exposure level

Technical safeguards for compliant edtech data pipelines

if your product touches any edge of this landscape, these controls aren’t optional:

1. scope your scraper to public HTML only — no authenticated sessions, no API keys issued to a user account rather than a service account with explicit institutional authorization
2. strip or avoid collecting student identifiers — name + email + university together is enough to constitute an education record under some interpretations
3. document your data lineage — know which fields came from which source and whether any originated from an institutional feed
4. purge on schedule — retain only what you need; FERPA’s “legitimate educational interest” standard implies proportionality
5. review the ToS — Canvas and Blackboard both explicitly prohibit automated access for non-institutional purposes

a minimal compliant scraper for public faculty data looks like this:

import httpx
import time
from urllib.robotparser import RobotFileParser

def can_fetch(url: str, user_agent: str = "ResearchBot/1.0") -> bool:
    rp = RobotFileParser()
    rp.set_url(url.rstrip("/") + "/robots.txt")
    rp.read()
    return rp.can_fetch(user_agent, url)

def scrape_faculty_page(url: str) -> str | None:
    if not can_fetch(url):
        return None
    headers = {"User-Agent": "ResearchBot/1.0 (research use; contact@example.com)"}
    resp = httpx.get(url, headers=headers, timeout=10)
    resp.raise_for_status()
    time.sleep(1.5)
    return resp.text

no session cookies, no student-facing endpoints, rate-limited, robots.txt-compliant. the contact address in User-Agent is worth adding for .edu targets — it signals intent and lowers the chance of an IP block escalating to a legal letter.

How FERPA compares to other sectoral regulations

FERPA is often read as weaker than healthcare or financial regs because it lacks a private right of action. students can’t sue you directly. but that misses how enforcement actually works — loss of federal funding is an existential threat for any institution, which makes university compliance officers extremely aggressive about vendor contracts. a FERPA violation by your product can end the relationship and create downstream liability that lands on you.

the same dynamic plays out across every sector that has sensitive records. HIPAA and Web Scraping: When PHI Risk Bites (2026) covers health data the same way. PCI DSS and Web Scraping: Payment Card Data Risk Patterns (2026) covers payment records. international frameworks do the same thnig — the India DPDP Act and Web Scraping in 2026: Compliance Patterns and Australia Privacy Act and Web Scraping in 2026 both apply proportionality tests once you’re touching education-adjacent personal data. none of this is jurisdiction-specific quirk. it’s just how privacy law works now: consent, purpose limitation, and data minimization once PII is in scope.

Bottom line

public .edu content is scrapable. student records — direct or derived — aren’t, and the exposure is institutional-grade liability, not just a takedown notice. stay on the public side of the authentication wall, strip identifiers you don’t need, and document your sources. DRT covers this compliance landscape in depth because the line between “public data” and “protected record” is exactly where most enforcement actions originate.

Scraping medical and clinic data for healthcare research

Scrape medical data and you operate in one of the most consequential and compliance-sensitive verticals in commercial scraping. Healthcare data spans provider directories (Healthgrades, Vitals, ZocDoc, NPPES), clinical trial registries (ClinicalTrials.gov, EudraCT), drug pricing databases (Medicare’s Drug Spending Dashboard, GoodRx), and hospital quality metrics (CMS Hospital Compare, Leapfrog). Each source serves a different research question and each has its own access pattern, but the core compliance principle is the same: medical data requires more careful handling than commercial data, and the line between provider information (generally fair to scrape) and patient data (essentially never fair to scrape) is the line you must always respect.

This guide focuses on provider and facility data because that is the slice with broad analytical applicability and clear public-information status. Patient-level data sits behind HIPAA and equivalent international frameworks and is out of scope for ethical scraping projects.

The NPI registry as a foundation dataset

The U.S. National Plan and Provider Enumeration System (NPPES) publishes the canonical National Provider Identifier registry as a free, downloadable dataset. Every U.S. healthcare provider has a 10-digit NPI, and the dataset includes provider name, taxonomy code (specialty), practice address, and credential information. The dataset is updated daily and the full file is roughly 8 GB compressed.

import requests

NPPES_URL = "https://download.cms.gov/nppes/NPPES_Data_Dissemination_<date>.zip"

def download_nppes(date_str: str, target_path: str):
    url = NPPES_URL.replace("<date>", date_str)
    response = requests.get(url, stream=True)
    with open(target_path, "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)

For most healthcare research projects, the NPPES file is the canonical foundation. Build your provider universe from NPPES and join scraped data from the directory sites against the NPI as the canonical key. This approach saves substantial scraping effort because the directory sites are themselves built on top of NPPES plus their own user-generated content.

Provider directory scraping patterns

Healthgrades, Vitals, and ZocDoc are the dominant U.S. consumer-facing provider directories. Each maintains its own provider profile pages with overlapping but not identical content. The valuable additions over NPPES are user reviews, accepted insurance lists, hospital affiliations, and book-an-appointment availability.

import httpx
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "text/html",
    "Accept-Language": "en-US,en;q=0.9",
}

async def fetch_healthgrades_profile(npi: str, proxy: str):
    url = f"https://www.healthgrades.com/physician/dr-{npi}"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url, follow_redirects=True)
        if r.status_code == 200:
            soup = BeautifulSoup(r.text, "lxml")
            for script in soup.find_all("script", type="application/ld+json"):
                try:
                    import json
                    data = json.loads(script.string)
                    if data.get("@type") == "Physician":
                        return data
                except (json.JSONDecodeError, TypeError):
                    pass
        return None

The directory sites embed Schema.org Physician structured data in JSON-LD blocks, which is dramatically more reliable than parsing the rendered HTML. The structured data includes ratings, accepted insurance, languages spoken, and gender, all of which add useful dimensions to research.

ClinicalTrials.gov for clinical research data

ClinicalTrials.gov is the U.S. federal registry of publicly and privately funded clinical studies. It exposes a comprehensive REST API at https://clinicaltrials.gov/api/v2/studies that returns structured study records. No scraping is needed because the API is officially supported and well documented.

async def search_trials(condition: str, status: str = "RECRUITING"):
    url = "https://clinicaltrials.gov/api/v2/studies"
    params = {
        "query.cond": condition,
        "query.status": status,
        "pageSize": 100,
    }
    async with httpx.AsyncClient(timeout=20) as c:
        r = await c.get(url, params=params)
        if r.status_code == 200:
            return r.json().get("studies", [])
        return []

The corresponding European registry is EudraCT, which is less developer-friendly but exposes a similar dataset. Cross-referencing trial records across registries reveals duplicate registrations and gives you a single canonical view of global trial activity per indication.

Hospital quality metrics from CMS

The Centers for Medicare and Medicaid Services publish a comprehensive set of hospital quality metrics through the Hospital Compare program. The data is available as direct downloads at data.cms.gov/provider-data and includes mortality rates, readmission rates, patient safety indicators, and patient experience scores per facility. Like NPPES, these are free downloads rather than scraping targets, but they integrate naturally with scraped directory data through facility identifiers.

For broader pattern guidance, see our residential proxy provider ranking and our GDPR compliance guide for scraping.

Detecting and routing around bot challenges

When directory sites and registries flag your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment....

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of vertical. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        return sum(1 for _, ok in bucket if ok) / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Pipeline orchestration and scheduling

For any non-trivial medical data scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and sources.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_source(source_id: str, page: int):
    return crawl_one_page(source_id, page)

@flow(name="medical-data-daily-sweep")
def daily_sweep(source_ids: list):
    futures = []
    for sid in source_ids:
        for page in range(1, 30):
            futures.append(fetch_source.submit(sid, page))
    return [f.result() for f in futures]

Run the flow on a cadence aligned to how dynamic the underlying data is. For medical data where records change intraday, a 4-6 hour cadence catches meaningful movements. For longer-cycle data, daily is sufficient and the cost saving is meaningful.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. Structural checks verify that every required field is present and of the expected type. A snapshot row missing the canonical identifier is not a real snapshot. Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer records than yesterday, something broke either in collection or in the upstream source. Semantic checks compare related fields for consistency.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size below threshold")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: serve cached responses when consumers ask for the same record within the same hour. The second is conditional GET using ETag or If-Modified-Since headers when supported. The third is selective field hydration when the upstream API supports field selection.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers: collection, parsing, storage, and serving. The collection layer handles the network conversation and knows nothing about data shape. The parsing layer transforms raw bytes into structured records and owns the schema. The storage layer holds the canonical snapshots in a query-optimized format like DuckDB, ClickHouse, or BigQuery. The serving layer exposes the data to consumers and should be denormalized and pre-aggregated where possible.

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Legal and compliance considerations

Public medical data data is generally treated as fair to scrape in most jurisdictions, but always confine your collection to non-personal data: identifiers, structured attributes, and aggregates. Avoid collecting personally identifying details, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and the OECD guidance on AI training data sourcing remain useful starting points for documenting your approach.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across medical data datasets:

-- Volume trend over the last 30 days
SELECT date_trunc('day', snapshot_at) AS day, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY 1 ORDER BY 1;

-- New entities first seen in the last 14 days
SELECT entity_id, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY entity_id
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

-- Source distribution
SELECT source, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY source
ORDER BY records DESC;

Add a category share view, a source concentration view, and a price-volatility view (where applicable) and you have a solid foundation for a medical data intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Versioning your scraper for source evolution

Every medical data source evolves its schema regularly. New fields appear, old fields are deprecated, and display logic changes. Stamp every snapshot row with the scraper version that produced it. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

Pair this with a small registry table that documents what each scraper version did differently. When a downstream user asks why a particular metric jumped on a specific date, the version registry usually has the answer. This habit pays for itself dramatically the first time a parser change introduces a subtle metric drift.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with source size, which becomes expensive at multi-million record scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle: freshness deadline, volatility, and business priority. Records that downstream users actually query get higher refresh priority than dormant records that nobody has looked at in months.

Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the records that actually matter to the business.

Building a provider quality dashboard

The most common analytical product on top of medical scraping is a provider quality dashboard that combines NPPES baseline data, scraped review data from Healthgrades and Vitals, and CMS quality metrics where applicable. The dashboard tracks per-provider average rating, review count trend, accepted insurance, and (for hospital-affiliated providers) the linked facility’s CMS quality scores.

def provider_summary(npi):
    nppes = nppes_lookup(npi)
    healthgrades = scrape_healthgrades_profile(npi)
    cms = cms_facility_scores(nppes['facility_id']) if nppes.get('facility_id') else None
    return {
        'name': nppes['name'],
        'specialty': nppes['taxonomy'],
        'rating': healthgrades.get('aggregateRating', {}).get('ratingValue'),
        'review_count': healthgrades.get('aggregateRating', {}).get('reviewCount'),
        'cms_score': cms.get('overall_score') if cms else None,
    }

The combined view is dramatically more valuable than any single source on its own. Healthgrades reviews give you the patient-experience signal; CMS scores give you the clinical-quality signal; NPPES gives you the canonical professional identity.

For broader context, layer in malpractice claim data where available (Medical Malpractice Payment Reports through the National Practitioner Data Bank, with restricted access), and disciplinary action data from state medical boards. The complete provider intelligence stack supports use cases from health insurer network design to consumer-facing provider-finder products.

International medical scraping notes

Outside the U.S., the structures differ but the principles transfer. The UK has the GMC Specialist Register for doctors and the NMC register for nurses, both publicly searchable. Germany has Bundesarztregister maintained by the Bundesarztkammer. Each country’s framework treats provider data as public-information-by-default while keeping patient data strictly protected. For international healthcare research, build per-country adapters and maintain a unified canonical schema centered on a country-plus-provider-id composite key.

For research projects spanning multiple jurisdictions, also account for the GDPR special categories of data. Even though provider data is not health data of patients, the linkage between providers and the conditions they treat sometimes brings analytical outputs uncomfortably close to special-category territory. Document your purpose limitation carefully and consult specialized counsel for any commercial deployment.

Common pitfalls when scraping medical clinic data

Three issues dominate medical-data scraping. The first is HIPAA-adjacent risk creep. The directory data itself (clinic name, address, hours, accepted insurance) is publicly listed and outside HIPAA. The moment a dataset combines clinic data with patient-review text that names individuals or describes specific medical conditions, the analytical surface enters a more sensitive zone. Strip patient-identifying language at ingest, not at the report layer.

The second is NPI vs DEA conflation. Provider directories use NPI (National Provider Identifier) as the canonical key. DEA numbers identify prescribing authority. They are not interchangeable; a clinic can have many NPIs for individual practitioners under one DEA number. Joining on the wrong key inflates provider counts.

The third is in-network status staleness. Insurance-network membership changes monthly but most directory pages cache it for 30-90 days. A scraper that reports the listed in-network status as current can mislead patients. For research-grade data, snapshot the directory monthly and treat the in-network field as a 30-day moving observation.

FAQ

Is scraping doctor directory sites legal?
Provider information (name, specialty, practice address, NPI) is public information from federal registries, and the directory sites publish it as a value-added service. Scraping the basic directory information is generally fair. User-generated reviews require more care because they include personal opinions; treat the review text as the personal data of the reviewer.

What about HIPAA and patient data?
HIPAA restricts the use of Protected Health Information (PHI), which includes anything that identifies a patient combined with health information. Provider data does not include PHI. If your scraping pipeline somehow captures patient-identifying information, that is a serious problem requiring immediate remediation.

Can I scrape pharmacy or drug pricing data?
GoodRx and similar drug-pricing sites expose pharmacy-specific pricing for prescription medications. The data is public but the sites enforce aggressive bot defenses. Medicare’s Drug Spending Dashboard provides similar data through direct download with no scraping needed.

How do I handle international medical data?
Each country has its own provider registry and its own privacy framework. UK has the GMC register, Germany has the Bundesarztregister, and so on. The data is generally public but the access patterns differ. EU GDPR adds extra constraints on processing health-adjacent data even when the underlying records are technically public.

What about telehealth platform scraping?
Telehealth platforms like Teladoc or Amwell are commercial services with explicit terms of service that prohibit unauthorized scraping. The provider directory aspects overlap with general directory scraping; the booking and consultation aspects are out of bounds.

Is it legal to scrape a clinic directory?
Public-facing directory pages are generally scrapeable for research purposes. Aggregating patient-identifiable review content has different legal exposure and should be reviewed with counsel before any redistribution.

How do I deduplicate the same physician across multiple clinic affiliations?
NPI is the canonical key. Many physicians work at 3-5 affiliated practices. Treat the (NPI, clinic_id) pair as the row primary key and aggregate by NPI for physician-level metrics.

Which directories carry the most reliable specialty and credential data?
The CMS NPI registry is the canonical source for primary specialty and license state. Vitals, Healthgrades, and Doximity supplement with patient-rating signal but their specialty taxonomies do not always match CMS exactly. For research-grade analyses, anchor on CMS for credentials and treat consumer directories as secondary observations layered on top.

To build broader healthcare research pipelines, browse the cybersecurity-osint category for tooling reviews and framework deep dives.

Scraping real estate listings: Zillow, Redfin, Rightmove

Scrape real estate listings and you tap into one of the most analytically rich datasets on the public web. Zillow alone publishes more than 135 million U.S. property records, Redfin overlaps with a similar national footprint, and Rightmove dominates the UK with more than a million active listings. Each platform exposes a different schema, a different anti-bot posture, and a different update cadence, but the core analytical questions are the same: what is on the market, how is it priced, how long does it sit, and how do those signals trend over time. The scraping landscape is shaped by three things: aggressive bot detection on Zillow specifically, an MLS-feed structure that constrains what can legally be republished, and geographic specificity in URL patterns that requires per-region scraping rather than a single global crawl.

This guide covers the U.S. (Zillow and Redfin) and the UK (Rightmove). The patterns transfer to Realtor.com, Trulia, and continental European portals like ImmoScout24 with minor adjustments.

Source taxonomy and listing identifiers

Each platform uses its own identifier scheme but they all anchor on a property address. Zillow uses a numeric zpid (Zillow Property ID) that persists for the life of the property. Redfin uses a numeric propertyId. Rightmove uses a numeric propertyId with a different namespace. Cross-platform deduplication relies on the canonical address rather than any platform-specific ID.

Address normalization is the first hard problem in real estate scraping. The same property can be listed as “123 Main St, Apt 4B, New York, NY 10001” or “123 Main Street #4B, New York 10001” or “123 MAIN ST APT 4B NEW YORK NY 10001”. Use a structured address parser like libpostal or the USPS API to normalize before deduplication.

import httpx

ZILLOW_HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-US,en;q=0.9",
}

async def fetch_zillow_property(zpid: int, proxy: str):
    url = f"https://www.zillow.com/graphql/?zpid={zpid}"
    payload = {
        "operationName": "FullPropertyDetailsQuery",
        "variables": {"zpid": zpid},
        "query": "query FullPropertyDetailsQuery($zpid: ID!) { property(zpid: $zpid) { zpid streetAddress city state zipcode price beds baths livingArea zestimate rentZestimate yearBuilt homeStatus } }",
    }
    async with httpx.AsyncClient(proxy=proxy, headers=ZILLOW_HEADERS, timeout=20) as c:
        r = await c.post(url, json=payload)
        if r.status_code == 200:
            return r.json().get("data", {}).get("property")
        return None

Zillow’s GraphQL endpoint is the canonical access path for structured property data. The response includes the canonical fields plus zestimate, rent zestimate, school assignments, and historical price data. Zillow gates access aggressively, so this endpoint requires U.S. residential proxies for sustained scraping.

Proxy strategy for the major platforms

Zillow has the most aggressive bot detection of any major real estate platform. The site fingerprints TLS, HTTP/2 frames, header order, and behavioral patterns. U.S. residential or mobile IPs are required, and even with clean IPs, request rates above 1 per 5 seconds per IP trigger soft blocks. Plan for a substantial proxy budget if you intend to scrape Zillow at scale.

Redfin is moderately defended. U.S. datacenter IPs sometimes work for short bursts, but residential is strongly preferred for sustained operation. Rate limits are similar to Zillow but the challenge frequency is lower.

Rightmove is the lightest of the three. UK datacenter IPs work for moderate workloads, and UK residential pools handle high-volume scraping comfortably. The platform does enforce rate limits per IP but does not aggressively challenge.

Geographic crawl strategy

Real estate is hyperlocal, which means the natural unit of crawling is a geographic region (a city, a county, a school district, or a postcode). Each platform exposes region-level listing endpoints that you can iterate to enumerate active properties.

For Zillow, the region search endpoint accepts a polygon or a city name and returns listings within that region. For Rightmove, the search URL takes a locationIdentifier parameter that maps to their internal geographic taxonomy. Rightmove publishes the location identifier dictionary at a static endpoint that you can cache.

async def search_rightmove(location_id: str, page: int, proxy: str):
    url = "https://www.rightmove.co.uk/api/_search"
    params = {
        "locationIdentifier": location_id,
        "numberOfPropertiesPerPage": 24,
        "index": (page - 1) * 24,
        "sortType": 6,
        "channel": "BUY",
    }
    async with httpx.AsyncClient(proxy=proxy, timeout=20) as c:
        r = await c.get(url, params=params, headers={"User-Agent": "Mozilla/5.0"})
        if r.status_code == 200:
            return r.json().get("properties", [])
        return []

For comprehensive U.S. coverage, decompose by ZIP code. There are roughly 33,000 ZIP codes in the U.S., and a daily snapshot at the ZIP level gives you complete national coverage. For UK coverage, decompose by Rightmove location identifier (roughly 50,000 nodes). For most analytical use cases, the major metro areas (top 50 U.S. metros, top 20 UK regions) capture 80%+ of the meaningful market activity.

Schema for cross-platform property snapshots

CREATE TABLE property_listing_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    canonical_address_hash VARCHAR(64) NOT NULL,
    source VARCHAR(16) NOT NULL,
    platform_id VARCHAR(64) NOT NULL,
    asking_price DECIMAL(14,2),
    currency VARCHAR(3),
    beds INT,
    baths DECIMAL(4,1),
    living_area_sqft INT,
    list_date DATE,
    days_on_market INT,
    status VARCHAR(16),
    PRIMARY KEY (snapshot_at, canonical_address_hash, source)
);

The canonical_address_hash is a SHA-256 of the normalized address tuple. This lets you deduplicate the same property across multiple platforms without storing the raw address as the join key. For analytics that need address text (rare), you can join back to a separate address dictionary table.

Days-on-market and price-history derivation

The two most analytically valuable derived metrics are days-on-market and price-history. Most platforms expose days-on-market directly, but the value drifts over time because listings get re-listed or status-changed. The cleanest approach is to derive your own days-on-market from the first appearance of a property in your snapshots.

For price-history, compare the asking_price across consecutive snapshots and emit a price-change event whenever it differs by more than 0.5% (to filter rounding noise). Aggregating price-change events at the metro level reveals the headline real estate market story far more clearly than averaging the listed prices.

For broader pattern guidance on real estate scraping, see our residential proxy provider ranking and our headless browser frameworks ranking.

Detecting and routing around bot challenges

When Zillow and similar real estate sources flag your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment....

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of vertical. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Pipeline orchestration and scheduling

For any non-trivial real estate scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_source(source_id: str, page: int):
    return crawl_one_page(source_id, page)

@flow(name="real-estate-daily-sweep")
def daily_sweep(source_ids: list):
    futures = []
    for sid in source_ids:
        for page in range(1, 30):
            futures.append(fetch_source.submit(sid, page))
    return [f.result() for f in futures]

Run the flow on a cadence aligned to how dynamic the underlying data is. For real estate where records change intraday, a 4-6 hour cadence catches meaningful movements. For longer-cycle data, daily is sufficient.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. Structural checks verify that every required field is present and of the expected type. Distributional checks compare the current snapshot against recent history. Semantic checks compare related fields for consistency.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: serve cached responses when consumers ask for the same record within the same hour. The second is conditional GET using ETag or If-Modified-Since headers. The third is selective field hydration when the upstream API supports it.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers: collection, parsing, storage, and serving. The collection layer handles the network conversation and knows nothing about data shape. The parsing layer transforms raw bytes into structured records and owns the schema. The storage layer holds the canonical snapshots in a query-optimized format like DuckDB or ClickHouse. The serving layer exposes the data to consumers and should be denormalized and pre-aggregated.

Decoupling these layers enables independent scaling. The collection layer is bound by proxy capacity. The parsing layer is CPU-bound. The storage layer is bound by I/O. The serving layer is bound by query concurrency.

Legal and compliance considerations

Public real estate data is generally treated as fair to scrape in most jurisdictions, but always confine your collection to non-personal data: identifiers, structured attributes, and aggregates. Avoid collecting personally identifying details, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Sample analytics queries

-- Volume trend over the last 30 days
SELECT date_trunc('day', snapshot_at) AS day, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY 1 ORDER BY 1;

-- New entities first seen in the last 14 days
SELECT entity_id, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY entity_id
HAVING MIN(snapshot_at) > now() - interval '14 days';

-- Source distribution
SELECT source, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY source
ORDER BY records DESC;

Add a category share view, a source concentration view, and a price-volatility view (where applicable) and you have a solid foundation for a real estate intelligence product.

Versioning your scraper for source evolution

Every real estate source evolves its schema regularly. New fields appear, old fields are deprecated, and display logic changes. Stamp every snapshot row with the scraper version that produced it. Downstream analytics can filter by version when they need consistent semantics across a time range. Pair this with a small registry table that documents what each scraper version did differently so debugging unexpected metric jumps becomes tractable.

Building a metro-level dashboard from the dataset

The most common analytical product on top of real estate scraping is a metro-level market-temperature dashboard. The dashboard tracks median asking price, median days-on-market, inventory count, and price-change-event count for each major metro on a daily basis. Across the U.S. top 100 metros, this is roughly 10,000 data points per day, which is trivially storable and queryable.

def daily_metro_summary(snap_df, metro_col):
    return snap_df.groupby([metro_col, 'snapshot_date']).agg(
        median_price=('asking_price', 'median'),
        active_count=('canonical_address_hash', 'nunique'),
        median_dom=('days_on_market', 'median'),
    )

The headline metric for most consumer-facing products is the median asking price trend. The headline metric for most professional-facing products is days-on-market because it leads price changes by 4-6 weeks. Build both views and let the consumer choose.

For richer analytics, layer in school district overlays, walkability scores, and crime data. Each of these is its own scraping or licensing challenge but the combined dataset is dramatically more valuable than property listings alone.

Cross-platform pricing arbitrage signal

When the same property is listed on Zillow, Redfin, and the listing brokerage’s own site at different prices, you have a small but meaningful signal of either listing-staleness or active price-testing by the agent. Track the cross-platform price-spread per property as a derived metric and surface the largest spreads to subscribers as alerts.

The pattern works particularly well in the UK where Rightmove and Zoopla often show different prices for the same property because different agents post the property on different platforms. The price spread itself is uninteresting, but the time-derivative of the spread (how the spread changes when one platform updates and the other lags) is a strong indicator of which platform is the agent’s primary channel.

International real estate scraping notes

Outside the U.S. and UK, the dominant real estate portals shift but the patterns transfer. ImmoScout24 dominates Germany, Idealista dominates Spain and Italy, SeLoger dominates France, and PropertyGuru dominates Singapore and Malaysia. Each has its own anti-bot posture and its own URL structure, but the canonical fields (address, price, beds, baths, area, list date) are universal.

For multi-country pipelines, build a per-country adapter pattern with a shared canonical schema. The shared schema is the integration point; the adapters handle the source-specific quirks. This pattern scales cleanly to 10-20 countries without becoming an unmaintainable mess.

Common pitfalls when scraping real estate listings

Three issues account for most production incidents. The first is listing-status staleness. MLS data flows through aggregators (Zillow, Realtor.com, Redfin) with 15-60 minute lags. A listing marked Active on the public site can be Pending or Closed in the underlying MLS feed. For absorption-rate or days-on-market analytics, cross-check the listing status against the public records layer rather than trusting the portal status field alone.

The second is duplicate-listing inflation. The same property frequently appears with different listing IDs across portals when the listing agreement changes (relisting after expiration). A scraper that deduplicates by listing id double-counts the property. Use the parcel ID or address hash as the canonical property key and treat listing id as a child.

The third is square-footage source ambiguity. The portal can pull square footage from the MLS, the county tax record, the appraisal, or the listing agent’s input, and these often disagree by 5-20%. Capture which source produced the value (sqft_source field where available) and prefer county-record values for analytics that compare across markets.

FAQ

Are Zillow and Redfin listings legal to scrape?
Property listings are mostly aggregated from MLS feeds, which are licensed datasets with redistribution restrictions. Public listing pages are generally fair to scrape for analytical use, but redistributing the listings or building a competing portal raises licensing complexity. Confine your collection to non-personal data and consult counsel for commercial use cases.

What about Zestimate values? Are those reliable?
Zestimates are Zillow’s automated valuation model output. They have well-documented accuracy issues, especially in rural and atypical-property segments. For most analytical purposes the asking price and the actual sold price (when available) are stronger signals than the Zestimate.

How do I track sold properties vs. active listings?
Both Zillow and Redfin expose sold listings as a separate filter. Snapshot active and sold separately and join on the property identifier to compute time-from-list-to-sale. This metric is one of the strongest indicators of market temperature.

Can I scrape MLS data directly?
MLS data is licensed through regional Multiple Listing Services and requires an agent or broker membership. Direct MLS scraping is generally not feasible legally. Public-facing portals like Zillow and Redfin remain the realistic source for analytical work.

Does Rightmove expose sold-price history?
Yes. Rightmove has a separate sold-price section that exposes UK Land Registry data. The data is available without aggressive bot defenses and is dramatically more comprehensive than U.S. equivalents because the UK Land Registry publishes sold prices publicly.

Is scraping Zillow’s public site legal in 2026?
Public listing data is widely accepted as scrapeable for analytical purposes per the hiQ ruling lineage. Reselling raw listings or contact information falls into a different regulatory zone and requires direct MLS or syndication agreements.

How do I track price reductions over time on a single listing?
Snapshot daily and store every price observation with the scrape timestamp. Compute price-reduction events by detecting downward changes in the rolling daily series.

To build broader real estate intelligence pipelines, browse the ecommerce scraping category for tooling reviews and framework deep dives.

Scraping car dealership inventory data in 2026

Scrape dealership inventory and you tap into one of the largest single-vertical datasets in commercial scraping. The U.S. has roughly 18,000 franchised dealerships and another 35,000 independent used-car dealers, each maintaining a public inventory feed. The aggregator sites (AutoTrader, Cars.com, CarGurus, TrueCar) sit on top, and OEM dealer locators (BMW, Toyota, Ford) sit alongside. The combined dataset gives you near-real-time visibility into pricing, model mix, days-on-lot, and regional inventory distribution. The scraping landscape is shaped by three things: an aggressive layer of bot detection on the major aggregator sites, a long tail of dealer-by-dealer scraping required for full coverage, and an inventory schema that varies meaningfully across sources.

This guide focuses on the U.S. market because it is the largest and best-documented. The patterns transfer to UK and EU dealership scraping with minor adjustments.

Source taxonomy and data shapes

The dealership inventory data ecosystem has three distinct source types, each with its own scraping characteristics.

Aggregator sites consolidate inventory from thousands of dealers into a single browseable catalogue. AutoTrader, Cars.com, and CarGurus are the dominant U.S. aggregators. They expose listing search APIs (most undocumented) that return VIN, make, model, year, mileage, asking price, dealer name, and location. The advantage is breadth in a single source. The disadvantage is aggressive bot defenses because the aggregators are themselves businesses that monetize the data.

OEM dealer locators (BMW, Mercedes, Toyota, Ford, GM brand sites) expose new-vehicle inventory across the manufacturer’s authorized dealer network. These tend to be less aggressively defended than aggregators because they are designed for consumers shopping for a specific brand. The schema is brand-specific and includes manufacturer-specific options like build configurations.

Direct dealership websites are the long-tail source. Most dealerships use one of a handful of website platforms (DealerOn, Dealer.com, DealerInspire, AutoTrader’s own platform). Each platform has its own URL structure and inventory feed format, but within a platform the structure is consistent.

import httpx
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15",
    "Accept": "text/html,application/json",
    "Accept-Language": "en-US,en;q=0.9",
}

async def scrape_dealer_inventory(dealer_url: str, proxy: str):
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(f"{dealer_url}/inventory")
        if r.status_code != 200:
            return []
        soup = BeautifulSoup(r.text, "lxml")
        # Most dealer platforms embed an inventory JSON in script tags
        for script in soup.find_all("script"):
            if script.string and "vehicleInventory" in (script.string or ""):
                return parse_dealer_json(script.string)
        return []

For comprehensive coverage, scrape both the aggregators and a sample of direct dealer sites. The aggregators give you breadth fast; direct dealer sites give you the most current pricing because aggregators cache for 12-24 hours.

VIN as the canonical identifier

The Vehicle Identification Number is the universal canonical identifier for any specific vehicle. Every car has a unique 17-character VIN. Aggregator listings, dealer feeds, and OEM dealer locators all expose VINs, which makes cross-source deduplication straightforward.

CREATE TABLE vehicle_listing_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    vin VARCHAR(17) NOT NULL,
    source VARCHAR(32) NOT NULL,
    dealer_id VARCHAR(64),
    dealer_zip VARCHAR(10),
    asking_price_usd INT,
    mileage INT,
    days_on_lot INT,
    PRIMARY KEY (snapshot_at, vin, source)
);
CREATE INDEX vin_idx ON vehicle_listing_snapshot(vin);

Tracking the same VIN across sources reveals interesting patterns. The same vehicle often sits at different asking prices on different aggregators because dealers list at different price points across channels. The dealer’s own website often has the freshest price; aggregators lag by 12-24 hours.

Pricing normalization across sources

Aggregator sites display “asking price” but the meaning varies. Some include freight and dealer prep fees; others exclude them. Some show MSRP minus advertised incentives; others show the dealer’s actual posted price. Build a normalization step that captures both the raw advertised price and a normalized “out-the-door estimate” that adds estimated taxes and fees.

For brand monitoring use cases, store the raw fields from each source and compute the normalized comparison at query time. Hard-coding normalization at scrape time loses the underlying signal and makes downstream debugging harder when the upstream definitions change.

Days-on-lot and price-history derivation

The two most analytically valuable derived metrics are days-on-lot (how long a vehicle has been listed) and price-history (the sequence of price changes during the listing). Neither is exposed directly by most sources, but both can be derived from snapshot diffs.

For days-on-lot, track the first appearance of each VIN in your snapshots and compute the difference in days against the current snapshot. For price-history, compare the asking_price field across consecutive snapshots and emit a price-change event whenever it differs.

def derive_price_changes(prev_snap: dict, curr_snap: dict) -> list:
    changes = []
    for vin, curr_row in curr_snap.items():
        prev_row = prev_snap.get(vin)
        if not prev_row:
            continue
        if prev_row["asking_price"] != curr_row["asking_price"]:
            changes.append({
                "vin": vin,
                "old_price": prev_row["asking_price"],
                "new_price": curr_row["asking_price"],
                "delta": curr_row["asking_price"] - prev_row["asking_price"],
                "changed_at": now(),
            })
    return changes

The price-change event stream is the foundation for the dealer-pricing-strategy reports that a finance team or a brand team actually wants to consume.

Proxy strategy for dealership scraping

Aggregator sites enforce aggressive bot detection and require U.S. residential or mobile IPs for sustained scraping. Direct dealer sites are much less aggressive and often work from datacenter IPs as long as you respect basic rate limits. OEM dealer locators sit in between.

For workloads under 5,000 listings per day, a small U.S. residential pool is sufficient. For comprehensive daily snapshots covering 500,000+ active listings, a dedicated mobile proxy pool with 50+ ports is the production-grade approach. The math works out at $300-500 per month for the proxy infrastructure, which is small relative to the analytical value of the dataset.

For deeper proxy strategy guidance, see our residential vs mobile proxy comparison and our best web scraping APIs ranking.

Detecting and routing around bot challenges

When automotive inventory sources flag your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment....

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. For pages that absolutely must be fetched, have a fallback path that uses a headless browser. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of vertical. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Pipeline orchestration and scheduling

For any non-trivial automotive inventory scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and sources.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_source(source_id: str, page: int):
    return crawl_one_page(source_id, page)

@flow(name="automotive-inventory-daily-sweep")
def daily_sweep(source_ids: list):
    futures = []
    for sid in source_ids:
        for page in range(1, 30):
            futures.append(fetch_source.submit(sid, page))
    return [f.result() for f in futures]

Run the flow on a cadence aligned to how dynamic the underlying data is. For automotive inventory where pricing or availability changes intraday, a 4-6 hour cadence catches meaningful movements without driving up proxy costs.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. Structural checks verify that every required field is present and of the expected type. Distributional checks compare the current snapshot against recent history. Semantic checks compare related fields for consistency.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same record within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads. The collection layer handles the network conversation. The parsing layer transforms raw bytes into structured records. The storage layer holds the canonical snapshots in a query-optimized format. The serving layer exposes the data to consumers.

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Legal and compliance considerations

Public data across automotive inventory sources is generally treated as fair to scrape in most jurisdictions, but always confine your collection to non-personal data: identifiers, prices, structured attributes, and aggregates. Avoid collecting personally identifying details, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Sample analytics queries

Once your snapshots are landing reliably, the analytics layer is where the value materializes:

-- Trend over the last 30 days
SELECT date_trunc('day', snapshot_at) AS day,
       COUNT(*) AS records,
       AVG(price) AS avg_price
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY 1
ORDER BY 1;

-- New entities first seen in the last 14 days
SELECT entity_id, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY entity_id
HAVING MIN(snapshot_at) > now() - interval '14 days';

Add a category share view, a source concentration view, and a price-volatility view and you have a solid foundation for a automotive inventory intelligence product.

Versioning your scraper for source evolution

Every dealership inventory source evolves its schema regularly. New fields appear, old fields are deprecated, and pricing display logic changes. Your scraper code has to evolve with these changes, and a versioning pattern that keeps old data interpretable is critical. Stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads. The collection layer handles the network conversation. The parsing layer transforms raw bytes into structured records. The storage layer holds the canonical snapshots in a query-optimized format. The serving layer exposes the data to consumers.

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same VIN within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output.

Building a regional dealer panel for ongoing intelligence

The most defensible analytical product on top of dealership data is a regional dealer panel: a curated set of 500-1,000 dealers across geographic and brand segments that you snapshot consistently every day. The panel approach has three advantages over scraping the entire universe.

First, panel-based intelligence has cleaner longitudinal continuity. The same set of dealers appearing in every snapshot lets you compute month-over-month and year-over-year changes without having to deduplicate against the broader population.

Second, the panel reduces proxy cost dramatically. Instead of scraping 50,000 dealer sites daily, you scrape 1,000 with high reliability. The cost reduction is 50x and the analytical signal is often stronger because the panel is balanced by region and brand.

Third, the panel approach lets you weight the results to match the underlying U.S. population of dealers. With known weights, you can produce population-level estimates from the panel that are more credible to downstream consumers than ad-hoc full-population scrapes that may have coverage gaps.

def weighted_panel_metric(panel_df, weights_df, metric_col):
    merged = panel_df.merge(weights_df, on=["region", "brand"])
    return (merged[metric_col] * merged["weight"]).sum() / merged["weight"].sum()

The panel design itself is the analytical asset. Document the inclusion criteria, the weighting scheme, and the refresh cadence in a methodology doc that you publish alongside the data product.

Common pitfalls when scraping dealership inventory

Three failure modes show up across nearly every dealership scraping project. The first is VIN duplication across dealer groups. A vehicle that moves between dealers in the same auto group keeps its VIN but appears under a new listing id. A scraper that deduplicates by listing id alone double-counts inventory. Always include VIN as a secondary deduplication key and reconcile at the VIN level downstream.

The second is incentive vs sticker price confusion. Manufacturer incentives, dealer cash, and trade-in bonuses are layered separately on the listing page. The MSRP, dealer_price, and out_the_door_price are three different numbers that can differ by $3,000-$8,000. Capture all three and let the analytics layer decide which is canonical for the question being answered.

The third is days-on-lot misattribution. Most platforms compute days-on-lot from the first scrape, not from the original listing date. A vehicle that was on the lot for 60 days before your scraper started shows up as a fresh listing. Backfill the original listing date from VIN history APIs (Carfax, AutoCheck) for vehicles you care about, or accept that the first 90 days of your dataset will undercount aging inventory.

FAQ

Are dealership inventory pages legal to scrape?
Vehicle inventory data is generally considered public commercial information. Aggregator sites have terms of service that often prohibit automated access; their enforcement focuses on commercial competitors. Confine your collection to non-personal data and document your lawful basis for processing.

What’s the typical refresh rate I should target?
Once daily catches the major price-change signals. Twice daily (early morning and late evening) catches dealer-set pricing changes that happen at the start and end of the business day. For real-time alerting on price changes, hourly is feasible but increases proxy costs significantly.

How should I handle private-seller listings on Craigslist or Facebook Marketplace?
Private-seller data has stronger personal-data implications. Limit collection to the listing description and price; avoid storing seller contact details. For most analytical use cases, dealer inventory alone provides sufficient signal without the privacy complexity of private-seller data.

Does CARFAX or AutoCheck data overlap with what I can scrape?
CARFAX and AutoCheck sell vehicle history reports per VIN and are not realistically scrapable. They are licensed data products. For analytics that need vehicle history, license the data; for analytics that focus on listing dynamics, the public listings are the relevant dataset.

Can I use the data to compute a fair-market-value model?
Yes. With 30+ days of daily snapshots across 100,000+ listings, you have enough data to fit a regression model for price as a function of make, model, year, mileage, region, and days-on-lot. KBB and Edmunds publish similar models commercially; building your own is a substantial undertaking but feasible.

How do I track price drops on a specific VIN over time?
Hash on VIN as the primary key and store every price observation as a row, not a column. This makes time-series queries trivial in any SQL backend.

Do dealer websites use dynamic pricing engines?
Increasingly yes. vAuto, Dealer.com, and several others reprice inventory automatically based on market data. Expect price changes within minutes of competitor movements on hot SKUs.

To build broader vertical scraping pipelines, browse the ecommerce scraping category for tooling reviews and framework deep dives.

How to scrape Takealot South Africa in 2026

Scrape Takealot South Africa and you tap into the dominant ecommerce platform in South Africa, owned by Naspers and the regional incumbent against which all other South African online retailers benchmark. Takealot operates a hybrid first-party and marketplace model with its own logistics network and the Mr D Food delivery sub-brand. The scraping landscape is shaped by three things: a JSON product API that powers the front end, a moderate Cloudflare layer that profiles non-South African traffic, and a relatively scrape-friendly architecture compared to Western marketplaces of similar scale.

This guide focuses on Takealot at takealot.com as the canonical example.

Mapping Takealot URL and JSON structure

Takealot product URLs follow the pattern https://www.takealot.com/<product-slug>/PLID<plid>. The trailing PLID (Product Listing ID) is the canonical SKU identifier. Behind every product page sits a JSON endpoint at https://api.takealot.com/rest/v-1-12-0/product-details/PLID<plid>. The endpoint returns price, stock, full description, images, seller information, and the offer stack.

import httpx

def api(plid: str) -> str:
    return f"https://api.takealot.com/rest/v-1-12-0/product-details/PLID{plid}"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-ZA,en;q=0.9",
}

async def fetch_takealot(plid: str, proxy: str):
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(api(plid))
        if r.status_code == 200:
            return r.json()
        return None

The response includes core (canonical product), buybox (the winning seller offer with price, stock, fulfillment), gallery (image set), attributes, variants, and reviews_summary. For most analytical use cases the API alone is sufficient and you do not need to fetch the rendered HTML.

South African proxy strategy

Takealot’s bot detection profiles visitor IP geography. South African residential or mobile IPs through Vodacom, MTN South Africa, or Cell C are strongly preferred for sustained scraping. Pan-African residential pools work for light loads but degrade at higher volumes. European residential pools work surprisingly well for short bursts because of historical CDN routing patterns, but the success rate degrades quickly under sustained load.

For workloads under 5,000 product reads per day, a small South African residential pool with sticky 15-minute sessions is sufficient. For higher volumes, dedicated South African mobile ports through Vodacom are the cleaner path.

Crawling the category tree

Takealot exposes a category tree at https://api.takealot.com/rest/v-1-12-0/category/all. Each category has a url_key and an id. The listing endpoint at https://api.takealot.com/rest/v-1-12-0/searches/products accepts category, sort, and pagination parameters, with practical limits of 100 pages of 36 products each.

async def crawl_category(category_slug: str, proxy_pool, max_pages: int = 100):
    results = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = "https://api.takealot.com/rest/v-1-12-0/searches/products"
        params = {
            "filter": f"Category:{category_slug}",
            "sort": "Relevance",
            "rows": 36,
            "page": page,
        }
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url, params=params)
            if r.status_code != 200:
                break
            results.extend(r.json().get("results", []))
    return results

For broader categories, decompose by brand or price-band facets exposed in the search response.

Buybox vs. all offers

Takealot follows the Amazon-style buybox model where one seller wins the default offer position on a product page. The product detail JSON exposes both the buybox winner and the full offer stack. For brand monitoring, the buybox tracking is the primary signal because it determines what most buyers see, but the full offer stack reveals the gray-market and parallel-import landscape.

Schema for Takealot snapshots

CREATE TABLE takealot_snapshot (
    snapshot_at TIMESTAMP NOT NULL,
    plid VARCHAR(16) NOT NULL,
    seller_id VARCHAR(64) NOT NULL,
    is_buybox BOOLEAN,
    price_zar DECIMAL(12,2),
    list_price_zar DECIMAL(12,2),
    in_stock BOOLEAN,
    fulfillment VARCHAR(32),
    PRIMARY KEY (snapshot_at, plid, seller_id)
);

For dynamic-pricing competitors, snapshot every 4-6 hours captures meaningful changes. For weekly category reports, daily is sufficient. Take care to preserve the buybox winner per snapshot so you can compute buybox-flip frequency, which is one of the most useful signals for sellers competing for placement.

Detecting and routing around CAPTCHA challenges on Takealot

When Takealot flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real South Africa residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle.

Working with ZAR pricing and FX normalization

Pricing on Takealot is denominated in ZAR, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local ZAR and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread.

Comparing Takealot to other regional marketplaces

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Takealot but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal.

Operational monitoring and alerting

Every production scraper needs three monitoring layers. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for South Africa

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but South Africa has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@flow(name="takealot-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    return [f.result() for f in futures]

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is.

Sample analytics queries

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days';

These queries power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product.

Building robust deduplication across noisy listings

The long-tail catalogue is full of near-duplicate listings. The standard deduplication approach uses a three-pass funnel: exact match on EAN, normalized title plus brand TF-IDF similarity, then perceptual image hash similarity.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge or under-merge.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp using freshness deadline, volatility, and business priority signals to decide what to refetch on each cycle. Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads.

The collection layer handles the network conversation: HTTP requests, proxy assignment, retry logic, and rate limit enforcement. It should know nothing about the data shape and nothing about how the data will eventually be queried. Its only job is to fetch raw bytes reliably and hand them off to the next layer with metadata about which IP, which user agent, and which timestamp produced them.

The parsing layer transforms raw bytes into structured records. It owns the schema, the field normalization, and the validation rules. When the upstream HTML or JSON structure changes, only the parsing layer needs to adapt. Keep parsers idempotent and version them aggressively so old raw bytes can be re-parsed when you discover bugs.

The storage layer holds the canonical snapshots in a query-optimized format. For most ecommerce datasets, a column-oriented store like DuckDB, ClickHouse, or BigQuery outperforms row-oriented Postgres at analytical scale. The trade-off is write latency and update support; column stores prefer append-only and bulk loads, which fits the snapshot model naturally.

The serving layer exposes the data to consumers, whether that is a BI dashboard, an API for downstream systems, or an alerting pipeline. Keep the serving layer denormalized and pre-aggregated where possible. Recomputing complex analytics on every dashboard load wastes resources and hurts responsiveness.

# Pseudo-code for the four-layer split
async def collect(url: str, proxy_pool) -> RawFetch:
    proxy = proxy_pool.next()
    response = await http_get(url, proxy)
    return RawFetch(url=url, body=response.text, fetched_at=now(), ip=proxy.ip)

def parse(raw: RawFetch) -> Snapshot:
    data = json.loads(raw.body)
    return Snapshot(sku=data["id"], price=data["price"], ...)

def store(snapshot: Snapshot, db) -> None:
    db.append("snapshots", snapshot)

def serve(query: str, db) -> list:
    return db.query(query)

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. The checks fall into three categories: structural, distributional, and semantic.

Structural checks verify that every required field is present and of the expected type. A snapshot row missing the price field is not a real snapshot. A row with a negative price is not a real price.

Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer SKUs than yesterday, something broke either in collection or in the upstream catalogue. Either way, the on-call engineer needs to investigate before downstream consumers see broken data.

Semantic checks compare related fields for consistency. If a SKU shows in_stock = true but stock_quantity = 0, one of the fields is wrong. If the discount percentage is computed from list_price and selling_price, the computed value should match the stated discount field.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    invalid = [r for r in snapshot if r.get("price", -1) < 0]
    if invalid:
        errors.append(f"{len(invalid)} rows have invalid price")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same SKU within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

Common pitfalls when scraping Takealot

Three issues catch most teams. The first is plid vs tsin confusion. Takealot uses plid (product listing id) and tsin (Takealot stock-keeping number) interchangeably in URLs and APIs. The plid identifies the product page; the tsin identifies a specific variant. Joining datasets on the wrong key collapses variants into the parent product and loses color/size pricing.

The second is Daily Deals vs Blue Dot Sale staleness. Takealot’s headline promotions expire on a fixed cadence but the cached product detail JSON can lag by 5-15 minutes after expiry. A snapshot taken at the boundary captures a price that is no longer purchasable. Validate active promotions by cross-checking the promotion_end_time epoch against the scrape timestamp.

The third is third-party seller marketplace dilution. Takealot’s marketplace lets third-party sellers list against the same parent listing. The Buy Box price can flip between Takealot first-party and a marketplace seller within minutes. Capture buy_box_seller on every snapshot or your time series will look noisier than the underlying market is.

FAQ

Is the Takealot API officially documented?
The api.takealot.com endpoints are the same endpoints used by the public web site. They have been stable for several years but are not contractually supported.

Can I scrape Takealot from European or US IPs?
For light occasional reads, yes. For sustained scraping, Takealot blocks non-South African IPs after a few hours of activity. South African residential or mobile IPs are strongly preferred.

Does Takealot expose stock counts in the API?
The API returns availability boolean and a low-stock indicator but not exact stock counts for most SKUs. For SKUs with very low stock (under 5 units), Takealot sometimes shows the exact count in the buybox response.

How does Takealot handle the Mr D Food sub-brand?
Mr D Food uses a separate API surface focused on hyperlocal restaurant delivery. The patterns here apply to the main Takealot retail catalogue. Plan for a separate code path if your project covers Mr D.

What about Takealot’s marketplace seller restrictions?
Takealot vets marketplace sellers and has different fulfillment options (FBT for Fulfilled by Takealot, FBM for Fulfilled by Merchant). The fulfillment field in the API exposes which option each seller uses, which matters for delivery promise analytics.

Does Takealot block non-South African IPs?
Casual lookups succeed from most regions. Sustained scraping at production volume requires South African residential or mobile IPs. JNB and CPT proxies perform best in our testing.

How do I separate Takealot’s first-party stock from marketplace stock?
The merchant_id field identifies Takealot’s house merchant (typically id 1) versus third-party sellers. Filter on merchant_id == 1 to isolate first-party stock for retail analytics.

To build a broader South Africa ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives.

How to scrape Konga Nigeria product listings

Scrape Konga Nigeria and you tap into the second-largest general-merchandise marketplace in Nigeria, alongside Jumia. Konga has shifted ownership multiple times since its founding in 2012 and has rebuilt its catalogue and logistics under each phase. By 2026 the platform serves a strong base in electronics, fashion, and groceries with a hybrid first-party and third-party seller model. The scraping landscape is shaped by three things: a JSON catalogue API that powers the front end, an aggressive Cloudflare front end that profiles non-Nigerian traffic strictly, and a SKU schema that overlaps partially with Jumia for cross-marketplace analytics.

This guide focuses on Konga at konga.com as the canonical example. The patterns also apply to KongaPay merchant subsystems with minor adjustments.

Mapping Konga URL and JSON structure

Konga product URLs follow the pattern https://www.konga.com/product/<product-slug>-<productId>. The trailing productId is the canonical SKU. Behind every product page sits a JSON endpoint at https://api.konga.com/v1/catalog/products/sku/<productId>. The endpoint returns price, stock, full description, images, and seller information.

import httpx

API = "https://api.konga.com/v1/catalog/products/sku"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-NG,en;q=0.9",
}

async def fetch_konga(sku: str, proxy: str):
    url = f"{API}/{sku}"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return None

The response includes the canonical product object with sku, name, brand, original_price, special_price, stock_status, seller_name, seller_id, categories, and rating_summary. For most analytical use cases the API is sufficient.

Nigerian proxy strategy

Konga’s Cloudflare front end is aggressive against non-Nigerian IPs. Nigerian residential or mobile IPs through MTN, Airtel, 9mobile, or Glo are required for sustained scraping. Pan-African residential pools work for light loads but degrade at volume.

For workloads under 2,000 product reads per day, a small Nigerian residential pool with sticky 15-minute sessions is sufficient. For higher volumes, dedicated Nigerian mobile ports are the cleaner path because they sustain higher request rates without challenges.

Crawling the category tree

Konga exposes a category tree at https://api.konga.com/v1/catalog/categories. Each category has a url_key and an id. The listing endpoint at https://api.konga.com/v1/catalog/products accepts category, sort, and pagination parameters, with practical limits of 25 pages of 40 products each.

async def crawl_category(category_id: int, proxy_pool, max_pages: int = 25):
    results = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = "https://api.konga.com/v1/catalog/products"
        params = {
            "category_id": category_id,
            "page": page,
            "limit": 40,
            "sort": "relevance",
        }
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url, params=params)
            if r.status_code != 200:
                break
            items = r.json().get("products", [])
            if not items:
                break
            results.extend(items)
    return results

For broader categories like Phones and Tablets, decompose by brand and price band facets exposed in the search response.

Cross-checking Konga pricing against Jumia

The most analytically interesting Nigerian ecommerce signal is the price differential between Konga and Jumia for the same SKU. They overlap heavily on consumer electronics, household goods, and fashion. For brand monitoring, the price gap often signals which platform is running a promotion or which platform a particular seller is using as the price-leader channel.

For SKU matching across the two platforms, group by EAN where available, then by normalized title plus brand for the long tail. Match rates on Nigerian ecommerce specifically tend to be lower than other markets because EAN coverage is voluntary and many sellers use product names that emphasize Nigerian-specific marketing claims.

Detecting and routing around CAPTCHA challenges on Konga

When Konga flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Nigeria residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle.

Working with NGN pricing and FX normalization

Pricing on Konga is denominated in NGN, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local NGN and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread.

Comparing Konga to other regional marketplaces

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Konga but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal.

Operational monitoring and alerting

Every production scraper needs three monitoring layers. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for Nigeria

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Nigeria has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@flow(name="konga-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    return [f.result() for f in futures]

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is.

Sample analytics queries

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days';

These queries power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product.

Building robust deduplication across noisy listings

The long-tail catalogue is full of near-duplicate listings. The standard deduplication approach uses a three-pass funnel: exact match on EAN, normalized title plus brand TF-IDF similarity, then perceptual image hash similarity.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge or under-merge.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp using freshness deadline, volatility, and business priority signals to decide what to refetch on each cycle. Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads.

The collection layer handles the network conversation: HTTP requests, proxy assignment, retry logic, and rate limit enforcement. It should know nothing about the data shape and nothing about how the data will eventually be queried. Its only job is to fetch raw bytes reliably and hand them off to the next layer with metadata about which IP, which user agent, and which timestamp produced them.

The parsing layer transforms raw bytes into structured records. It owns the schema, the field normalization, and the validation rules. When the upstream HTML or JSON structure changes, only the parsing layer needs to adapt. Keep parsers idempotent and version them aggressively so old raw bytes can be re-parsed when you discover bugs.

The storage layer holds the canonical snapshots in a query-optimized format. For most ecommerce datasets, a column-oriented store like DuckDB, ClickHouse, or BigQuery outperforms row-oriented Postgres at analytical scale. The trade-off is write latency and update support; column stores prefer append-only and bulk loads, which fits the snapshot model naturally.

The serving layer exposes the data to consumers, whether that is a BI dashboard, an API for downstream systems, or an alerting pipeline. Keep the serving layer denormalized and pre-aggregated where possible. Recomputing complex analytics on every dashboard load wastes resources and hurts responsiveness.

# Pseudo-code for the four-layer split
async def collect(url: str, proxy_pool) -> RawFetch:
    proxy = proxy_pool.next()
    response = await http_get(url, proxy)
    return RawFetch(url=url, body=response.text, fetched_at=now(), ip=proxy.ip)

def parse(raw: RawFetch) -> Snapshot:
    data = json.loads(raw.body)
    return Snapshot(sku=data["id"], price=data["price"], ...)

def store(snapshot: Snapshot, db) -> None:
    db.append("snapshots", snapshot)

def serve(query: str, db) -> list:
    return db.query(query)

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. The checks fall into three categories: structural, distributional, and semantic.

Structural checks verify that every required field is present and of the expected type. A snapshot row missing the price field is not a real snapshot. A row with a negative price is not a real price.

Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer SKUs than yesterday, something broke either in collection or in the upstream catalogue. Either way, the on-call engineer needs to investigate before downstream consumers see broken data.

Semantic checks compare related fields for consistency. If a SKU shows in_stock = true but stock_quantity = 0, one of the fields is wrong. If the discount percentage is computed from list_price and selling_price, the computed value should match the stated discount field.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    invalid = [r for r in snapshot if r.get("price", -1) < 0]
    if invalid:
        errors.append(f"{len(invalid)} rows have invalid price")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same SKU within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

Common pitfalls when scraping Konga

Three issues recur. The first is KongaPay-financing price masking. Konga shows the headline price on the product card and a financed monthly figure on the detail page. Some scrapers extract the monthly figure and treat it as the SKU price, which understates the cash price by an order of magnitude. Always pull the selling_price field from the product JSON, not from rendered HTML.

The second is seller-type confusion. Konga operates a first-party (Konga Retail) and a third-party seller pool on the same SKU. Pricing dynamics differ: first-party prices are stable for weeks, third-party prices reprice daily. Aggregating both into one population smears the trend signal. Segment by seller_type before computing any time-series metric.

The third is FX-driven price stale state. The Naira moves frequently against the USD. Sellers reprice imported electronics on a 1-3 day lag after FX shocks. A scraper that stores only the local-currency price loses the FX-attribution signal. Capture the scrape-time FX rate from a stable reference (CBN or Bloomberg) and compute USD-equivalent prices in a derived column.

FAQ

Is the Konga API officially documented?
The api.konga.com endpoints are the same endpoints used by the public web site. They have been stable for several years but are not contractually supported.

Can I scrape Konga from a UK or US residential IP?
For occasional product lookups, yes. For sustained scraping, Konga blocks non-Nigerian IPs aggressively. Nigerian residential or mobile IPs are strongly preferred.

Does Konga distinguish between first-party and third-party sellers?
Yes. The seller_id and seller_name fields identify the seller. KongaCare-flagged sellers receive verification badges. For analytics, separating first-party Konga listings from third-party sellers is essential because the price dynamics differ.

How does Konga handle the Naira’s frequent revaluation?
Konga prices update with the underlying Naira movements, but lag behind FX shocks by 24-72 hours as sellers reprice. For longitudinal analyses, normalize prices to a stable currency using daily reference FX rates.

What about Konga’s logistics promise vs. actual delivery?
The product page shows the logistics promise but the API does not expose delivery success rates. For analytics that need to assess seller delivery reliability, you have to scrape buyer reviews and aggregate the delivery-related sentiment, which requires more text processing.

Are there days when Konga rate-limits scrapers more aggressively?
Yes. Black Friday week (late November) and December gift season (Dec 15-24) carry the tightest rate limits. Pre-stage baseline snapshots in October to avoid being throttled during the actual event.

How do I match Konga SKUs to Jumia for cross-marketplace analytics?
Use brand + model + storage capacity as the canonical join. Konga’s GTIN coverage is sparse, so fuzzy title matching with a similarity threshold of 0.85 is the practical fallback.

Does Konga’s mobile app expose a different API surface than the web site?
Yes. The Android app uses protobuf endpoints under mobileapi.konga.com for personalization, while the public web JSON layer under api.konga.com covers catalog and pricing. For catalog and pricing intelligence the web endpoints are sufficient and easier to maintain. Reverse-engineering the protobuf surface only pays back when the analytics design requires the personalization signal.

To build a broader Nigeria ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives.

How to scrape Tiki Vietnam ecommerce

Scrape Tiki Vietnam and you tap into one of the three dominant ecommerce platforms in Vietnam, alongside Shopee Vietnam and Lazada Vietnam. Tiki was founded in 2010 and has built a strong reputation around fast 2-hour delivery (TikiNow) in major cities and a curated catalogue that leans into electronics, books, and household goods. The scraping landscape is shaped by three things: a publicly accessible JSON API that powers most of the site, Vietnamese-language content with Latin characters but heavy diacritics, and a moderate Cloudflare front end that profiles non-Vietnamese traffic.

This guide focuses on Tiki at tiki.vn as the canonical example. The patterns transfer to TikiNow city-specific catalogues with minor adjustments.

Mapping Tiki URL and JSON structure

Tiki product URLs follow the pattern https://tiki.vn/<product-slug>-p<productId>.html. The trailing productId (an integer) is the canonical SKU identifier. Behind every product page sits a JSON endpoint at https://tiki.vn/api/v2/products/<productId>. The endpoint accepts query parameters for store_id and platform and returns price, stock, full description, images, and seller information.

import httpx

API = "https://tiki.vn/api/v2/products"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15",
    "Accept": "application/json",
    "Accept-Language": "vi-VN,vi;q=0.9,en;q=0.8",
}

async def fetch_tiki(product_id: int, proxy: str):
    url = f"{API}/{product_id}"
    params = {"platform": "web", "spid": ""}
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url, params=params)
        if r.status_code == 200:
            return r.json()
        return None

The response includes id, name, price, original_price, discount_rate, stock_item.qty, inventory_status, seller.id, seller.name, brand.name, categories, and rating_average. For most analytical use cases the API alone is sufficient.

Vietnamese proxy strategy

Tiki bot detection profiles visitor IP at the country level. For light scraping under 5,000 product reads per day, clean Asian datacenter IPs from Singapore or Tokyo work. For higher volumes, Vietnamese residential IPs through Viettel, VNPT, or FPT Telecom dramatically improve success rates and avoid Cloudflare interstitials.

For full catalogue sweeps, dedicated Vietnamese mobile inventory pays for itself. The cost differential against pan-Asian residential is meaningful but the success rate at scale is significantly higher.

Crawling the category tree

Tiki exposes a category tree at https://api.tiki.vn/raiden/v2/menu-config. Each category has a url_key and an id that you can use to query the listing API at https://tiki.vn/api/personalish/v1/blocks/listings. Pagination uses page and limit query parameters, with practical limits of 50 pages of 50 products each.

async def crawl_category(category_id: int, proxy_pool, max_pages: int = 50):
    results = []
    for page in range(1, max_pages + 1):
        proxy = proxy_pool.next()
        url = "https://tiki.vn/api/personalish/v1/blocks/listings"
        params = {
            "limit": 50,
            "page": page,
            "category": category_id,
            "aggregations": 2,
        }
        async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
            r = await c.get(url, params=params)
            if r.status_code != 200:
                break
            items = r.json().get("data", [])
            if not items:
                break
            results.extend(items)
    return results

For deeper coverage of large categories, decompose by brand or price band facets. The aggregations response includes the available facets and counts per facet.

Handling Vietnamese diacritics

Vietnamese product names use Latin characters with extensive diacritics. Always store the original UTF-8 text without normalizing diacritics, because diacritic stripping changes the meaning of many words and breaks brand matches. For full-text search, use Postgres with the unaccent extension or Elasticsearch with the Vietnamese analyzer, which both handle diacritic-aware matching correctly.

Cross-checking Tiki pricing against Shopee Vietnam

The most analytically interesting Vietnamese ecommerce signal is the price differential between Tiki and Shopee for the same SKU. They overlap heavily on electronics, beauty, and household goods. For brand monitoring, the price gap on a given SKU often signals which platform is running a flash promotion at any given time.

For SKU matching across platforms, group by EAN where available and by normalized title plus brand for the long tail.

Detecting and routing around CAPTCHA challenges on Tiki

When Tiki flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Vietnam residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle.

Working with VND pricing and FX normalization

Pricing on Tiki is denominated in VND, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local VND and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread.

Comparing Tiki to other regional marketplaces

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on Tiki but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal.

Operational monitoring and alerting

Every production scraper needs three monitoring layers. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails.

Legal and compliance considerations for Vietnam

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Vietnam has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@flow(name="tiki-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    return [f.result() for f in futures]

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is.

Sample analytics queries

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days';

These queries power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product.

Building robust deduplication across noisy listings

The long-tail catalogue is full of near-duplicate listings. The standard deduplication approach uses a three-pass funnel: exact match on EAN, normalized title plus brand TF-IDF similarity, then perceptual image hash similarity.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge or under-merge.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp using freshness deadline, volatility, and business priority signals to decide what to refetch on each cycle. Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

End-to-end pipeline architecture

A production-grade scraping pipeline has four layers that work together: collection, parsing, storage, and serving. Each layer has its own failure modes and its own scaling characteristics, and treating them as a single monolith is the most common architectural mistake teams make when scaling beyond hobby workloads.

The collection layer handles the network conversation: HTTP requests, proxy assignment, retry logic, and rate limit enforcement. It should know nothing about the data shape and nothing about how the data will eventually be queried. Its only job is to fetch raw bytes reliably and hand them off to the next layer with metadata about which IP, which user agent, and which timestamp produced them.

The parsing layer transforms raw bytes into structured records. It owns the schema, the field normalization, and the validation rules. When the upstream HTML or JSON structure changes, only the parsing layer needs to adapt. Keep parsers idempotent and version them aggressively so old raw bytes can be re-parsed when you discover bugs.

The storage layer holds the canonical snapshots in a query-optimized format. For most ecommerce datasets, a column-oriented store like DuckDB, ClickHouse, or BigQuery outperforms row-oriented Postgres at analytical scale. The trade-off is write latency and update support; column stores prefer append-only and bulk loads, which fits the snapshot model naturally.

The serving layer exposes the data to consumers, whether that is a BI dashboard, an API for downstream systems, or an alerting pipeline. Keep the serving layer denormalized and pre-aggregated where possible. Recomputing complex analytics on every dashboard load wastes resources and hurts responsiveness.

# Pseudo-code for the four-layer split
async def collect(url: str, proxy_pool) -> RawFetch:
    proxy = proxy_pool.next()
    response = await http_get(url, proxy)
    return RawFetch(url=url, body=response.text, fetched_at=now(), ip=proxy.ip)

def parse(raw: RawFetch) -> Snapshot:
    data = json.loads(raw.body)
    return Snapshot(sku=data["id"], price=data["price"], ...)

def store(snapshot: Snapshot, db) -> None:
    db.append("snapshots", snapshot)

def serve(query: str, db) -> list:
    return db.query(query)

Decoupling these layers also enables independent scaling. The collection layer is bound by proxy capacity and network bandwidth. The parsing layer is CPU-bound. The storage layer is bound by I/O and disk capacity. The serving layer is bound by query concurrency. Each layer can scale horizontally without coupling to the others.

Data quality monitoring patterns

Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. The checks fall into three categories: structural, distributional, and semantic.

Structural checks verify that every required field is present and of the expected type. A snapshot row missing the price field is not a real snapshot. A row with a negative price is not a real price.

Distributional checks compare the current snapshot against recent history. If today’s snapshot has 30% fewer SKUs than yesterday, something broke either in collection or in the upstream catalogue. Either way, the on-call engineer needs to investigate before downstream consumers see broken data.

Semantic checks compare related fields for consistency. If a SKU shows in_stock = true but stock_quantity = 0, one of the fields is wrong. If the discount percentage is computed from list_price and selling_price, the computed value should match the stated discount field.

def quality_check(snapshot: list[dict]) -> list[str]:
    errors = []
    if not snapshot:
        errors.append("empty snapshot")
        return errors
    avg_yesterday = get_yesterday_avg_size()
    if len(snapshot) < avg_yesterday * 0.7:
        errors.append(f"snapshot size {len(snapshot)} is 30% below yesterday")
    invalid = [r for r in snapshot if r.get("price", -1) < 0]
    if invalid:
        errors.append(f"{len(invalid)} rows have invalid price")
    return errors

Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review, not silently published.

Cost optimization strategies

Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: if two consumers ask for the same SKU within the same hour, the system should serve the cached response rather than refetching. The second is conditional GET: when the upstream supports ETag or If-Modified-Since headers, conditional requests transfer no body when the resource has not changed. The third is selective field hydration: when the upstream API supports field selection, requesting only the fields you need reduces payload size dramatically.

For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort to implement them is modest and the payback period is usually under a month at production volume.

Common pitfalls when scraping Tiki

Three issues recur on Tiki scrapers. The first is TikiNOW vs Marketplace separation. TikiNOW (2-hour delivery in HCMC and Hanoi) and Tiki Marketplace (multi-day fulfillment elsewhere) coexist on the same product page. The is_tikinow flag and seller_id distinguish them. For delivery-speed analytics, segment by both fields.

The second is Vietnamese diacritic normalization. Product titles include Vietnamese diacritics (e.g., điện thoại). Some pipelines strip diacritics for ASCII compatibility, which collapses distinct Vietnamese words into the same key and corrupts brand-level aggregates. Preserve diacritics end-to-end and only strip them at the final analytics layer if your downstream BI tool cannot handle UTF-8.

The third is VAT inclusion drift. Tiki shows VAT-inclusive prices on most SKUs but VAT-exclusive prices for some B2B-flagged SKUs. The vat_included field on the API response tells you which is which. Naive scrapers store the raw price and downstream analytics compare apples to oranges.

FAQ

Is the Tiki API officially documented?
The endpoints described here are the unauthenticated APIs that the public web site uses. They have been stable for several years but they are not contractually supported. Build defensively with schema drift alerts.

Can I scrape Tiki from Singapore or Hong Kong residential IPs?
Yes for light loads under 5,000 product reads per day. For sustained scraping at higher volumes, Vietnamese IPs are strongly preferred because they sustain higher request rates without challenges.

Does Tiki distinguish between Tiki Trading and third-party sellers?
Yes. The seller.id field identifies the seller; sellers with id 1 are Tiki’s own first-party Trading. Third-party sellers have their own IDs. For analytics, separating first-party from third-party listings is essential because the price dynamics and turnover characteristics differ significantly.

How does TikiNow city-specific delivery affect the data?
TikiNow availability depends on the destination postcode. The API returns a generic stock value but the actual same-day delivery promise is computed against the buyer location. For most analytical use cases, treat the stock value as the canonical inventory signal.

What about Vietnamese tax and VAT in pricing?
Tiki prices are displayed VAT-inclusive at 10%. The displayed price is what the buyer pays. For brand teams comparing against MAP policies set in net-of-tax terms, you need to back out the VAT to align with the brand reference price.

Does Tiki publish an official API?
Tiki Open Platform serves sellers, not analysts. For market-intelligence work, public-page scraping is the operational path. The endpoints under tiki.vn/api/v2/ are stable in practice.

How do I track Tiki’s flash sales accurately?
The flash_sale block carries start_time and end_time epochs. Sample at 5-15 minute intervals during the active window to capture stock burndown.

To build a broader Vietnam ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives.

How to scrape PChome Taiwan product data

Scrape PChome Taiwan and you are scraping the dominant general-merchandise marketplace in Taiwan, famous for its 24-hour delivery service across the island and operated by PChome Online Inc. The platform is the largest player in Taiwanese B2C ecommerce by GMV, ahead of Momoshop and Yahoo Shopping Taiwan, and it has a relatively scrape-friendly architecture for a marketplace of its size. The scraping landscape is shaped by three things: a publicly available JSON API that powers the front end, a category structure that maps cleanly to URL paths, and a Cloudflare front end that profiles non-Taiwanese traffic less aggressively than most Asian marketplaces.

This guide covers PChome 24h (the express-delivery property at 24h.pchome.com.tw) and notes where the patterns extend to PChome Shopping Mall (shopping.pchome.com.tw).

Mapping PChome URL and JSON structure

PChome 24h product URLs follow the pattern https://24h.pchome.com.tw/prod/<productCode>. The productCode is a 12-character alphanumeric identifier that is the canonical SKU. Behind every product page sits a JSON endpoint at https://ecapi.pchome.com.tw/ecshop/prodapi/v2/prod/<productCode>. The endpoint returns price, stock, full description, images, and category path in a single response.

import httpx

API = "https://ecapi.pchome.com.tw/ecshop/prodapi/v2/prod"

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "application/json",
    "Accept-Language": "zh-TW,zh;q=0.9",
}

async def fetch_pchome(product_code: str, proxy: str):
    url = f"{API}/{product_code}"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return None

The response includes Id (productCode), Name, Brand, Price.P (current price in TWD), Price.M (member price), Stock, Slogan, Description, Pic (image filenames), and Cate (category path as a list). For most analytical use cases the API is sufficient and you do not need to render the HTML.

Taiwanese proxy strategy

PChome’s bot detection is moderate compared to other regional marketplaces. The site profiles visitor IP at the country level but does not aggressively block non-Taiwan traffic for the API endpoints. For light scraping (under 5,000 product reads per day), a clean datacenter IP from a Tokyo or Singapore region works. For higher volumes, Taiwanese residential or mobile IPs through Chunghwa Telecom or Far EasTone produce dramatically better success rates and avoid the occasional Cloudflare interstitial.

For full catalogue sweeps that touch hundreds of thousands of SKUs per day, dedicated Taiwan inventory pays for itself. The cost differential against generic Asian residential is meaningful but the success rate differential at scale is larger.

Crawling the category tree

PChome exposes the category tree at https://ecapi.pchome.com.tw/cdn/ecshop/prodapi/v2/cateinfo/<rootCateId>/category. Categories are organized as a six-level deep hierarchy. The API returns child nodes with their own categoryId values that you can recursively walk to enumerate the full tree.

async def fetch_category(cate_id: str, proxy: str):
    url = f"https://ecapi.pchome.com.tw/cdn/ecshop/prodapi/v2/cateinfo/{cate_id}/category"
    async with httpx.AsyncClient(proxy=proxy, headers=HEADERS, timeout=20) as c:
        r = await c.get(url)
        if r.status_code == 200:
            return r.json()
        return []

For each leaf category, the listing endpoint at https://ecapi.pchome.com.tw/ecshop/prodapi/v2/cateprod/<cateId>/prod returns the products in that category. Pagination uses a start and rows parameter, with practical limits of 1,000 products per category. For broader categories, decompose by brand or price band facet.

Working with Traditional Chinese text

PChome catalogue text is in Traditional Chinese. If your downstream pipeline expects Simplified Chinese, plan a conversion step using the OpenCC library. Conversion is generally lossless from Traditional to Simplified for product catalogue text, though edge cases (region-specific brand names, technical jargon) sometimes need a dictionary override.

from opencc import OpenCC

cc = OpenCC("t2s")  # traditional to simplified
simplified_title = cc.convert(traditional_title)

For full-text search across the dataset, both Postgres and Elasticsearch handle Chinese text well with the right analyzer. Use the chinese text search configuration in Postgres or the IK analyzer in Elasticsearch. Avoid space-tokenized full-text search because Chinese has no word boundaries and naive tokenization fails badly.

Cross-checking PChome pricing against Momoshop

The most analytically interesting Taiwan ecommerce signal is the price differential between PChome and Momoshop, the two dominant marketplaces, for the same SKU. They overlap heavily on electronics, home appliances, and beauty, and the price gap on a given SKU often signals which platform is running a promotion at any given time.

To match SKUs across platforms, group by EAN or by normalized title plus brand. The match rate is roughly 60% on EAN and an additional 20% on title-plus-brand, leaving a long tail of platform-exclusive SKUs.

For brand monitoring use cases, build a master SKU table that joins PChome productCode and Momoshop i_code via EAN where available. For SKUs without EAN matches, run a daily fuzzy-matching job and surface the matches for human review.

Detecting and routing around CAPTCHA challenges on PChome

When PChome flags your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment.... Treat any of these as a soft block.

def is_challenged(response) -> bool:
    if response.status_code in (403, 503):
        return True
    if "cf-mitigated" in response.headers:
        return True
    if "__cf_chl_" in response.headers.get("set-cookie", ""):
        return True
    body = response.text[:2000].lower()
    return "just a moment" in body or "checking your browser" in body

When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend and can lead to long-term blacklisting of your subnet. For pages that absolutely must be fetched, have a fallback path that uses a headless browser with a real Taiwan residential IP. The browser path costs more per page but solves the small percentage of challenges that the API path cannot handle. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.

Working with TWD pricing and FX normalization

Pricing on PChome is denominated in TWD, and any cross-market analysis requires careful FX normalization. The naive approach of converting at scrape time using a live FX feed introduces noise into your trend lines because exchange rate movements get conflated with real price changes. Store the price in local TWD and apply FX conversion at query time using a daily reference rate.

CREATE TABLE fx_rates (
    rate_date DATE NOT NULL,
    base_ccy VARCHAR(3) NOT NULL,
    quote_ccy VARCHAR(3) NOT NULL,
    rate DECIMAL(18,8) NOT NULL,
    PRIMARY KEY (rate_date, base_ccy, quote_ccy)
);

Source the daily rates from a reliable feed such as the European Central Bank reference rates or your bank wholesale feed. Avoid scraping retail FX rates because they include the bank spread and produce inconsistent comparisons. For analyses that span multiple years, also account for currency revaluation events that occasionally happen in emerging markets.

Comparing PChome to other regional marketplaces

Cross-marketplace analyses help separate platform-specific dynamics from genuine market trends. If a price drops on PChome 24h but stays flat across the comparable competitors, that is a platform-driven event rather than a market-wide signal. Your scraping pipeline should ingest from at least three platforms in any market where you intend to publish category insights.

Operational monitoring and alerting

Every production scraper needs three monitoring layers regardless of target. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.

import time
from collections import deque

class IPHealthTracker:
    def __init__(self, window_seconds: int = 300):
        self.window = window_seconds
        self.events = {}

    def record(self, ip: str, success: bool):
        bucket = self.events.setdefault(ip, deque())
        now = time.time()
        bucket.append((now, success))
        while bucket and bucket[0][0] < now - self.window:
            bucket.popleft()

    def success_rate(self, ip: str) -> float:
        bucket = self.events.get(ip)
        if not bucket:
            return 1.0
        successes = sum(1 for _, ok in bucket if ok)
        return successes / len(bucket)

Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations against PChome, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.

Legal and compliance considerations for Taiwan

Public product, price, and availability data are generally treated as fair to scrape in most jurisdictions, but Taiwan has its own consumer protection and personal data frameworks. Confine your collection to non-personal data: SKU identifiers, prices, descriptions, ratings as aggregates, and seller display names. Avoid collecting individual buyer reviews with names, phone numbers, or email addresses attached, and avoid pulling any data behind a login.

For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. Most data protection regimes treat scraped public data more favorably when there is a clear lawful basis and the data is not used for direct marketing to identified individuals. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.

Pipeline orchestration and scheduling

For any non-trivial scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster. Both handle the patterns you need: DAG dependencies, retries, observability, secret management, and dynamic fan-out across IPs and categories.

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch_category(category_id: int, page: int):
    return crawl_one_page(category_id, page)

@task
def store_pages(pages: list):
    write_to_db(pages)

@flow(name="pchome-daily-sweep")
def daily_sweep(category_ids: list):
    futures = []
    for cid in category_ids:
        for page in range(1, 50):
            futures.append(fetch_category.submit(cid, page))
    pages = [f.result() for f in futures]
    store_pages(pages)

Run the flow on a 6-hour or 24-hour schedule depending on how dynamic the underlying catalogue is. For seasonal markets like apparel where pricing changes daily, a 6-hour cadence catches the meaningful movements without driving up proxy costs unnecessarily. For long-tail categories like books or industrial supplies, daily is sufficient and the cost saving is meaningful.

Sample analytics queries on the collected dataset

Once your snapshots are landing reliably, the analytics layer is where the value materializes. A few queries that consistently come up across PChome datasets:

-- Top 50 SKUs by price drop in the last 7 days
SELECT sku, MIN(selling_price) - MAX(selling_price) AS price_drop
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY sku
ORDER BY price_drop ASC
LIMIT 50;

-- Stock-out frequency per category
SELECT category_id,
       SUM(CASE WHEN in_stock = 0 THEN 1 ELSE 0 END)::float / COUNT(*) AS oos_rate
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY category_id
ORDER BY oos_rate DESC;

-- New SKUs first seen in the last 14 days
SELECT sku, MIN(snapshot_at) AS first_seen
FROM snapshot
GROUP BY sku
HAVING MIN(snapshot_at) > now() - interval '14 days'
ORDER BY first_seen DESC;

These three queries alone power most of the dashboards a category manager wants. Add a brand share view, a seller concentration view, and a campaign-frequency view and you have a competitive intelligence product. The collection layer is the prerequisite; the analytics layer is where you create defensible value.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks.

The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups).

Versioning your scraper for catalogue evolution

Every ecommerce site evolves its catalogue structure regularly. Stamp every snapshot row with the scraper version that produced it. When you deploy a new version of the parser, increment the version number. Downstream analytics can filter by version when they need consistent semantics across a time range, or join across versions when they want long-running trend analysis.

Building robust deduplication across noisy listings

When you scrape any ecommerce marketplace at scale, the long-tail catalogue is full of near-duplicate listings. The same physical product appears under different titles, different sellers, slight variations in pack size, and slightly different image sets. For analytics that try to compute brand share or category trends, deduplication is mandatory and it is harder than it looks. The standard approach uses a three-pass funnel. The first pass groups by exact match on EAN or GTIN where present. The second pass groups by normalized title plus brand using a TF-IDF cosine similarity threshold of 0.85. The third pass groups by image hash similarity using perceptual hashing.

import imagehash
from PIL import Image

def perceptual_hash(image_path: str) -> str:
    img = Image.open(image_path)
    return str(imagehash.phash(img, hash_size=16))

Tune the similarity thresholds against a hand-labeled gold set of 500 to 1,000 known duplicate clusters. Without a gold set, you will either over-merge (collapsing distinct variants) or under-merge (leaving the same product in many groups). Both failure modes break downstream analytics in subtle ways that take weeks to detect.

Caching strategy and incremental crawls

Full daily snapshots scale linearly with catalogue size, which becomes expensive at multi-million SKU scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp. The pattern uses three signals to decide what to refetch on each cycle. The first signal is freshness deadline. Every SKU has a maximum staleness budget, and any SKU older than its budget gets refreshed. The second signal is volatility. SKUs that have changed price recently get higher refresh priority. The third signal is business priority. SKUs that downstream users actually query get higher refresh priority than dormant SKUs that nobody has looked at in months.

This kind of priority-driven scheduler reduces total request volume by 60-80% compared to blind full snapshots, while keeping the data fresh on the SKUs that actually matter to the business.

Common pitfalls when scraping PChome 24h

Three issues dominate PChome scraping. The first is the prod ID vs SKU ID confusion. PChome uses a prod_id for the product page and a separate sku_id for variants like color and capacity. The same prod_id can have 3-10 sku_ids with different prices. Naive scrapers join on prod_id and report the cheapest variant as the product price, which understates the average sale price by 8-20% for electronics.

The second is 24-hour delivery vs marketplace mixing. PChome 24h’s flagship promise is Taiwan-wide same-day or next-day delivery, but the marketplace (PChome Mall) sits on the same domain with longer fulfillment. The is_24h flag separates them. Aggregating both into the same dataset distorts delivery-speed analytics.

The third is Traditional vs Simplified Chinese normalization. PChome stores product titles in Traditional Chinese. Cross-referencing with Mainland data sources requires a TC->SC conversion via OpenCC or similar. Encoding-only conversion (UTF-8 reading) does not produce a Mainland-readable string.

FAQ

Is the PChome API officially documented?
The ecapi.pchome.com.tw endpoints are the same endpoints used by the PChome web front end. They have been stable for several years but are not contractually supported as a public API. Build defensively with schema drift alerts.

Does PChome offer an affiliate API I should use instead?
PChome operates an affiliate program but the affiliate feed lacks real-time pricing and does not include the full catalogue. For competitive intelligence purposes, scraping the public API remains the higher-fidelity path.

Can I scrape PChome from China mainland IPs?
PChome does not specifically block China mainland IPs but the success rate is meaningfully lower because the cross-strait routing introduces high latency and connection instability. Taiwan or Hong Kong IPs are strongly preferred.

How does PChome handle the difference between PChome 24h and PChome Shopping Mall?
24h is the express-delivery property with a smaller curated catalogue and PChome’s own logistics. Shopping Mall is the larger marketplace with third-party sellers and longer delivery times. The APIs share patterns but use different endpoint paths and product code schemes. Plan for separate code paths if your project covers both.

Are Taiwan-specific holidays a meaningful factor for snapshot scheduling?
Taiwan’s major shopping events (Double 11, Double 12, Lunar New Year, 618) drive significant pricing and promotion activity. During these windows, increase your snapshot frequency to capture the rapid price changes. The default 24-hour cadence misses important intraday movements during peak promotional periods.

Does PChome block non-Taiwan IPs aggressively?
Casual lookups succeed from most regions. Sustained scraping above 100 requests per hour from non-TW IPs triggers rate limits. Taiwan residential IPs are the safe path for production.

How do I track PChome Double 11 promotions?
Capture daily baselines from October 1 onwards. PChome staggers promotions across two weeks leading into 11/11 with daily reveals, so a single pre-event baseline misses the early discount waves.

To build a broader Taiwan ecommerce intelligence stack, browse the ecommerce scraping category for tooling reviews, proxy comparisons, and framework deep dives that pair with the patterns above.