Scraping ecommerce catalogues at scale

Why catalogue scraping is a different problem

A single product page is easy. Fetch it, parse the DOM or the embedded JSON, done. Ecommerce scraping at catalogue scale is a different job entirely, because the target isn’t one page, it’s tens of thousands of pages that change price, stock, and structure on their own schedule, sitting behind infrastructure that’s specifically built to notice repeated automated traffic.

Most of the engineering effort in a real catalogue scrape doesn’t go into parsing. It goes into keeping a crawl running for days without falling over, without getting a growing share of its requests blocked, and without producing a dataset that’s already stale by the time it lands in a database. That’s the part that doesn’t show up in a “how to scrape with BeautifulSoup” tutorial.

Pagination and listing state

Product listings paginate in a few different ways, and each one changes how you have to crawl it.

Offset or page-number pagination (?page=4) is the simplest to walk but the least stable. If items are inserted or removed while you’re crawling, page boundaries shift and you either skip products or reprocess them. Cursor-based pagination (an opaque token pointing to “the next batch”) is more common on larger catalogues because it’s resilient to that kind of drift, but it means your crawl state has to persist the cursor, not just a page count, or a resume after a failure starts back at page one.

Infinite scroll is the hardest case, because the listing is built by client-side JavaScript firing XHR or fetch calls as the user scrolls, and the HTML you get from a plain HTTP request often just won’t contain the product grid at all. Handling that means either driving a real browser (Playwright or Puppeteer) to trigger those scroll events, or reverse engineering the underlying API calls the frontend makes and hitting those directly. The second option is far cheaper at scale, since a headless browser instance costs orders of magnitude more CPU and memory per page than a direct HTTP request, but it only works when the API responses are stable and not obfuscated behind session-bound tokens that expire quickly.

Variants, options, and the normalization problem

A “product” in an ecommerce catalogue is rarely one row of data. A t-shirt listing might expand into a dozen SKUs across size and color, each with its own price, stock count, and sometimes its own images. Some sites render every variant on the same URL and swap data via JavaScript when you pick an option; others give each variant a distinct URL or query parameter.

The practical effect: you can’t design a catalogue scraper around “one row per page.” You need a data model that separates the parent listing (title, description, category, brand) from the variant-level attributes (SKU, option values, price, availability), and you need to decide up front whether out-of-stock variants get scraped at all, since some storefronts stop rendering them in the DOM entirely once inventory hits zero. Get this wrong on day one and you end up re-crawling the whole catalogue later just to backfill a field you didn’t know you needed.

Proxy infrastructure for catalogue-scale crawls

Once a crawl runs past a few thousand requests to the same domain, IP-based rate limiting becomes the binding constraint, not your own bandwidth or CPU. This is where proxy choice actually matters, and it’s worth being specific instead of hand-wavy about it.

Datacenter IPs are cheap and fast, but they’re also the easiest class of IP for a site to flag, because datacenter ASNs are well known and easy to blocklist or throttle wholesale. Residential and mobile IPs route through real consumer ISPs, so they blend into normal traffic patterns in a way datacenter ranges don’t, which is why they’re the default choice for sustained catalogue crawls against sites with active anti-bot tooling. That doesn’t make them invisible. It changes the cost and detection profile, not the fact that a detection system exists on the other end.

Session behavior matters as much as IP class. Some catalogue crawls need a sticky session, same exit IP across a sequence of requests, when a site ties pagination cursors or A/B-tested category structures to a session cookie. Others benefit from rotating IPs per request specifically to avoid a single IP accumulating enough request volume to trip a velocity threshold. Geo-targeted proxy pools matter for catalogues with region-specific pricing or currency, since fetching from the wrong country’s exit IP silently gives you the wrong price, not an error you’d notice.

Concurrency has to be tuned per target, not set once globally. A crawl that’s polite by one site’s standards can still look like an attack to a smaller storefront with less request headroom. Backoff on 429 and 503 responses, honoring Retry-After headers when they’re present, and capping concurrent connections per domain are baseline engineering discipline here, not optional extras.

How the other side actually defends against this

It’s worth understanding catalogue defense from the site operator’s perspective, because it explains why brittle scrapers break the way they do.

Modern anti-bot systems don’t rely on a single signal. They combine IP reputation (is this ASN associated with datacenter or proxy traffic, has this IP hit us before) with TLS and HTTP fingerprinting (does the client’s negotiation order and header set match a real browser, or a library’s default), and with behavioral signals (mouse movement, scroll timing, request cadence that’s too regular to be human). A request that nails the User-Agent string but sends headers in the wrong order, or opens a raw TCP connection with a TLS fingerprint that doesn’t match any real browser, gets flagged regardless of what the User-Agent claims.

Rate anomaly detection looks at request velocity and pattern rather than any single request. A crawler hitting every product page in strict URL order, at a perfectly even interval, is a much easier pattern to flag than organic browsing, even if each individual request looks legitimate. This is defensive architecture worth understanding on its own terms: it’s how production sites protect infrastructure cost, pricing data, and inventory signals from being harvested by competitors, and it’s not a puzzle to be defeated so much as a cost function to respect. Nothing in this pipeline, proxy or otherwise, makes a scraper undetectable or a target’s terms of service irrelevant. Every anti-bot vendor’s own marketing will tell you their detection rate isn’t 100%, and no proxy vendor’s will tell you their evasion rate is either. Treat both claims with equal skepticism.

Storage, dedup, and change detection

Catalogue data is only useful if you can tell what changed. Re-scraping ten thousand products and dumping ten thousand new rows every run turns your database into an append-only log nobody can query. A workable pipeline hashes each product’s relevant fields (price, stock, title) and only writes a new row when the hash changes, with a last_seen timestamp updated on every pass regardless. That gives you both a change history and a live snapshot from the same table, and it keeps write volume proportional to actual catalogue churn instead of crawl frequency.

Scheduling should match how fast the data actually moves. Price and stock are volatile and might warrant daily or hourly checks on a subset of high-priority SKUs. Descriptions, categories, and images change rarely and re-crawling them daily is wasted request budget that would be better spent on politeness margin elsewhere in the crawl.

Staying inside the lines

Product catalogues, prices, and stock counts on public storefronts sit in a different category from anything behind a login wall or a paywall. Scraping account data, customer reviews tied to identifiable people, or any content that requires authentication to view crosses into territory this kind of infrastructure shouldn’t be pointed at. Check a site’s terms of service and robots.txt before building anything durable against it, and treat a site’s explicit request not to be crawled as the end of the conversation, not an obstacle to route around.

Where this fits

We build and write about the infrastructure layer of this problem at Data Research Tools: proxy pools, orchestration frameworks, and the pipeline patterns that hold up past a few thousand requests. If you’re evaluating tools for a catalogue project, start with what your target actually enforces before picking infrastructure to match it.

Read more scraping infrastructure breakdowns on the Data Research Tools home page.

Get new guides and videos first — join the Telegram channel.