Sitemaps and feeds: the cheapest data source you’re not using

Discovery is the expensive part

Most people budget their scraping costs around the fetch: proxy bandwidth, request volume, maybe headless browser time if the site renders client side. That’s real cost, but it’s not usually the biggest one. The biggest one is discovery, figuring out which URLs exist on a site in the first place.

If you’re crawling a site the naive way, you start at the homepage, extract links, queue them, fetch each page, extract more links, dedupe, repeat. Every one of those discovery hops is a request that burns a proxy IP and rotation budget, and it’s a request that produces zero business value on its own. You’re not extracting data, you’re just finding out where the data might be. On a site with deep pagination, faceted category pages, or infinite scroll, discovery can dwarf the actual extraction traffic.

Sitemaps and feeds skip that phase almost entirely. A site that publishes a sitemap has already done your discovery work and handed you the URL list in one flat file. That’s the whole pitch, and it’s why sitemap scraping is worth building into any pipeline before you write a single crawler rule.

What a sitemap actually contains

A standard XML sitemap is a urlset with a loc entry per URL, plus optional lastmod, changefreq, and priority fields. lastmod is the one that matters operationally, it’s an ISO 8601 timestamp the site claims the page was last modified.

Large sites don’t ship one flat sitemap, they ship a sitemap index: a file whose entries point to other sitemap files instead of pages, often split by content type or date range (sitemap-products-2026-01.xml, sitemap-products-2026-02.xml, and so on). Your parser needs to handle both shapes: fetch the index, recurse into each child sitemap, then flatten to a URL list. Files are frequently gzipped (.xml.gz), so decompression is part of the pipeline, not an edge case.

None of this is exotic. It’s the same protocol Google, Bing, and every other search crawler consumes, which is exactly why it exists: sites publish sitemaps so crawlers don’t have to guess their structure.

Where to find them

Check robots.txt first. It’s supposed to contain a Sitemap: directive pointing to the canonical location, and well-behaved crawlers read it from there rather than guessing. If it’s not listed, /sitemap.xml and /sitemap_index.xml are the common conventions, but they’re conventions, not guarantees. Some CMS platforms nest sitemaps under /sitemap/ or generate per-section files with non-obvious names, and some sites don’t publish one at all. Absence of a sitemap isn’t a signal of anything, it just means you’re back to link-based discovery for that target.

Feeds cover the other half

RSS and Atom feeds do a related but different job. A sitemap is a snapshot of everything that exists. A feed is a stream of what’s new, usually capped at the last 10 to 50 items. For news sites, blogs, changelogs, and job boards, that’s the more useful primitive: instead of re-crawling the whole site to check for new posts, you poll the feed and only see what’s actually changed since your last read.

JSON Feed is a smaller but growing alternative to RSS/Atom, same idea, easier to parse since it’s just JSON instead of XML namespaces. If a site offers both a sitemap and a feed, they’re not redundant, use the sitemap for full-catalog backfill and the feed for ongoing monitoring.

The real saving: incremental fetching

Here’s where sitemaps stop being just a convenience and start being a cost control. Once you’ve done a full sitemap parse and stored the URL list alongside each lastmod value, every subsequent run is a diff, not a full crawl. You compare the new sitemap’s lastmod values against what you stored last time, and only enqueue URLs that actually changed.

That’s the difference between refetching ten thousand pages every night and refetching the two hundred that were actually updated. Every page you don’t have to fetch is bandwidth you don’t spend, proxy rotations you don’t burn, and requests that can’t get you rate limited on the destination site. In a pipeline where proxy cost scales with request volume, this is usually the single biggest lever available, bigger than switching proxy providers or tuning concurrency.

The caveat: lastmod is self-reported by whoever generates the sitemap. Some CMSs set it accurately from the database’s updated_at column. Others just stamp the current time on every generation run regardless of whether anything changed, which makes the field useless for diffing on that particular site. You find this out by observing the field over a few runs, not by trusting it on day one. Treat it as a hint that needs validation, not a contract.

What sitemaps don’t solve

A sitemap gives you a URL and maybe a timestamp. It doesn’t give you the data on the page. You still have to fetch each URL and parse the response, with all the normal considerations that come with that: rendering if the content is client side, proxy rotation if the target rate limits or blocks based on IP reputation, and respecting whatever the site’s terms of service and robots.txt disallow rules say about that specific path.

It’s also worth being clear about what a sitemap is not: it’s not a bypass for content the site has deliberately not made public. Paywalled articles, gated account pages, and personal data behind authentication generally don’t show up in public sitemaps precisely because the site doesn’t want them indexed or crawled. If a page isn’t in the sitemap and isn’t linked from anywhere public, that’s a signal about the publisher’s intent, not a gap to route around.

Where bot detection still applies

Fetching the sitemap file itself is close to risk free from a detection standpoint. It’s a static file that the site published specifically for automated consumption, and server logs are already full of Googlebot, Bingbot, and a dozen SEO tools hitting it constantly. A WAF or bot management layer has little reason to flag a GET request to /sitemap.xml.

That protection ends the moment you start fetching the URLs listed inside it. If a sitemap index contains fifty thousand product pages and you fetch all of them back to back from one IP, you look exactly like a scraper doing bulk extraction, because that’s what’s happening. The site’s rate limiting, WAF rules, and fingerprinting systems evaluate that traffic on its own terms: request rate, header consistency, TLS fingerprint, and behavioral patterns like sequential ID access. Sitemaps make discovery cheap, they don’t make fetching invisible. Any pipeline that treats a sitemap URL list as an excuse to hammer a site at full concurrency will run into the same defenses as any other undisciplined crawler.

Building it into a pipeline

The concrete flow looks like this: fetch robots.txt, extract the Sitemap: line, fetch that file, check if it’s an index or a urlset, recurse if it’s an index, decompress if gzipped, parse loc and lastmod out of every entry, diff against your last stored state, and hand the changed URLs to your normal fetch layer with whatever proxy rotation, rate limiting, and rendering logic you already run.

Most scraping frameworks have this half built already. Scrapy ships a SitemapSpider that handles index recursion and gzip out of the box, and it’s a reasonable starting point to read even if you end up writing your own parser for a specific pipeline’s storage format.

Sitemap and feed parsing won’t replace the rest of your scraping stack. It replaces the part of it that was never adding value in the first place, the part where you were paying proxy and compute cost just to figure out where the pages were.

If you’re building out a pipeline and want more on structuring the fetch layer, proxy rotation, or the frameworks that handle this well, check out the rest of Data Research Tools.

Get new guides and videos first — join the Telegram channel.