Public bulk data you can download instead of crawling

Check for a dump before you build a scraper

Every scraping project starts with the same question nobody asks early enough: does this data already exist somewhere as a file? If you’re about to spin up a proxy pool, write parsing logic for a target’s HTML, and build retry and backoff handling for a site that doesn’t want to be crawled, it’s worth ten minutes to check whether someone has already done that work and published the result as a download.

This matters for the same reason proxy infrastructure costs money. Every request you send through residential or mobile IPs has a per-GB or per-request cost, plus the engineering time to keep the pipeline healthy when a target changes its markup or tightens its bot detection. A bulk download has none of that. You fetch a file once, from a server that wants you to have it, and you’re done with the network layer entirely. The rest of the job becomes parsing and storage, which is usually the easier half anyway.

The catch is that bulk dumps are never a full substitute for a live scraper. They’re a snapshot, they’re licensed for specific uses, and they cover what the publisher decided to include, not what you actually need. Knowing which is which up front saves a lot of wasted infrastructure.

Common Crawl: the web crawl you don’t have to run yourself

Common Crawl publishes petabyte-scale monthly crawls of the open web as WARC, WAT, and WET files hosted on AWS S3, free to pull with no auth wall. If your project is something like “find every page that mentions X domain” or “build a training corpus from a broad web sample,” this is usually the first thing to check, because it means you’re not the one operating the crawler, rotating IPs, or absorbing rate limits from millions of individual sites.

The tradeoffs are real. Common Crawl’s coverage is broad but not exhaustive and not guaranteed to include any specific site you care about, since it crawls based on its own seed list and politeness rules. The data is also a month or more stale by the time it’s published, so it’s a poor fit for anything that needs current state. And WARC files are large and require real parsing infrastructure (something like a Spark or DuckDB job against the index) to be useful at scale. You’re trading crawler infrastructure for a different kind of infrastructure, not eliminating infrastructure entirely.

Wikipedia and Wikidata dumps

Wikimedia publishes full database dumps of every language edition of Wikipedia, plus Wikidata’s structured knowledge base, on a regular schedule at dumps.wikimedia.org. These are XML and JSON exports containing full revision history, current article text, and in Wikidata’s case, structured entity relationships. For any project that would otherwise involve scraping Wikipedia article by article, this is a straightforward win: you get the entire encyclopedia in one download, licensed under Creative Commons terms, with no rate limiting to manage because you’re not hitting the live site at all.

The practical hurdle is size and format. The full English dump is tens of gigabytes compressed, and parsing wikitext markup into clean text is its own non-trivial job, since Wikipedia’s markup includes templates, infoboxes, and citations that don’t map cleanly to plain text. Wikidata’s JSON dump solves a different problem (structured facts, not prose) and is worth pulling separately if that’s what you actually need.

Government and public records data

Most national statistics agencies publish their own data as structured downloads rather than expecting anyone to scrape a web frontend. The US has data.gov as an index across federal agencies, the Census Bureau publishes API and bulk file access to its surveys, and Eurostat does the same for EU member states. State and local governments vary a lot in quality, but many publish open data portals (usually running Socrata or CKAN) with CSV or API exports for things like business licenses, property records, and permit filings.

This is the category where bulk data most clearly beats scraping, because government sites are frequently the worst possible scraping target: inconsistent HTML, session-based pagination, and legal terms that are stricter than commercial sites in some jurisdictions since the underlying records can carry their own use restrictions. If the same data is sitting in a downloadable CSV from the same agency, there’s no reason to touch the web frontend at all.

Company filings and financial data

The SEC’s EDGAR system provides full-text search and bulk access to every filing companies submit, including quarterly and annual reports, insider trading disclosures, and prospectuses, going back decades. It’s a public dataset by design, since disclosure is the point of the filing requirement, and the SEC publishes both an API and periodic bulk archives rather than relying on people scraping the filing viewer.

The IRS also publishes bulk data on nonprofit Form 990 filings through the AWS Registry of Open Data, which covers financial details for hundreds of thousands of tax-exempt organizations. Between EDGAR and the 990 archive, a lot of what people build scrapers to extract from investor-relations pages or nonprofit directories is already sitting in a structured, official download.

Developer and code activity: GH Archive

GH Archive records every public GitHub event (pushes, issues, pull requests, stars) and publishes it as hourly JSON archives going back to 2011, mirrored into BigQuery for querying at scale. If a project involves tracking open source activity, dependency trends, or contributor patterns, this replaces what would otherwise be a very fast way to get your IP range blocked by GitHub’s API rate limits. It doesn’t cover private repos or content that isn’t a public event, so it’s an activity log, not a full code archive.

Geographic data: OpenStreetMap

OpenStreetMap publishes its entire global dataset as a “planet file” through planet.openstreetmap.org, plus regional extracts through third parties like Geofabrik, covering roads, buildings, points of interest, and administrative boundaries contributed under an open license. Anyone building something that needs address data, POI locations, or map geometry should check this before scraping a mapping service, both because OSM’s terms are built for bulk reuse and because most commercial map providers explicitly prohibit scraping their tiles or place data in their terms of service.

What bulk data doesn’t solve

None of this replaces a scraper for data that’s inherently live: current pricing on a specific retailer’s site, real-time inventory, a competitor’s changing product catalog, or anything behind a login wall that no one has agreed to publish. Bulk dumps exist because a publisher decided to make that data available on their terms. When the data you need isn’t covered by any of those terms, that’s a different problem, and it’s the one actual scraping infrastructure is built to solve, within whatever the target’s terms of service and applicable law allow.

It’s also worth being honest about staleness and completeness. A monthly Common Crawl snapshot won’t have yesterday’s page changes. A government CSV export is only as current as the agency’s publishing schedule. Treat bulk downloads as a specific tool for a specific job (broad, static, or slowly-changing datasets) and keep scraping infrastructure for what’s actually live and not otherwise published.

Where this fits in a real pipeline

In practice, the projects that run cheapest are the ones that check for a bulk source first, use it to cover as much ground as possible, and reserve live scraping infrastructure, proxies, request pacing, parsing logic tuned to a specific site, for the narrow slice of data that genuinely isn’t available any other way. That’s not a purity argument, it’s a cost argument: every request through a proxy pool has a price, and every page you don’t have to fetch live is infrastructure you don’t have to build or maintain.

If you’re weighing when a project needs real scraping infrastructure versus when a bulk download will do, that’s exactly the kind of tradeoff we cover on the channel and in the rest of the site.

Explore more breakdowns like this on the Data Research Tools homepage.

Get new guides and videos first — join the Telegram channel.