Scraping to MongoDB: Schema-Less Storage for Variable Web Data

I’ll write the article directly.

—

Web scrapers that collect variable data structures — job listings, e-commerce products, news articles — run into relational databases like a wall. scraping to MongoDB solves this by letting each document carry its own shape, so a product with 3 attributes and another with 30 can live in the same collection without a migration ticket.

the tradeoff is real: you gain flexibility and insert speed, you give up strict consistency and ad-hoc aggregation performance. this article covers when that trade is worth making, how to structure your pipeline, and what to watch out for before you put this in production.

when MongoDB fits a scraping pipeline

the core case is structural variation. a scraper hitting 15 e-commerce sites will encounter products with wildly different attribute sets: some have voltage, some have fabric_care, some have neither. forcing this into a relational schema means either an anemic table with hundreds of nullable columns or a slow JSON column workaround.

MongoDB’s document model handles this natively. each document is a BSON object with arbitrary depth, so you store exactly what you scraped without a translation layer. it also has a genuine write throughput advantage over Postgres at high insert rates — benchmarks on Atlas M30 (2026 pricing: ~$0.54/hr) show around 40,000 inserts/sec for small documents, versus ~12,000 for Postgres on comparable hardware.

where MongoDB loses: complex aggregations across documents, strict schema enforcement, and joins. if your downstream use case is analytical queries, consider Scraping to ClickHouse: Real-Time Analytics Pipeline for Web Data (2026) instead, which handles analytical workloads significantly better. for local prototyping without infrastructure, Scraping to DuckDB: Local Analytics Pipeline for Web Data (2026) is often faster to set up.

pipeline architecture

a minimal production-ready scraping-to-MongoDB pipeline has three stages:

fetch — HTTP client with proxy rotation and retry logic
parse — extract structured fields from HTML/JSON
write — upsert into MongoDB with an idempotency key (usually the source URL or item ID)

the upsert step is critical. scrapers re-visit pages. without an idempotency key you get duplicate documents at scale. use update_one with upsert=True and filter on your natural key:

from pymongo import MongoClient, UpdateOne
from datetime import datetime, timezone

client = MongoClient("mongodb+srv://user:pass@cluster.mongodb.net/")
collection = client["scraper"]["products"]

def upsert_product(item: dict) -> None:
    key = {"source_url": item["source_url"]}
    payload = {
        "$set": {**item, "updated_at": datetime.now(timezone.utc)},
        "$setOnInsert": {"first_seen": datetime.now(timezone.utc)},
    }
    collection.update_one(key, payload, upsert=True)

# bulk variant for throughput
def bulk_upsert(items: list[dict]) -> None:
    ops = [
        UpdateOne({"source_url": i["source_url"]}, {"$set": i}, upsert=True)
        for i in items
    ]
    collection.bulk_write(ops, ordered=False)

ordered=False on bulk writes lets MongoDB continue past individual errors, which matters when scraping noisy data with occasional malformed documents.

for orchestration at scale, both Scraping with Dagster: Orchestrating Web Scraping at Scale (2026) and Scraping with Prefect: Modern Workflow Orchestration for Scrapers (2026) integrate cleanly with pymongo — Dagster’s IO managers can wrap a collection, while Prefect tasks compose naturally around the bulk_upsert function above.

indexing strategy for scraped collections

MongoDB reads are only fast if you index correctly. a collection with 10 million documents and no index on source_url will full-scan on every upsert filter — that’s the difference between 1ms and 4 seconds per query.

recommended index set for a scraping collection:

source_url — unique index, used as the upsert key
scraped_at — TTL index if you want documents to expire (e.g., keep 90 days of data)
(category, price) — compound index if you query by facet

// mongosh
db.products.createIndex({ source_url: 1 }, { unique: true })
db.products.createIndex({ scraped_at: 1 }, { expireAfterSeconds: 7776000 })
db.products.createIndex({ category: 1, price: 1 })

avoid indexing every field that lands in a document. each index adds ~10-15% write overhead and consumes RAM. the working set (indexes + hot documents) needs to fit in RAM or Atlas will start swapping and latency spikes.

MongoDB Atlas vs self-hosted: honest comparison

factor	MongoDB Atlas	self-hosted (Ubuntu + mongod)
ops overhead	near-zero	moderate (backups, upgrades, monitoring)
cost at 100GB	~$57/mo (M10)	~$15-20/mo (VPS)
connection limits	plan-gated	configurable
change streams	yes	yes (replica set required)
free tier	512MB M0	unlimited (your hardware)
latency to scraper	depends on region	co-locate for <5ms

for most scraping workloads under 50GB, Atlas M0 (free) or M10 ($57/mo) is the correct choice — the ops savings outweigh the price premium. self-hosting makes sense when you’re archiving terabytes of raw HTML or need to co-locate the database with the scraper fleet to minimize round-trip time.

schema design patterns for variable data

schema-less does not mean schema-free. the best-performing MongoDB scraping setups enforce a loose schema at the application layer:

required fields pattern — every document must have source_url, scraped_at, and domain. everything else is optional. this keeps aggregation queries sane even when product attributes vary wildly.

versioned snapshots — instead of $set overwriting all fields, some pipelines use insert-only mode with a version counter, keeping full history. useful for price tracking but collections grow fast (plan for 3-5x your data volume).

attribute normalization — for e-commerce, normalize the most common attributes (price, brand, sku) into top-level fields, dump the rest into a nested attributes object. this lets you index the important fields without polluting the document root.

before scraping any site at scale, check the legal posture of your target. the ongoing litigation documented in Reddit Lawsuit and Web Scraping: Legal Implications for Data Collectors illustrates how quickly acceptable-use policies can become liability exposure, particularly for commercial data collection.

bottom line

MongoDB is the right default storage layer for scrapers collecting structurally inconsistent data, especially when you need fast writes and flexible downstream querying. use Atlas for anything under a few hundred GB, enforce a loose schema at the application layer, and index on your upsert key from day one. DRT covers the full scraping pipeline stack — storage, orchestration, and legal considerations — so check the rest of the site if you’re assembling this infrastructure end to end.