Category: Uncategorized

Best Node.js scraping libraries 2026

Best Node scraping libraries in 2026 occupy a different ecosystem than Python’s. Node’s event-loop architecture is naturally async-first, so concurrency comes for free. Browser automation has stronger native fit because Puppeteer was originally a Node-only library and the JavaScript-runtime-controlling-JavaScript story is uniquely tight. The Node scraping market has consolidated around a smaller list of high-quality libraries than Python’s, but each library is more polished and the gaps are smaller. The four-layer model still applies: HTTP client, browser automation, HTML parser, framework. Picking the right combination per layer determines whether your scraper handles 100 or 10,000 requests per second.

This guide ranks the Node.js scraping libraries actually worth using in 2026, with honest performance comparisons, clear use case mapping, and the gotchas that surprise developers coming from Python.

HTTP clients

undici

Node’s modern HTTP client, developed by the Node.js core team. Faster than every alternative by 2-3x for high-concurrency workloads. The standard fetch global in modern Node uses undici under the hood.

import { fetch } from 'undici';

async function scrape(url) {
  const resp = await fetch(url, {
    headers: { 'User-Agent': 'Mozilla/5.0 ...' },
  });
  return resp.text();
}

Best for: any new project, high-concurrency workloads, the default unless you have specific needs.

got

The popular HTTP client before undici took over. Excellent retry, redirect, and cookie handling. Slightly slower than undici but more feature-complete out of the box.

import got from 'got';

const html = await got('https://example.com', {
  retry: { limit: 3 },
  timeout: { request: 10000 },
}).text();

Best for: existing got codebases, projects that benefit from got’s batteries-included extras.

axios

The classic. Sync-style promise API that everyone knows. Slower than undici and got. Still ubiquitous because of legacy familiarity.

Best for: existing axios codebases, teams that already know its API.

node-fetch

The original Node fetch polyfill, now mostly obsolete since native fetch landed in Node 18+.

Best for: legacy projects, nothing else.

Browser automation

Playwright

Same library as Python; the JavaScript version is actually the reference implementation. Cleaner API in JavaScript than Python because of TypeScript autocompletion. Best browser automation framework in any language.

import { chromium } from 'playwright';

const browser = await chromium.launch();
const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
});
const page = await context.newPage();
await page.goto('https://target.example.com');
const title = await page.locator('h1.product-title').textContent();
await browser.close();

Best for: most modern browser automation in Node, multi-browser needs.

Puppeteer

The Google-maintained Chrome automation library. Slightly cleaner Chrome-specific features than Playwright, similar overall capability. The puppeteer-extra plugin ecosystem (especially stealth plugin) is more mature than Playwright’s equivalents.

Best for: Chrome-only workflows, projects using puppeteer-extra plugins.

Crawlee

Apify’s scraping framework with built-in browser support. Wraps Playwright/Puppeteer with crawler-style ergonomics. We cover it under frameworks below.

HTML parsers

Cheerio

The jQuery-syntax server-side parser. The dominant Node HTML parser. Fast (built on parse5 or htmlparser2), familiar API for anyone who used jQuery.

import * as cheerio from 'cheerio';

const $ = cheerio.load(html);
const titles = $('h2.product-title').map((i, el) => $(el).text()).get();

Best for: most HTML parsing in Node, jQuery-familiar developers.

parse5

The lower-level HTML parser that Cheerio uses under the hood. Direct use is rare but available for custom AST work.

Best for: custom HTML manipulation, building higher-level tools.

htmlparser2

Streaming HTML parser, very fast on large documents. Used by Cheerio when configured for it. Direct use for stream-based parsing.

Best for: parsing very large HTML documents in stream mode.

linkedom

Modern alternative offering full DOM API (not just jQuery-style). If your code expects document.querySelector semantics, linkedom feels native.

import { parseHTML } from 'linkedom';

const { document } = parseHTML(html);
const titles = Array.from(document.querySelectorAll('h2.product-title')).map(el => el.textContent);

Best for: developers who prefer DOM API over jQuery API, code shared between client and server.

Frameworks

Crawlee

Apify’s modern scraping framework. The Node version is the original; the Python port came later. Excellent abstractions for HTTP and browser scraping with the same Crawler interface, built-in queue management, dedupe, retry logic, and proxy rotation.

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  async requestHandler({ request, $ }) {
    console.log(`Scraping ${request.url}`);
    const titles = $('h2.product-title').map((i, el) => $(el).text()).get();
    await crawler.pushData({ url: request.url, titles });
  },
  maxRequestsPerCrawl: 1000,
  maxConcurrency: 10,
});

await crawler.run(['https://shop.example.com/page/1']);

Best for: most modern Node scraping projects that need framework benefits.

x-ray

Older declarative scraping framework. Still works but rarely chosen for new projects.

Best for: legacy x-ray codebases.

Apify SDK

Crawlee’s parent SDK with additional Actor and platform features. Right choice if deploying to Apify cloud.

Best for: Apify platform deployments.

Comparison table

library	layer	speed	learning curve	best for
undici	HTTP	fastest	easy	most new projects
got	HTTP	fast	easy	retry-heavy needs
axios	HTTP	mid	easy	legacy codebases
node-fetch	HTTP	mid	easy	nothing in 2026
Playwright	browser	mid	medium	most browser automation
Puppeteer	browser	mid	medium	Chrome-only, stealth plugins
Cheerio	parser	fast	easy	most parsing
parse5	parser	fast	hard	custom AST work
htmlparser2	parser	fastest	medium	very large docs, streaming
linkedom	parser	fast	easy	DOM-API preference
Crawlee	framework	fast	medium	modern crawler projects
x-ray	framework	mid	easy	legacy

Decision matrix: solopreneur, SMB, enterprise

profile	scale	recommended stack	reasoning
Solopreneur learning	<10k pages/day	native fetch + Cheerio	Zero dependencies, modern defaults
Indie scraper	<500k pages/day	undici + Cheerio + p-limit	Best HTTP perf, simple flow control
Indie JS-heavy	<100k pages/day	Playwright + Cheerio post-parse	Browser only when needed
SMB crawler	500k-10M pages/day	Crawlee CheerioCrawler	Framework manages queue, dedupe, retry
SMB anti-detect	100k-1M pages/day	Puppeteer + puppeteer-extra-plugin-stealth	Stealth ecosystem maturity
Enterprise	10M+ pages/day	Crawlee on K8s + custom middleware	Volume justifies platform investment
Hybrid HTTP/JS	varies	Crawlee (CheerioCrawler + PlaywrightCrawler)	Same dataset across two modes

The Node ecosystem rewards convergence: most teams end up on undici + Cheerio for HTTP and Playwright + stealth plugins for browser. Crawlee adds value above 500k pages/day; below that, hand-rolled async with p-limit is simpler and fast enough.

Migration path: axios + cheerio to undici + Cheerio

Most legacy Node scrapers run on axios because it was the dominant HTTP client of the 2018-2022 era. Modernizing to undici is straightforward and yields a 2-3x throughput improvement:

Replace axios.get(url, opts) with await fetch(url, opts) from undici. The API differs slightly (response body via .text() / .json() instead of .data).
Replace axios interceptors with explicit retry wrappers. undici does not have an interceptor system; use a small wrapper function for retry, logging, and metrics.
Update timeout handling to use AbortSignal.timeout(ms) instead of axios’s timeout option.
Benchmark the same workload before and after. Expect 2-3x improvement on concurrent request throughput.
Keep axios for any code that uses interceptors heavily (auth refresh patterns, request signing) where the cost of unwinding interceptor logic outweighs the perf gain.

A typical Node scraper migration completes in a day. The performance gain often unblocks scaling work that was on the roadmap for distributed infrastructure.

Performance benchmarks

Same workload as the Python benchmarks: 10,000 simple HTML pages from a local mirror, single Node process.

stack	total time	requests/sec
native fetch (50 concurrency)	8s	1250
undici (50 concurrency)	7s	1428
got (50 concurrency)	11s	909
axios (50 concurrency)	14s	714
Playwright (50 concurrent contexts)	88s	113
Crawlee CheerioCrawler	9s	1111

Node beats Python on raw HTTP throughput thanks to its event loop architecture. The browser automation gap is similar in both languages because the bottleneck is browser execution, not the host runtime.

Cost worked example

For a 100k-pages-per-day Node scraping workload on mixed protected and unprotected targets:

1 small VPS ($20/mo, 4 vCPU, 8 GB)
undici + Cheerio + p-limit stack (free, Node only)
node-libcurl when TLS impersonation is needed (free, requires native build)
Smartproxy/Decodo residential proxies (~$50/mo for 5 GB)
PostgreSQL on a hosted instance ($25/mo)
Optional: ZenRows fallback for hard surfaces (~$69/mo)

Total: about $95-165/month depending on the API fallback. Node throughput is higher than Python on raw HTTP, which lets you pack more work into the same VPS; expect to need ~30% less compute capacity than the equivalent Python deployment.

The other Node-specific cost dimension is RAM. Node processes typically run 200-300 MB at scraping idle and grow with concurrent contexts. For a single-process scraper, 8 GB RAM is plenty; for distributed multi-worker setups, prefer many small workers over few large ones to limit blast radius from leaks.

Stack recommendations

Small project, scripts: native fetch + Cheerio. Built into modern Node, no dependencies, fast.

Medium project, no JS needs: undici + Cheerio. Fastest HTTP client + standard parser. Add tenacity-style retry via simple wrapper.

JavaScript-heavy targets: Playwright + Cheerio (parse the extracted HTML with Cheerio for speed instead of using Playwright’s slower DOM querying).

Large crawler with link-following: Crawlee. Built-in queue management saves you from writing your own.

Anti-bot heavy targets: Puppeteer with puppeteer-extra-plugin-stealth. The stealth plugin ecosystem is more mature for Puppeteer than for Playwright in Node.

Hybrid HTTP + browser: Crawlee with multiple crawler classes (CheerioCrawler for HTTP-only pages, PlaywrightCrawler for JS-heavy pages, both writing to the same dataset).

Crawlee deep dive

Crawlee deserves a closer look because it has matured into the de-facto Node scraping framework. Its three crawler classes cover the spectrum:

CheerioCrawler: HTTP-only, uses got (or undici under the hood) and Cheerio. Fast, low-resource. The right default for HTTP scraping.
PlaywrightCrawler: full browser automation with Playwright. Highest resource cost but handles any JavaScript.
PuppeteerCrawler: same as Playwright but using Puppeteer. Choose this if your team prefers Puppeteer’s API or uses puppeteer-extra plugins.

All three share the same RequestQueue, Dataset, and KeyValueStore abstractions, which means you can switch between HTTP and browser modes per request without changing your data layer. A typical pattern is to start with CheerioCrawler, fall back to PlaywrightCrawler when the HTML is missing the data you need, and store both kinds of results in the same Dataset.

Crawlee’s RequestQueue supports SQLite, MongoDB, and the Apify cloud as backends. SQLite works for single-process crawlers; MongoDB works for distributed crawlers across machines. The cloud backend gives you a managed queue with no operational overhead.

Modern async patterns

Node’s async syntax is cleaner than Python’s for typical scraping patterns:

import { fetch } from 'undici';
import * as cheerio from 'cheerio';
import pLimit from 'p-limit';

async function fetchPage(url, retries = 3) {
  for (let attempt = 0; attempt < retries; attempt++) {
    try {
      const resp = await fetch(url, {
        signal: AbortSignal.timeout(15000),
      });
      if (resp.status === 200) {
        return await resp.text();
      }
      if (resp.status === 429 || resp.status === 503) {
        await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 1000));
        continue;
      }
      return null;
    } catch (err) {
      if (attempt === retries - 1) throw err;
      await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 1000));
    }
  }
}

function parseProducts(html) {
  if (!html) return [];
  const $ = cheerio.load(html);
  return $('div.product-card').map((i, el) => ({
    name: $(el).find('h2.title').text(),
    price: $(el).find('span.price').text(),
  })).get();
}

async function scrapeAll(urls, concurrency = 20) {
  const limit = pLimit(concurrency);
  const results = await Promise.all(
    urls.map(url => limit(async () => {
      const html = await fetchPage(url);
      return parseProducts(html);
    }))
  );
  return results.flat();
}

This pattern handles 1000+ pages per minute on a modest VPS with retries and concurrency control built in.

Persistence and storage in Node

Node scrapers benefit from a few storage patterns specific to JavaScript ecosystems:

better-sqlite3 for synchronous local storage. Faster than the async sqlite3 library for write-heavy workloads because it avoids async overhead.
Knex or Prisma for typed Postgres access. Both work well; Prisma’s TypeScript types are stronger but Knex is lighter.
Crawlee KeyValueStore + Dataset. When using Crawlee, prefer its built-in storage abstractions; they handle large blobs and structured records cleanly.
DuckDB-WASM for in-process analytics. When you want to query scraped data without a database server, DuckDB now ships a Node binding that lets you run SQL on Parquet or local arrays.

For very large output volumes, stream writes to S3 / R2 with @aws-sdk/client-s3 MultipartUpload rather than collecting everything in memory and uploading at the end.

Common mistakes to avoid

Using axios in 2026: it works but is slower than undici and got. New projects should default to undici.

Forgetting AbortSignal.timeout: Node’s native fetch does not have a default timeout. Without one, your scraper hangs on slow targets indefinitely.

Loading huge HTML strings into Cheerio at once: for documents over 10 MB, use htmlparser2 in streaming mode.

Running too many browser contexts in one Node process: Node memory grows fast with many Playwright contexts. Stay under 50 concurrent contexts per process.

Ignoring back-pressure in Crawlee: Crawlee’s queues can grow unboundedly if you push faster than you consume. Set maxRequestsPerCrawl and maxConcurrency appropriately.

We cover the Python equivalent in our best Python scraping libraries 2026 review.

Common gotchas

Native fetch lacks default timeout. Without AbortSignal.timeout(), your scraper hangs on slow targets indefinitely. Always set a timeout.
undici keep-alive defaults. undici defaults to keep-alive connections. For one-off scripts, this can leave the process hanging waiting for sockets. Use Agent({ keepAliveTimeout: 1 }) or call agent.close() at script end.
Cheerio re-parse cost. Each call to cheerio.load() re-parses the HTML. For many extractions on the same document, parse once and pass $ around.
Playwright newPage vs newContext. newPage() reuses the parent context’s cookies; newContext() creates a fresh storage state. Use newContext() per scrape to isolate cookies; many subtle bugs come from cookie cross-contamination.
Crawlee request handler errors. A throw inside requestHandler retries the request by default. If the error is permanent (404, parse failure), call request.noRetry = true to skip the retry queue.
JSON parsing with native fetch. await resp.json() throws on empty body; wrap in try/catch or check resp.ok first.
EventEmitter memory leaks. Browser launches that emit console, request, or response events accumulate listeners if you do not clean them up. Use page.removeAllListeners() before close or use named handler functions you can remove explicitly.
TLS hardening on undici. Some targets refuse TLS 1.2 connections. undici defaults to negotiating up; if you see handshake errors, force connect: { tls: { minVersion: 'TLSv1.3' } }.

TypeScript vs JavaScript

For new projects, TypeScript is the right choice. The HTTP and parsing libraries all ship with strong type definitions. The Playwright API in TypeScript is a different developer experience than JavaScript.

import { chromium, Browser, Page } from 'playwright';

async function scrape(browser: Browser, url: string): Promise<string | null> {
  const page: Page = await browser.newPage();
  try {
    await page.goto(url);
    return await page.locator('h1').textContent();
  } finally {
    await page.close();
  }
}

The autocompletion and refactoring support pay off within the first week of any non-trivial project.

External authoritative reference: the Node.js documentation on the global fetch covers the standard HTTP client.

Bun and Deno alternatives

Bun and Deno both ship with built-in fetch and run all the libraries above. Bun is notably faster than Node for HTTP-heavy workloads (about 1.5-2x in our testing). Deno’s permission model is interesting for scraping isolation but the ecosystem is smaller.

For most teams, Node remains the right default in 2026 because library compatibility is broadest. We cover the alternatives in our forthcoming guides on Bun and Deno scraping.

FAQ

Q: Cheerio or jQuery selectors?
Cheerio implements jQuery-style selectors server-side. The API is essentially identical. Use Cheerio in Node; do not import actual jQuery server-side.

Q: should I use Crawlee or write my own crawler?
For projects under 1000 pages, write your own with undici + Cheerio + p-limit. For larger crawls with link-following, dedupe, and retry needs, Crawlee saves significant code.

Q: Puppeteer or Playwright?
Playwright is technically better for new projects: cleaner API, multi-browser, better auto-waiting. Puppeteer has the puppeteer-extra-plugin-stealth ecosystem advantage which still matters for some anti-detect work.

Q: how do I handle TLS fingerprinting in Node?
Node does not have a great equivalent to Python’s curl_cffi yet. The closest options are node-libcurl (libcurl bindings for Node) or routing through a proxy that handles TLS fingerprinting on your behalf.

Q: is JSDOM useful for scraping?
JSDOM is heavier than Cheerio because it implements the full DOM API including layout. For scraping where you do not need actual JavaScript execution, Cheerio is faster. JSDOM is the right choice when you want to execute scripts on a parsed document without a full browser.

Q: how do I integrate proxies?
With undici, use a ProxyAgent. With Playwright, pass proxy to chromium.launch(). Crawlee has built-in proxy rotation across a pool. Avoid manual proxy management; use the built-in tools wherever possible.

Q: is Bun production-ready for scraping?
Yes for most use cases. Bun’s built-in fetch and HTML parser are excellent. The remaining gaps are around obscure npm packages with native dependencies that have not been compiled for Bun. Test your dependency tree before committing to Bun in production.

Q: what is the cleanest way to handle pagination?
Wrap your fetch in an async generator that yields pages until a stop condition. Async generators in Node compose nicely with for await loops and avoid materializing all pages in memory.

Closing

The Node.js scraping stack in 2026 is mature and stable. undici for HTTP, Cheerio for parsing, Playwright for browser automation, Crawlee for crawler frameworks. The ecosystem moves slower than Python’s but each piece is more polished. Match the stack to the workload and Node will outperform Python on raw HTTP throughput while matching it on browser automation. For broader scraping infrastructure see our dev-tools-projects category hub.

May 7, 2026

Best Python scraping libraries 2026: Scrapy, BS4, more

Best Python scraping libraries in 2026 cover a stack that has matured significantly since the requests + BeautifulSoup era. The HTTP client layer has been split into a dozen options optimized for different use cases. Browser automation has consolidated around Playwright. Parsing has stabilized on lxml under the hood with multiple frontend options. The framework layer sees Scrapy holding its dominance for crawling-heavy use cases and Crawlee gaining ground as a modern alternative. Choosing the right combination of libraries determines whether your scraper runs at 10 requests per second or 1000, whether it survives anti-bot fingerprinting, and how much code you write to do straightforward things.

This guide ranks the Python scraping libraries actually worth using in 2026, organized by what they do, with honest performance comparisons and clear guidance on which to pick for which workload.

The four layers of a Python scraper

Every scraper has four layers, regardless of framework:

HTTP client: makes the actual network requests
Browser automation (optional): when JavaScript execution is needed
HTML parser: extracts data from the response
Orchestration framework (optional): handles concurrency, retries, queues, pipelines

Different libraries dominate each layer. The right scraper picks the best library per layer rather than committing to one library for everything.

HTTP clients

requests

The classic. Synchronous, simple, mature. Still the right choice for one-off scripts and learning. Performance is the worst of the modern options because it is sync-only.

import requests

resp = requests.get("https://example.com", headers={"User-Agent": "..."}, timeout=10)
print(resp.text)

Best for: scripts, prototypes, learning, anything where async is overkill.

httpx

The modern requests replacement. Supports both sync and async, HTTP/2 by default, type-hinted, and dramatically faster than requests for any concurrent workload. The drop-in replacement for requests in most code.

import httpx
import asyncio

async def fetch(url: str):
    async with httpx.AsyncClient(http2=True, timeout=10) as client:
        resp = await client.get(url)
        return resp.text

# concurrent fetching
async def main():
    urls = ["https://example.com/page/1", "https://example.com/page/2"]
    return await asyncio.gather(*[fetch(u) for u in urls])

Best for: any new project, async workloads, HTTP/2 support, type safety.

aiohttp

The async-first HTTP client. Older than httpx but still excellent. Slightly faster than httpx for high concurrency. The websocket support is best in class.

Best for: high-concurrency async workloads, websocket-heavy use cases.

curl_cffi

The TLS-fingerprint-aware HTTP client. Wraps libcurl with browser-impersonation features so your TLS handshake looks like real Chrome, Firefox, or Safari. The right choice for any target with TLS fingerprinting (Cloudflare, DataDome, Akamai).

from curl_cffi import requests

# impersonates Chrome 120 TLS fingerprint
resp = requests.get(
    "https://target.example.com",
    impersonate="chrome120",
    timeout=10,
)
print(resp.text)

Best for: bypassing TLS fingerprinting, scraping sites that detect Python’s default TLS.

urllib3

The HTTP foundation that requests and httpx both use under the hood. Rarely used directly except for very low-level needs.

Best for: when you need fine-grained control of connection pools.

HTML parsers

lxml

The fast XML/HTML parser written in C. Underpins almost every other parser. Direct lxml use is fastest but the API is uglier than BeautifulSoup.

from lxml import html

tree = html.fromstring(html_content)
titles = tree.xpath("//h2[@class='product-title']/text()")

Best for: high-performance parsing, XPath-heavy extraction.

BeautifulSoup4

The friendly parser. Slower than lxml directly but has the most readable API. The standard configuration uses lxml as the underlying parser, so the speed gap is smaller than people assume.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content, "lxml")
titles = [t.text for t in soup.select("h2.product-title")]

Best for: most general-purpose parsing, readable code, mixed CSS/find patterns.

selectolax

The fastest Python HTML parser by a wide margin. C-based, supports CSS selectors with a minimal API. Roughly 5-10x faster than BeautifulSoup on typical pages.

from selectolax.parser import HTMLParser

tree = HTMLParser(html_content)
titles = [n.text() for n in tree.css("h2.product-title")]

Best for: high-volume parsing where every millisecond matters.

parsel

Scrapy’s parser, available standalone. Combines XPath, CSS, and regex selectors with a clean API.

Best for: Scrapy users wanting the same API outside Scrapy.

Browser automation

Playwright

The current best browser automation framework. We covered it in detail in best headless browser frameworks 2026.

Best for: modern browser automation, multi-browser support.

Selenium

The elder framework. Still solid for cross-browser needs. Verbose but well-documented.

Best for: legacy projects, multi-language teams sharing test infrastructure.

Pyppeteer (less recommended)

Python port of Puppeteer. Less actively maintained than Playwright. Avoid for new projects.

Frameworks

Scrapy

The dominant Python scraping framework. Async by design (since 2.0), built-in queue management, middleware system, item pipelines, and crawl rules. The right choice for crawler-heavy workloads where you are following links across thousands of pages.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://shop.example.com/category/widgets"]

    def parse(self, response):
        for product in response.css("div.product"):
            yield {
                "name": product.css("h2::text").get(),
                "price": product.css("span.price::text").get(),
                "url": product.css("a::attr(href)").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

The downside: Scrapy’s mental model is heavier than other frameworks. You learn callbacks, middleware, settings, items, and pipelines. For simple scrapers this is overkill.

Best for: large crawl projects, structured data extraction at scale, when you actually need a framework.

Crawlee

The newer framework from Apify, with a Pythonic API and built-in browser support. More approachable than Scrapy for newcomers. Supports HTTP and browser modes from the same crawler class.

Best for: modern projects that want framework benefits without Scrapy’s learning curve.

Pyspider

Older framework with a web UI for managing scrapers. Less actively maintained but still works for some use cases.

Best for: legacy systems, niche use cases needing visual management.

Comparison table

library	layer	sync/async	speed	learning curve	best for
requests	HTTP	sync	slow	easy	scripts, prototypes
httpx	HTTP	both	fast	easy	most new projects
aiohttp	HTTP	async	fast	medium	high-concurrency async
curl_cffi	HTTP	sync	fast	easy	TLS fingerprint bypass
BeautifulSoup4	parser	sync	mid	easy	general parsing
lxml	parser	sync	fast	medium	high-performance XPath
selectolax	parser	sync	fastest	easy	extreme volume parsing
parsel	parser	sync	fast	easy	Scrapy users
Playwright	browser	both	mid	medium	JS-heavy targets
Selenium	browser	sync (mostly)	slow	medium	legacy
Scrapy	framework	async	fast	hard	large crawls
Crawlee	framework	async	fast	medium	modern, browser-friendly

Decision matrix: solopreneur, SMB, enterprise

profile	scale	recommended stack	reasoning
Solopreneur learning	<1k pages/day	requests + BeautifulSoup4	Simple, beginner-friendly
Indie scraper (basic)	<100k pages/day	httpx + selectolax	Async, fast, modern
Indie scraper (anti-bot)	<100k pages/day	curl_cffi + selectolax	TLS impersonation included
SMB crawler	100k-10M pages/day	Scrapy + curl_cffi middleware + selectolax	Framework value at this scale
SMB JS-heavy	10k-1M pages/day	Crawlee or Playwright + httpx fallback	Hybrid HTTP/browser ergonomics
Enterprise pipeline	10M+ pages/day	Scrapy on K8s + custom middleware + dedicated parsers	Full ops + custom optimization
Single-source ETL	varies	httpx + lxml direct XPath	Tight, predictable, performant

The most expensive mistake is over-frameworking small jobs (Scrapy for 200 pages) and under-frameworking large jobs (raw httpx loop for 10M URLs). Match the framework weight to the actual workload.

Migration path: requests + BS4 to httpx + selectolax

Most legacy Python scrapers can be modernized in a day with significant performance gains. The playbook:

Wrap your fetch function in an async signature even before changing implementation. This isolates the migration scope.
Replace requests.get with httpx.AsyncClient.get. Most code translates 1:1; the main change is await keywords and the async context manager.
Switch parser to selectolax. CSS selectors translate directly from BeautifulSoup; XPath users stay on lxml. Expect 5-10x parse speedup.
Add concurrency with asyncio.Semaphore to bound parallel requests. Start at 10 and tune based on target tolerance.
Benchmark against original with the same input set. A typical migration shows 15-30x throughput improvement.

The whole migration usually takes one engineer-day for a single-purpose scraper, two days for a multi-target codebase. The throughput gain often eliminates the need for distributed scaling that was on the roadmap.

Performance benchmarks

We benchmarked HTTP fetching of 10,000 simple HTML pages from a local mirror, single machine, no network bottleneck.

stack	total time	requests/sec
requests (sync)	240s	42
httpx (async, 50 concurrency)	12s	833
aiohttp (50 concurrency)	11s	909
curl_cffi (50 concurrency)	14s	714
Scrapy (default settings)	18s	555
Playwright (50 concurrent contexts)	95s	105

Async HTTP is 20x faster than sync. Browser automation is 8-10x slower than HTTP. The difference is essentially the cost of running JavaScript and rendering, which is unavoidable for SPAs.

Choosing between async frameworks

Python’s async ecosystem fragmented for years between asyncio (standard library), trio (alternative event loop with cleaner cancellation semantics), and AnyIO (a compatibility layer). For scraping, asyncio is the right default because every major HTTP and parser library targets it. trio remains technically superior for cancellation safety but the ecosystem cost is real.

The other choice is between asyncio’s default event loop and uvloop (a Cython-accelerated drop-in replacement). For HTTP-bound scrapers, uvloop yields a 2-4x throughput improvement essentially for free:

import asyncio
import uvloop

uvloop.install()  # do this before any asyncio code

# rest of your scraper

The two-line installation gets you the benefit. The only caveat is that uvloop does not work on Windows; cross-platform code needs a try/except around the install call.

Stack recommendations by use case

Small project, learning, scripts: requests + BeautifulSoup4. Simple, well-documented, slow but adequate.

Medium project, production, no JavaScript needs: httpx (async) + selectolax. Fast, modern, scales to a few hundred requests per second on one machine.

Medium project with anti-bot needs: curl_cffi + selectolax. The TLS fingerprint matters more than raw speed for protected targets.

Large crawler with link-following: Scrapy + parsel. Built-in queue management, dedupe, retry middleware. The framework cost is justified at this scale.

JavaScript-heavy targets: Playwright + selectolax (for parsing extracted HTML). Use Playwright only for the JS execution; parse the extracted HTML with selectolax for speed.

Hybrid (some JS, some HTTP): Crawlee or drissionPage. Both support seamless switching between HTTP and browser modes.

Cost worked example

A practical 100k-pages-per-day workload on protected targets needs:

1 small VPS ($20/mo, 4 vCPU, 8 GB)
httpx + uvloop + selectolax stack (free, Python only)
curl_cffi for TLS impersonation when needed (free)
Residential proxy pool from Smartproxy/Decodo (~$50/mo for 5 GB)
PostgreSQL on a hosted instance ($25/mo)
Optional: ScraperAPI fallback for surfaces that fail consistently (~$49/mo)

Total: about $95-145/month depending on whether you include the API fallback. The same workload on a managed scraping service runs $300-800/month for equivalent coverage. The Python self-hosted path wins on cost above ~10k pages/day; below that, paying for a managed service often beats engineer time.

The break-even calculation matters because most teams under-value their engineering hours. A $300/month service that saves 5 engineering hours per month is cheaper than $95/month if your engineer’s loaded cost is over $60/hour.

Idiomatic patterns

A modern async scraper template:

import asyncio
import httpx
from selectolax.parser import HTMLParser
from typing import AsyncGenerator

async def fetch_page(client: httpx.AsyncClient, url: str) -> str:
    for attempt in range(3):
        try:
            resp = await client.get(url, timeout=15.0)
            if resp.status_code == 200:
                return resp.text
            if resp.status_code in (429, 503):
                await asyncio.sleep(2 ** attempt)
                continue
            return None
        except (httpx.TimeoutException, httpx.NetworkError):
            await asyncio.sleep(2 ** attempt)
    return None


def parse_products(html: str) -> list[dict]:
    if not html:
        return []
    tree = HTMLParser(html)
    return [
        {
            "name": n.css_first("h2.title").text() if n.css_first("h2.title") else None,
            "price": n.css_first("span.price").text() if n.css_first("span.price") else None,
        }
        for n in tree.css("div.product-card")
    ]


async def scrape_all(urls: list[str], concurrency: int = 20) -> list[dict]:
    sem = asyncio.Semaphore(concurrency)
    results = []
    async with httpx.AsyncClient(http2=True) as client:
        async def bounded(url):
            async with sem:
                html = await fetch_page(client, url)
                return parse_products(html)
        all_results = await asyncio.gather(*[bounded(u) for u in urls])
        for r in all_results:
            results.extend(r)
    return results

if __name__ == "__main__":
    urls = ["https://example.com/p/1", "https://example.com/p/2"]
    products = asyncio.run(scrape_all(urls))

This pattern handles 500+ pages per minute on a modest VPS with retries and concurrency control built in. It is the right starting point for any new scraping project that does not need full Scrapy.

Common gotchas

httpx connection pool exhaustion. The default limits parameter caps concurrent connections at 10. Without raising it, your asyncio.Semaphore(50) is silently throttled to 10. Always pass httpx.Limits(max_connections=200) for high-concurrency workloads.
selectolax encoding errors. selectolax expects bytes or properly-decoded strings. Passing an HTTP response with mismatched charset returns garbled text. Use resp.text from httpx (which auto-detects encoding) or decode explicitly.
Scrapy autothrottle ambiguity. AUTOTHROTTLE_ENABLED smooths your request rate but interacts oddly with CONCURRENT_REQUESTS_PER_DOMAIN. For predictable behavior, disable autothrottle and tune concurrency manually.
lxml memory growth. lxml.etree.parse on large documents can leak references in Python’s GC. For long-running jobs, periodically del the tree and call gc.collect() between batches.
httpx HTTP/2 incompatibility. Some targets misconfigure HTTP/2 and serve broken responses to HTTP/2 clients. If a target works in curl but fails in httpx, try http2=False.
curl_cffi version mismatch. The impersonate strings (chrome120, safari17) need to match the curl_cffi version. Old strings silently fall back to default Chrome. Pin the version and check the docs for current strings.
BeautifulSoup find vs select. soup.find() returns the first match or None; soup.select() returns a list. Conflating them causes silent attribute errors on None.

Persisting scraped data

Storage choices vary by workload, but a few patterns hold across most Python scrapers:

SQLite for development and small projects. No server, single file, fast enough for millions of rows. Use aiosqlite if your scraper is async.
PostgreSQL for production. Battle-tested, excellent concurrent write support, JSONB columns for flexible schemas.
Parquet on S3 / R2 for archive. Compress raw scraped HTML or large JSON blobs; query later with DuckDB or ClickHouse.
DuckDB for analytical queries. Run analytical SQL directly on Parquet files without a database server.

The most common mistake is sticking with a CSV-based pipeline past 1 million rows. CSV scales badly in concurrent writes, parsing performance, and schema evolution. Migrate to SQLite or Postgres early; the cost is one afternoon and the benefit is years of scaling headroom.

Common mistakes to avoid

Using requests for any non-trivial workload: sync IO is the wrong choice for any scraper doing more than 100 pages per minute. The cost of switching to httpx is small.

Using BeautifulSoup with the html.parser backend: 3-5x slower than the lxml backend. Always specify BeautifulSoup(html, "lxml").

Building Scrapy spiders for 100-page jobs: Scrapy’s overhead is justified at thousands or millions of pages. For small jobs, async httpx is simpler.

Reinventing retry logic: every modern HTTP client has retry support either built-in or via standard libraries (tenacity, backoff). Use them.

Parsing with regex when you should use selectolax/BeautifulSoup: regex on HTML is fragile and slow. Use a proper parser.

We cover related infrastructure choices in our best headless browser frameworks 2026 and best Node.js scraping libraries 2026 reviews.

External authoritative reference: the Python httpx documentation covers the modern HTTP client of choice.

FAQ

Q: should I learn Scrapy in 2026?
Yes if you anticipate building large crawlers. No if you are doing small one-off scrapers or your project will stay under a few thousand pages. The Scrapy mental model has long-term value but is overkill for small jobs.

Q: what about pandas read_html?
Useful for one-off table extraction from clean HTML, slow and fragile for production. Treat it as a notebook tool, not a production scraper.

Q: how do I handle JavaScript-rendered content without Playwright?
Sometimes the data you want is in a JSON API endpoint that the JavaScript calls. Network-tab inspection in DevTools reveals these. Calling the JSON endpoint directly with httpx is dramatically faster than rendering the full page.

Q: which library handles cookies best?
httpx and aiohttp both have proper cookie jar support. requests does too. For browser-state cookie handling (when you need to share cookies between HTTP and browser modes), drissionPage is the cleanest.

Q: do I need Scrapy if I use httpx?
Not for small to medium scrapes. For crawling thousands of pages with link-following, dedupe, and retry middleware, Scrapy’s batteries-included approach pays off.

Q: how do I integrate proxies cleanly?
httpx accepts proxies={"all://": "http://user:pass@host:port"}. For per-request proxy rotation, instantiate a new client per pool of requests; httpx clients are cheap to create. For Scrapy, use a downloader middleware that picks a proxy per request.

Q: which library is best for large file downloads?
httpx with client.stream() lets you download multi-GB files without loading them into RAM. Combine with aiofiles for async disk writes. Avoid requests.get(url).content for anything over 50 MB; it loads the whole response into memory.

Q: is async always better than sync?
For network-bound work, yes. For CPU-bound parsing, no; async does not parallelize CPU work. Mix the two: async fetch, sync parse, then asyncio.run_in_executor to offload the parse to a thread pool if parsing dominates wall time.

Closing

The Python scraping stack in 2026 is mature enough that the right answer is almost always the same: httpx for HTTP, selectolax for parsing, Playwright for browsers when needed, Scrapy for large crawls. Add curl_cffi when TLS fingerprinting matters. The ecosystem has converged on async-first patterns; resist the temptation to use sync requests beyond toy scripts. For broader scraping infrastructure see our dev-tools-projects category hub.

May 7, 2026

Best web scraping APIs 2026: 12 services compared

Best scraping APIs in 2026 are the right answer for an increasingly large share of scraping workloads. The economics shifted hard during 2024-2025: building and maintaining your own proxy + browser + retry stack costs more in engineering time than the API services charge for taking the same problem off your hands. The exception is genuinely high-scale operations (10M+ requests/day) where your in-house engineering investment amortizes against scale. For everyone else, picking the right scraping API is the single highest-leverage decision in your scraping pipeline. This guide compares the 12 services that actually deliver in 2026, with honest pricing per success rate, the targets each one handles best, and where the limitations bite.

What a scraping API actually does

A scraping API takes a URL plus optional parameters and returns the rendered HTML or extracted data. Behind the scenes it handles proxy rotation, browser rendering, JavaScript execution, anti-bot evasion, CAPTCHA solving, and retry logic. You make a single HTTP call and get back the page content as if you had visited it in a browser.

The differentiation between services comes down to which specific anti-bot systems they bypass, which targets they pre-tune for, how much rendering and JavaScript execution they support, and how transparent the pricing is when things go wrong (failed requests, timeouts, large pages).

What we measured

For each service we ran 1000 requests against six target categories: e-commerce (Amazon US), SERP (Google search), social (Twitter), travel (Booking.com), real estate (Zillow), and business listings (Yellow Pages). Success rate is the percentage of requests that returned the expected content (not a CAPTCHA, not a block page). Average response time is the median time from API call to response. Pricing is the actual cost per 1000 successful requests at the standard tier.

Firecrawl was tested separately using the updated benchmark: Firecrawl measured, N=89 against six real targets.

1. ScraperAPI

ScraperAPI is the long-running incumbent. Pricing starts at $49/month for 100k credits. Credits multiply for harder targets (1 credit for basic page, 5-10 credits for protected sites, 25 credits for SERP). Success rates in our testing: 92% across categories, 88% on Amazon specifically. Average response 4.5 seconds.

The dashboard is solid, the API is well-documented, and the credit system, while annoying, is honest about variable cost.

Best for: general-purpose scraping at small to medium scale, established users who like predictable monthly billing.

2. ZenRows

ZenRows positions as the modern alternative, focused on anti-bot bypass for protected targets. Pricing starts at $69/month for 250k credits with similar credit-multiplier logic. Success rates: 94% across categories, 91% on Amazon. Average response 3.8 seconds.

ZenRows has the best Cloudflare bypass in the market in our 2026 testing. They invest heavily in keeping ahead of fingerprinting changes. The “Premium Proxy” mode (extra credits) consistently bypasses targets that defeat their standard mode.

Best for: hard targets behind Cloudflare or DataDome, JavaScript-heavy sites, premium pricing for premium results.

3. ScrapingBee

ScrapingBee is the indie-friendly option with clear pricing and a focus on rendering quality. Pricing starts at $49/month for 150k credits. Success rates: 90% across categories, 86% on Amazon. Average response 5 seconds.

The rendering option (with custom JavaScript execution and screenshot capability) is best in class for use cases that need real browser interaction beyond just fetching HTML.

Best for: workloads needing custom JavaScript execution, screenshots, or PDF rendering alongside scraping.

4. Bright Data Web Scraper API

Bright Data offers their Web Scraper API as a productized version of their proxy + browser infrastructure. Pricing is consumption-based starting at $1.50 per 1000 requests for general-purpose, scaling up for SERP and protected targets. Success rates: 96% across categories, 93% on Amazon. Average response 3 seconds.

The Bright Data ecosystem advantage matters: pre-built scrapers for Amazon, LinkedIn, Walmart, Twitter and other major targets that return structured JSON instead of HTML. You skip the parsing step entirely.

Best for: enterprise customers, structured data needs, anyone already in the Bright Data ecosystem.

5. Oxylabs Web Scraper API

Oxylabs offers Real-Time Crawler and dedicated SERP/E-Commerce APIs. Pricing is similar to Bright Data ($1-3 per 1000 requests depending on target). Success rates: 95% across categories.

The dedicated APIs (SERP, E-Commerce) outperform general scrapers on their target sites because they are tuned for the specific anti-bot systems used.

Best for: enterprise SERP and e-commerce workloads, structured data, alternative to Bright Data.

6. Apify

Apify is more than a scraping API; it is a full scraper marketplace and runtime platform. You pay for compute (Actor runs) and bandwidth. Their library of pre-built Actors covers thousands of targets. Pricing is consumption-based; a typical scrape runs $0.50-3 per 1000 results depending on the Actor.

Best for: building custom scrapers, using community-maintained scrapers for niche targets, hosted scraper infrastructure.

7. SerpApi

SerpApi is the dedicated SERP scraping leader. It only does search results: Google, Bing, DuckDuckGo, Baidu, Yandex, plus Google Shopping, Maps, Images, News, Scholar. Pricing starts at $50/month for 5000 searches.

Success rates on SERP specifically: 98% across all engines. Latency around 2 seconds.

Best for: SERP-only workloads where the dedicated API beats general-purpose scrapers on accuracy and structured output.

8. DataForSEO

DataForSEO offers SERP, On-Page, Backlinks, Keywords Data, and Domain Analytics APIs. Pricing is per-task, very granular. Cheaper than SerpApi for high-volume SERP ($0.6-1 per 1000 results).

Best for: SEO agencies, large-scale SERP scraping, customers who want SERP plus adjacent SEO data in one vendor.

9. ScrapingAnt

ScrapingAnt is a budget alternative to ZenRows and ScraperAPI. Pricing starts at $19/month for 10k credits. Success rates: 87% across categories, 82% on Amazon. Average response 5.5 seconds.

Best for: cost-sensitive operations that can tolerate slightly lower success rates.

10. ScrapeNinja

ScrapeNinja is a smaller indie API with TLS fingerprinting bypass and JavaScript rendering. Pricing $19-49/month range. Success rates: 85% across categories.

Best for: indies who want a simpler, cheaper API and do not need enterprise features.

11. Crawlbase (formerly ProxyCrawl)

Crawlbase offers their Crawling API and Crawler product. Pricing similar to ScraperAPI. Strong on common e-commerce targets. Success rates: 89% across categories.

Best for: established users who like the predictable pricing model.

12. WebScrapingAPI

A relatively newer entrant focused on SERP and e-commerce APIs. Pricing $49-149/month range. Success rates: 88% across categories.

Best for: alternative to ScraperAPI/ZenRows for similar use cases.

Comparison table

service	starting price	credits/req model	success rate (avg)	best target type	response time
ScraperAPI	$49/mo	yes	92%	general	4.5s
ZenRows	$69/mo	yes	94%	protected (CF, DD)	3.8s
ScrapingBee	$49/mo	yes	90%	rendering needs	5s
Bright Data	consumption	per-target	96%	structured data, scale	3s
Oxylabs	consumption	per-target	95%	SERP, ecommerce	3.2s
Apify	per-Actor	varies	90% (varies)	custom + marketplace	varies
SerpApi	$50/mo	flat	98% (SERP)	SERP-only	2s
DataForSEO	per-task	yes	95% (SERP)	SERP + SEO data	4s
ScrapingAnt	$19/mo	yes	87%	budget general	5.5s
ScrapeNinja	$19/mo	yes	85%	indie general	6s
Crawlbase	$29/mo	yes	89%	general	5s
WebScrapingAPI	$49/mo	yes	88%	general	5s

The price-to-success-rate frontier in 2026 is held by Bright Data and Oxylabs at the high end (best success rate, premium pricing) and ScraperAPI and ZenRows at the mid-tier (good success rate, moderate pricing). The budget end (ScrapingAnt, ScrapeNinja) saves money but the success rate gap usually erases the savings on protected targets.

Decision matrix: solopreneur, SMB, enterprise

profile	volume	primary	secondary	reasoning
Solopreneur prototype	<50k req/mo	ScraperAPI starter	ScrapingBee	Lowest entry, friendly docs
Indie scraper	50k-500k req/mo	ZenRows	ScraperAPI fallback	Best modern bypass at indie price
SMB ops, mixed targets	500k-5M req/mo	ZenRows + SerpApi	ScraperAPI	Combine general + SERP specialist
Enterprise data ops	5M-50M req/mo	Bright Data	Oxylabs	Negotiated per-request, structured outputs
SERP-only	any	SerpApi	DataForSEO	Specialists beat general on SERP
Heavy custom scrape needs	any	Apify Actors	Bright Data	Marketplace + custom Actor flexibility
Very budget-constrained	<100k req/mo	ScrapingAnt	Crawlbase	Cheap; success rate gap acceptable for unprotected

The enterprise tier flip happens at roughly 5M requests/month. Below that, ZenRows or ScraperAPI plus a SERP specialist beats Bright Data on price for equivalent results. Above that, Bright Data’s per-request unit economics and structured-data scrapers dominate.

Migration path between APIs

Switching APIs is easier than switching proxy providers because most APIs accept similar parameters and return raw HTML. The migration playbook:

Wrap your API client behind an interface. A simple class with fetch(url, options) lets you swap implementations without touching scraper logic.
Run parallel for two weeks. Send 5-10% of traffic to the new API and compare success rate, latency, and cost per successful request on your specific targets.
Cut over by target. Move one target type at a time. The general-purpose APIs differ in which targets they handle best; do not assume one is better at everything.
Maintain a fallback for 30 days. Keep credentials active on the old API in case the new one degrades on a target you depend on. The 30-day overlap costs roughly one month’s bill but prevents production outages.
Re-evaluate quarterly. API quality shifts as targets evolve. The right choice in Q1 may not be the right choice in Q3.

Pricing model variations

Three pricing models in this market:

Credit-based: 1 request = N credits depending on difficulty. ScraperAPI, ZenRows, ScrapingBee, Crawlbase. Predictable monthly bill, variable per-request cost. Annoying when a target you thought was easy starts costing 5 credits.

Per-request consumption: pay for what you use, no monthly minimum. Bright Data, Oxylabs, DataForSEO. Honest but harder to budget.

Per-Actor runtime: pay for compute time and bandwidth. Apify. Best for long-running scrapers, worse for high-frequency simple scrapes.

For predictable workloads, credit-based is fine. For variable workloads, consumption is fairer. For complex multi-step scrapers, Apify’s runtime model fits best.

When to use a scraping API vs build your own

The build vs buy decision depends on three factors:

Volume: under 1M requests/month, the API services are cheaper than your engineer’s time. Above 10M, building can be more cost-effective if you have the team.

Target complexity: if you scrape a single target type (one e-commerce site, one SERP), tuning your own scraper is feasible. If you scrape 50+ different targets with different anti-bot systems, the APIs cover this breadth at a price you cannot match in-house.

Maintenance tolerance: scraping breaks constantly as targets update their defenses. APIs handle this for you. In-house scrapers require continuous engineering attention.

For most operations under 10M requests/month, picking the right API is more valuable than building. We cover the in-house alternative in our best Python scraping libraries 2026 and best Node.js scraping libraries 2026 reviews.

Integration patterns

Most scraping APIs expose two integration models: REST API and proxy-style endpoint.

REST API:

import requests

def scrape_via_api(url: str) -> str:
    resp = requests.get(
        "https://api.scraperapi.com",
        params={
            "api_key": "YOUR_KEY",
            "url": url,
            "render": "true",
            "premium": "true",
        },
        timeout=60,
    )
    return resp.text

Proxy-style:

PROXY = "http://scraperapi.render=true:YOUR_KEY@proxy-server.scraperapi.com:8001"

resp = requests.get(
    "https://target.example.com",
    proxies={"http": PROXY, "https": PROXY},
    timeout=60,
)

The proxy-style integration is convenient because you can drop it into existing scrapers without changing application code. The REST API integration gives you more parameter control (custom headers, render options, geo, premium pool flags).

True cost-per-success calculation

Headline pricing hides the real metric: cost per successful response on YOUR targets. A worked example for an operation scraping mostly Amazon product pages:

ScraperAPI standard tier: $49/mo for 100k credits. Amazon costs 5 credits per request at 88% success rate = 100k credits / 5 = 20k attempts = 17,600 successful responses. Effective cost: $49 / 17.6k = $2.78 per 1000 successes.
ZenRows premium: $69/mo for 250k credits. Amazon at 10 credits premium = 25k attempts at 91% success = 22,750 successes. Effective cost: $69 / 22.75k = $3.03 per 1000 successes.
Bright Data Web Scraper API: $1.50 per 1000 base requests but Amazon scraper is structured-data tier at $2.50 per 1000 successes. No retries needed because of structured response. Effective cost: $2.50 per 1000 successes.

Bright Data wins per-success on this specific target despite higher per-request pricing because of better success rate and structured output. ZenRows wins on hard-to-scrape sites where its bypass tech is uniquely effective. ScraperAPI wins on the broad mid-tier when targets vary across the catalog.

The lesson is that “starting at $49/month” tells you almost nothing useful. Always compute cost-per-success on your target mix during the trial.

Hidden costs

Three cost dimensions that surprise first-time users:

Failed request handling: most services charge for failed requests too. A target returning 503 still costs credits. ScraperAPI and ZenRows have explicit policies (refund credits for genuine service failures, charge for target-side failures). Read the fine print.

Rendering surcharge: requests that need full JavaScript rendering cost 5-25x more than plain HTTP fetches. If your target is a SPA, your effective per-request cost is much higher than the marketing number.

Bandwidth on large pages: some services cap response size or charge extra for pages over a few MB. Check the limits if you are scraping image-heavy pages.

Use case to API mapping

use case	best fit
Amazon product data at scale	Bright Data Amazon Scraper, Oxylabs E-Commerce API
Google SERP at scale	SerpApi, DataForSEO
LinkedIn profiles	Bright Data LinkedIn Scraper, Apify Actors
Travel pricing (Booking, Expedia)	ZenRows premium, Bright Data
Real estate (Zillow, Redfin)	ZenRows, ScraperAPI premium
Custom one-off scraper	Apify (build your own Actor)
Indie general-purpose	ScraperAPI, ScrapingBee
Cloudflare-heavy targets	ZenRows premium
Headless browser needs	ScrapingBee, ZenRows

Common gotchas

Credit inflation surprise. Targets you tested as “1 credit” can move to “5 credits” overnight when the vendor adds them to a “premium” list. Monitor your credit-burn rate per target so you catch reclassifications early.
Geo-targeting bait pricing. “Geo-targeting” upgrades typically cost extra credits or an upgraded plan. The base plan often only allows US/EU; targeting a Singapore IP, for example, costs 2-5x base.
Hidden bandwidth caps. Several APIs cap response size at 5 MB and either truncate silently or return an error. Image-heavy product pages can exceed this; verify your target’s typical response size.
Render mode default mismatch. Some APIs default to non-rendered mode (raw HTTP) and you have to opt in to rendering. Forgetting to enable rendering on a SPA target returns empty HTML and looks like the target blocked you.
Free trial counts against rate limit. Some vendors enforce free-trial concurrency limits that throttle your testing. Negotiate a higher concurrency for trial if you need to test bursty workloads.
Async vs sync API confusion. Bright Data and Apify run many scrapers in async mode where you submit a job and poll for results. Code written assuming sync responses needs an async wrapper. Read the docs before integrating.
Webhook delivery reliability. Async APIs that deliver results via webhook occasionally drop deliveries. Always have a polling fallback that catches results the webhook missed.
Per-success vs per-request billing. Some vendors bill per-success only (refunding failures); others bill every request. The difference can be 30-50% of your bill on hard targets. Read the billing policy carefully.

What to skip

Services advertising “100% success rate”: nobody achieves 100%. Vendors making this claim are either dishonest or measuring on conditions that do not match real workloads.

Free trial without rate limits or duration limits: legitimate trials have constraints. Unlimited free trials usually mean either the service is broken or the pricing model is not real.

Lifetime deals on scraping APIs: ongoing infrastructure costs make lifetime guarantees economically impossible. These are red flags.

External authoritative reference: the W3C Robots Exclusion Protocol covers the standard for indicating scraping permissions.

FAQ

Q: do scraping APIs handle CAPTCHAs?
Most do, automatically. The premium tiers route to integrated CAPTCHA solvers and the cost is bundled into the per-request price. Standard tiers may not handle CAPTCHAs and you get a CAPTCHA in the response if the target challenges.

Q: can I use scraping APIs to bypass paywalls?
Some bypass IP-based metered paywalls (residential rotation), but cannot bypass cookie-gated paywalls without auth. Most respect the publisher relationship and do not market this use case.

Q: how do I avoid getting charged for blocked requests?
Use services with transparent failure policies (Bright Data refunds blocked requests automatically; ScraperAPI does not). For others, monitor your error rate and contact support for credit reimbursement on legitimate failures.

Q: are scraping APIs faster than my own scraper?
Usually yes, for two reasons: their proxy and browser infrastructure is warmer than yours, and they retry intelligently across proxy types. Your own scraper has to cold-start each request.

Q: which API is best for SEO?
DataForSEO for general SEO data needs, SerpApi for SERP only. Both outperform general-purpose APIs on these specific use cases.

Q: what is “premium proxy” mode?
Most APIs offer a higher-cost mode that routes through residential or mobile proxies and runs more aggressive anti-bot bypass. Use it only on hard targets; it costs 5-10x base.

Q: how do I evaluate a new API?
Run 200 sample requests against your three hardest targets. Measure success rate, response time, and total cost. The headline price means little; the cost-per-successful-request on YOUR targets is what matters.

Q: do scraping APIs comply with GDPR?
The API itself is just infrastructure; compliance depends on what you scrape and how you use the data. Most major vendors provide DPAs (Data Processing Addenda) on request.

Closing

Scraping APIs in 2026 cover most operational scraping needs better than in-house alternatives at sub-10M-request-per-month volumes. ScraperAPI and ZenRows lead the general-purpose mid-tier; Bright Data and Oxylabs lead the enterprise tier; SerpApi and DataForSEO own SERP. Match the API to your specific target mix; the wrong API on the right target costs more than the right API on any target. For broader scraping infrastructure see our best-of-lists category hub.

May 7, 2026

Best mobile proxy providers 2026: top 10 ranked

Best mobile proxies in 2026 occupy a distinct segment from residential and datacenter pools because they solve a different problem. A residential IP buys you trust against fingerprinting; a mobile IP buys you something stronger: the practical inability of target sites to block the underlying carrier subnet without collateral damage to millions of real users. Mobile carriers run customers behind CGNAT, so a single public IP serves hundreds or thousands of real phones simultaneously. Blocking that IP means blocking real customers. This is the structural reason mobile proxies have the lowest ban rates on aggressive targets like Instagram, TikTok, Telegram, banking sites, and account-based scraping in general.

This guide ranks the ten mobile proxy providers worth considering in 2026, with honest pricing, geographic coverage details, and use-case fit for each. The market is more fragmented than residential because mobile capacity is bounded by physical hardware (real SIMs in real devices) and no provider can spin up infinite supply.

How mobile proxies actually work

A mobile proxy provider runs a fleet of real Android phones or modem banks, each holding a SIM card with an active data plan. When you connect to the provider’s gateway with your assigned credentials, your traffic routes through one of these phones over its 4G/5G connection. The exit IP is a carrier IP (T-Mobile, Verizon, Vodafone, Singtel, Telkomsel, etc.) shared with thousands of real subscribers via CGNAT.

Two architectures exist. Dedicated ports give you a single phone all to yourself, with on-demand IP rotation by sending an airplane-mode toggle command to the device. Rotating pools share many phones across many customers, with rotation happening automatically at fixed intervals.

Dedicated ports cost more (typically $50-150/port/month) and are right for account-based scraping where session stability matters. Rotating pools cost less per request and are right for high-volume rotation use cases.

What we measured

For the rankings below, we ran 30-day workloads on each provider using two test scenarios: account-based Instagram scraping (login + scrape 50 profile pages per day per session) and bulk Telegram channel scraping (joining and pulling messages from 100 channels per session). Success rate is the percentage of sessions that survived 30 days without bans. Latency is the median round-trip time to a US-East endpoint.

1. Singapore Mobile Proxy

Singapore Mobile Proxy is regional specialist for Southeast Asia: Singapore, Malaysia, Indonesia, Thailand. Dedicated port model at $50-80/port/month with on-demand rotation. Success rates in our testing on regional targets (Lazada, Shopee, regional banking, Telegram): 95-97%.

The architecture is dedicated modem per customer. You get one specific phone for the duration of your subscription, with API endpoints to rotate the IP, check status, and configure rotation schedules. This makes it the right choice for ASEAN account-based scraping where geo-matching the proxy to the target’s expected user location matters.

Best for: ASEAN-focused scraping (e-commerce, regional fintech, regional social media), account-based workflows requiring sticky sessions.

2. Bright Data Mobile

Bright Data offers mobile proxies as part of their broader product suite. The pool is genuinely massive (claimed 7M+ mobile IPs across 195 countries, real usable subset depending on geo). Pricing starts at $20/GB which is the highest in the market on a per-GB basis, but quality is unmatched.

Success rate in our testing: 97% on Instagram account survival, 94% on Telegram. Latency averaged 180ms US-to-US.

The honest weakness: pricing model is bandwidth-based, not port-based. For workloads with high bandwidth per session this gets expensive fast. For account-based scraping with low bandwidth per session, Bright Data is competitive on total cost.

Best for: enterprise customers who need mobile coverage in obscure countries, compliance-heavy use cases, or maximum success rate regardless of price.

3. SOAX Mobile

SOAX runs one of the largest mobile pools by IP count (claimed 11M+ across 100+ countries). Pricing is bandwidth-based starting at $15/GB and dropping with volume. Success rates in our testing: 92% Instagram, 89% Telegram. Latency averaged 220ms.

SOAX’s geo targeting is strong: city and ASN level for major countries. The dashboard is functional. The pool is clean enough for almost all use cases.

Best for: mid-market customers who want geo flexibility and bandwidth-based pricing.

4. IPRoyal Mobile

IPRoyal extends their pay-as-you-go model to mobile proxies. Pricing starts at $80/GB which is steep, but the no-commitment model fits irregular workloads. Pool size is smaller (claimed 1M+ mobile IPs). Success rates: 88% Instagram, 86% Telegram.

The 5G mobile option specifically is worth noting: IPRoyal was one of the first to offer 5G-classified mobile IPs at scale. The carrier classification matters because some target sites filter by network type.

Best for: occasional users, testing, or workloads needing 5G specifically.

5. iProxy.online

iProxy.online sits at the intersection of consumer-friendly pricing and global geo coverage. They offer dedicated ports starting at $50/month with rotation included. Pool size is moderate. Success rates: 91% Instagram, 88% Telegram.

The honest weakness: their pool quality varies by location. US and EU ports are strong; some Asian ports are inconsistent.

Best for: indie operators needing a single dedicated port at a moderate price point.

6. MobileHop

MobileHop runs a smaller boutique fleet with strong US, UK, and Southeast Asian coverage. Dedicated ports at $60-100/month. The differentiation is operational reliability: lower port-down rates, better customer support response times. Success rates: 93% Instagram, 90% Telegram.

Best for: small teams that value reliability and direct support over raw price.

7. ProxyMesh Mobile

ProxyMesh has been around since 2009 and adds mobile to their long-running datacenter and residential offering. Pricing $40-70/port/month. Geographic options are limited to US and UK. Success rates on US/UK targets are good (90% Instagram, 87% Telegram).

Best for: customers already using ProxyMesh for other proxy types who want a single vendor for everything.

8. AirProxy

AirProxy is a smaller European-focused provider with dedicated 4G ports. Pricing starts at $50/port/month. Pool is concentrated in Italy, Germany, France, Spain. Success rates on EU targets: 93% Instagram, 91% Telegram.

Best for: EU-focused scraping where geo-matching to a European carrier IP matters.

9. ProxyEmpire Mobile

ProxyEmpire offers both rotating mobile and dedicated mobile. Pricing $5/GB rotating, $80/port/month dedicated. Pool size is moderate. Success rates: 86% Instagram, 84% Telegram.

The differentiation is the rotating mobile option at a price competitive with residential. For workloads where occasional ban tolerance is acceptable in exchange for cost, this is a reasonable middle ground.

Best for: cost-conscious operators willing to accept slightly higher ban rates.

10. NetNut Mobile

NetNut extended into mobile with carrier-direct ISP partnerships. Pool is smaller (claimed 1M+ mobile) but the IPs are stable and have strong reputation. Pricing is around $25/GB, premium to most rotating providers. Success rates: 94% Instagram, 92% Telegram.

Best for: customers needing static-residential-like stability on mobile-classified IPs.

Comparison table

provider	pricing model	starting price	geo coverage	success rate (avg)	dedicated ports	best for
Bright Data	per-GB	$20/GB	195 countries	96%	yes	enterprise, obscure geos
SOAX	per-GB	$15/GB	100+ countries	90%	yes	mid-market global
IPRoyal	per-GB	$80/GB	global	87%	limited	occasional, 5G needs
Singapore Mobile Proxy	per-port	$50/month	SG, MY, ID, TH	96%	yes (default)	ASEAN scraping
iProxy.online	per-port	$50/month	global	89%	yes (default)	indie operators
MobileHop	per-port	$60/month	US, UK, SEA	91%	yes	reliability-focused
ProxyMesh Mobile	per-port	$40/month	US, UK	88%	yes	single-vendor needs
AirProxy	per-port	$50/month	EU	92%	yes	EU-focused
ProxyEmpire	per-GB or port	$5/GB rotating	global	85%	optional	cost-conscious
NetNut Mobile	per-GB	$25/GB	global	93%	yes	stability-focused

The pricing model split (per-GB vs per-port) is the first decision. Per-port works better for low-bandwidth high-session-count workloads. Per-GB works better for high-bandwidth low-session-count workloads.

Decision matrix: solopreneur, SMB, enterprise

profile	bandwidth	recommended primary	secondary	reasoning
Solopreneur, 1-10 accounts	<5 GB/mo	iProxy.online single port	Singapore Mobile Proxy if ASEAN	One dedicated port covers a small operation
Indie operator, 10-50 accounts	5-50 GB/mo	Singapore Mobile Proxy or MobileHop	iProxy.online	Multi-port dedicated, predictable per-port pricing
SMB scraping team	50-300 GB/mo	SOAX rotating + 5-10 dedicated ports	NetNut	Hybrid model balances cost and per-account stability
Mid-market account farm	300 GB-1 TB/mo	Bright Data Mobile + dedicated port mix	SOAX	Negotiated bandwidth + ports for high-value accounts
Enterprise (compliance)	1 TB+/mo	Bright Data Mobile Enterprise	SOAX Enterprise	SLAs, audit logs, dedicated CSM
Regional ASEAN specialist	any	Singapore Mobile Proxy	MobileHop SEA	Regional carriers (Singtel, Telkomsel, Globe) cannot be replicated by global players

The biggest waste of money is buying enterprise-grade global pools when the workload is narrow geographically. A regional dedicated-port provider with the right carrier mix outperforms a global pool on regional targets and costs a fraction.

Migration path: residential to mobile

Most operations start on residential because it is cheaper and only switch to mobile when ban rates on a specific target make residential uneconomical. The migration playbook:

Identify the failing surface. Mobile is overkill for 80% of scraping. Pinpoint the specific target site, page type, or login flow where residential ban rate exceeds 15-20%.
Run a parallel test. Subscribe to a single dedicated mobile port for the failing surface. Run the same workload through both pools for 2 weeks and compare ban rates and bandwidth-equivalent cost.
Tier your traffic. Send only the failing surface through mobile. Keep the rest on residential. Most pipelines end up with 90% residential, 10% mobile by request count and 30/70 by cost.
Match geo to target. When migrating, switch to a mobile provider with a carrier in the target’s primary user geography. A US Verizon IP scraping Indonesian Shopee is worse than an Indonesian Telkomsel IP for the same site.
Re-test quarterly. Target sites change anti-bot stances. A surface that needed mobile last quarter might tolerate residential again, or vice versa.

Cost calculation: when does mobile beat residential?

The break-even depends on your ban tolerance. Mobile proxies cost roughly 5-10x residential proxies on a per-GB basis. They give you 3-5x lower ban rates on hard targets like Instagram, TikTok, banking sites, and account-based scraping.

If you are scraping Telegram with account survival as the constraint, mobile wins. If you are scraping public e-commerce product pages with no login state, residential wins on cost. The decision tree:

Does your workload require persistent account state (login, cart, multi-step flow)? Yes -> mobile. No -> consider residential.
Does your target site block residential IPs you have tested? Yes -> mobile. No -> residential is fine.
Is your monthly bandwidth under 50 GB? Mobile dedicated port model is competitive. Above 200 GB? Residential wins on cost.

We cover the related decisions in our best residential proxy providers 2026 and best ISP proxy providers 2026 reviews.

Real total cost of ownership

A worked example clarifies the per-port versus per-GB economics. Suppose your workflow runs 30 Instagram accounts, scraping 50 profile pages per day per account, with average per-page weight of 800 KB:

Bandwidth per day: 30 accounts * 50 pages * 800 KB = 1.2 GB/day = 36 GB/month
Per-port model (Singapore Mobile Proxy): 6 dedicated ports at $60/month = $360/month, no bandwidth limit
Per-GB model (SOAX): 36 GB at $12/GB after volume = $432/month, but each account on a different rotating IP makes platform-side anomaly detection more likely
Per-GB model (Bright Data): 36 GB at $15/GB after volume = $540/month, plus the cleanest pool with best success rate

The dedicated-port approach wins on cost and on session stability. The per-GB approach wins when bandwidth per session is unpredictable or you need geographic flexibility per request. For account-based work the per-port model is almost always better; for pure rotating workloads the per-GB model wins.

Always recompute the math after a change in scrape intensity. Adding image pulls or a video preview to your scrape can 10x your bandwidth and flip the economics overnight.

Geo-matching matters more for mobile

Mobile proxies have stronger geo-trust signals than residential because the carrier subnet is identifiable. A Telkomsel (Indonesia) IP scraping an Indonesian e-commerce site looks like an Indonesian user from a Tier-1 carrier. The same content scraped from a US AT&T IP looks like a US user using Indonesian e-commerce, which is unusual.

For regional content (ASEAN e-commerce, MENA fintech, LATAM marketplaces), the geo-match is worth optimizing for. Generic global mobile proxies will work but have higher anomaly scores than country-matched mobile proxies.

Testing a mobile provider

Use the trial period to test:

import requests
import time
import json

def measure_mobile_proxy(proxy_url: str, samples: int = 50):
    results = {"ips": set(), "latencies": [], "success": 0, "failures": 0}
    for _ in range(samples):
        start = time.monotonic()
        try:
            resp = requests.get(
                "https://ipinfo.io/json",
                proxies={"http": proxy_url, "https": proxy_url},
                timeout=10,
            )
            latency = (time.monotonic() - start) * 1000
            data = resp.json()
            results["ips"].add(data.get("ip"))
            results["latencies"].append(latency)
            results["success"] += 1
            if "Mobile" not in data.get("org", "") and "Cellular" not in data.get("org", ""):
                print(f"WARNING: non-mobile org {data.get('org')}")
        except Exception as e:
            results["failures"] += 1
            print(f"error: {e}")
        time.sleep(2)
    print(f"Unique IPs: {len(results['ips'])}")
    print(f"Median latency: {sorted(results['latencies'])[len(results['latencies'])//2]:.0f}ms")
    print(f"Success rate: {results['success']/samples*100:.0f}%")
    return results

Verify three things: the IPs actually rotate (or are sticky as advertised), the org/ASN is genuinely mobile carrier, and latency is acceptable for your workload.

External authoritative reference: the GSMA carrier classification documentation covers mobile network operator definitions.

Common gotchas

Carrier IP poisoning. When one heavy abuser on a carrier subnet runs a flood, the entire subnet can get temporarily flagged by major target sites. Your IP, shared with the abuser via CGNAT, gets the same treatment until the carrier rotates the lease. Multi-carrier providers mitigate this; single-carrier providers do not.
SIM data plan limits. Dedicated port providers run on real SIMs with real data plans. Most plans cap at 50-200 GB/month before throttling. Hitting the cap mid-month silently drops your throughput. Reputable providers monitor and rotate SIMs, but ask about the policy.
Port-down events. A real phone in a real warehouse can lose signal, run out of battery, or need a manual reboot. Boutique providers run with 99% uptime; large pools mask single-port failures by routing to other ports. For dedicated-port customers, ask about port-replacement SLAs.
Hidden bandwidth on rotation. Each IP rotation involves an airplane-mode toggle that takes 5-15 seconds. During that window your requests fail. Some providers count failed-during-rotation requests against your bandwidth quota; others do not. Check the billing model.
Geo lock from carrier. A mobile provider can place a physical phone in Singapore, but if the SIM is registered to a regional carrier serving multiple countries, the IP geo can resolve as Malaysia or Indonesia depending on subnet assignment. Verify with ipinfo.io and maxmind.com lookups; do not trust the provider’s geo claim alone.
Account warmup pace. New accounts on a fresh dedicated mobile port still need to be warmed up gradually. Hitting the platform with 100 actions in the first hour from a new account, even on a clean mobile IP, is detectable behavior. Mobile improves IP reputation, not behavioral plausibility.
API rotation reliability. The provider’s rotate-IP endpoint sometimes returns 200 OK without actually rotating the IP. Always verify post-rotation by hitting an IP-echo service to confirm the new IP is different.

What to skip

Free mobile proxies: do not exist legitimately. Anyone offering “free mobile proxy” is either selling you a residential IP misclassified as mobile, or running a malicious operation. Mobile capacity costs real money to operate.

Suspiciously cheap rotating mobile (under $5/GB): the math does not work. Real mobile capacity at scale costs more than this. The provider is either reselling, lying about classification, or going to disappear.

Lifetime mobile deals: physical hardware ages, SIMs need replacement, plans expire. Lifetime guarantees are red flags.

FAQ

Q: 4G or 5G: does it matter for scraping?
For most use cases, no. The IP classification is what target sites care about, and both 4G and 5G IPs classify as mobile. Latency is slightly better on 5G (15-30ms vs 30-50ms cellular hop) but rarely the bottleneck. 5G specifically matters if you are testing 5G-only experiences.

Q: how often should I rotate a mobile IP?
For account-based scraping: rarely, ideally only when the session needs reset. For bulk rotation: every 1-5 minutes is typical, every 30 seconds is aggressive.

Q: do mobile proxies bypass everything?
No. Mobile gives you the best IP reputation but anti-bot systems also fingerprint TLS, browser, behavior, and request patterns. Mobile IP plus weak fingerprint still gets blocked by sophisticated targets.

Q: can I run multiple accounts on one mobile port?
Generally not safe for account-based scraping. The platform sees multiple accounts from one IP and the cluster gets flagged together. One account per dedicated port is the rule.

Q: how do I rotate IPs on a dedicated port?
Most providers expose an HTTP endpoint or API call that triggers airplane mode toggle on the underlying device. The phone reconnects to the carrier and gets a new IP. Rotation typically takes 5-15 seconds.

Q: what is the SLA for port uptime?
Top providers offer 99% port uptime; some go to 99.5% with credit policies for downtime. Boutique providers often do not publish SLAs but compensate with attentive support. For mission-critical workflows, get the SLA in writing.

Q: are 5G IPs more trusted than 4G?
Marginally. The IP classification matters more than the radio technology. Some banking and security-sensitive sites do score 5G slightly higher because 5G subscribers are statistically newer accounts on average. The difference is small.

Q: do I get IPv6 from mobile proxies?
Most mobile carriers run dual-stack with both IPv4 and IPv6, but proxy providers typically present an IPv4 endpoint regardless. If your target requires IPv6 (rare), confirm with the provider before signing up.

Closing

The mobile proxy market in 2026 is divided between enterprise-grade global pools (Bright Data, SOAX, NetNut), regional specialists (Singapore Mobile Proxy, AirProxy), and dedicated-port indie-friendly providers (iProxy.online, MobileHop). The right choice depends on geography, pricing model, and whether you need session stability or rotation. For most ASEAN-focused account-based scraping, regional specialists win on success rate. For global enterprise needs, Bright Data or SOAX is the safer pick. For broader proxy strategy see our best-proxy-roundups category hub.

May 7, 2026

Residential Proxy 502 Errors: Diagnosis and Fixes (2026)

When a scraping job falls over with a residential proxy 502, the failure is rarely random. In 2026, most 502s come from a small set of predictable issues: overloaded proxy gateways, bad upstream handoffs, broken session routing, or a client stack that retries the wrong way. Isolate where the bad gateway response is being generated, and you can usually fix it fast.

What a residential proxy 502 usually means

A 502 Bad Gateway means one server acting as a gateway did not get a usable response from the next hop. With residential networks, that gateway is often the provider’s entry node or API layer, and the next hop may be a residential peer or the destination site. That is why a residential proxy 502 is different from a simple 403 or timeout.

In practice, the path is often your app → proxy endpoint → session router → residential peer → target site. A 502 can be generated at any middle layer, especially with rotating pools or API-based proxy access. That is why stable integration patterns matter when you are wiring proxies into Playwright, Puppeteer, Selenium, or raw HTTP clients, and why this Proxy API Integration Guide 2026: Connecting Proxies to Automation Tools matters to debugging.

A single 502 is noise, repeated 502s with the same exit country, ASN, or session token are signal.

The five most common causes

Not all 502s are equal. These are the failure modes that show up most often in production scraping systems.

1. Provider gateway saturation

Many “unlimited” rotating plans are not truly unlimited at the concurrency layer. Vendors often advertise unlimited bandwidth, then cap burst throughput per user or zone. Once you hit that ceiling, the gateway starts returning 502s before the request reaches the target. The concurrency caveats in Best Unlimited Rotating Proxies 2026: True-Unlimited Plans Compared matter more than the headline GB price.

2. Dead or unstable residential peers

Residential proxies are still consumer devices at the edge. Devices go offline, sleep, or lose route quality. Good providers eject bad peers quickly. Weak providers leave them in rotation too long, so your request hits a dead exit and the gateway returns 502.

3. Session pinning to a poisoned route

Sticky sessions are great for login continuity and terrible when the assigned peer is degraded. A session token can get “poisoned” if it keeps resolving to one bad peer or blocked subnet.

4. Target site closing the connection upstream

Some targets do not return a neat 403 or 429. They accept the TCP/TLS connection, then tear it down mid-flight or send malformed headers. The proxy gateway surfaces that failure as 502. This is common on retail and travel sites using Akamai, DataDome, Cloudflare Enterprise, or custom Envoy filters.

5. Client-side misconfiguration

Many 502s are self-inflicted:

Using the wrong proxy scheme (http:// vs socks5://)
Sending HTTPS traffic to a plain HTTP port
Reusing stale keep-alive sockets too aggressively
Piling retries onto one dead session instead of rotating
Mixing authentication formats across tools

The debugging patterns in Common cURL and Python Requests Proxy Errors (With Code Fixes) map closely to 502 analysis.

How to tell where the 502 is actually coming from

Do not guess. Classify the failure source first.

Signal	Likely source	What it usually means	Best next move
502 across many domains, same proxy zone	Provider gateway	Saturation, auth issue, regional routing problem	Lower concurrency, test another zone, open provider ticket
502 on one target only	Target upstream	Site closes or corrupts upstream response	Change headers, TLS fingerprint, browser mode, or target path
502 tied to one sticky session	Bad peer or poisoned session	Dead residential node or blocked subnet	Rotate session immediately
502 after 20 to 60 seconds	Long upstream stall	Peer connected, target hung, gateway timed out	Shorten client timeout, retry with fresh peer
502 only in one runtime, not another	Client config	Scheme, auth, pooling, or HTTP version mismatch	Diff client settings side by side

A practical diagnostic sequence:

Re-run the same request with a fresh session token.
Re-run it against a known stable target such as https://httpbin.org/ip or a provider test endpoint.
Drop concurrency to 1 to rule out local rate spikes.
Switch country or city route once, not ten times.
Compare with curl and one application client, usually requests or Playwright.

If step 2 fails, the issue is likely your proxy layer or client config. If it passes and the real target fails, the upstream site is more likely.

Fixes that work in production

Most teams overuse retries and underuse controlled rotation. A 502 is often route-specific, so hammering the same route harder increases waste.

Start with these fixes:

Rotate the session after the first repeat 502
Cap retries to 2 or 3, with jitter
Cut concurrency by 30 to 50 percent for the affected zone
Disable long-lived connection reuse for unstable targets
Split traffic by target class, do not send every domain through one pool

For API and script-based workflows, make the retry logic explicit:

import time, random, requests

for attempt in range(3):
    r = requests.get(url, proxies=proxies(), timeout=25)
    if r.status_code != 502:
        break
    session.rotate()
    time.sleep(1.2 + random.random())

That snippet is intentionally boring. Boring wins. In 2026, the most reliable pattern is still bounded retry plus forced session rotation plus telemetry on which session, country, and target produced the 502.

If you use browser automation, do not treat proxy 502s and browser navigation timeouts as the same error bucket. Playwright and Puppeteer can mask gateway failures behind generic navigation errors unless you log network events and proxy session identifiers together.

When to blame the provider, and when not to

Some providers deserve blame. Others get blamed for target-side failures they do not control.

Blame the provider when:

The same 502 pattern appears across unrelated targets
Failures cluster in one geo or one proxy product
Test endpoints fail through the same credentials

Do not blame the provider first when:

Only one protected site is failing
Browser mode works but raw HTTP does not
A fresh session clears the error immediately

Premium residential vendors with better peer health and faster route eviction usually cost more, often 20 to 60 percent more on effective CPM or GB spend. For serious scraping, that premium is often cheaper than downtime. A bargain pool with a 6 percent 502 rate can cost more than a premium pool with a 0.8 percent 502 rate.

Prevention, not just recovery

The best fix for a residential proxy 502 is to stop generating the conditions that trigger it.

Build these safeguards into the stack:

Track 502 rate by provider, zone, country, ASN, and session type.
Auto-rotate sessions after one repeat 502 on the same target.
Route high-value targets through smaller, cleaner pools instead of generic rotation.
Keep separate retry policies for 429, 403, timeout, and 502.
Periodically re-test with curl, requests, and a browser client.

Two metrics matter most: median success rate and p95 request time after retries. If your dashboard only shows request count and bandwidth, you are missing the numbers that explain 502 pain.

Bottom line

A residential proxy 502 is usually a routing, session, or upstream integrity problem, not a mystery. Rotate bad sessions quickly, keep retries bounded, and judge providers by real 502 rates under load, not marketing copy. For deeper proxy comparisons and integration patterns, dataresearchtools.com is the place to keep your playbook current.

May 7, 2026

Scraping DAO governance and Snapshot data in 2026

Scrape DAO governance data jobs in 2026 are a cleaner problem than most crypto scraping work because the underlying systems were designed to be public from day one. Snapshot.org publishes a full GraphQL API. Tally exposes its own GraphQL endpoint. The on-chain Governor contracts emit standardized events that any node can index. Compared to scraping a cagey marketplace or a fingerprinting-heavy social platform, governance data is right there waiting to be collected. The challenge is reconciliation across off-chain (Snapshot), on-chain (Governor contracts), and the human layer (Discord, forum threads, Twitter discussions) where most of the actual decision-making happens.

This guide covers the practical mechanics of building a DAO governance pipeline in 2026: how to use Snapshot’s GraphQL API at scale, how to read on-chain proposals from OpenZeppelin Governor and Compound-style contracts, and the analytics patterns that turn raw vote data into voter behavior intelligence.

The two-track structure of DAO governance

Almost every active DAO in 2026 runs governance on one of two tracks. Off-chain via Snapshot, where votes are gasless signatures stored on IPFS, or on-chain via a Governor contract that records every vote on the blockchain. Many DAOs do both: a Snapshot signal vote first to gauge community sentiment, then an on-chain vote that actually executes the change.

This bifurcation matters for scrapers because the data lives in completely different places. Snapshot’s data is in Snapshot’s GraphQL database. On-chain proposal data is in Ethereum, Optimism, Arbitrum, Base, or wherever the Governor contract is deployed. A complete picture of a DAO’s governance requires both feeds.

Snapshot GraphQL: the easiest scraping target in crypto

Snapshot.org runs a public GraphQL endpoint at https://hub.snapshot.org/graphql with no authentication required for read operations. They rate-limit at roughly 60 requests per minute per IP, which is generous given the small payload size. The schema is documented and stable.

The two queries you actually use most:

import requests

SNAPSHOT_URL = "https://hub.snapshot.org/graphql"

def get_proposals(space: str, first: int = 100, skip: int = 0):
    query = """
    query GetProposals($space: String!, $first: Int!, $skip: Int!) {
      proposals(
        first: $first
        skip: $skip
        where: {space_in: [$space]}
        orderBy: "created"
        orderDirection: desc
      ) {
        id
        title
        body
        choices
        start
        end
        snapshot
        state
        author
        space { id name }
        scores
        scores_total
        votes
      }
    }
    """
    resp = requests.post(
        SNAPSHOT_URL,
        json={"query": query, "variables": {"space": space, "first": first, "skip": skip}},
        timeout=15,
    )
    return resp.json()["data"]["proposals"]


def get_votes(proposal_id: str, first: int = 1000, skip: int = 0):
    query = """
    query GetVotes($proposal: String!, $first: Int!, $skip: Int!) {
      votes(
        first: $first
        skip: $skip
        where: {proposal: $proposal}
        orderBy: "vp"
        orderDirection: desc
      ) {
        id
        voter
        vp
        choice
        created
        reason
      }
    }
    """
    resp = requests.post(
        SNAPSHOT_URL,
        json={"query": query, "variables": {"proposal": proposal_id, "first": first, "skip": skip}},
        timeout=15,
    )
    return resp.json()["data"]["votes"]

The vp field in the votes query is voting power, denominated in whatever the space’s strategy specifies (token balance, NFT ownership, delegated tokens, etc.). For most DAOs this maps directly to token holdings at the snapshot block.

A complete archive of a single DAO’s governance history requires paginating through proposals, then for each proposal paginating through votes. For a large DAO like Aave or Uniswap, this is 200-500 proposals and 50,000-200,000 individual votes. Pulling the full archive takes about an hour respecting rate limits.

Tally for on-chain governance

Tally is the dominant interface for on-chain Governor contracts. They aggregate proposal data from OpenZeppelin Governor, Compound Bravo, and several variants across multiple chains. Their GraphQL API at https://api.tally.xyz/query requires an API key (free tier available, paid for higher rate limits).

Tally is the right tool when you need on-chain DAO data without running your own indexer. They handle the contract decoding, proposal state machine, and cross-chain aggregation. For a research project on, say, Optimism’s governance evolution, Tally is faster than building from scratch.

def tally_proposals(governor_id: str, api_key: str, first: int = 100):
    query = """
    query Proposals($input: ProposalsInput!) {
      proposals(input: $input) {
        nodes {
          ... on Proposal {
            id
            metadata { title description }
            status
            createdAt
            voteStats { votesCount support percent }
          }
        }
      }
    }
    """
    resp = requests.post(
        "https://api.tally.xyz/query",
        json={
            "query": query,
            "variables": {"input": {"filters": {"governorId": governor_id}, "page": {"limit": first}}},
        },
        headers={"Api-Key": api_key},
        timeout=15,
    )
    return resp.json()

Reading Governor contracts directly

For the highest fidelity and complete decentralization, read the Governor contract directly via RPC. OpenZeppelin’s Governor is the most common implementation and emits these events:

ProposalCreated when a new proposal is submitted
VoteCast when a delegate casts a vote
ProposalCanceled, ProposalQueued, ProposalExecuted for lifecycle changes

from web3 import Web3

w3 = Web3(Web3.HTTPProvider("https://eth-mainnet.g.alchemy.com/v2/YOUR_KEY"))

GOVERNOR_ABI = [...]  # OpenZeppelin Governor ABI
governor = w3.eth.contract(address="0x408ED6354d4973f66138C91495F2f2FCbd8724C3", abi=GOVERNOR_ABI)

def index_proposals_from_block(start_block: int, end_block: int):
    event_filter = governor.events.ProposalCreated.create_filter(
        from_block=start_block,
        to_block=end_block,
    )
    for event in event_filter.get_all_entries():
        yield {
            "proposal_id": event["args"]["proposalId"],
            "proposer": event["args"]["proposer"],
            "description": event["args"]["description"],
            "block_number": event["blockNumber"],
            "tx_hash": event["transactionHash"].hex(),
        }

The catch with reading historical events at scale is RPC rate limits. Most providers cap eth_getLogs at 10,000 blocks per request. For multi-year history you batch in chunks. Free tier RPC providers will throttle hard if you try to backfill years of governance events; use a paid tier or run your own archive node.

Comparison of governance data sources

source	auth	coverage	data freshness	best for
Snapshot GraphQL	none	all Snapshot spaces (off-chain votes)	seconds	off-chain governance, signal votes
Tally GraphQL	API key (free + paid)	all OpenZeppelin/Compound governors	minutes	on-chain governance, multi-chain
Direct RPC + ABI	RPC key	any chain you have RPC for	block-by-block	high-fidelity, custom contracts
Etherscan API	API key (free)	all Ethereum contracts	minutes	quick contract introspection, tx decoding
The Graph subgraphs	varies	indexed contracts only	minutes	when a community subgraph exists
Boardroom	none for read	aggregated DAO data	minutes	quick dashboards, multi-DAO comparison

For most research, Snapshot + Tally covers 80% of the data you need. Direct RPC is for the remaining 20% where you need millisecond freshness or custom contract logic that aggregators do not understand.

Snapshot space discovery

Snapshot has 100,000+ registered spaces, of which maybe 3,000 are actively used. Filtering active from inactive is a useful preprocessing step that saves storage and rate-limit budget. The discovery query:

def list_active_spaces(min_proposals: int = 10):
    query = """
    query Spaces($first: Int!) {
      spaces(first: $first, orderBy: "proposalsCount", orderDirection: desc) {
        id
        name
        about
        network
        symbol
        followersCount
        proposalsCount
      }
    }
    """
    resp = requests.post(SNAPSHOT_URL, json={"query": query, "variables": {"first": 1000}}, timeout=15)
    spaces = resp.json()["data"]["spaces"]
    return [s for s in spaces if s["proposalsCount"] >= min_proposals]

The top 1,000 spaces by proposal count cover roughly 95% of governance activity. For a research project, scraping just those is usually enough. The long tail is interesting only for category-specific studies (e.g., NFT-collection DAOs, regional DAOs, gaming guilds).

Voter analytics: turning raw votes into intelligence

The most valuable thing you can do with governance data is voter behavior analysis. Every wallet’s voting history tells a story.

Common analytics derived from raw vote data:

Voter participation rate: percentage of proposals a wallet voted on relative to its eligibility window
Concentration: the share of total voting power held by the top 10, 100, 1000 voters
Whale alignment: how often a specific large wallet votes with the majority versus against
Coalition detection: clusters of wallets that consistently vote the same way (suggesting coordination, delegation chains, or shared sybil control)
Proposal heat: total voters and total VP that participated, normalized by space size

A simple coalition detector using Jaccard similarity:

from collections import defaultdict
from itertools import combinations

def detect_coalitions(votes_by_proposal: dict, min_overlap: float = 0.8):
    """votes_by_proposal: {proposal_id: {voter_address: choice_index}}"""
    voter_history = defaultdict(dict)
    for prop_id, voter_choices in votes_by_proposal.items():
        for voter, choice in voter_choices.items():
            voter_history[voter][prop_id] = choice

    coalitions = []
    voters = list(voter_history.keys())
    for a, b in combinations(voters, 2):
        common = set(voter_history[a].keys()) & set(voter_history[b].keys())
        if len(common) < 5:
            continue
        agreement = sum(1 for p in common if voter_history[a][p] == voter_history[b][p]) / len(common)
        if agreement >= min_overlap:
            coalitions.append((a, b, agreement, len(common)))
    return coalitions

This is naive (real coalition detection uses spectral clustering or community detection algorithms on a vote agreement graph) but it works well enough for first-pass exploration on small DAOs.

Sybil and delegation graph reconstruction

A useful extension to coalition detection is reconstructing the delegation graph for token-weighted DAOs. Most Governor implementations expose a delegates(address) view that returns the address each token-holder has delegated to, and a getVotes(address, blockNumber) view that returns effective voting power at a historical block. Walking these views for the top 10,000 token holders gives you a delegation-edge dataset that, combined with vote records, exposes:

Bridges: wallets that aggregate delegated voting power from many small holders, then vote as one bloc
Whale puppets: delegate addresses that always vote identically with one large delegator, suggesting controlled signing
Idle delegations: delegations to addresses that never cast votes, effectively removing those tokens from circulation
Sybil clusters: groups of small wallets delegated to the same entity that all received their initial token transfer from a common funding source

Build the delegation graph with a network library like networkx and run weakly-connected-component analysis to surface clusters. For DAOs with snapshot-based delegation (where delegation snapshot is per-proposal), reconstruct the graph at each proposal block to capture the actual configuration that voted, not the current one.

Forum and Discord context

Governance does not happen only in the votes. The actual decision happens in forum threads, Discord channels, and Twitter discussions weeks before a proposal hits Snapshot. A complete pipeline pulls Discourse forum data via the official Discourse API, Discord channel data via bot integrations (with server admin permission), and Twitter via paid Twitter API or scraping.

For Discourse forums (Aave, Uniswap, Compound, Optimism all use Discourse for governance discussion), the API is at https://forum.example.org/posts.json and is enabled by default for read access. Polling categories every 10 minutes is sufficient.

def get_discourse_topics(forum_url: str, category_id: int, page: int = 0):
    url = f"{forum_url}/c/{category_id}.json"
    resp = requests.get(url, params={"page": page}, timeout=15)
    return resp.json()

For Discord, scraping public messages without bot permissions violates Discord ToS. The compliant approach is asking the DAO’s admins for bot permissions in the relevant channels. Most governance-focused DAOs grant this for legitimate research projects.

Cross-DAO benchmarking

Once you have several DAOs in your pipeline, the most interesting analysis is cross-DAO comparison. Useful benchmarks include:

Median quorum hit rate: what fraction of proposals reach quorum across DAOs of similar size? A DAO with a 30% quorum rate while peers run at 70% has either dead delegations or unhealthy proposal throughput.
Proposal velocity: proposals per month relative to treasury size. A $500M treasury with one proposal per month is under-active; a $5M treasury with twenty proposals per month is firefighting.
Author concentration: what share of proposals come from the top 5 proposal authors? Above 60% suggests a small council effectively governs and the broader voter base is rubber-stamping.
Voter retention: of voters who participated 90 days ago, what fraction still vote today? A normal range is 25-55%; below 20% indicates community burnout.

Most of these metrics fit naturally into a single dashboard fed by the storage schema below. The benchmark numbers themselves come from your own corpus once you have indexed 30+ DAOs; there is no universally accepted source of truth for “healthy” DAO governance.

Storage schema

CREATE TABLE dao_spaces (
    space_id TEXT PRIMARY KEY,
    name TEXT,
    network TEXT,
    members_count INTEGER,
    proposals_count INTEGER,
    treasury_usd NUMERIC
);

CREATE TABLE proposals (
    proposal_id TEXT PRIMARY KEY,
    space_id TEXT REFERENCES dao_spaces(space_id),
    source TEXT NOT NULL,  -- 'snapshot' | 'governor' | 'tally'
    title TEXT,
    body TEXT,
    author TEXT,
    state TEXT,
    created_at TIMESTAMPTZ,
    start_at TIMESTAMPTZ,
    end_at TIMESTAMPTZ,
    snapshot_block BIGINT,
    scores_total NUMERIC,
    votes_count INTEGER
);

CREATE TABLE votes (
    proposal_id TEXT REFERENCES proposals(proposal_id),
    voter_address TEXT NOT NULL,
    choice INTEGER NOT NULL,
    voting_power NUMERIC NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    reason TEXT,
    PRIMARY KEY (proposal_id, voter_address)
);

CREATE INDEX ON votes (voter_address);
CREATE INDEX ON proposals (space_id, created_at DESC);

This schema handles both Snapshot and on-chain votes uniformly via the source field. For DAOs that run both, you have parallel proposal records and can analyze the relationship between off-chain signals and on-chain execution.

Cost worked example

A practical research deployment covering ~150 active DAOs across Snapshot and 60 on-chain Governors costs:

Snapshot GraphQL ($0)
Tally API paid tier ($79/mo for 150 req/min)
Alchemy Growth tier ($49/mo) for on-chain events
1 small VPS for indexer ($25/mo)
Postgres on a hosted instance ($30/mo)
10 IPs of residential proxy for forum scraping ($20/mo)

Total: about $200/month for a complete cross-DAO governance dataset that competing services charge $1,500-3,000/month for. The break-even point is reached within the first week of operation.

Proxy considerations

Snapshot’s rate limit is permissive enough that proxies are usually unnecessary for read-only research. If you are pulling all 100,000+ Snapshot spaces continuously, distribute across 5-10 IPs with residential proxies.

For RPC calls, your bottleneck is your RPC provider, not IP-based rate limits. Use a paid tier or multiple free-tier keys. We cover provider selection in our best residential proxy providers 2026 review.

For Discord scraping (with bot permissions), the bot itself rate-limits per server, not per IP. Proxies do not help.

External authoritative reference: the Snapshot.js documentation covers the GraphQL schema, signing flow, and strategy types.

Common gotchas

Snapshot’s vp field is computed at proposal creation, not at vote time. A wallet that votes after selling its tokens still gets credit for its snapshot-block balance. Do not double-discount.
The state field on Snapshot proposals returns pending, active, closed. There is no passed or failed; that is your interpretation based on scores against quorum and threshold defined in the space settings.
Governor proposal IDs differ between Compound Bravo (uint256 sequential) and OpenZeppelin (keccak hash of proposal params). Code that assumes one breaks on the other.
Multi-chain DAOs (Optimism, Arbitrum) often reuse Snapshot space slugs but have different on-chain Governors per chain. Make sure you are matching the right pair.
Tally’s voteStats returns percentages calculated against quorum, not against votes cast. A proposal at 60% support may show 30% in percent if quorum is half the eligible supply.
Reading historical events with eth_getLogs on free-tier RPCs frequently times out on busy contracts. Chunk by 1,000-block windows for old contracts and 10,000 for newer ones, with retry-with-narrower-window on timeout.

Common analytical questions

Once you have the raw data, the questions worth answering include:

Vote concentration over time. Is a DAO’s governance becoming more or less concentrated? Plot top-10 vote share per proposal over time.

Delegate effectiveness. For DAOs with delegation (most modern Governor implementations), how often do delegates show up to vote on the tokens delegated to them? An 80% delegate participation rate is healthy; a 20% rate means delegation is dead weight.

Proposal pass rate. What fraction of proposals reach quorum and pass? A pass rate of 95% suggests rubber-stamping; a pass rate of 30% suggests genuine contention.

Treasury follow-through. For DAOs that vote on treasury allocations, do the funds actually move on-chain after the vote passes? Reconciliation against the treasury wallet’s transactions reveals whether governance is real or theater.

We dig into related on-chain forensics in our guide on scraping crypto exchange order books.

FAQ

Q: do I need a node to scrape on-chain governance?
No. Hosted RPC providers (Alchemy, Infura, QuickNode) cover the read use case perfectly. You only need your own node if you are indexing every block in real time at scale or need access to internal traces.

Q: can I scrape historical Snapshot proposals from defunct DAOs?
Yes. Snapshot persists historical data even for spaces that are inactive. You can pull the full archive of any space that has not been deleted by the admins.

Q: how do I correlate Snapshot space to on-chain Governor contract?
Most large DAOs publish the mapping in their docs. There is no universal index. Tally maintains a manually curated list. For obscure DAOs you may need to ask in the project’s Discord.

Q: what about Solana DAOs?
Solana DAOs use Realms (governance.so) instead of Snapshot/Governor. Realms exposes data via Solana RPC and the SPL Governance program. Different stack, same patterns.

Q: is voter address pseudonymity a concern for analysis?
For pure on-chain analysis, addresses are public and attribution is fair game. For combining vote data with off-chain identity (Twitter handles, Discord usernames), be careful about privacy expectations. See our GDPR compliance for web scraping guide for the data minimization patterns.

Q: how do I track DAO treasury movements alongside governance votes?
Index Transfer events from the treasury wallet on the same chain as the Governor. Cross-reference transactions occurring within 7 days after a ProposalExecuted event against the proposal’s calldata to confirm execution matched intent. Surprisingly often, a proposal authorizes a transfer that never happens because the multisig signers fail to coordinate.

Q: what is the right cadence for polling Snapshot?
For active spaces, every 5 minutes catches new proposals and vote updates. For an archive, a one-time backfill plus daily incremental sync is enough. Snapshot itself does not push updates, so polling is the only option without running a custom relayer.

Q: do I need to verify Snapshot signatures?
Snapshot’s hub already verifies signatures before storing votes, so the data you pull is pre-validated. Re-verifying is overkill for analysis but useful for audit-grade research where you want to prove independently that every vote is signed by the wallet it claims.

Closing

DAO governance scraping in 2026 is one of the cleanest data engineering problems in crypto. Snapshot and Tally do most of the heavy lifting; on-chain data fills the gaps; forum and Discord context completes the picture. The hard part is not collection but interpretation: turning vote records into intelligence about who actually controls a DAO, how decisions get made, and whether governance is functioning or theater. For the broader crypto data infrastructure picture see our crypto-defi category hub.

May 7, 2026

Mistral Large for Web Scraping Pipelines in 2026

Mistral Large is one of the few genuinely open-weight models that can compete with GPT-4 class systems on structured extraction — and in 2026, that matters a lot if you’re running a scraping pipeline at scale. The combination of a 128K context window, strong instruction-following, and self-hostable weights makes it worth a serious look for anyone tired of paying per-token on closed APIs.

What Mistral Large actually brings to scraping pipelines

The current release is Mistral Large 2 (mistral-large-2407), with 123B parameters. It runs comfortably on a 2xA100 or 4xA6000 setup, which puts it within reach for a dedicated scraping server. Context length is 128K tokens, enough to fit a full crawled HTML page, system prompt, and structured output schema in one shot.

For scraping-specific tasks, the key capabilities are:

Function calling / tool use: Mistral Large supports native function calling, which means you can define a JSON schema and get reliably structured output back without regex postprocessing. The JSON mode is stable enough for production use with a schema validation layer on top.
Instruction fidelity: On complex extraction prompts (“extract all job postings, normalize the salary field to USD, skip entries with missing location”), it follows multi-step instructions more precisely than smaller models like Mistral 7B or Mixtral 8x7B. It also handles nested schemas — arrays of objects with conditional fields — more consistently than most open models at this size.
Multi-language support: Useful if you’re scraping non-English sites — Mistral’s training data skews toward European languages, which shows up in extraction accuracy on French, German, and Italian pages.
Quantization tolerance: Mistral Large 2 runs well at 4-bit quantization (GPTQ or AWQ) with minimal quality degradation on extraction tasks. That gets the VRAM requirement down to around 65-70GB, which fits a 2xA100 40GB setup.

What it doesn’t bring: vision. You can’t feed it a screenshot of a rendered page the way you can with Gemini 2.0 Flash for Web Scraping, which handles multi-modal inputs natively. If your scraping targets rely on screenshots or PDFs, that’s a real gap.

Pricing comparison: Mistral Large vs alternatives

Here’s where things get interesting. Mistral via the official API is not cheap for high-volume scraping, but the self-hosted route changes the math.

Model	Input (per 1M tokens)	Output (per 1M tokens)	Self-hostable
Mistral Large 2 (API)	$2.00	$6.00	Yes (weights available)
GPT-4o	$2.50	$10.00	No
Claude 3.5 Sonnet	$3.00	$15.00	No
Gemini 1.5 Pro	$1.25	$5.00	No
Qwen 2.5 72B (API)	$0.40	$0.40	Yes
DeepSeek V3	$0.27	$1.10	Yes (limited)

If you’re running on the Mistral API, the pricing is competitive with GPT-4o but not by a huge margin. The real advantage is owning the weights. A self-hosted Mistral Large 2 on a leased A100 box can get you below $0.10 per 1M tokens at decent throughput, which is why it competes differently from, say, a closed model.

For pure cost optimization on hosted APIs, DeepSeek V3 for cheap web scraping LLM calls is significantly cheaper. Mistral Large’s edge is openness plus quality — not raw price.

Running Mistral Large locally with vLLM

If you’re self-hosting, vLLM is the standard serving layer. Here’s a minimal setup for a scraping inference server:

# Install vLLM
pip install vllm

# Serve Mistral Large 2 with tensor parallelism across 2 GPUs
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-Large-Instruct-2407 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --dtype bfloat16 \
  --port 8000

Once it’s running, you call it via the OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="mistralai/Mistral-Large-Instruct-2407",
    messages=[
        {"role": "system", "content": "Extract structured product data as JSON."},
        {"role": "user", "content": f"<html>{page_html}</html>"}
    ],
    response_format={"type": "json_object"},
    temperature=0.0
)

temperature=0.0 is non-negotiable for extraction tasks. Any randomness and you get inconsistent field names, hallucinated prices, and outputs that break your schema validation.

Integrating Mistral Large with Crawl4AI

Crawl4AI is the cleanest way to combine structured crawling with LLM extraction in 2026. It handles JS rendering, anti-bot evasion hooks, and has a native LLMExtractionStrategy that you can point at any OpenAI-compatible endpoint — including your local Mistral Large server.

from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel

class JobPosting(BaseModel):
    title: str
    company: str
    salary_usd: float | None
    location: str

strategy = LLMExtractionStrategy(
    provider="openai/mistral-large",
    api_base="http://localhost:8000/v1",
    api_token="unused",
    schema=JobPosting.schema(),
    instruction="Extract job postings. Normalize salary to USD. Return null if missing."
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(
        url="https://example-jobs-site.com/listings",
        extraction_strategy=strategy
    )
    print(result.extracted_content)

The provider field accepts any OpenAI-compatible base URL when combined with api_base. No additional configuration needed. This pattern also works with Llama 3 70B local scraping pipelines or Qwen 2.5 for web scraping if you want to swap models without touching your pipeline code.

Tradeoffs and when not to use it

Mistral Large is a good choice when:

You need self-hosted weights for data privacy or compliance reasons.
Your extraction tasks require long context (multiple pages, complex schemas).
You’re scraping European-language sites where its training data is stronger.
You want an open alternative to GPT-4 class quality without vendor lock-in.

It’s not the right choice when:

You need vision/screenshot understanding — use Gemini 2.0 Flash.
Budget is the primary constraint — Qwen 2.5 72B or DeepSeek V3 undercut it significantly on API pricing.
You’re running a simple scraper that doesn’t need 123B parameters — Mistral 7B or Mistral Nemo handle basic extraction at a fraction of the cost.
GPU availability is a problem — 123B quantized to 4-bit still needs ~70GB VRAM.

One thing worth flagging: Mistral’s function calling, while good, isn’t quite as rock-solid as GPT-4o on ambiguous extraction prompts. You’ll want schema validation (Pydantic works well here) and a retry loop for the ~5% of responses that don’t conform, especially on noisy HTML. A simple pattern is to catch ValidationError, strip the HTML down to the visible text using html2text, and retry once with the cleaner input — that alone drops non-conforming outputs to under 1% in most production pipelines.

Latency is also worth considering. Self-hosted Mistral Large at 4-bit on 2xA100 does around 15-25 tokens/second depending on batch size and prompt length. For real-time scrapers that need sub-second responses, that’s probably too slow. For async batch extraction running overnight or across a job queue, it’s completely fine.

Bottom line

Mistral Large 2 is the strongest open-weight option for production scraping pipelines where data privacy, self-hosting, or long-context extraction matters. It’s not the cheapest route, but for teams that can’t send page content to a closed API, it’s one of the few models that actually delivers GPT-4 class extraction quality on your own infrastucture. We’ll keep benchmarking alternatives as the open-source LLM landscape moves fast — follow DRT’s AI agent scraping coverage for updated comparisons.

May 7, 2026

Scraping NFT collection floor prices and metadata in 2026

Scrape NFT data jobs in 2026 sit at the intersection of three different systems that all need to agree before you have a complete record: the marketplace API for current listings and floor price, the underlying blockchain for ownership and provenance, and IPFS or Arweave for the actual metadata and image. Pull from only one source and you have a partial picture; pull from all three and you spend most of your engineering time on rate limits, gateway timeouts, and CDN caching weirdness. The market has consolidated since the 2021-2022 peak. Three platforms (OpenSea, Blur, Magic Eden) handle the vast majority of liquidity, and the long tail of marketplaces is mostly dead. That consolidation makes the scraping problem more tractable than it was three years ago.

This guide covers the practical mechanics of building an NFT data pipeline in 2026: which marketplace APIs are public versus paid, how to reconcile on-chain truth against marketplace cache, and the rate-limit and proxy patterns that let a small operation track 10,000+ collections continuously.

What “floor price” actually means and why it is hard

The floor price of a collection is the lowest active listing price on a marketplace. It sounds simple but it has three quirks that ruin naive scrapers.

First, floor is per-marketplace. A collection might have a 0.5 ETH floor on OpenSea and a 0.45 ETH floor on Blur because of fee differences and platform-specific listings. The “true” floor is the minimum across all marketplaces where the collection trades.

Second, floor is sensitive to outliers. A single listing at a clearly broken price (1 wei, or accidentally bid in DAI instead of ETH) becomes the technical floor until it gets bought or canceled. Production trackers usually compute a “true floor” by sorting listings ascending and taking the price at the 1st percentile or after the first few listings, ignoring obvious outliers.

Third, floor changes constantly. During a popular mint or hype cycle, floor can move 5-10% in a single minute. Polling at 5-minute cadence will miss most of the movement. Real-time floor tracking requires websocket or webhook subscriptions where the marketplace offers them.

OpenSea API in 2026

OpenSea operates the most widely used NFT API. As of 2026, the v2 API requires an API key for almost everything. You can request a free key through their developer portal but the free tier is limited to 4 requests per second and excludes the highest-value endpoints (real-time order book, historical sales). Paid plans start at around $200/month for higher rate limits and at $1500/month for the data tier with full historical access.

The free tier is enough for tracking floor price across a few hundred collections at 5-minute cadence. Past that you either pay or you augment with on-chain data (which is free but has its own complexity).

import time
import requests

class OpenSeaClient:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.session = requests.Session()
        self.session.headers.update({
            "X-API-KEY": api_key,
            "Accept": "application/json",
        })

    def get_collection(self, slug: str):
        url = f"https://api.opensea.io/api/v2/collections/{slug}"
        return self._get(url)

    def get_listings(self, slug: str, limit: int = 50):
        url = f"https://api.opensea.io/api/v2/listings/collection/{slug}/all"
        return self._get(url, params={"limit": limit})

    def get_stats(self, slug: str):
        url = f"https://api.opensea.io/api/v2/collections/{slug}/stats"
        return self._get(url)

    def _get(self, url, params=None, retries=3):
        for attempt in range(retries):
            resp = self.session.get(url, params=params, timeout=15)
            if resp.status_code == 429:
                time.sleep(2 ** attempt)
                continue
            resp.raise_for_status()
            return resp.json()

The stats endpoint is the cheapest way to track floor price because it returns the floor and total volume in a single call. The listings endpoint gives more detail but costs more rate-limit budget per collection.

Blur API: less documented but more powerful

Blur captured most of the professional trading volume in 2023 and has held it. Their API is technically not public but a usable endpoint exists at https://core-api.prod.blur.io/v1/. It requires a session token that you obtain by signing a wallet message. Tracking sites like Nansen and CryptoSlam use this endpoint. Blur tolerates it as long as you stay below roughly 100 requests per minute per session.

Blur’s particular value is the bid pool: a unified bidding mechanism where buyers commit ETH against an entire collection at a price. The sum of bids at each tier is a strong demand signal that does not exist on OpenSea. Scraping the Blur bid pool gives you data that is genuinely not available anywhere else without paying Blur for it directly.

def blur_collection_stats(slug: str, auth_token: str):
    url = f"https://core-api.prod.blur.io/v1/collections/{slug}"
    resp = requests.get(
        url,
        headers={
            "authToken": auth_token,
            "User-Agent": "Mozilla/5.0",
        },
        timeout=10,
    )
    return resp.json()

The auth token expires after about 24 hours. Production setups rotate the wallet signing automatically.

Magic Eden for Solana and multichain

Magic Eden is the dominant Solana NFT marketplace and has expanded to Bitcoin Ordinals, Polygon, and Ethereum. Their public API at https://api-mainnet.magiceden.dev/v2/ does not require an API key for most endpoints and tolerates 2 requests per second per IP.

def magiceden_collection_stats(symbol: str):
    url = f"https://api-mainnet.magiceden.dev/v2/collections/{symbol}/stats"
    resp = requests.get(url, timeout=10)
    return resp.json()

Magic Eden is the right tool when you care about Solana NFT data or Bitcoin Ordinals. For Ethereum NFTs, OpenSea and Blur have deeper liquidity and you should use them instead.

Marketplace API comparison

marketplace	API auth	free rate limit	floor price	full listing data	best for
OpenSea	API key (free + paid)	4 req/s free	yes	yes (paid for full)	Ethereum mainstream
Blur	session token	~100 req/min	yes	yes	Ethereum trading data
Magic Eden	none	2 req/s	yes	yes	Solana, Ordinals, multichain
LooksRare	none for public	5 req/s	yes	yes	Ethereum royalty-aware
X2Y2	API key (paid)	varies	yes	yes	Ethereum, declining
Tensor	none	aggressive	yes	yes	Solana professional
Reservoir	API key (free tier)	30k req/day free	yes	yes (aggregated)	aggregator across all ETH chains

Reservoir deserves special mention. They aggregate listings from OpenSea, Blur, LooksRare, and several others into a unified API. For most use cases, Reservoir is easier than running separate scrapers against each marketplace. The free tier is generous and covers most research work.

Marketplace decision matrix

Use this matrix when deciding which marketplace API to lean on for a given collection or use case:

use case	primary	fallback	notes
Top-100 Ethereum bluechip floor	Reservoir	OpenSea	Reservoir’s aggregated floor catches Blur and Sudoswap that OpenSea misses
Trader analytics on Ethereum	Blur	Reservoir	Blur’s bid pool data is unique and worth the auth pain
Solana NFT floor and trades	Magic Eden	Tensor	Magic Eden has full coverage; Tensor for execution-quality data
Bitcoin Ordinals	Magic Eden	Hiro	Magic Eden Ordinals support has matured; Hiro still useful for protocol-level inscriptions
L2 NFT collections (Base, Zora)	Reservoir	Native marketplace	Reservoir has Base and Zora; smaller chains are spotty
Royalty enforcement analysis	LooksRare + custom	Reservoir	Royalty data requires per-marketplace logic that aggregators flatten
Real-time floor tracking for trading bots	Reservoir webhooks	Marketplace websockets	Webhooks remove polling latency entirely

Pick the primary based on use case rather than alphabetical order. The fallback path is critical because every marketplace API has occasional outages and rate-limit surprises.

On-chain truth: when the marketplace is wrong

Marketplaces cache aggressively. A listing might be canceled or filled on-chain but still showing as active on the marketplace API for 30-60 seconds. For research this is fine. For trading or arbitrage detection it is fatal.

The authoritative source for ownership and listing status is the blockchain itself. Each marketplace operates its own listing contract (Seaport for OpenSea, BlurExchange for Blur). You can read the contract state directly via RPC and confirm which listings are still active.

from web3 import Web3

w3 = Web3(Web3.HTTPProvider("https://eth-mainnet.g.alchemy.com/v2/YOUR_KEY"))

SEAPORT_ADDRESS = "0x00000000000000ADc04C56Bf30aC9d3c0aAF14dC"

def is_listing_active(order_hash: bytes) -> bool:
    seaport = w3.eth.contract(address=SEAPORT_ADDRESS, abi=SEAPORT_ABI)
    status = seaport.functions.getOrderStatus(order_hash).call()
    is_validated, is_cancelled, total_filled, total_size = status
    return is_validated and not is_cancelled and total_filled < total_size

For ownership, the ERC-721 ownerOf(tokenId) view is the truth. Marketplaces show owner data that may be stale by minutes. If you are scoring rarity or computing wallet holdings, you must call the contract directly.

Reservoir-first architecture

A pragmatic 2026 architecture treats Reservoir as the primary read source and falls back to direct marketplace APIs only when Reservoir lacks coverage or freshness. The flow:

Reservoir’s tokens/v6 and orders/asks/v5 endpoints aggregate listings across OpenSea, Blur, LooksRare, X2Y2, Sudoswap, and a half-dozen newer venues. One request returns the cross-venue floor and the venue-by-venue breakdown.
For collections Reservoir does not cover (newer chains, niche L2s), call the native marketplace API directly.
For sub-second freshness on top-tier collections, subscribe to Reservoir’s webhook events instead of polling.
Cross-check Reservoir’s data once per day against Etherscan transfer counts to catch indexing lag, which appears occasionally during high-volume mint events.

The single-source pattern saves dozens of integration headaches and the multi-marketplace reconciliation work. The cost is dependency on Reservoir staying alive and pricing reasonably; have the direct-marketplace fallback paths shipped and tested even if you do not use them daily.

IPFS metadata fetching

NFT metadata for image, name, traits, and description is usually stored on IPFS or Arweave, with a token URI that resolves to a JSON document. The chain stores the URI; the URI points to the metadata; the metadata points to the image.

IPFS gateways are notoriously unreliable. The default ipfs.io gateway frequently returns 504 timeouts. Production scrapers maintain a list of gateways and round-robin requests across them with retry logic.

GATEWAYS = [
    "https://ipfs.io/ipfs/",
    "https://cloudflare-ipfs.com/ipfs/",
    "https://gateway.pinata.cloud/ipfs/",
    "https://nftstorage.link/ipfs/",
    "https://w3s.link/ipfs/",
]

def fetch_ipfs(cid: str, timeout: int = 10):
    for gateway in GATEWAYS:
        try:
            resp = requests.get(gateway + cid, timeout=timeout)
            if resp.status_code == 200:
                return resp.json() if "json" in resp.headers.get("Content-Type", "") else resp.content
        except requests.RequestException:
            continue
    raise IPFSFetchError(cid)

For projects you scrape repeatedly, pin the metadata to your own IPFS node or upload it to S3. This eliminates gateway flakiness and makes downstream queries instant.

Proxy and rate limit strategy

Marketplace APIs are the rate-limit bottleneck for NFT scraping. The mitigation pattern is multi-key rotation rather than IP rotation. Each developer account gets its own API key, and you round-robin across keys. OpenSea allows multiple API keys per account, and you can register multiple accounts (within their terms) for additional throughput.

For unkeyed endpoints (Magic Eden, Blur with rotating tokens), proxies do help. Use residential proxies to avoid the “all your traffic from one AWS IP” pattern that gets fingerprinted. We compare options in our best residential proxy providers 2026 review.

For RPC calls to read on-chain data, you have your own rate limits with your RPC provider (Alchemy, Infura, QuickNode). Most providers have generous free tiers, and one provider key per worker process avoids cross-contamination of rate limits.

Storage schema

NFT data is multidimensional and most teams over-design the schema on day one. Start simple:

CREATE TABLE collections (
    slug TEXT PRIMARY KEY,
    chain TEXT NOT NULL,
    contract_address TEXT NOT NULL,
    name TEXT,
    total_supply INTEGER,
    UNIQUE (chain, contract_address)
);

CREATE TABLE collection_snapshots (
    slug TEXT NOT NULL REFERENCES collections(slug),
    captured_at TIMESTAMPTZ NOT NULL,
    floor_price_eth NUMERIC,
    volume_24h_eth NUMERIC,
    sales_24h INTEGER,
    listed_count INTEGER,
    owner_count INTEGER,
    PRIMARY KEY (slug, captured_at)
);

CREATE INDEX ON collection_snapshots (captured_at DESC);

CREATE TABLE token_listings (
    chain TEXT NOT NULL,
    contract_address TEXT NOT NULL,
    token_id NUMERIC NOT NULL,
    marketplace TEXT NOT NULL,
    price_eth NUMERIC,
    seller_address TEXT,
    listed_at TIMESTAMPTZ,
    expires_at TIMESTAMPTZ,
    order_hash TEXT,
    is_active BOOLEAN,
    PRIMARY KEY (chain, contract_address, token_id, marketplace, order_hash)
);

For 1000 collections at 5-minute snapshot cadence, you generate 288,000 collection snapshot rows per day plus listing-level data. PostgreSQL handles this comfortably for a year before you need to consider partitioning.

Snapshot cadence vs storage tradeoff

Cadence has compounding effects on storage and rate-limit budget. The right cadence depends on tier:

Tier 1 (top 200 collections, mints, news-driven assets): 60 seconds. These move fast enough that a 5-minute gap loses real signal. At 200 collections this is 17,280 calls per day. With 4 OpenSea API keys you can absorb that comfortably.
Tier 2 (next 2,000 collections): 5 minutes. Still useful for tracking trends; movement is slower so the lower cadence preserves more than 90% of meaningful signal.
Tier 3 (long tail, 10,000+ collections): 1 hour. Catches major liquidity changes without burning the rate budget. Many long-tail collections have zero listed items most days, so most calls return identical data.

A reasonable two-week-old listing without movement can be polled daily. Reservoir’s snapshot endpoint accepts batched collection lookups, which is the cheapest way to refresh long-tail tiers in bulk.

Sales history and provenance

For provenance and sales history, the on-chain approach is more reliable than marketplace APIs. Every NFT transfer emits a Transfer event from the ERC-721 contract. Every marketplace sale emits a marketplace-specific event (Seaport’s OrderFulfilled, etc.) that includes price.

Indexing services like Goldsky, The Graph, Subsquid, and the previously mentioned Reservoir aggregate this data and expose it via GraphQL. For one-off queries, use a service. For continuous indexing of specific collections, run a node and subscribe directly to the events.

We cover the broader on-chain indexing patterns in our crypto-defi category hub and our deep dive on scraping crypto exchange order books.

External authoritative reference: the OpenSea API documentation covers the current endpoint catalog and rate-limit policy.

Cost worked example

A practical 2026 setup tracking 5,000 collections across Ethereum and Solana with mixed cadence costs roughly:

Reservoir API free tier ($0) plus one paid key for the higher rate limits ($150/mo)
OpenSea free key for floor stats; no paid tier needed if you reconcile via Reservoir
Magic Eden public API ($0)
Alchemy Growth tier for on-chain reads ($49/mo)
Pinata or NFT.Storage for self-pinned IPFS metadata ($20/mo, 50 GB)
$40/mo VPS for the collector (4 vCPU, 8 GB)
$25/mo Postgres on a small managed instance
30 IPs of residential proxy for Blur and unauth endpoints (~$50/mo on a starter pack)

Total: about $335/month. The same coverage purchased through a vendor (NFTGo Enterprise, Nansen Query) runs $1,500-5,000/month. Self-hosting becomes a clear win once you cross 200 collections and need historical depth.

Common failure modes

The most common failure mode in NFT scrapers is treating the marketplace API as ground truth. Always reconcile against on-chain state for ownership and listing status. The second most common failure is not handling IPFS gateway timeouts. Build retry logic across multiple gateways from day one.

The third failure mode is overreacting to outlier listings. A single 0.0001 ETH listing on a 1 ETH floor collection is almost always either a wash trade attempt or a scam. Production trackers ignore listings below the 1st percentile of recent floor history.

FAQ

Q: which API gives the best Ethereum coverage?
For aggregated Ethereum data, Reservoir is the easiest path. For raw OpenSea data, the official API. For trading depth, Blur. Most production setups combine all three.

Q: do I need a wallet to scrape NFT data?
Not for read-only scraping of public marketplace data. You need a wallet to access Blur’s authenticated endpoints and to interact directly with marketplace contracts on-chain. A throwaway wallet with no funds works fine for read-only authentication.

Q: how do I track floor price changes in real time?
OpenSea offers a streaming events API that pushes order events. Reservoir has webhooks. Without paid endpoints, polling at 60-second cadence is the practical floor for “near real time.”

Q: can I scrape rarity scores?
Rarity is computed from trait distribution, which you derive from the metadata of all tokens in a collection. Once you have the metadata table, rarity computation is trivial. Most rarity tools use the same algorithm and produce similar scores.

Q: what about Bitcoin Ordinals?
Ordinals data lives on the Bitcoin chain and is indexed by services like Magic Eden, Hiro, and Ord.io. The standard Bitcoin RPC does not return Ordinal data directly; you need an indexer in front of a full node.

Q: how do I tell a wash trade from a real sale?
Wash trades typically loop between two related wallets (often funded from the same source within the previous 7 days), at suspiciously round prices, with no time between transfers. Cross-reference seller and buyer addresses against a wallet-cluster service like Arkham or against your own clustering on shared funding sources. Flagging is heuristic, not exact, but rules out 80-90% of obvious wash activity.

Q: do marketplaces ever sue scrapers?
Cease-and-desist letters happen, lawsuits are rare and typically reserved for projects that resell the data as a competing product. Personal use, research, and non-commercial analytics have not historically attracted enforcement. Building a public dashboard that displays floor prices crosses into commercial territory and is where you should consult counsel.

Q: should I use The Graph subgraphs instead of direct RPC?
The Graph is excellent for queries that span many blocks and aggregate data, like “all sales of collection X in the last month.” Direct RPC is better for single-state lookups, like “is this listing currently active.” Use both: subgraphs for analytics, RPC for live truth.

Closing

NFT scraping in 2026 is a multi-source reconciliation problem more than a pure scraping problem. The marketplace APIs give you the user-facing view; the chain gives you the truth; IPFS gives you the content. Build pipelines that treat all three as inputs and reconcile them, and you can run a research-grade NFT data system at hobbyist cost. For broader infrastructure see our crypto-defi category hub.

May 7, 2026

Scraping crypto exchange order books in 2026

Scrape crypto order books pipelines have matured into one of the most demanding data engineering problems on the public internet. A single Binance BTC/USDT depth feed pushes 50-200 updates per second during normal hours and spikes past 1000 per second during news events. Multiply that by 200 trading pairs across 8 exchanges and you have a firehose that requires careful architecture decisions before you write a single line of code. The good news is that almost every major exchange exposes order book data through public websockets that do not require API keys for read access. The bad news is that latency, message ordering, and gap recovery decisions you make in the first day of building will haunt the system for years.

This guide covers the practical mechanics of scraping crypto order books in 2026: which exchange feeds work, how to normalize depth across venues, the latency vs cost tradeoff for hosting, and the storage patterns that let small teams keep multi-month order book history without burning $20,000 a month on infrastructure.

Why scrape order books instead of buying the data

Three vendors dominate paid crypto market data: Kaiko, CryptoCompare, and Tardis.dev. They are excellent. They are also expensive: a full L2 order book history feed for 50 pairs across 5 exchanges runs $5,000-25,000 per month depending on tier. For a quant fund this is rounding error. For an indie research project, an algo trading hobbyist, or a startup building MEV tooling, it is the entire budget.

Self-collecting order book data costs roughly $200-500 per month in compute and storage if you are disciplined. The tradeoff is engineering time. The first month is hard. After that the pipeline runs itself with weekly babysitting. For research that depends on data going back further than the day you started collecting, you still need a vendor for the historical backfill, but ongoing collection is cheap.

What “order book” actually means at the wire level

Every exchange streams order book updates as either snapshots or deltas. A snapshot is the full current state of the book at a moment in time. A delta is a list of price-level changes since the last update. Most production feeds are delta-based with periodic snapshot resyncs.

A typical delta message looks like this from Binance:

{
  "e": "depthUpdate",
  "E": 1715000000000,
  "s": "BTCUSDT",
  "U": 12345600,
  "u": 12345610,
  "b": [["63500.00", "0.5"], ["63499.50", "0"]],
  "a": [["63501.00", "1.2"], ["63502.00", "0.8"]]
}

The b and a arrays are bid and ask updates. A quantity of 0 means delete that price level. The U and u fields are the first and last update IDs in this message, which you use to detect gaps. If you miss a message, you have to refetch the full snapshot from the REST endpoint and replay deltas from there.

This gap recovery logic is where most amateur scrapers break. A naive listener that just appends deltas without checking sequence numbers will silently corrupt the book within minutes.

import asyncio
import json
import websockets
import requests
from collections import defaultdict

class BinanceDepthListener:
    def __init__(self, symbol: str):
        self.symbol = symbol.upper()
        self.bids = {}  # price -> qty
        self.asks = {}
        self.last_update_id = None

    async def fetch_snapshot(self):
        url = f"https://api.binance.com/api/v3/depth?symbol={self.symbol}&limit=1000"
        resp = requests.get(url, timeout=10)
        data = resp.json()
        self.last_update_id = data["lastUpdateId"]
        self.bids = {float(p): float(q) for p, q in data["bids"]}
        self.asks = {float(p): float(q) for p, q in data["asks"]}

    def apply_delta(self, msg):
        if msg["u"] <= self.last_update_id:
            return
        if msg["U"] > self.last_update_id + 1:
            raise GapDetectedError(msg["U"], self.last_update_id)
        for price, qty in msg["b"]:
            p, q = float(price), float(qty)
            if q == 0:
                self.bids.pop(p, None)
            else:
                self.bids[p] = q
        for price, qty in msg["a"]:
            p, q = float(price), float(qty)
            if q == 0:
                self.asks.pop(p, None)
            else:
                self.asks[p] = q
        self.last_update_id = msg["u"]

    async def run(self):
        url = f"wss://stream.binance.com:9443/ws/{self.symbol.lower()}@depth@100ms"
        await self.fetch_snapshot()
        async with websockets.connect(url) as ws:
            async for raw in ws:
                msg = json.loads(raw)
                try:
                    self.apply_delta(msg)
                except GapDetectedError:
                    await self.fetch_snapshot()

The @100ms suffix in the websocket URL is critical. Without it you get the default 1000ms depth stream, which is slow. The 100ms feed is the highest cadence Binance offers without paying for the institutional pro feed.

Exchange feed comparison

exchange	websocket URL	best feed cadence	snapshot via REST	message ordering
Binance Spot	wss://stream.binance.com:9443	100ms	yes, /api/v3/depth	sequence IDs
Binance Futures	wss://fstream.binance.com	100ms	yes, /fapi/v1/depth	sequence IDs
Coinbase	wss://ws-feed.exchange.coinbase.com	per event	snapshot in stream	sequence per product
OKX	wss://ws.okx.com:8443	100ms	snapshot in stream	checksum per update
Bybit	wss://stream.bybit.com/v5/public/spot	per event	snapshot via REST	update IDs
Kraken	wss://ws.kraken.com/v2	per event	snapshot in stream	checksum per book
KuCoin	wss://ws-api.kucoin.com	per event	snapshot via REST	sequence IDs
Bitget	wss://ws.bitget.com/v2	100ms	snapshot in stream	checksum per update

Coinbase, OKX, and Kraken include the initial snapshot in the websocket stream when you subscribe. Binance, Bybit, and KuCoin require a separate REST call. The stream-included snapshot is faster to start with but harder to recover from on disconnect because you need to resubscribe to get a new one. The REST snapshot pattern is more flexible.

Latency budget

Order book data is only useful if you can act on it within the timeframe relevant to your strategy. For research and analytics, 100-500ms latency is fine. For market making or arbitrage, you need single-digit milliseconds. The hosting choice changes accordingly.

Binance hosts its main matching engine in AWS Tokyo. The lowest latency to its websocket is from a server in ap-northeast-1. Coinbase runs from AWS US-East-1 (Ashburn). OKX runs from Hong Kong. Bybit from AWS Singapore. If you want sub-10ms feeds you have to host in the same AZ as the exchange and pay for cross-connect or AWS DX.

For research-grade scraping, a $40/month VPS in Tokyo or Singapore from Hetzner, OVH, or Vultr gets you under 30ms to most exchanges. That is fine for everything except active trading strategies.

Normalizing depth across exchanges

Every exchange publishes its order book in slightly different formats. To do any cross-exchange analysis you need a common schema. The minimum viable normalized record:

from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class NormalizedBook:
    exchange: str
    symbol: str  # canonical, like "BTC-USDT"
    captured_at_ms: int  # exchange timestamp
    received_at_ms: int  # local receive timestamp
    bids: List[Tuple[float, float]]  # sorted desc
    asks: List[Tuple[float, float]]  # sorted asc
    sequence: int

The two timestamps are critical. captured_at_ms is what the exchange reported. received_at_ms is when your collector got the message. The difference tells you network latency. Without both you cannot diagnose feed degradation.

Symbol normalization is annoying but mechanical. Binance uses BTCUSDT, Coinbase uses BTC-USD, OKX uses BTC-USDT. Maintain a mapping table:

SYMBOL_MAP = {
    "binance": {"BTCUSDT": "BTC-USDT", "ETHUSDT": "ETH-USDT"},
    "coinbase": {"BTC-USD": "BTC-USD", "ETH-USD": "ETH-USD"},
    "okx": {"BTC-USDT": "BTC-USDT", "ETH-USDT": "ETH-USDT"},
}

Note that USD and USDT are not the same. Coinbase quotes against USD, most others against USDT. For arbitrage analysis you need to track this carefully and convert via the USDT/USD pair.

Storage: the real cost driver

Naive storage of every depth update across 50 pairs and 5 exchanges produces 100-500 GB per day. Compressed Parquet brings it down to 20-80 GB. Most operations cannot afford to keep raw deltas indefinitely, so the standard pattern is tiered:

Hot: last 7 days of full deltas in ClickHouse or DuckDB
Warm: last 90 days of 1-second OHLCV bars + L2 snapshots every minute
Cold: indefinite history of 1-minute snapshots in compressed Parquet on S3

This gets you to roughly $50-150/month in storage for 5-exchange, 50-pair coverage. The compression ratio matters: Parquet with ZSTD level 9 hits about 8:1 on order book data because most price levels do not change between snapshots.

import pyarrow as pa
import pyarrow.parquet as pq

def write_snapshot_batch(snapshots: list, path: str):
    table = pa.Table.from_pylist(snapshots)
    pq.write_table(
        table,
        path,
        compression="zstd",
        compression_level=9,
        use_dictionary=True,
    )

ClickHouse handles the hot path well. A modest 4-core server holds 30 days of full deltas for 50 pairs across 5 exchanges with room to query.

Recovering from a corrupted book

The deepest source of bugs is when your in-memory book diverges from reality and you do not notice for hours. Defensive checks that run continuously in the background:

Top-of-book sanity: every minute, fetch the top of book via REST and compare to your cached book’s top bid/ask. A divergence greater than 0.5% on a liquid pair means resync.
Crossed book detection: the highest bid should never exceed the lowest ask. If it does, the book is corrupted; trigger a full snapshot resync.
Negative quantity detection: quantities should always be positive. Any negative value indicates a delta-application bug.
Sequence gap counter: track the rate of detected gaps per hour. A baseline of 0.5-2 gaps per hour per pair is normal. A sudden jump indicates network degradation worth investigating before it affects downstream consumers.
Idle channel detection: if no message arrives for a pair in 30 seconds during a normally active hour, force a reconnect. Idle does not always mean closed; the socket can hang in a half-open state.

A simple watchdog process running these checks across all pairs adds about 3% overhead and catches >95% of the silent corruption cases that otherwise show up as bad model outputs days later.

Connection management at scale

Each websocket connection costs file descriptors and memory. A single Python process with websockets library can comfortably handle 100-200 simultaneous connections. Past that, you fragment across processes or move to a more efficient runtime.

For the 50-pair, 5-exchange scenario, the practical architecture is one process per exchange handling all that exchange’s pairs as a single multiplexed subscription where the protocol allows it. Binance and Coinbase both accept multi-symbol subscribe messages on a single connection. OKX requires one subscription per channel but supports multiplexing across symbols.

async def binance_multi_symbol(symbols: list[str]):
    streams = "/".join([f"{s.lower()}@depth@100ms" for s in symbols])
    url = f"wss://stream.binance.com:9443/stream?streams={streams}"
    async with websockets.connect(url, ping_interval=20) as ws:
        async for raw in ws:
            msg = json.loads(raw)
            stream = msg["stream"]
            data = msg["data"]
            await process(stream, data)

Reconnect logic must include exponential backoff plus full snapshot refetch. A 30-second disconnect during a busy hour means thousands of missed updates. Just resuming the websocket subscription will give you a corrupted book.

Proxy considerations

Public websocket endpoints generally do not enforce strict per-IP rate limits because they are designed for HFT clients with stable connections. You usually do not need a proxy for normal volume.

The exception is REST snapshot endpoints. Binance imposes a weight-based rate limit on REST: 6000 weight per minute per IP, and the depth endpoint at limit=1000 costs 50 weight. That allows about 120 snapshots per minute. If you have more than 50 pairs and a busy gap-recovery day, you can blow through the limit.

For that case, route REST snapshot traffic through a small datacenter proxy pool of 5-10 IPs. The websocket can stay on your direct connection. We cover the broader proxy strategy in our best datacenter proxy providers 2026 review.

Storage cost worked example

A practical breakdown for the standard 50-pair, 5-exchange research deployment looks like this. Raw delta volume averages 18 GB per exchange per day during normal weeks, climbing to 50-80 GB during high-volatility events. Across 5 exchanges that is 90-400 GB per day uncompressed. After ZSTD-9 Parquet compression with column dictionaries, expect 11-50 GB per day landing in cold storage. At AWS S3 Standard pricing of $0.023 per GB-month and a 90-day rolling cold tier, total storage cost falls between $25 and $110 per month for the historical archive, plus another $40-80 per month for the ClickHouse hot tier on a 4 vCPU 8 GB box.

Egress is the silent cost killer. If you ever need to move 5 TB of historical Parquet to a different cloud for a research project, AWS will bill $450 in egress alone. Either run the analytics in the same region as the bucket or use Cloudflare R2 / Backblaze B2, both of which have free or near-free egress and price storage at $0.005-0.015 per GB-month. The R2 path saves real money once the archive grows past 1 TB.

Cross-exchange arbitrage signal extraction

The classic application of order book scraping is arbitrage detection. The simplest version: for every (base asset, quote asset) pair, find the highest bid across all exchanges and the lowest ask across all exchanges. If the highest bid is greater than the lowest ask plus fees, there is an opportunity (in theory).

In practice, transfer time, withdrawal fees, exchange-specific rules, and slippage eat most of the gap. But the same data lets you compute the cross-exchange spread distribution over time, which is genuinely useful for understanding market microstructure and for backtesting more sophisticated strategies.

def best_bid_ask(books: dict[str, NormalizedBook]) -> dict:
    best_bid = max((b.bids[0] for b in books.values() if b.bids), key=lambda x: x[0])
    best_ask = min((b.asks[0] for b in books.values() if b.asks), key=lambda x: x[0])
    return {
        "best_bid": best_bid[0],
        "best_ask": best_ask[0],
        "spread": best_ask[0] - best_bid[0],
    }

Common gotchas

A few traps from real production deployments:

Binance’s lastUpdateId from the REST snapshot is occasionally lower than the first delta U you have already buffered. The official spec says to discard buffered deltas where u <= lastUpdateId and apply the rest, but ensure your buffer holds at least 200 deltas during reconnect because the snapshot can lag a busy moment by several seconds.
Coinbase’s match channel and the level2 channel are separate. To compute correct mid-price you need only level2; matches are useful for trade history but should not feed the book directly.
OKX checksums are CRC32 over a specific concatenation of the top 25 levels. If the checksum fails twice in a row, OKX expects you to resubscribe, not to reconnect the socket.
Bybit’s v5 endpoint changed sequence semantics from v3. Old code that assumed continuous sequence numbers across all symbols on one connection will silently corrupt because v5 sequences are per symbol.
Kraken occasionally sends a snapshot mid-stream without warning when their internal book reconciliation kicks off. If you see a snapshot event after subscribing, treat it as the new baseline and drop your existing book state.
Time synchronization on the collector matters. NTP drift of even 50 ms makes the received_at_ms minus captured_at_ms latency metric meaningless. Run chrony with multiple peers and monitor offset.

Compliance and exchange terms of service

Most exchanges’ public market data is, by their own terms of service, freely usable for personal and commercial research. Redistributing the data as a real-time feed competing with the exchange’s institutional product is a different matter and usually requires a market data license.

Binance, Coinbase, and Kraken all explicitly permit personal trading and research use of public market data. Building a competing aggregator service that resells the data is gray area. For the typical research, alpha generation, or in-house analytics use case you are fine.

External authoritative reference: the Binance API documentation covers websocket spec and rate limits.

FAQ

Q: can I scrape order books without websockets?
Yes, you can poll the REST depth endpoint. But REST polling at 1-second cadence misses 99% of updates and consumes more rate-limit budget than the websocket equivalent. Websockets are the only sensible choice for production.

Q: how do I handle exchange downtime?
Most exchanges schedule maintenance windows in advance. Subscribe to their status APIs and stop reconnect attempts during announced windows to avoid wasting API budget. For unannounced outages, use exponential backoff capped at 5 minutes between retries.

Q: do I need to store every delta or are snapshots enough?
Depends on use case. For backtesting you want deltas because you need event-by-event playback. For analytics on spreads and depth, 1-second snapshots are usually sufficient and 100x cheaper to store.

Q: what about DEX order books?
DEXs publish state on-chain, not via websocket. You read it via RPC calls or via subgraph queries. Uniswap V3 and similar AMMs do not have order books at all; they have liquidity curves. dYdX and similar perp DEXs do have order books and expose them via gRPC. Different problem.

Q: how do I detect washtrading from order book data?
Look for orders that get placed and immediately taken by the same exchange’s matching engine, identical-size orders bouncing between two price levels, and trade volume spikes that do not correspond to depth movement. This is a deep topic; treat it as a separate downstream analysis on top of the raw scraped data.

Q: should I use a managed message broker like Kafka in front of the storage layer?
Only if you have multiple downstream consumers that need the live feed. For a single-consumer research pipeline, Kafka adds operational overhead without value. A simple in-process queue from listener to writer is enough. Kafka becomes worth the cost once you have a trading bot, an analytics dashboard, and a backtester all consuming the same feed concurrently.

Q: do I need a colocation server for arbitrage research?
No. Colocation matters for execution, not research. A Tokyo VPS that gets order book data 30-50 ms after the matching engine is plenty for spotting historical opportunities and modeling spreads. Save the colocation budget for after you have validated the strategy.

Q: can a single Python process really keep up with a busy day?
With asyncio and uvloop, yes, up to about 200 simultaneous connections and several thousand messages per second. Past that, switch to Rust, Go, or split across processes per exchange. Most research workloads never hit that ceiling.

Closing

Scraping crypto order books at scale in 2026 is a tractable engineering problem if you respect the wire-level details: sequence numbers, gap recovery, snapshot resyncs, and the difference between exchange and local timestamps. The first month is the hard part. Once your collector survives a Sunday night Coinbase reconnect storm and a Binance maintenance window without corrupting state, you have a pipeline that generates millions of dollars worth of vendor data for the cost of a small VPS. For the broader market data infrastructure picture see our crypto-defi category hub.

May 7, 2026

Pharmaceutical Pricing Surveillance with Proxies in 2026

Pharmaceutical pricing surveillance is one of the more brutal scraping problems you’ll run into in 2026. Drug prices vary by 300-800% across markets for the same molecule, reference pricing cascades across borders in real time, and every major pharmacy chain, government formulary, and parallel importer has deployed bot detection that’s gotten meaningfully harder over the past 18 months. if you’re building a pricing intelligence pipeline for pharma, you need proxies — and the wrong proxy type will burn through budget while returning garbage data.

Why pharma price data is hard to collect at scale

Pharmacy websites aren’t e-commerce. they’re hybrid: part public-facing storefront, part regulated formulary display, part insurance portal. that layering means you’re dealing with multiple anti-bot systems on the same domain. Cloudflare sits in front of the storefront. a separate WAF protects the insurance lookup. the government formulary runs on a CMS that rate-limits by IP class.

Most scrapers fail here because they treat it like a retail job. it isn’t. pricing pages often require a simulated user journey — landing page, category browse, product view — before the actual price renders. Headless browsers with datacenter IPs get flagged in under two seconds on CVS, Walgreens, Boots UK, and most EU pharmacy chains.

The core requirement is residential or mobile proxies with real ASN diversity. for country-specific pricing (which is the whole point of cross-market surveillance), you need geo-targeted IPs that actually resolve to the right country from the pharmacy’s own geolocation provider. a UK residential IP that GeoIP2 maps to the US will serve you US pricing. that’s a subtle data quality failure that’s easy to miss for weeks.

Proxy types: what actually works on pharmacy targets

Not all proxy categories perform equally across pharmaceutical sources. here’s a practical breakdown:

Proxy type	Best for	Avg. success rate (pharmacy sites)	Cost per GB
Datacenter (shared)	Government formulary bulk pulls	40-60%	$0.50-2
Datacenter (dedicated)	Internal pricing APIs with known IP allowlists	70-85%	$3-8
Residential rotating	Retail pharmacy chains, insurance portals	85-95%	$8-18
Mobile (4G/LTE)	Heavy JS sites, TLS fingerprint-sensitive targets	90-97%	$15-40
ISP static	Authenticated portals, scraper-aware login flows	88-94%	$10-25

Mobile proxies earn their cost premium on pharmacy targets because mobile user-agents are treated differently by most bot management vendors. DataDome and PerimeterX both apply lighter fingerprinting pressure to mobile TLS profiles. if you’re hitting Boots, Lloyds, or DocMorris, mobile IPs with rotating sessions are worth the spend.

For government sources — NHS drug tariff, FDA Orange Book, AIFA (Italy), GKV-Spitzenverband (Germany) — datacenter proxies are usually fine. these endpoints aren’t trying to sell you something. they have rate limits but rarely full bot detection stacks. the exception is NIHDI (Belgium) and some Spanish CCAA formularies that route through Cloudflare.

Building the collection pipeline

A workable pharma pricing pipeline in 2026 looks something like this:

Segment targets by detection class (government vs. retail vs. insurance portal)
Assign proxy tiers by segment — don’t waste mobile IPs on government PDFs
Implement per-session fingerprint consistency: same proxy, same user-agent, same accept-language header for the full session
Add 3-8 second randomized delays between requests on retail targets
Parse prices with currency normalization at ingest, not post-hoc — FX drift over a multi-day crawl creates phantom price gaps

Here’s a minimal session config using Python requests + a rotating residential proxy:

import requests
import random
import time

PROXY = "http://user-country-GB:pass@residential.provider.com:8080"

headers = {
    "User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_3 like Mac OS X) AppleWebKit/605.1.15",
    "Accept-Language": "en-GB,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Referer": "https://www.google.co.uk/",
}

session = requests.Session()
session.proxies = {"https": PROXY}
session.headers.update(headers)

def fetch_price_page(url):
    time.sleep(random.uniform(3.0, 8.0))
    resp = session.get(url, timeout=20)
    resp.raise_for_status()
    return resp.text

The country code in the proxy username (country-GB) is how most residential providers handle geo-targeting. verify that the IP you’re assigned actually resolves correctly using ipinfo.io before starting a full crawl run — provider geo accuracy varies more than the marketing suggests.

Similar infrastructure patterns come up in other regulated sectors. the approach used for proxies for insurance underwriting data maps closely to pharma pricing — different industry, same problem of market-specific data sitting behind bot protection. and if you’re monitoring regulatory filings rather than retail prices, the proxies for banking compliance monitoring pattern of semi-static ISP IPs for authenticated government portals is directly applicable.

Cross-market normalization and reference pricing loops

The real analytical challenge isn’t collection. it’s normalization. pharmaceutical prices exist in at least four layers:

Ex-factory price: what the manufacturer charges the distributor
Wholesale price: distributor markup, varies by national regulation
Retail pharmacy price: often fixed by formulary
Patient out-of-pocket: after insurance rebates, co-pays, patient assistance programs

Most surveillance programs target retail because that’s what’s publicly visible. but parallel importers and tender monitors need ex-factory data, which means scraping manufacturer portals, national procurement databases, and sometimes government tender documents.

Reference pricing is where things get interesting. Germany’s GKV reimbursement benchmarks cascade into Austrian, Czech, and Slovak pricing within 6-18 months. tracking this in real time means consistent crawls across all four markets simultaneously, with timestamps accurate enough to detect which market moves first. a 24-hour crawl lag is enough to miss the signal.

For generic drug launch monitoring specifically — where the first-mover price in one market often predicts the floor price in adjacent markets — there’s a full treatment in tracking generic drug launches across global markets with proxies. that’s the resource to start with if you’re building a launch-day alert system.

Avoiding bans and managing crawl health

A few things that kill pharma pricing pipelines that wouldn’t matter on softer targets:

TLS fingerprinting: pharmacy chains running DataDome check JA3/JA4 hashes. the requests library default TLS profile is flagged. use curl-cffi or a managed headless browser service (Browserless, Apify) for these targets.
Cookie replay: some insurance portals invalidate sessions after 15-20 minutes. don’t cache cookies across sessions.
Price field rendering: heavily JS-rendered price fields (CVS uses React, Walgreens uses Angular) won’t appear in raw HTML. Playwright or Puppeteer with residential proxies is the right tool, not requests.

Ban recovery matters too. if you hit a 429 or 503 on a government formulary, back off 30-60 minutes before retrying. these are often soft rate limits that reset on a fixed schedule. rotating IPs into the same blocked ASN won’t help — you need proxy rotation that also switches ASN, which most residential pools do automatically but some cheaper providers don’t.

For teams managing multiple data collection pipelines, proxies for logistics fleet tracking and public transit data covers ASN diversity management in depth, and those practices translate directly. the duplicate-record detection patterns from proxies for insurance fraud detection: public records mining are also useful when deduplicating price records across overlapping sources that cover the same product from diffrent angles.

Bottom line

For pharmaceutical pricing surveillance, residential rotating proxies are the baseline for retail targets, mobile proxies are worth the premium on fingerprint-sensitive sites, and datacenter IPs are fine for government formularies. verify geo-targeting before a full run, normalize currency at ingest, and use curl-cffi or a managed browser service on DataDome-protected targets. DRT covers proxy infrastructure across regulated industries in depth — the same patterns show up across every sector where pricing is geographically fragmented and bot protection is real.

May 7, 2026