Category: Uncategorized

Running scrapers on Cloudflare Workers in 2026

Running scrapers on Cloudflare Workers is the kind of architectural decision that sounds clever in a slack thread and either pays off massively or crashes against runtime limits within a week. Workers give you a globally-distributed serverless runtime with sub-50ms cold starts, free egress to anywhere, and tight integration with KV/R2/D1/Durable Objects for state. The catch is that each Worker invocation has a 30-second CPU limit (or 5 minutes with longer-duration plans), 128 MB of memory, no persistent disk, and an aggressive limit on subrequests. For some scraping workloads these limits are fine. For others they are fatal.

This guide covers what Workers can and cannot do for scraping in 2026, the cases where the architecture wins, the limits that bite, and a complete working scraper implementation including state management via Durable Objects, queue management, and the new Browser Rendering API for JS-heavy pages.

What Workers offer that other serverless does not

The Workers value proposition for scrapers:

Sub-50ms cold start: each invocation feels instant, no Lambda-style 5-second cold starts
Free egress: outbound HTTPS to anywhere is free, no per-GB egress charges
300+ POPs: requests originate from whichever POP is closest to the target, naturally distributing IPs
Built-in KV, R2, D1: persistent state without separate infrastructure
Durable Objects: stateful coordination for queues, rate limits, locks
Browser Rendering API: real Chromium rendering integrated with Workers in 2024
Cron Triggers: scheduled execution without separate scheduler

For official Cloudflare Workers docs, see Cloudflare’s developer site.

What Workers cannot do for scrapers

The limits that catch teams:

30 sec CPU limit per invocation (Workers Standard) or 5 min (Workers Unbound, costs more)
128 MB memory (Standard) or 1 GB (Unbound)
50 subrequests (Standard) or 1000 (Unbound) per invocation
No persistent disk: no SQLite files, no temp directories
Limited Node.js compat: many npm packages do not work
Outbound IP is Cloudflare’s: no proxy support (requires fetch-via-proxy patterns)
No long-lived connections: WebSockets supported but not for scraping use
Browser Rendering API is metered: $0.20 per browser-minute, not free

The CPU limit is the most cutting one. A scraper that fetches 100 pages and parses them all in one invocation will hit 30 seconds easily. The pattern that works is: each Worker invocation does one small unit of work, persists state to KV/D1/Durable Object, and either schedules the next invocation or returns.

When Workers win for scraping

Workers shine when:

High-volume, light-per-page scraping (1000s of pages, each <1 sec to fetch)
Geo-distributed scraping (target wants to see traffic from many countries)
Scheduled lightweight jobs (cron-triggered, completes in seconds)
API endpoints that need a global edge layer wrapping internal scrapers
Webhook receivers that trigger scraping flows
Real-time price monitoring at scale

When Workers do not win:

Heavy JavaScript-rendered pages (Browser Rendering API helps but adds latency)
Sites with strict TLS fingerprinting (Worker fetch uses Cloudflare’s fingerprint)
Pages requiring proxies (no native proxy support)
Long-running scraping that needs minutes per page
Stateful scraping with complex local state

The pattern that wins: small, frequent, distributed scraping. The pattern that loses: long, complex, single-machine.

Setup: a basic Worker scraper

Install Wrangler:

npm install -g wrangler
wrangler login

Create a Worker:

wrangler init my-scraper --type javascript
cd my-scraper

The basic Worker that fetches a page:

// src/index.js
export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url).searchParams.get("url");
    if (!url) return new Response("Missing url parameter", { status: 400 });

    try {
      const resp = await fetch(url, {
        headers: {
          "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                       + "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
          "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
          "Accept-Language": "en-US,en;q=0.9",
        },
      });
      const html = await resp.text();

      // Parse with HTMLRewriter (Worker's built-in HTML parser)
      const titles = [];
      const rewriter = new HTMLRewriter().on("h1", {
        text(text) {
          if (text.text.trim()) titles.push(text.text);
        },
      });
      await rewriter.transform(new Response(html)).text();

      return new Response(JSON.stringify({ url, titles }), {
        headers: { "Content-Type": "application/json" },
      });
    } catch (err) {
      return new Response(JSON.stringify({ error: err.message }), { status: 500 });
    }
  },
};

Deploy:

wrangler deploy

This Worker accepts a url query parameter, fetches it, extracts H1 titles, and returns JSON. Try it: curl https://my-scraper.your-subdomain.workers.dev?url=https://example.com.

Storage layer: KV, R2, D1, Durable Objects

Each storage option has tradeoffs:

storage	use case	latency	cost
KV	global k-v store, eventually consistent	<50ms read	$0.50/M reads
R2	object storage (S3-like)	varies	$0.36/M operations + storage
D1	SQL database (SQLite at edge)	<50ms	$0.001/1k rows read
Durable Objects	strongly consistent state, stateful	<10ms	$0.15/M requests
Workers Cache	HTTP-style cache	<10ms	included

For scraping state:

Crawl queue: Durable Object (FIFO, strongly consistent)
Visited URLs (dedup): KV (eventually consistent is fine)
Scraped data (records): D1 or R2
Per-host rate limits: Durable Object (atomic counters)
Page snapshots (HTML, screenshots): R2

A Durable Object for queue management:

// src/queue.js
export class CrawlQueue {
  constructor(state, env) {
    this.state = state;
    this.env = env;
  }

  async fetch(request) {
    const url = new URL(request.url);
    const action = url.searchParams.get("action");

    if (action === "push") {
      const body = await request.json();
      const queue = (await this.state.storage.get("queue")) || [];
      queue.push(...body.urls);
      await this.state.storage.put("queue", queue);
      return new Response(JSON.stringify({ size: queue.length }));
    }

    if (action === "pop") {
      const queue = (await this.state.storage.get("queue")) || [];
      const item = queue.shift();
      await this.state.storage.put("queue", queue);
      return new Response(JSON.stringify({ url: item, remaining: queue.length }));
    }

    return new Response("unknown action", { status: 400 });
  }
}

Bind it in wrangler.toml:

[[durable_objects.bindings]]
name = "QUEUE"
class_name = "CrawlQueue"

[[migrations]]
tag = "v1"
new_classes = ["CrawlQueue"]

Use from a Worker:

const id = env.QUEUE.idFromName("global");
const stub = env.QUEUE.get(id);
const resp = await stub.fetch("http://queue/?action=pop");
const { url } = await resp.json();

The Durable Object guarantees that two Workers calling pop simultaneously get different items (no race condition).

Browser Rendering API

For JavaScript-heavy pages, Cloudflare’s Browser Rendering API gives you Chromium in a Worker. It launched in 2024 and stabilized through 2025.

// wrangler.toml binding
// [browser]
// binding = "MYBROWSER"

import puppeteer from "@cloudflare/puppeteer";

export default {
  async fetch(request, env) {
    const url = new URL(request.url).searchParams.get("url");
    const browser = await puppeteer.launch(env.MYBROWSER);
    const page = await browser.newPage();
    await page.goto(url, { waitUntil: "networkidle0" });
    const html = await page.content();
    const screenshot = await page.screenshot();
    await browser.close();

    // Store screenshot in R2
    await env.SCREENSHOTS.put(`${Date.now()}.png`, screenshot);

    return new Response(html, {
      headers: { "Content-Type": "text/html" },
    });
  },
};

Pricing: $0.20 per browser-minute. A 5-second page render costs ~$0.017. For 1000 pages, that is $17. Compare to running your own browser pool: usually cheaper at high volume but with operational overhead.

Stealth on Workers

Worker fetch uses Cloudflare’s network. The TLS fingerprint is whatever Cloudflare’s outbound HTTP client uses, which is distinctive. Targets that fingerprint TLS see “Cloudflare Worker” patterns.

Mitigations:

For HTTP-only sites without TLS fingerprinting: standard fetch is fine
For sites with light fingerprinting: Browser Rendering API uses real Chromium
For heavy targets: Workers are not the right tool. Use them as orchestration, route the actual fetches through proxied scrapers elsewhere

Cloudflare also offers Smart Placement which routes Workers to a POP near the upstream service, reducing latency. For scraping, you usually want POPs distributed close to your targets.

Subrequest limits

Each Worker invocation can make up to 50 subrequests on Standard plan, 1000 on Unbound. A subrequest is any external fetch, KV operation, D1 query, etc.

For high-volume scraping, this means each Worker invocation can fetch up to 50 pages in parallel. Beyond that, you need to chain invocations:

async function scrapeWithChunking(urls, env) {
  const CHUNK_SIZE = 30;  // leave headroom under 50
  for (let i = 0; i < urls.length; i += CHUNK_SIZE) {
    const chunk = urls.slice(i, i + CHUNK_SIZE);
    await Promise.all(chunk.map(url => scrapeOne(url, env)));
  }
}

If a single chunk takes 30+ seconds CPU time, split into separate Worker invocations triggered by Cron or by Durable Object scheduling.

Cron Triggers for scheduled scraping

Workers support scheduled execution via Cron Triggers in wrangler.toml:

[triggers]
crons = ["*/5 * * * *", "0 0 * * *"]  # every 5 min and daily at midnight

Handler:

export default {
  async scheduled(event, env, ctx) {
    if (event.cron === "*/5 * * * *") {
      // Run every 5 minutes
      ctx.waitUntil(scrapePriceUpdates(env));
    }
    if (event.cron === "0 0 * * *") {
      // Daily full crawl
      ctx.waitUntil(fullCrawl(env));
    }
  },
};

ctx.waitUntil lets the work continue past the response return, useful for fire-and-forget scrapers.

Comparison: Workers vs other serverless for scraping

platform	cold start	max duration	egress cost	proxy support	browser support
Cloudflare Workers	<50ms	30s (5min Unbound)	free	none native	Browser Rendering API
AWS Lambda	100-3000ms	15 min	$0.09/GB	yes	via container
Google Cloud Functions	100-3000ms	9 min	$0.12/GB	yes	via container
Azure Functions	200-3000ms	10 min	$0.087/GB	yes	via container
Vercel Functions	100-1000ms	60s (Pro)	$0.40/GB	limited	yes
Deno Deploy	<50ms	50s	free	yes	no

For free egress and global edge distribution, Workers and Deno Deploy lead. For long-running tasks and proxy support, Lambda or GCF win. For browser rendering, Workers’ Browser Rendering API or Lambda with custom container.

For broader serverless-scraping patterns in JS, see Deno scraping libraries 2026 reviewed.

Architecture: Workers + Durable Object + R2

A complete distributed scraper:

   Cron Trigger (every 1 min)
            |
            v
   +----------------+
   |   Worker       |  reads queue, dispatches subrequests
   +----------------+
            |
   +--------+--------+
   |                 |
   v                 v
 Durable Object    Worker subrequests
 (queue, state)   (parallel fetches)
                       |
                       v
                  KV (dedup)
                  D1 (data)
                  R2 (HTML, screenshots)

Each Cron tick: Worker reads up to 50 URLs from the Durable Object queue, fetches them in parallel, parses, writes results to D1 and R2, marks URLs as visited in KV. If queue still has items, schedule another invocation immediately.

This pattern handles tens of thousands of URLs per hour on Workers Standard, scales to millions on Unbound.

Cost analysis

A scraper doing 1 million page fetches per month:

component	cost
Workers requests (1M)	$0.30
Workers CPU time	varies, ~$5-20
KV reads (visited check)	$0.50
D1 writes (results)	$1-3
R2 storage (HTML at 50 KB avg)	$0.75
Durable Object requests	$0.15
Browser Rendering (10% of pages, 5 sec each)	$1,000
Total without browser	~$10/month
Total with browser on 10% of pages	~$1,010/month

Without browser rendering, Workers are extremely cheap. With browser rendering, they are still competitive but no longer the absolute bargain. For pages that need browser rendering at scale, consider running Playwright on dedicated VMs and reserving Workers for HTTP-only flows.

Common pitfalls

Cold storage state on first invocation: Durable Objects start empty; handle nil/empty cases
HTMLRewriter is streaming, not DOM: cannot do complex selector queries
No npm packages with native code: pure JS only
Subrequest limit: hits silently when exceeded, your fetch returns an error
CPU limit applies to async work too: even waiting for a fetch counts if your Worker is processing
Time limit measured in CPU time: 30 seconds CPU != 30 seconds wall clock
No persistent disk: cannot save SQLite files, must use D1 or KV

For the Cloudflare Workers limits reference, see Workers limits.

Operational checklist

For production scrapers on Workers in 2026:

Wrangler 3+ for deploys
Workers Unbound plan if you need >30s CPU or >50 subrequests
Durable Objects for queues and rate limits
KV for dedup and lightweight state
D1 or R2 for results storage
Cron Triggers for scheduling
Monitor CPU time, subrequest count, error rate per Worker
Use Workers Logpush to stream logs to R2 or external SIEM
Reserve Browser Rendering API for pages that truly need JS
For TLS-fingerprinted targets, route through external proxy infrastructure
Test with wrangler dev --remote for true edge testing

FAQ

Q: can I use proxies with Workers?
Not natively. Workers fetch from Cloudflare’s network. To use proxies, you would need to route to a proxy service via fetch (a custom proxy that accepts HTTPS and forwards), which is awkward. For proxy-required scraping, run your scrapers elsewhere.

Q: how does Worker fetch’s TLS fingerprint look?
Distinct from any browser. Cloudflare’s outbound HTTP client has its own JA4. Targets that check TLS fingerprints can identify it. For TLS-sensitive targets, do not use Workers for the fetch.

Q: is Browser Rendering API a real Chrome?
Yes, it runs Chromium in Cloudflare’s infrastructure. Your Worker connects to it via the puppeteer-compatible API. It supports stealth patches but is metered.

Q: can I do long-running scraping on Workers?
Not in a single invocation. Use Durable Objects to break work into small chunks, with each Worker invocation handling one chunk. Coordinate via Cron or self-scheduled triggers.

Q: how do Workers compare to Vercel Edge Functions for scraping?
Workers have lower cold start and free egress; Vercel has Edge Functions in similar shape but with usage-based pricing including egress. For raw scraping cost at scale, Workers are cheaper. For Vercel projects already using their platform, Edge Functions integrate natively.

Common pitfalls in production Workers scraping

The first failure mode is the subrequest fan-out cliff. Workers Standard caps subrequests at 50 per invocation; Workers Unbound raises this to 1000. A scraper that processes a category page and queues 60 product fetches from a single Worker invocation hits the cap silently: the 51st fetch returns an error in the response object but does not throw. Your loop continues, your error counter never increments because you assumed fetch() would throw, and you end up with 49 successful fetches and 11 silently dropped requests. The fix is two-part: first, always check response.ok and response.status after every fetch; second, batch outbound fetches into chunks of 40 (leaving headroom for retries) and queue overflow into the Durable Object for the next Cron tick.

The second pitfall is HTMLRewriter buffering on slow upstreams. HTMLRewriter is streaming, but if your handler does any async work inside an element callback, it pauses the stream. A handler that fetches a related URL on every <a href> element it encounters effectively serializes the entire HTML parse, turning a 200ms streaming parse into a 30+ second one (one fetch per anchor). The fix is to collect the URLs synchronously into an array during the parse, then dispatch fetches in parallel after the parse completes:

const collectedUrls = [];
const rewriter = new HTMLRewriter().on("a[href]", {
  element(el) {
    collectedUrls.push(el.getAttribute("href"));  // sync only
  },
});
await rewriter.transform(response).text();
// Now fan out
const subResponses = await Promise.all(
  collectedUrls.slice(0, 40).map((u) => fetch(new URL(u, baseUrl)))
);

The third pitfall is Durable Object hibernation losing in-memory state. DOs hibernate after about 10 seconds of inactivity, and any state stored in instance variables (not in state.storage) is lost on wake. Scrapers that maintain a “currently processing” set in this.processing = new Set() find that the set is empty after every cron-triggered wake, leading to duplicate processing. Always persist coordination state via state.storage.put() and re-hydrate via state.storage.get() on every method entry. Hibernation is documented but easy to forget when your local wrangler dev keeps state in memory across reloads.

Real-world example: 10M-URL crawl with $40 monthly cost

A scraper team built a price-monitoring system that fetched 10 million product URLs per month across 200 ecommerce sites. The architecture used Workers Cron + Durable Objects + KV + D1 + R2:

// cron.js — runs every minute
export default {
  async scheduled(event, env, ctx) {
    const queueDO = env.QUEUE_DO.get(env.QUEUE_DO.idFromName("global"));
    const batch = await queueDO.fetch("https://internal/dequeue?count=40").then(r => r.json());

    if (!batch.urls.length) return;

    const results = await Promise.all(batch.urls.map(async (url) => {
      try {
        const r = await fetch(url, {
          headers: { "user-agent": "Mozilla/5.0 ..." },
          cf: { cacheTtl: 0 },  // never cache, always fresh
        });
        if (!r.ok) return { url, error: `status ${r.status}` };
        const html = await r.text();
        const price = extractPrice(html);  // sync HTMLRewriter elsewhere
        return { url, price, fetched_at: Date.now() };
      } catch (e) {
        return { url, error: e.message };
      }
    }));

    // Write results to D1 in one batched insert
    const stmt = env.DB.prepare(
      "INSERT INTO prices (url, price, fetched_at) VALUES (?, ?, ?)"
    );
    await env.DB.batch(results
      .filter(r => r.price)
      .map(r => stmt.bind(r.url, r.price, r.fetched_at))
    );
  },
};

Monthly billing breakdown:

Workers requests: 14.4M (43,200 cron ticks at 40 fetches + 14M subrequests) = $4.80
Workers CPU time at avg 80ms per invocation = $11.50
Durable Object requests = $1.20
D1 writes (10M rows) = $15
KV reads (dedup checks) = $5
R2 storage (sampled raw HTML, 100GB) = $1.50

Total: $39 per month for 10M monitored URLs. The same workload on AWS Lambda + RDS would have cost approximately $850 per month, dominated by RDS instance fees and Lambda invocation costs. The lesson: Workers wins decisively when the workload fits its constraints (no proxies needed, sub-second per page, results structured enough for D1).

Detection: when Workers are the wrong choice

Five signals that your scraping workload should NOT live on Workers:

Per-request proxy required: Workers cannot rotate through residential proxy pools efficiently. Move to Lambda or VMs.
Long-running JS challenges: Cloudflare’s own Under Attack mode takes 5-10 seconds, and other JS challenges take similar time. Workers Unbound max is 5 minutes but CPU time is metered separately. Use Browserbase or VM-based Playwright.
Heavy data parsing: A 5MB HTML page parsed via DOMParser easily exceeds the 30s Workers Standard CPU budget. HTMLRewriter handles streaming HTML but is awkward for non-trivial extraction.
Stateful sessions across many requests: Cookies and login flows that span dozens of requests work better in long-lived processes.
Custom TLS fingerprinting: Workers cannot impersonate Chrome’s TLS handshake. Targets that check JA4 see Cloudflare’s outbound fingerprint instead.

If any two of these apply, run your scraper on Fargate, EKS, or self-hosted infrastructure instead.

Wrapping up

Cloudflare Workers are a niche but powerful scraping platform when the workload fits: high volume, distributed, HTTP-light, with state in KV/D1/Durable Objects. Browser Rendering API closes the JS gap at the cost of metered pricing. Pair this with our scrapers vs Playwright integration patterns and Deno scraping libraries for the full serverless picture, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.

May 7, 2026

Scrapy + Playwright integration in 2026

Scrapy + Playwright integration is the standard answer when Scrapy’s speed and pipeline ergonomics meet pages that need a real browser to render. Scrapy alone runs at thousands of pages per second through asynchronous Twisted callbacks but cannot execute JavaScript. Playwright can drive a real browser through any page but is a single-page-at-a-time tool that does not give you Scrapy’s middleware, item pipelines, or deduplication. The scrapy-playwright plugin bridges them: Scrapy still handles the orchestration, scheduling, and pipelines, but pages route through Playwright when needed.

This guide covers scrapy-playwright 0.0.40+ on Scrapy 2.11+ in 2026, the setup steps, the page coroutines pattern, proxy and stealth integration, and production patterns that scale. Code is Python 3.12 throughout. By the end you will have a Scrapy project that can mix HTML-only requests (fast) and Playwright-rendered requests (slow but JS-capable) in one spider, with shared middleware and pipelines.

Why this combination wins

Scrapy alone is unbeatable for HTML-only scraping at scale. Its async model, request prioritization, deduplication, and pipeline architecture are mature. The weakness is that any JavaScript-rendered page returns empty HTML.

Playwright alone handles JS perfectly but has no concept of crawl orchestration. You write your own request queue, deduplication, retry logic, and item processing.

scrapy-playwright lets each page in a Scrapy spider opt into Playwright via a metadata flag. Pages without the flag stay as fast HTML requests. Pages with the flag route through a Playwright pool, render fully, and return rendered HTML to your callback as if Scrapy fetched it directly.

For Scrapy’s official docs, see Scrapy 2.11+ documentation. For Playwright’s, see Playwright Python documentation.

Installation

pip install scrapy scrapy-playwright playwright
playwright install chromium

Verify versions:

scrapy version       # 2.11.0 or later
python -c "import scrapy_playwright; print(scrapy_playwright.__version__)"
# 0.0.40 or later

Project setup

Create a Scrapy project:

scrapy startproject dynamic_scraper
cd dynamic_scraper

Edit dynamic_scraper/settings.py to enable scrapy-playwright:

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

PLAYWRIGHT_BROWSER_TYPE = "chromium"
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "args": [
        "--disable-blink-features=AutomationControlled",
        "--disable-features=IsolateOrigins",
    ],
}

# How many concurrent Playwright contexts to maintain
PLAYWRIGHT_MAX_CONTEXTS = 8
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 30000

# Reasonable defaults for mixed crawls
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0
RETRY_TIMES = 3

# Standard middleware for header rotation, etc
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
    "scrapy_user_agents.middlewares.RandomUserAgentMiddleware": 400,
    "dynamic_scraper.middlewares.ProxyMiddleware": 410,
}

# Item pipeline (whatever you need)
ITEM_PIPELINES = {
    "dynamic_scraper.pipelines.JsonExportPipeline": 300,
}

The download handlers route every HTTP request through scrapy-playwright. Pages without the playwright meta flag are still fetched via regular HTTP (scrapy-playwright detects this automatically), so you do not pay the browser cost for pages that do not need it.

A first spider mixing HTML and Playwright requests

Most real spiders mix request types. The product listing might be static HTML, but the product detail might require JavaScript. scrapy-playwright handles this cleanly:

# spiders/products.py
import scrapy
from scrapy_playwright.page import PageMethod


class ProductSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        # Listing page is plain HTML, no Playwright needed
        for product_url in response.css("a.product-link::attr(href)").getall():
            yield response.follow(
                product_url,
                callback=self.parse_product,
                meta={
                    "playwright": True,
                    "playwright_include_page": False,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", "div.product-detail"),
                    ],
                },
            )

        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

    def parse_product(self, response):
        yield {
            "title": response.css("h1.product-title::text").get(),
            "price": response.css("span.price-current::text").get(),
            "stock": response.css("div.stock-info::text").get(),
            "description": response.css("div.product-description").get(),
        }

Key points:

The listing page is fetched normally (no playwright: True in meta)
Product detail pages are fetched via Playwright (playwright: True)
playwright_page_methods runs on the Playwright page before returning the HTML
wait_for_selector ensures the JavaScript-rendered detail is present before scrapy-playwright captures the HTML

This pattern keeps the spider’s overall structure conventional while opting specific pages into the heavyweight rendering path.

Interacting with the page

Some scraping requires interaction (click a “Show more” button, scroll to load infinite content, fill a search form). For this, request the actual Playwright Page object back via playwright_include_page=True:

import scrapy
from scrapy_playwright.page import PageMethod


class InfiniteScrollSpider(scrapy.Spider):
    name = "infinite"
    start_urls = ["https://example.com/feed"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "playwright": True,
                    "playwright_include_page": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", "article.feed-item"),
                    ],
                },
                callback=self.parse_feed,
                errback=self.errback_close_page,
            )

    async def parse_feed(self, response):
        page = response.meta["playwright_page"]
        # Scroll to load more items
        for _ in range(5):
            await page.evaluate("window.scrollBy(0, document.body.scrollHeight)")
            await page.wait_for_timeout(2000)
        html = await page.content()
        await page.close()

        # Parse the now-loaded HTML with Scrapy selectors
        from scrapy.http import HtmlResponse
        new_response = HtmlResponse(url=response.url, body=html, encoding="utf-8")
        for article in new_response.css("article.feed-item"):
            yield {
                "title": article.css("h2::text").get(),
                "url": article.css("a::attr(href)").get(),
                "summary": article.css("p.summary::text").get(),
            }

    async def errback_close_page(self, failure):
        page = failure.request.meta.get("playwright_page")
        if page:
            await page.close()

The playwright_include_page=True means the Page object is attached to the response. Your callback can interact with it (scroll, click, wait) before extracting HTML. Always close the page in your callback or errback to avoid leaking browser contexts.

Proxy integration

Pass proxies to Playwright either at the launch level (one proxy for all contexts) or per-context (different proxy per request). Per-context is the more flexible pattern.

# settings.py addition
PLAYWRIGHT_CONTEXTS = {
    "default": {
        "proxy": {
            "server": "http://squid.internal:3128",
            "username": "scraper_user",
            "password": "secret",
        },
    },
}

For per-request proxies via a custom middleware:

# middlewares.py
import random


class ProxyRotationMiddleware:
    PROXIES = [
        "http://user1:pass1@proxy1.provider.com:8080",
        "http://user2:pass2@proxy2.provider.com:8080",
        "http://user3:pass3@proxy3.provider.com:8080",
    ]

    def process_request(self, request, spider):
        if request.meta.get("playwright"):
            # For Playwright requests, set the context with a specific proxy
            proxy = random.choice(self.PROXIES)
            user_pass, host_port = proxy.split("//")[1].split("@")
            user, password = user_pass.split(":")
            host, port = host_port.split(":")
            request.meta["playwright_context"] = f"proxy_{hash(proxy)}"
            request.meta["playwright_context_kwargs"] = {
                "proxy": {
                    "server": f"http://{host}:{port}",
                    "username": user,
                    "password": password,
                },
            }
        else:
            # For regular HTTP requests, set the standard proxy meta
            request.meta["proxy"] = random.choice(self.PROXIES)

scrapy-playwright caches contexts by name, so reusing the same proxy uses the same context. This avoids spawning a new browser context per request, which is expensive.

Stealth: integrating patchright

For sites that fingerprint, use patchright (Playwright stealth fork) instead of vanilla Playwright. patchright is a drop-in replacement, so you can swap it in via setting the playwright module:

# In your settings.py or as an environment variable
import patchright as playwright_module

PLAYWRIGHT_BROWSER_TYPE = "chromium"
# scrapy-playwright will use whatever playwright module is installed

Or install patchright as the playwright package (it shadows vanilla playwright):

pip install patchright
playwright install chromium

After install, scrapy-playwright automatically uses patchright’s launch logic, which applies stealth patches.

For fine-grained stealth on a per-request basis, inject scripts via init_script:

yield scrapy.Request(
    url,
    meta={
        "playwright": True,
        "playwright_context_kwargs": {
            "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                          "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        },
        "playwright_page_init_script": """
            Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
            Object.defineProperty(navigator, 'plugins', { get: () => [1, 2, 3] });
        """,
    },
)

For the full picture on browser fingerprinting, see canvas fingerprinting bypass and WebGL fingerprinting bypass.

Performance: how much does Playwright slow you down

Throughput comparison on a single worker:

request type	requests/sec
pure Scrapy HTML	200-500
scrapy-playwright (cached context, simple page)	5-10
scrapy-playwright (new context, complex SPA)	1-2
scrapy-playwright with full humanization	0.3-0.5

The cost is real. Use Playwright only for pages that need it. The mixed pattern (HTML for listings, Playwright for details) is the right balance for most sites.

To scale Playwright throughput:

Increase PLAYWRIGHT_MAX_CONTEXTS (each context costs ~100 MB RAM)
Reuse contexts across requests by giving them stable names
Run multiple Scrapy instances on different machines
Keep navigation timeout tight (30 seconds is usually enough)

Comparison: scrapy-playwright vs alternatives

approach	strengths	weaknesses
scrapy-playwright	mixed HTML/JS in one project, native Scrapy pipelines	adds Playwright overhead even for simple sites
scrapy-splash	older, lighter than Playwright	Splash is essentially abandoned, JS engine is dated
scrapy + selenium-wire	works	slow, fragile
scrapy + standalone Playwright service	clean separation	extra service to operate
pure Playwright	most JS-capable	no Scrapy pipelines, write your own everything
Crawlee (Node)	similar pattern in Node ecosystem	requires switching off Python

For most Python teams in 2026, scrapy-playwright is the right pick. For Node teams, Crawlee. For very simple JS sites, scrapy-splash still works but is on its way out.

For the alternative Crawlee path, see Scrapy Cloud vs Crawlee Cloud in 2026.

Item pipelines and Playwright

Item pipelines work identically whether the request was HTML-only or Playwright. The pipeline sees a Scrapy Item (or dict) and processes it. No changes needed:

# pipelines.py
import json
from datetime import datetime


class JsonExportPipeline:
    def open_spider(self, spider):
        self.file = open(f"output/{spider.name}_{datetime.now():%Y%m%d_%H%M%S}.jsonl", "w")

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(dict(item)) + "\n"
        self.file.write(line)
        return item

This is one of the main wins of scrapy-playwright over rolling your own: you get to keep all your existing Scrapy pipelines and middleware.

Common patterns

Wait for specific elements before extracting:

yield scrapy.Request(
    url,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod("wait_for_selector", "div.product-loaded"),
            PageMethod("wait_for_load_state", "networkidle"),
        ],
    },
)

Click a button to load content:

yield scrapy.Request(
    url,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod("click", "button.show-more"),
            PageMethod("wait_for_selector", "div.expanded-content"),
        ],
    },
)

Take a screenshot for debugging:

yield scrapy.Request(
    url,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod("screenshot", path="debug.png", full_page=True),
        ],
    },
)

Intercept network requests:

async def parse(self, response):
    page = response.meta["playwright_page"]
    # Capture all XHR responses
    captured_data = []
    page.on("response", lambda r: captured_data.append(r.url))
    await page.wait_for_load_state("networkidle")
    await page.close()
    yield {"all_xhr_urls": captured_data}

Production deployment

A production scrapy-playwright deployment looks like:

Compute: container with Chromium installed (use playwright-base images)
Concurrency: tune CONCURRENT_REQUESTS based on RAM (each Playwright context is ~100 MB)
Proxy: routed through Squid or commercial pool
Stealth: patchright as drop-in
Monitoring: Scrapy stats + Playwright context counter
Persistence: items to a database or queue, not just files
Scheduling: scrapyd, scrapy-cluster, or Airflow

A reasonable container Dockerfile:

FROM mcr.microsoft.com/playwright/python:v1.45.0-jammy

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

ENV PLAYWRIGHT_BROWSERS_PATH=/ms-playwright

CMD ["scrapy", "crawl", "products"]

Use the playwright-python base image to skip installing Chromium and dependencies separately.

Operational checklist

For production scrapy-playwright deployments in 2026:

Scrapy 2.11+, scrapy-playwright 0.0.40+
Use playwright-python base image for containers
patchright for stealth
Per-request opt-in to Playwright via meta flag
Always close pages in callbacks and errbacks
Use named contexts to reuse across requests
Tune CONCURRENT_REQUESTS based on RAM
Set navigation timeouts (30s default is reasonable)
Log per-spider stats including Playwright vs HTML request count
Pair with proxy rotation middleware
Monitor RAM usage closely (Playwright contexts leak if not closed)

FAQ

Q: can I use Playwright with Scrapy without scrapy-playwright?
You can, but it is painful. You would have to write your own download handler, manage the browser lifecycle, and integrate with Scrapy’s async model. scrapy-playwright handles all this for you.

Q: what about scrapy-splash?
Splash is essentially unmaintained as of 2024-2025. The JS engine is older and many modern SPAs do not render correctly. Use scrapy-playwright unless you have legacy reasons to stay on Splash.

Q: how do I handle Playwright authentication and cookies?
Pass cookies via the request meta or set them on the context:

meta={"playwright": True, "playwright_context_kwargs": {"storage_state": "auth.json"}}

auth.json is generated by saving Playwright’s context.storage_state() after login.

Q: can I run scrapy-playwright on AWS Lambda?
Technically yes via Playwright’s Lambda layers, but the cold start is brutal (10+ seconds per invocation just to launch Chromium). For Lambda, prefer dedicated browser services like Browserbase or run scrapy-playwright on Fargate or EKS instead.

Q: does scrapy-playwright support Firefox?
Yes. Set PLAYWRIGHT_BROWSER_TYPE = "firefox" and run playwright install firefox. Most patches work on Firefox too, though some stealth libraries are Chromium-focused.

Common pitfalls in production scrapy-playwright

The first failure mode that catches teams off guard is the page-object leak. scrapy-playwright passes the open Page to your callback via response.meta["playwright_page"], but if your parse function raises an exception or yields requests without closing the page, the browser context retains a reference. After 50-100 leaked pages, Chromium hits its per-context limit (typically 30 tabs depending on RAM) and new requests stall waiting for tab slots. The fix is a try/finally pattern in every parse method that touches playwright_page:

async def parse_product(self, response):
    page = response.meta.get("playwright_page")
    try:
        await page.wait_for_selector(".product-loaded", timeout=10000)
        title = await page.text_content("h1")
        yield {"title": title}
    except Exception as e:
        self.logger.error(f"parse failed: {e}")
        raise
    finally:
        if page:
            await page.close()

Add an errback that also closes the page on download failure, otherwise pages leak from network errors that never reach your parse callback.

The second pitfall is the headers-vs-Playwright disconnect. Setting headers on a scrapy.Request does not automatically pass them to Playwright when playwright=True. Scrapy treats those headers as the eventual download request, but Playwright bypasses Scrapy’s downloader entirely and uses its own. The result: your User-Agent rotation middleware appears to set a Chrome 124 UA but Playwright launches with whatever UA the browser instance has. The fix is to set headers via playwright_context_kwargs on each request:

yield scrapy.Request(
    url,
    meta={
        "playwright": True,
        "playwright_context_kwargs": {
            "user_agent": rotated_ua,
            "extra_http_headers": rotated_headers,
        },
    },
)

Each new context spawn now gets the rotated UA and headers. Beware: spawning a new context per request defeats Playwright’s connection reuse, so pair this with named contexts to amortize the cost across same-UA requests.

The third pitfall is silent JavaScript navigation that bypasses Scrapy’s URL tracking. When a page does window.location.href = "/next-page" mid-load, Playwright follows it and ends up at a URL different from what you originally requested. Scrapy’s response.url reflects the post-navigation URL, but your duplicate-detection middleware only saw the original URL. You can end up scraping the same final page repeatedly through different entry-point URLs. The fix is to track final URLs in your DupeFilter:

from scrapy.dupefilters import RFPDupeFilter
import hashlib

class FinalUrlDupeFilter(RFPDupeFilter):
    def request_seen(self, request):
        # If response has been received, dedupe on response.url too
        final_url = request.meta.get("playwright_final_url")
        if final_url:
            fp = hashlib.sha256(final_url.encode()).hexdigest()
            if fp in self.fingerprints:
                return True
            self.fingerprints.add(fp)
        return super().request_seen(request)

Set playwright_final_url in your parse callback after capturing page.url, then re-yield the request through the dedupe filter.

Real-world example: 50,000-product crawl with mixed HTML and Playwright

A scrapy-playwright project crawled an electronics retailer with 50,000 products across 800 category pages. Initial implementation used Playwright for everything and took 18 hours per full crawl with 90 percent CPU usage on a 16-core machine. The optimization that cut this to 2.5 hours was a tiered request strategy that classified URL patterns and used the cheapest method per pattern:

class TieredSpider(scrapy.Spider):
    name = "products_tiered"

    def start_requests(self):
        # Category pages: pure HTML, server-rendered
        for cat_url in self.category_urls:
            yield scrapy.Request(cat_url, callback=self.parse_category)

    def parse_category(self, response):
        # Extract product URLs from server-rendered HTML
        for url in response.css(".product-link::attr(href)").getall():
            full_url = response.urljoin(url)
            # Tier 1: standard product page (HTML works)
            if "/p/" in full_url:
                yield scrapy.Request(full_url, callback=self.parse_product_html)
            # Tier 2: SPA product detail (Playwright needed)
            elif "/spa-product/" in full_url:
                yield scrapy.Request(
                    full_url,
                    meta={
                        "playwright": True,
                        "playwright_page_methods": [
                            PageMethod("wait_for_selector", ".price-loaded"),
                        ],
                    },
                    callback=self.parse_product_playwright,
                )

    def parse_product_html(self, response):
        yield {
            "url": response.url,
            "title": response.css("h1::text").get(),
            "price": response.css(".price::text").get(),
        }

    async def parse_product_playwright(self, response):
        page = response.meta["playwright_page"]
        try:
            title = await page.text_content("h1")
            price = await page.text_content(".price")
            yield {"url": response.url, "title": title, "price": price}
        finally:
            await page.close()

Of the 50,000 products, only 8,000 needed Playwright (the SPA product pages). The remaining 42,000 went through pure Scrapy HTML at 200+ requests/second. Playwright handled the 8,000 SPA pages at 5 requests/second. Total throughput: 5.5 hours for the HTML tier and 27 minutes for the SPA tier, running concurrently in 2.5 hours total. The lesson: classify your URLs by render type and route accordingly. Default-to-Playwright is the most expensive choice you can make.

Comparison: scrapy-playwright settings tuning by site type

Different site types need different tuning profiles. A reference table from a 2026 production deployment:

site type	CONCURRENT_REQUESTS	PLAYWRIGHT_MAX_CONTEXTS	nav timeout	wait strategy
ecommerce SPA	16	8	30s	wait_for_selector on price
news article	32	16	15s	wait_for_load_state domcontentloaded
listing with infinite scroll	8	4	60s	manual scroll + wait_for_response
dashboard behind login	4	2	45s	wait_for_url + wait_for_selector
pricing page (heavy JS)	16	8	30s	wait_for_function on data-loaded attr

For listing pages with infinite scroll, use a manual scroll loop:

async def scroll_until_done(page, max_scrolls=20):
    for i in range(max_scrolls):
        prev_height = await page.evaluate("document.body.scrollHeight")
        await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        await page.wait_for_timeout(1500)
        new_height = await page.evaluate("document.body.scrollHeight")
        if new_height == prev_height:
            break  # no more content loaded

Without the scroll-until-done pattern, you get only the first viewport’s worth of items even when the listing has 500+. Pair with a per-spider MAX_SCROLL_ITERATIONS setting so a runaway page does not loop forever.

Memory profiling: detecting Playwright leaks

Production scrapy-playwright deployments suffer from RAM growth that takes hours to manifest. The cause is almost always context reuse without bounded lifetime. Each long-lived context accumulates DOM nodes, IndexedDB entries, and service worker state. After 4-6 hours of constant scraping through one context, RAM usage doubles. The detection pattern:

import psutil
import os

class MemoryMonitorMiddleware:
    def __init__(self):
        self.process = psutil.Process(os.getpid())
        self.baseline_mb = self.process.memory_info().rss / 1024 / 1024
        self.requests_processed = 0

    def process_response(self, request, response, spider):
        self.requests_processed += 1
        if self.requests_processed % 100 == 0:
            current_mb = self.process.memory_info().rss / 1024 / 1024
            growth = current_mb - self.baseline_mb
            spider.logger.info(
                f"requests={self.requests_processed} "
                f"rss={current_mb:.0f}MB growth={growth:.0f}MB"
            )
            if growth > 2000:  # 2GB growth, recycle contexts
                spider.logger.warning("memory bloat detected, recycling contexts")
                # Trigger context recycling in your context manager
        return response

Recycle contexts every 1000-2000 requests by closing them and letting scrapy-playwright spawn fresh ones. This caps RAM growth at the cost of one cold-context launch per recycle (about 800ms). For long-running spiders, this is a worthwhile trade.

Wrapping up

Scrapy + Playwright is the right combination for any Python scraping project that mixes simple HTML pages with JavaScript-heavy pages. scrapy-playwright glues them cleanly enough that you keep all of Scrapy’s pipeline ergonomics while getting Playwright’s rendering capability where you need it. Pair this guide with our Scrapy Cloud vs Crawlee Cloud and best Python scraping libraries 2026 writeups, and browse the framework-tutorials category on DRT for related tutorials.

May 7, 2026

Self-hosted proxy infrastructure: complete 2026 guide

Self-hosted proxy infrastructure is a category that ranges from “I run Squid in front of five purchased upstream proxies” to “I operate a 200-modem mobile proxy farm with ISP business contracts and four-9s uptime.” This guide is about the latter end of the spectrum: the build that actually saves money compared to buying from Bright Data or Oxylabs at scale, the operational reality of running it, and the break-even point where self-hosting starts paying off.

The numbers in this guide come from operating real proxy infrastructure on the SMP (Singapore Mobile Proxy) and similar farms in the Singapore-Malaysia region. The pricing, hardware choices, and ISP dynamics are 2026 reality. If you are scraping under 1 million pages per month, self-hosting is probably not worth it. Above 10 million, the unit economics are compelling. In between is the gray zone where it depends on your engineering capacity.

Why self-host

The case for self-hosted proxies in 2026:

Cost: at scale, self-hosted mobile proxies cost $0.30-$1 per GB versus $5-15/GB from premium vendors
Control: you choose the ISPs, IP ranges, geo distribution, and rotation behavior
Reliability: no upstream vendor outages affecting your business
Privacy: your traffic does not pass through a third-party gateway that logs everything
Customer billing: you can resell capacity if you have spare throughput

The case against:

Capex: hardware, racks, network equipment cost upfront ($50k+ for a serious build)
Opex: data plans, electricity, internet uplink, maintenance hours
ISP relationships: requires business contracts, sometimes legal entity in the country
24/7 ops: things break at 3am, you fix them
Capacity inflexibility: cannot quickly add 500 more IPs the way Bright Data can

Most teams should buy proxies. Self-hosting is for the small number of teams whose volume justifies the operational burden.

Architecture: the SMP-style mobile proxy farm

A typical mid-scale mobile proxy farm:

              + customers (scrapers, ad-verification, OSINT) +
                              |
                              | HTTPS to public IP
                              v
                  +-------------------------+
                  |   Edge proxy (Squid +   |
                  |   ProxySmart controller) |
                  +-------------------------+
                              |
                              | private 10.x routing
                              v
              +-------+-------+-------+-------+
              |       |       |       |       |
           server1 server2 server3 server4 ...
              |       |       |       |       |
              | USB  | USB  | USB  | USB  |
              v       v       v       v       v
            modem1  modem2  modem3  modem4  modem5  ...
            (LTE)  (LTE)  (LTE)  (LTE)  (LTE)
              |       |       |       |       |
              v       v       v       v       v
            ISP A    ISP B    ISP A    ISP C    ISP B   (carrier APNs)

Each modem produces one mobile IP via PPP-over-LTE through the carrier’s APN. The server hosts a process that exposes that modem as an HTTP/SOCKS5 port to the controller. The controller assigns ports to customers, handles rotation, and meters usage.

This pattern scales modularly: add a server with 50 modems, expose 50 more ports, customers see them as additional capacity. Each modem is one IP with full carrier-grade NAT obfuscation, which is the reason mobile IPs are valuable for scraping (they look like millions of real users sharing the carrier’s NAT pool).

Hardware: per-server build

Per-server hardware for a 50-modem build:

component	spec	purpose
chassis	4U server, 12-bay or rackmount mining frame	physical housing
CPU	Intel i5-12500 or Ryzen 5 5600	enough for 50 modem PPP daemons
RAM	32 GB DDR4	one PPP daemon per modem, ~500 MB each
storage	500 GB NVMe SSD	OS, logs, modem state
USB hubs	4x 16-port industrial USB 3.0 hubs	64 USB ports total
modems	50x Huawei E3372 or similar 4G/LTE dongles	the actual modems
SIM cards	50x carrier SIMs with data plans	network access
network	2x 1 Gbit Ethernet (LACP)	uplink to switch
PSU	redundant 850W	reliability

Cost per server: roughly $4,000-6,000 in hardware plus $300-500/month in SIM data plans (varies by country and plan size).

For the full multi-server build:

4-server cluster: ~$20,000 hardware + $1,500/month opex
Yields 200 mobile IPs
Throughput: ~10 Mbit/s per modem aggregate, so 2 Gbit/s peak across the farm
Bandwidth budget: usually 50-200 GB per modem per month (carrier-dependent)

Modems are the failure-prone component. Plan for 5-10% modem replacement per year. Keep spare modems and SIMs on hand.

Software stack

Each server runs:

Linux: Ubuntu 22.04 LTS or 24.04 LTS (most modem driver compatibility)
PPP daemon: pppd per modem with carrier-specific APN config
ProxySmart: per-modem HTTP/SOCKS proxy daemon exposing the PPP IP as a port
Controller agent: communicates with the central controller for port assignment, rotation, telemetry

ProxySmart specifically wraps each modem’s PPP interface and exposes it as an HTTP CONNECT proxy. Customers connect to a port (e.g., server2.example.com:8123) and traffic egresses through the modem’s mobile IP.

For DNS within the proxy farm, run a local resolver (unbound or dnsmasq) so that DNS queries do not leak to ISP resolvers, which can throttle or log them.

Network: ISP business contracts

The hardest part of self-hosting mobile proxies is not the technology, it is the SIM card supply. Personal-line carriers throttle or terminate SIMs that are clearly used for proxying. You need:

Business SIM contracts: a corporate account with 50+ lines, possibly negotiated bulk pricing
Data plans: typically 100-200 GB/month per line, billed as a corporate package
APN access: business APN (sometimes different from consumer APN)
Rotating contract terms: some carriers prohibit “automation” in their AUP; you need terms that allow this use

Country-specific dynamics in 2026:

region	ISP-friendliness	typical contract
Singapore	high (Singtel, Starhub, M1 all have business plans)	$30-50 per 100 GB
Malaysia	medium (Maxis, Digi, Celcom; some restrictions)	$20-40 per 100 GB
Indonesia	medium (Telkomsel, Indosat; data caps strict)	$15-30 per 100 GB
Thailand	high (AIS, Truemove, dtac all have plans)	$25-45 per 100 GB
US	low (Verizon, AT&T, T-Mobile have hostile AUPs)	$40-80 per 100 GB
Europe	varies (Vodafone, Orange friendlier; Telekom hostile)	$20-50 per 100 GB

Without a business relationship, the SIMs get shut down within weeks of heavy use. With a relationship, they last months or years.

Carrier rotation: rebooting modems for fresh IPs

Mobile IPs rotate on PPP reconnection. To get a new IP:

Send AT command to modem to reset
Wait for PPP to drop and reestablish
New IP is assigned by the carrier

Time to rotate: 10-30 seconds per modem. Customers see this as a “rotate” API call that returns a new IP within 30 seconds.

# Sketch of modem rotation script
import subprocess
import time

def rotate_modem(modem_id: int) -> str:
    # Send AT command via the modem's serial interface
    subprocess.run(
        ["echo", "AT+CFUN=4", ">", f"/dev/ttyUSB{modem_id}_at"],
        shell=True,
    )
    time.sleep(2)
    subprocess.run(
        ["echo", "AT+CFUN=1", ">", f"/dev/ttyUSB{modem_id}_at"],
        shell=True,
    )
    # Wait for PPP to come back up
    for _ in range(30):
        time.sleep(1)
        ip = subprocess.check_output(
            ["ip", "addr", "show", f"ppp{modem_id}"]
        ).decode()
        if "inet " in ip:
            new_ip = ip.split("inet ")[1].split("/")[0]
            return new_ip
    raise TimeoutError(f"modem {modem_id} did not reconnect")

In production, ProxySmart handles this via a REST API exposed per modem. Customers call POST /rotate/<port_token> and get a fresh IP within 30 seconds.

Customer-facing API

A typical customer interaction:

# Customer's scraper uses the proxy port
curl -x http://user:pass@proxy.example.com:8123 https://target.example.com

# When they need rotation
curl -X POST -H "Authorization: Bearer api_key" \
    https://proxy.example.com/api/v1/rotate/port_8123

# Check current IP
curl -x http://user:pass@proxy.example.com:8123 https://api.ipify.org

The controller exposes a REST API for customer self-service:

POST /api/v1/rotate/<port>: trigger rotation
GET /api/v1/status/<port>: current IP, data used, plan limit
POST /api/v1/swap/<port>: swap to a different modem (different ISP/IP range)
GET /api/v1/usage: per-customer billing info

For an example of this kind of platform, see Singapore Mobile Proxy which exposes mobile proxies via this pattern.

Costs and break-even

Real numbers for a 200-modem mobile farm in Singapore:

line item	monthly cost
SIM data plans (200 lines, 100 GB each)	$7,000
Server power (4 servers, 24/7)	$400
Internet uplink (1 Gbit business)	$300
Rack space (colo, 2U each * 4)	$500
Hardware amortization ($20k over 36 months)	$560
Engineering time (1 day/week)	$2,000
Total	$10,760/month

Capacity: 200 IPs * 100 GB/month = 20,000 GB/month total bandwidth. At a customer price of $1.50/GB, revenue is $30,000/month at full utilization. Margin: 65% gross.

For comparison, buying 20,000 GB of mobile proxy from Bright Data at $8/GB costs $160,000/month. Self-hosting saves $150k/month at this scale.

Break-even: roughly 30 modems / 3,000 GB/month is the point where self-hosting beats buying. Below that, the engineering and ops overhead is not worth the savings.

Monitoring

Critical metrics:

Per-modem PPP uptime: alert when any modem is down >5 minutes
Per-modem data usage: track against carrier plan limits
Per-port latency: median and p95 from edge proxy
Per-port throughput: bytes/sec by direction
Carrier IP reputation: periodically check IPs against known-bot lists
Modem hardware health: temperature, USB hub status

Tooling:

Prometheus + node_exporter on each server
Custom exporter for modem state (PPP uptime, IP, data usage)
Grafana dashboards for operator view
AlertManager routing to PagerDuty or Telegram

Sample dashboard panels:

panel	query
modem uptime	sum(up{job=”modem”}) / count(up{job=”modem”})
data usage by modem	sum_over_time(modem_bytes_total[7d])
latency p95 by port	histogram_quantile(0.95, port_latency_seconds_bucket)
customer requests/sec	rate(controller_requests_total[1m])

Failure modes

Real failure modes from running this kind of farm:

USB hub disconnect: a 16-port hub stops responding, takes down 16 modems. Fix: replace hub or USB-reset.
Modem firmware crash: single modem stops responding. Fix: USB-reset or physical reboot.
PPP daemon hang: ppp0 interface up but no traffic. Fix: kill pppd, restart.
Carrier IP block: ISP blocks one of your IP ranges. Fix: rotate to different APN or wait.
SIM data plan exhausted: SIM stops working. Fix: switch to backup SIM or wait for plan reset.
DNS poisoning by carrier: ISP injects DNS responses. Fix: use local resolver via DoT/DoH to bypass ISP.
Server crash: takes down all modems on that server. Fix: redundancy, multiple servers.
Network uplink flap: brief outages drop customer connections. Fix: dual uplinks with LACP.

The annual maintenance burden is roughly 100-200 engineering hours per 200-modem farm.

Comparison: self-host vs commercial

dimension	self-hosted	commercial
cost per GB	$0.30-1 at scale	$5-15
scale time	weeks to add capacity	minutes
reliability	99-99.9% (you own it)	99.9-99.99% (vendor SLA)
geo coverage	one country usually	global
IP rotation	per modem, your control	massive pools, vendor control
customer trust	depends on your reputation	well-known brand
ops burden	high	zero
break-even	~3,000 GB/month	starts at zero

Self-host when: high volume in one geo, regulatory needs to keep traffic local, custom rotation requirements, or building a product.

Commercial when: low volume, multi-geo needed, no engineering capacity for ops, want zero burden.

For broader proxy strategy, see best mobile proxy providers 2026 and building a custom rotating proxy pool with Squid.

Legal and compliance

Operating proxy infrastructure has compliance touchpoints:

Local telecoms law: in some countries, providing internet access requires a license
Customer KYC: depending on jurisdiction, you may need to verify customer identity
Logging requirements: some countries require ISPs to retain connection logs
AUP enforcement: customer abuse (CSAM, fraud) can implicate you if not addressed
Tax: B2B services have VAT/GST in many jurisdictions

For OWASP guidance on related security topics, see OWASP Top 10. For the specifics of running a regulated telecom service in your country, consult local counsel.

Operational checklist

For running a self-hosted mobile proxy farm in 2026:

Hardware sized for 50-200 modems per server
Business SIM contracts with carriers
Local DNS resolver (unbound)
Per-modem PPP daemons + ProxySmart
Central controller for port assignment
Customer REST API
Prometheus + Grafana monitoring
24/7 alerting on PPP uptime and data usage
Spare modems on hand (10% of fleet)
Regular IP reputation checks against deny lists
Customer billing system (Stripe, BMS, custom)
Legal review of AUP and terms

Smaller-scale alternative: residential at home

If you cannot justify mobile farm capex, a smaller residential setup at scale:

Get 5-10 residential ISP lines at home or in a colo with multiple ISPs
Run Squid or HAProxy on a single machine routing across the lines
Each line gives you one residential IP
Total cost: $200-500/month for the lines plus colo

Capacity is small (maybe 1-10 GB/day per line) but the IPs are real residential, very high quality. Useful for low-volume high-value scraping (account creation, captcha solving) where IP quality matters more than capacity.

FAQ

Q: how long do mobile SIMs last before getting flagged or shut down?
With a business contract: months to years. Without one: weeks. The relationship with the carrier is the durability factor, not the technology.

Q: can I run this on AWS or GCP?
Mobile proxies require physical USB-connected modems, so no. You can run the controller and edge proxy in cloud, but the modems themselves need physical hosting (your office, a colo, or a residential location).

Q: what about IPv6?
Most carriers in 2026 are dual-stack. ProxySmart supports both. For target sites that prefer IPv4 (most do), force IPv4 routing. For target sites with IPv6, expose the v6 address.

Q: is self-hosting worth it for residential proxies (not mobile)?
Less so. Residential ISP lines are slower to provision, harder to multi-tenant, and have hostile AUPs. Mobile is the better fit for self-hosting because of carrier-grade NAT and the established business-line market.

Q: how do I price for customers?
Common patterns: per-GB ($1-3/GB), per-port-month ($30-100 per dedicated port), or hybrid. Mobile proxies command higher prices than datacenter or residential. Match competitor pricing in your geo and undercut by 20% to get traction.

Common pitfalls in production self-hosted proxy infrastructure

The first failure mode is USB hub contention. Mobile proxy farms typically connect 16-32 USB modems per host through powered USB 3.0 hubs. Cheap hubs (under $40) commonly share one USB controller across all 16 ports, which means total throughput across the hub caps at roughly 480 Mbit/s shared. With 16 LTE modems each capable of 50-100 Mbit/s, the hub bandwidth becomes the bottleneck before the modems do. The fix is per-port-controlled hubs (Yepkit YKUSH3 or VIA-chipset hubs from D-Link DUB-H7 generation) where each port has its own USB controller chain. Budget $80-120 per 7-port hub. Verify with lsusb -t that each modem reports under a separate Bus rather than a shared one.

The second pitfall is carrier-side IP rotation policy mismatch. Different carriers rotate the LTE-assigned IP at different intervals: Singtel rotates every 4 hours by default, M1 rotates every 24 hours, StarHub rotates only on PDP context teardown (so it can stay sticky for days). If your scraper expects a stable IP per session and you provision a Singtel modem, your customer’s session token bound to the original IP becomes invalid after 4 hours. Match carrier choice to use case: use StarHub for sticky sessions, use Singtel for high-rotation requirements. Document each modem’s carrier in your inventory and route customer traffic accordingly.

The third pitfall is modem firmware silent regression. Quectel and Fibocom modems push firmware updates over the air for some operators. A firmware bump from EM05CEFAR06A07M4G to EM05CEFAR07A02M4G can change the AT command set, the PPP negotiation timing, or the TLS-handshake-passthrough behavior. Your provisioning scripts that worked yesterday produce silent failures today. The mitigation is firmware pinning where the modem allows it (AT+QFOTADL=0 to disable Quectel OTA on Quectel EM05/EM06/EC25 modems), or a weekly automated firmware-version inventory that flags any modem whose AT+CGMR output changed since last week.

Real-world example: month-end IP exhaustion recovery

A self-hosted mobile farm running 60 modems across three Singapore carriers hit a customer-facing outage on month-end of February 2026. Sixteen Vivifi modems exhausted their 100GB monthly data caps within the first 12 hours of February 28 because three large customers had over-rotated through the same proxy pool. Carrier behavior on cap exhaustion: the modem still attaches, PPP comes up, but every TCP connection times out or returns 503 from a carrier portal page. The pool’s automated health checks reported the modems “online” because PPP was up, but real customer traffic was 100 percent failing.

The recovery had three steps:

# Step 1: detect data exhaustion via active probe (not just PPP state)
async def is_modem_alive(port: int) -> bool:
    try:
        resp = await fetch_via_proxy(
            f"http://192.168.1.1:{port}",
            target_url="https://1.1.1.1/cdn-cgi/trace",
            timeout=5,
        )
        return resp.status == 200 and "ip=" in resp.text
    except Exception:
        return False

# Step 2: auto-rotate exhausted modems out of the active pool
exhausted = [m for m in MODEMS if not await is_modem_alive(m.port)]
for modem in exhausted:
    await mark_modem_unavailable(modem)

# Step 3: rotate to LTESpace reserve SIMs from the spare inventory
reserve = await get_reserve_sims(carrier="vivifi", min_data_gb=80)
for spare_sim, dead_modem in zip(reserve, exhausted):
    await swap_sim(dead_modem.imei, spare_sim.iccid)

The recovery took 47 minutes from first alert to full pool restoration. Lessons: PPP-state monitoring is necessary but not sufficient (must also probe an active TCP destination), reserve SIM inventory needs to be maintained at 20 percent of fleet size to absorb month-end exhaustion spikes, and customer billing should track per-port data usage daily so heavy users can be throttled before they exhaust shared pool capacity.

Wrapping up

Self-hosted proxy infrastructure pays off above ~3,000 GB/month if you have engineering capacity for ops. Below that threshold, buying from commercial vendors is the right call. For teams that build into self-hosting, the unit economics are very good: 65% gross margins, durable competitive advantage, customer trust through ownership of the stack. Pair this with our building a custom rotating proxy pool with Squid and best mobile proxy providers 2026 guides for both the build-vs-buy decision and the technical implementation, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.

Related comparison: See how Bright Data stacks up against a dedicated Singapore mobile network in our Singapore Mobile Proxy vs Bright Data comparison.

May 7, 2026

Building a custom rotating proxy pool with Squid in 2026

Building a custom rotating proxy pool with Squid is one of the cheapest ways to gate egress traffic across a fleet of upstream proxies without paying for a premium gateway. You buy 50 residential or mobile proxies from various providers, point Squid at all of them as parent peers, expose a single internal port to your scraping fleet, and let Squid rotate which upstream takes each request. Done correctly, this gives you per-customer rate limiting, per-target proxy affinity, automatic failover when an upstream dies, and the ability to drop in or remove upstream peers without touching your scraper code.

This guide walks through Squid 6.x on Ubuntu 22.04, the upstream peer configuration that does rotation and failover, ACLs that keep your pool from being abused, monitoring that actually works, and a Python client that integrates cleanly. Every config snippet is production-tested and the Squid ACL choices reflect lessons from running scraper farms that survived three years of growth.

Why Squid for proxy pooling

Squid is overkill for many use cases but well-suited for proxy pooling specifically because:

It supports multiple parent peers with rotation and failover
ACL system is mature and powerful
Logging is verbose enough for forensic debugging
Performance is excellent (handles 10k+ requests/second on modest hardware)
Free and battle-tested since 1996
Well-documented, with the official wiki covering most edge cases

What Squid does not do well: SOCKS5 forwarding (Squid is HTTP-first), per-request authentication to different upstreams, or websocket forwarding. For SOCKS5 use 3proxy or a custom Go service. For websockets use a forward proxy designed for it.

Architecture overview

   scrapers (multiple servers)
         |
         | HTTP CONNECT / GET on internal port
         v
   +-----------+
   |   Squid   |  <- rotates across upstream peers
   +-----------+
         |
   +-----+-----+--------+----------+
   |     |     |        |          |
  ISP1  ISP2 mobile1 residential  ...  (upstream proxies)
   |     |     |        |
   +--> internet

Scrapers connect to one Squid endpoint (port 3128 by default). Squid picks an upstream peer per request, forwards through it, and returns the response. Scrapers see a stable single endpoint while Squid handles rotation and failover internally.

Installing Squid 6.x on Ubuntu

sudo apt-get update
sudo apt-get install -y squid

# Verify version (should be 6.x in 2026 on Ubuntu 22.04 with backports or 24.04 baseline)
squid -v | head -1

For the latest Squid features, especially TLS interception and improved peer health checks, use Ubuntu 24.04 or build from source.

Base configuration

Replace /etc/squid/squid.conf with this base. We will layer on parent peers and ACLs after.

# /etc/squid/squid.conf — base scraper proxy pool

http_port 3128

# Logging
access_log /var/log/squid/access.log squid
cache_log /var/log/squid/cache.log

# Cache disabled (this is a forward proxy pool, not an HTTP cache)
cache deny all
cache_store_log none

# DNS
dns_v4_first on
positive_dns_ttl 1 hour
negative_dns_ttl 1 minute

# Connection limits
client_lifetime 1 hour
read_timeout 60 seconds
connect_timeout 30 seconds
request_timeout 30 seconds

# Header forwarding hygiene (do not leak client IP via X-Forwarded-For)
forwarded_for delete
via off

# Reject CONNECT to non-standard ports
acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 443
acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports
http_access deny !Safe_ports

The key choices: cache disabled (we are forwarding, not caching), forwarded_for delete (do not leak client IPs to upstreams), via off (do not advertise Squid in headers).

Adding upstream peers

Each upstream proxy is a cache_peer directive. The example below adds five parent peers with round-robin selection and per-peer authentication.

# /etc/squid/squid.conf — parent peer pool

# Parent peers: format is host port http_port options
cache_peer proxy1.provider.com parent 8080 0 round-robin no-query \
    login=user1:pass1 name=peer_1
cache_peer proxy2.provider.com parent 8080 0 round-robin no-query \
    login=user2:pass2 name=peer_2
cache_peer proxy3.provider.com parent 8080 0 round-robin no-query \
    login=user3:pass3 name=peer_3
cache_peer proxy4.provider.com parent 8080 0 round-robin no-query \
    login=user4:pass4 name=peer_4
cache_peer proxy5.provider.com parent 8080 0 round-robin no-query \
    login=user5:pass5 name=peer_5

# Force all traffic through parent peers (never go direct)
never_direct allow all

Options explained:

parent: this is an upstream HTTP proxy
8080: port to connect to on upstream
0: ICP port (we set to 0 because no ICP)
round-robin: rotate across peers with this option in round-robin
no-query: do not send ICP queries
login=user:pass: HTTP Basic auth credentials for the upstream
name=peer_N: human-readable name for logs and metrics

The never_direct allow all line is critical. Without it, Squid falls back to direct connection if all peers are down, which leaks your origin IP. With it, requests fail when peers are exhausted, which is the safer behavior for scrapers.

Reload and verify

sudo squid -k parse  # syntax check
sudo systemctl reload squid

# Test by curling through it
curl -x http://localhost:3128 https://httpbin.org/ip
# Expected: an IP from one of your upstream pools, not your origin

If you see your origin IP, never_direct is not working. Check that the line is present and Squid has been reloaded.

Rotation strategies

Squid supports several peer selection strategies via cache_peer options:

option	behavior
`round-robin`	each request goes to next peer in order
`weighted-round-robin`	round-robin with per-peer weight
`carp`	consistent hash by URL (same URL always goes to same peer)
`userhash`	consistent hash by client IP
`sourcehash`	consistent hash by client IP and port

For most scraper pools, round-robin is the default and gives even distribution. For situations where you want the same target site to always egress through the same proxy (some sites set IP-bound cookies), use carp:

cache_peer proxy1.provider.com parent 8080 0 carp no-query login=user1:pass1
cache_peer proxy2.provider.com parent 8080 0 carp no-query login=user2:pass2
# ... etc

CARP hashes each URL and consistently picks the same peer for the same URL. This means a session of requests to the same target uses the same proxy, which avoids tripping IP-binding checks.

Failover and health checks

Squid pings parent peers periodically and removes failing ones from the pool. Configure the failure threshold:

# Mark a peer dead after 3 consecutive failures
connect_timeout 30 seconds
peer_connect_timeout 10 seconds
dead_peer_timeout 5 minutes

When a peer is dead, Squid skips it until the dead_peer_timeout elapses. After that, it tries again. If the peer responds, it is brought back into the pool.

For more aggressive failover, write a custom health check that probes upstreams every minute and rewrites the Squid config. Sketch:

#!/usr/bin/env python3
# squid_peer_healthcheck.py
import requests
import subprocess
import time

PEERS = [
    ("proxy1.provider.com", 8080, "user1", "pass1"),
    ("proxy2.provider.com", 8080, "user2", "pass2"),
    # ...
]

def check_peer(host, port, user, password):
    try:
        proxy_url = f"http://{user}:{password}@{host}:{port}"
        r = requests.get(
            "https://httpbin.org/ip",
            proxies={"https": proxy_url},
            timeout=10,
        )
        return r.status_code == 200
    except Exception:
        return False

def write_squid_config(healthy_peers):
    with open("/etc/squid/peers.conf", "w") as f:
        for i, (host, port, user, password) in enumerate(healthy_peers, 1):
            f.write(
                f"cache_peer {host} parent {port} 0 round-robin no-query "
                f"login={user}:{password} name=peer_{i}\n"
            )

def reload_squid():
    subprocess.run(["sudo", "systemctl", "reload", "squid"], check=True)

def main():
    healthy = [p for p in PEERS if check_peer(*p)]
    print(f"{len(healthy)}/{len(PEERS)} peers healthy")
    write_squid_config(healthy)
    reload_squid()

if __name__ == "__main__":
    main()

Run via cron every minute. Include peers.conf in your main squid.conf via:

include /etc/squid/peers.conf

This pattern auto-removes failing peers without manual intervention.

ACL hygiene

By default, Squid is open to everything once you grant http_access. For a scraper pool, restrict access to your scraper IPs:

# Define allowed clients
acl scraper_clients src 10.0.0.0/8 192.168.0.0/16 172.16.0.0/12

# Allow only those IPs
http_access allow scraper_clients
http_access deny all

For authenticated client access (when scrapers run on internet IPs), use HTTP Basic auth:

auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwd
auth_param basic realm "Scraper Proxy Pool"

acl authenticated proxy_auth REQUIRED
http_access allow authenticated
http_access deny all

Generate the password file:

sudo htpasswd -c /etc/squid/passwd scraper_user
sudo chown proxy:proxy /etc/squid/passwd
sudo chmod 640 /etc/squid/passwd

Per-tenant rate limiting

If you serve multiple internal teams from one Squid pool, rate limit per team:

# Define teams by subnet
acl team_a src 10.1.0.0/16
acl team_b src 10.2.0.0/16

# Delay pools (rate limiting)
delay_pools 2

# Pool 1: team_a, 10 MB/s aggregate, 1 MB/s per host
delay_class 1 2
delay_parameters 1 10000000/10000000 1000000/1000000
delay_access 1 allow team_a
delay_access 1 deny all

# Pool 2: team_b, 5 MB/s aggregate
delay_class 2 1
delay_parameters 2 5000000/5000000
delay_access 2 allow team_b
delay_access 2 deny all

This caps total bandwidth per team, useful when one runaway scraper would otherwise saturate your upstream contracts.

Logging

Default Squid log format is fine for debugging. For monitoring, switch to a structured log format:

logformat scraperjson { \
    "ts": "%ts.%03tu", \
    "client_ip": "%>a", \
    "duration_ms": %tr, \
    "status": "%>Hs", \
    "bytes_sent": %<st, \
    "method": "%rm", \
    "url": "%ru", \
    "peer_used": "%<a", \
    "user_agent": "%{User-Agent}>h" \
}

access_log /var/log/squid/access.log scraperjson

Then ingest via Vector, Fluent Bit, or a similar log shipper into Loki or Elasticsearch for querying. Common queries:

Per-peer success rate: count of status=200 group by peer_used
Per-team bandwidth: sum of bytes_sent group by client_ip block
Slow upstreams: p95 of duration_ms group by peer_used

These queries highlight which upstream peers are degrading and which scrapers are hammering the pool.

Monitoring with Prometheus

Squid exposes runtime stats via squidclient mgr:info. Convert to Prometheus metrics with a small exporter:

# squid_exporter.py
from prometheus_client import start_http_server, Gauge
import subprocess
import re
import time

requests_total = Gauge("squid_requests_total", "Total requests", ["state"])
peer_status = Gauge("squid_peer_status", "Peer up/down", ["peer"])

def parse_squid_info():
    out = subprocess.check_output(["squidclient", "mgr:info"]).decode()
    # Parse lines like "Number of HTTP requests received: 12345"
    m = re.search(r"Number of HTTP requests received:\s+(\d+)", out)
    if m:
        requests_total.labels(state="received").set(int(m.group(1)))

def parse_peer_status():
    out = subprocess.check_output(["squidclient", "mgr:server_list"]).decode()
    # Parse peer status lines
    for match in re.finditer(r"Host\s*:\s*(\S+).*?Status\s*:\s*(\w+)", out, re.DOTALL):
        peer, status = match.group(1), match.group(2)
        peer_status.labels(peer=peer).set(1 if status == "ALIVE" else 0)

if __name__ == "__main__":
    start_http_server(9301)
    while True:
        parse_squid_info()
        parse_peer_status()
        time.sleep(15)

Scrape from Prometheus and visualize peer health in Grafana. Alert when any peer flips to dead or aggregate request rate drops below baseline.

Python client integration

Scrapers integrate with the Squid pool by setting it as the HTTPS proxy. Most clients support this via standard environment variables or per-call config.

For curl_cffi:

from curl_cffi import requests

resp = requests.get(
    "https://target.example.com",
    impersonate="chrome124",
    proxies={
        "http": "http://scraper_user:secret@squid.internal:3128",
        "https": "http://scraper_user:secret@squid.internal:3128",
    },
)

For Playwright:

from patchright.async_api import async_playwright

async def fetch():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": "http://squid.internal:3128",
                "username": "scraper_user",
                "password": "secret",
            },
        )
        # ...

The single Squid endpoint masks the entire upstream pool from your scraper code. Add or remove upstream peers without changing scraper config.

Comparison: Squid vs alternatives

tool	strength	weakness
Squid	mature, ACLs, monitoring	HTTP-first, complex config
HAProxy	very fast, modern config	less ACL flexibility for proxy use
Nginx (with stream module)	fast, widely known	weaker for proxy pooling specifically
3proxy	SOCKS5 support	smaller community, less feature-rich
custom Go service	exactly what you want	maintenance burden
commercial gateway (Bright Data, Oxylabs)	zero infra	expensive at scale

For HTTP/HTTPS proxy pooling, Squid is the right pick. For SOCKS5, 3proxy or a custom Go service. For very high throughput, HAProxy or a custom service.

For broader proxy infrastructure patterns, see self-hosted proxy infrastructure: complete 2026 guide.

Operational considerations

A Squid pool in production needs:

Hardware: 4 vCPU, 8 GB RAM handles 10k req/s for typical scraper traffic
Bandwidth: budget for outgoing + incoming, rule of thumb 1-2 Mbit/s per active scraper worker
Disk: 100 GB for logs (rotate daily, keep 30 days)
Monitoring: peer status, request rate, error rate, p95 latency
Alerting: any peer dead, error rate >5%, latency p95 >1s
Backup: config and password file in version control
Security: ACL restricting access, never expose 3128 to the internet

For scaling, run multiple Squid instances behind a load balancer (HAProxy or a TCP load balancer). Each Squid instance has the same peer pool, so requests distribute evenly across the cluster.

Common failure modes

All peers marked dead: usually a transient upstream provider issue or DNS failure. Check upstream connectivity directly.
High latency: an upstream is slow. Check per-peer p95 latency in your logs.
High error rate: upstream is degraded. Health check script should auto-remove.
Connection refused: Squid is not running or wrong port. Check systemctl.
407 Proxy Auth Required: ACL is blocking. Check auth_param and http_access order.
403 Forbidden from Squid itself: the target hit a Squid-side ACL. Check http_access rules.
Memory growing without bound: a buggy Squid version or misconfigured cache. Check that cache deny all is set.

For OWASP guidance on proxy security, see the OWASP Proxy security cheat sheet.

Operational checklist

Squid 6.x on Ubuntu 22.04 or 24.04
Cache disabled, forwarded_for delete, via off
Parent peers with login auth and round-robin
never_direct allow all
ACL restricting client access
Health check cron rotating peers in/out
Structured JSON access log
Prometheus exporter + Grafana dashboard
Alerting on peer status and error rate
Config in version control
Multiple Squid instances behind LB for scale

FAQ

Q: how many upstream peers do I need?
Match your concurrency. If you run 100 concurrent scraper workers and each makes 1 request/second, you have 100 req/s. With 5 peers in round-robin, each peer handles 20 req/s. Most residential proxy plans handle that comfortably. For high-volume work, 20-50 peers gives you headroom and failover capacity.

Q: can Squid rotate proxies on every request to a single domain?
Yes with round-robin. Squid picks the next peer per request regardless of destination. For per-domain affinity (same domain always uses same peer), use carp.

Q: does Squid support SOCKS5 upstreams?
Squid 6.x added SOCKS5 support via the cache_peer connection-auth=off type=parent options, but it is less mature than HTTP support. For SOCKS5-heavy work, 3proxy is more reliable.

Q: how do I add or remove peers without restart?
Edit /etc/squid/peers.conf and reload Squid: sudo systemctl reload squid. Reload is graceful and does not drop active connections.

Q: can I use Squid for HTTPS interception?
Yes with SSL bumping, but it requires installing a CA certificate on every client. For scrapers, this is usually unnecessary because you do not need to inspect HTTPS content, just forward it. Stick to CONNECT method for HTTPS.

Common pitfalls in production Squid pools

The first failure mode is silent peer exhaustion under high concurrency. Squid’s cache_peer with round-robin does not automatically queue requests when peers are busy: it picks the next peer in sequence regardless of that peer’s current load. If your scraper fleet sends 500 concurrent requests through 5 peers, each peer receives 100 simultaneous connections. Most residential providers cap concurrent connections at 50-100 per credential set, so half your requests stall or fail with upstream timeouts. The fix is to add max-conn=N to each cache_peer line (Squid 5.0+) so Squid stops sending new requests to peers at their capacity:

cache_peer proxy1.provider.com parent 8080 0 round-robin no-query \
    login=user1:pass1 name=peer_1 max-conn=50

The second pitfall is DNS leakage to upstream proxies. By default Squid resolves hostnames locally before forwarding, which means the upstream proxy receives an IP address rather than a hostname. Many residential proxy providers route by hostname (their internal load balancer chooses an exit IP based on the destination domain), so receiving a pre-resolved IP causes them to skip routing optimizations or return 502s. Add client_dst_passthru off plus host_verify_strict off and configure Squid to send the original CONNECT host:

host_verify_strict off
client_dst_passthru off
forwarded_for delete

For HTTP CONNECT (HTTPS tunneling), Squid passes the original host to the upstream by default. Verify by checking the upstream’s logs that they receive CONNECT example.com:443 rather than CONNECT 93.184.216.34:443.

The third pitfall is Squid’s stale connection pool to dead upstreams. When a peer dies and dead_peer_timeout expires, Squid retries it. If the peer is still dead, Squid waits for the connect_timeout (30s default) before failing the request. During that wait, upstream requests through that peer hang, and your scrapers see 30-second latency spikes. Set peer_connect_timeout 5 seconds to fail fast on dead peers, and pair with the active health check script that runs every 60 seconds to remove dead peers from the config entirely:

peer_connect_timeout 5 seconds
connect_timeout 15 seconds

Real-world example: 95th percentile latency tuning

A scraper farm running 80 concurrent workers through a Squid pool of 12 residential upstreams saw average latency of 800ms but p95 of 14 seconds. Investigation showed two of the 12 upstreams were experiencing intermittent 10-15 second response delays during certain hours, but Squid’s round-robin still routed traffic to them. The fix combined three changes: first, lowering peer_connect_timeout from 30s to 5s so dead-feeling peers were skipped faster; second, adding max-conn=40 on each peer so a slow peer did not accumulate stuck connections; third, switching from round-robin to weighted-round-robin and assigning the two slow peers weight=1 while the fast peers got weight=10:

cache_peer fast_proxy1.provider.com parent 8080 0 weighted-round-robin no-query \
    login=user1:pass1 name=fast_1 weight=10 max-conn=40
cache_peer slow_proxy1.provider.com parent 8080 0 weighted-round-robin no-query \
    login=user2:pass2 name=slow_1 weight=1 max-conn=40

After deployment, p95 latency dropped from 14s to 2.1s within an hour and p99 from 22s to 4.3s. The slow peers still received 1/10th of traffic (useful for monitoring whether they recovered) without dragging down the overall pool experience. The lesson: Squid’s defaults assume homogeneous peers, but real-world residential pools are heterogeneous and need weight-based shaping plus aggressive timeouts to keep the tail latency bounded.

Wrapping up

A Squid-based rotating proxy pool gives you per-tenant control, automatic failover, and simple integration with any scraping client for the cost of a small VM and 50 lines of config. It pays for itself within weeks compared to commercial gateway pricing if you have moderate volume. Pair this with our self-hosted proxy infrastructure and best residential proxy providers 2026 guides for the full stack, and browse the dev-tools-projects category on DRT for related infrastructure deep-dives.

May 7, 2026

Best Go scraping libraries 2026 ranked

Best Go scraping libraries in 2026 occupy a niche that is small but unusually high-leverage. Go’s concurrency model (goroutines and channels) maps almost perfectly to the scraping problem, and Go’s compiled binary makes deployment dramatically simpler than Python or Node alternatives. The downside is library breadth: the Go scraping ecosystem has fewer options than Python’s, and the existing libraries are less actively maintained on average. For specific workloads (high-throughput HTTP scraping, distributed crawler workers, scraping infrastructure embedded in Go services), Go is the right choice and the libraries that exist are excellent. For one-off scrapers or projects that benefit from a rich ecosystem, Python or Node remain easier.

This guide ranks the Go scraping libraries actually worth using in 2026, with honest performance comparisons, clear use case mapping, and the gotchas specific to Go’s approach.

Why Go for scraping

Three reasons Go is interesting for scrapers:

Concurrency: a goroutine costs about 2 KB of stack memory. You can run 10,000+ concurrent goroutines on a modest server. Compared to Python’s coroutine overhead and Node’s event loop limits, Go’s concurrency is genuinely different in scale.

Compile-once deploy-anywhere: a Go binary is a single static file. Deployment to a new server or container is scp and ./scraper. No virtualenv, no node_modules, no version drift between dev and prod.

HTTP performance: Go’s net/http standard library is fast enough that “scraping” and “high-performance HTTP service” use the same toolkit. fasthttp pushes performance even further for extreme throughput needs.

Three reasons Go is sometimes wrong:

Smaller library ecosystem: fewer parsers, fewer pre-built scrapers, less community content.

No native browser automation: Chromedp and Rod are good but not as polished as Playwright in Python or JavaScript.

Verbose for one-offs: Python’s requests two-line scraper has no clean Go equivalent.

HTTP clients

net/http (standard library)

The standard library client. Production-grade, well-documented, fast. Right choice for most scraping HTTP needs.

package main

import (
    "io"
    "net/http"
    "time"
)

func fetch(url string) ([]byte, error) {
    client := &http.Client{Timeout: 10 * time.Second}
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return nil, err
    }
    req.Header.Set("User-Agent", "Mozilla/5.0 ...")
    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    return io.ReadAll(resp.Body)
}

Best for: most Go HTTP work. Default unless you have specific needs.

fasthttp

Aggressive performance-oriented HTTP library that bypasses some net/http abstractions for raw speed. 5-10x faster than net/http on benchmarks. The API is different (uses fasthttp.Request and fasthttp.Response instead of net/http types).

import "github.com/valyala/fasthttp"

func fetchFast(url string) ([]byte, error) {
    req := fasthttp.AcquireRequest()
    resp := fasthttp.AcquireResponse()
    defer fasthttp.ReleaseRequest(req)
    defer fasthttp.ReleaseResponse(resp)

    req.SetRequestURI(url)
    req.Header.SetUserAgent("Mozilla/5.0 ...")
    if err := fasthttp.Do(req, resp); err != nil {
        return nil, err
    }
    return resp.Body(), nil
}

Best for: extreme throughput needs (10k+ requests/sec), low-latency requirements.

resty

The popular convenience HTTP client wrapping net/http with a nicer API. Fluent builder pattern, JSON serialization, retry support. Slightly slower than raw net/http but more readable.

import "github.com/go-resty/resty/v2"

client := resty.New().SetTimeout(10 * time.Second)
resp, err := client.R().
    SetHeader("User-Agent", "Mozilla/5.0").
    Get("https://example.com")

Best for: developer ergonomics, projects that benefit from convenience over absolute performance.

HTML parsers

GoQuery

The jQuery-style HTML parser. Cleanest API for Go HTML manipulation. Built on golang.org/x/net/html under the hood.

import (
    "github.com/PuerkitoBio/goquery"
    "strings"
)

func parseTitles(html string) []string {
    doc, err := goquery.NewDocumentFromReader(strings.NewReader(html))
    if err != nil {
        return nil
    }
    var titles []string
    doc.Find("h2.product-title").Each(func(i int, s *goquery.Selection) {
        titles = append(titles, s.Text())
    })
    return titles
}

Best for: most Go HTML parsing. Default choice.

golang.org/x/net/html

The standard parser GoQuery wraps. Direct use is verbose but available for custom AST manipulation.

Best for: low-level parsing needs, when you want zero dependencies.

colly’s parser

Colly framework includes its own HTML traversal which is less verbose than GoQuery for callback-driven scraping. Used in conjunction with Colly only.

Browser automation

Chromedp

The dominant Go browser automation library. Uses Chrome DevTools Protocol directly without intermediate libraries. Fast, well-maintained, but the API is verbose compared to Playwright.

import (
    "context"
    "github.com/chromedp/chromedp"
    "time"
)

func scrapeWithChrome(url string) (string, error) {
    ctx, cancel := chromedp.NewContext(context.Background())
    defer cancel()
    ctx, cancel = context.WithTimeout(ctx, 30*time.Second)
    defer cancel()

    var title string
    err := chromedp.Run(ctx,
        chromedp.Navigate(url),
        chromedp.WaitVisible("h1.product-title"),
        chromedp.Text("h1.product-title", &title),
    )
    return title, err
}

Best for: most Go browser automation. The default choice when you need a real browser.

Rod

A modern alternative to Chromedp with a more fluent API. Active development, strong feature parity with Playwright.

import "github.com/go-rod/rod"

browser := rod.New().MustConnect()
page := browser.MustPage("https://example.com").MustWaitLoad()
title := page.MustElement("h1.product-title").MustText()

Best for: developers who prefer Rod’s API ergonomics over Chromedp’s.

Playwright-go

The Microsoft Playwright API for Go. Newer and less mature than Chromedp/Rod but offers cross-browser (Firefox, WebKit) support that the Chrome-only alternatives lack.

Best for: cross-browser needs in Go, teams using Playwright in other languages.

Frameworks

Colly

The dominant Go scraping framework. Built-in caching, concurrency, request rate limiting, and HTML parsing callbacks. The right choice for crawler-heavy Go scrapers.

import "github.com/gocolly/colly/v2"

c := colly.NewCollector(
    colly.AllowedDomains("example.com"),
    colly.Async(true),
)
c.Limit(&colly.LimitRule{
    DomainGlob:  "*",
    Parallelism: 10,
    Delay:       100 * time.Millisecond,
})

c.OnHTML("div.product", func(e *colly.HTMLElement) {
    fmt.Println(e.ChildText("h2.title"))
})

c.OnHTML("a.next", func(e *colly.HTMLElement) {
    e.Request.Visit(e.Attr("href"))
})

c.Visit("https://shop.example.com/page/1")
c.Wait()

Best for: large crawlers, the standard Go scraping framework.

Geziyor

Another Go scraping framework with similar feature set to Colly. Less popular but actively maintained.

Best for: Colly alternatives.

Comparison table

library	layer	speed	learning curve	best for
net/http	HTTP	fast	easy	most HTTP work
fasthttp	HTTP	fastest	medium	extreme throughput
resty	HTTP	fast	easy	developer ergonomics
GoQuery	parser	fast	easy	most HTML parsing
golang.org/x/net/html	parser	fast	hard	custom AST work
Chromedp	browser	mid	medium	most browser automation
Rod	browser	mid	medium	Chromedp alternative
Playwright-go	browser	mid	medium	cross-browser
Colly	framework	fast	medium	most crawler work
Geziyor	framework	fast	medium	Colly alternative

Decision matrix: solopreneur, SMB, enterprise

profile	scale	recommended stack	reasoning
Solopreneur Go-curious	<10k pages/day	net/http + GoQuery	Standard library + the one parser
Indie scraper, single binary	<500k pages/day	net/http + GoQuery + Colly	Framework value at this scale
Indie extreme throughput	<1M pages/day	fasthttp + GoQuery	When net/http becomes a bottleneck
SMB scraping infra	1-10M pages/day	Colly + Redis queue + custom workers	Distribute across N binaries
SMB JS-heavy	<500k pages/day	Chromedp + GoQuery post-parse	Browser only when needed
Embedded scraping in service	varies	net/http only	Avoid framework imports inside larger services
Enterprise data pipeline	10M+ pages/day	Custom Go workers + Kafka + GoQuery	Maximum control, minimum dependencies

The right pattern for Go at scale is custom workers reading from a queue rather than a monolithic Colly process. Goroutines do the concurrency; Redis or NATS does the work distribution. This pattern scales linearly with worker count and survives single-machine failures cleanly.

Migration path: Python or Node to Go

Most Go migrations happen when Python or Node scrapers hit infrastructure limits at scale. The playbook:

Identify the throughput bottleneck. If your Python scraper saturates one CPU core at 800 req/s, Go can run the same workload at 4000+ req/s on one core. If you are not CPU-bound, the migration may not pay off.
Port one scraper end-to-end. Choose the highest-throughput single-target scraper as the migration pilot. Validate output equivalence on a sample.
Keep Python or Node for orchestration. Many teams use Go for the scraper workers and Python for the data pipeline (Pandas, ML preprocessing). The tools do not have to match.
Containerize and deploy in parallel. Run Go workers alongside Python workers reading from the same queue. Cut over by reducing Python worker count over a few weeks.
Re-evaluate at six months. If the Go workers are stable and the throughput gain is real, migrate the rest. If they are not, the original choice was right.

The migration is rarely binary. Most production scrapers end up polyglot with Go for hot-path workers and Python for one-off and analytical work.

Performance benchmarks

Same workload as Python and Node benchmarks: 10,000 simple HTML pages from a local mirror, single Go binary.

stack	total time	requests/sec
net/http (50 goroutines)	6s	1666
fasthttp (50 goroutines)	3s	3333
resty (50 goroutines)	8s	1250
Colly (default)	7s	1428
Chromedp (50 contexts)	110s	90

Go HTTP throughput is the highest of the three languages we benchmarked. fasthttp specifically is faster than even Node’s undici for this workload. Browser automation is similar across all languages because the bottleneck is browser execution.

Cost worked example

For a 1M-pages-per-day Go scraping workload (roughly 12 req/s sustained):

1 medium VPS ($40/mo, 8 vCPU, 16 GB)
net/http + GoQuery + Colly stack (free)
uTLS for TLS fingerprint impersonation when needed (free)
Smartproxy/Decodo residential proxies (~$200/mo for ~25 GB)
Redis on a small managed instance ($10/mo) for distributed work coordination
PostgreSQL on a hosted instance ($25/mo)

Total: about $275/month for a workload that handles 30 million pages per month. The Python or Node equivalent would need 2-3x the compute capacity for the same throughput, raising infrastructure cost by $80-120/month. Go’s compiled binary also reduces deployment complexity (no language runtime, no virtualenv) and operational toil.

The break-even point where Go’s lower compute cost overcomes its higher development cost typically sits around 10M pages/month. Below that, Python or Node ergonomics usually win on total team productivity.

Stack recommendations

Most Go scraping: net/http + GoQuery + Colly. Standard library plus the two best community libraries. Adequate for almost everything.

Extreme throughput: fasthttp + GoQuery + Colly. When you need 10k+ HTTP requests per second per machine.

Browser-required scraping: Chromedp + GoQuery (for parsing extracted HTML). Use Chromedp for JS execution, GoQuery for the parsing because it is more ergonomic.

Distributed scraping: net/http + GoQuery + custom code with Redis queues. Colly does not have great distributed support; for multi-machine scrapers you build the coordination layer yourself.

Scraping inside a larger Go service: net/http directly. Avoid pulling in framework overhead for embedded scraping inside a service that does other things.

Idiomatic Go scraper template

A modern Go scraper using standard libraries:

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "strings"
    "sync"
    "time"

    "github.com/PuerkitoBio/goquery"
)

type Product struct {
    Name  string
    Price string
    URL   string
}

func fetchPage(ctx context.Context, url string) (string, error) {
    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
    if err != nil {
        return "", err
    }
    req.Header.Set("User-Agent", "Mozilla/5.0")

    client := &http.Client{Timeout: 15 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    return string(body), err
}

func parseProducts(html string) []Product {
    doc, err := goquery.NewDocumentFromReader(strings.NewReader(html))
    if err != nil {
        return nil
    }
    var products []Product
    doc.Find("div.product-card").Each(func(i int, s *goquery.Selection) {
        products = append(products, Product{
            Name:  s.Find("h2.title").Text(),
            Price: s.Find("span.price").Text(),
            URL:   s.Find("a").AttrOr("href", ""),
        })
    })
    return products
}

func scrapeAll(urls []string, concurrency int) []Product {
    sem := make(chan struct{}, concurrency)
    var wg sync.WaitGroup
    var mu sync.Mutex
    var allProducts []Product

    for _, url := range urls {
        wg.Add(1)
        go func(u string) {
            defer wg.Done()
            sem <- struct{}{}
            defer func() { <-sem }()

            html, err := fetchPage(context.Background(), u)
            if err != nil {
                fmt.Println("error:", err)
                return
            }
            products := parseProducts(html)
            mu.Lock()
            allProducts = append(allProducts, products...)
            mu.Unlock()
        }(url)
    }
    wg.Wait()
    return allProducts
}

func main() {
    urls := []string{"https://example.com/p/1", "https://example.com/p/2"}
    products := scrapeAll(urls, 20)
    for _, p := range products {
        fmt.Printf("%+v\n", p)
    }
}

This pattern handles 1500+ pages per minute on a small VPS with proper concurrency control.

Distributed scraper architecture

For workloads that exceed one machine, the canonical Go scraper architecture is:

Coordinator service that pushes URLs to a queue (Redis Streams, NATS JetStream, or Kafka).
Worker pool of N stateless Go binaries, each consuming from the queue, scraping in parallel goroutines, and writing results to a sink (Postgres, S3, Kafka).
Health and metrics exposed via Prometheus endpoints on each worker; scraped via a central Prometheus + Grafana stack.
Dead-letter queue for URLs that fail repeatedly, picked up by a slower retry process or surfaced for manual investigation.

This pattern scales linearly: doubling worker count doubles throughput up until the target rate-limits or the queue itself bottlenecks. With NATS or Kafka, the queue layer easily handles 100k messages/sec, far beyond what most scrapers need.

Common mistakes to avoid

Forgetting to close response bodies: every HTTP response body must be closed or you leak file descriptors. The defer resp.Body.Close() pattern is essential.

Unbounded goroutine spawning: launching one goroutine per URL without a semaphore exhausts memory and overwhelms target sites. Use a buffered channel as a semaphore.

Using fasthttp when you do not need it: fasthttp’s API is different from net/http and the integration cost is real. For most workloads, net/http is fast enough.

Ignoring context cancellation: pass context.Context through your scraper functions so you can cancel cleanly on shutdown signals.

Trying to use Python-style async patterns: Go’s concurrency primitives (goroutines, channels, sync.WaitGroup) are different from async/await. Embrace them rather than fighting them.

We cover the Python and Node alternatives in our best Python scraping libraries 2026 and best Node.js scraping libraries 2026 reviews.

External authoritative reference: the Go net/http documentation covers the standard library client.

Common gotchas

Goroutine leaks. Goroutines started without a clear exit path can leak forever if their channel never closes. Always have a context.Done() check or a select with a timeout case.
net/http connection reuse defaults. The default Transport reuses connections, which is good for performance but bad if you want each request from a fresh proxy. For per-request isolation, set Transport.DisableKeepAlives = true.
fasthttp’s API allocations. fasthttp.Request and Response are pooled; you must Acquire and Release them. Forgetting Release causes memory growth that looks like a leak.
GoQuery selector syntax differences. GoQuery uses CSS selectors but does not support all jQuery extensions. :contains() is supported, :has() is not. Test your selectors against the actual DOM before assuming.
Chromedp context cancellation. Cancelling the parent context kills all in-flight Chrome operations, but the headless Chrome process can survive. Always call chromedp.Cancel(ctx) explicitly to ensure cleanup.
JSON unmarshaling silent failures. Unknown fields are silently dropped by json.Unmarshal. If your target’s response shape changes, you may not notice. Use DisallowUnknownFields() on the decoder during development.
Slice append concurrency. Multiple goroutines appending to the same slice corrupt it. Use a sync.Mutex or a channel-based aggregator.
Colly OnHTML callback ordering. Multiple OnHTML handlers for overlapping selectors fire in registration order, not in DOM order. Test handler ordering if you depend on it.

When to use Go vs Python vs Node

consideration	best language
highest HTTP throughput per machine	Go
richest ecosystem	Python
best browser automation	Python or Node (Playwright)
simplest deployment	Go (single binary)
smallest learning curve	Python
best for embedded scraping in services	Go
largest community of scraping content	Python
AI/LLM integration	Python

For dedicated scraping projects that scale and benefit from compiled performance, Go is excellent. For one-offs and projects requiring rich ecosystem support, Python wins. For Node-shop infrastructure, Node Crawlee fits naturally.

FAQ

Q: Colly or write my own?
For projects with link-following, deduplication, and rate-limiting needs across thousands of pages, Colly saves significant code. For simple scrapers with a known URL list, raw net/http + GoQuery is enough.

Q: Chromedp or Rod or Playwright-go?
Chromedp is the safe default with the most production usage. Rod has a nicer API. Playwright-go is right when you need Firefox or WebKit. Performance is similar across all three.

Q: how do I handle TLS fingerprinting in Go?
Go’s TLS stack does not have first-class fingerprint impersonation. The closest options are utls (uTLS) which mimics specific browser TLS handshakes, and routing through a proxy that handles fingerprinting.

Q: is fasthttp worth the complexity?
For most workloads, no. fasthttp gives you an extra 2-5x throughput at the cost of API divergence from the standard library. Use it when you have measured a performance need that net/http cannot meet.

Q: does Go have an equivalent to Scrapy?
Colly is the closest. Less batteries-included than Scrapy but covers the core crawler patterns.

Q: how do I handle proxies in Go?
Set Transport.Proxy on your http.Client. For per-request proxy rotation, build a custom Transport that selects a proxy from a pool. Colly accepts a proxy switcher function natively.

Q: are there structured-data extraction libraries?
A few exist (go-rod/rod for browser, tdewolff/parse for streaming HTML). For strongly typed extraction, write a struct and unmarshal CSS selectors into it manually with reflection or code generation.

Q: is Go’s gc a problem for long-running scrapers?
Generally no. Go’s GC has been excellent since 1.14 with sub-millisecond pauses. The main GC concern is allocating in tight loops; reuse buffers and pool objects with sync.Pool if you see GC pressure.

Closing

Go scraping in 2026 is the right choice for high-throughput dedicated scraping infrastructure, distributed crawler workers, and embedded scraping inside Go services. The ecosystem is smaller than Python’s but the libraries that exist are excellent. net/http + GoQuery + Colly is the standard stack; fasthttp and Chromedp cover specialized needs. For broader scraping infrastructure see our dev-tools-projects category hub.

May 7, 2026

Best Node.js scraping libraries 2026

Best Node scraping libraries in 2026 occupy a different ecosystem than Python’s. Node’s event-loop architecture is naturally async-first, so concurrency comes for free. Browser automation has stronger native fit because Puppeteer was originally a Node-only library and the JavaScript-runtime-controlling-JavaScript story is uniquely tight. The Node scraping market has consolidated around a smaller list of high-quality libraries than Python’s, but each library is more polished and the gaps are smaller. The four-layer model still applies: HTTP client, browser automation, HTML parser, framework. Picking the right combination per layer determines whether your scraper handles 100 or 10,000 requests per second.

This guide ranks the Node.js scraping libraries actually worth using in 2026, with honest performance comparisons, clear use case mapping, and the gotchas that surprise developers coming from Python.

HTTP clients

undici

Node’s modern HTTP client, developed by the Node.js core team. Faster than every alternative by 2-3x for high-concurrency workloads. The standard fetch global in modern Node uses undici under the hood.

import { fetch } from 'undici';

async function scrape(url) {
  const resp = await fetch(url, {
    headers: { 'User-Agent': 'Mozilla/5.0 ...' },
  });
  return resp.text();
}

Best for: any new project, high-concurrency workloads, the default unless you have specific needs.

got

The popular HTTP client before undici took over. Excellent retry, redirect, and cookie handling. Slightly slower than undici but more feature-complete out of the box.

import got from 'got';

const html = await got('https://example.com', {
  retry: { limit: 3 },
  timeout: { request: 10000 },
}).text();

Best for: existing got codebases, projects that benefit from got’s batteries-included extras.

axios

The classic. Sync-style promise API that everyone knows. Slower than undici and got. Still ubiquitous because of legacy familiarity.

Best for: existing axios codebases, teams that already know its API.

node-fetch

The original Node fetch polyfill, now mostly obsolete since native fetch landed in Node 18+.

Best for: legacy projects, nothing else.

Browser automation

Playwright

Same library as Python; the JavaScript version is actually the reference implementation. Cleaner API in JavaScript than Python because of TypeScript autocompletion. Best browser automation framework in any language.

import { chromium } from 'playwright';

const browser = await chromium.launch();
const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
});
const page = await context.newPage();
await page.goto('https://target.example.com');
const title = await page.locator('h1.product-title').textContent();
await browser.close();

Best for: most modern browser automation in Node, multi-browser needs.

Puppeteer

The Google-maintained Chrome automation library. Slightly cleaner Chrome-specific features than Playwright, similar overall capability. The puppeteer-extra plugin ecosystem (especially stealth plugin) is more mature than Playwright’s equivalents.

Best for: Chrome-only workflows, projects using puppeteer-extra plugins.

Crawlee

Apify’s scraping framework with built-in browser support. Wraps Playwright/Puppeteer with crawler-style ergonomics. We cover it under frameworks below.

HTML parsers

Cheerio

The jQuery-syntax server-side parser. The dominant Node HTML parser. Fast (built on parse5 or htmlparser2), familiar API for anyone who used jQuery.

import * as cheerio from 'cheerio';

const $ = cheerio.load(html);
const titles = $('h2.product-title').map((i, el) => $(el).text()).get();

Best for: most HTML parsing in Node, jQuery-familiar developers.

parse5

The lower-level HTML parser that Cheerio uses under the hood. Direct use is rare but available for custom AST work.

Best for: custom HTML manipulation, building higher-level tools.

htmlparser2

Streaming HTML parser, very fast on large documents. Used by Cheerio when configured for it. Direct use for stream-based parsing.

Best for: parsing very large HTML documents in stream mode.

linkedom

Modern alternative offering full DOM API (not just jQuery-style). If your code expects document.querySelector semantics, linkedom feels native.

import { parseHTML } from 'linkedom';

const { document } = parseHTML(html);
const titles = Array.from(document.querySelectorAll('h2.product-title')).map(el => el.textContent);

Best for: developers who prefer DOM API over jQuery API, code shared between client and server.

Frameworks

Crawlee

Apify’s modern scraping framework. The Node version is the original; the Python port came later. Excellent abstractions for HTTP and browser scraping with the same Crawler interface, built-in queue management, dedupe, retry logic, and proxy rotation.

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  async requestHandler({ request, $ }) {
    console.log(`Scraping ${request.url}`);
    const titles = $('h2.product-title').map((i, el) => $(el).text()).get();
    await crawler.pushData({ url: request.url, titles });
  },
  maxRequestsPerCrawl: 1000,
  maxConcurrency: 10,
});

await crawler.run(['https://shop.example.com/page/1']);

Best for: most modern Node scraping projects that need framework benefits.

x-ray

Older declarative scraping framework. Still works but rarely chosen for new projects.

Best for: legacy x-ray codebases.

Apify SDK

Crawlee’s parent SDK with additional Actor and platform features. Right choice if deploying to Apify cloud.

Best for: Apify platform deployments.

Comparison table

library	layer	speed	learning curve	best for
undici	HTTP	fastest	easy	most new projects
got	HTTP	fast	easy	retry-heavy needs
axios	HTTP	mid	easy	legacy codebases
node-fetch	HTTP	mid	easy	nothing in 2026
Playwright	browser	mid	medium	most browser automation
Puppeteer	browser	mid	medium	Chrome-only, stealth plugins
Cheerio	parser	fast	easy	most parsing
parse5	parser	fast	hard	custom AST work
htmlparser2	parser	fastest	medium	very large docs, streaming
linkedom	parser	fast	easy	DOM-API preference
Crawlee	framework	fast	medium	modern crawler projects
x-ray	framework	mid	easy	legacy

Decision matrix: solopreneur, SMB, enterprise

profile	scale	recommended stack	reasoning
Solopreneur learning	<10k pages/day	native fetch + Cheerio	Zero dependencies, modern defaults
Indie scraper	<500k pages/day	undici + Cheerio + p-limit	Best HTTP perf, simple flow control
Indie JS-heavy	<100k pages/day	Playwright + Cheerio post-parse	Browser only when needed
SMB crawler	500k-10M pages/day	Crawlee CheerioCrawler	Framework manages queue, dedupe, retry
SMB anti-detect	100k-1M pages/day	Puppeteer + puppeteer-extra-plugin-stealth	Stealth ecosystem maturity
Enterprise	10M+ pages/day	Crawlee on K8s + custom middleware	Volume justifies platform investment
Hybrid HTTP/JS	varies	Crawlee (CheerioCrawler + PlaywrightCrawler)	Same dataset across two modes

The Node ecosystem rewards convergence: most teams end up on undici + Cheerio for HTTP and Playwright + stealth plugins for browser. Crawlee adds value above 500k pages/day; below that, hand-rolled async with p-limit is simpler and fast enough.

Migration path: axios + cheerio to undici + Cheerio

Most legacy Node scrapers run on axios because it was the dominant HTTP client of the 2018-2022 era. Modernizing to undici is straightforward and yields a 2-3x throughput improvement:

Replace axios.get(url, opts) with await fetch(url, opts) from undici. The API differs slightly (response body via .text() / .json() instead of .data).
Replace axios interceptors with explicit retry wrappers. undici does not have an interceptor system; use a small wrapper function for retry, logging, and metrics.
Update timeout handling to use AbortSignal.timeout(ms) instead of axios’s timeout option.
Benchmark the same workload before and after. Expect 2-3x improvement on concurrent request throughput.
Keep axios for any code that uses interceptors heavily (auth refresh patterns, request signing) where the cost of unwinding interceptor logic outweighs the perf gain.

A typical Node scraper migration completes in a day. The performance gain often unblocks scaling work that was on the roadmap for distributed infrastructure.

Performance benchmarks

Same workload as the Python benchmarks: 10,000 simple HTML pages from a local mirror, single Node process.

stack	total time	requests/sec
native fetch (50 concurrency)	8s	1250
undici (50 concurrency)	7s	1428
got (50 concurrency)	11s	909
axios (50 concurrency)	14s	714
Playwright (50 concurrent contexts)	88s	113
Crawlee CheerioCrawler	9s	1111

Node beats Python on raw HTTP throughput thanks to its event loop architecture. The browser automation gap is similar in both languages because the bottleneck is browser execution, not the host runtime.

Cost worked example

For a 100k-pages-per-day Node scraping workload on mixed protected and unprotected targets:

1 small VPS ($20/mo, 4 vCPU, 8 GB)
undici + Cheerio + p-limit stack (free, Node only)
node-libcurl when TLS impersonation is needed (free, requires native build)
Smartproxy/Decodo residential proxies (~$50/mo for 5 GB)
PostgreSQL on a hosted instance ($25/mo)
Optional: ZenRows fallback for hard surfaces (~$69/mo)

Total: about $95-165/month depending on the API fallback. Node throughput is higher than Python on raw HTTP, which lets you pack more work into the same VPS; expect to need ~30% less compute capacity than the equivalent Python deployment.

The other Node-specific cost dimension is RAM. Node processes typically run 200-300 MB at scraping idle and grow with concurrent contexts. For a single-process scraper, 8 GB RAM is plenty; for distributed multi-worker setups, prefer many small workers over few large ones to limit blast radius from leaks.

Stack recommendations

Small project, scripts: native fetch + Cheerio. Built into modern Node, no dependencies, fast.

Medium project, no JS needs: undici + Cheerio. Fastest HTTP client + standard parser. Add tenacity-style retry via simple wrapper.

JavaScript-heavy targets: Playwright + Cheerio (parse the extracted HTML with Cheerio for speed instead of using Playwright’s slower DOM querying).

Large crawler with link-following: Crawlee. Built-in queue management saves you from writing your own.

Anti-bot heavy targets: Puppeteer with puppeteer-extra-plugin-stealth. The stealth plugin ecosystem is more mature for Puppeteer than for Playwright in Node.

Hybrid HTTP + browser: Crawlee with multiple crawler classes (CheerioCrawler for HTTP-only pages, PlaywrightCrawler for JS-heavy pages, both writing to the same dataset).

Crawlee deep dive

Crawlee deserves a closer look because it has matured into the de-facto Node scraping framework. Its three crawler classes cover the spectrum:

CheerioCrawler: HTTP-only, uses got (or undici under the hood) and Cheerio. Fast, low-resource. The right default for HTTP scraping.
PlaywrightCrawler: full browser automation with Playwright. Highest resource cost but handles any JavaScript.
PuppeteerCrawler: same as Playwright but using Puppeteer. Choose this if your team prefers Puppeteer’s API or uses puppeteer-extra plugins.

All three share the same RequestQueue, Dataset, and KeyValueStore abstractions, which means you can switch between HTTP and browser modes per request without changing your data layer. A typical pattern is to start with CheerioCrawler, fall back to PlaywrightCrawler when the HTML is missing the data you need, and store both kinds of results in the same Dataset.

Crawlee’s RequestQueue supports SQLite, MongoDB, and the Apify cloud as backends. SQLite works for single-process crawlers; MongoDB works for distributed crawlers across machines. The cloud backend gives you a managed queue with no operational overhead.

Modern async patterns

Node’s async syntax is cleaner than Python’s for typical scraping patterns:

import { fetch } from 'undici';
import * as cheerio from 'cheerio';
import pLimit from 'p-limit';

async function fetchPage(url, retries = 3) {
  for (let attempt = 0; attempt < retries; attempt++) {
    try {
      const resp = await fetch(url, {
        signal: AbortSignal.timeout(15000),
      });
      if (resp.status === 200) {
        return await resp.text();
      }
      if (resp.status === 429 || resp.status === 503) {
        await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 1000));
        continue;
      }
      return null;
    } catch (err) {
      if (attempt === retries - 1) throw err;
      await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 1000));
    }
  }
}

function parseProducts(html) {
  if (!html) return [];
  const $ = cheerio.load(html);
  return $('div.product-card').map((i, el) => ({
    name: $(el).find('h2.title').text(),
    price: $(el).find('span.price').text(),
  })).get();
}

async function scrapeAll(urls, concurrency = 20) {
  const limit = pLimit(concurrency);
  const results = await Promise.all(
    urls.map(url => limit(async () => {
      const html = await fetchPage(url);
      return parseProducts(html);
    }))
  );
  return results.flat();
}

This pattern handles 1000+ pages per minute on a modest VPS with retries and concurrency control built in.

Persistence and storage in Node

Node scrapers benefit from a few storage patterns specific to JavaScript ecosystems:

better-sqlite3 for synchronous local storage. Faster than the async sqlite3 library for write-heavy workloads because it avoids async overhead.
Knex or Prisma for typed Postgres access. Both work well; Prisma’s TypeScript types are stronger but Knex is lighter.
Crawlee KeyValueStore + Dataset. When using Crawlee, prefer its built-in storage abstractions; they handle large blobs and structured records cleanly.
DuckDB-WASM for in-process analytics. When you want to query scraped data without a database server, DuckDB now ships a Node binding that lets you run SQL on Parquet or local arrays.

For very large output volumes, stream writes to S3 / R2 with @aws-sdk/client-s3 MultipartUpload rather than collecting everything in memory and uploading at the end.

Common mistakes to avoid

Using axios in 2026: it works but is slower than undici and got. New projects should default to undici.

Forgetting AbortSignal.timeout: Node’s native fetch does not have a default timeout. Without one, your scraper hangs on slow targets indefinitely.

Loading huge HTML strings into Cheerio at once: for documents over 10 MB, use htmlparser2 in streaming mode.

Running too many browser contexts in one Node process: Node memory grows fast with many Playwright contexts. Stay under 50 concurrent contexts per process.

Ignoring back-pressure in Crawlee: Crawlee’s queues can grow unboundedly if you push faster than you consume. Set maxRequestsPerCrawl and maxConcurrency appropriately.

We cover the Python equivalent in our best Python scraping libraries 2026 review.

Common gotchas

Native fetch lacks default timeout. Without AbortSignal.timeout(), your scraper hangs on slow targets indefinitely. Always set a timeout.
undici keep-alive defaults. undici defaults to keep-alive connections. For one-off scripts, this can leave the process hanging waiting for sockets. Use Agent({ keepAliveTimeout: 1 }) or call agent.close() at script end.
Cheerio re-parse cost. Each call to cheerio.load() re-parses the HTML. For many extractions on the same document, parse once and pass $ around.
Playwright newPage vs newContext. newPage() reuses the parent context’s cookies; newContext() creates a fresh storage state. Use newContext() per scrape to isolate cookies; many subtle bugs come from cookie cross-contamination.
Crawlee request handler errors. A throw inside requestHandler retries the request by default. If the error is permanent (404, parse failure), call request.noRetry = true to skip the retry queue.
JSON parsing with native fetch. await resp.json() throws on empty body; wrap in try/catch or check resp.ok first.
EventEmitter memory leaks. Browser launches that emit console, request, or response events accumulate listeners if you do not clean them up. Use page.removeAllListeners() before close or use named handler functions you can remove explicitly.
TLS hardening on undici. Some targets refuse TLS 1.2 connections. undici defaults to negotiating up; if you see handshake errors, force connect: { tls: { minVersion: 'TLSv1.3' } }.

TypeScript vs JavaScript

For new projects, TypeScript is the right choice. The HTTP and parsing libraries all ship with strong type definitions. The Playwright API in TypeScript is a different developer experience than JavaScript.

import { chromium, Browser, Page } from 'playwright';

async function scrape(browser: Browser, url: string): Promise<string | null> {
  const page: Page = await browser.newPage();
  try {
    await page.goto(url);
    return await page.locator('h1').textContent();
  } finally {
    await page.close();
  }
}

The autocompletion and refactoring support pay off within the first week of any non-trivial project.

External authoritative reference: the Node.js documentation on the global fetch covers the standard HTTP client.

Bun and Deno alternatives

Bun and Deno both ship with built-in fetch and run all the libraries above. Bun is notably faster than Node for HTTP-heavy workloads (about 1.5-2x in our testing). Deno’s permission model is interesting for scraping isolation but the ecosystem is smaller.

For most teams, Node remains the right default in 2026 because library compatibility is broadest. We cover the alternatives in our forthcoming guides on Bun and Deno scraping.

FAQ

Q: Cheerio or jQuery selectors?
Cheerio implements jQuery-style selectors server-side. The API is essentially identical. Use Cheerio in Node; do not import actual jQuery server-side.

Q: should I use Crawlee or write my own crawler?
For projects under 1000 pages, write your own with undici + Cheerio + p-limit. For larger crawls with link-following, dedupe, and retry needs, Crawlee saves significant code.

Q: Puppeteer or Playwright?
Playwright is technically better for new projects: cleaner API, multi-browser, better auto-waiting. Puppeteer has the puppeteer-extra-plugin-stealth ecosystem advantage which still matters for some anti-detect work.

Q: how do I handle TLS fingerprinting in Node?
Node does not have a great equivalent to Python’s curl_cffi yet. The closest options are node-libcurl (libcurl bindings for Node) or routing through a proxy that handles TLS fingerprinting on your behalf.

Q: is JSDOM useful for scraping?
JSDOM is heavier than Cheerio because it implements the full DOM API including layout. For scraping where you do not need actual JavaScript execution, Cheerio is faster. JSDOM is the right choice when you want to execute scripts on a parsed document without a full browser.

Q: how do I integrate proxies?
With undici, use a ProxyAgent. With Playwright, pass proxy to chromium.launch(). Crawlee has built-in proxy rotation across a pool. Avoid manual proxy management; use the built-in tools wherever possible.

Q: is Bun production-ready for scraping?
Yes for most use cases. Bun’s built-in fetch and HTML parser are excellent. The remaining gaps are around obscure npm packages with native dependencies that have not been compiled for Bun. Test your dependency tree before committing to Bun in production.

Q: what is the cleanest way to handle pagination?
Wrap your fetch in an async generator that yields pages until a stop condition. Async generators in Node compose nicely with for await loops and avoid materializing all pages in memory.

Closing

The Node.js scraping stack in 2026 is mature and stable. undici for HTTP, Cheerio for parsing, Playwright for browser automation, Crawlee for crawler frameworks. The ecosystem moves slower than Python’s but each piece is more polished. Match the stack to the workload and Node will outperform Python on raw HTTP throughput while matching it on browser automation. For broader scraping infrastructure see our dev-tools-projects category hub.

May 7, 2026

Best Python scraping libraries 2026: Scrapy, BS4, more

Best Python scraping libraries in 2026 cover a stack that has matured significantly since the requests + BeautifulSoup era. The HTTP client layer has been split into a dozen options optimized for different use cases. Browser automation has consolidated around Playwright. Parsing has stabilized on lxml under the hood with multiple frontend options. The framework layer sees Scrapy holding its dominance for crawling-heavy use cases and Crawlee gaining ground as a modern alternative. Choosing the right combination of libraries determines whether your scraper runs at 10 requests per second or 1000, whether it survives anti-bot fingerprinting, and how much code you write to do straightforward things.

This guide ranks the Python scraping libraries actually worth using in 2026, organized by what they do, with honest performance comparisons and clear guidance on which to pick for which workload.

The four layers of a Python scraper

Every scraper has four layers, regardless of framework:

HTTP client: makes the actual network requests
Browser automation (optional): when JavaScript execution is needed
HTML parser: extracts data from the response
Orchestration framework (optional): handles concurrency, retries, queues, pipelines

Different libraries dominate each layer. The right scraper picks the best library per layer rather than committing to one library for everything.

HTTP clients

requests

The classic. Synchronous, simple, mature. Still the right choice for one-off scripts and learning. Performance is the worst of the modern options because it is sync-only.

import requests

resp = requests.get("https://example.com", headers={"User-Agent": "..."}, timeout=10)
print(resp.text)

Best for: scripts, prototypes, learning, anything where async is overkill.

httpx

The modern requests replacement. Supports both sync and async, HTTP/2 by default, type-hinted, and dramatically faster than requests for any concurrent workload. The drop-in replacement for requests in most code.

import httpx
import asyncio

async def fetch(url: str):
    async with httpx.AsyncClient(http2=True, timeout=10) as client:
        resp = await client.get(url)
        return resp.text

# concurrent fetching
async def main():
    urls = ["https://example.com/page/1", "https://example.com/page/2"]
    return await asyncio.gather(*[fetch(u) for u in urls])

Best for: any new project, async workloads, HTTP/2 support, type safety.

aiohttp

The async-first HTTP client. Older than httpx but still excellent. Slightly faster than httpx for high concurrency. The websocket support is best in class.

Best for: high-concurrency async workloads, websocket-heavy use cases.

curl_cffi

The TLS-fingerprint-aware HTTP client. Wraps libcurl with browser-impersonation features so your TLS handshake looks like real Chrome, Firefox, or Safari. The right choice for any target with TLS fingerprinting (Cloudflare, DataDome, Akamai).

from curl_cffi import requests

# impersonates Chrome 120 TLS fingerprint
resp = requests.get(
    "https://target.example.com",
    impersonate="chrome120",
    timeout=10,
)
print(resp.text)

Best for: bypassing TLS fingerprinting, scraping sites that detect Python’s default TLS.

urllib3

The HTTP foundation that requests and httpx both use under the hood. Rarely used directly except for very low-level needs.

Best for: when you need fine-grained control of connection pools.

HTML parsers

lxml

The fast XML/HTML parser written in C. Underpins almost every other parser. Direct lxml use is fastest but the API is uglier than BeautifulSoup.

from lxml import html

tree = html.fromstring(html_content)
titles = tree.xpath("//h2[@class='product-title']/text()")

Best for: high-performance parsing, XPath-heavy extraction.

BeautifulSoup4

The friendly parser. Slower than lxml directly but has the most readable API. The standard configuration uses lxml as the underlying parser, so the speed gap is smaller than people assume.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content, "lxml")
titles = [t.text for t in soup.select("h2.product-title")]

Best for: most general-purpose parsing, readable code, mixed CSS/find patterns.

selectolax

The fastest Python HTML parser by a wide margin. C-based, supports CSS selectors with a minimal API. Roughly 5-10x faster than BeautifulSoup on typical pages.

from selectolax.parser import HTMLParser

tree = HTMLParser(html_content)
titles = [n.text() for n in tree.css("h2.product-title")]

Best for: high-volume parsing where every millisecond matters.

parsel

Scrapy’s parser, available standalone. Combines XPath, CSS, and regex selectors with a clean API.

Best for: Scrapy users wanting the same API outside Scrapy.

Browser automation

Playwright

The current best browser automation framework. We covered it in detail in best headless browser frameworks 2026.

Best for: modern browser automation, multi-browser support.

Selenium

The elder framework. Still solid for cross-browser needs. Verbose but well-documented.

Best for: legacy projects, multi-language teams sharing test infrastructure.

Pyppeteer (less recommended)

Python port of Puppeteer. Less actively maintained than Playwright. Avoid for new projects.

Frameworks

Scrapy

The dominant Python scraping framework. Async by design (since 2.0), built-in queue management, middleware system, item pipelines, and crawl rules. The right choice for crawler-heavy workloads where you are following links across thousands of pages.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://shop.example.com/category/widgets"]

    def parse(self, response):
        for product in response.css("div.product"):
            yield {
                "name": product.css("h2::text").get(),
                "price": product.css("span.price::text").get(),
                "url": product.css("a::attr(href)").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

The downside: Scrapy’s mental model is heavier than other frameworks. You learn callbacks, middleware, settings, items, and pipelines. For simple scrapers this is overkill.

Best for: large crawl projects, structured data extraction at scale, when you actually need a framework.

Crawlee

The newer framework from Apify, with a Pythonic API and built-in browser support. More approachable than Scrapy for newcomers. Supports HTTP and browser modes from the same crawler class.

Best for: modern projects that want framework benefits without Scrapy’s learning curve.

Pyspider

Older framework with a web UI for managing scrapers. Less actively maintained but still works for some use cases.

Best for: legacy systems, niche use cases needing visual management.

Comparison table

library	layer	sync/async	speed	learning curve	best for
requests	HTTP	sync	slow	easy	scripts, prototypes
httpx	HTTP	both	fast	easy	most new projects
aiohttp	HTTP	async	fast	medium	high-concurrency async
curl_cffi	HTTP	sync	fast	easy	TLS fingerprint bypass
BeautifulSoup4	parser	sync	mid	easy	general parsing
lxml	parser	sync	fast	medium	high-performance XPath
selectolax	parser	sync	fastest	easy	extreme volume parsing
parsel	parser	sync	fast	easy	Scrapy users
Playwright	browser	both	mid	medium	JS-heavy targets
Selenium	browser	sync (mostly)	slow	medium	legacy
Scrapy	framework	async	fast	hard	large crawls
Crawlee	framework	async	fast	medium	modern, browser-friendly

Decision matrix: solopreneur, SMB, enterprise

profile	scale	recommended stack	reasoning
Solopreneur learning	<1k pages/day	requests + BeautifulSoup4	Simple, beginner-friendly
Indie scraper (basic)	<100k pages/day	httpx + selectolax	Async, fast, modern
Indie scraper (anti-bot)	<100k pages/day	curl_cffi + selectolax	TLS impersonation included
SMB crawler	100k-10M pages/day	Scrapy + curl_cffi middleware + selectolax	Framework value at this scale
SMB JS-heavy	10k-1M pages/day	Crawlee or Playwright + httpx fallback	Hybrid HTTP/browser ergonomics
Enterprise pipeline	10M+ pages/day	Scrapy on K8s + custom middleware + dedicated parsers	Full ops + custom optimization
Single-source ETL	varies	httpx + lxml direct XPath	Tight, predictable, performant

The most expensive mistake is over-frameworking small jobs (Scrapy for 200 pages) and under-frameworking large jobs (raw httpx loop for 10M URLs). Match the framework weight to the actual workload.

Migration path: requests + BS4 to httpx + selectolax

Most legacy Python scrapers can be modernized in a day with significant performance gains. The playbook:

Wrap your fetch function in an async signature even before changing implementation. This isolates the migration scope.
Replace requests.get with httpx.AsyncClient.get. Most code translates 1:1; the main change is await keywords and the async context manager.
Switch parser to selectolax. CSS selectors translate directly from BeautifulSoup; XPath users stay on lxml. Expect 5-10x parse speedup.
Add concurrency with asyncio.Semaphore to bound parallel requests. Start at 10 and tune based on target tolerance.
Benchmark against original with the same input set. A typical migration shows 15-30x throughput improvement.

The whole migration usually takes one engineer-day for a single-purpose scraper, two days for a multi-target codebase. The throughput gain often eliminates the need for distributed scaling that was on the roadmap.

Performance benchmarks

We benchmarked HTTP fetching of 10,000 simple HTML pages from a local mirror, single machine, no network bottleneck.

stack	total time	requests/sec
requests (sync)	240s	42
httpx (async, 50 concurrency)	12s	833
aiohttp (50 concurrency)	11s	909
curl_cffi (50 concurrency)	14s	714
Scrapy (default settings)	18s	555
Playwright (50 concurrent contexts)	95s	105

Async HTTP is 20x faster than sync. Browser automation is 8-10x slower than HTTP. The difference is essentially the cost of running JavaScript and rendering, which is unavoidable for SPAs.

Choosing between async frameworks

Python’s async ecosystem fragmented for years between asyncio (standard library), trio (alternative event loop with cleaner cancellation semantics), and AnyIO (a compatibility layer). For scraping, asyncio is the right default because every major HTTP and parser library targets it. trio remains technically superior for cancellation safety but the ecosystem cost is real.

The other choice is between asyncio’s default event loop and uvloop (a Cython-accelerated drop-in replacement). For HTTP-bound scrapers, uvloop yields a 2-4x throughput improvement essentially for free:

import asyncio
import uvloop

uvloop.install()  # do this before any asyncio code

# rest of your scraper

The two-line installation gets you the benefit. The only caveat is that uvloop does not work on Windows; cross-platform code needs a try/except around the install call.

Stack recommendations by use case

Small project, learning, scripts: requests + BeautifulSoup4. Simple, well-documented, slow but adequate.

Medium project, production, no JavaScript needs: httpx (async) + selectolax. Fast, modern, scales to a few hundred requests per second on one machine.

Medium project with anti-bot needs: curl_cffi + selectolax. The TLS fingerprint matters more than raw speed for protected targets.

Large crawler with link-following: Scrapy + parsel. Built-in queue management, dedupe, retry middleware. The framework cost is justified at this scale.

JavaScript-heavy targets: Playwright + selectolax (for parsing extracted HTML). Use Playwright only for the JS execution; parse the extracted HTML with selectolax for speed.

Hybrid (some JS, some HTTP): Crawlee or drissionPage. Both support seamless switching between HTTP and browser modes.

Cost worked example

A practical 100k-pages-per-day workload on protected targets needs:

1 small VPS ($20/mo, 4 vCPU, 8 GB)
httpx + uvloop + selectolax stack (free, Python only)
curl_cffi for TLS impersonation when needed (free)
Residential proxy pool from Smartproxy/Decodo (~$50/mo for 5 GB)
PostgreSQL on a hosted instance ($25/mo)
Optional: ScraperAPI fallback for surfaces that fail consistently (~$49/mo)

Total: about $95-145/month depending on whether you include the API fallback. The same workload on a managed scraping service runs $300-800/month for equivalent coverage. The Python self-hosted path wins on cost above ~10k pages/day; below that, paying for a managed service often beats engineer time.

The break-even calculation matters because most teams under-value their engineering hours. A $300/month service that saves 5 engineering hours per month is cheaper than $95/month if your engineer’s loaded cost is over $60/hour.

Idiomatic patterns

A modern async scraper template:

import asyncio
import httpx
from selectolax.parser import HTMLParser
from typing import AsyncGenerator

async def fetch_page(client: httpx.AsyncClient, url: str) -> str:
    for attempt in range(3):
        try:
            resp = await client.get(url, timeout=15.0)
            if resp.status_code == 200:
                return resp.text
            if resp.status_code in (429, 503):
                await asyncio.sleep(2 ** attempt)
                continue
            return None
        except (httpx.TimeoutException, httpx.NetworkError):
            await asyncio.sleep(2 ** attempt)
    return None


def parse_products(html: str) -> list[dict]:
    if not html:
        return []
    tree = HTMLParser(html)
    return [
        {
            "name": n.css_first("h2.title").text() if n.css_first("h2.title") else None,
            "price": n.css_first("span.price").text() if n.css_first("span.price") else None,
        }
        for n in tree.css("div.product-card")
    ]


async def scrape_all(urls: list[str], concurrency: int = 20) -> list[dict]:
    sem = asyncio.Semaphore(concurrency)
    results = []
    async with httpx.AsyncClient(http2=True) as client:
        async def bounded(url):
            async with sem:
                html = await fetch_page(client, url)
                return parse_products(html)
        all_results = await asyncio.gather(*[bounded(u) for u in urls])
        for r in all_results:
            results.extend(r)
    return results

if __name__ == "__main__":
    urls = ["https://example.com/p/1", "https://example.com/p/2"]
    products = asyncio.run(scrape_all(urls))

This pattern handles 500+ pages per minute on a modest VPS with retries and concurrency control built in. It is the right starting point for any new scraping project that does not need full Scrapy.

Common gotchas

httpx connection pool exhaustion. The default limits parameter caps concurrent connections at 10. Without raising it, your asyncio.Semaphore(50) is silently throttled to 10. Always pass httpx.Limits(max_connections=200) for high-concurrency workloads.
selectolax encoding errors. selectolax expects bytes or properly-decoded strings. Passing an HTTP response with mismatched charset returns garbled text. Use resp.text from httpx (which auto-detects encoding) or decode explicitly.
Scrapy autothrottle ambiguity. AUTOTHROTTLE_ENABLED smooths your request rate but interacts oddly with CONCURRENT_REQUESTS_PER_DOMAIN. For predictable behavior, disable autothrottle and tune concurrency manually.
lxml memory growth. lxml.etree.parse on large documents can leak references in Python’s GC. For long-running jobs, periodically del the tree and call gc.collect() between batches.
httpx HTTP/2 incompatibility. Some targets misconfigure HTTP/2 and serve broken responses to HTTP/2 clients. If a target works in curl but fails in httpx, try http2=False.
curl_cffi version mismatch. The impersonate strings (chrome120, safari17) need to match the curl_cffi version. Old strings silently fall back to default Chrome. Pin the version and check the docs for current strings.
BeautifulSoup find vs select. soup.find() returns the first match or None; soup.select() returns a list. Conflating them causes silent attribute errors on None.

Persisting scraped data

Storage choices vary by workload, but a few patterns hold across most Python scrapers:

SQLite for development and small projects. No server, single file, fast enough for millions of rows. Use aiosqlite if your scraper is async.
PostgreSQL for production. Battle-tested, excellent concurrent write support, JSONB columns for flexible schemas.
Parquet on S3 / R2 for archive. Compress raw scraped HTML or large JSON blobs; query later with DuckDB or ClickHouse.
DuckDB for analytical queries. Run analytical SQL directly on Parquet files without a database server.

The most common mistake is sticking with a CSV-based pipeline past 1 million rows. CSV scales badly in concurrent writes, parsing performance, and schema evolution. Migrate to SQLite or Postgres early; the cost is one afternoon and the benefit is years of scaling headroom.

Common mistakes to avoid

Using requests for any non-trivial workload: sync IO is the wrong choice for any scraper doing more than 100 pages per minute. The cost of switching to httpx is small.

Using BeautifulSoup with the html.parser backend: 3-5x slower than the lxml backend. Always specify BeautifulSoup(html, "lxml").

Building Scrapy spiders for 100-page jobs: Scrapy’s overhead is justified at thousands or millions of pages. For small jobs, async httpx is simpler.

Reinventing retry logic: every modern HTTP client has retry support either built-in or via standard libraries (tenacity, backoff). Use them.

Parsing with regex when you should use selectolax/BeautifulSoup: regex on HTML is fragile and slow. Use a proper parser.

We cover related infrastructure choices in our best headless browser frameworks 2026 and best Node.js scraping libraries 2026 reviews.

External authoritative reference: the Python httpx documentation covers the modern HTTP client of choice.

FAQ

Q: should I learn Scrapy in 2026?
Yes if you anticipate building large crawlers. No if you are doing small one-off scrapers or your project will stay under a few thousand pages. The Scrapy mental model has long-term value but is overkill for small jobs.

Q: what about pandas read_html?
Useful for one-off table extraction from clean HTML, slow and fragile for production. Treat it as a notebook tool, not a production scraper.

Q: how do I handle JavaScript-rendered content without Playwright?
Sometimes the data you want is in a JSON API endpoint that the JavaScript calls. Network-tab inspection in DevTools reveals these. Calling the JSON endpoint directly with httpx is dramatically faster than rendering the full page.

Q: which library handles cookies best?
httpx and aiohttp both have proper cookie jar support. requests does too. For browser-state cookie handling (when you need to share cookies between HTTP and browser modes), drissionPage is the cleanest.

Q: do I need Scrapy if I use httpx?
Not for small to medium scrapes. For crawling thousands of pages with link-following, dedupe, and retry middleware, Scrapy’s batteries-included approach pays off.

Q: how do I integrate proxies cleanly?
httpx accepts proxies={"all://": "http://user:pass@host:port"}. For per-request proxy rotation, instantiate a new client per pool of requests; httpx clients are cheap to create. For Scrapy, use a downloader middleware that picks a proxy per request.

Q: which library is best for large file downloads?
httpx with client.stream() lets you download multi-GB files without loading them into RAM. Combine with aiofiles for async disk writes. Avoid requests.get(url).content for anything over 50 MB; it loads the whole response into memory.

Q: is async always better than sync?
For network-bound work, yes. For CPU-bound parsing, no; async does not parallelize CPU work. Mix the two: async fetch, sync parse, then asyncio.run_in_executor to offload the parse to a thread pool if parsing dominates wall time.

Closing

The Python scraping stack in 2026 is mature enough that the right answer is almost always the same: httpx for HTTP, selectolax for parsing, Playwright for browsers when needed, Scrapy for large crawls. Add curl_cffi when TLS fingerprinting matters. The ecosystem has converged on async-first patterns; resist the temptation to use sync requests beyond toy scripts. For broader scraping infrastructure see our dev-tools-projects category hub.

May 7, 2026

Best web scraping APIs 2026: 12 services compared

Best scraping APIs in 2026 are the right answer for an increasingly large share of scraping workloads. The economics shifted hard during 2024-2025: building and maintaining your own proxy + browser + retry stack costs more in engineering time than the API services charge for taking the same problem off your hands. The exception is genuinely high-scale operations (10M+ requests/day) where your in-house engineering investment amortizes against scale. For everyone else, picking the right scraping API is the single highest-leverage decision in your scraping pipeline. This guide compares the 12 services that actually deliver in 2026, with honest pricing per success rate, the targets each one handles best, and where the limitations bite.

What a scraping API actually does

A scraping API takes a URL plus optional parameters and returns the rendered HTML or extracted data. Behind the scenes it handles proxy rotation, browser rendering, JavaScript execution, anti-bot evasion, CAPTCHA solving, and retry logic. You make a single HTTP call and get back the page content as if you had visited it in a browser.

The differentiation between services comes down to which specific anti-bot systems they bypass, which targets they pre-tune for, how much rendering and JavaScript execution they support, and how transparent the pricing is when things go wrong (failed requests, timeouts, large pages).

What we measured

For each service we ran 1000 requests against six target categories: e-commerce (Amazon US), SERP (Google search), social (Twitter), travel (Booking.com), real estate (Zillow), and business listings (Yellow Pages). Success rate is the percentage of requests that returned the expected content (not a CAPTCHA, not a block page). Average response time is the median time from API call to response. Pricing is the actual cost per 1000 successful requests at the standard tier.

1. ScraperAPI

ScraperAPI is the long-running incumbent. Pricing starts at $49/month for 100k credits. Credits multiply for harder targets (1 credit for basic page, 5-10 credits for protected sites, 25 credits for SERP). Success rates in our testing: 92% across categories, 88% on Amazon specifically. Average response 4.5 seconds.

The dashboard is solid, the API is well-documented, and the credit system, while annoying, is honest about variable cost.

Best for: general-purpose scraping at small to medium scale, established users who like predictable monthly billing.

2. ZenRows

ZenRows positions as the modern alternative, focused on anti-bot bypass for protected targets. Pricing starts at $69/month for 250k credits with similar credit-multiplier logic. Success rates: 94% across categories, 91% on Amazon. Average response 3.8 seconds.

ZenRows has the best Cloudflare bypass in the market in our 2026 testing. They invest heavily in keeping ahead of fingerprinting changes. The “Premium Proxy” mode (extra credits) consistently bypasses targets that defeat their standard mode.

Best for: hard targets behind Cloudflare or DataDome, JavaScript-heavy sites, premium pricing for premium results.

3. ScrapingBee

ScrapingBee is the indie-friendly option with clear pricing and a focus on rendering quality. Pricing starts at $49/month for 150k credits. Success rates: 90% across categories, 86% on Amazon. Average response 5 seconds.

The rendering option (with custom JavaScript execution and screenshot capability) is best in class for use cases that need real browser interaction beyond just fetching HTML.

Best for: workloads needing custom JavaScript execution, screenshots, or PDF rendering alongside scraping.

4. Bright Data Web Scraper API

Bright Data offers their Web Scraper API as a productized version of their proxy + browser infrastructure. Pricing is consumption-based starting at $1.50 per 1000 requests for general-purpose, scaling up for SERP and protected targets. Success rates: 96% across categories, 93% on Amazon. Average response 3 seconds.

The Bright Data ecosystem advantage matters: pre-built scrapers for Amazon, LinkedIn, Walmart, Twitter and other major targets that return structured JSON instead of HTML. You skip the parsing step entirely.

Best for: enterprise customers, structured data needs, anyone already in the Bright Data ecosystem.

5. Oxylabs Web Scraper API

Oxylabs offers Real-Time Crawler and dedicated SERP/E-Commerce APIs. Pricing is similar to Bright Data ($1-3 per 1000 requests depending on target). Success rates: 95% across categories.

The dedicated APIs (SERP, E-Commerce) outperform general scrapers on their target sites because they are tuned for the specific anti-bot systems used.

Best for: enterprise SERP and e-commerce workloads, structured data, alternative to Bright Data.

6. Apify

Apify is more than a scraping API; it is a full scraper marketplace and runtime platform. You pay for compute (Actor runs) and bandwidth. Their library of pre-built Actors covers thousands of targets. Pricing is consumption-based; a typical scrape runs $0.50-3 per 1000 results depending on the Actor.

Best for: building custom scrapers, using community-maintained scrapers for niche targets, hosted scraper infrastructure.

7. SerpApi

SerpApi is the dedicated SERP scraping leader. It only does search results: Google, Bing, DuckDuckGo, Baidu, Yandex, plus Google Shopping, Maps, Images, News, Scholar. Pricing starts at $50/month for 5000 searches.

Success rates on SERP specifically: 98% across all engines. Latency around 2 seconds.

Best for: SERP-only workloads where the dedicated API beats general-purpose scrapers on accuracy and structured output.

8. DataForSEO

DataForSEO offers SERP, On-Page, Backlinks, Keywords Data, and Domain Analytics APIs. Pricing is per-task, very granular. Cheaper than SerpApi for high-volume SERP ($0.6-1 per 1000 results).

Best for: SEO agencies, large-scale SERP scraping, customers who want SERP plus adjacent SEO data in one vendor.

9. ScrapingAnt

ScrapingAnt is a budget alternative to ZenRows and ScraperAPI. Pricing starts at $19/month for 10k credits. Success rates: 87% across categories, 82% on Amazon. Average response 5.5 seconds.

Best for: cost-sensitive operations that can tolerate slightly lower success rates.

10. ScrapeNinja

ScrapeNinja is a smaller indie API with TLS fingerprinting bypass and JavaScript rendering. Pricing $19-49/month range. Success rates: 85% across categories.

Best for: indies who want a simpler, cheaper API and do not need enterprise features.

11. Crawlbase (formerly ProxyCrawl)

Crawlbase offers their Crawling API and Crawler product. Pricing similar to ScraperAPI. Strong on common e-commerce targets. Success rates: 89% across categories.

Best for: established users who like the predictable pricing model.

12. WebScrapingAPI

A relatively newer entrant focused on SERP and e-commerce APIs. Pricing $49-149/month range. Success rates: 88% across categories.

Best for: alternative to ScraperAPI/ZenRows for similar use cases.

Comparison table

service	starting price	credits/req model	success rate (avg)	best target type	response time
ScraperAPI	$49/mo	yes	92%	general	4.5s
ZenRows	$69/mo	yes	94%	protected (CF, DD)	3.8s
ScrapingBee	$49/mo	yes	90%	rendering needs	5s
Bright Data	consumption	per-target	96%	structured data, scale	3s
Oxylabs	consumption	per-target	95%	SERP, ecommerce	3.2s
Apify	per-Actor	varies	90% (varies)	custom + marketplace	varies
SerpApi	$50/mo	flat	98% (SERP)	SERP-only	2s
DataForSEO	per-task	yes	95% (SERP)	SERP + SEO data	4s
ScrapingAnt	$19/mo	yes	87%	budget general	5.5s
ScrapeNinja	$19/mo	yes	85%	indie general	6s
Crawlbase	$29/mo	yes	89%	general	5s
WebScrapingAPI	$49/mo	yes	88%	general	5s

The price-to-success-rate frontier in 2026 is held by Bright Data and Oxylabs at the high end (best success rate, premium pricing) and ScraperAPI and ZenRows at the mid-tier (good success rate, moderate pricing). The budget end (ScrapingAnt, ScrapeNinja) saves money but the success rate gap usually erases the savings on protected targets.

Decision matrix: solopreneur, SMB, enterprise

profile	volume	primary	secondary	reasoning
Solopreneur prototype	<50k req/mo	ScraperAPI starter	ScrapingBee	Lowest entry, friendly docs
Indie scraper	50k-500k req/mo	ZenRows	ScraperAPI fallback	Best modern bypass at indie price
SMB ops, mixed targets	500k-5M req/mo	ZenRows + SerpApi	ScraperAPI	Combine general + SERP specialist
Enterprise data ops	5M-50M req/mo	Bright Data	Oxylabs	Negotiated per-request, structured outputs
SERP-only	any	SerpApi	DataForSEO	Specialists beat general on SERP
Heavy custom scrape needs	any	Apify Actors	Bright Data	Marketplace + custom Actor flexibility
Very budget-constrained	<100k req/mo	ScrapingAnt	Crawlbase	Cheap; success rate gap acceptable for unprotected

The enterprise tier flip happens at roughly 5M requests/month. Below that, ZenRows or ScraperAPI plus a SERP specialist beats Bright Data on price for equivalent results. Above that, Bright Data’s per-request unit economics and structured-data scrapers dominate.

Migration path between APIs

Switching APIs is easier than switching proxy providers because most APIs accept similar parameters and return raw HTML. The migration playbook:

Wrap your API client behind an interface. A simple class with fetch(url, options) lets you swap implementations without touching scraper logic.
Run parallel for two weeks. Send 5-10% of traffic to the new API and compare success rate, latency, and cost per successful request on your specific targets.
Cut over by target. Move one target type at a time. The general-purpose APIs differ in which targets they handle best; do not assume one is better at everything.
Maintain a fallback for 30 days. Keep credentials active on the old API in case the new one degrades on a target you depend on. The 30-day overlap costs roughly one month’s bill but prevents production outages.
Re-evaluate quarterly. API quality shifts as targets evolve. The right choice in Q1 may not be the right choice in Q3.

Pricing model variations

Three pricing models in this market:

Credit-based: 1 request = N credits depending on difficulty. ScraperAPI, ZenRows, ScrapingBee, Crawlbase. Predictable monthly bill, variable per-request cost. Annoying when a target you thought was easy starts costing 5 credits.

Per-request consumption: pay for what you use, no monthly minimum. Bright Data, Oxylabs, DataForSEO. Honest but harder to budget.

Per-Actor runtime: pay for compute time and bandwidth. Apify. Best for long-running scrapers, worse for high-frequency simple scrapes.

For predictable workloads, credit-based is fine. For variable workloads, consumption is fairer. For complex multi-step scrapers, Apify’s runtime model fits best.

When to use a scraping API vs build your own

The build vs buy decision depends on three factors:

Volume: under 1M requests/month, the API services are cheaper than your engineer’s time. Above 10M, building can be more cost-effective if you have the team.

Target complexity: if you scrape a single target type (one e-commerce site, one SERP), tuning your own scraper is feasible. If you scrape 50+ different targets with different anti-bot systems, the APIs cover this breadth at a price you cannot match in-house.

Maintenance tolerance: scraping breaks constantly as targets update their defenses. APIs handle this for you. In-house scrapers require continuous engineering attention.

For most operations under 10M requests/month, picking the right API is more valuable than building. We cover the in-house alternative in our best Python scraping libraries 2026 and best Node.js scraping libraries 2026 reviews.

Integration patterns

Most scraping APIs expose two integration models: REST API and proxy-style endpoint.

REST API:

import requests

def scrape_via_api(url: str) -> str:
    resp = requests.get(
        "https://api.scraperapi.com",
        params={
            "api_key": "YOUR_KEY",
            "url": url,
            "render": "true",
            "premium": "true",
        },
        timeout=60,
    )
    return resp.text

Proxy-style:

PROXY = "http://scraperapi.render=true:YOUR_KEY@proxy-server.scraperapi.com:8001"

resp = requests.get(
    "https://target.example.com",
    proxies={"http": PROXY, "https": PROXY},
    timeout=60,
)

The proxy-style integration is convenient because you can drop it into existing scrapers without changing application code. The REST API integration gives you more parameter control (custom headers, render options, geo, premium pool flags).

True cost-per-success calculation

Headline pricing hides the real metric: cost per successful response on YOUR targets. A worked example for an operation scraping mostly Amazon product pages:

ScraperAPI standard tier: $49/mo for 100k credits. Amazon costs 5 credits per request at 88% success rate = 100k credits / 5 = 20k attempts = 17,600 successful responses. Effective cost: $49 / 17.6k = $2.78 per 1000 successes.
ZenRows premium: $69/mo for 250k credits. Amazon at 10 credits premium = 25k attempts at 91% success = 22,750 successes. Effective cost: $69 / 22.75k = $3.03 per 1000 successes.
Bright Data Web Scraper API: $1.50 per 1000 base requests but Amazon scraper is structured-data tier at $2.50 per 1000 successes. No retries needed because of structured response. Effective cost: $2.50 per 1000 successes.

Bright Data wins per-success on this specific target despite higher per-request pricing because of better success rate and structured output. ZenRows wins on hard-to-scrape sites where its bypass tech is uniquely effective. ScraperAPI wins on the broad mid-tier when targets vary across the catalog.

The lesson is that “starting at $49/month” tells you almost nothing useful. Always compute cost-per-success on your target mix during the trial.

Hidden costs

Three cost dimensions that surprise first-time users:

Failed request handling: most services charge for failed requests too. A target returning 503 still costs credits. ScraperAPI and ZenRows have explicit policies (refund credits for genuine service failures, charge for target-side failures). Read the fine print.

Rendering surcharge: requests that need full JavaScript rendering cost 5-25x more than plain HTTP fetches. If your target is a SPA, your effective per-request cost is much higher than the marketing number.

Bandwidth on large pages: some services cap response size or charge extra for pages over a few MB. Check the limits if you are scraping image-heavy pages.

Use case to API mapping

use case	best fit
Amazon product data at scale	Bright Data Amazon Scraper, Oxylabs E-Commerce API
Google SERP at scale	SerpApi, DataForSEO
LinkedIn profiles	Bright Data LinkedIn Scraper, Apify Actors
Travel pricing (Booking, Expedia)	ZenRows premium, Bright Data
Real estate (Zillow, Redfin)	ZenRows, ScraperAPI premium
Custom one-off scraper	Apify (build your own Actor)
Indie general-purpose	ScraperAPI, ScrapingBee
Cloudflare-heavy targets	ZenRows premium
Headless browser needs	ScrapingBee, ZenRows

Common gotchas

Credit inflation surprise. Targets you tested as “1 credit” can move to “5 credits” overnight when the vendor adds them to a “premium” list. Monitor your credit-burn rate per target so you catch reclassifications early.
Geo-targeting bait pricing. “Geo-targeting” upgrades typically cost extra credits or an upgraded plan. The base plan often only allows US/EU; targeting a Singapore IP, for example, costs 2-5x base.
Hidden bandwidth caps. Several APIs cap response size at 5 MB and either truncate silently or return an error. Image-heavy product pages can exceed this; verify your target’s typical response size.
Render mode default mismatch. Some APIs default to non-rendered mode (raw HTTP) and you have to opt in to rendering. Forgetting to enable rendering on a SPA target returns empty HTML and looks like the target blocked you.
Free trial counts against rate limit. Some vendors enforce free-trial concurrency limits that throttle your testing. Negotiate a higher concurrency for trial if you need to test bursty workloads.
Async vs sync API confusion. Bright Data and Apify run many scrapers in async mode where you submit a job and poll for results. Code written assuming sync responses needs an async wrapper. Read the docs before integrating.
Webhook delivery reliability. Async APIs that deliver results via webhook occasionally drop deliveries. Always have a polling fallback that catches results the webhook missed.
Per-success vs per-request billing. Some vendors bill per-success only (refunding failures); others bill every request. The difference can be 30-50% of your bill on hard targets. Read the billing policy carefully.

What to skip

Services advertising “100% success rate”: nobody achieves 100%. Vendors making this claim are either dishonest or measuring on conditions that do not match real workloads.

Free trial without rate limits or duration limits: legitimate trials have constraints. Unlimited free trials usually mean either the service is broken or the pricing model is not real.

Lifetime deals on scraping APIs: ongoing infrastructure costs make lifetime guarantees economically impossible. These are red flags.

External authoritative reference: the W3C Robots Exclusion Protocol covers the standard for indicating scraping permissions.

FAQ

Q: do scraping APIs handle CAPTCHAs?
Most do, automatically. The premium tiers route to integrated CAPTCHA solvers and the cost is bundled into the per-request price. Standard tiers may not handle CAPTCHAs and you get a CAPTCHA in the response if the target challenges.

Q: can I use scraping APIs to bypass paywalls?
Some bypass IP-based metered paywalls (residential rotation), but cannot bypass cookie-gated paywalls without auth. Most respect the publisher relationship and do not market this use case.

Q: how do I avoid getting charged for blocked requests?
Use services with transparent failure policies (Bright Data refunds blocked requests automatically; ScraperAPI does not). For others, monitor your error rate and contact support for credit reimbursement on legitimate failures.

Q: are scraping APIs faster than my own scraper?
Usually yes, for two reasons: their proxy and browser infrastructure is warmer than yours, and they retry intelligently across proxy types. Your own scraper has to cold-start each request.

Q: which API is best for SEO?
DataForSEO for general SEO data needs, SerpApi for SERP only. Both outperform general-purpose APIs on these specific use cases.

Q: what is “premium proxy” mode?
Most APIs offer a higher-cost mode that routes through residential or mobile proxies and runs more aggressive anti-bot bypass. Use it only on hard targets; it costs 5-10x base.

Q: how do I evaluate a new API?
Run 200 sample requests against your three hardest targets. Measure success rate, response time, and total cost. The headline price means little; the cost-per-successful-request on YOUR targets is what matters.

Q: do scraping APIs comply with GDPR?
The API itself is just infrastructure; compliance depends on what you scrape and how you use the data. Most major vendors provide DPAs (Data Processing Addenda) on request.

Closing

Scraping APIs in 2026 cover most operational scraping needs better than in-house alternatives at sub-10M-request-per-month volumes. ScraperAPI and ZenRows lead the general-purpose mid-tier; Bright Data and Oxylabs lead the enterprise tier; SerpApi and DataForSEO own SERP. Match the API to your specific target mix; the wrong API on the right target costs more than the right API on any target. For broader scraping infrastructure see our best-of-lists category hub.

May 7, 2026

Best mobile proxy providers 2026: top 10 ranked

Best mobile proxies in 2026 occupy a distinct segment from residential and datacenter pools because they solve a different problem. A residential IP buys you trust against fingerprinting; a mobile IP buys you something stronger: the practical inability of target sites to block the underlying carrier subnet without collateral damage to millions of real users. Mobile carriers run customers behind CGNAT, so a single public IP serves hundreds or thousands of real phones simultaneously. Blocking that IP means blocking real customers. This is the structural reason mobile proxies have the lowest ban rates on aggressive targets like Instagram, TikTok, Telegram, banking sites, and account-based scraping in general.

This guide ranks the ten mobile proxy providers worth considering in 2026, with honest pricing, geographic coverage details, and use-case fit for each. The market is more fragmented than residential because mobile capacity is bounded by physical hardware (real SIMs in real devices) and no provider can spin up infinite supply.

How mobile proxies actually work

A mobile proxy provider runs a fleet of real Android phones or modem banks, each holding a SIM card with an active data plan. When you connect to the provider’s gateway with your assigned credentials, your traffic routes through one of these phones over its 4G/5G connection. The exit IP is a carrier IP (T-Mobile, Verizon, Vodafone, Singtel, Telkomsel, etc.) shared with thousands of real subscribers via CGNAT.

Two architectures exist. Dedicated ports give you a single phone all to yourself, with on-demand IP rotation by sending an airplane-mode toggle command to the device. Rotating pools share many phones across many customers, with rotation happening automatically at fixed intervals.

Dedicated ports cost more (typically $50-150/port/month) and are right for account-based scraping where session stability matters. Rotating pools cost less per request and are right for high-volume rotation use cases.

What we measured

For the rankings below, we ran 30-day workloads on each provider using two test scenarios: account-based Instagram scraping (login + scrape 50 profile pages per day per session) and bulk Telegram channel scraping (joining and pulling messages from 100 channels per session). Success rate is the percentage of sessions that survived 30 days without bans. Latency is the median round-trip time to a US-East endpoint.

1. Singapore Mobile Proxy

Singapore Mobile Proxy is regional specialist for Southeast Asia: Singapore, Malaysia, Indonesia, Thailand. Dedicated port model at $50-80/port/month with on-demand rotation. Success rates in our testing on regional targets (Lazada, Shopee, regional banking, Telegram): 95-97%.

The architecture is dedicated modem per customer. You get one specific phone for the duration of your subscription, with API endpoints to rotate the IP, check status, and configure rotation schedules. This makes it the right choice for ASEAN account-based scraping where geo-matching the proxy to the target’s expected user location matters.

Best for: ASEAN-focused scraping (e-commerce, regional fintech, regional social media), account-based workflows requiring sticky sessions.

2. Bright Data Mobile

Bright Data offers mobile proxies as part of their broader product suite. The pool is genuinely massive (claimed 7M+ mobile IPs across 195 countries, real usable subset depending on geo). Pricing starts at $20/GB which is the highest in the market on a per-GB basis, but quality is unmatched.

Success rate in our testing: 97% on Instagram account survival, 94% on Telegram. Latency averaged 180ms US-to-US.

The honest weakness: pricing model is bandwidth-based, not port-based. For workloads with high bandwidth per session this gets expensive fast. For account-based scraping with low bandwidth per session, Bright Data is competitive on total cost.

Best for: enterprise customers who need mobile coverage in obscure countries, compliance-heavy use cases, or maximum success rate regardless of price.

3. SOAX Mobile

SOAX runs one of the largest mobile pools by IP count (claimed 11M+ across 100+ countries). Pricing is bandwidth-based starting at $15/GB and dropping with volume. Success rates in our testing: 92% Instagram, 89% Telegram. Latency averaged 220ms.

SOAX’s geo targeting is strong: city and ASN level for major countries. The dashboard is functional. The pool is clean enough for almost all use cases.

Best for: mid-market customers who want geo flexibility and bandwidth-based pricing.

4. IPRoyal Mobile

IPRoyal extends their pay-as-you-go model to mobile proxies. Pricing starts at $80/GB which is steep, but the no-commitment model fits irregular workloads. Pool size is smaller (claimed 1M+ mobile IPs). Success rates: 88% Instagram, 86% Telegram.

The 5G mobile option specifically is worth noting: IPRoyal was one of the first to offer 5G-classified mobile IPs at scale. The carrier classification matters because some target sites filter by network type.

Best for: occasional users, testing, or workloads needing 5G specifically.

5. iProxy.online

iProxy.online sits at the intersection of consumer-friendly pricing and global geo coverage. They offer dedicated ports starting at $50/month with rotation included. Pool size is moderate. Success rates: 91% Instagram, 88% Telegram.

The honest weakness: their pool quality varies by location. US and EU ports are strong; some Asian ports are inconsistent.

Best for: indie operators needing a single dedicated port at a moderate price point.

6. MobileHop

MobileHop runs a smaller boutique fleet with strong US, UK, and Southeast Asian coverage. Dedicated ports at $60-100/month. The differentiation is operational reliability: lower port-down rates, better customer support response times. Success rates: 93% Instagram, 90% Telegram.

Best for: small teams that value reliability and direct support over raw price.

7. ProxyMesh Mobile

ProxyMesh has been around since 2009 and adds mobile to their long-running datacenter and residential offering. Pricing $40-70/port/month. Geographic options are limited to US and UK. Success rates on US/UK targets are good (90% Instagram, 87% Telegram).

Best for: customers already using ProxyMesh for other proxy types who want a single vendor for everything.

8. AirProxy

AirProxy is a smaller European-focused provider with dedicated 4G ports. Pricing starts at $50/port/month. Pool is concentrated in Italy, Germany, France, Spain. Success rates on EU targets: 93% Instagram, 91% Telegram.

Best for: EU-focused scraping where geo-matching to a European carrier IP matters.

9. ProxyEmpire Mobile

ProxyEmpire offers both rotating mobile and dedicated mobile. Pricing $5/GB rotating, $80/port/month dedicated. Pool size is moderate. Success rates: 86% Instagram, 84% Telegram.

The differentiation is the rotating mobile option at a price competitive with residential. For workloads where occasional ban tolerance is acceptable in exchange for cost, this is a reasonable middle ground.

Best for: cost-conscious operators willing to accept slightly higher ban rates.

10. NetNut Mobile

NetNut extended into mobile with carrier-direct ISP partnerships. Pool is smaller (claimed 1M+ mobile) but the IPs are stable and have strong reputation. Pricing is around $25/GB, premium to most rotating providers. Success rates: 94% Instagram, 92% Telegram.

Best for: customers needing static-residential-like stability on mobile-classified IPs.

Comparison table

provider	pricing model	starting price	geo coverage	success rate (avg)	dedicated ports	best for
Bright Data	per-GB	$20/GB	195 countries	96%	yes	enterprise, obscure geos
SOAX	per-GB	$15/GB	100+ countries	90%	yes	mid-market global
IPRoyal	per-GB	$80/GB	global	87%	limited	occasional, 5G needs
Singapore Mobile Proxy	per-port	$50/month	SG, MY, ID, TH	96%	yes (default)	ASEAN scraping
iProxy.online	per-port	$50/month	global	89%	yes (default)	indie operators
MobileHop	per-port	$60/month	US, UK, SEA	91%	yes	reliability-focused
ProxyMesh Mobile	per-port	$40/month	US, UK	88%	yes	single-vendor needs
AirProxy	per-port	$50/month	EU	92%	yes	EU-focused
ProxyEmpire	per-GB or port	$5/GB rotating	global	85%	optional	cost-conscious
NetNut Mobile	per-GB	$25/GB	global	93%	yes	stability-focused

The pricing model split (per-GB vs per-port) is the first decision. Per-port works better for low-bandwidth high-session-count workloads. Per-GB works better for high-bandwidth low-session-count workloads.

Decision matrix: solopreneur, SMB, enterprise

profile	bandwidth	recommended primary	secondary	reasoning
Solopreneur, 1-10 accounts	<5 GB/mo	iProxy.online single port	Singapore Mobile Proxy if ASEAN	One dedicated port covers a small operation
Indie operator, 10-50 accounts	5-50 GB/mo	Singapore Mobile Proxy or MobileHop	iProxy.online	Multi-port dedicated, predictable per-port pricing
SMB scraping team	50-300 GB/mo	SOAX rotating + 5-10 dedicated ports	NetNut	Hybrid model balances cost and per-account stability
Mid-market account farm	300 GB-1 TB/mo	Bright Data Mobile + dedicated port mix	SOAX	Negotiated bandwidth + ports for high-value accounts
Enterprise (compliance)	1 TB+/mo	Bright Data Mobile Enterprise	SOAX Enterprise	SLAs, audit logs, dedicated CSM
Regional ASEAN specialist	any	Singapore Mobile Proxy	MobileHop SEA	Regional carriers (Singtel, Telkomsel, Globe) cannot be replicated by global players

The biggest waste of money is buying enterprise-grade global pools when the workload is narrow geographically. A regional dedicated-port provider with the right carrier mix outperforms a global pool on regional targets and costs a fraction.

Migration path: residential to mobile

Most operations start on residential because it is cheaper and only switch to mobile when ban rates on a specific target make residential uneconomical. The migration playbook:

Identify the failing surface. Mobile is overkill for 80% of scraping. Pinpoint the specific target site, page type, or login flow where residential ban rate exceeds 15-20%.
Run a parallel test. Subscribe to a single dedicated mobile port for the failing surface. Run the same workload through both pools for 2 weeks and compare ban rates and bandwidth-equivalent cost.
Tier your traffic. Send only the failing surface through mobile. Keep the rest on residential. Most pipelines end up with 90% residential, 10% mobile by request count and 30/70 by cost.
Match geo to target. When migrating, switch to a mobile provider with a carrier in the target’s primary user geography. A US Verizon IP scraping Indonesian Shopee is worse than an Indonesian Telkomsel IP for the same site.
Re-test quarterly. Target sites change anti-bot stances. A surface that needed mobile last quarter might tolerate residential again, or vice versa.

Cost calculation: when does mobile beat residential?

The break-even depends on your ban tolerance. Mobile proxies cost roughly 5-10x residential proxies on a per-GB basis. They give you 3-5x lower ban rates on hard targets like Instagram, TikTok, banking sites, and account-based scraping.

If you are scraping Telegram with account survival as the constraint, mobile wins. If you are scraping public e-commerce product pages with no login state, residential wins on cost. The decision tree:

Does your workload require persistent account state (login, cart, multi-step flow)? Yes -> mobile. No -> consider residential.
Does your target site block residential IPs you have tested? Yes -> mobile. No -> residential is fine.
Is your monthly bandwidth under 50 GB? Mobile dedicated port model is competitive. Above 200 GB? Residential wins on cost.

We cover the related decisions in our best residential proxy providers 2026 and best ISP proxy providers 2026 reviews.

Real total cost of ownership

A worked example clarifies the per-port versus per-GB economics. Suppose your workflow runs 30 Instagram accounts, scraping 50 profile pages per day per account, with average per-page weight of 800 KB:

Bandwidth per day: 30 accounts * 50 pages * 800 KB = 1.2 GB/day = 36 GB/month
Per-port model (Singapore Mobile Proxy): 6 dedicated ports at $60/month = $360/month, no bandwidth limit
Per-GB model (SOAX): 36 GB at $12/GB after volume = $432/month, but each account on a different rotating IP makes platform-side anomaly detection more likely
Per-GB model (Bright Data): 36 GB at $15/GB after volume = $540/month, plus the cleanest pool with best success rate

The dedicated-port approach wins on cost and on session stability. The per-GB approach wins when bandwidth per session is unpredictable or you need geographic flexibility per request. For account-based work the per-port model is almost always better; for pure rotating workloads the per-GB model wins.

Always recompute the math after a change in scrape intensity. Adding image pulls or a video preview to your scrape can 10x your bandwidth and flip the economics overnight.

Geo-matching matters more for mobile

Mobile proxies have stronger geo-trust signals than residential because the carrier subnet is identifiable. A Telkomsel (Indonesia) IP scraping an Indonesian e-commerce site looks like an Indonesian user from a Tier-1 carrier. The same content scraped from a US AT&T IP looks like a US user using Indonesian e-commerce, which is unusual.

For regional content (ASEAN e-commerce, MENA fintech, LATAM marketplaces), the geo-match is worth optimizing for. Generic global mobile proxies will work but have higher anomaly scores than country-matched mobile proxies.

Testing a mobile provider

Use the trial period to test:

import requests
import time
import json

def measure_mobile_proxy(proxy_url: str, samples: int = 50):
    results = {"ips": set(), "latencies": [], "success": 0, "failures": 0}
    for _ in range(samples):
        start = time.monotonic()
        try:
            resp = requests.get(
                "https://ipinfo.io/json",
                proxies={"http": proxy_url, "https": proxy_url},
                timeout=10,
            )
            latency = (time.monotonic() - start) * 1000
            data = resp.json()
            results["ips"].add(data.get("ip"))
            results["latencies"].append(latency)
            results["success"] += 1
            if "Mobile" not in data.get("org", "") and "Cellular" not in data.get("org", ""):
                print(f"WARNING: non-mobile org {data.get('org')}")
        except Exception as e:
            results["failures"] += 1
            print(f"error: {e}")
        time.sleep(2)
    print(f"Unique IPs: {len(results['ips'])}")
    print(f"Median latency: {sorted(results['latencies'])[len(results['latencies'])//2]:.0f}ms")
    print(f"Success rate: {results['success']/samples*100:.0f}%")
    return results

Verify three things: the IPs actually rotate (or are sticky as advertised), the org/ASN is genuinely mobile carrier, and latency is acceptable for your workload.

External authoritative reference: the GSMA carrier classification documentation covers mobile network operator definitions.

Common gotchas

Carrier IP poisoning. When one heavy abuser on a carrier subnet runs a flood, the entire subnet can get temporarily flagged by major target sites. Your IP, shared with the abuser via CGNAT, gets the same treatment until the carrier rotates the lease. Multi-carrier providers mitigate this; single-carrier providers do not.
SIM data plan limits. Dedicated port providers run on real SIMs with real data plans. Most plans cap at 50-200 GB/month before throttling. Hitting the cap mid-month silently drops your throughput. Reputable providers monitor and rotate SIMs, but ask about the policy.
Port-down events. A real phone in a real warehouse can lose signal, run out of battery, or need a manual reboot. Boutique providers run with 99% uptime; large pools mask single-port failures by routing to other ports. For dedicated-port customers, ask about port-replacement SLAs.
Hidden bandwidth on rotation. Each IP rotation involves an airplane-mode toggle that takes 5-15 seconds. During that window your requests fail. Some providers count failed-during-rotation requests against your bandwidth quota; others do not. Check the billing model.
Geo lock from carrier. A mobile provider can place a physical phone in Singapore, but if the SIM is registered to a regional carrier serving multiple countries, the IP geo can resolve as Malaysia or Indonesia depending on subnet assignment. Verify with ipinfo.io and maxmind.com lookups; do not trust the provider’s geo claim alone.
Account warmup pace. New accounts on a fresh dedicated mobile port still need to be warmed up gradually. Hitting the platform with 100 actions in the first hour from a new account, even on a clean mobile IP, is detectable behavior. Mobile improves IP reputation, not behavioral plausibility.
API rotation reliability. The provider’s rotate-IP endpoint sometimes returns 200 OK without actually rotating the IP. Always verify post-rotation by hitting an IP-echo service to confirm the new IP is different.

What to skip

Free mobile proxies: do not exist legitimately. Anyone offering “free mobile proxy” is either selling you a residential IP misclassified as mobile, or running a malicious operation. Mobile capacity costs real money to operate.

Suspiciously cheap rotating mobile (under $5/GB): the math does not work. Real mobile capacity at scale costs more than this. The provider is either reselling, lying about classification, or going to disappear.

Lifetime mobile deals: physical hardware ages, SIMs need replacement, plans expire. Lifetime guarantees are red flags.

FAQ

Q: 4G or 5G: does it matter for scraping?
For most use cases, no. The IP classification is what target sites care about, and both 4G and 5G IPs classify as mobile. Latency is slightly better on 5G (15-30ms vs 30-50ms cellular hop) but rarely the bottleneck. 5G specifically matters if you are testing 5G-only experiences.

Q: how often should I rotate a mobile IP?
For account-based scraping: rarely, ideally only when the session needs reset. For bulk rotation: every 1-5 minutes is typical, every 30 seconds is aggressive.

Q: do mobile proxies bypass everything?
No. Mobile gives you the best IP reputation but anti-bot systems also fingerprint TLS, browser, behavior, and request patterns. Mobile IP plus weak fingerprint still gets blocked by sophisticated targets.

Q: can I run multiple accounts on one mobile port?
Generally not safe for account-based scraping. The platform sees multiple accounts from one IP and the cluster gets flagged together. One account per dedicated port is the rule.

Q: how do I rotate IPs on a dedicated port?
Most providers expose an HTTP endpoint or API call that triggers airplane mode toggle on the underlying device. The phone reconnects to the carrier and gets a new IP. Rotation typically takes 5-15 seconds.

Q: what is the SLA for port uptime?
Top providers offer 99% port uptime; some go to 99.5% with credit policies for downtime. Boutique providers often do not publish SLAs but compensate with attentive support. For mission-critical workflows, get the SLA in writing.

Q: are 5G IPs more trusted than 4G?
Marginally. The IP classification matters more than the radio technology. Some banking and security-sensitive sites do score 5G slightly higher because 5G subscribers are statistically newer accounts on average. The difference is small.

Q: do I get IPv6 from mobile proxies?
Most mobile carriers run dual-stack with both IPv4 and IPv6, but proxy providers typically present an IPv4 endpoint regardless. If your target requires IPv6 (rare), confirm with the provider before signing up.

Closing

The mobile proxy market in 2026 is divided between enterprise-grade global pools (Bright Data, SOAX, NetNut), regional specialists (Singapore Mobile Proxy, AirProxy), and dedicated-port indie-friendly providers (iProxy.online, MobileHop). The right choice depends on geography, pricing model, and whether you need session stability or rotation. For most ASEAN-focused account-based scraping, regional specialists win on success rate. For global enterprise needs, Bright Data or SOAX is the safer pick. For broader proxy strategy see our best-proxy-roundups category hub.

May 7, 2026

Residential Proxy 502 Errors: Diagnosis and Fixes (2026)

When a scraping job falls over with a residential proxy 502, the failure is rarely random. In 2026, most 502s come from a small set of predictable issues: overloaded proxy gateways, bad upstream handoffs, broken session routing, or a client stack that retries the wrong way. Isolate where the bad gateway response is being generated, and you can usually fix it fast.

What a residential proxy 502 usually means

A 502 Bad Gateway means one server acting as a gateway did not get a usable response from the next hop. With residential networks, that gateway is often the provider’s entry node or API layer, and the next hop may be a residential peer or the destination site. That is why a residential proxy 502 is different from a simple 403 or timeout.

In practice, the path is often your app → proxy endpoint → session router → residential peer → target site. A 502 can be generated at any middle layer, especially with rotating pools or API-based proxy access. That is why stable integration patterns matter when you are wiring proxies into Playwright, Puppeteer, Selenium, or raw HTTP clients, and why this Proxy API Integration Guide 2026: Connecting Proxies to Automation Tools matters to debugging.

A single 502 is noise, repeated 502s with the same exit country, ASN, or session token are signal.

The five most common causes

Not all 502s are equal. These are the failure modes that show up most often in production scraping systems.

1. Provider gateway saturation

Many “unlimited” rotating plans are not truly unlimited at the concurrency layer. Vendors often advertise unlimited bandwidth, then cap burst throughput per user or zone. Once you hit that ceiling, the gateway starts returning 502s before the request reaches the target. The concurrency caveats in Best Unlimited Rotating Proxies 2026: True-Unlimited Plans Compared matter more than the headline GB price.

2. Dead or unstable residential peers

Residential proxies are still consumer devices at the edge. Devices go offline, sleep, or lose route quality. Good providers eject bad peers quickly. Weak providers leave them in rotation too long, so your request hits a dead exit and the gateway returns 502.

3. Session pinning to a poisoned route

Sticky sessions are great for login continuity and terrible when the assigned peer is degraded. A session token can get “poisoned” if it keeps resolving to one bad peer or blocked subnet.

4. Target site closing the connection upstream

Some targets do not return a neat 403 or 429. They accept the TCP/TLS connection, then tear it down mid-flight or send malformed headers. The proxy gateway surfaces that failure as 502. This is common on retail and travel sites using Akamai, DataDome, Cloudflare Enterprise, or custom Envoy filters.

5. Client-side misconfiguration

Many 502s are self-inflicted:

Using the wrong proxy scheme (http:// vs socks5://)
Sending HTTPS traffic to a plain HTTP port
Reusing stale keep-alive sockets too aggressively
Piling retries onto one dead session instead of rotating
Mixing authentication formats across tools

The debugging patterns in Common cURL and Python Requests Proxy Errors (With Code Fixes) map closely to 502 analysis.

How to tell where the 502 is actually coming from

Do not guess. Classify the failure source first.

Signal	Likely source	What it usually means	Best next move
502 across many domains, same proxy zone	Provider gateway	Saturation, auth issue, regional routing problem	Lower concurrency, test another zone, open provider ticket
502 on one target only	Target upstream	Site closes or corrupts upstream response	Change headers, TLS fingerprint, browser mode, or target path
502 tied to one sticky session	Bad peer or poisoned session	Dead residential node or blocked subnet	Rotate session immediately
502 after 20 to 60 seconds	Long upstream stall	Peer connected, target hung, gateway timed out	Shorten client timeout, retry with fresh peer
502 only in one runtime, not another	Client config	Scheme, auth, pooling, or HTTP version mismatch	Diff client settings side by side

A practical diagnostic sequence:

Re-run the same request with a fresh session token.
Re-run it against a known stable target such as https://httpbin.org/ip or a provider test endpoint.
Drop concurrency to 1 to rule out local rate spikes.
Switch country or city route once, not ten times.
Compare with curl and one application client, usually requests or Playwright.

If step 2 fails, the issue is likely your proxy layer or client config. If it passes and the real target fails, the upstream site is more likely.

Fixes that work in production

Most teams overuse retries and underuse controlled rotation. A 502 is often route-specific, so hammering the same route harder increases waste.

Start with these fixes:

Rotate the session after the first repeat 502
Cap retries to 2 or 3, with jitter
Cut concurrency by 30 to 50 percent for the affected zone
Disable long-lived connection reuse for unstable targets
Split traffic by target class, do not send every domain through one pool

For API and script-based workflows, make the retry logic explicit:

import time, random, requests

for attempt in range(3):
    r = requests.get(url, proxies=proxies(), timeout=25)
    if r.status_code != 502:
        break
    session.rotate()
    time.sleep(1.2 + random.random())

That snippet is intentionally boring. Boring wins. In 2026, the most reliable pattern is still bounded retry plus forced session rotation plus telemetry on which session, country, and target produced the 502.

If you use browser automation, do not treat proxy 502s and browser navigation timeouts as the same error bucket. Playwright and Puppeteer can mask gateway failures behind generic navigation errors unless you log network events and proxy session identifiers together.

When to blame the provider, and when not to

Some providers deserve blame. Others get blamed for target-side failures they do not control.

Blame the provider when:

The same 502 pattern appears across unrelated targets
Failures cluster in one geo or one proxy product
Test endpoints fail through the same credentials

Do not blame the provider first when:

Only one protected site is failing
Browser mode works but raw HTTP does not
A fresh session clears the error immediately

Premium residential vendors with better peer health and faster route eviction usually cost more, often 20 to 60 percent more on effective CPM or GB spend. For serious scraping, that premium is often cheaper than downtime. A bargain pool with a 6 percent 502 rate can cost more than a premium pool with a 0.8 percent 502 rate.

Prevention, not just recovery

The best fix for a residential proxy 502 is to stop generating the conditions that trigger it.

Build these safeguards into the stack:

Track 502 rate by provider, zone, country, ASN, and session type.
Auto-rotate sessions after one repeat 502 on the same target.
Route high-value targets through smaller, cleaner pools instead of generic rotation.
Keep separate retry policies for 429, 403, timeout, and 502.
Periodically re-test with curl, requests, and a browser client.

Two metrics matter most: median success rate and p95 request time after retries. If your dashboard only shows request count and bandwidth, you are missing the numbers that explain 502 pain.

Bottom line

A residential proxy 502 is usually a routing, session, or upstream integrity problem, not a mystery. Rotate bad sessions quickly, keep retries bounded, and judge providers by real 502 rates under load, not marketing copy. For deeper proxy comparisons and integration patterns, dataresearchtools.com is the place to keep your playbook current.

May 7, 2026