Cloudflare Workers is one of the most tempting deployment targets for scrapers — edge network, zero cold starts, generous free tier — but it hits hard limits fast. here’s the full picture.
—
Cloudflare Workers sounds like a scraper’s dream: code runs in 300+ PoPs worldwide, spins up in under 1ms, and the free tier gives you 100,000 requests per day. but if you’ve tried deploying a real scraper there, you’ve already hit the walls. the runtime strips out most of what makes deploying scrapers on Cloudflare Workers workable at scale — no filesystem, no native modules, a strict CPU time cap, and zero support for headless browsers. this guide covers what actually works in 2026, what doesn’t, and how teams are routing around the constraints.
What Cloudflare Workers can and can’t do for scrapers
Workers run on V8 isolates, not Node.js. that distinction matters more than people expect. you get the Web Platform APIs (fetch, crypto, streams) but you lose:
child_process,fs,net,http(the Node built-ins)- native addons (no Puppeteer, no Playwright, no libxml2 bindings)
- persistent local state (no disk, no SQLite)
- long-running jobs (CPU time hard cap: 30s on paid, 10ms on free)
what you can do: fetch, parse HTML with a WASM-compiled parser, run regex over response bodies, call external APIs, and fan out requests across the edge. for certain scraping patterns — lightweight crawls, API harvesting, structured data extraction from fast-loading pages — Workers is genuinely useful. for anything requiring a real browser or heavy processing, look elsewhere.
| Capability | Workers (free) | Workers (paid) | Node/VPS |
|---|---|---|---|
| Fetch + parse HTML | yes | yes | yes |
| Headless browser | no | no | yes |
| CPU per request | 10ms | 30s | unlimited |
| Cron triggers | no | yes | yes |
| Filesystem access | no | no | yes |
| Outbound IP control | no | no | yes |
| Memory per isolate | 128MB | 128MB | configurable |
The CPU cap is the real constraint
the 30-second CPU limit on paid Workers sounds generous until you realise it’s CPU time, not wall time. waiting on a fetch doesn’t burn CPU — but parsing a large HTML document with a WASM parser, running xpath queries across dozens of nodes, or decompressing a gzip body absolutely does. a scraper processing 50 product pages in one invocation will trip the limit reliably.
the workaround is to make Workers do less per invocation. fetch one URL, extract minimal structured data, write to a queue (Cloudflare Queues or an external webhook), and return. move aggregation and heavy parsing downstream to a worker that has real compute — something like a persistent background worker. if you need cron-based crawl scheduling without the CPU wall, Deploying Scrapers on Railway 2026: Cron + Background Workers covers a simpler model where the runtime constraints don’t fight you.
Routing around the no-browser limitation
no headless browser means no JavaScript-rendered pages, no cookie jars, no human-like interaction flows. teams handle this two ways:
- pair Workers with an external rendering service — Browserless.io, ScrapingBee, or a self-hosted Playwright cluster — and have the Worker call the render endpoint, receive HTML, then do extraction locally
- use Workers purely as a proxy layer: accept the crawl job, forward to a full compute environment, return the result asynchronously
option 1 adds latency and cost per render (~$0.001-0.003/page depending on provider). option 2 just moves the problem, but keeps your edge logic clean. for teams that need GPU-backed headless rendering at scale, Deploying Scrapers on Modal Labs 2026: Serverless GPU Headless Browsers is worth reading — Modal spins Playwright on GPU-backed containers with no cold-start penalty on warm pools.
What actually works: a fetch-parse-queue pattern
here’s a minimal Workers scraper that stays within the constraints — it fetches one URL, parses title and meta description using the built-in HTMLRewriter, and pushes to a Cloudflare Queue:
export default {
async fetch(request, env) {
const { url } = await request.json();
const resp = await fetch(url, {
headers: { "User-Agent": "Mozilla/5.0 (compatible; DRTbot/1.0)" }
});
const result = { url, title: null, description: null };
await new HTMLRewriter()
.on("title", { text(t) { result.title = (result.title || "") + t.text; } })
.on('meta[name="description"]', {
element(el) { result.description = el.getAttribute("content"); }
})
.transform(resp)
.arrayBuffer();
await env.SCRAPE_QUEUE.send(result);
return Response.json({ ok: true, url });
}
};HTMLRewriter is streaming and WASM-native — it won’t blow your CPU budget on typical pages. avoid pulling innerHTML of full documents and running regex over them; that pattern chews CPU fast.
IP blocks, rate limits, and Cloudflare’s own anti-bot
here’s the irony: Cloudflare Workers egress traffic shares IP ranges with Cloudflare’s CDN. many anti-bot systems (including Cloudflare itself on customer sites) flag CDN ranges aggressively. you’ll see 403s and CAPTCHA challenges on targets that are themselves Cloudflare-protected at a much higher rate than you’d get scraping from a residential or datacenter IP.
the fix is to route your fetch calls through a proxy rather than hitting targets directly. residential proxies with sticky sessions work best for sites with session-bound anti-bot. for cost-sensitive crawls of public APIs or non-JS pages, datacenter proxies are fine. Workers can pass a proxy URL to fetch using the cf options object:
const resp = await fetch(targetUrl, {
cf: { resolveOverride: "your-proxy-host.example.com" }
});note this is a partial workaround — full CONNECT tunneling through SOCKS5 isn’t supported in the Workers runtime as of mid-2026. for workloads where IP quality is critical, a VPS gives you full control. Deploying Scrapers on Hetzner: Cheapest Production Stack 2026 benchmarks the cost difference when you need to control outbound IPs completely.
Cron triggers and stateful crawl management
Workers Cron Triggers (paid plan) fire on a schedule but each invocation is still a single stateless Worker call. you can’t maintain a crawl frontier in memory across invocations. the options for state:
- Cloudflare KV — eventually consistent, good for URL deduplication at low frequency
- Cloudflare D1 — SQLite-compatible, consistent, 10GB max, fits most crawl queues
- Cloudflare Queues — pull-based, exactly-once delivery, max message size 128KB
for large-scale crawls with complex retry logic and dependency chains, Workers’ stateful primitives get awkward fast. Deploying Scrapers on Render 2026: Background Worker Patterns shows how persistent background workers simplify queue management when you need more than KV can offer.
Bottom line
use Cloudflare Workers for the narrow cases where it fits: lightweight fetch-and-parse of fast, non-JS pages, API harvesting, or as an edge fanout layer in front of heavier compute. don’t use it as a general scraping runtime — the CPU cap, no-browser constraint, and CDN IP reputation issues rule it out for most production crawlers. we cover the full deployment landscape across serverless and dedicated options at dataresearchtools.com if you’re still choosing a stack.