Permission denied for Desktop. here’s the full article markdown body:
—
Render quietly became one of the better homes for persistent scrapers in 2026. deploying scrapers on Render with background workers gives you always-on processes, Redis-backed queues, and a simple deploy story — without the operational overhead of managing your own fleet. if you’re running anything heavier than a 60-second cron job, you need to understand how Render’s worker model actually behaves before you commit.
how Render’s background worker service type works
Render splits services into web services (which bind a port) and background workers (which don’t). background workers are persistent processes — they start on deploy and stay running. Render will restart them on crash, but there’s no HTTP health check, so you don’t need to fake a server just to keep the dyno alive.
this is already an improvement over some competitors. when deploying scrapers on Railway 2026, you’re working inside Railway’s cron + worker model which is solid but requires you to think in job intervals. on Render, your worker just runs. you control the loop internally.
a minimal Python worker looks like this:
import time
import redis
import os
r = redis.from_url(os.environ["REDIS_URL"])
while True:
job = r.blpop("scrape_queue", timeout=30)
if job:
_, url = job
scrape(url.decode())
time.sleep(0.1)blpop blocks for up to 30 seconds before returning None, which keeps CPU near zero when the queue is empty. this pattern scales cleanly when you add more workers.
queue architecture and Redis on Render
Render offers managed Redis as a first-party add-on (currently $10/mo for the starter tier, 25MB). for most scraping queues this is plenty — a URL string with metadata rarely exceeds 1KB, so 25MB holds ~25,000 jobs before you need to think about eviction policy.
for heavier workloads, use the allkeys-lru eviction policy and keep job payloads lean. don’t serialize the entire HTML response into Redis. write it to Render’s ephemeral disk or an external store (S3, Supabase, R2) and only queue the reference.
if you need job priorities or dead-letter queues, drop in RQ or Celery rather than hand-rolling list operations. RQ in particular has a minimal footprint and its dashboard worker can run as a second background service on Render.
concurrency, scaling, and the free-tier trap
Render’s free tier does not support background workers. workers require at minimum the Starter instance ($7/mo per service). this catches people when they prototype on a web service (which does have a free tier) and then try to convert it.
scaling options on Render in 2026:
| approach | use case | cost implication |
|---|---|---|
| single worker, threaded | low-volume, I/O-bound scraping | 1x Starter instance |
| multiple worker replicas | parallel queue draining | Nx Starter instances |
| horizontal + Redis partitioning | high-throughput, site-specific queues | N instances + Redis tier |
| cron service + worker | scheduled crawls + async processing | 2 services minimum |
Render doesn’t have auto-scaling for background workers the way Fly.io does with machine-level scale-to-zero. if you need region-pinned elastic workers, deploying scrapers on Fly.io 2026 gives you finer control over where your machines run and when they spin down.
for most scraping workloads in the 10k-100k URL/day range, two Render worker replicas sharing a Redis queue is the simplest architecture that works.
disk, memory, and headless browser constraints
Render’s Starter instance gives you 512MB RAM and no persistent disk by default. headless Chromium at idle consumes around 120-150MB, and a page render with JavaScript hydration can spike to 350MB+ depending on the site. this means you cannot reliably run Playwright or Puppeteer on a 512MB worker without careful process management.
practical options:
- upgrade to the Standard instance (2GB RAM, $25/mo) and run Playwright directly
- use a lightweight HTTP-only scraper on Starter and offload JS rendering to a dedicated service
- call out to a serverless renderer — deploying scrapers on Modal Labs 2026 covers GPU-backed headless browser execution for sites that need it
persistent disk is available on Render at $0.25/GB/month. mount it at /data and use it for cookie jars, rotating proxy credential caches, or raw HTML before parsing. don’t rely on the ephemeral filesystem surviving a deploy or restart.
environment, secrets, and deploy configuration
Render uses environment groups, which let you share variables across services without duplicating them. this matters for scraping deployments where multiple workers and a scheduler share the same proxy credentials, API keys, and database URLs.
a typical render.yaml for a two-service scraper setup:
services:
- type: worker
name: scrape-worker
runtime: python
buildCommand: pip install -r requirements.txt
startCommand: python worker.py
envVarGroups:
- scraper-shared
scaling:
minInstances: 1
maxInstances: 3
- type: cron
name: scrape-scheduler
runtime: python
schedule: "*/15 * * * *"
buildCommand: pip install -r requirements.txt
startCommand: python scheduler.py
envVarGroups:
- scraper-shareda few things to note:
envVarGroupspulls from a named group you create in the Render dashboard — keep proxy credentials here, not hardcoded- the cron service only runs the
startCommandon schedule, then exits — it is not a persistent process scaling.maxInstanceson a worker service requires the Team plan; on individual plans you set replicas manually
for secrets rotation (proxy pool credentials, API keys), Render’s dashboard allows env var updates without a full redeploy. the worker process needs to re-read env vars to pick them up, so either restart the service or build a reload signal into your worker loop.
unlike Cloudflare’s edge execution environment, you’re working with a standard Linux container here. there are no fetch() API restrictions, no 128MB memory ceilings, and no 30-second CPU limits that make deploying scrapers on Cloudflare Workers 2026 a puzzle of workarounds for anything stateful.
Bottom line
Render’s background worker model is the right default for teams that want persistent scrapers without infrastructure management. use a Redis queue, size your instances to your actual memory footprint (don’t underspec for headless browsers), and use environment groups to keep credentials clean across services. DRT covers the full range of scraper deployment targets — Render sits in the middle of the spectrum: more control than serverless, less ops than self-hosted.