Caching responses to cut proxy spend

The bill scales with requests, not with data

Proxy providers charge by bandwidth or by request count, and either way the number goes up every time your scraper hits the network. Most pipelines don’t need to hit the network nearly as often as they do. A price monitor that checks the same product page every ten minutes is paying for ten fresh fetches an hour when the price probably changed once, if at all. A lead-gen crawler that re-walks a directory site’s category pages every run is buying bandwidth for HTML it already has sitting on disk from yesterday.

Caching doesn’t reduce how much data you need. It reduces how many times you pay a proxy to go get data you already have. That’s the whole value proposition, and it’s worth being precise about it, because caching gets oversold as a performance trick when its real job here is a cost-control lever sitting between your scraper and your IP pool.

What’s actually worth caching

Not everything you scrape is equally volatile, and that’s the entire basis for a caching strategy. Split your targets into three rough buckets before you write any caching code:

Near-static. Category taxonomies, “about” pages, terms of service, historical archive pages, anything with a URL that implies it won’t change again. These can sit in cache for days or weeks with almost no risk of serving stale data that matters.

Slow-moving. Product listings, job postings, review counts, search result pages for stable queries. These change on the order of hours, sometimes once a day. A cache with a TTL matched to your actual monitoring cadence saves the bulk of your proxy spend here, because this bucket is usually the largest share of request volume in a monitoring pipeline.

Volatile. Live pricing during a sale, inventory counts, anything time-sensitive enough that a stale read is a wrong read. This bucket should mostly bypass cache, or use a TTL measured in seconds rather than minutes.

The mistake I see most often is treating every URL as volatile by default, which means every URL pays full proxy cost on every run. Sort your target list into these buckets first. The caching layer only pays for itself on the URLs where staleness is tolerable.

A cache key that doesn’t lie to you

The cache key has to represent everything that changes the response, or the cache will confidently hand back the wrong page. URL path and query string are the obvious components, but they’re not sufficient. If your scraper sends a different Accept-Language header per region, or your proxy pool rotates through residential exits in different countries that get geo-targeted content, the same URL can return materially different HTML depending on which parameters and which context produced the fetch.

Build the key from URL plus the specific request parameters that affect the response body: query params, the country or region you’re routing through if the target geo-targets, and any pagination or sort parameters. Leave out things that don’t affect content, like a cache-busting timestamp param the target site ignores, or your own request ID header. A key that’s too narrow causes cache misses you didn’t need. A key that’s too broad causes cache hits that return the wrong content for the request that’s asking, which is worse, because it fails silently.

Matching TTL to how fast the target actually moves

Once URLs are bucketed, TTL is a direct translation of “how often do I actually need fresh data” into seconds. If your monitoring job runs every ten minutes but a category page only reshuffles once a day, a TTL of a few hours doesn’t cost you any real freshness and it collapses dozens of would-be fetches into one.

Set TTL per bucket, not globally. A single blanket TTL either wastes proxy spend on the near-static bucket, because you’re re-fetching pages that never change, or serves stale data from the volatile bucket, because a two-hour TTL is fine for a category listing and useless for a flash-sale price. If you can observe how often a given page actually changes, by diffing cached copies over a week, use that to tune the TTL instead of guessing. Pages that turned out static in practice can move to the near-static bucket even if you assumed otherwise going in.

Where the cache sits in the pipeline

The cache belongs between the request scheduler and the proxy client, as a layer the scheduler checks before it ever assigns a proxy to a job. Concretely: the scheduler looks up the cache key, and only if it’s a miss or expired does the job go into the queue that consumes a proxy. This is a different role from an HTTP cache sitting behind a CDN, because you control both ends and you’re optimizing for proxy cost, not for latency to an end user.

For storage, a key-value store like Redis works well when you need shared state across multiple scraper workers, so two workers don’t independently fetch the same URL in the same window. A flat store keyed by a hash of the cache key, with the response body and a TTL, is enough for a single-machine pipeline. Whatever you use, store enough metadata alongside the body to make debugging possible later: the fetch timestamp, the proxy or exit node that served it, and the response status code. When something looks wrong three days from now, that metadata is what tells you whether the problem is the cache or the fetch.

The failure mode nobody mentions: caching a block page

This is the one that actually costs money instead of saving it. If your fetch gets a CAPTCHA challenge, a rate-limit response, or a soft-block page that still returns HTTP 200, and your caching logic only checks status code before writing to cache, you’ve just cached the block page. Every subsequent read for that TTL window returns a page with no data in it, and your pipeline reports the target as unchanged or empty instead of blocked. You won’t notice until someone asks why a page that’s been live for months suddenly shows no products.

Guard the write path, not just the read path. Before writing a response to cache, check for the markers that indicate a challenge or block: known CAPTCHA page signatures, unexpected redirect targets, a response body far shorter than the page’s normal size, or the absence of the DOM elements your parser expects to find. Treat those as fetch failures that don’t get cached, not as valid responses to store. A cache is only trustworthy if what goes into it is verified to be real content, and that check has to run before the write, not after something downstream complains.

Conditional requests, where the target supports them

Some sites still honor If-Modified-Since or If-None-Match with a 304 Not Modified response, which costs far less bandwidth than a full page fetch and, depending on your provider’s pricing model, may cost less against your proxy quota too. If a target returns ETag or Last-Modified headers on its responses, store them alongside the cached body and send them back on the next check. A 304 confirms your cached copy is still good without paying for the full transfer. This isn’t something you can rely on across a target list, plenty of sites don’t set these headers meaningfully, but where it works it’s close to free freshness verification.

Deduplicating requests that never needed a proxy at all

Separate from time-based caching, watch for the case where your own pipeline issues the same request twice in the same run, two different scraper jobs both need the same category page, or a retry queue re-adds a job that’s already in flight. A short-lived in-flight lock, keyed the same way as the cache, lets the second caller wait for the first fetch to complete and read its result instead of starting a duplicate fetch through a second proxy. This has nothing to do with target volatility. It’s just not paying twice for a fetch you were already about to make once.

What caching doesn’t fix

A cache reduces how often you go back to a target for data that hasn’t changed. It does nothing for a pipeline that’s scraping more targets than it needs, storing full page bodies when it only needs three fields, or running on a schedule tighter than the business case actually requires. Those are separate cost problems, and they’re usually the bigger ones. Caching is the layer that stops you from paying for the same answer twice. It’s not a substitute for asking whether you needed to ask the question that often in the first place.

If you’re building out scraping infrastructure and want more on proxy architecture, pipeline design, and how bot detection actually works from the defensive side, browse the rest of the site here.

Get new guides and videos first — join the Telegram channel.