Your cart is currently empty!
Blocking images, fonts and trackers in Playwright to cut bandwidth
The bytes you pay for and never read
Open any retail product page in a normal browser with the network panel showing and scroll to the bottom of the request list. You’ll usually find the HTML document near the top, then a long tail of images, web fonts, analytics beacons, ad scripts, video poster frames, and a tag manager payload that pulls in still more. The price, the title and the stock flag you actually want sit inside the document or inside one or two JSON calls. Everything else is decoration for a human eye your collector doesn’t have.
When that browser runs through a per-gigabyte proxy, the decoration is billed. Residential and mobile bandwidth is sold by the byte, and a headless render fetches every asset the page references unless you tell it otherwise. So the same job can cost very different amounts depending on one filter function that most people write in five minutes and never revisit.
This piece is about that filter. We already covered the bigger question of whether a page needs a browser at all in the guide on scraping JavaScript-heavy sites without wasting money, and the question of what to store so you don’t refetch in the caching guide. Both of those decide which requests happen at the job level. This is one layer lower: once a page is rendering, which of the forty or four hundred requests it fires should be allowed out of the browser.
Playwright gives you a clean hook for it. The hard part is deciding what’s safe to drop, because an aggressive filter that saves bandwidth and silently empties half your fields is worse than no filter. Most of what follows is about that decision, and about checking it with evidence instead of hope.
We run collection jobs through our own mobile proxy farm and through bought residential pools, so bandwidth on a render is a line item we watch. None of the numbers below are benchmarks, because page weight varies too much between targets for a single figure to mean anything. The method is what transfers.
How page.route actually intercepts a request
Playwright lets you register a handler for any request whose URL matches a pattern. The handler receives a route object and a request object, and it has to do one of a small number of things with the route: continue it to the network, abort it, fulfill it with a response you make up, or fall back to the next matching handler.
async def filter_assets(route):
if route.request.resource_type in {"image", "media", "font"}:
await route.abort()
else:
await route.continue_()
await page.route("**/*", filter_assets)
That’s the whole mechanism. You can register it on a single page with page.route or on the whole browser context with context.route, in which case every page and popup opened in that context inherits it. Page-level routes take precedence over context-level ones when both match, and when several handlers match the same URL, the most recently registered one runs first. route.fallback() hands the request down to the next matching handler instead of deciding yourself, which is how you layer a general rule under a site-specific exception.
Three details in the documentation matter more than they look. Enabling routing disables the browser’s HTTP cache for that page or context, so a script shared across every page on a site gets fetched again each time instead of coming from cache. Requests made by a service worker aren’t intercepted by page routes at all unless you block service workers on the context. And every routed request makes a round trip between the browser and your Playwright driver process, which adds a small delay per request that grows with the number of requests you route.
None of those are reasons to avoid routing. They’re reasons to route deliberately rather than wrapping every URL in a handler out of habit.
Resource types are the filter you want first
Every request Playwright sees carries a resource_type, which is the browser’s own label for why it was requested. The documented values are document, stylesheet, image, media, font, script, texttrack, xhr, fetch, eventsource, websocket, manifest, plus a catch-all called other. That label comes from how the page asked for the thing: an img tag produces an image request, a CSS url() for a font produces a font request, and a JavaScript fetch() call produces a fetch request regardless of what comes back.
This is more reliable than filtering on file extensions, which is the first thing most people try. Plenty of images are served from URLs with no extension at all, from resizing endpoints with query strings, or from content delivery paths that end in an ID. A glob like **/*.{png,jpg,webp} misses all of them, and it also catches the occasional JSON endpoint someone unhelpfully named with an image suffix.
The resource type does have one blind spot that bites. If a page loads an image through fetch() and turns it into a blob, it arrives as a fetch request, not an image. If a page loads its product data through an ordinary script tag that assigns a global variable, the data arrives as a script request. The label tells you how the browser requested something, not what the bytes contain.
So the sensible first cut is by type, and the sensible types to start with are image, media and font. Those three almost never carry the fields a collector extracts, and on a typical commercial page they’re where most of the weight sits. Media covers video and audio, which can dwarf everything else on a page with autoplaying previews. Fonts are small individually but a site loading six weights of two families adds up across thousands of pages.
Stylesheets and scripts are a different conversation, and they get their own sections below.
Deciding what is safe to drop
The honest answer to “what can I block” is: whatever doesn’t change the record you extract. That sounds obvious, and it gives you a test you can actually run, which beats a rule of thumb copied from a forum.
Start with a baseline. Pick a sample of pages from the target that covers its variety: a normal listing, one with lots of variants, one out of stock, one with a sale price, a category page if you collect those. Twenty pages is a reasonable start for a site with a handful of templates. Render each with nothing blocked and save two things: the extracted record, and a log of every request with its URL, resource type and response size.
Then block one category and run the same sample again. Compare the new records to the baseline field by field. If every field on every page matches, that category is safe for that target at that moment. If anything differs, you’ve found a dependency, and you look at which requests you removed to see which one mattered.
Do it one category at a time, because blocking several at once and seeing a difference tells you nothing about which block caused it. Images first, then media, then fonts, then third-party scripts by domain, then stylesheets if you’re feeling brave.
This test also catches a subtle class of failure: a block that doesn’t break extraction but changes it. A page that reads the viewport layout to decide how many items to render per row, or a lazy loader that only fetches the next batch once earlier images have finished, can return fewer items without throwing any error. The field-by-field diff shows that as a shorter list. A check that only asks “did the page load” never sees it.
It’s slower than guessing. It’s also the only version that lets you tell a client, or yourself, that the filter doesn’t cost you data.
Measuring what a block actually saves
You can’t justify a filter by feel either, and Playwright gives you the numbers without any extra tooling. Listen for requestfinished on the page or context, and for each finished request call request.sizes(), which returns the request and response header and body sizes the browser recorded. Sum the response body sizes by resource type and by domain and you have a per-page weight breakdown.
Run that on the same baseline sample. You’ll usually find the weight is lopsided: a few categories or a couple of third-party domains carry most of the bytes, and a long tail of small requests carries almost nothing. That tells you where filtering is worth the risk and where it isn’t. Blocking a two-kilobyte beacon saves you close to nothing and might break a consent flow. Blocking the video previews on a page might remove most of its weight in one rule.
Two cautions about the numbers. Body sizes are what the browser saw after decompression handling, and what your proxy bills can differ because of transfer encoding, TLS overhead and headers, so treat the breakdown as a relative guide to where weight lives rather than an invoice. And remember that routing disables the HTTP cache, so a baseline measured with a no-op route installed will look heavier than the same page in a normal browser session, because shared scripts refetch on every page.
If you want the proxy side of the ledger, most vendors expose a usage counter per credential. Running the baseline sample and then the filtered sample through a dedicated credential, and reading the counter before and after, gives you the number that lands on your bill. That’s the figure to put in front of anyone asking whether the filter was worth writing.
We keep the per-type breakdown as a saved artifact next to each target’s config. When a target redesigns and the weight profile shifts, the old breakdown shows exactly what changed.
Images: usually safe, occasionally load-bearing
Images are the category everyone blocks first, and most of the time they’re right to. The product data on a modern page is text in the DOM or JSON behind it, and the pictures are there for people.
The exceptions are worth knowing so you recognise them. Some pages trigger their next step from an image event: a lazy-load script that waits for the hero image’s load event before fetching the rest of the listing, or a gallery component that only builds its variant selector after thumbnails arrive. Abort the image and the event that fires is error instead of load, and code that only listens for load never runs. You’ll see it in the diff as missing variants or a truncated list.
When that happens, there’s a gentler option than abort. route.fulfill() lets you answer the request yourself with a tiny valid response, such as a one-pixel transparent image with the right content type. The page’s load handler fires, the script carries on, and no bytes cross the proxy for the real image. It’s more code, but it keeps the page’s own logic intact while still saving the weight.
The second exception is rarer: data that only exists as an image. Some sites render a price or phone number as a picture specifically so it isn’t sitting in the HTML. If a field you need is one of those, you can’t block the image, and you should also think about whether that site is signalling it doesn’t want that field collected.
The third is image URLs themselves. If your record needs the product image link, you don’t need the image bytes. The URL is in the src or srcset attribute and survives an aborted request just fine. People sometimes allow images because they need image links, which pays for the bytes for no reason.
So the default for images is block, with fulfill as the fallback when the diff shows a dependency.
Fonts and media: the easy wins with one catch each
Fonts are close to free to block. A missing web font means the browser falls back to a system font, the text still renders, and every character your selectors read is the same. On sites that load several weights or families, fonts add up to real weight per page across a big crawl.
The catch is icon fonts. Some interfaces draw their buttons, ratings or stock indicators with a font where a private-use character maps to a star or a tick. Block the font and the DOM still contains the same character, so text extraction is unaffected. But if your parser reads a rating by counting rendered stars, or your extraction relies on a screenshot, the icons vanish. That’s another thing the baseline diff catches, and it’s rare enough that fonts stay in the default block list.
Media (video and audio) is usually the single heaviest thing on a page that has any. Autoplaying product videos, background loops on landing pages and preview clips on listing grids can each outweigh the rest of the page. Blocking media is almost always safe for data collection.
The catch with media is that it isn’t always labelled as media. Video players that stream in segments often fetch those segments with fetch() or XHR, so they show up as fetch or xhr rather than media. Your type filter lets them through. If the weight breakdown shows a large volume of fetch traffic to a video host domain, that’s streaming segments, and you block it by domain instead of by type.
Poster frames, the still image shown before a video plays, arrive as images and are handled by the image rule. Captions arrive as texttrack. Neither matters to a collector.
Stylesheets: where aggressive filters break things
Stylesheets are the category where the savings are small and the breakage is quiet. CSS files are rarely heavy compared with images or video, and blocking them changes how the page lays out, which changes what the page’s own JavaScript does.
A surprising amount of front-end behaviour depends on layout. Infinite scroll and lazy loading typically use an intersection observer, which fires when an element becomes visible in the viewport. With stylesheets gone, elements can collapse to zero height or stack in ways the designer never intended, and the observer that should fire as you scroll either fires for everything at once or never fires. Tab panels and accordions that hide content with CSS can expose all their content, which sounds helpful until your selector matches three copies of the same price from three hidden panels.
Playwright’s own actions are affected too. Before clicking, Playwright checks that the element is visible, stable and able to receive events. Without styles, a button can end up covered by another element, rendered off screen, or sized to nothing, and the click waits and then times out. If your collection flow involves clicking “load more” or selecting a variant, blocking CSS can turn a working job into one that hangs.
There’s also a detection angle worth understanding in the third person. A page can observe whether its own stylesheets applied by reading computed styles, and a client that renders pages with no styling at all behaves unlike any ordinary visitor. We take it as one more reason the savings from blocking CSS rarely justify the risk.
Our default is to leave stylesheets alone. If the weight breakdown shows a single enormous stylesheet on a target and the diff proves it’s safe, block it for that target only.
Third-party scripts and trackers, handled carefully
Third-party scripts are where the real judgement lives. Analytics, advertising, session recording, heatmaps, chat widgets and A/B testing tools are often loaded from other domains, and each can pull further scripts, beacons and iframes. On some commercial pages they’re a large share of the requests even when they aren’t the largest share of the bytes.
Filtering them by domain is the usual approach: a set of hostnames you abort, checked against urlparse(route.request.url).hostname. Community-maintained ad and tracker lists exist and some teams import them, but a hand-built list per target, built from the weight breakdown, is easier to reason about when something breaks.
Some third-party scripts do real work. Tag managers are a common trap: the page loads one tag manager script, and the tag manager loads everything else, including sometimes the consent banner that has to be dismissed before content appears, or the A/B framework that decides which layout you get. Block the manager and you might get a page stuck behind a banner, or a layout variant your parser has never seen.
Then there’s the category we deliberately leave out of any block list: scripts a site uses to decide whether to serve the page. Bot-management and challenge scripts, whether first-party or from a vendor, are the site’s defence, and blocking them isn’t a bandwidth optimisation. In practice a site that doesn’t get its check back usually serves a challenge or nothing, so the collector gains nothing and the operator has signalled exactly what the defence looks for. If a target runs that kind of protection, the right conversation is about whether and how that site should be collected at all, which our guides on how sites detect automation cover from the defender’s side.
The workable rule: block third-party domains that appear in the weight breakdown, that aren’t on the page’s critical path, and whose removal the diff shows to be harmless.
Allow-lists versus block-lists
There are two ways to write the filter, and they fail in opposite directions.
A block-list names what to abort, such as image, media and font types plus a set of tracker domains, and lets everything else through. It fails open: when the site adds a new analytics vendor or a new video host, the new traffic flows until someone notices the bill. Its advantage is that it rarely breaks a page, because anything you haven’t thought about still loads.
An allow-list names what to permit, such as document, script, xhr and fetch from the target’s own domains and its known API hosts, and aborts everything else. It fails closed: when the site moves its API to a new subdomain or starts loading product data from a new CDN host, the request is aborted and your fields go empty. Its advantage is tight bandwidth control, because nothing new sneaks through.
Which one to use depends on how you monitor. If you have a field completeness check on every run, the kind we describe in the guide on catching silent failures, an allow-list’s failures show up quickly as a drop in populated fields, and you get the tighter control. If you don’t, a block-list’s failures are cheaper, because the cost of a missed tracker is some wasted bytes while the cost of a missed API host is a run of empty records.
We run block-lists on most targets and allow-lists on a few high-volume ones where the weight savings justify the extra monitoring. Either way, log every aborted request’s hostname at a low sampling rate. When something breaks, that log answers “what did we block” in seconds.
Where the filter should live in your code
A filter that lives inside each scraper as an inline lambda drifts. One target gets a fix, another doesn’t, and six months later nobody knows why two nearly identical jobs have different bandwidth profiles.
The structure that holds up is a shared filter module with a default policy and a per-target override. The default policy is the conservative one: abort anything typed as image, media or font, block a short list of well-known heavy third-party domains, allow everything else. Each target’s config can add domains to block, add domains that must never be blocked, switch to allow-list mode, or swap abort for fulfill on images. The route handler reads that config, so changing a target’s policy is a config change with a clear history.
Install it at context level with context.route, so popups and new pages opened by the site inherit it. If a particular flow needs an exception, a page-level route registered afterwards takes precedence and can call route.fallback() for everything it doesn’t care about, which passes those requests down to the shared handler.
Keep the handler fast. Every routed request pauses in the browser until your handler answers, so a handler that does a regex against a list of thousands of domains on every request adds latency to every page. A set lookup on the hostname costs almost nothing. Anything heavier belongs outside the hot path.
And block service workers on the context if the target registers one. Otherwise requests the service worker makes on the page’s behalf skip your routes entirely, and the weight breakdown shows traffic your filter never had a chance to see.
Failure modes and how they show up
A few failures are common enough to name, because each has a signature you learn to recognise in monitoring.
The empty list. Records arrive with titles and prices but an empty variants array, or a category page returns the first twelve items instead of forty-eight. Cause, almost always: a lazy loader waiting on an image load event or an intersection observer that never fires. The completeness check shows a field whose average length dropped, not a field that went null.
The hang. Jobs that used to finish in a predictable time start hitting their timeout on a click or a wait. Cause: a blocked stylesheet or script changed layout or removed an element the flow waits for. Playwright’s trace viewer, recorded on a failing page, shows the action retrying against an element that isn’t actionable.
The banner wall. Every page extracts the same text, which turns out to be a cookie consent message. Cause: a blocked tag manager or consent script left the overlay in place with no script to dismiss it, or the content never loaded because it waited for a consent decision.
The phantom saving. The filter is in place, the per-page breakdown looks light, but the proxy bill didn’t move. Usual causes: a service worker fetching assets outside the routes, a video player streaming segments as fetch traffic, or the cache loss from enabling routing making shared scripts refetch on every page and eating the saving.
Each of these is caught by the same pair of habits: a field-level diff against a known-good baseline whenever the filter changes, and a bandwidth reading per credential after each run.
Keeping the filter honest after it ships
A filter that was safe on the day you tested it isn’t safe forever. Sites redesign and swap vendors. They move APIs and add widgets. Each change can turn a harmless block into one that removes data, or turn a lean filter into one that lets a new heavy asset through.
Keep the baseline sample and rerun the unfiltered-versus-filtered diff on a schedule. Monthly is reasonable for stable targets. Weekly makes sense for targets that redesign often or where a missing field has real downstream cost. The run costs the bandwidth of one unfiltered sample, which is small next to a full crawl, and it catches a broken dependency before it has quietly emptied a month of records.
Alert on two numbers per target. One is bytes per page, from the requestfinished sums, which catches new weight sneaking through. The other is field completeness, which catches blocks that started removing data. Either moving by more than its usual variation is worth a look.
When the diff does show a difference, resist the urge to loosen the whole policy. Find the one request that matters, allow or fulfill just that one, and rerun. The goal is a filter that’s as tight as the target allows, and loosening everything because one lazy loader broke throws away most of the saving.
Finally, write down why each exception exists. “Allow cdn-x.example because variant data loads from it” is a comment that saves an hour the next time someone tidies the config and removes what looks like an unnecessary entry.
What to do on Monday
If you’re running Playwright through a metered proxy today with no filter, here’s the order we’d do it in.
Pick one target, the one with the highest proxy spend. Build a baseline sample of twenty or so pages that covers its templates, render them unfiltered, and save the extracted records alongside a per-request log with resource type, host and body size.
Add a context-level route that aborts image, media and font types. Rerun the sample and diff the records, then compare the weight breakdown. If the diff is clean, ship that policy for this target and read the proxy credential’s usage on the next real run to confirm the bill moved.
Look at the weight breakdown for the largest remaining third-party domains. Add them to the block list one at a time, rerunning the diff after each. Stop when the remaining domains are small or when the next one breaks something.
Leave stylesheets and first-party scripts alone unless the numbers make a strong case. Never touch scripts the site uses to decide whether to serve you.
Move the policy into a shared module with per-target overrides, block service workers where the target uses them, and schedule the baseline diff to rerun monthly. Then do the next most expensive target.
Most of the saving comes from the first two steps. The rest is keeping it from rotting.
For more breakdowns on rendering costs, proxy spend and the infrastructure around production collection, the rest of the guides live at Data Research Tools.
Get new guides and videos first — join the Telegram channel.
Leave a Reply