Headless Browser vs HTTP Client: When You Actually Need Playwright

Most scrapers reach for a headless browser by default, and it is the most expensive habit in this work. They spin up a full Chromium for a page that would have handed the same data to a plain http request, then pay for that browser on every page, forever. The decision that matters isn’t which browser tool to use. It’s whether you need a browser at all, and most of the time you don’t.

I run production scrapers, so I’ve paid the browser bill when I didn’t have to and watched it show up in the monthly invoice. This is how I decide, where the real cost hides, and the hybrid I run so a job stays cheap where it can and only pays for a browser where a site forces one.

The one test that decides it

Open the page in your browser, view the raw source, and search that source for the thing you want to collect. If the price, the title, the number is already sitting in the html the server sent, you almost never need a browser. If the source is a near empty shell and the content only fills in after the scripts run, then you need something that executes those scripts.

That one look at the raw source settles the large majority of these calls, and it takes about ten seconds. Everything below is really just the consequences of getting that answer right or wrong.

What an http client does, and what it costs

An http client is the simple side. It opens a connection, sends a request, and hands you back exactly the bytes the server returned. It doesn’t run javascript, it doesn’t build a page, it doesn’t know what a button is. It’s just the raw response.

That’s why it’s so light. There’s no browser eating memory, no rendering, no waiting for scripts, so one modest machine can run hundreds of these requests at once. When the data lives in the server html, an http client is the whole job, and it’s fast and cheap. Libraries like requests and httpx do this, and a framework like Scrapy is built on the same idea at scale.

What a headless browser does, and what it costs

A headless browser is the heavy side. It launches a real Chromium engine with no window drawn on screen, loads the page the way a person’s browser would, runs all the javascript, builds the dom, and lets you read the page after it has fully rendered. Playwright, Puppeteer, and Selenium all drive a browser like this.

That full render is the entire reason it exists, and it’s the entire reason it costs so much. A raw http request costs a few milliseconds and a sliver of memory. A full browser page costs hundreds of megabytes of memory, real cpu, and seconds instead of milliseconds.

The gap in real numbers

The difference isn’t small. Per page, a browser routinely costs ten to fifty times the resources of a raw request, and that gap compounds at scale.

Put a job on it. A million pages with a raw http client on a modest server finishes in hours for a few dollars. The same million pages with a full browser each can need a fleet of machines and a bill many times larger, plus the constant work of keeping all those browsers healthy.

Check for a background api first

Before you assume a page needs a browser, open the developer tools and watch the network tab while the page loads. Very often the content you want arrives as a clean json response from an api the page quietly calls in the background.

When that’s happening, you don’t need to render anything. You call that json endpoint directly with a plain http request and skip the browser entirely. This one habit has saved me more compute than any tool choice, because the cheapest page to collect is the one you never render.

The hybrid: http first, browser only where forced

In practice it’s rarely all browser or all client. I run a fast http client across the whole site for the pages that hand over their data plainly, and I fall back to a browser only for the specific pages that come up empty. So the heavy cost lands on the handful of pages that genuinely need a render, and everything else stays cheap.

A browser really is the only door in a few cases: when the content is rendered on the client and never appears in the raw source, when the data hides behind an interaction like a click or an infinite scroll, or when the page is a single page app that builds itself entirely from javascript. In those cases an http client sees the empty shell and there’s nothing to parse. The rule I follow is simple: crawl wide and cheap with raw requests, render narrow and expensive only where the html arrives empty.

If you do land on a browser, the tool matters less than people think. Playwright and Puppeteer both drive Chromium cleanly, Selenium works but feels older, and I default to Playwright. The expensive part is never which one you pick, it’s the browser itself.

The maintenance burden of browsers at scale

The cost that doesn’t show up in a benchmark shows up at three in the morning. An http client is boring in the best way: it breaks when a site changes its html, and the fix is a quick selector update.

A browser breaks in far more ways. Chromium updates and a flag you relied on changes. A page gets slower and a timing assumption that held last month starts flaking. Memory leaks creep in over a long run until the process bloats and dies. Running browsers at scale becomes its own operations job: managing a pool of them, restarting them before they leak, killing the ones that hang, catching zombie processes, pinning a browser version, and wrapping every page in a timeout so it can’t hang forever. None of that exists on the http side.

There’s a subtler tax too. An http response either arrives or it doesn’t, so you know when you’re done. A browser page is never obviously finished, because scripts keep firing and content keeps arriving, so you write waits, and every wait is time you spend on every single page.

The detection angle, honestly

People ask which side is harder to detect, and it’s mostly the wrong question. Sites decide on the network you arrive from, the handshake your client sends, and the rhythm of your behavior, and both a client and a browser can be loud or quiet depending on how you run them. A real browser engine does produce a more browser like fingerprint, which is one honest reason browser automation exists.

But none of that makes anything undetectable, and this isn’t a guide to evading anything. It cuts the other way too: a headless browser in its default mode leaves its own tells, from the automation flags it sets to the too clean, repeatable shape of its fingerprint. A browser isn’t automatically safer than a client, it just fails in different places. The durable move on either side is to collect public data at a polite rate, respect the robots file, prefer an official api, and stay off the adversarial path entirely.

How I decide, every time

The ladder is short. View the raw source and check whether the data is already in the server html. If it is, use a plain http client and stop there. If it isn’t, open the network tab and look for a background json api you can call directly, and if it exists, use that. Only if both of those fail, when the data truly only exists after the browser renders it, reach for a headless browser, and even then render only the pages that need it.

Whichever side you land on, validate what comes back. The failure that hurts most is the one that doesn’t throw an error, where a client gets a soft block page or a browser reads the dom too early and hands you rows full of nulls. Confirm the fields you expect are present and in a sane range, and fail loudly when they aren’t.

I run this exact setup in production, so the frameworks and tradeoffs here are the ones I actually deal with, not theory. If you want the full written guides, the hybrid configs, and the picks I use and test, read them at dataresearchtools.com. No undetectable promises, no guaranteed results, just the cheapest tool that gets the data.

Get new guides and videos first — join the Telegram channel.