Headless vs headed browser runs: why the same scraper gives you different output

The same script, two different outputs

Anyone who has moved a scraper from a laptop into a production container has hit this. The script works perfectly in headed Chrome, while you’re watching it click through pages. Then it ships to a headless environment and comes back with empty fields, truncated content, or an outright block page. Nothing in the code changed. The environment did. That’s usually enough.

Most of the time, it’s a mix of rendering differences, timing differences, and the fact that headless and headed Chrome present different fingerprints to any page that bothers to look. Your selectors are rarely the actual problem. Understanding where those differences come from makes debugging a lot faster. It also explains why bot detection vendors bother checking for headless at all.

What “headless” actually changes under the hood

Headless just means Chrome runs without a visible window and without a compositor drawing pixels to a screen. Since Chrome 109 (January 2023), Google shipped “headless=new.” It runs the same Blink rendering engine and V8 JavaScript engine as the full browser, through the same code paths. That closed a lot of gaps that existed in the old headless mode, which was a separate, more stripped-down binary target with its own quirks.

Before that switch, old headless Chrome had visible tells. Its user agent string literally contained “HeadlessChrome.” Its plugin and MIME type arrays were empty, and several DOM APIs behaved differently than in a real browser window. New headless mode fixed the obvious ones, but it didn’t make headless and headed identical.

It made them closer.

Signals sites use to tell the two apart

Bot detection scripts don’t rely on one signal. They correlate several. Worth knowing what they’re actually checking, purely so you understand why your output changes.

navigator.webdriver is the most direct one. Automation frameworks like Selenium, Playwright, and Puppeteer set this to true by default. It has nothing to do with headless vs headed specifically, it flags automated control either way. But it’s usually checked alongside headless indicators.

window.chrome and permissions.query round out the browser-object checks: real Chrome exposes a window.chrome object and returns specific values from navigator.permissions.query for things like notifications. Automated or headless contexts have historically diverged here, though new headless narrowed this too.

WebGL and canvas renderer strings are one of the more durable signals, because they reflect actual hardware rather than a spoofable header. A headless instance running in a container or VM without a GPU typically reports a software renderer like SwiftShader or llvmpipe instead of a real GPU name. Any site that reads WEBGL_debug_renderer_info can see this immediately.

Font enumeration is another tell. Headless environments, especially minimal Docker images, often have a much smaller font set installed than a real desktop. Font fingerprinting (measuring how text renders) picks this up directly. Screen and viewport defaults matter too, since headless runs default to a fixed viewport unless you explicitly set one, and that default doesn’t match common real-world screen resolutions.

None of these individually prove “this is a bot.” A real user with an unusual GPU or a locked-down font set exists too. Detection systems weigh these signals together with behavioral data like mouse movement, timing between actions, and navigation patterns. That’s why config tweaks alone don’t make automation invisible to them.

Why output differs even without any detection happening at all

This is the part that trips people up. Your output can differ between headless and headed runs on a page with zero bot defense whatsoever, purely because of rendering mechanics.

Lazy-loaded and viewport-triggered content. A lot of modern pages use IntersectionObserver or scroll listeners to load content only when it enters the viewport. If your headless run uses a small or default viewport and never scrolls, that content simply never fires. Your scraper reads a DOM that’s missing sections a human visitor would see.

Render timing. Headless runs faster because there’s no compositor painting frames to a screen. If your scraper reads the DOM based on a fixed sleep or a domcontentloaded event, rather than waiting on actual network idle or a specific element, you can read the page before async content finishes loading. In headless, that race resolves differently than it does in headed mode, where paint work adds latency.

Font substitution changing layout. If a page’s JS reads computed positions, or if you’re taking a screenshot to OCR or crop something, a missing font means the browser substitutes another one. That changes line wrapping and element positions. Text content extraction is usually unaffected. But any layout-dependent logic can break.

Fingerprint-conditional content. Some sites use canvas or WebGL fingerprints as part of A/B test bucketing or personalization, not just bot blocking. A software-rendered canvas hash in headless can land you in a different bucket than a hardware-rendered one in headed mode. You might legitimately be served a different variant of a page.

Not a block. Just a different experience.

How this shows up in a production pipeline

Running fleets of scraping jobs across proxy infrastructure, the failure pattern is usually the same. A job tested and validated headed on a dev machine ships to a headless worker in production, and starts returning partial data on a subset of targets. The instinct is to blame the proxy, or assume a block. But the first thing worth checking is whether the DOM you’re getting back is even the same shape as what you saw during testing.

The fix that actually helps is treating headless/headed parity as a thing you test for, not something you assume. Run the same job in both modes against the same target. Diff the resulting DOM or extracted fields, and log fingerprint-relevant properties (renderer string, viewport, font count) as part of your pipeline’s health telemetry. If a target starts returning different output only in headless, that’s a useful signal. Either your wait conditions are wrong, or the site is doing something conditional on the signals above.

Worth investigating before you assume the scraper logic itself is broken.

Running headed in a container is possible with a virtual display like Xvfb, and it’s a reasonable way to sanity-check output parity. But a virtual display doesn’t add a real GPU. It doesn’t change the fact that the browser is being driven by automation. It closes some rendering gaps without closing detection gaps.

What operators actually do about it

The practical fixes are boring, and mostly about consistency rather than cleverness. Pin the same Chrome version and launch flags across dev and production. Set an explicit viewport size that matches what you tested with. Install a reasonable font set in your production image, if you can, instead of relying on whatever a minimal base image ships with. Replace fixed sleeps with waits on the specific network request or DOM condition your extraction actually depends on.

That alone closes most of the “it worked headed but not headless” timing gaps.

A defensive note on bot detection

It’s worth being straightforward about what detection systems like Cloudflare, DataDome, PerimeterX/Human, and Akamai are doing. They combine headless signals, automation flags, and behavioral analysis into a risk score, and they update those checks continuously. No configuration change makes automated browsing undetectable or a proxy unbannable. Treating any vendor’s claim to the contrary at face value isn’t something we’d sign off on without seeing it tested. If you’re scraping, the safer frame is sticking to publicly available data, respecting robots.txt and a site’s terms, and using headless/headed testing to build reliable pipelines rather than to defeat a specific defense.

If you’re building or debugging a scraping pipeline and want more of this kind of infrastructure-level breakdown, from proxy behavior to fingerprinting to orchestration, you’ll find more on Data Research Tools.

Get new guides and videos first — join the Telegram channel.