Header rotation and TLS profiles for production scrapers

Header rotation and TLS profiles are the two halves of looking like a real browser at the network layer. Either one alone is detectable. Header rotation without TLS alignment ships Chrome-style headers over a Python TLS handshake, which is an obvious mismatch. TLS impersonation without header alignment ships a perfect Chrome ClientHello followed by Python’s idiosyncratic header order, which is also obvious. The two must move together for a scraper to look genuinely like a browser to enterprise bot detection.

This guide covers what real Chrome and real Firefox headers look like in 2026, how to align headers with TLS profiles, common rotation patterns, and the production code that ties it all together. Everything below targets curl_cffi and tls-client because those are the two libraries that handle both surfaces, but the principles apply to any scraping stack.

Why headers and TLS must align

Bot detection vendors compute TLS fingerprints (JA4) at the connection layer and header fingerprints (header order, presence of specific headers, casing) at the request layer. The vendor’s risk model checks consistency across these signals: a Chrome 124 JA4 with Chrome-style headers in the right order produces low risk. A Chrome 124 JA4 with Python-style headers (different order, missing headers, extra headers) produces high risk because the inconsistency is itself anomalous.

What changes between real browsers:

A scraper using Chrome TLS impersonation must also ship Chrome’s exact header order. A Firefox-impersonating scraper needs Firefox’s headers. Mixing them creates a third profile that matches no real browser, which is the worst of both worlds.

For the IETF reference on HTTP semantics, see RFC 9110, which defines what headers mean but not what order they appear in. Order is implementation-specific, which is exactly why it is fingerprintable.

What real Chrome 124 headers look like

A captured request from Chrome 124 stable to a public site:

GET /products HTTP/2
Host: example.com
sec-ch-ua: "Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
sec-fetch-site: none
sec-fetch-mode: navigate
sec-fetch-user: ?1
sec-fetch-dest: document
accept-encoding: gzip, deflate, br, zstd
accept-language: en-US,en;q=0.9
priority: u=0, i

Critical observations:

1. All headers are lowercase (HTTP/2 requires lowercase pseudo-headers and Chrome lowercases everything else)
2. sec-ch-ua group comes before user-agent
3. accept comes after user-agent
4. sec-fetch-* group comes after accept
5. accept-encoding includes zstd (Chrome 124+)
6. priority header is present (Chrome 124 uses RFC 9218 signaling)

A subsequent same-origin navigation has slightly different sec-fetch-* values:

sec-fetch-site: same-origin
sec-fetch-mode: navigate
sec-fetch-user: ?1
sec-fetch-dest: document
referer: https://example.com/

For an API call (XHR/fetch from page JavaScript):

sec-fetch-site: same-origin
sec-fetch-mode: cors
sec-fetch-dest: empty
accept: */*
accept-language: en-US,en;q=0.9
content-type: application/json

These contextual differences are themselves fingerprinted. A scraper that ships sec-fetch-mode: navigate for an API endpoint is anomalous.

What real Firefox 124 headers look like

Firefox 124 ships headers in a noticeably different shape:

GET /products HTTP/2
Host: example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; rv:124.0) Gecko/20100101 Firefox/124.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
priority: u=0, i

Key differences from Chrome:

• user-agent comes first (after Host)
• No sec-ch-ua-* headers (Firefox does not implement Client Hints)
• accept-language uses q=0.5 (Chrome uses q=0.9)
• accept-encoding does not include zstd (Firefox added it in 126)
• Header casing is preserved as-sent (lowercase in HTTP/2)

A scraper claiming to be Firefox must ship these specific headers in this order. Chrome-style sec-ch-ua headers from a Firefox profile is a flag.

What real Safari 17 headers look like

Safari is more conservative:

GET /products HTTP/2
Host: example.com
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
sec-fetch-site: none
sec-fetch-dest: document
accept-language: en-US,en;q=0.9
sec-fetch-mode: navigate
accept-encoding: gzip, deflate, br
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15

Differences from Chrome:

• accept is shorter (no image/avif, no application/signed-exchange)
• No sec-ch-ua-*
• No upgrade-insecure-requests
• No priority
• user-agent comes after accept and sec-fetch-* headers

Safari mobile (iOS) is shorter still:

accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-language: en-US,en;q=0.9
accept-encoding: gzip, deflate, br
user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1

Each browser’s header set is distinct enough to fingerprint independently of TLS. Match them.

Header rotation strategies

Two patterns work in production:

Pattern 1: profile pool. Maintain a pool of complete browser profiles (Chrome 122, Chrome 124, Firefox 124, Safari 17, Edge 124). Each profile has a matched TLS impersonation, header set, and User-Agent. Rotate across the pool by request.

import random
from curl_cffi import requests

PROFILES = [
    {
        "tls": "chrome124",
        "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
              "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "sec_ch_ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
        "platform": "Windows",
    },
    {
        "tls": "chrome124",
        "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
              "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "sec_ch_ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
        "platform": "macOS",
    },
    {
        "tls": "firefox124",
        "ua": "Mozilla/5.0 (Windows NT 10.0; rv:124.0) Gecko/20100101 Firefox/124.0",
        "sec_ch_ua": None,  # Firefox does not send this
        "platform": "Windows",
    },
    {
        "tls": "safari17",
        "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
              "(KHTML, like Gecko) Version/17.4 Safari/605.1.15",
        "sec_ch_ua": None,
        "platform": "macOS",
    },
]

def build_headers(profile, url):
    h = {}
    if profile["sec_ch_ua"]:
        h["sec-ch-ua"] = profile["sec_ch_ua"]
        h["sec-ch-ua-mobile"] = "?0"
        h["sec-ch-ua-platform"] = f'"{profile["platform"]}"'
    h["user-agent"] = profile["ua"]
    h["accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
    h["sec-fetch-site"] = "none"
    h["sec-fetch-mode"] = "navigate"
    h["sec-fetch-dest"] = "document"
    h["accept-encoding"] = "gzip, deflate, br, zstd" if "Chrome" in profile["ua"] else "gzip, deflate, br"
    h["accept-language"] = "en-US,en;q=0.9"
    return h

def fetch_with_random_profile(url, proxies=None):
    profile = random.choice(PROFILES)
    headers = build_headers(profile, url)
    return requests.get(url, headers=headers, impersonate=profile["tls"], proxies=proxies)

Pattern 2: stable profile per session. Pick a profile when you start a scraping session and stick with it for the duration. This is more realistic because a single user does not switch browsers mid-session.

class ScraperSession:
    def __init__(self, profile=None, proxy=None):
        self.profile = profile or random.choice(PROFILES)
        self.proxy = proxy
        self.cookies = {}

    def fetch(self, url, **kwargs):
        headers = build_headers(self.profile, url)
        headers.update(kwargs.get("headers", {}))
        return requests.get(
            url,
            headers=headers,
            impersonate=self.profile["tls"],
            proxies={"https": self.proxy} if self.proxy else None,
            cookies=self.cookies,
        )

For most scraping, pattern 2 is more authentic. Per-request profile rotation creates an unusual session shape (one user, multiple browsers).

Header order matters more than header values

Most scrapers focus on header values (User-Agent, Accept, etc.) and ignore order. Bot detection vendors increasingly check order because order is harder to fake.

Default Python requests produces this order:

User-Agent
Accept-Encoding
Accept
Connection

Default Chrome:

sec-ch-ua
sec-ch-ua-mobile
sec-ch-ua-platform
upgrade-insecure-requests
user-agent
accept
sec-fetch-site
sec-fetch-mode
sec-fetch-user
sec-fetch-dest
accept-encoding
accept-language
priority

The orders are completely different. Even if you set every Chrome header in your requests call, requests sorts them alphabetically before sending, breaking the fingerprint.

curl_cffi and tls-client both preserve header insertion order by default. Use them to control order. In curl_cffi:

from curl_cffi import requests

# Headers are sent in the order you provide them
headers = [
    ("sec-ch-ua", '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"'),
    ("sec-ch-ua-mobile", "?0"),
    ("sec-ch-ua-platform", '"Windows"'),
    ("upgrade-insecure-requests", "1"),
    ("user-agent", "Mozilla/5.0 ..."),
    ("accept", "text/html,..."),
    ("sec-fetch-site", "none"),
    ("sec-fetch-mode", "navigate"),
    ("sec-fetch-user", "?1"),
    ("sec-fetch-dest", "document"),
    ("accept-encoding", "gzip, deflate, br, zstd"),
    ("accept-language", "en-US,en;q=0.9"),
    ("priority", "u=0, i"),
]

resp = requests.get(url, headers=dict(headers), impersonate="chrome124")

curl_cffi preserves Python dict insertion order (Python 3.7+ dicts are ordered) when shipping headers. Verify with a wire capture or with tls.peet.ws’s http_headers field.

Comparison: header sets across browsers

For a full request, what each browser ships:

Match every header to the claimed browser. Missing a header that the browser sends is a flag, sending one that the browser does not is also a flag.

Validating your headers

Public sites that show what headers you sent:

Check that your scraper’s output at httpbin.org/headers matches what a real Chrome shows when visiting the same site. If your scraper’s headers differ in order or set, fix them.

Production header refresh cycle

Real browsers ship updates every 4-6 weeks. Each release can change:

• User-Agent string
• Sec-CH-UA brand list
• Accept-Encoding (e.g., adding zstd)
• Accept value structure
• Priority signaling

Your scraper’s profile pool needs the same refresh cadence. Plan a quarterly review:

1. Pull latest stable Chrome, Firefox, Safari User-Agents from a real install or from useragents.io
2. Capture latest header set from each browser via mitmproxy or DevTools
3. Update profile definitions
4. Verify with tls.peet.ws and httpbin.org/headers
5. Run regression tests against your top 20 target sites
6. Roll out the new profile pool

Without this cycle, your scraper drifts: claiming to be Chrome 122 when Chrome is on 130 means the User-Agent is itself anomalous, even with perfect TLS.

For broader scraping infrastructure patterns, see building a custom rotating proxy pool with Squid and self-hosted proxy infrastructure.

Common header mistakes

• Setting User-Agent only: leaves all other headers as Python defaults, easy to detect
• Wrong Accept value: Chrome’s Accept is distinctive; Python’s default is bare */*
• Including X-Forwarded-For unless you really need to: trips proxy detection
• Missing sec-fetch-*: real browsers always send these for navigations
• Sec-CH-UA on a Firefox-claimed UA: only Chrome and Edge send this
• Priority header on Safari claim: Safari does not send this
• HTTP/1.1-style Connection: keep-alive in HTTP/2 requests: HTTP/2 has no Connection header

Header rotation for API scraping vs page scraping

API endpoints often have looser header expectations because real browser code (XHR, fetch) sends different headers than navigations. For an API call:

api_headers = {
    "user-agent": profile["ua"],
    "accept": "*/*",  # XHR default
    "accept-language": "en-US,en;q=0.9",
    "accept-encoding": "gzip, deflate, br, zstd",
    "sec-ch-ua": profile["sec_ch_ua"],
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": f'"{profile["platform"]}"',
    "sec-fetch-site": "same-origin",
    "sec-fetch-mode": "cors",
    "sec-fetch-dest": "empty",
    "referer": "https://target.example.com/",
    "origin": "https://target.example.com",
    "content-type": "application/json",  # for POST
}

Note sec-fetch-mode: cors and sec-fetch-dest: empty for XHR vs navigate and document for page loads. Match the header set to the request type.

Operational checklist

• Use curl_cffi or tls-client (libraries that preserve header order)
• Maintain a profile pool with TLS + header set per profile
• Match TLS impersonation to claimed User-Agent
• Validate header order with tls.peet.ws or wire capture
• Refresh profiles quarterly with current browser versions
• Use page-style headers for navigations, XHR-style for APIs
• Set Origin and Referer correctly for cross-origin POSTs
• Avoid sending headers that real browsers do not send (X-Forwarded-For, X-Real-IP, custom defaults from your library)
• Verify against httpbin.org/headers in CI

FAQ

Q: do I need to match every header exactly?
The major signals (User-Agent, Sec-CH-UA presence, Accept value, Accept-Encoding, header order) matter most. Minor details (specific quality values in Accept-Language) matter less but cumulatively add up.

Q: how often do real browsers change headers?
Major changes (new headers, removed headers) happen every few major versions. Minor changes (User-Agent string, brand list) happen every release. Plan to refresh quarterly to stay current.

Q: can I use a single User-Agent for all my scrapers?
Within a session yes, across sessions no. Vendors fingerprint repeated User-Agents from the same IP space and treat them as a coordinated bot fleet. Rotate User-Agent across sessions but keep it stable within one.

Q: does header casing matter?
In HTTP/1.1 servers are case-insensitive but capture original casing. Chrome lowercases all custom headers. Capitalize-Each-Word style is a Python requests default that flags scrapers. In HTTP/2 lowercase is required.

Q: what about cookies?
Cookies are headers but with their own logic. Manage them via session cookie jars rather than as raw headers. The order is enforced by the cookie jar, not by your code.

Common pitfalls in production header alignment

The first failure mode is the priority header value mismatch. Chrome 124 sends priority: u=0, i for top-level navigations and priority: u=1, i for subresources, but Chrome 126+ stable started omitting the i parameter in some configurations. If you pin a Chrome 124 profile but your scraping fleet visits sites with strict server-push HTTP/2 deployments, the priority value gets compared against the User-Agent’s expected behavior. A scraper claiming Chrome 126 with u=0, i is anomalous because real Chrome 126 sends u=0 only. Update the priority value when you bump the profile’s claimed Chrome version.

The second pitfall is the accept-encoding order on Brotli vs zstd handshakes. Chrome 124 advertises gzip, deflate, br, zstd in that exact order. If your library reorders to gzip, deflate, zstd, br (a common bug in older curl_cffi releases), Cloudflare’s content-encoding negotiation logs the alphabetical order as anomalous because alphabetical-sort is what Python requests produces by default. Servers respond identically (they pick br or zstd regardless of order), but the fingerprint differs. Verify with a wire capture that your accept-encoding string matches Chrome byte-for-byte, including the spaces after commas.

The third pitfall is referer policy mismatch on cross-origin POSTs. Chrome 124 honors a Referrer-Policy: strict-origin-when-cross-origin default, which means a POST from https://app.example.com/checkout to https://api.example.com/v1/charge ships referer: https://app.example.com/ (origin only, no path). A scraper that hardcodes the full URL as referer (referer: https://app.example.com/checkout) violates the policy that real Chrome would have applied, which is itself a flag for vendors that compute the expected referer from the page URL plus the policy. Compute referer dynamically based on the claimed origin policy, not by copying the page URL verbatim.

Real-world example: alignment-driven recovery on Akamai

A scraper team running a 40-node Playwright fleet against an Akamai-protected airline booking site experienced a sudden block-rate jump from 8 percent to 71 percent over 48 hours with no code changes. The culprit was a transparent proxy upgrade upstream that started rewriting the accept-language header from en-US,en;q=0.9 to en-US,en;q=0.9,en-CA;q=0.8 (the proxy added a regional fallback). Chrome 124 never sends en-CA, so the modified header diverged from any plausible Chrome profile. Akamai’s header-shape model flagged it within hours of the proxy rollout.

The fix involved two parts: bypass the upstream proxy for header-sensitive requests and add a CI check that captures outbound headers via a passive sniffer and diffs them against the canonical Chrome reference:

import json
import subprocess

CANONICAL_CHROME_124 = {
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "accept-encoding": "gzip, deflate, br, zstd",
    "accept-language": "en-US,en;q=0.9",
    "sec-ch-ua-platform": '"Windows"',
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "none",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
}

def diff_headers(actual: dict, canonical: dict) -> dict:
    diffs = {}
    for k, v in canonical.items():
        if actual.get(k) != v:
            diffs[k] = {"expected": v, "actual": actual.get(k)}
    return diffs

# In CI: run scraper against httpbin, capture, diff
result = subprocess.check_output(
    ["python", "scrape_one.py", "https://httpbin.org/headers"]
)
captured = json.loads(result)["headers"]
diffs = diff_headers({k.lower(): v for k, v in captured.items()}, CANONICAL_CHROME_124)
assert not diffs, f"Header drift: {json.dumps(diffs, indent=2)}"

After deployment of the CI check, the team caught two more upstream-proxy-induced drifts within the next quarter before they reached production scrapers. The lesson: header alignment is not a one-time setup, it is an ongoing surveillance task because anything between your code and the wire can rewrite headers without telling you.

Comparison: header order across libraries

A wire-capture comparison of how each Python HTTP client orders the headers you provide:

For full control, use curl_cffi or tls-client with explicit ordering. For zero effort, use Playwright’s page.request which matches the launched browser. Anything else introduces unpredictable order that requires per-library workarounds.

Detection in production logs: header-shape correlation

When you suspect a target is fingerprinting headers, you can confirm by correlating block rate against header changes. Log every outbound header set with a stable hash:

import hashlib
import json

def header_shape_hash(headers: dict) -> str:
    # Hash on the ordered keys, not the values, to capture shape
    keys_in_order = list(headers.keys())
    return hashlib.sha256(json.dumps(keys_in_order).encode()).hexdigest()[:8]

def log_request(url: str, headers: dict, status: int):
    shape = header_shape_hash(headers)
    print(json.dumps({
        "url": url,
        "header_shape": shape,
        "status": status,
    }))

Aggregate by header_shape over a 24h window. If one shape has a 90 percent success rate and another has a 30 percent success rate, the difference is your fingerprint. Either pin the high-success shape or investigate why the low-success shape is leaking. This kind of shape-vs-status correlation is invisible without the structured logging.

Wrapping up

Header rotation and TLS profiles are two halves of the same problem. Get them aligned and your scraper looks like a real browser at the network layer. Get them mismatched and you broadcast “Python pretending to be Chrome” to every modern bot detector. The fix is a profile pool, a library that preserves header order (curl_cffi, tls-client), quarterly profile refreshes, and validation in CI. Pair this with our TLS fingerprinting guide and HTTP/2 fingerprinting writeups for the full network-layer picture, and browse the anti-detect-browsers category on DRT for related tactics.