User-Agent Rotation: Best Practices for Web Scraping

By /

User-Agent Rotation: Best Practices for Web Scraping

The User-Agent header is the first thing anti-bot systems check. Sending the same user agent across thousands of requests — or worse, sending Python’s default python-requests/2.31.0 — is the fastest way to get blocked.

But user-agent rotation isn’t just about having a list of random strings. Done wrong, it creates new detection signals. This guide covers how to rotate user agents correctly in 2026.

Why User-Agent Rotation Matters

Every HTTP request includes a User-Agent header that identifies the client software. Browsers send detailed UA strings:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Anti-bot systems use user agents for:

  1. Bot identification — Non-browser UAs are blocked immediately
  2. Consistency checking — UA should match other signals (TLS fingerprint, headers, behavior)
  3. Frequency analysis — Same UA across thousands of requests = bot
  4. Version analysis — Outdated browser versions are suspicious

Building a User-Agent Pool

Current Browser User Agents (March 2026)

USER_AGENTS = {
    "chrome_windows": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    ],
    "chrome_mac": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36",
    ],
    "chrome_linux": [
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
    ],
    "firefox_windows": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0",
    ],
    "firefox_mac": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:123.0) Gecko/20100101 Firefox/123.0",
    ],
    "safari_mac": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.3 Safari/605.1.15",
    ],
    "edge_windows": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0",
    ],
    "chrome_mobile": [
        "Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Mobile Safari/537.36",
        "Mozilla/5.0 (iPhone; CPU iPhone OS 17_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/122.0.6261.62 Mobile/15E148 Safari/604.1",
    ],
}

Weighted Distribution

Real-world browser market share isn’t uniform. Your rotation should reflect actual usage patterns:

import random

# Approximate browser market share (desktop, 2026)
UA_WEIGHTS = {
    "chrome_windows": 45,
    "chrome_mac": 15,
    "chrome_linux": 3,
    "firefox_windows": 8,
    "firefox_mac": 3,
    "safari_mac": 12,
    "edge_windows": 10,
    "chrome_mobile": 4,  # Include some mobile
}

def get_weighted_user_agent():
    """Select a user agent based on real market share."""
    categories = list(UA_WEIGHTS.keys())
    weights = list(UA_WEIGHTS.values())

    category = random.choices(categories, weights=weights, k=1)[0]
    return random.choice(USER_AGENTS[category]), category

Matching User Agent to Other Signals

This is the critical part most guides miss. Your user agent must be consistent with everything else about your request.

Matching TLS Fingerprint

If your UA says Chrome but your TLS fingerprint says Python, you’re instantly detected:

from curl_cffi import requests

# CORRECT: TLS fingerprint matches user agent
session = requests.Session(impersonate="chrome120")
# curl_cffi automatically sets matching UA, but you can override:
session.headers["User-Agent"] = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36"
)

# WRONG: Chrome UA with Python TLS fingerprint
import requests as plain_requests
plain_requests.get(url, headers={
    "User-Agent": "Mozilla/5.0 ... Chrome/120.0.0.0 ..."
})
# Server sees: Chrome UA + Python TLS = FAKE

Matching Headers

Chrome, Firefox, and Safari send different header sets. If your UA says Firefox but you’re sending Chrome-specific Sec-Fetch-* headers, that’s a mismatch.

CHROME_HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Sec-Ch-Ua": '"Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
    "Upgrade-Insecure-Requests": "1",
}

FIREFOX_HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    # Note: Firefox doesn't send Sec-Ch-Ua headers
}

Matching Platform

If your UA says Windows Chrome, your navigator.platform should be “Win32”, not “Linux”:

# For browser automation
PLATFORM_MAP = {
    "chrome_windows": "Win32",
    "chrome_mac": "MacIntel",
    "chrome_linux": "Linux x86_64",
    "firefox_windows": "Win32",
    "firefox_mac": "MacIntel",
    "safari_mac": "MacIntel",
}

Complete Rotation Implementation

import random
from curl_cffi import requests

class UserAgentRotator:
    def __init__(self):
        self.profiles = self._build_profiles()

    def _build_profiles(self):
        return [
            {
                "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
                "impersonate": "chrome120",
                "headers": {
                    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
                    "Accept-Language": "en-US,en;q=0.9",
                    "Sec-Ch-Ua": '"Chromium";v="122", "Google Chrome";v="122"',
                    "Sec-Ch-Ua-Mobile": "?0",
                    "Sec-Ch-Ua-Platform": '"Windows"',
                    "Sec-Fetch-Dest": "document",
                    "Sec-Fetch-Mode": "navigate",
                    "Sec-Fetch-Site": "none",
                    "Upgrade-Insecure-Requests": "1",
                },
                "weight": 45,
            },
            {
                "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
                "impersonate": "chrome120",
                "headers": {
                    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
                    "Accept-Language": "en-US,en;q=0.9",
                    "Sec-Ch-Ua": '"Chromium";v="122", "Google Chrome";v="122"',
                    "Sec-Ch-Ua-Mobile": "?0",
                    "Sec-Ch-Ua-Platform": '"macOS"',
                    "Sec-Fetch-Dest": "document",
                    "Sec-Fetch-Mode": "navigate",
                    "Sec-Fetch-Site": "none",
                    "Upgrade-Insecure-Requests": "1",
                },
                "weight": 25,
            },
            {
                "ua": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0",
                "impersonate": "chrome120",  # Firefox TLS not always available
                "headers": {
                    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
                    "Accept-Language": "en-US,en;q=0.5",
                    "Sec-Fetch-Dest": "document",
                    "Sec-Fetch-Mode": "navigate",
                    "Sec-Fetch-Site": "none",
                    "Upgrade-Insecure-Requests": "1",
                },
                "weight": 15,
            },
        ]

    def get_session(self, proxy=None):
        """Create a session with a consistent profile."""
        weights = [p["weight"] for p in self.profiles]
        profile = random.choices(self.profiles, weights=weights, k=1)[0]

        session = requests.Session(impersonate=profile["impersonate"])
        session.headers.update(profile["headers"])
        session.headers["User-Agent"] = profile["ua"]

        if proxy:
            session.proxies = {
                "http": proxy,
                "https": proxy
            }

        return session

# Usage
rotator = UserAgentRotator()

for url in urls:
    session = rotator.get_session(
        proxy="http://user:pass@residential.example.com:7777"
    )
    response = session.get(url)
    print(f"{url}: {response.status_code}")

Common Rotation Mistakes

Mistake 1: Using Outdated UAs

# BAD: This UA is from 2020
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/85.0.4183.121 Safari/537.36"

# GOOD: Recent version
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/122.0.0.0 Safari/537.36"

Anti-bot systems know that virtually no real users run Chrome 85 in 2026.

Mistake 2: Random UA per Request

# BAD: Different UA every single request
for url in urls:
    headers["User-Agent"] = random.choice(ua_list)
    session.get(url, headers=headers)

# GOOD: Same UA per session, rotate between sessions
session = rotator.get_session()
for url in batch:
    session.get(url)  # Consistent UA within session

Real users don’t change their browser between page loads.

Mistake 3: Mixing Desktop and Mobile

# BAD: Switching between desktop and mobile in the same session
requests_data = [
    {"ua": "... Chrome/122.0.0.0 Safari/537.36", "url": "/page/1"},
    {"ua": "... Mobile Safari/537.36", "url": "/page/2"},  # Suddenly mobile?
]

Mistake 4: Ignoring the Sec-Ch-Ua Header

Modern Chrome sends Sec-Ch-Ua which must match the Chrome version in the User-Agent:

# User-Agent says Chrome 122
ua = "... Chrome/122.0.0.0 Safari/537.36"

# Sec-Ch-Ua MUST also say 122
sec_ch_ua = '"Chromium";v="122", "Google Chrome";v="122", "Not(A:Brand";v="24"'

Keeping UAs Updated

Browser versions update every 4-6 weeks. Your UA pool needs regular updates:

def generate_chrome_ua(version, platform="windows"):
    """Generate a Chrome UA for a specific version."""
    platforms = {
        "windows": "Windows NT 10.0; Win64; x64",
        "mac": "Macintosh; Intel Mac OS X 10_15_7",
        "linux": "X11; Linux x86_64",
    }

    return (
        f"Mozilla/5.0 ({platforms[platform]}) "
        f"AppleWebKit/537.36 (KHTML, like Gecko) "
        f"Chrome/{version}.0.0.0 Safari/537.36"
    )

# Generate UAs for recent Chrome versions
for version in [120, 121, 122]:
    for platform in ["windows", "mac", "linux"]:
        ua = generate_chrome_ua(version, platform)
        print(ua)

FAQ

How many user agents do I need in my rotation pool?

For most scraping, 10-20 well-configured user agents (with matching headers and TLS fingerprints) are sufficient. Quality matters more than quantity — 5 properly matched Chrome profiles outperform 100 random UA strings. Focus on the 3-4 most common browsers with their 2-3 most recent versions.

Should I include mobile user agents?

Include mobile UAs only if your target site serves different content to mobile users that you need to scrape. Otherwise, stick to desktop UAs since they’re simpler to configure consistently. If you do use mobile UAs, ensure your viewport, touch support, and device memory values match a mobile device.

How often should I rotate my user agent?

Rotate between sessions or batches, not between individual requests. A real user doesn’t change their browser mid-session. Use the same UA for 50-200 requests (matching a realistic browsing session), then switch to a different profile for the next batch.

Does user-agent rotation alone prevent blocking?

No. User-agent rotation is necessary but not sufficient. You also need IP rotation, proper TLS fingerprinting, rate limiting, and ideally browser fingerprint management. Think of UA rotation as one layer of defense.

Can websites detect UA rotation?

Yes. If you rotate UAs but other signals (IP, TLS fingerprint, cookies, behavior) remain constant, the rotation itself becomes a detection signal. Always rotate UAs as part of a complete profile change (new IP, new session, matching headers).

Conclusion

User-agent rotation is a fundamental scraping technique, but it must be done with consistency in mind. Every UA you send should be paired with matching TLS fingerprints, headers, and platform signals. Use weighted distributions based on real browser market share, keep your UA pool current, and rotate at the session level rather than per-request.

Useful Resources

Related Reading

Scroll to Top