How Anti-Bot Systems Detect Scrapers (Cloudflare, Akamai, PerimeterX)

The Arms Race You Are Already Part Of

If you collect data from the web, you are engaged in an arms race whether you know it or not. On one side are anti-bot vendors (Cloudflare, Akamai, HUMAN, DataDome, Kasada) with billion-dollar incentives to detect and block automated traffic. On the other side are data practitioners, SEO professionals, market researchers, and developers who need programmatic access to web data.

Understanding how these systems work is not about finding silver bullets. There are none. It is about understanding the detection layers so you can make informed infrastructure decisions that maximize your access reliability.

This guide provides a technical breakdown of each detection layer, how the major vendors implement them differently, and where the technology is heading.

Layer 1: IP Reputation

IP reputation is the first and most impactful detection layer. Before your request even reaches the website’s server, the anti-bot system has already assessed your IP and assigned it a trust score.

How IP Reputation Works

Anti-bot vendors maintain massive databases of IP addresses categorized by:

  • IP type: Data center, residential ISP, mobile carrier, VPN, Tor exit node
  • Historical behavior: Has this IP been associated with bot traffic before?
  • Abuse reports: Has this IP been reported for spam, scraping, or attacks?
  • Subnet reputation: If other IPs in the same /24 subnet have been flagged, the entire subnet gets a reduced score
  • ASN reputation: The autonomous system number identifies the ISP or hosting provider. Some ASNs are associated primarily with proxy or hosting services.

IP Type Scoring

The general trust hierarchy, from lowest to highest:

  1. Tor exit nodes: Lowest trust. Almost always challenged or blocked.
  2. Data center IPs: Low trust. Legitimate users rarely browse from data centers.
  3. VPN IPs: Low-medium trust. Known VPN exit IPs are flagged.
  4. Residential IPs: Medium-high trust. Real ISP assignments, but shared IPs from proxy providers accumulate negative reputation.
  5. Mobile carrier IPs: Highest trust. CGNAT architecture means blocking a mobile IP affects thousands of legitimate users.

Why Mobile Proxies Score Highest

Mobile IPs benefit from carrier-grade NAT (CGNAT), where a single IP address is shared among thousands of concurrent mobile users. Anti-bot systems know that blocking a mobile IP will affect legitimate users who share that IP. This creates a structural advantage for mobile proxy traffic.

Additionally, mobile IP assignments rotate naturally as devices move between cell towers and network sessions expire. Anti-bot systems expect high IP diversity from mobile carriers and treat it as normal behavior.

DataResearchTools’ Singapore mobile proxies leverage real carrier connections on major Singapore mobile networks, providing the same IP addresses used by genuine mobile subscribers.

Vendor Differences

  • Cloudflare: Maintains one of the largest IP reputation databases, fed by traffic data from millions of websites behind Cloudflare’s network. Their data advantage is significant.
  • Akamai: Leverages traffic patterns from their CDN network (which handles 15-30% of global web traffic) to build IP reputation profiles.
  • HUMAN: Focuses heavily on IP reputation as a primary signal, with particular emphasis on detecting known proxy provider IP ranges.
  • DataDome: Combines IP reputation with real-time behavioral signals, weighting IP reputation less than some competitors.

Layer 2: TLS Fingerprinting

TLS fingerprinting analyzes the characteristics of your TLS (HTTPS) handshake to determine what client is making the request. This happens before any HTTP data is exchanged.

How TLS Fingerprinting Works

When your client initiates a TLS connection, it sends a ClientHello message that includes:

  • Supported TLS versions
  • Cipher suites (in a specific order)
  • TLS extensions (and their order)
  • Supported elliptic curves
  • Supported point formats
  • ALPN protocols
  • Signature algorithms

Each browser and HTTP library produces a distinctive ClientHello. Chrome 120 on Windows has a different TLS fingerprint than Chrome 120 on macOS, which has a different fingerprint than Python’s requests library, which has a different fingerprint than Node.js fetch.

JA3 and JA4 Fingerprints

JA3 is a widely adopted method for fingerprinting TLS clients. It creates a hash of the ClientHello parameters (TLS version, cipher suites, extensions, elliptic curves, and point formats). JA4 is the successor with improved granularity.

Anti-bot systems maintain databases of JA3/JA4 fingerprints mapped to known clients. When they see a request claiming to be Chrome (via User-Agent header) but with a JA3 fingerprint matching Python’s requests library, the request is flagged as spoofed.

Defeating TLS Fingerprinting

  • Use a real browser: Headless Chrome produces the same TLS fingerprint as headed Chrome because it is the same TLS stack.
  • TLS fingerprint spoofing libraries: Libraries like curl-impersonate and tls-client can mimic the TLS fingerprint of specific browsers.
  • Match fingerprint to User-Agent: If your User-Agent says Chrome 120, your TLS fingerprint must match Chrome 120.

Vendor Implementation

  • Cloudflare: Pioneered the use of JA3 fingerprinting for bot detection. Their implementation checks for fingerprint-User-Agent mismatches and maintains a database of known bot fingerprints.
  • Akamai: Uses TLS fingerprinting as one of many signals, weighting it in combination with other layers.
  • HUMAN: Relies heavily on TLS fingerprinting, particularly for detecting headless browsers and HTTP libraries.

Layer 3: JavaScript Challenges

JavaScript challenges test whether the client can execute JavaScript in a real browser environment. This layer separates simple HTTP scrapers from browser-based automation.

How JS Challenges Work

When the anti-bot system suspects a request might be automated, it serves a challenge page instead of the actual content. This page contains JavaScript that:

  1. Computes a cryptographic proof-of-work (forces the client to spend CPU time)
  2. Collects browser environment data (APIs, properties, rendering capabilities)
  3. Reports results back to the anti-bot server
  4. If the results pass validation, the server issues a cookie/token that grants access to the actual content

Types of Challenges

Invisible challenges: Run automatically without user interaction. The page appears to load normally but executes checks in the background. Cloudflare’s “managed challenge” is an example.

Interactive challenges: Require user action, such as clicking a checkbox (Turnstile), solving a CAPTCHA (reCAPTCHA), or performing a behavioral task. These are more disruptive to legitimate users and are used when the system has higher suspicion.

Proof-of-work challenges: Require the client to solve a computational puzzle. This slows down automated clients by consuming CPU time proportional to the number of requests. Kasada is known for aggressive proof-of-work challenges.

What JS Challenges Collect

The JavaScript executing during a challenge collects extensive client data:

  • navigator properties (userAgent, platform, language, plugins, hardwareConcurrency, deviceMemory)
  • window properties (screen dimensions, color depth, devicePixelRatio)
  • Canvas fingerprint (rendering a hidden canvas element and hashing the pixel data)
  • WebGL fingerprint (GPU vendor, renderer, supported extensions)
  • Audio context fingerprint
  • Font enumeration
  • Timing data (how long computations take, which reveals the execution environment)
  • DOM API availability (certain APIs differ between headless and headed browsers)
  • Automation flags (navigator.webdriver, __selenium_unwrapped, Puppeteer-specific properties)

Bypassing JS Challenges

  • Headless browser with stealth: A properly configured headless browser with stealth patches can pass most JS challenges. See our headless browser proxy setup guide.
  • Challenge token reuse: Some challenges produce tokens that can be reused for multiple requests. Solve the challenge once in a browser, then use the resulting cookies/tokens for subsequent HTTP requests.
  • Challenge solver services: Third-party services that solve challenges at scale (similar to CAPTCHA solving services).

Layer 4: Browser Fingerprinting

Browser fingerprinting goes beyond JS challenges to create a unique identifier for each browser instance. This layer detects when the same browser visits repeatedly, even across IP changes.

Fingerprint Components

A comprehensive browser fingerprint combines:

  • Canvas fingerprint: Subtle rendering differences between GPUs and operating systems produce unique canvas output.
  • WebGL fingerprint: GPU vendor string, renderer string, supported extensions, and shader precision formats.
  • AudioContext fingerprint: Differences in audio processing hardware and software produce unique audio fingerprints.
  • Font fingerprint: The set of installed fonts varies between systems and can identify specific OS versions and configurations.
  • Plugin enumeration: Browser plugins and their versions create a unique combination.
  • Screen and display: Resolution, color depth, pixel ratio, available screen space (accounting for taskbar/dock).
  • Timezone and locale: Timezone offset, language preferences, date formatting conventions.

Fingerprint Consistency Detection

Anti-bot systems check for internal consistency within the fingerprint:

  • A Chrome User-Agent on Windows should have Windows-specific fonts, a DirectX-capable GPU, and a Windows-standard screen resolution.
  • A Safari User-Agent should report WebKit-specific CSS rendering quirks and macOS-specific system fonts.
  • A mobile User-Agent should have touch capabilities, appropriate screen dimensions, and mobile-specific API behavior.

Inconsistencies indicate fingerprint spoofing, which is itself a bot signal.

Cross-Session Fingerprint Tracking

Even when you rotate IPs and clear cookies, a stable browser fingerprint can link your sessions together. If the same fingerprint appears from 50 different IPs over a week, all making similar requests, the system identifies the traffic as automated.

Mitigation: Rotate fingerprints alongside IP rotation. Each new “user session” should have a unique, internally consistent fingerprint. Maintain a library of pre-built fingerprint profiles that correspond to real device configurations.

Layer 5: Behavioral Analysis

Behavioral analysis is the most sophisticated detection layer and the hardest to defeat. It analyzes how the client interacts with the page, looking for patterns that distinguish humans from bots.

What Gets Analyzed

Mouse dynamics: Real users move their mouse in natural curves with variable speed. They overshoot targets, correct course, and have characteristic acceleration and deceleration patterns. Bots either have no mouse movement or move in perfectly straight lines at constant speed.

Scroll behavior: Humans scroll at variable speeds, often overshooting and scrolling back. They pause at content that interests them. Bots scroll at constant speed or jump to specific positions.

Typing patterns: Keystroke dynamics (time between key presses, hold duration) are highly individual and difficult to fake consistently.

Navigation patterns: Humans browse non-linearly. They go back, they click on unrelated links, they spend variable time on different pages. Bots navigate systematically through predefined paths.

Timing: Humans have variable reaction times, typically 200-500ms for simple actions. They take longer for complex decisions. Perfectly consistent timing is a bot signal.

Machine Learning Models

Modern anti-bot systems use machine learning to classify traffic:

  • Supervised learning: Trained on labeled datasets of known human and bot traffic
  • Anomaly detection: Identifies traffic patterns that deviate from the baseline of normal human behavior
  • Clustering: Groups similar traffic patterns to identify bot networks even when individual sessions look legitimate
  • Real-time scoring: Assigns a bot probability score to each session in real time, updating as new behavioral data arrives

Vendor Behavioral Approaches

  • Cloudflare: Uses behavioral signals primarily as a risk multiplier. High behavioral risk combined with medium IP risk triggers challenges.
  • HUMAN: Behavioral analysis is HUMAN’s core differentiation. They collect extensive client-side telemetry and process it through ML models.
  • DataDome: Emphasizes real-time behavioral detection, claiming to detect bots within the first request of a session.
  • Akamai: Combines behavioral signals with their massive network data for traffic pattern analysis.

Detection Scoring Models

Anti-bot systems do not make binary decisions. They compute risk scores that determine the response.

Multi-Signal Scoring

Each detection layer contributes to an overall risk score:

SignalWeight (Typical)Score Range
IP reputationHigh0-30 points
TLS fingerprintMedium0-15 points
JS challenge resultHigh0-25 points
Browser fingerprint consistencyMedium0-15 points
Behavioral analysisHigh0-20 points
Historical session dataMedium0-10 points

A score above a threshold (e.g., 50 out of 100) triggers a challenge. A score above a higher threshold (e.g., 80) results in a block.

Adaptive Thresholds

Thresholds are not static. They adjust based on:

  • Overall traffic volume (tighter during traffic spikes)
  • Endpoint sensitivity (login pages have lower thresholds than blog posts)
  • Client configuration (the site operator can adjust aggressiveness)
  • Historical attack patterns (thresholds tighten after detected attacks)

How Mobile Proxies Affect Scoring

Mobile proxy IPs start with the lowest possible IP reputation risk score (0-5 out of 30), compared to data center IPs (20-30 out of 30). This gives mobile proxy traffic a significant head start. Combined with a properly configured headless browser (0-5 TLS score, 0-5 fingerprint score), the overall risk score stays well below challenge thresholds.

This is why mobile proxies from DataResearchTools consistently outperform other proxy types against anti-bot systems. The structural advantage of mobile IP trust scores compounds with every other layer of the detection model.

How Each Vendor Differs

Cloudflare

Market position: The largest anti-bot provider by website coverage (millions of sites behind Cloudflare).

Strengths:

  • Massive data network for IP reputation
  • Turnstile (their CAPTCHA replacement) provides smooth legitimate user experience
  • Fast deployment (DNS-level integration)
  • Bot Score API available to site operators for custom logic

Weaknesses:

  • Lower thresholds for free/lower-tier plans mean more false positives
  • Widely studied by the bot detection bypass community
  • Managed Challenge can be bypassed by well-configured headless browsers

Akamai Bot Manager

Market position: Dominant among large enterprises, especially financial services, airlines, and e-commerce.

Strengths:

  • Deep network visibility from their CDN (handles enormous traffic share)
  • Sensor data collection (client-side JavaScript) is highly sophisticated
  • Strong in detecting credential stuffing and account takeover attacks

Weaknesses:

  • More expensive, primarily serving enterprise clients
  • Sensor script is large and performance-impacting
  • Less frequently updated than Cloudflare’s detection

HUMAN (formerly PerimeterX)

Market position: Used by many mid-to-large e-commerce and ticketing platforms.

Strengths:

  • Behavioral analysis is their primary differentiation
  • Pre-built integrations with major e-commerce platforms
  • Strong in detecting sophisticated bots that bypass other vendors

Weaknesses:

  • Smaller network data advantage compared to Cloudflare/Akamai
  • Can be resource-intensive on client side

DataDome

Market position: Growing European-based vendor with strong presence in e-commerce.

Strengths:

  • Claims first-request detection (no initial challenge page)
  • Server-side detection reduces client-side impact
  • Fast integration (CDN-agnostic)

Weaknesses:

  • Smaller market presence means less traffic data for reputation scoring
  • Less publicly documented, making research harder

Kasada

Market position: Niche vendor focusing on proof-of-work challenges.

Strengths:

  • Proof-of-work challenges make automated access computationally expensive
  • Effective against high-volume scraping operations
  • Obfuscated JavaScript challenges that resist reverse engineering

Weaknesses:

  • Proof-of-work can impact legitimate user experience
  • Less sophisticated behavioral analysis compared to HUMAN

The Future of Bot Detection

AI-Based Detection

Anti-bot systems are increasingly using AI/ML models that:

  • Analyze traffic patterns across millions of sessions to identify subtle bot signatures
  • Adapt in real-time to new bot techniques without manual rule updates
  • Detect bot networks by correlating behavior across seemingly unrelated sessions
  • Generate novel challenge types designed to be difficult for current-generation bots

Hardware Attestation

Apple’s Private Access Tokens and Google’s Privacy Pass are early examples of hardware-backed device attestation. These cryptographic tokens prove that a request originates from a genuine device without revealing the user’s identity. As adoption grows, anti-bot systems will increasingly require hardware attestation, which is extremely difficult to fake.

Behavioral Biometrics

The next frontier in behavioral analysis is continuous behavioral biometric authentication. Rather than checking behavior at specific checkpoints, the system continuously analyzes mouse, keyboard, and touch patterns to maintain a real-time confidence score that the user is human.

Implications for Scraping

These advances will make scraping progressively harder. The response for data practitioners:

  • Invest in high-quality infrastructure (mobile proxies, real browser automation) rather than trying to circumvent detection cheaply
  • Consider API access and legitimate data partnerships as the detection bar rises
  • Build resilient architectures that can adapt to changing detection methods
  • Focus on the structural advantage of mobile proxies, which will remain effective as long as mobile users exist on CGNAT networks

Practical Recommendations

Based on how detection systems work, here are the infrastructure decisions that matter most:

  1. Start with mobile proxies: IP reputation is the highest-weighted signal. Starting with high-trust IPs gives you the most headroom across all other detection layers. Explore DataResearchTools’ mobile proxy options.
  1. Use real browsers: Headless browsers with stealth patches pass TLS fingerprinting, JS challenges, and browser fingerprinting simultaneously. See our headless browser guide.
  1. Implement realistic rate limiting: Behavioral analysis detects machine-speed request patterns. Rate limiting with randomized timing is essential.
  1. Rotate fingerprints with IPs: Use our rotation strategy guide to coordinate IP and fingerprint rotation for consistent sessions.
  1. Monitor continuously: Detection systems evolve. What works today may trigger blocks next month. Build monitoring that detects degradation early.

The anti-bot arms race favors defenders with more data and resources. But by understanding how detection works and investing in the right infrastructure, data practitioners can maintain reliable access to the web data they need.

Related Reading

Scroll to Top