When a Site Asks You to Stop: Reading and Respecting Anti-Scraping Signals

The moment it happens

It rarely shows up as a sentence. Most of the time it’s a status code, a blank page where data used to be, or a CAPTCHA wall that wasn’t there last week. Sometimes it’s louder than that: an actual email from someone’s legal team, or a note from a hosting provider saying a target has complained about traffic from your IP range. All of these are the same message delivered through different channels, and any team running scraping infrastructure at scale needs to be able to read every one of them, not just the loud ones.

I run production pipelines against a lot of targets, on proxy infrastructure I maintain myself, and I’ve watched this signal arrive in every form below. None of them are ambiguous once you know what you’re looking at.

The quiet way: robots.txt and terms of service

The first and quietest signal is robots.txt. A Disallow directive on a path, or a Crawl-delay value, is a site operator writing down, in a machine-readable file, exactly what they’re willing to have automated tools touch and how fast. It isn’t legally enforceable on its own, but it’s documented intent, and treating it as noise because it’s “just a text file” misreads what it actually is: the operator telling you their preference before anything else happens.

Terms of service go further, usually with an explicit clause on automated access, scraping, or data reuse. ToS language varies wildly in enforceability by jurisdiction and by what you actually do with the data, and that’s a legal question, not an engineering one. What’s consistent is that a ToS clause prohibiting scraping is a stop signal stated in plain language, before you’ve sent a single request that trips a rate limiter.

The technical way: rate limits and soft blocks

Next comes the layer built into the infrastructure itself. A 429 Too Many Requests with a Retry-After header is a server telling a specific client, in the HTTP spec’s own vocabulary, to slow down or stop for a defined window. On the backend, this is usually implemented as a token bucket or sliding window counter keyed on IP, session, or API key, with thresholds that adapt based on traffic patterns for that route. When you see a 429 pattern, it isn’t random. Someone configured a limit, and your traffic crossed it.

Softer versions of the same signal exist below the 429. Some sites serve degraded or stale content to traffic they’ve flagged as automated rather than blocking it outright, so the response looks successful but the data quietly stops being useful. Others throttle response times for suspect clients. These are deliberate design choices by the site’s engineering team to discourage high-volume automated access without an outright ban, and they’re worth treating with the same seriousness as a hard block, because they’re often the first stage before one.

The hard way: fingerprinting and full blocks

Above rate limiting sits bot management infrastructure, and this is where a lot of the actual detection engineering lives. Commercial WAF and bot-management stacks (Cloudflare, Akamai Bot Manager, DataDome, Human/PerimeterX, and similar vendors) combine several signal sources: TLS handshake fingerprints like JA3 or JA4, which expose differences between how a real browser’s TLS stack negotiates a connection versus an HTTP client library; HTTP header order and casing, which differ between browser engines and most scraping frameworks; canvas and WebGL rendering fingerprints collected via JavaScript challenges; timing between requests; and IP and ASN reputation, where ranges known for hosting or proxy traffic get scored differently than residential or mobile ranges from the start.

None of these signals are decisive alone. What actually flags a session is usually a scored combination, run against a model trained on real traffic from that specific site, which is why the same request pattern gets waved through on one target and challenged on another. It’s worth understanding this as a detection system, not a single trick to route around, because that’s genuinely how it’s built and how the vendors who sell it describe it in their own documentation. A challenge page or a hard 403 at this layer is the site’s infrastructure making an automated determination that your traffic doesn’t look like a browser being driven by a person, and pushing past that determination with a workaround doesn’t change the underlying signal, it just delays the next detection.

The direct way: a human tells you

The clearest version of “stop” needs no interpretation. A cease and desist letter, an account suspension email, a message from a site’s legal or trust and safety team naming your traffic or your company specifically. There’s no ambiguity to parse here, and no technical workaround is a reasonable response to it. If this arrives, the scraping activity it names stops, full stop, while you get a lawyer to tell you what happens next.

Why the arms race is a bad bet, not just a bad look

Set the ethics aside for a second and look at it as an operator managing infrastructure and budget. A residential or mobile proxy pool costs real money per IP, and IP reputation on shared infrastructure is a depreciating asset: once a range gets flagged by a major bot-management vendor’s shared reputation database, that damage often follows the range across every site using that vendor, not just the one you were scraping. Burning a proxy pool trying to outlast one target’s detection layer is money spent degrading infrastructure you’ll want for legitimate work later.

Then there’s the fact that the other side usually has more resources allocated to this exact problem than you do. A company running Akamai or DataDome in front of a high-value dataset is paying a vendor whose entire product is detection, updated continuously against traffic from thousands of scrapers, not just you. Treating a stop signal as a puzzle to solve is a fight against a team built to win it, on their own turf, with a budget scoped to the problem.

What a production pipeline should actually do

The operational answer is boring, and that’s the point. Respect robots.txt by default in your crawler configuration, and require a deliberate override, not a silent one, if a target’s ToS explicitly permits otherwise. Implement backoff on 429s that honors the Retry-After header rather than retrying immediately from a different IP. Set conservative default request rates per target and per session, and treat a rising block or challenge rate on a target as a signal to reduce load automatically, not a trigger to rotate proxies harder. Where a site publishes an official API, use it instead of scraping the same data from rendered pages; it’s usually more stable and it’s the access method the site actually wants automated traffic to take. And where a target’s terms are explicit about prohibiting automated access at any scale, that’s the ceiling, not a starting negotiating position.

If a cease and desist actually lands

Stop the named activity immediately. Preserve your logs rather than deleting anything, since you may need them. Talk to a lawyer before responding, since the specifics of what was scraped, how, and under what jurisdiction all change the actual legal exposure, and none of that is something a blog post can tell you in the general case. What it isn’t is a prompt to switch identities and keep the pipeline running unchanged.

Building the habit before you’re asked

The operators who handle this well aren’t the ones with the cleverest workaround ready for the day a target pushes back. They’re the ones who built rate limits, robots.txt compliance, and ToS review into the pipeline before the first request went out, so “the site asked us to stop” is a normal, expected branch in the system rather than an emergency. Treat every one of these signals, quiet or loud, as data about what a target is willing to allow, and design the infrastructure to respect that from the start.

If you’re building or auditing a scraping pipeline and want to see how we think about proxy infrastructure, rate limiting, and detection from the defensive side, take a look at what we’ve written and reviewed here.

Get new guides and videos first — join the Telegram channel.