What a robots.txt file asks for and what it does not

Every time someone on the team asks “can we scrape this site,” the first thing that gets pulled up is the robots.txt file. It’s the fastest signal you can get about a site’s crawling posture, and it takes about ten seconds to read. But it also gets misunderstood constantly, in both directions. Some people treat it as a legal wall. Others treat it as nothing at all. Neither is right, and getting this wrong shapes decisions that matter, from what your pipeline is allowed to touch to what your crawl-delay logic should look like.

Here’s what the file actually is, what it does, and where its authority ends.

What robots.txt actually is

robots.txt is a plain text file, sitting at the root of a domain (example.com/robots.txt), that follows the Robots Exclusion Protocol. It’s been around since 1994. It’s not a config file the web server enforces. It’s not a firewall rule. It’s a set of instructions that well-behaved automated clients are expected to read and follow voluntarily before crawling a site.

That word “voluntarily” is the whole story. Nothing on the server side checks whether your HTTP client fetched robots.txt first, and nothing blocks a request just because robots.txt says to disallow it. The file is a request, not a lock. Google’s crawler respects it because Google has committed to respecting it, and because ignoring it would get Search Console flooded with webmaster complaints. Your own scraper respects it (or doesn’t) purely based on what you build into it.

The syntax, briefly

A typical file looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /blog/
Crawl-delay: 10

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

User-agent targets a specific crawler by its declared name, or * for everyone. Disallow lists path prefixes that agent should not fetch. Allow carves out exceptions within a disallowed section. Crawl-delay is a non-standard but widely respected suggestion for seconds between requests. Sitemap points to an XML sitemap, which is honestly the most useful line in the file if you’re trying to discover a site’s public content quickly instead of crawling blind.

Rules are matched by the longest path prefix, not by order in the file, and matching is case-sensitive. A site can have different rule blocks for different user-agents, which is how you get the increasingly common pattern of a general Allow policy alongside a hard Disallow: / aimed specifically at AI training crawlers like GPTBot or CCBot.

What it does not do

Three things people routinely get wrong.

It does not stop a request from reaching the server. If /cart/ is disallowed and you send a GET to /cart/, the server responds exactly as it would for any other request, same status code, same content, same latency. Nothing about robots.txt changes the server’s behavior. Any actual enforcement, blocking, rate limiting, CAPTCHA, comes from a separate layer entirely: WAF rules, reverse proxy config, application middleware, or a bot detection vendor sitting in front of the origin. robots.txt and enforcement are two unrelated systems that happen to live on the same domain.

It’s not a technical access control, and it’s not designed to be one. The protocol has no concept of authentication, no way to actually gate a resource. If a site wants a path to be genuinely inaccessible, it needs a login wall, an IP allowlist, or it needs to just not serve the content publicly. Putting something behind a robots.txt Disallow while leaving it on the open internet is closer to putting a “staff only” sign on an unlocked door than actually locking it.

It’s not a settled legal instrument either, and I’d be careful with anyone who tells you flatly that ignoring it is “legal” or “illegal.” Courts in different jurisdictions have looked at robots.txt directives as evidence of a site owner’s stated intent, relevant in some disputes around unauthorized access or breach of terms of service, but the file itself has no statutory force. Terms of service, the CFAA and equivalent laws elsewhere, and the nature of the data being collected all matter more than whether a Disallow line existed. If you’re scraping anything involving personal data, paywalled material, or content a site has gone out of its way to restrict, that’s a legal and ethical question that robots.txt does not answer for you, and no engineering trick resolves it either.

Why we still check it first

Given all that, it might sound like robots.txt is irrelevant to a real pipeline. It isn’t. We check it before touching a new domain for a few concrete reasons.

It’s the fastest map of what a site considers safe for automated traffic versus what it doesn’t want indexed or hit at volume. Disallowed paths often line up with admin panels, cart and checkout flows, internal search results, or anything that’s expensive to render server-side. Respecting those isn’t just politeness, it keeps your crawler off pages that were never meant for bulk automated traffic and that tend to be poorly cached, meaning every hit is a real database query on their end.

Crawl-delay, where present, is a direct signal for how to pace requests against that origin without needing to reverse-engineer their rate limits by trial and error. We treat it as a floor, not a target. Sites publishing an aggressive Disallow: / for a named agent, particularly AI crawlers, are telling you plainly that traffic identifying as that agent will not be tolerated. A production pipeline that respects those declarations avoids a class of avoidable conflict with site owners entirely.

It also matters for how bot detection systems get built and tuned, which is a topic worth understanding from the defensive side even if you never operate one yourself. Some detection stacks log requests to disallowed paths as a scoring input, on the reasoning that a client hitting /admin/ after presumably fetching robots.txt (or without fetching it at all) looks less like a browser and more like an indiscriminate crawler. It’s one signal among many, alongside TLS fingerprinting, header ordering, timing patterns, and JS execution checks, and on its own it rarely triggers a block. But it’s a real input into how these systems build a risk score, which is one more reason serious scraping operations treat robots.txt as an actual constraint on crawl design rather than a file to ignore.

Where this leaves you

robots.txt tells you what a site owner would prefer automated clients do. It does not tell you what’s technically possible, and it does not settle what’s legally or ethically fine to collect. Those are three separate questions, and conflating them is how projects end up either overly cautious about publicly available, harmless data, or genuinely out of line on something that mattered. Read the file, respect what it says, use the sitemap if it’s there, and treat the crawl-delay as a real number. Then make the actual legal and ethical call, about the data itself, the site’s terms, and what you’re going to do with what you collect, as its own separate decision.

If you’re building out scraping infrastructure and want the rest of the stack explained with the same level of detail, from proxy architecture to how fingerprinting actually works under the hood, that’s what we cover on Data Research Tools.

Get new guides and videos first — join the Telegram channel.