What Is robots.txt? Syntax, Rules, and RFC 9309 Explained
robots.txt is a plain-text file at a site's root that tells crawlers which paths they may or may not request. Since September 2022, the format is a formal IETF standard — RFC 9309 — which finally resolved decades of inconsistent, informally-extended behavior across different crawlers by defining exactly how the core directives, matching rules, and error handling are supposed to work.
⚡ Key Takeaways
- robots.txt must live at
/robots.txt, all lowercase, at the top level of the site, UTF-8 encoded, served astext/plain. - RFC 9309 standardized the Robots Exclusion Protocol in 2022, formalizing rules that had existed informally since 1994.
- Within a matching group, the longest matching rule wins; if an Allow and a Disallow tie in length, Allow wins.
- 4xx responses for robots.txt mean "fully crawlable"; a persistent 5xx for over 30 days is treated the same way, though Google pauses crawling in the short term.
- Only two special characters matter for matching:
*(wildcard) and$(end-of-path anchor). - Many widely-used directives — including crawl-delay, host, and noindex — are not part of the standard and are ignored by Google's parser even though they appear in countless real robots.txt files.
What Is robots.txt?
robots.txt is the file websites use to tell visiting crawlers which parts of the site they're permitted to request. It's not an access-control mechanism — a well-behaved crawler chooses to respect it, but nothing prevents a non-compliant bot from ignoring the file entirely and requesting disallowed paths anyway.
The format was first published in 1994 by Martijn Koster and adopted informally across the web for nearly three decades. In September 2022, Google worked with Koster and IETF contributors to formalize it as RFC 9309, giving the protocol an authoritative definition of its parsing rules, error handling, and caching behavior for the first time.
Syntax and Matching Rules
A robots.txt file is organized into groups, each starting with one or more User-agent lines followed by Allow and Disallow rules, with an optional Sitemap line that applies file-wide rather than to a specific group.
| Line | Meaning |
|---|---|
User-agent: Googlebot | Starts a group of rules that apply only to Google's crawler. |
Disallow: /private/ | Blocks any path beginning with /private/. |
Allow: /private/public-notice.html | Carves out an exception within a disallowed path. |
Sitemap: https://example.com/sitemap.xml | Points to the sitemap; applies regardless of which group it's placed near. |
Within a matching group, RFC 9309 requires the longest matching rule to win, regardless of the order the rules appear in the file — length beats position. When an Allow and a Disallow rule match a URL with equal length, Allow wins the tie. Only two special characters carry meaning in path patterns: * matches any sequence of characters, and $ anchors the match to the exact end of the path.
Supported vs. Unsupported Directives
Google's parser documentation explicitly names a growing list of directives it does not process, despite their frequent appearance in real robots.txt files: crawl-delay, host, noarchive, nofollow, noindex, and clean-param were already on that list, and an April 2026 update added content-signal, content-usage, domain, request-rate, revisit-after, and visit-time — tags identified as common in real files through HTTP Archive analysis. Only User-agent, Allow, Disallow, and Sitemap are part of the RFC 9309 core; anything else is, at best, honored inconsistently by other crawlers.
Error handling is also standardized: a 4xx response when fetching robots.txt means the crawler should treat the site as fully crawlable, since "not found" is interpreted as "no restrictions." A persistent 5xx lasting more than roughly 30 days is eventually treated the same way, though Google pauses crawling in the near term while it waits to see if the error resolves. This is why serving a valid file — even a minimal one containing just User-agent: * and a sitemap line — is safer than letting the request fail.
Fetching at scale? Respect the rules automatically
Nstdata Crawl checks and respects robots.txt rules as part of its fetch pipeline, so your own crawling stays compliant without manual parsing logic.
Try Nstdata Crawl →robots.txt vs. Adjacent Concepts
robots.txt controls crawling access at the path level; it does not reliably control indexing — a disallowed URL that's linked to from elsewhere can still appear in results without a description, since Google never crawled it to see a noindex tag in the first place. A noindex meta tag or HTTP header is the more definitive way to keep a page out of the index, but it requires the page to be crawlable so the tag can actually be seen. The crawl-delay directive, sometimes placed inside a robots.txt group, is a related but separate mechanism aimed at request pacing rather than access control, and Google's parser ignores it entirely.
Limits
robots.txt is fundamentally advisory. It has no effect on crawlers that choose not to honor it, and it provides no security — anything referenced in a disallowed path is still technically reachable by direct request, so it should never be relied on to hide genuinely sensitive content. The standardization from RFC 9309 improved consistency for compliant crawlers, but the AI-crawler landscape has since exposed a new gap: the standard doesn't define AI-specific directives for training or synthesis use, and some AI crawlers reportedly ignore robots.txt altogether, which is why IETF discussions are exploring follow-up extensions for AI usage preferences.
Conclusion
robots.txt is a simple, voluntary access-control signal that finally has a formal specification in RFC 9309 after nearly thirty years of informal, inconsistent behavior. Stick to the four standardized directives — User-agent, Allow, Disallow, Sitemap — for anything that needs to work reliably across crawlers, and use noindex rather than robots.txt when the goal is actually keeping a page out of search results.
If your own crawling needs to respect target sites' robots.txt automatically at scale, evaluate Nstdata Crawl against your workflow.
Further Reading
Sources
Try Nstdata Crawl for compliant, large-scale fetching
robots.txt-aware crawling with proxy-backed access and automatic retries.
Try Nstdata for Free →FAQ
Q: Is robots.txt legally binding?
No. It's a voluntary standard that compliant crawlers choose to follow. It has no legal force on its own and doesn't prevent a non-compliant bot from requesting disallowed paths.
Q: Does robots.txt prevent a page from being indexed?
Not reliably. It prevents crawling, which usually prevents indexing, but a disallowed URL linked to from elsewhere can still appear in results without a description. Use a noindex tag for a definitive block.
Q: What happens if my robots.txt file returns an error?
A 4xx response is treated as "fully crawlable, no restrictions." A persistent 5xx for over roughly 30 days is eventually treated the same way, though Google pauses crawling in the short term.
Q: Does Google support the crawl-delay directive in robots.txt?
No. Google's parser documentation explicitly lists crawl-delay as unsupported, along with host, noindex, and several other commonly used but non-standard directives.
Q: What are the only directives RFC 9309 actually standardizes?
User-agent, Allow, Disallow, and Sitemap. Anything else — including crawl-delay, host, and clean-param — sits outside the formal standard and is handled inconsistently across different crawlers.
Was this guide helpful?
Your choice is saved on this device.
