What Is Crawl-Delay? The robots.txt Directive Google Ignores

Crawl-delay is a robots.txt directive that asks a crawler to wait a set number of seconds between requests to a site. It is not part of the core Robots Exclusion Protocol standardized by RFC 9309, so support is inconsistent: Bing and Yandex honor it, while Google ignores it entirely and manages crawl rate algorithmically instead.

What Is Crawl-Delay?

Crawl-delay is a directive placed inside a User-agent block in robots.txt that specifies a minimum number of seconds a crawler should wait before requesting another page from the same host. The syntax looks like this:

It was designed to prevent an aggressive crawler from overwhelming server resources by spacing out requests, functioning as a politeness throttle rather than an access restriction.

Which Search Engines Actually Support It

Google's public robots.txt parser documentation was updated in April 2026 to explicitly name several long-used but unsupported fields, adding content-signal, content-usage, domain, request-rate, revisit-after, and visit-time to an existing list that already included clean-param, crawl-delay, host, noarchive, nofollow, and noindex. The change is documentation clarity, not a behavior change — Google states plainly that it never processed these fields, and identified them as common in real robots.txt files via HTTP Archive analysis.

How to Use Crawl-Delay Correctly

Scope the directive to specific user agents rather than a blanket wildcard, since a wildcard block affects every crawler that recognizes the extension, including SEO tools like Ahrefs or Semrush that also honor it. A typical configuration might set a short delay for Bing on shared hosting and a longer one for lower-priority third-party crawlers, while leaving Googlebot alone entirely since the directive has no effect on it.

Reasonable values sit between one and five seconds for cooperative crawlers on typical hosting, and five to ten seconds on constrained shared hosting. Values above thirty seconds meaningfully slow discovery and recrawling and are rarely justified outside emergency load situations.

Crawl-Delay vs. Adjacent Concepts

Crawl-delay is a request-pacing directive; crawl budget is the broader resource allocation — capacity plus demand — that governs how many URLs a crawler fetches in total. Setting crawl-delay for Bing narrows the pace within whatever budget Bing has already allocated; it doesn't expand or shrink that budget itself. For Google specifically, the directive is a dead letter: Google Search Console's crawl-rate emergency controls and general server-health signals are the only levers that actually move Googlebot's pace.

Limits

Crawl-delay is a voluntary instruction that only cooperative crawlers honor. It has no effect on non-compliant scrapers or malicious bots ignoring robots.txt entirely — server-side rate limiting, authentication, firewall rules, or IP-level blocking are the actual controls for that traffic. It also does not function as an access-control mechanism: a search engine or tool that doesn't recognize the directive will simply crawl at whatever pace its own logic determines.

Conclusion

Crawl-delay is a narrow, voluntary throttle that works for Bing and Yandex but does nothing for Google, which manages its own pace algorithmically and has now formally documented crawl-delay as unsupported. Use it scoped to specific user agents on hosting that genuinely needs the relief, and rely on server-side controls — not robots.txt — for any traffic that actually needs to be stopped rather than merely slowed.

If your own workflows need to fetch pages at scale without tripping a target site's rate limits, evaluate Nstdata Crawl, which handles pacing and retries as part of the request pipeline.

FAQ