How to Implement the Crawl-Delay Directive in robots.txt
The Crawl-delay directive goes inside a robots.txt User-agent block to ask a crawler to wait a set number of seconds between requests. It only affects crawlers that choose to honor it — Bing and Yandex do, Google and Baidu don't — so implementing it correctly means scoping it to the right user agents and verifying the result against your actual server logs, not just the file's syntax.
⚡ Key Takeaways
- Place Crawl-delay inside a specific User-agent block, not a wildcard, unless you genuinely want it to apply to every crawler that recognizes it.
- Bing treats the value as a time window; Yandex treats it as a minimum gap between requests — the same number behaves slightly differently on each.
- Google and Baidu ignore the directive entirely, so setting it for Googlebot has zero effect regardless of the value chosen.
- One to five seconds is reasonable for cooperative crawlers on typical hosting; values above thirty seconds meaningfully slow discovery.
- Verify the implementation against server logs, not just by re-reading the robots.txt file — logs show whether the target crawler is actually respecting the pacing.
- For crawlers that don't support the directive at all, server-side rate limiting is the only real control.
What the Crawl-Delay Directive Does
Crawl-delay is a line placed inside a robots.txt User-agent group that specifies a minimum number of seconds a crawler should wait before making its next request. It's not part of the core Robots Exclusion Protocol standardized by RFC 9309 — it's a widely-adopted but informal extension, which is exactly why support and interpretation vary by crawler rather than following one universal rule.
Implementing Crawl-Delay Step by Step
Step 1: Identify which crawlers actually need throttling
Check server logs for the user agents generating the load you want to reduce. There's no reason to add Crawl-delay for Googlebot or Baiduspider specifically, since neither reads it — target Bingbot, YandexBot, or any SEO tool crawler (Ahrefs, Semrush) that's contributing to the load and is documented to support the directive.
Step 2: Scope the directive to a specific User-agent block
Write a dedicated group for each crawler that needs a distinct pace, rather than relying on a wildcard block that could unintentionally throttle crawlers you didn't intend to affect:
| robots.txt block | Effect |
|---|---|
User-agent: Bingbot | Bing fetches at most one page per 5-second window. |
User-agent: YandexBot | Yandex waits at least 10 seconds between two consecutive requests. |
User-agent: Googlebot | No Crawl-delay needed — Google ignores the directive entirely. |
Step 3: Choose a value proportional to the actual problem
Start with one to five seconds for cooperative crawlers on typical hosting, and five to ten on constrained shared hosting where multiple bots crawling simultaneously are degrading performance for real users. Reserve values above thirty seconds for genuine emergencies, since that range meaningfully slows discovery and recrawling for the affected crawler with limited additional benefit to server load.
Step 4: Validate the file's syntax
Confirm the directive sits correctly inside its intended User-agent group and that the file as a whole still parses as valid robots.txt — a malformed group boundary can cause a Crawl-delay line to apply to the wrong crawler, or to none at all.
Step 5: Confirm the effect in server logs, not just the file
Writing the directive correctly doesn't guarantee the target crawler is honoring it. Filter access logs by the relevant user agent over a few days and check whether request intervals actually reflect the configured delay — this is the only way to distinguish "the directive is correct" from "the directive is having the intended effect."
Need controlled pacing on the requesting side instead?
If you're the one doing the crawling rather than receiving it, Nstdata Crawl manages request pacing, concurrency, and retries automatically against target sites' actual tolerance — no manual Crawl-delay tuning required.
Try Nstdata Crawl →Common Implementation Mistakes
- Applying Crawl-delay to a wildcard User-agent when the goal was only to throttle one specific crawler, inadvertently slowing down every compliant bot that recognizes the directive.
- Setting a Crawl-delay for Googlebot expecting it to reduce Google's crawl rate — it has no effect; Google Search Console's crawl-rate guidance for emergencies is the actual lever for Google specifically.
- Choosing an arbitrarily high value "just to be safe," which slows legitimate discovery and recrawling far more than the server-load problem actually warranted.
- Assuming the directive stops malicious or non-compliant scrapers — it's voluntary, so it does nothing against bots that don't parse or respect robots.txt at all.
Crawl-Delay vs. Adjacent Concepts
Crawl-delay paces requests within whatever crawl budget a search engine has already allocated to a site — it doesn't expand or shrink that budget, it just spreads the same requests out over more time for crawlers that honor it. For traffic that ignores Crawl-delay entirely, server-side rate limiting, IP-based throttling, or authentication are the actual controls, since robots.txt directives of any kind are advisory rather than enforced.
Conclusion
Implementing Crawl-delay correctly is less about syntax — the line itself is simple — and more about scoping it to crawlers that actually support it, choosing a value proportional to the real server-load problem, and verifying the effect in logs rather than assuming the file works as written. For Google specifically, skip it entirely and use Search Console's crawl-rate tools instead.
If your own systems are the ones doing the fetching and need to pace requests against a target site's tolerance, evaluate Nstdata Crawl, which handles that pacing as part of its request pipeline.
Further Reading
Sources
Try Nstdata Crawl for managed request pacing
Automatic pacing and retries when you're the one doing the fetching.
Try Nstdata for Free →FAQ
Q: Where exactly does the Crawl-delay line go in robots.txt?
Inside a User-agent block, alongside Allow and Disallow rules for that same crawler. It applies to whichever User-agent group it's placed in.
Q: Can I set different Crawl-delay values for different crawlers?
Yes. Create a separate User-agent block for each crawler that needs its own value, rather than relying on one wildcard block for all of them.
Q: How do I check if my Crawl-delay directive is actually working?
Filter your server access logs by the target crawler's user agent and check whether the observed request intervals match the configured delay. The robots.txt file being syntactically correct doesn't confirm the crawler is honoring it.
Q: What should I use instead of Crawl-delay for Googlebot?
Google Search Console's crawl-rate settings, intended for emergency situations, since Google's parser does not process the Crawl-delay directive at all.
Q: Is there a maximum useful Crawl-delay value?
Values above thirty seconds are rarely justified — they significantly slow discovery and recrawling for the affected crawler without a proportional reduction in server load beyond what a much smaller value already achieves.
Was this guide helpful?
Your choice is saved on this device.
