What Is Crawl Budget? How Google Decides What Gets Crawled
Crawl budget is the number of URLs Googlebot can and wants to crawl on a site within a given period. It is not a fixed allowance Google assigns to a domain, and on most sites it is not why a page is missing from search results. Google rewrote its crawl budget documentation on 22 July 2026, splitting the concept into two independent components that move separately: crawl capacity and crawl demand.
⚡ Key Takeaways
- Crawl budget is the number of URLs Googlebot can and wants to crawl on a site within a given period — not a static allowance.
- Google's July 2026 rewrite splits it into two components: the crawl capacity limit (what your server can handle) and crawl demand (how much Google wants your URLs).
- Every site starts from the same conservative default capacity. Google raises it only when demand exists and the server stays healthy.
- Crawl capacity is shared across all of Google's crawlers, including those feeding AI Overviews and Gemini grounding.
- Crawl budget mainly matters on large sites, roughly above ten thousand crawlable URLs — below that, unindexed pages are usually a quality or linking issue.
noindexdoes not save crawl budget. Googlebot still has to fetch the page to see the tag.
What Is Crawl Budget?
Crawl budget is the set of URLs Googlebot can and wants to crawl on a given site within a given window of time. Google originally defined it by combining crawl rate and crawl demand into the number of URLs Googlebot can and wants to crawl, and flagged low-value URLs as a drag on that budget from the start.
Google rewrote the underlying documentation on 22 July 2026, and the newer framing separates what moves independently more clearly. Crawl budget is now defined through two components: the crawl capacity limit and crawl demand. In practice, crawl budget behaves as the minimum of what a site's infrastructure can absorb and what Google is motivated to fetch — a large, fast server with low demand still gets crawled lightly, and a popular site on a slow server still gets throttled.
Mechanism: Capacity and Demand
Crawl capacity limit. This is the total time a server spends holding connections open for Google, based on the number of parallel connections and their duration. It adjusts automatically: response times under roughly 200ms typically let Google push the pace up, while 5xx errors or slow responses pull it back down. Every site starts with the same conservative default capacity limit, and Google raises it only when demand exists and the site stays healthy — there is no carried-over baseline. The rewrite also made explicit that crawl capacity is shared across all of Google's crawlers, so heavy crawling by one reduces what's available to the others — a detail that matters more now that part of that shared pool also feeds AI Overviews and Gemini grounding rather than classic search alone.
Crawl demand. This is Google's appetite for a site's URLs, shaped by perceived inventory, popularity, and staleness. A page that earns external links or steady traffic gets revisited more; a page nobody links to or updates gets deprioritized regardless of how much spare server capacity exists.
Crawl budget optimization is, in practice, a triage exercise: fix whichever side is the real constraint. If Search Console's Crawl Stats show the server absorbing every request Google sends, demand is the bottleneck — publish, earn links, and prune stale inventory. If response times are climbing or 5xx errors are showing up, capacity is the bottleneck — that's an infrastructure fix, not a content one.
When Crawl Budget Actually Matters
Crawl budget is a genuine constraint mainly on large or fast-changing sites: big e-commerce catalogs, marketplaces, news publishers, and sites with heavy faceted navigation or pagination. Below that scale, a missing page is far more often a quality, duplication, or internal-linking failure wearing a crawl-budget costume. One practical threshold used by practitioners — not an official Google figure — is roughly ten thousand crawlable URLs as the point where crawl budget starts to deserve its own investigation.
The clearest diagnostic signal is Search Console's "Discovered – currently not indexed" report, which shows URLs Google has found but hasn't yet crawled. A large and growing count there, alongside valuable pages that stay unindexed, points to a real crawl budget issue rather than a content quality problem. Cross-checking the Crawl Stats report and raw Googlebot access logs confirms whether the server is the bottleneck or Google simply isn't interested yet.
Types of Low-Value URLs That Drain Budget
| Pattern | Why it drains budget |
|---|---|
| Faceted navigation / session IDs | Generates near-duplicate URLs for the same underlying inventory. |
| Soft 404s and thin content | Google keeps re-checking these instead of dropping them from the queue. |
| Hacked or spammed pages | Injected content that can also trigger sudden, unexplained crawl spikes. |
| Infinite spaces | Calendar pages or paginated chains only reachable by following "next" links sequentially. |
| Duplicate parameterized URLs | Near-identical pages differentiated only by tracking or filter parameters. |
Deep pages behind these patterns — reachable only after several link hops — are crawled less often by construction, since crawl depth and crawl budget compound each other on large sites.
Fetching at scale runs into the same constraint
Nstdata Crawl handles large-scale, reliable page fetching with proxy-backed access and automatic retries, so your own collection jobs don't get throttled the way an under-provisioned crawler does.
Try Nstdata Crawl →Crawl Budget vs. Adjacent Concepts
Crawl budget is frequently confused with two neighboring terms. Crawl rate / crawl capacity is only the server-side half of the equation — what Googlebot can fetch without overloading the site; crawl budget is the combined outcome of capacity and demand. Crawl frequency is the observable result — how often a given page or section actually gets revisited. It's downstream of crawl budget, not a synonym for it.
noindex is a common point of confusion too: it doesn't conserve crawl budget, because Googlebot still has to request the page to discover the tag. Blocking the path in robots.txt, or removing the internal links that generate it, are the actual budget-saving moves.
Limits of the Concept
Crawl budget is not a mysterious score, and Google has been explicit that the July 2026 rewrite didn't change how crawling actually works — it wrote down a mechanism that was previously left to inference. It's also not a ranking factor in itself: being crawled is a precondition for indexing and ranking, not a signal that improves either. Optimizing crawl budget on a small, well-structured site is unlikely to move rankings, because the site was never budget-constrained to begin with. The return on this work scales with site size and change velocity, not with effort spent.
Conclusion
Crawl budget is the practical ceiling on how much of a site Google will fetch, set by the smaller of server capacity and Google's interest in the content — and since July 2026, Google's own documentation names those two halves explicitly rather than leaving them implicit. Diagnose with the "Discovered – currently not indexed" report before assuming a budget problem, fix the side that's actually constrained, and keep in mind that shared crawl capacity now also feeds AI systems reading the same pages.
Teams building their own crawling or monitoring pipelines against large sites face a related constraint from the other direction — fetching at scale without getting rate-limited or blocked. Evaluate Nstdata Crawl against your own target pages if that's the problem you're solving.
Further Reading
Sources
Try Nstdata Crawl for large-scale fetching
Bounded, proxy-backed crawling with automatic retries — built for the same reliability constraints crawl budget describes from Google's side.
Try Nstdata for Free →FAQ
Q: Does crawl budget affect small websites?
Rarely in a meaningful way. Below roughly ten thousand crawlable URLs, an unindexed page is more likely a quality, duplication, or internal-linking issue than a genuine crawl budget shortage.
Q: Can I set my crawl budget manually in Search Console?
No. Google removed the manual crawl-rate control; capacity and demand are both algorithmic and adjust automatically based on server health and content signals.
Q: Does noindex save crawl budget?
No. Googlebot must fetch a URL to see the noindex tag, so the request still counts. Removing internal links to the URL or disallowing it in robots.txt are the actual ways to stop Google from spending budget on it.
Q: How is crawl budget different from crawl rate?
Crawl rate (now called the crawl capacity limit) is only the server-side constraint. Crawl budget is the combined result of that capacity limit and Google's separate demand for the content.
Q: What's the fastest way to check if I have a crawl budget problem?
Open Search Console's Page Indexing report and look at "Discovered – currently not indexed." Cross-check it against the Crawl Stats report and server logs before concluding capacity is the bottleneck.
Was this guide helpful?
Your choice is saved on this device.
