What is crawl budget?
Crawl budget refers to the number of pages Googlebot will crawl on your site within a given timeframe. It is determined by crawl rate limit (how fast Google can crawl without overloading your server) and crawl demand (how much Google wants to crawl your content).
How to improve crawl efficiency
Crawl budget matters for large sites (10,000+ pages) where Google may not crawl every page regularly. Sites with significant crawl waste — low-value URLs like faceted navigation combinations, parameter spam, soft 404s, and thin pages — spend their budget on pages that add no ranking value.
Improving crawl efficiency means ensuring robots.txt, canonical tags, and your sitemap are aligned so Googlebot focuses on your highest-value pages. For most sites under 1,000 pages, crawl budget is rarely a practical constraint.
Example
Example
An e-commerce site with 15,000 category filter combinations (color+size+material) can waste most of its crawl budget on non-indexable URLs. Disallowing these in robots.txt or using canonical tags concentrates crawling on core category pages.
Frequently asked questions
When does crawl budget actually matter?
Mostly above 10,000 URLs, or on sites generating unbounded URL spaces through facets and parameters. For a 500-page site, crawl budget is almost never the bottleneck.
How do I see how Google spends crawl budget on my site?
The Crawl Stats report in Google Search Console, and server log analysis for the full picture. Logs show exactly which URLs Googlebot fetches and how often, exposing waste that no crawler simulation can.