What Is Crawl Budget?

Crawl budget refers to the number of pages Googlebot will crawl on your site within a given period. Google has limited resources — it can’t crawl every page of every website every day. So it prioritises based on a combination of the page’s perceived value and how frequently the site is crawled based on its authority and speed.

Two components affect crawl budget: crawl rate limit (how fast Googlebot crawls to avoid overloading your server) and crawl demand (how much Google wants to crawl your site based on popularity and content freshness).

When Does Crawl Budget Actually Matter?

For most small to medium websites — under 1,000 pages — crawl budget is rarely the primary problem. Google is generally capable of crawling and indexing a well-structured small site without issues.

Crawl budget becomes a real concern when:

• You have a large site (thousands of pages) and notice important pages aren’t getting indexed
• Your site has lots of duplicate content or URL parameters generating near-identical pages
• You’ve recently done a major migration and pages that should be indexed aren’t showing up
• Your GSC Coverage report shows a large number of “Discovered — currently not indexed” URLs

How to Check If Crawl Budget Is a Problem

Start with Google Search Console. Go to Settings → Crawl Stats to see how many pages Googlebot is crawling per day and whether there are any response code errors. If your total crawled pages is consistently low relative to your site size, that’s a signal worth investigating.

Also check your Coverage report. A large volume of “Discovered — currently not indexed” pages compared to your indexed pages often points to crawl or quality issues.

Common Crawl Budget Wasters

URL Parameters

E-commerce and faceted navigation sites often generate hundreds of thousands of near-duplicate URLs through filtering parameters (e.g., ?colour=red&size=large&sort=price). Googlebot can waste enormous amounts of crawl budget on these. Use canonical tags and GSC parameter handling to address this.

Redirect Chains

When Page A redirects to Page B which redirects to Page C, Google has to follow each step. Long redirect chains slow down crawling and can cause Googlebot to give up before reaching the final URL. Clean these up to point directly to the final destination.

Low-Quality or Thin Pages

If you have lots of thin, low-quality pages on your site, Google may deprioritise crawling. This is especially common on sites that have auto-generated category or tag pages with little unique content.

What to Do If You Have Crawl Issues

The most effective things to address crawl budget problems: implement canonical tags correctly on duplicate pages, fix redirect chains, block genuinely useless URLs via robots.txt, and improve the overall quality of your site’s content so Google treats it as worth crawling more frequently.

For most small sites: don’t worry about crawl budget and focus on content quality and basic technical health instead. For larger sites, it’s worth adding to your regular monitoring list.