Crawl Budget

Crawl budget — the number of pages a search engine bot will crawl on a site within a certain period and the amount of resources it will consume.

Crawl Budget is the total number of pages a search engine bot (e.g. Googlebot) will crawl on a site within a given period, along with the server resources it will consume.

Crawl budget consists of two components:

• Crawl Capacity Limit: The site server’s capacity to handle bot traffic.
• Crawl Demand: The bot’s desire to crawl this site (high for popular sites that are updated frequently).

Crawl budget becomes especially critical in the following cases:

• Large sites with many pages (e-commerce, forums, news)
• Pages with filtering/sorting parameters
• Sites with frequent content updates

To prevent crawl budget waste, block unnecessary URLs with robots.txt, manage duplicate content with canonical tags, and use noindex tags correctly.

Tip: The “Crawl Stats” report in Search Console shows how Googlebot crawls your site. You can identify crawl errors and unnecessary crawl traffic there.