Crawl budget

What is crawl budget?

Crawl budget is the number of pages Googlebot visits on a website in a given period. It depends on how much Google can crawl without overloading your server and on how much Google wants to crawl. According to Google, it is mainly a concern for very large websites.

Why is crawl budget important?

• New pages found faster: Googlebot spends its time on pages that matter.
• Less waste: duplicate pages, filter links and error pages use up crawl capacity.
• Healthier site: a fast server and a clean structure let Google visit more pages.

Examples of crawl budget

• An online store with filters creates thousands of URL variations of the same category.
• A calendar page with an endless series of empty months.
• Long chains of redirects that Googlebot has to follow step by step.

How do you manage crawl budget?

Small and medium sites rarely need to adjust it. If your site is large, block useless URLs in robots.txt, fix error pages, keep your sitemap up to date and improve server speed. This is part of technical SEO.