Keeping Google’s index user-friendly and up-to-date requires the search engine giant to send web crawler bots to search the internet for new information.

The Crawl Budget found within Google’s Search Console specifies how many pages of a site are crawled and submitted to the index. The domain popularity and trust rank factor determines the number of pages which are crawled, but there are limits for each website.

The performance of a website determines the crawl rate, so if you have fast loading pages, you respond quickly to your users and the crawl rate increases. When your site has a long load time, the Google bot assumes you host pages on a weak server so, therefore, the crawl rate decreases. This means that fewer content is crawled and submitted to the index.

However, even if your crawl limit is high, it doesn’t necessarily mean Googlebot crawls more URLs of your site. Google still decides this itself, but how exactly can you influence the crawl rate?

The crawl demand

There are two important factors that make Googlebot increase your crawl rate on your site:

Popularity – Popular URLs from large sites are crawled more frequently to keep Google’s index fresh.

Freshness – Google doesn’t like lazy slow websites. So if you work on the pages and keep them fresh, the Googlebot will be alerted and will look more often.

If you combine the crawl rate and the crawl demand, you get the Crawl Budget. If this budget decreases, fewer pages of your site are crawled and they get found less in search result pages.

If you don’t want your Crawl Budget to shrink, you must avoid the following:

• Website / Server Errors (404, page not found error messages for example).
• Hacked or Compromised Pages.
• Infinite Space – Pages without content.
• Bad Content or Spam.
• Confusing Navigation – The navigation must remain user-friendly.
• On-Site duplicate content – same content across different URLs.

To ensure a consistent crawl budget, your site needs to be optimised continuously.

The loading times of each URL play a crucial role. The crawl rate is not a ranking factor by Google because it is not about where you land in the search results, but whether and with how many URL’s you show up.

Why care about it?

The Googlebot determines what, how much and how often it crawls.

However, you can still control the crawl budget by generating good and fresh content which is important for your business and your users. Pages with poor information for the users or error pages can be identified and will be excluded from crawling.