Good News from Google on Crawl Budget and Indexing

01 September 2022

Recent information on Google’s algorithm update has evoked many emotional responses. Truth be told, the impact of Helpful Content on website ranking is still yet to be discovered. However, things are different with the site indexing and Crawl Budget. Undoubtedly, the latter isn’t as mysterious as the Helpful Content update.

News

Kasia Walczyk

2min.

Comments:0

(No Ratings Yet)

On the Google Search Central podcast, Gary Illyes and Martin Splitt clear up common doubts and misconceptions on Crawl Budget. Here is what they said.

Who Came Up With the Crawl Budget?

Interestingly, according to Garry Illyes, the Crawl Budget concept emerged from the Google users themselves. When the buzz about it grew bigger, Crawl Budget was just the resultant of several metrics - it wasn’t an algorithm or rule that handled one purpose. Garry said:

…for the longest time, we were saying that we don’t have the concept of Crawl Budget. And it was true [...]. And then we worked with two or three or four teams – I don’t remember – where we tried to come up with at least a few internal metrics that could map together into something that people externally define as Crawl Budget.

What Is the Crawl Budget for Google?

While taking into consideration what Garry said, it appears that Crawl Budget depends on a few practical metrics. Some of them include a limited number of URLs that Googlebot can index without overloading the memory. Gary Illyes and Martin Splitt also confirmed that this is one of the reasons for defining Crawl Budget as a number of URLs that Googlebot can index on a given website.

If you feel like learning more about Crawl Budget, read our blog post: How to Optimize Crawl Budget

Shortly after that, those two guys published a tweet, saying that there are some pieces of online content that aren’t intended for indexing. The reason for that is simple. By doing so, there is more space left for content that is actually relevant and useful for the users. They also revealed that Google doesn’t index, and has never indexed every single piece of content that was published on the internet. Google storage capacity is limited, so the robots can’t index everything people put on their websites.

What Should We Know About Indexing Our Websites?

Just like the Crawl Budget, all the insecurities connected with indexing and non-indexing come from Google users. Gary and Martin pointed out that the majority of website owners - over 90% - shouldn’t be worried at all. They added that the concerns are caused mainly by misinformation, and the urge some people have to control what is uncontrollable.

How Does Google Distinguish Index Eligible Websites?

After listening to the podcast, one may come to the conclusion that there are some factors favoring one item of content over another. This happens when Google receives signals - e.g. the number of backlinks a website receives - that a particular piece of content is valuable, or when the topic is popular among the users. If you want to know more, listen to Google's podcast: [embed]https://www.youtube.com/watch?v=LIMqmzuzpVg[/embed]