d-tags 
Loading...Who Came Up With the Crawl Budget?
Interestingly, according to Garry Illyes, the Crawl Budget concept emerged from the Google users themselves. When the buzz about it grew bigger, Crawl Budget was just the resultant of several metrics - it wasn’t an algorithm or rule that handled one purpose. Garry said:…for the longest time, we were saying that we don’t have the concept of Crawl Budget. And it was true [...]. And then we worked with two or three or four teams – I don’t remember – where we tried to come up with at least a few internal metrics that could map together into something that people externally define as Crawl Budget.
What Is the Crawl Budget for Google?
While taking into consideration what Garry said, it appears that Crawl Budget depends on a few practical metrics. Some of them include a limited number of URLs that Googlebot can index without overloading the memory. Gary Illyes and Martin Splitt also confirmed that this is one of the reasons for defining Crawl Budget as a number of URLs that Googlebot can index on a given website.
If you feel like learning more about Crawl Budget, read our blog post: How to Optimize Crawl Budget
Shortly after that, those two guys published a tweet, saying that there are some pieces of online content that aren’t intended for indexing. The reason for that is simple. By doing so, there is more space left for content that is actually relevant and useful for the users. They also revealed that Google doesn’t index, and has never indexed every single piece of content that was published on the internet. Google storage capacity is limited, so the robots can’t index everything people put on their websites.
What Should We Know About Indexing Our Websites?
Just like the Crawl Budget, all the insecurities connected with indexing and non-indexing come from Google users. Gary and Martin pointed out that the majority of website owners - over 90% - shouldn’t be worried at all. They added that the concerns are caused mainly by misinformation, and the urge some people have to control what is uncontrollable.
