The Latest Information on Googlebots Work
A recent documentation update has brought new information about the work of Googlebots, specifically, their limited operation. Google reported that:Googlebot can crawl the first 15MB of an HTML file or supported text-based file. Any resources referenced in the HTML such as images, videos, CSS, and JavaScript are fetched separately. After the first 15MB of the file, Googlebot stops crawling and only considers the first 15MB of the file for indexing. The file size limit is applied on the uncompressed data. Other crawlers may have different limits. - https://developers.google.com/search/docs/advanced/crawling/googlebotIt means that anything outside the 15 MB file limit will not be included in the ranking calculations, and thus will not be indexed. The information caused some confusion, but John Mueller dispelled some of the doubts on Twitter. He informed that, first of all, this limit is not new - according to him, it existed much earlier, but only now users have been informed about it - and secondly, it applies to the HTML file itself. The limit does not include any resources or content embedded with IMG tags.
