In January 2017, Gary Illyes from Google published a document on the Google blog explaining how crawl budget works to webmasters and SEOs. He has now updated the document to make clear that AJAX (i.e. XHR) calls, can and will consume a site’s crawl budget.
The clarification. The update (in italics added here) reads, “Generally, any URL that Googlebot crawls will count towards a site’s crawl budget. Alternate URLs, like AMP or hreflang, as well as embedded content, such as CSS and JavaScript, including AJAX (i.e. XHR) calls, may have to be crawled and will consume a site’s crawl budget. Similarly, long redirect chains may have a negative effect on crawling.”
What is crawl budget? Google defines crawl budget as “the number of URLs Googlebot can and wants to crawl.” In short, crawl budget is “taking crawl rate and crawl demand together.” You can read more about it here.
Did anything change? No, Google just clarified that AJAX/XHR calls do consume crawl budget and that has always been the case.
Why we should care. Generally Google has said time and time again that crawl budget is not something to worry about for most webmasters and SEOs. Google will crawl most sites without an issue. Really large web sites that need to prioritize what Google crawls may want to double check their AJAX/XHR calls to verify if Google is crawling them and more importantly, if they want those calls to take up their site’s crawl budget.
About The Author
Barry Schwartz is Search Engine Land’s News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on SEM topics.