Google now renders and indexes client-side AJAX-style JavaScript POST requests. This is good news for those who use modern JavaScript to query online resource APIs and produce dynamic content in pages because such dynamic content is now finally indexed along with the rest of the static content on a page. This is a development with Evergreen Chromium, although some AJAX link crawling was supported earlier. Google deprecated its own previous advice that: “The browser can execute JavaScript and produce content on the fly – the crawler cannot.”
HTTP Background
The HTTP standard describes a number of request types, of which GET is most widely used. Browsers, for example, use GET to retrieve a URL when you type in the URL address bar, or click links and bookmarks etc. The POST request type differs from GET in that it comes with a payload definition that is meant to be unpacked on the server for use with an application program. HTML forms use POST to send text from input fields for processing on the server.
POST Problems
After early experimentation, search engines generally avoid seeding input fields for making POST requests on their own. If a site is created with valuable database content accessible via a site search engine field without easy discovery of links to its results pages, we wouldn’t expect it to get indexed – even by today’s Googlebot. These traditional indexing problems affect pages with client-side XHR POST requests, too.
With the rapid evolution of JavaScript, client-side POST requests have been programmatically available through the browser’s XMLHttpRequest (XHR) object since before the jQuery days. XHR allows JavaScript in the browser to conduct a sub-request for querying an online resource API and retrieve information to produce content “on the fly.”
Some POST Requests Now Work with Google
Google’s new Evergreen Googlebot can now crawl and index XHR POST requests. The question of whether it does or doesn’t was prompted by technical SEO Valentin Pletzer, who follows the Evergreen Googlebot closely, by examining browser support for burgeoning JavaScript features. Be aware that other crawlers still do not have this capability yet.
Proof of Concept
Google developer Martin Splitt at first expressed doubts as to whether XHR POST requests would work with the new Evergreen Googlebot, but curiosity caused him to write a proof of concept and he found that it works. The interesting thing to note is that he didn’t write it using old vanilla JavaScript XHR patterns, and he didn’t write it using jQuery. He used ES6’s new fetch() method. View the source code on Glitch.
There are likely going to be a lot of JavaScript pages written using older patterns, especially jQuery, but it’s definitely more forward looking to use ES6. Logic follows that if Googlebot can interpret ES6 JavaScript patterns for XHR then it can surely interpret older JavaScript patterns for accessing the same browser object.
Things to Note
When Google renders dynamic content driven by the XHR POST request method, each additional sub-request will count against your crawl budget. The content from the POST event isn’t cached as part of the page, which reduces your crawl budget by the number of XHR requests to assemble the page. If you had a crawl budget of 100 pages, for example, and your template for them used one XHR POST request each for content on the fly, it appears that only 50 of your pages would get cached for use with Google’s search index.
About The Author
Detlef Johnson is Editor at Large for Third Door Media. He writes a column for Search Engine Land entitled “Technical SEO for Developers.” Detlef is one of the original group of pioneering webmasters who established the professional SEO field more than 20 years ago. Since then he has worked for major search engine technology providers, managed programming and marketing teams for Chicago Tribune, and consulted for numerous entities including Fortune 500 companies. Detlef has a strong understanding of Technical SEO and a passion for Web programming. As a noted technology moderator at our SMX conference series, Detlef will continue to promote SEO excellence combined with marketing-programmer features and webmaster tips.