• Search Engine Land
  • Sections
    • SEO
    • SEM
    • Local
    • Retail
    • Google
    • Bing
    • Social
    • Resources
    • More
    • Home
  • Follow Us
    • Follow
  • Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • Live
  • More
  • Events
    • Follow
  • SUBSCRIBE

Search Engine Land

Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • More
  • Newsletters
  • Home
SEO

Googlebot renders, indexes AJAX-style dynamic content driven by XHR POST

More of your dynamic content can now be indexed by Google.

Detlef Johnson on May 21, 2019 at 9:17 am
  • More

Google now renders and indexes client-side AJAX-style JavaScript POST requests. This is good news for those who use modern JavaScript to query online resource APIs and produce dynamic content in pages because such dynamic content is now finally indexed along with the rest of the static content on a page. This is a development with Evergreen Chromium, although some AJAX link crawling was supported earlier. Google deprecated its own previous advice that: “The browser can execute JavaScript and produce content on the fly – the crawler cannot.”

HTTP Background

The HTTP standard describes a number of request types, of which GET is most widely used. Browsers, for example, use GET to retrieve a URL when you type in the URL address bar, or click links and bookmarks etc. The POST request type differs from GET in that it comes with a payload definition that is meant to be unpacked on the server for use with an application program. HTML forms use POST to send text from input fields for processing on the server.

POST Problems

After early experimentation, search engines generally avoid seeding input fields for making POST requests on their own. If a site is created with valuable database content accessible via a site search engine field without easy discovery of links to its results pages, we wouldn’t expect it to get indexed – even by today’s Googlebot. These traditional indexing problems affect pages with client-side XHR POST requests, too.

With the rapid evolution of JavaScript, client-side POST requests have been programmatically available through the browser’s XMLHttpRequest (XHR) object since before the jQuery days. XHR allows JavaScript in the browser to conduct a sub-request for querying an online resource API and retrieve information to produce content “on the fly.”

Some POST Requests Now Work with Google

Google’s new Evergreen Googlebot can now crawl and index XHR POST requests. The question of whether it does or doesn’t was prompted by technical SEO Valentin Pletzer, who follows the Evergreen Googlebot closely, by examining browser support for burgeoning JavaScript features. Be aware that other crawlers still do not have this capability yet.

Proof of Concept

Google developer Martin Splitt at first expressed doubts as to whether XHR POST requests would work with the new Evergreen Googlebot, but curiosity caused him to write a proof of concept and he found that it works. The interesting thing to note is that he didn’t write it using old vanilla JavaScript XHR patterns, and he didn’t write it using jQuery. He used ES6’s new fetch() method. View the source code on Glitch.

There are likely going to be a lot of JavaScript pages written using older patterns, especially jQuery, but it’s definitely more forward looking to use ES6. Logic follows that if Googlebot can interpret ES6 JavaScript patterns for XHR then it can surely interpret older JavaScript patterns for accessing the same browser object.

Things to Note

When Google renders dynamic content driven by the XHR POST request method, each additional sub-request will count against your crawl budget. The content from the POST event isn’t cached as part of the page, which reduces your crawl budget by the number of XHR requests to assemble the page. If you had a crawl budget of 100 pages, for example, and your template for them used one XHR POST request each for content on the fly, it appears that only 50 of your pages would get cached for use with Google’s search index.

More development tips for SEOs

  • Our SEO for Developers Column
  • Google's Lighthouse is now recommending JavaScript library alternates
  • How to show Lighthouse Scores in Google Sheets with a custom function
  • SEO for reactive JavaScript using React or Vue with NodeJS and other backend stacks


About The Author

Detlef Johnson
Detlef Johnson is the SEO for Developers Expert for Search Engine Land and SMX. He is also a member of the programming team for SMX events and writes the SEO for Developers series on Search Engine Land. Detlef is one of the original group of pioneering webmasters who established the professional SEO field more than 20 years ago. Since then he has worked for major search engine technology providers, managed programming and marketing teams for Chicago Tribune, and consulted for numerous entities including Fortune 500 companies. Detlef has a strong understanding of Technical SEO and a passion for Web programming.

Related Topics

Channel: SEOGoogleGoogle: SEOSEO for Developers

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter search marketers rely on.
See terms.

ATTEND OUR EVENTS

Lorem ipsum doler this is promo text about SMX events.

February 23, 2021: SMX Report

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

October 2021: SMX Advanced Europe

December 17, 2021: SMX Code

Available On-Demand: SMX

×


Learn More About Our SMX Events

Discover actionable tactics that can help you overcome crucial marketing challenges. Our next conference will be held:

MarTech 2021: March 16-17

MarTech 2021: Sept. 14-15

MarTech 2020: Watch On-Demand

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • The State of Local Marketing Report 2020-2021
  • Quality CRM Data: The Key to Delivering Great Customer Experiences
  • How the Microsoft Search Network Can Maximize Your Search Campaigns
  • The Marketer’s Playbook for Customer Acquisition
  • How To Optimize SEO With UGC
See More Whitepapers

Webinars

  • How to Avoid the Digital Transformation Trap
  • How to Build a Marketing System of Record
  • Meet BIMI: The brand-boosting email security marketers must have for 2021
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

h
Receive daily search news and analysis.
Search Engine Land
Download the Search Engine Land App on iTunes Download the Search Engine Land App on Google Play

Channels

  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social

Our Events

  • SMX
  • MarTech

Resources

  • White Papers
  • Research
  • Webinars
  • Search Marketing Expo
  • MarTech Conference

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • Youtube
  • iOS App
  • Google Play

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok