Search Engine Land
  • SEO
    • > All SEO
    • > What Is SEO?
    • > SEO Periodic Table
    • > Google: SEO
    • > Bing SEO
    • > Google Algorithm Updates
  • PPC
    • > All PPC
    • > What is PPC?
    • > Google Ads
    • > Microsoft Ads
    • > The Periodic Tables of PPC
  • Focuses
    • > Local
    • > Commerce
    • > Shopify SEO Guide
    • > Content
    • > Email Marketing Periodic Table
    • > Social Media Marketing
    • > Analytics
    • > Search Engine Land Awards
    • > All Focuses
  • SMX
  • Webinars
  • Intelligence Reports
  • White Papers
  • About
    • > About Search Engine Land
    • > Newsletter
    • > Third Door Media
    • > Advertise

Processing...Please wait.

Search Engine Land » Google » Google: SEO » Googlebot renders, indexes AJAX-style dynamic content driven by XHR POST

Googlebot renders, indexes AJAX-style dynamic content driven by XHR POST

More of your dynamic content can now be indexed by Google.

Detlef Johnson on May 21, 2019 at 9:17 am

Google now renders and indexes client-side AJAX-style JavaScript POST requests. This is good news for those who use modern JavaScript to query online resource APIs and produce dynamic content in pages because such dynamic content is now finally indexed along with the rest of the static content on a page. This is a development with Evergreen Chromium, although some AJAX link crawling was supported earlier. Google deprecated its own previous advice that: “The browser can execute JavaScript and produce content on the fly – the crawler cannot.”

HTTP Background

The HTTP standard describes a number of request types, of which GET is most widely used. Browsers, for example, use GET to retrieve a URL when you type in the URL address bar, or click links and bookmarks etc. The POST request type differs from GET in that it comes with a payload definition that is meant to be unpacked on the server for use with an application program. HTML forms use POST to send text from input fields for processing on the server.

POST Problems

After early experimentation, search engines generally avoid seeding input fields for making POST requests on their own. If a site is created with valuable database content accessible via a site search engine field without easy discovery of links to its results pages, we wouldn’t expect it to get indexed – even by today’s Googlebot. These traditional indexing problems affect pages with client-side XHR POST requests, too.

With the rapid evolution of JavaScript, client-side POST requests have been programmatically available through the browser’s XMLHttpRequest (XHR) object since before the jQuery days. XHR allows JavaScript in the browser to conduct a sub-request for querying an online resource API and retrieve information to produce content “on the fly.”

Some POST Requests Now Work with Google

Google’s new Evergreen Googlebot can now crawl and index XHR POST requests. The question of whether it does or doesn’t was prompted by technical SEO Valentin Pletzer, who follows the Evergreen Googlebot closely, by examining browser support for burgeoning JavaScript features. Be aware that other crawlers still do not have this capability yet.

Proof of Concept

Google developer Martin Splitt at first expressed doubts as to whether XHR POST requests would work with the new Evergreen Googlebot, but curiosity caused him to write a proof of concept and he found that it works. The interesting thing to note is that he didn’t write it using old vanilla JavaScript XHR patterns, and he didn’t write it using jQuery. He used ES6’s new fetch() method. View the source code on Glitch.

There are likely going to be a lot of JavaScript pages written using older patterns, especially jQuery, but it’s definitely more forward looking to use ES6. Logic follows that if Googlebot can interpret ES6 JavaScript patterns for XHR then it can surely interpret older JavaScript patterns for accessing the same browser object.

Things to Note

When Google renders dynamic content driven by the XHR POST request method, each additional sub-request will count against your crawl budget. The content from the POST event isn’t cached as part of the page, which reduces your crawl budget by the number of XHR requests to assemble the page. If you had a crawl budget of 100 pages, for example, and your template for them used one XHR POST request each for content on the fly, it appears that only 50 of your pages would get cached for use with Google’s search index.


New on Search Engine Land

    9 ways to become an SEO problem-solver

    More FAQ rich results being displayed in Google Search

    Webinar: Benchmark your social media performance for a competitive edge

    Google releases May 2022 broad core update

    Spotify, Meta update political ad offerings for 2022 election cycle

About The Author

Detlef Johnson
Detlef Johnson is the SEO for Developers Expert for Search Engine Land and SMX. He is also a member of the programming team for SMX events and writes the SEO for Developers series on Search Engine Land. Detlef is one of the original group of pioneering webmasters who established the professional SEO field more than 25 years ago. Since then he has worked for major search engine technology providers such as PositionTech, managed programming and marketing teams for Chicago Tribune, and advised numerous entities including several Fortune companies. Detlef lends a strong understanding of Technical SEO and a passion for Web development to company reports and special freelance services.

Related Topics

GoogleGoogle: SEOSEO

Get the daily newsletter search marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS

The SMX Conference logo.

Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.

March 8-9, 2022: Master Classes (virtual)

June 14-15, 2022: SMX Advanced (virtual)

November 15-16, 2022: SMX Next (virtual)

Learn More About Our SMX Events

The MarTech Conference logo.

Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.

Start Discovering Now: Spring (virtual)

September 28-29, 2022: Fall (virtual)

Learn More About Our MarTech Events

Webinars

Benchmark Your Social Media Performance For a Competitive Edge

Take a Crawl, Walk, Run Approach to Multi-Channel ABM

Content Comes First: Transform Your Operations With DAM

See More Webinars

Intelligence Reports

Enterprise SEO Platforms: A Marketer’s Guide

Enterprise Identity Resolution Platforms

Email Marketing Platforms: A Marketer’s Guide

Enterprise Sales Enablement Platforms: A Marketer’s Guide

Enterprise Digital Experience Platforms: A Marketer’s Guide

Enterprise Call Analytics Platforms: A Marketer’s Guide

See More Intelligence Reports

White Papers

The State of Influencer Pricing

How to Measure Influencer Performance

Reputation Management For Healthcare Organizations

Unlock the App Marketing Potential of QR Codes

Realising the power of virtual events for demand generation

See More Whitepapers

Receive daily search news and analysis.

Processing...Please wait.

Topics

  • SEO
  • PPC

Our Events

  • Search Marketing Expo - SMX
  • MarTech

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2022 Third Door Media, Inc. All rights reserved.