Search Engine Land
  • SEO
    • > All SEO
    • > What Is SEO?
    • > SEO Periodic Table
    • > Google: SEO
    • > Bing SEO
    • > Google Algorithm Updates
  • PPC
    • > All PPC
    • > What is PPC?
    • > Google Ads
    • > Microsoft Ads
    • > The Periodic Tables of PPC
  • Focuses
    • > Local
    • > Commerce
    • > Shopify SEO Guide
    • > Content
    • > Email Marketing Periodic Table
    • > Social Media Marketing
    • > Analytics
    • > Search Engine Land Awards
    • > All Focuses
  • SMX
  • Webinars
  • Intelligence Reports
  • White Papers
  • About
    • > About Search Engine Land
    • > Newsletter
    • > Third Door Media
    • > Advertise

Processing...Please wait.

Search Engine Land » Channel » SEO » IndexNow – new initiative by Microsoft and Yandex to push content to search engines

IndexNow – new initiative by Microsoft and Yandex to push content to search engines

Google seems to currently not be participating in this initiative.

Barry Schwartz on October 18, 2021 at 12:30 pm

Microsoft and Yandex announced a new initiative today named IndexNow, a protocol that any search engine can participate in to enable site owners to have their pages and content instantly indexed by the search engine. Currently, Microsoft Bing and Yandex are the two search engines fully participating in the initiative but others are welcome to adopt this open protocol.

IndexNow allows “websites to easily notify search engines whenever their website content is created, updated, or deleted,” Microsoft wrote on its blog. The goal is to make for a “more efficient Internet,” the company said, by reducing the dependency on search engine spiders having to go out into the web and crawl each URL they find. Instead, the goal is for site owners to push these details and URL changes to the search engines directly. “By telling search engines whether an URL has been changed, website owners provide a clear signal helping search engines to prioritize crawl for these URLs, thereby limiting the need for exploratory crawl to test if the content has changed,” Microsoft wrote.

How it works. The protocol is very simple — all you need to do is create a key on your server, and then post a URL to the search engine to notify IndexNow-participating search engines of the change. The steps include:

  1. Generate a key supported by the protocol using the online key generation tool.
  2. Host the key in text file named with the value of the key at the root of your web site.
  3. Start submitting URLs when your URLs are added, updated, or deleted. You can submit one URL or a set of URLs per API call.

Submit one URL is easy as sending a simple HTTP request containing the URL changed and your key.
https://www.bing.com/IndexNow?url=url-changed&key=your-key and the same would work by using https://yandex.com/indexnow?url=url-changed&key=your-key

You can see more details instructions at the Microsoft Bing IndexNow site or the IndexNow protocol web site.

They work together. If you use the Bing method, then both Bing and Yandex (or other participating search engines) will get the update. You do not need to submit to both Bing and Yandex’s URLs, you just need to pick one and all search engines that are part of this initiative will pick up on the change.

The search engines are sharing this IndexNow system, so if you notify one, that search engine will immediately re-ping each other engine in the background, notifying them all. In fact, it is a requirement of IndexNow that any search engines adopting the IndexNow protocol must agree that submitted URLs will be automatically shared with all other participating search engines. To participate, search engines must have a noticeable presence in at least one market, Microsoft told Search Engine Land.

Similar to Bing URL submission API. Is this similar to the Bing URL submission API? Yes, in that the aim is to reduce crawling requirements and improve efficiency. But, it is different in that this is a completely different protocol. If you are using the Bing URL submission API or the Bing content submission API, technically Bing will get your URLs and content changes immediately but these two APIs do not work with the IndexNow protocol, so the other search engines won’t get the changes.

Will these APIs go away if and when the IndexNow initiative becomes more popular? That is unclear. The URL submission API would be somewhat redundant to IndexNow but the content submission API is unique.

Integrations. IndexNow is gaining support among third-party websites like eBay — as well as Microsoft-owned LinkedIn, MSN and GitHub — to integrate with the IndexNow API. Microsoft said many have adopted the Microsoft Bing Webmaster URL submission API and are planning a migration to IndexNow.

Microsoft said it is encouraging all Web Content Management Systems to adopt IndexNow to help their users get their latest website content immediately indexed and minimize crawl load on their websites. In fact, Microsoft provided WordPress code it can use to integrate IndexNow into its CMS. Wix, Duda and others plan to integrate with IndexNow soon as well. CDNs like CloudFlare and Akamai are also working with the IndexNow protocol and so are SEO tools like Botify, OnCrawl and others.

What about Google. We were told that Google is aware of the IndexNow initiative and the company was asked to participate. At this point Google is not an active IndexNow participant.

Why we care. Instant indexing is an SEO’s dream when it comes to giving search engines the most updated content on a site. Google has been very strict about its applications indexing API, used for job postings and livestream content only now. So while it seems Google may not participate in IndexNow in the near future, search engines like Microsoft Bing and Yandex are aiming to push this initiative hard.

The protocol is very simple and it requires very little developer effort to add this to your site, so it makes sense to implement this if you care about speedy indexing. It seems more and more search engines will participate but in terms of the big one, Google, that remains unclear.


New on Search Engine Land

    What are your secrets to overcoming marketing challenges? Take our survey

    10 reasons to join us at SMX Advanced online this June

    How to ensure influencers help your SEO campaigns

    Webinar: Dominate your competition with Google auction insights and search intelligence

    Vimeo adds structured data to all public videos to improve Google Search visibility

About The Author

Barry Schwartz
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Related Topics

Bing SEOLinkedInSEOTools and SoftwareWordPress

Get the daily newsletter search marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS

Learn actionable search marketing tactics that can help you drive more traffic, leads, and revenue.

March 8-9, 2022: Master Classes (virtual)

June 14-15, 2022: SMX Advanced (virtual)

November 15-16, 2022: SMX Next (virtual)

Learn More About Our SMX Events

Discover time-saving technologies and actionable tactics that can help you overcome crucial marketing challenges.

Start Discovering Now: Spring (virtual)

September 28-29, 2022: Fall (virtual)

Learn More About Our MarTech Events

Webinars

Data-Driven Answers to Achieve Omnichannel Success

Is Your Marketing Stack Ready for Omnichannel CX?

Outrank in Organic Search with These 5 Core Tactics

See More Webinars

Intelligence Reports

Enterprise SEO Platforms: A Marketer’s Guide

Enterprise Identity Resolution Platforms

Email Marketing Platforms: A Marketer’s Guide

Enterprise Sales Enablement Platforms: A Marketer’s Guide

Enterprise Digital Experience Platforms: A Marketer’s Guide

Enterprise Call Analytics Platforms: A Marketer’s Guide

See More Intelligence Reports

White Papers

Realising the power of virtual events for demand generation

The Progressive Marketer’s Ultimate Events Strategy 2022 Worksheet

CMO Guide: How to Plan Smart and Pivot Fast

The Retail Renaissance Report, USA Edition: 4 Keys to Predicting Online & In-Store Demand Across Global Markets

Thinking Bigger About Marketing Budgets

See More Whitepapers

Receive daily search news and analysis.

Processing...Please wait.

Topics

  • SEO
  • PPC

Our Events

  • Search Marketing Expo - SMX
  • MarTech

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2022 Third Door Media, Inc. All rights reserved.