• Search Engine Land
  • Sections
    • SEO
    • SEM
    • Local
    • Retail
    • Google
    • Bing
    • Social
    • Resources
    • More
    • Home
  • Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • Live
  • More
  • Events
  • SUBSCRIBE

Search Engine Land

Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • More
  • Newsletters
  • Home
SEO

Search Engines Unite On Sitemaps Autodiscovery

Last November, Google, Microsoft and Yahoo united to support sitemaps, a standardized method of submitting web pages through feeds to the search engines. Today, the three are now joined by Ask.com in supporting the system and an extension of it called autodiscovery. This is where the major search engines will automatically locate your sitemaps file […]

Danny Sullivan on April 11, 2007 at 8:07 am
  • More

Last November,

Google
,

Microsoft
and
Yahoo
united to support sitemaps, a
standardized method of submitting web pages through feeds to the search engines.
Today, the three are now joined by Ask.com in supporting the system and an
extension of it called autodiscovery. This is where the major search engines
will automatically locate your sitemaps file if the location is listed in a
robots.txt file. Announcements are up from

Google
and Ask
now Yahoo and
Microsoft.

Information on how to create sitemaps files can be found at the
Sitemaps.org site. Aside from the
sitemaps XML formal, you can also provide RSS 2.0 or Atom 0.3 or 1.0 feeds.
That’s handy for those with blogs that already generate these feeds.

Sitemaps XML files too complicated? Don’t run a blog? Note that the site has
newly expanded information on how you can submit a simple list of URLs in a text
file.

In the past, if you created a sitemaps file, you then had to manually tell
the search engines where to find it. With today’s announcement, search engines
will check your
robots.txt file
for a link to a sitemaps file, then get the file from that
location. This is a big plus because all the major search engines regularly
check robots.txt files as part of their ordinary crawling.

To add the location, just put a line like this anywhere in your robots.txt
file:

Sitemap: LOCATION-OF-SITEMAPS-FILE

Replace the LOCATION-OF-SITEMAPS-FILE with the actual location. For example,
if you ran a site at mywonderfulsite.com and had a sitemaps file called
allmypages.xml in your top level, the reference would be like this:

Sitemap: http://mywonderfulsite.com/allmypages.xml

Have more than one sitemaps file? Ideally, you’d create a special "sitemaps
index" file that links to all of them, then put a link to the sitemaps index
file in your robots.txt file. If that sounds like too much work, you can have
more than one sitemaps URL listed in the robots.txt file.

Aside from autodiscovery, you can also ping Google and Yahoo with the
location of your file. The Sitemaps.org site has more instructions on this in
general. For specifics:

  • Google: See

    here
    . Note that this pinging is different than the pinging Google
    also
    supports
    for blog search.
     
  • Yahoo: See

    here
    and
    here
    . Unlike Google, the same pinging system is used for both web and blog
    search, to my understanding.

Both Google and Yahoo also allow you to manually submit sitemaps files. In
both cases, doing this via their
Google Webmaster Central
or
Yahoo Site Explorer
systems gives you access to specialized monitoring and
reporting tools or information on how they crawl you.

For more about these tools, or how each individual search engine handles
sitemaps files, please see the links below:

  • Google: Google
    Webmaster Central
    or

    sitemaps info
    .
  • Yahoo: Yahoo Site
    Explorer
    or

    sitemaps info
    .
  • Microsoft Live:
    submit page
    or

    submit help info
  • Ask:
    webmaster help info

Keep in mind that Microsoft and Ask are still lacking references to sitemaps
information, but I expect this will change over time.

For related coverage, see here and here on Techmeme.



About The Author

Danny Sullivan
Danny Sullivan was a journalist and analyst who covered the digital and search marketing space from 1996 through 2017. He was also a cofounder of Third Door Media, which publishes Search Engine Land, Marketing Land, MarTech Today and produces the SMX: Search Marketing Expo and MarTech events. He retired from journalism and Third Door Media in June 2017. You can learn more about him on his personal site & blog He can also be found on Facebook and Twitter.

Related Topics

Bing SEOChannel: SEOGoogle: Search ConsoleGoogle: SEO

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter search marketers rely on.

Processing...Please wait.

See terms.

ATTEND OUR EVENTS

Lorem ipsum doler this is promo text about SMX events.

Available On-Demand: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

June 21-22, 2021: SMX Advanced Europe

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

December 14, 2021: SMX Code

Available On-Demand: SMX

Available On-Demand: SMX Report

×


Learn More About Our SMX Events

Discover actionable tactics that can help you overcome crucial marketing challenges. Our next conference will be held:

Next Event: Sept. 14-15, 2021

Available On-Demand: March 2021

Available On-Demand: October 2020

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • Gartner Magic Quadrant for Digital Experience Platforms
  • Selecting a Customer Data Platform For Your Organization: The 2020 Gartner Market Guide
  • The Complete Guide to Web Core Vitals
  • The New Era of Automation in SEO
  • Nielsen Annual Marketing Report: Era of Adaptation
See More Whitepapers

Webinars

  • Drive Customer Engagement with the Power of Personalization
  • 7 Use Cases That Prove Why You Should Implement DAM
  • Accelerate Your SEO & Content Marketing Program with 4 Key Milestones
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

Attend SMX For Only $199

h
Receive daily search news and analysis.

Channels

  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social

Our Events

  • SMX
  • MarTech

Resources

  • White Papers
  • Research
  • Webinars

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok