Search Engines Unite On Sitemaps Autodiscovery

Last November, Google, Microsoft and Yahoo united to support sitemaps, a standardized method of submitting web pages through feeds to the search engines. Today, the three are now joined by Ask.com in supporting the system and an extension of it called autodiscovery. This is where the major search engines will automatically locate your sitemaps file if the location is listed in a robots.txt file. Announcements are up from Google and Ask now Yahoo and Microsoft.

Information on how to create sitemaps files can be found at the Sitemaps.org site. Aside from the sitemaps XML formal, you can also provide RSS 2.0 or Atom 0.3 or 1.0 feeds. That’s handy for those with blogs that already generate these feeds.

Sitemaps XML files too complicated? Don’t run a blog? Note that the site has newly expanded information on how you can submit a simple list of URLs in a text file.

In the past, if you created a sitemaps file, you then had to manually tell the search engines where to find it. With today’s announcement, search engines will check your robots.txt file for a link to a sitemaps file, then get the file from that location. This is a big plus because all the major search engines regularly check robots.txt files as part of their ordinary crawling.

To add the location, just put a line like this anywhere in your robots.txt file:

Sitemap: LOCATION-OF-SITEMAPS-FILE

Replace the LOCATION-OF-SITEMAPS-FILE with the actual location. For example, if you ran a site at mywonderfulsite.com and had a sitemaps file called allmypages.xml in your top level, the reference would be like this:

Sitemap: http://mywonderfulsite.com/allmypages.xml

Have more than one sitemaps file? Ideally, you’d create a special "sitemaps index" file that links to all of them, then put a link to the sitemaps index file in your robots.txt file. If that sounds like too much work, you can have more than one sitemaps URL listed in the robots.txt file.

Aside from autodiscovery, you can also ping Google and Yahoo with the location of your file. The Sitemaps.org site has more instructions on this in general. For specifics:

  • Google: See here. Note that this pinging is different than the pinging Google also supports for blog search.
     
  • Yahoo: See here and here. Unlike Google, the same pinging system is used for both web and blog search, to my understanding.

Both Google and Yahoo also allow you to manually submit sitemaps files. In both cases, doing this via their Google Webmaster Central or Yahoo Site Explorer systems gives you access to specialized monitoring and reporting tools or information on how they crawl you.

For more about these tools, or how each individual search engine handles sitemaps files, please see the links below:

Keep in mind that Microsoft and Ask are still lacking references to sitemaps information, but I expect this will change over time.

For related coverage, see here and here on Techmeme.

Related Topics: Ask: SEO | Channel: SEO | Google: SEO | Google: Webmaster Central | Microsoft: Bing SEO | SEO: Submitting & Sitemaps | Yahoo: SEO | Yahoo: Site Explorer

Sponsored


About The Author: is a Founding Editor of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also serves as Chief Content Officer for Third Door Media, which publishes Search Engine Land and produces the SMX: Search Marketing Expo conference series. He has a personal blog called Daggle (and keeps his disclosures page there). He can be found on Facebook, Google + and microblogs on Twitter as @dannysullivan.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://www.countryroadwebdesigns.com SueM

    Thank you. Information was very useful.

  • http://www.seo-theory.com/ Michael Martinez

    Danny, this is a great article and I welcome the news with open arms.

    Just one request for the future: Would you please say “XML sitemaps” to distinguish these types of files from “HTML sitemaps”, or otherwise indicate the difference (since .TXT files and ATOM feeds can also be used).

    People very easily confuse ambiguous references to XML Sitemaps with HTML sitemaps, that many sites include in their standard secondary navigation.

  • eCopt

    Good suggestion Michael.

    I have been having to distinguish between the many formats with clients too. It’s understandable since some formats are just beginning to be used in this manner. I guess if you didn’t keep up with it since the first uses of sitemaps, you wouldn’t really know the differences.

  • http://www.HippoIMT.com Hippo

    This is wonderful news for site owners and the industry of SEO. It’s nice to see that the search engines are working together and allowing us to “talk to them” in effect.

    What a welcome change this industry has had over the last five years or so. I remember attending SES in Boston back then and it seemed as if the attendees were just hanging on every word the search engine reps were saying and they had very little interest in what we were saying.

    The times are changing. It’s nice.

    Thanks Danny! You’ve been an integral part of “the revolution”.

  • http://www.ioncannon.net/ mcdonaec

    This is nice coming right after Google added support for embedding KML into sitemaps.

    http://googlemapsapi.blogspot.com/2007/01/get-more-traffic-to-your-maps-api-site.html

  • http://www.blizzardinternet.com Mary Bowling

    We’re arguing here at my outfit whether we should give the SE’s an XML sitemap telling them about all the pages on a site or simply upload the verification file from Google and then wait to see how they crawl the site and what errors they may find.
    There are good arguments on both sides. Do you have an opinion as to which method is best? Thanks

  • eCopt

    Easy, try both. Verify your site with Google and Yahoo and create a valid file type to upload to your robots.txt or use http requests.

    I think the greatest advantage of using autodiscovery would be the access it gives to Ask and Live.com. Those were the only two that didn’t have another way to discover your pages other than crawling or manual submit. Now they do.

    Both will make you accessible to all SE’s. Why start slow and see what happens, might as well do all you can. Think Vanessa Fox said if you can try to notify SE’s all the ways you can, if possible.

  • http://www.antezeta.com/google/sitemap-standard.html Sean Carlos

    Ask has since documented sitemap support here: http://about.ask.com/en/docs/about/webmasters.shtml#22 . While the ping mechanism does work, I haven’t seen them actually crawl the file, but that could be attributed to their low overall crawling rate.

    Before embracing sitemaps auto-discovery, one should ask the question: is it sensible to facilitate content scraping by providing a list of all of your URLs to the world? I believe it is wiser to stick with manual notification via Google’s Webmaster Dashboard and Yahoo!’s Site Explorer. To the extent there are few major search engines, manual pinging is not a major issue for the others, which for now is limited to Ask as Microsoft has not yet begun supporting sitemaps.

  • http://www.ourhometools.com odls

    Although submitting both URLs and Sitemaps to Google along with Yahoo is easy enough, I still don`t see how to submit either to Ask, nor a Sitemap to MSN. Or, am I missing something?

  • http://oiloffshoremarine.wordpress.com Paul

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide