Search Engines Unite On Sitemaps Autodiscovery

Apr 11, 2007 at 8:07am ET by Danny Sullivan

Last November, Google, Microsoft and Yahoo united to support sitemaps, a standardized method of submitting web pages through feeds to the search engines. Today, the three are now joined by Ask.com in supporting the system and an extension of it called autodiscovery. This is where the major search engines will automatically locate your sitemaps file if the location is listed in a robots.txt file. Announcements are up from Google and Ask now Yahoo and Microsoft.

Information on how to create sitemaps files can be found at the Sitemaps.org site. Aside from the sitemaps XML formal, you can also provide RSS 2.0 or Atom 0.3 or 1.0 feeds. That’s handy for those with blogs that already generate these feeds.

Sitemaps XML files too complicated? Don’t run a blog? Note that the site has newly expanded information on how you can submit a simple list of URLs in a text file.

In the past, if you created a sitemaps file, you then had to manually tell the search engines where to find it. With today’s announcement, search engines will check your robots.txt file for a link to a sitemaps file, then get the file from that location. This is a big plus because all the major search engines regularly check robots.txt files as part of their ordinary crawling.

To add the location, just put a line like this anywhere in your robots.txt file:

Sitemap: LOCATION-OF-SITEMAPS-FILE

Replace the LOCATION-OF-SITEMAPS-FILE with the actual location. For example, if you ran a site at mywonderfulsite.com and had a sitemaps file called allmypages.xml in your top level, the reference would be like this:

Sitemap: http://mywonderfulsite.com/allmypages.xml

Have more than one sitemaps file? Ideally, you’d create a special "sitemaps index" file that links to all of them, then put a link to the sitemaps index file in your robots.txt file. If that sounds like too much work, you can have more than one sitemaps URL listed in the robots.txt file.

Aside from autodiscovery, you can also ping Google and Yahoo with the location of your file. The Sitemaps.org site has more instructions on this in general. For specifics:

Google: See here. Note that this pinging is different than the pinging Google also supports for blog search.
Yahoo: See here and here. Unlike Google, the same pinging system is used for both web and blog search, to my understanding.

Both Google and Yahoo also allow you to manually submit sitemaps files. In both cases, doing this via their Google Webmaster Central or Yahoo Site Explorer systems gives you access to specialized monitoring and reporting tools or information on how they crawl you.

For more about these tools, or how each individual search engine handles sitemaps files, please see the links below:

Google: Google Webmaster Central or sitemaps info.
Yahoo: Yahoo Site Explorer or sitemaps info.
Microsoft Live: submit page or submit help info
Ask: webmaster help info

Keep in mind that Microsoft and Ask are still lacking references to sitemaps information, but I expect this will change over time.

For related coverage, see here and here on Techmeme.

Danny Sullivan is editor-in-chief of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also oversees Search Engine Land’s SMX: Search Marketing Expo conference series, maintains a personal blog called Daggle and microblogs on Twitter as @dannysullivan.

See more articles by Danny Sullivan >

Share, Bookmark & Discuss This Article
More:

10 COMMENTS ON Search Engines Unite On Sitemaps Autodiscovery

SueM, April 11th, 2007 at 9:14 am ET:

Thank you. Information was very useful.

Michael Martinez, April 11th, 2007 at 12:09 pm ET:

Danny, this is a great article and I welcome the news with open arms.

Just one request for the future: Would you please say “XML sitemaps” to distinguish these types of files from “HTML sitemaps”, or otherwise indicate the difference (since .TXT files and ATOM feeds can also be used).

People very easily confuse ambiguous references to XML Sitemaps with HTML sitemaps, that many sites include in their standard secondary navigation.

eCopt, April 11th, 2007 at 5:48 pm ET:

Good suggestion Michael.

I have been having to distinguish between the many formats with clients too. It’s understandable since some formats are just beginning to be used in this manner. I guess if you didn’t keep up with it since the first uses of sitemaps, you wouldn’t really know the differences.

Hippo, April 12th, 2007 at 8:17 am ET:

This is wonderful news for site owners and the industry of SEO. It’s nice to see that the search engines are working together and allowing us to “talk to them” in effect.

What a welcome change this industry has had over the last five years or so. I remember attending SES in Boston back then and it seemed as if the attendees were just hanging on every word the search engine reps were saying and they had very little interest in what we were saying.

The times are changing. It’s nice.

Thanks Danny! You’ve been an integral part of “the revolution”.

mcdonaec, April 12th, 2007 at 9:05 am ET:

This is nice coming right after Google added support for embedding KML into sitemaps.

http://googlemapsapi.blogspot.com/2007/01/get-more-traffic-to-your-maps-api-site.html

Mary Bowling, April 12th, 2007 at 1:23 pm ET:

We’re arguing here at my outfit whether we should give the SE’s an XML sitemap telling them about all the pages on a site or simply upload the verification file from Google and then wait to see how they crawl the site and what errors they may find.
There are good arguments on both sides. Do you have an opinion as to which method is best? Thanks

eCopt, April 12th, 2007 at 1:40 pm ET:

Easy, try both. Verify your site with Google and Yahoo and create a valid file type to upload to your robots.txt or use http requests.

I think the greatest advantage of using autodiscovery would be the access it gives to Ask and Live.com. Those were the only two that didn’t have another way to discover your pages other than crawling or manual submit. Now they do.

Both will make you accessible to all SE’s. Why start slow and see what happens, might as well do all you can. Think Vanessa Fox said if you can try to notify SE’s all the ways you can, if possible.

Sean Carlos, April 18th, 2007 at 3:12 am ET:

Ask has since documented sitemap support here: http://about.ask.com/en/docs/about/webmasters.shtml#22 . While the ping mechanism does work, I haven’t seen them actually crawl the file, but that could be attributed to their low overall crawling rate.

Before embracing sitemaps auto-discovery, one should ask the question: is it sensible to facilitate content scraping by providing a list of all of your URLs to the world? I believe it is wiser to stick with manual notification via Google’s Webmaster Dashboard and Yahoo!’s Site Explorer. To the extent there are few major search engines, manual pinging is not a major issue for the others, which for now is limited to Ask as Microsoft has not yet begun supporting sitemaps.

odls, July 15th, 2007 at 4:15 pm ET:

Although submitting both URLs and Sitemaps to Google along with Yahoo is easy enough, I still don`t see how to submit either to Ask, nor a Sitemap to MSN. Or, am I missing something?

Paul, July 17th, 2007 at 4:51 pm ET:

odls: for ask use http://submissions.ask.com/ping?sitemap=http://www.yourdomain.com/sitemap.xml

hope this helps :)

FREE DAILY SEARCH NEWS RECAP!