Report: Sitemaps Decrease Crawler Response Time

An SEOMoz post has documented statistics on how submitting a Sitemap to Google and Yahoo impacts the rate of crawling that web site. The results show that submitting a Sitemap to Google and Yahoo decreased the time it took Google and Yahoo to crawl the page.

It is important to note that the test was done “on 12 different posts, 6 with Sitemaps being submitted, and 6 with the Sitemaps not being submitted.” The results were that when a “Sitemap was submitted, the average time it took for the bot to visit the new post was 14 minutes for Google and 245 minutes for Yahoo.” In contrast, when “no Sitemap was submitted and the bot had to crawl to the post, it took 1375 minutes for Google and 1773 for Yahoo.” That is a huge difference in crawl time. It is true that many sites see their content indexed within minutes, with or without a sitemap file. But in some cases, it takes longer.

In the old days, I was not a fan of submitting sitemap files for sites that were very search engine friendly. But sometime last year, I became a fan of using sitemap files for search engines. Mainly because it gives me an extra metric in the webmaster tools area to glean information. But in addition, I have been hearing positive results from webmasters after using it over the course of 2008. Finally, I have been noticing that Google is pouring more and more time into these tools and using the data to improve the search results – so why not be there in most cases?

Related Topics: Channel: SEO | Google: SEO | Google: Webmaster Central | SEO: Submitting & Sitemaps


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


SMX - Search Marketing Expo

SearchCap:

Get all the top search stories emailed daily!  

Like This Story? Please Share!

Other ways to share:

Like Our Site? Follow Us!

Subscribe to Our Feed! Join our LinkedIn Group Check out our Tumblr! See us on Pinterest Get Search Engine Land on your mobile device!
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • 1witness

    Barry, great points. I stopped using a sitemap.xml file for one of my sites to test, but could never find time to benchmark results. Does putting your sitemap.xml link inside your robots.txt file help for other search engines read your sitemap such as Yahoo, or should you upload a file to them too?

    Great post!

  • http://www.seroundtable.com/ Barry Schwartz

    I think both work, but you won’t get the data in google webmaster tool without submitting directly, i think.

Get Our News, Everywhere!

 
  • Advertise With Us
 

Click to watch SMX conference video

Join us at an upcoming SMX event:

North America

EMEA

APAC

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.

SMX Site » | SMX Difference » | SMX News »




 

Search Engine Land Periodic Table of SEO Ranking Factors

Get Your Copy
Read The Full SEO Guide