Report: Sitemaps Decrease Crawler Response Time
Jan 7, 2009 at 9:32am ET by Barry Schwartz
An SEOMoz post has documented statistics on how submitting a Sitemap to Google and Yahoo impacts the rate of crawling that web site. The results show that submitting a Sitemap to Google and Yahoo decreased the time it took Google and Yahoo to crawl the page.
It is important to note that the test was done “on 12 different posts, 6 with Sitemaps being submitted, and 6 with the Sitemaps not being submitted.” The results were that when a “Sitemap was submitted, the average time it took for the bot to visit the new post was 14 minutes for Google and 245 minutes for Yahoo.” In contrast, when “no Sitemap was submitted and the bot had to crawl to the post, it took 1375 minutes for Google and 1773 for Yahoo.” That is a huge difference in crawl time. It is true that many sites see their content indexed within minutes, with or without a sitemap file. But in some cases, it takes longer.
In the old days, I was not a fan of submitting sitemap files for sites that were very search engine friendly. But sometime last year, I became a fan of using sitemap files for search engines. Mainly because it gives me an extra metric in the webmaster tools area to glean information. But in addition, I have been hearing positive results from webmasters after using it over the course of 2008. Finally, I have been noticing that Google is pouring more and more time into these tools and using the data to improve the search results – so why not be there in most cases?
Share, Bookmark & Discuss This Article
More:
Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter
See more stories like this in the Members Library! Check out the Google: SEO, Google: Webmaster Central, SEO: Submitting & Sitemaps sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!
2 COMMENTS ON Report: Sitemaps Decrease Crawler Response Time
TOP STORIES
SEARCH NEWS BRIEFS
FEATURES & ANALYSIS
RECENT COMMENTS
- ezaaron said " Great post Debra. The idea of a product comparison site is great, as it is great fodder to send to t"
- Grantastic said " Nothing mentioned about Video SEO? Newspaper website are putting out a lot more video, and should ab"
- dmarsch said " Interesting stuff... and if anything, this underestimates the importance of google to News Corp as a"
FREE DAILY SEARCH NEWS RECAP!
Stay on top of all the search news with our daily summary, the SearchCap newsletter. View a sample ›
STAY CURRENT THROUGHOUT THE DAY
Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.
SMX Web Site » | SMX Difference » | SMX News »
Join us at an upcoming SMX event:
- SMX West - March 2-4, 2010
- SMX Munich - March 23-24, 2010
- SMX Toronto - April 8-9, 2010
- SMX Sydney - April 22-23, 2010
- SMX Advanced London - May 17-18, 2010
- SMX Advanced Seattle - June 8-9, 2010
- SMX East - October 4-6, 2010
- SMX Stockholm - October, 2010
Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:
- PPC Advertising & Geo-Targeting - November 12, 2009
- Making Online Marketing Relevant and Responsive: Tactics and A Tool for Success - November 17, 2009
- Ask the Search Engines: War Stories from the World Tour - December 1, 2009
TRACK US SOCIALLY
- Wall Street Journal Recommended Reading
- Ad Age Power 150 Blog
- Technorati Top 100 Blog
- Techmeme Leaderboard Source
- Best Search Engine Community/Forum
Featured sites from our Blogroll
- Shaving Occam – Tony Wright’s Blog
- Brent Csutoras
- An SEO & Web Marketing Blog: Scatterings
- BLADAM: Life, Liberty, Love and Stuff – (Full-text feed)
- WebProNews – eBusiness News, Search News, and Business Videos
Get Your Search Engine Land
Premium Membership!
Become a premium member today and receive:
- Express commenting privileges & photo.
- Exclusive videos & newsletters.
- Discounts to our SMX conferences.
- Access to "How To" & Other Archives.
© 2009 Third Door Media, Inc. All rights reserved. Google, Yahoo & Microsoft logos trademark respective companies and used with permission.
Barry, great points. I stopped using a sitemap.xml file for one of my sites to test, but could never find time to benchmark results. Does putting your sitemap.xml link inside your robots.txt file help for other search engines read your sitemap such as Yahoo, or should you upload a file to them too?
Great post!
Premium member since 01/2009
I think both work, but you won’t get the data in google webmaster tool without submitting directly, i think.