Irony: If Google Can’t Reach Your Robots.txt File, It Might Not List Your Site
Jun 18, 2008 at 8:34am ET by Barry Schwartz
I reported at the Search Engine Roundtable this morning that Google said if your robots.txt is unreachable, your site might not make it into the Google index. By unreachable, Google means that if your server simply times out and does not return any server response when Googlebot attempts to access your robots.txt file, then it might not include any of your pages in their index.
Googler John Mueller explained that Google tends to lean on the “safe” side when this situation pops up. When I showed this to Danny, he felt it was ironic that if Google can’t read what you want to block, it might block everything. But if you think about it, with all the legal woos Google has to deal with about indexing content, should they risk indexing a site that might have a nofollow directive in their robots.txt file?
It is important to clarify that a robots.txt file is not required in order to be listed with Google. If you don’t have one and Google sees a normal server status response such as a 404 not found, all’s good. It’s only if Google asks for a robots.txt file and gets no response at all where this might be an issue. Rare case, but good to know.
Share, Bookmark & Discuss This Article
More:
Keep Updated: News Via Email | News Via RSS Feed | News Via Twitter
See more stories like this in the Members Library! Check out the Google: SEO, SEO: Blocking Spiders sections of the Members Library where this story is filed. Members also get access to exclusive video content, a members-only weekly & monthly newsletter, plus more. Check out all the benefits!
TOP STORIES
SEARCH NEWS BRIEFS
FEATURES & ANALYSIS
RECENT COMMNENTS
- Buy Advertising said " I've been experimenting with the merger of advertising and entertainment. I think that it can be bot"
- nickstamoulis said " Wow, this is very interesting, I was not aware of the the Google Books case at all, I will be sure t"
- nickstamoulis said " These are all very cool, my personal favorite 4th logo is the Ask.com layout, it is very creative!"
FREE DAILY SEARCH NEWS RECAP!
Stay on top of all the search news with our daily summary, the SearchCap newsletter. View a sample ›
STAY CURRENT THROUGHOUT THE DAY
Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.
SMX Web Site » | SMX Difference » | SMX News »
Join us at an upcoming SMX event:
- SMX Singapore - July 2-3, 2009
- SMX São Paulo - August 4-5
- SMX East - October 5-7, 2009
- SMX Stockholm - 12-13 October, 2009
- SMX Mexico - November 11, 2009
Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:
- How Large Offline Marketers Drive Superior Search Marketing Results - July 21, 2009
- Trademarks, Brand Terms and PPC Advertising: Updates You Must Know - July
22, 2009
- Attribution Management Buyers Guide - July 28, 2009
TRACK US SOCIALLY
- Wall Street Journal Recommended Reading
- Ad Age Power 150 Blog
- Technorati Top 100 Blog
- Techmeme Leaderboard Source
- Best Search Engine Community/Forum
Featured sites from our Blogroll
- Internet Stocks on Seeking Alpha
- Techdirt
- SEOptimise
- johnon.com
- Matt Cutts: Gadgets, Google, and SEO
Get Your Search Engine Land
Premium Membership!
Become a premium member today and receive:
- Express commenting privileges & photo.
- Exclusive videos & newsletters.
- Discounts to our SMX conferences.
- Access to "How To" & Other Archives.
© 2009 Third Door Media, Inc. All rights reserved. Google, Yahoo & Microsoft logos trademark respective companies and used with permission.
