Google Search Descriptions Now Say If GoogleBot Is Blocked

When you block GoogleBot, Google’s web crawler, the page that is blocked can still show up in the Google search results because of links pointing to the page. When that happens, the search listing looks awkward, because Google isn’t able to crawl the page and make a title and description. Instead, Google will make a best guess at the page title and often leaves the description (search snippet) blank.

Now, instead of showing a blank search snippet, Google is showing a message that communicates why Google cannot show a search snippet.

The message reads:

A description for this result is not available because of this site’s robots.txt – learn more.

This clearly describes that Google is now able to show a description for the page because something is preventing GoogleBot from crawling it. The learn more link takes you to a help document with more details on why this is the case.

Not only does this give the searcher a better search experience, it also provides a webmaster with another notification of site issues outside of Google Webmaster Tools.

Related Topics: Channel: SEO | Google: SEO | Google: Web Search


About The Author: is Search Engine Land's News Editor and owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named Cartoon Barry and he can be followed on Twitter here. For more background information on Barry, see his full bio over here.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Bryan Ceguerra

    This one, is a good example that Google is not de-indexing links.

  • aaronrobb

    And how would you remove the page fully if you can’t control the links?

  • Franklyn Galusha

     We had this very same problem with a new client who for one reason or another did not want certain pages on his website to show up in the results.  Blocking Google’s little robot is more challenging than you might think.

  • w3origin

    Its a serious issue, what is the best way to resolve it ?

  • Joe Youngblood

    make a robots noindex,nofollow meta tag, let the page be indexed. easy cheesy breezy beautiful

  • Joe Youngblood

    google stopped obeying robots.txt last year. this is nothing new, they are just displaying a new message. if the page gets linked or posted to social somewhere gbot might find and index it. if you block gbot it can’t follow index directives. I dont think the title came from the page either, notice it’s not the same title as on the page or even teh same H1. Looks like a link Anchor or surrounding text was used.

  • BrewSEO

    Also, Google will index (and display) a page that is shared on Google+ regardless of it’s Robots status. They said they need to crawl the page to show it’s preview on Google+ 

  • aaronrobb

    Could that be directory-wide?

    And I think it is breezy, yes! :)

  • Joe Youngblood

    unfortunately no. it’s is on a page by page basis. there is no robots.txt noindex /directory command.

    however you can go into webmaster tools and remove the directory. it will get pulled out for 90 days (i think that’s the time frame) and during that time you can add the robots meta tags to the pages. once that is done they will get deindexed.

  • HomeMade14

    Heres a question for ya experts. I am changing my site ( slowly ) from html to php. So, on the older html, i added the no index no follow. The newer php has most of the same content. I thought i would have a problem with duplicute content and maybe have a problem… Any thoughts on what i should do? thanks so much

  • Jorge

    This message about robots.txt is difficult to remove too. I removed our robots.txt file days ago and it still appears that way in search results.

    I even used Google’s Webmaster tools to try to get it crawled again. That was 2 days ago.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide