Google Search Descriptions Now Say If GoogleBot Is Blocked

When you block GoogleBot, Google’s web crawler, the page that is blocked can still show up in the Google search results because of links pointing to the page. When that happens, the search listing looks awkward, because Google isn’t able to crawl the page and make a title and description. Instead, Google will make a […]

Chat with SearchBot

When you block GoogleBot, Google’s web crawler, the page that is blocked can still show up in the Google search results because of links pointing to the page. When that happens, the search listing looks awkward, because Google isn’t able to crawl the page and make a title and description. Instead, Google will make a best guess at the page title and often leaves the description (search snippet) blank.

Now, instead of showing a blank search snippet, Google is showing a message that communicates why Google cannot show a search snippet.

Google Snippet Blocked Robots Txt 1345119624

The message reads:

A description for this result is not available because of this site’s robots.txt – learn more.

This clearly describes that Google is now able to show a description for the page because something is preventing GoogleBot from crawling it. The learn more link takes you to a help document with more details on why this is the case.

Not only does this give the searcher a better search experience, it also provides a webmaster with another notification of site issues outside of Google Webmaster Tools.


About the author

Barry Schwartz
Staff
Barry Schwartz is a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Get the must-read newsletter for search marketers.