Everything you need to know about SEO, delivered every Thursday.
Google Search Descriptions Now Say If GoogleBot Is Blocked
When you block GoogleBot, Google’s web crawler, the page that is blocked can still show up in the Google search results because of links pointing to the page. When that happens, the search listing looks awkward, because Google isn’t able to crawl the page and make a title and description. Instead, Google will make a best guess at the page title and often leaves the description (search snippet) blank.
The message reads:
A description for this result is not available because of this site’s robots.txt – learn more.
This clearly describes that Google is now able to show a description for the page because something is preventing GoogleBot from crawling it. The learn more link takes you to a help document with more details on why this is the case.
Not only does this give the searcher a better search experience, it also provides a webmaster with another notification of site issues outside of Google Webmaster Tools.