Google Says Indexing Its Own Results Was Mistake, Will Be Fixed

Over the week, there were various reports of Google going against its own webmaster guidelines, by indexing its own search results. Last night, Google updated its robot.txt file to ensure it blocks its own search results from being in the Google search results. The guidelines read: Use robots.txt to prevent crawling of search results pages or other auto-generated pages that […]

Chat with SearchBot

google-g-logo-fade-1920

Over the week, there were various reports of Google going against its own webmaster guidelines, by indexing its own search results. Last night, Google updated its robot.txt file to ensure it blocks its own search results from being in the Google search results.

The guidelines read:

Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

Reports came from Chris Dyson and then I covered it yesterday and Google’s Gary Illyes commented on my Google+ saying “we’re going to look into what happened here.”

Then it made it on to Hacker News and we asked Google for a comment. Google responds in Google humor, “Indexing the index? We must go deeper!” Adding, “it’s a glitch with multiple slashes in web addresses that we’re working to fix now.”

Indeed, at around 6 p.m. EDT last night, Google updated their robots.txt file at google.com/robots.txt to prevent this from happening.

Here are before and after shots of the problem:

google-index-index-1410435967

google-index-indexs-1410435967

It is not common to find search results from other search engines and especially Google, within the Google search results. At least, Google doesn’t want to offer that as a search experience to its users.


About the author

Barry Schwartz
Staff
Barry Schwartz is a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Get the must-read newsletter for search marketers.