« SearchCap: The Day In Search, April 30, 2007 | Main | April 2007: Search Engine Land's Most Popular Stories »
Apr. 30, 2007 at 9:47pm Eastern by Danny Sullivan
From The Isn't It Ironic Dept: Google Product Search's Results Show Up In Google
Remember how Google said recently that it might crack down on listings pages that are simply search results themselves? Reader Michael Nguyen dropped an email today to point out how, ironically, Google is now listing pages from its own Google Product Search service exactly as it has warned others not to do.
OK, settle down back there, those of you having a chuckle. Embarrassing? Yes! Intentional? Almost certainly not. Let's take a look.
Try a search for snake light, and you'll get this:
See down there at the bottom? Two pages from Google Product Search showing up in the top results:
I can't resist. Let's dig the hole for Google just a bit deeper before I throw down a ladder, so it can climb out.
Google's recent warning about cracking down on search results showing up in its search results IS understandable. I mean, if you've done a search for beard trimmer, who wants to get a bunch of pages that just lead you to shopping sites, like this:
See all the results I've highlighted in red. Click on any of those, and you end up not at pages giving you information about beard trimmers or a particular beard trimmer product. Instead, you just get shopping search results, pages from shopping search engines listing a variety of beard trimmers and prices from merchants across the web.
For example, click on the number nine listing, and you get:
Oh, yeah, um -- shopping results from across the web, courtesy of Google Product Search.
So how did this listing for Google Product Search:
Wind up being listed by Google itself in Google's ordinary results? Is this a new conspiracy to compel searchers to try Google Product Search?
Nah. After all, Google already uses a product OneBox to push people to Google Product Search whenever it wants. With the beard trimmer search, you can see this at the top of the page:
Ah, but people might skip past this, so it's better to be in the "real" results. This is Google perhaps trying to slip a change like this past us.
Heh. Google doesn't need to slip that type of thing past anyone. Google's already started preparing a move for this publicly. Remember, just last week it stopped using OneBox display for news results, putting news into regular results (for more, see here and here. I've yet to see this change myself). This is expected to happen to other specialized Google search results, as well.
The explanation is easy. Google almost certainly forgot to block crawling of Google Product Search results by itself and other crawlers.
Look here at the Google Product Search home page:
See those links below the search box? Those are recent queries people have done (reload the page, and they change to different examples). Click on a link, and they generate search results. Click on them as a search spider, and you'll index search results -- unless you've been blocked.
To block those spiders, Google would need the right entries in its robots.txt file. Let's check!
Oops. No entries. Google does have these:
Disallow: /froogle? Disallow: /froogle_
Those were in there to stop queries from Google's Froogle shopping search engine from being indexed. Unfortunately, Google didn't update these entries to reflect how Froogle was renamed Google Product Search earlier this month.
That renaming shifted product results to this new URL:
http://www.google.com/products
As you can see in this URL for [beard trimmer]:
http://www.google.com/products?q=beard+trimmer
As soon as Google blocks the /products area via the robots.txt file, those 151,000 or so product pages that have been indexed will go away.
I am pinging Google about this, but I know exactly what they'll say. This was an oversight, and the robots.txt file will be updated soon. So go on, you've had your laugh for the night!
|
Like The Story? Vote For It On Yahoo Buzz!
Send me the monthly search newsletter too! (Learn more about our newsletters and feeds) |
Subscribe To Our Search Feed!         
|
| Share & Bookmark This Story! |
By Danny Sullivan
Permalink
Jump To Comments
See Related Stories In: Google: OneBox, Plus Box & Direct Answers, Google: Product Search, Google: SEO, SEO: Blocking Spiders, SEO: Spamming
Reader Comments
They've adjusted their robots.txt now to block "/products?". Those spammers ;)
Even though Vanessa Fox did all of the heavy lifting to mention this to the right people, I can't resist the punchline: Danny, this was an oversight, and the robots.txt file will be updated soon.
In fact, it already is. Thanks for noticing that in switching from Froogle to Product Search, the robots.txt didn't add "/products?" to match the "/froogle?".
SEL is allowing search results to be indexed?
![[TypeKey Profile Page]](http://searchengineland.com/nav-commenters.gif)
|
June 19, 2007 4:04 PMI just came across the original post, dated 4/30/07. I see some comments say the robots file has been updated to disallow the product searches, yet product searches are still showing up (as of today, 7/7/07). For example, do a Google search on: aluminum liner kit
|
July 9, 2007 6:59 PM
That's Hilarious Danny!!! I bet this makes it's rounds in the blogosphere before those results are out of the main index.
Good find.