The Test Begins: Do Google Shopping & Other Shopping Search Engines Give You The Best Deals?

google-shopping-products-featured

It sounds pretty damning. Two recent surveys suggest that Google Shopping isn’t leading searchers to the best prices on products. But the surveys weren’t well documented, nor did they include competitors like Bing, Shopzilla, PriceGrabber and Nextag, which have similar issues. So, Search Engine Land will be running its own fully documented tests. Every few days, we’ll search for an item and show what we found, starting with today’s test, a search for a toaster.

Testing For The Low Price Vs. Featured Price

This is a long story. It’s designed to be that way so that people can understand exactly what was tested and the reasoning for selecting prices to compare, which is lacking in the two other surveys. This type of detail is important, especially when lobbing around accusations to the US Federal Trade Commission about potential consumer misleading, as Consumer Watchdog has done.

Take the time to read the full story if you want to understand just how very complicated it is to conduct a survey of this nature. But for those who want a quick summary now, the first test shows that Google actually ranked best in featuring a price that matched the lowest price that could be found from a major merchant:

shopping.xls.pdf__1_page_

The survey also found that all the shopping search engines have issues in terms of disclosing to consumers why they rank results in the way they do. Several have some serious issues with pricing inaccuracies.

Remember, a single test isn’t enough to draw a conclusion across the entire industry. I’m not sure that running six tests as the Financial Times did is enough, nor the 14 that Consumer Watchdog did is, either. But far more important, the how you test and what you count can produce skewing that can mess up even a large sample.

How Much Is That Toaster In The Shopping Search Window?

Why look for a toaster? That’s one of several items where Consumer Watchdog said its survey found a $20 price gap between what was listed on Google and what was listed as the lowest price for the same item at Shopzilla, PriceGrabber or Nextag, when it looked for “Cuisinart Classic 2P Slice Toaster.”

Bing wasn’t surveyed, even though it has shopping search ads exactly like Google. Nor did Consumer Watchdog explain the exact query was used or list a model number for the toaster. Nothing was documented. We were only given the end result, and when it comes to comparing shopping search engines, as you’ll see, it’s essential to show your work.

We can’t search for the exact same toaster as Consumer Watchdog did, because we don’t know the model. Plus, even if we did know it, looking for it might cause some to assume Google’s now “fixed” the toaster price problem for that particular model. So, for our test, we’re seeking a Hamilton Beach 2-Slice Metal Toaster, model 22504. Why? It’s one of the best selling toasters at Amazon, so it seems a popular model worth testing.

Which Price Do You Compare To?

At Google, the search began by entering “Hamilton Beach 2-Slice Metal Toaster” into the main search box. That brought back a special area of the search page with shopping search results, as the arrow points to below:

The results in the box are drawn from Google Shopping, which controversially shifted to an all-ad model last year. Only people who advertise are included in Google Shopping listings. The same is true of all the other shopping search engines in today’s test, by the way. Google isn’t some odd exception.

Already, you can see the challenge in deciding how to survey if the move to all-ads is causing lower prices not to be found. Which price in the results do you count? The $26.99 one from Sears, because it’s first? The $24.28 one from Walmart, because it’s lowest? The $34.99 one from JCPenney, because it’s highest? And are these even all the same model? One’s a different color than the others.

Consumer Watchdog’s survey method was to pick one result out of this box and compare it to the lowest price for a particular item that it could find on another shopping search engine, where it may have “drilled down” into the results in order to find that better price. Or perhaps it didn’t. The survey doesn’t make this clear.

The Financial Times, which did its own survey, instead compared prices listed within these results to what searchers might find at Google itself, if they know how to drill-down to get more listings. Again, which price from the results box was used isn’t clear, which matters when you can’t tell easily if you’re comparing the exact same product.

Four Searches To Verify The Lowest Price At Google

Our survey will use what we think is the cheapest price shown out of the results box for the same exact product, as best we can tell. We’ll also compare to what we’ll call the featured merchant price and finally to the best price from any well-known merchant listed.

To do all this, it’s time to “drill-down” into Google’s shopping results. We already did one initial search, and now we’ll do more in the hunt for the best price. The second search happens by clicking on that “Shop for Hamilton Beach….” link the arrow points to in the screenshot above. Doing that brings up this:

That click brought up a list of various items Google thinks matches the search, drawn from Google Shopping, and results that are still all ads. You just get more of them.

The first listing is for the model we’re after, and clicking on it made the result get larger, with Walmart as the featured merchant offering it at $24.28, before tax and shipping.

Why’s Walmart featured? No idea — nor does the typical consumer know why. It might be they’re paying more. It might be that Google’s complicated system of showing ads based on advertiser “quality score” means Walmart gets a bump. It’s not because Walmart has the cheapest price of the four merchants shown in that box, because in other searches, I know that sometimes the higher price merchant will still get featured.

You can drill-down further, by clicking on the “Compare prices from 50+ stories” link in the box. That constitutes a third search, and doing so brings up this list:

On that list, Walmart still has the lowest base price. But we’re still not being shown all the listings, only the first ten, ranked by whatever mystery criteria Google is using. If you want the lowest price, you have to effectively do a fourth search by clicking on the “Base Price” column heading, which resorts the listings like this:

If you do that, you’ll discover that a merchant called “Unopened Savings” is offering the low, low price of $12.27 for the toaster. But as this is an unrated merchant, a merchant with no recognizable brand, we’re not going to count that for this test. Instead, Walmart still hangs in there with the lowest price.

That’s also a lesson why it makes sense that Google doesn’t just rank merchants by lowest price or even highest customer ratings. Using only those criteria can easily allow extremes that don’t correspond to a good purchase experience to emerge. An unknown retailer offering a too-good-to-be-true price might indeed be too good to be true. A merchant with only a few reviews could easily get the highest customer ratings, based off just a tiny sample. It would be good if Google better explained more about how relevancy is determined for consumers who care, in an easy-to-find location, but the obmission also not abnormal in the shopping search space, either.

Google: Bottom Line, Lowest Price Is Shown

So what does the survey show?

  • Lowest price in ad box: $24.28
  • Lowest featured price in drill down: $24.28
  • Lowest major brand price in drill down: $24.28

For this test, against its own listings, Google did perfect. But are there cheaper prices out there that competing search engines can find?

Bing: No Major Retailers Listed; No Lowest Price

Google’s biggest competitor is Bing. Here’s what you get for the same search there:

Just like Google, Bing only shows shopping results for those who pay, a change it made earlier this year, but without the controversy Google encountered. These ads appear in a box with a similar format to Google’s.

Unlike the search at Google, the results Bing displayed were terrible. None were for the model that we were after. Some aren’t even for the brand or for the specified two-slice capacity.

Changing to a search for “Hamilton Beach 22504″ to add the model number didn’t improve things much. The right toaster finally appeared, but apparently the only place it’s available is through eBay or a small merchant called Hayneedle, with the lowest price being $36.99:

Unlike with Google, there’s no way to “drill down” into Bing Shopping to search for a better price. That’s because Bing completely killed Bing Shopping earlier this year in favor of an all-ad format. What appears in that shopping ad box is all you get.

Of course, Bing would argue that it does provide “free” shopping results that allow for more inclusivity than Google’s model, through a system called “Rich Captions.” See that second arrow above, pointing to a Walmart listing? That’s Walmart appearing in Bing’s results — and for free — but using a system that displays the price of the toaster, $24.31 and in stock.

So, should that price be counted as Bing’s “low price” from its shopping results? Not if you’re trying to run a survey comparing Google’s shopping listings to Bing, as both Consumer Watchdog and the Financial Times did. That’s because they didn’t seem to use any pricing displayed through Google’s similar — and totally free — Rich Snippets system. Here’s an example of those from Google, for a search on ”Hamilton Beach 22504,” below:

Hamilton_Beach_22504_-_Google_Search-2

See how complicated trying to compare shopping search engines can be? With Bing, if you did our original search, you didn’t get pricing either through the ad-block or via Rich Captions, unlike with Google. You certainly didn’t get what Bing promised when it killed its shopping search engine earlier this year, a better experience:

You no longer need to waste time navigating to a dedicated “shopping” experience to find what you’re looking for. Based on your intent, we’ll serve the best results.

You absolutely don’t get the ability to see all the shopping results, to determine if Bing is showing the best price from all those it knows about — which is crucial for measuring Bing against the same accusations that Google faces.

For the purposes of our test, we’re going with the Bing pricing shown in the ad block and tagging that as N/A, as there’s no major direct retailer showing it.

PriceGrabber: Low Prices That Don’t Exist

Over to PriceGrabber, the toaster is listed multiple times, with various prices. As with Google, the challenge is knowing which is the right model for the price comparison:

The first of the listings promising a price “as low as $17.99″ seems good, so let’s drill down by clicking on it:

Where’s that $17.99 price? Oddly, PriceGrabber shoves it way down at the bottom of the page, choosing to push consumers toward K-Mart’s $28.99 listing first. Why? As with Google, who knows. A consumer certainly doesn’t, just as they also probably don’t know that PriceGrabber is almost certainly is using the same all-ads model as Google. I can’t say for certain, because PriceGrabber seems to lack any definitive disclosure page for consumers.

Which price to use for our survey? We could go with the Target price of $17.99, but there’s one problem with that. It doesn’t really exist:

Clicking on the offer opens PriceGrabber up to potential accusations of bait-and-switch, since the price Target is really selling the toaster for is $24.49. Amazon, which has the next lowest price of $19.99, turns out to really be selling it for $24.28. That’s cheaper than the third lowest price from PriceGrabber, Walmart at $24.31. But since PriceGrabber isn’t actually listing the correct Amazon price, for purposes of our survey, we’ll use the lowest correct price it actually shows: Walmart’s, and at the $24.28 actually shown when you go to Walmart’s site. As for the “featured price,” that would be the first price on the original list, Kmart at $28.99.

Nextag: Pushing Higher-Priced Amazon Listing Over Lower-Priced One

Searching for “Hamilton Beach 2-Slice Metal Toaster” at Nextag proved to be difficult. Well, impossible. No matter what I did, that search just generated no results. In the end, I resorted to a search for “Hamilton Beach 22504,” which came back with this:

As with Google — and PriceGrabber — the lowest price isn’t shown first. Instead, sorting is done by “Best Match” order, whatever that is. Nothing on the page explains it. The help page, if consumers go to it, has no help about it — though at least it discloses that Nextag gets paid by merchants to list them, just as Google does.

Changing the sort to lowest price brings this up:

Now we discover that the lowest price from a major retailer, according to Nextag, is Amazon for $23.77, not from Amazon for $29.99, as it first presented. Except, as it turns out, that $23.77 price doesn’t exist. Going to the offer finds the price is really $24.28 — which matches another Amazon listing that Nextag also shows.

For the survey, the low price will be $24.28, while the featured price will be $29.99 — both prices coming from Amazon, which yes, lists the identical product at two different price points.

Shopzilla: Inaccurate Prices Shown

At Shopzilla, “Hamilton Beach 2-Slice Metal Toaster” didn’t find the model we were after, so I did a search for “Hamilton Beach 22504,” which returned this list:

The model appears to be listed three times. If you were to click on any of these listings using the “go to store” link, rather than the small “Compare price at other stores,” you’d pay anywhere between $31.90 to $43.33 for the item. Walmart is listed first at $43.33.

Using Consumer Watchdog’s survey methodology, that’s the assumption of what most consumers would do — click on those links and not try to find better pricing. But let’s try the drill-down. Selecting the first of the listing brought up a page with prices from a variety of merchants:

Shopzilla_-_Hamilton_Beach_22504_Two-Slice_Toaster-3

As with Google, PriceGrabber and Nextag, prices aren’t shown in order of lowest-to-highest. Instead, the now familiar “Best Match” sort order is used. There’s no help page that explains what this means, just as there’s no help page providing any disclosure that all these listings are almost certainly ads.

The “Best Match” sort order means Target, with the lowest price of any major retailer at $17.99, is buried way down on the list. Instead, consumers are pointed first at a Sears offer to buy the toaster for $25.64.

Except that Target low price? It’s really $24.99. And that Sears offer? It’s actually $26.99:

As it turns out, the lowest price from a major retailer is from Walmart, at $24.32 – which actually turns out to be $24.28, when you go to the page.

Wait — wasn’t Walmart the first listing way back on Shopzilla, offering the toaster for $43.33? Yes, it was. Walmart has a one page where it oddly sells the same item for two different prices, cheaper if you buy from it directly and more expensive if you buy through a partner:

Hamilton_Beach_Toaster__22504_-_Walmart.com

For the purposes of our survey, we’re considering the featured price on Shopzilla to be the first price you get when you drill-down into the listings, that from Sears, at $26.99 that’s accurately shown in the landing page. The low price is Walmart at $24.28. Counting the Walmart featured listing before doing the drill-down is difficult, because it isn’t accurately showing the actual price.

The Big Wrap-Up: They All Kind Of Suck

As I said at the outset, we’ll run a few more tests like this. I have no doubt that Google, which came out the best in this one, may not do so well in another. But the bigger issue to me is that the entire state of shopping search feels like a big mess.

It’s pretty clear that there’s not a lot of consumer disclosure going on by anyone. That’s despite the fact that I called the FTC’s attention to this issue last year:

The FTC’s response was that of what I’ve come to expect from an agency that seems ill-prepared and equipped to understand the complicated search engine space. It did nothing more than encourage search engines to pretty-please disclose ads more:

Ironically, Google is coming under fresh fire for not disclosing enough when it seems to do far more than some of the shopping search engines that it’s competing with.

The mess of shopping search also, in my view, weakens the argument that Google somehow would be better if it was pointing people to competing shopping search engines even more. Given that several clearly have issues showing fresh, valid prices, that’s just going to jump consumers through even more hoops and frustration.

Rather, it would be good to see them all step-up efforts to improve relevancy in a space that feels largely forgotten, in terms of improving search relevancy. I’m an expert on search engines, and trying to ferret out what these shopping search engines are showing, and how to use them to double-check to get the best price, is exhausting. I feel sorry for the consumer relying on them for that purpose.

Related Topics: Channel: Retail | FairSearch | Features: Analysis | Google: Google Shopping | Legal: Regulation | Microsoft: Bing Shopping | Search Engines: Shopping Search Engines | Top News

Sponsored


About The Author: is a Founding Editor of Search Engine Land. He’s a widely cited authority on search engines and search marketing issues who has covered the space since 1996. Danny also serves as Chief Content Officer for Third Door Media, which publishes Search Engine Land and produces the SMX: Search Marketing Expo conference series. He has a personal blog called Daggle (and keeps his disclosures page there). He can be found on Facebook, Google + and microblogs on Twitter as @dannysullivan.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://www.orientalsharpton.com/ tiennguyen

    Thank you for putting this together. We work with hundreds of retailers that are listed on Google Shopping and the article was atrocious and provided 0 relevant insight, and is leveraging a “hatred” of Google to try to stir up controversy with no basis at all to its argument.

  • Paresh

    To make it even more complicate: do search for that toaster on Mobile. And where you are searching from may add more complication.

    It is interesting that the Shopping Comparison Engines like to complain when they are not getting as much of “FREE” traffic from Google as before. But these same guys have no problem charging huge sums per click to merchants but don’t want to pay Google on PPC.

    It’s like Microsoft Scroogle, if you complain loudly and enough times it must be true.

  • Movie Munce

    Here’s how you shop online. Type http://www.amazon.com, hit enter, then type what you want. It’s worked for me for over a decade.

  • Durant Imboden

    Your article does a good job of supporting your opinion that “the entire state of shopping search feels like a big mess.” Still, what can one expect from paid advertising? No one should mistake Google Shopping, Nextag, Shopzilla, etc. for Consumer Reports.

    In my opinion, the FTC should just make all of the shopping search engines slap a clearly visible “Advertising” above their results. Problem solved.

  • http://websitecash.net/ Scott McKirahan

    Why in the world would ANYBODY assume that they are going to get the
    best deals from places that are paying for advertising? I assume the
    exact opposite! I thought the original article was one of the dumbest
    articles I ever read (o.k. maybe not the article per se but the position
    of the Consumer Watchdog Group is certainly moronic). I pray for the
    day that people (and especially lawyers) who bring frivolous lawsuits
    will get heavily fined – maybe even disbarred!

    If you want the best prices on something, I guarantee you it will be found from a place
    that IS NOT paying for advertising. The problem is, you’ll never find
    them in Google’s search results anymore. Only the Big Boys that you
    already know about make the top of the search rankings and those who can
    afford to pay to play (which means you are paying extra in the long
    run). Long Live DuckDuckGo!

  • ashutosh rajput

    i am from India……and i think Bing is better i always search shopping deal in Bing…….

  • PStrohm

    “It’s like Microsoft Scroogle, if you complain loudly and enough times it must be true.”

    If its not grassroots no one will believed it.

  • Guest

    Disqus really sucks! On some posts comments work, and on others they don’t. No error message!

  • Martyn Wright

    Following the logic of Consumer Watchdog, if a company pays to advertise on TV but a competitor offers the same product cheaper, the network should not only give them an advertising slot for free, but that slot should be more prominent than the company that has actually paid them?

  • Chris

    I have worked with all the Shopping sites and search engines mentioned here over many years. The sad truth is that they don’t actually want you to find the cheapest price (quickly). They just want you to click as many paid links as possible. It’s more profitable for them to create confusion and multiple clicking back and forth in order to find the right product and the best price. They could do a better job, in fact many of them did a better job in the past, but it doesn’t generate as many clicks or advertising revenue.

    Also the small and vague use of words like “sponsored” and “merchants” means nothing to most people. They should be forced to write the words “ADVERTISMENT” at the top of each section/page just as in printed format where adverts are mixed with editorial content. In the industry we all know what these words mean and that all these sites are paid for but your average Joe has no idea.

  • jos3ph

    Here’s the CamelCamelCamel price tracker view of the toaster in the article:

    http://camelcamelcamel.com/Hamilton-Beach-2-Slice-Toaster-Brushed/product/B001ARQYRU?context=browse

    Amazon’s price has been all over the map (as it is for most products). How often does Google shopping allow price updates?

  • SneakyMedia

    Really enjoyed reading this. I’ve experienced myself, on many occasions, what you called bait-and-switch. It’s extremely frustrating!

    Thanks for tkaing the time to put this together.

  • http://searchengineland.com/ Danny Sullivan

    From this particular test, apparently better than the others. It had the Walmart price right.

  • http://searchengineland.com/ Danny Sullivan

    Sadly, the FTC completely ignored this issue when I raised it in my letter. Hence my conclusion that the agency really doesn’t understand the search engine space it’s supposed to help regulate.

  • http://searchengineland.com/ Danny Sullivan

    Yep. Almost put that in the story — that if I want to know the lowest price, I usually just do an Amazon search myself :)

  • Paresh

    It’s a true and tried method. Ask politicians.

  • http://www.michaelmerritt.org/ Michael Merritt

    For the record, the reason for two different Amazon price points on Nextag is because the higher-priced listing is from an Amazon Marketplace merchant, not Amazon itself.

    Anyway, the first thing I always do when going to one of these places is to sort by lowest price first and then look for reviews.

  • PStrohm

    No, politicians have some followers, scroogle campaign is purely artificial.

  • PStrohm

    No, politicians have some followers, scroogle campaign is purely artificial.

  • atatata

    I use Amazon.

    it’s the best option
    Whenever you search with Google or bing or other aggregators
    you always loss.
    Or you buy product but then mechants give you a call that product is unavailable right now.
    you too expensive shipping or somethign else.
    Amazon is
    A the best price
    B best shipping
    C best support with amazon gurantee
    D best reviews and similar products to make the best choice/decision
    E best product search
    No way Google, Bing,m nexttag and other will be comparable to Amazon
    they do not get what is shopping
    Here in Cupertino we had Google shopping in our local Whole foods market, they are gone by now. They told that nobody shops with Google shopping

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide