Judging Relevancy Is Tricky

I’ll go through some examples, but let me start with some caveats. If you dive into some forums, you’ll find plenty of people screaming. People tend to scream after any update that things have gotten worse because they’ve lost rankings. Few scream about how things have improved, much less provide examples. In short, relying just on forums can give you a skewed view.

That doesn’t mean you ignore what you might discover through forums, however. Few know the quality of any search engine’s search results as well as SEOs. They might not be happy if something outranks them, but good ones know if something better or worse has moved up. The same is true even for some of the “good” black hat people out there. They might deliberately violate Google’s guidelines, but that doesn’t mean they don’t know search quality.

Time for examples. To get these results, I was used “Incognito Mode” in Google Chrome to help eliminate personalization other than geographic targeting. That can’t be prevented.

Viagra: Missing Official Site & Many Other Problems

One of the first queries I did to check on the new algorithm’s impact was for “viagra,” because that search is a natural magnet for the type of spam that Google seeks to wipeout with this update. If that was the goal, Google had a big miss here:

Most glaringly, Google fails to list the official Viagra site in the top results, something that Bing gets right.

Postscript: John Andrews pointed out after this was posted that the official site has apparently been gone for 10 days or more, so this particular change seems unrelated to the new spam-fighting algorithm. I had asked Google about oddities with this particular search before posting my article but never heard back.

It’s also a good reminder to add that there’s no guarantee that any of the problems I’ll cover with these examples are due to the algorithm change. There’s no easy way to do a “before-and-after” for these searches to know what’s changed. However, you can look at the current state to assess whether they seem to be of good quality overall, which is part of what Google aims for with this current and all algorithm updates.

Almost as alarming, the four sites I’ve pointed at with red arrows have been hacked so that those visiting them get redirected to some place selling Viagra online. Three of these, Google even knows they’ve been hacked, because it shows a warning to searchers. Why it allows things to rank with those type of warnings makes no sense.

The two blue arrows point to two university web sites that have nothing to do with Viagra but which rank. Chances are, they were also hacked recently and fixed their problems. But if so, they’re still not relevant for Viagra and shouldn’t rank.

Bing: Better For Viagra

Here’s Bing for viagra, for comparison’s sake:

As mentioned, Bing gets the official Viagra site listed tops as you’d expect good search results to do. Bing also lists several informational sites, such as Drugs.com, How Stuff Works and Wikipedia. But all of these are also listed by Google, so there’s no big difference there.

Where Bing really pulls ahead is that instead of all those hacked sites, it lists a lot of online stores and affiliate sites. That’s certainly better than rewarding companies that have hacked other web sites, so it had the edge over Google. But whether these are legit online stores that require prescriptions is another matter.

Postscript: As noted in the comments below, one of these sites Bing lists seems likely to have been hacked.

Make Money Online: Google Has Very Slight Edge

One query I’ve seen go around in discussions, and which was tipped to us through email and in a comment, is for “make money online.” Is Google falling down here?

Honestly, it’s pretty tough to know what should rank well for that type of query, given that it’s so broad and so heavily filled with various schemes that purport on how you can earn money online (the answer, of course, is to create an online sharing service that makes pictures look old and has no income, then sell for 1 billion).

Still, let’s make the attempt. In this case, I’m going to put Google and Bing side-by-side, Google on the left, Bing on the right. I’ve removed the related searches from Bing’s listings and slightly spaced out the listings to even them up with Google, but the order is exactly as originally appeared:

The damning thing some point to with Google is a site on Blogger that has no content at all ranking tops. Yes, that’s terrible. It’s also exactly the same thing that happens on Bing, though Bing ranks it lower on the page (sometimes, Google also ranks it lower). Certainly it would be better for Google if its ranking algorithm wasn’t returning a useless site. But pick any search, on either search engine with any update you care to name. There’s always some weird outlier.

I’ve also seen some call out Google for having two older articles ranking. The same thing happens with Bing, for one of these. It’s easier to notice on Google because it lists dates for these articles. Bing has some outdated information but doesn’t show dates.

Google is heavier on listing blogs about making money, though many of these really just seem to be blogs about blogging. Bing is heavier on places that overtly pitch some “make money” type of courses. If I had to choose what’s better, I’d go with listing the blog sites over the course sites.

In the end, I’d give Google the slight edge here over Bing. None of the results are really that great, of course. But I guess that’s what I expect for that type of search, knowing the flood of “get rich online” content that’s out there.

Overdose Advice? Bing Beats Google

Another example from a comment on our original story was about “how many pills does it take to overdose,” where Google is taken to task for featuring Q&A sites, places like Yahoo Answers, where anyone rather than an actual expert can give advice.

Here’s the side-by-side, Google on the left, Bing on the right. Again, I spaced things out a bit, but the ranking order is the same:

As you can see, both Google and Bing favor Q&A sites. I’d agree, that’s not optimal for this type of search. You’d rather get information from respected medical web sites, and not information on how to commit an overdose but what to do if you suspect you’ve accidentally done that. Neither deliver.

One of the biggest reasons for this failure is probably that the medical web sites simple don’t have content written in this type of plain language. If there’s nothing but garbage out there, garbage is what you’re going to get back.

Still, Bing gets the edge because Google has three completely irrelevant answers coming up, pages with nothing to do with the search topic. It was also disappointing not to see Google kick in the special suicide prevention information that it does offer for some searches.

Payday Loans: A Toss-Up

Another example of Google failure that was suggested is for payday loans, with Check N Go — which I gather is a major known provider — not ranking in the top two pages of results. Let’s do the side-by-side. As I said before, I did some spacing to help make it easier to compare, but no order was changed:

Bing detected my location as being in El Segundo, California — actually about 40 miles from where I live, plus it tossed in a bunch of San Diego listings. That’s a strike against it right off the mark, but I decided to manually set my location to Costa Mesa, California for both search engines, to better even the comparison.

Both lead off with the same major provider of PayDay One. Bing does have Check N Go, while Google lacks that. But Google has many local providers, places I could walk in to, displayed. Bing buries these plus gives me lots of irrelevant local providers, including one in the UK.

It’s so bad with Bing that I kind of want to call this test in Google’s favor. But not knowing this space well, I get the impression that Check N Go probably is a provider it should be listing. Plus, listing what appears to be an affiliate site that probably has fake user reviews also was a knock against Google. So, I left the two tied.

Treadmill Reviews: Bing Gets The Edge

Earlier, I said that SEOs are often subject experts in their own areas. Daniel Deceuster, who commented on the changes, seems to be an example of this. He runs a site about treadmill reviews. The comment he left was full of the type of detail that a non-expert might not pick-up:

I monitor these rankings everyday, have for over a year. They have never been as bad as they are. #4 is an article reviewing just 8 machines from 2008, none of which are available for purchase any longer. Good luck finding anything through #3. Then #5 and #6 both link to each other and are obvious members of a huge internal link scheme. #7 doesn’t even have any reviews, just “discount codes” that aren’t codes but affiliate links that don’t get any discounts. #9 is a massive chart of numbers and ratings of random machines, some of which are no longer available. No reviews and their link to more reviews returns a 404 error.

I decided to take a closer look myself. The side-by-side:

I’m not an expert on treadmill reviews, so there are things Deceuster will see that I’m probably missing. But some of the sites he dislikes that Google is listing? Bing’s listing them, too. He focuses on an outdated article. That is annoying, and it doesn’t happen in this case with Bing. But it might happen in other searches with Bing.

Overall, I’d say that Bing does seem to do a better job of getting more sites that seem to offer fairly nice reviews of treadmills to the top. The one I liked the most, with my non-expert eye, was the one that Bing listed tops and that Google didn’t have at all.

Google Sucks For New Shoes? Not So Fast….

Another popular query I’ve been seeing going around for Google’s results apparently having gotten worse is a search for new shoes. The side-by-side is below, Google on the left, Bing on the right. Related searches and the news box from Bing have been removed, but the ranking order is the same:

The criticism is really about the second listing, which looks like some odd simulation thing that is totally irrelevant. It is relevant. It’s a marketing course called “NewShoes.”

Is it the most relevant thing for Google to list? I’d say not. I’d say most people who are searching for “new shoes” probably aren’t thinking about that course but rather actual shoes. Then again, a lot of people might be searching for a song called “New Shoes,” especially given that the official video has over 5 million views on YouTube. Google lists this, as well as how to buy it on Amazon. Bing doesn’t.

Both list a variety of major name-brand online retailers. Google doesn’t have Shoe Buy; Bing doesn’t have Foot Locker. Both have Zappos (which I’d have expected tops at both). Bing’s got some really specialized retailer that feels odd to reward. Then again, it also found what seems to be a gem of a site, one that lists new shoe release dates. Who knew!

I’d just barely hand this one to Bing, mainly because that marketing course, while relevant, does feel out of place to rank so highly. But on the whole, I don’t think you can look at this set of results and condemn Google’s entire algorithm change as flawed. As I said before, it’s easy to find outliers on any set of results with either Google or Bing.

Who Serves Results for “Autism Resources” Better?

Aunesty Reikofski commented about concerns she had with as search for “autism resources.” That resonated with me. Who wouldn’t want the best results coming up for a search on that?

Reikofski wrote:

Coming at this as a ‘mom’ and not an SEO professional….

Try searching for  “autism resources” #1 position is a hyphenated site that hasn’t been changed since 2006.  A total link directory, and all the national organizations rank below

Let’s do the side-by-side, Google on the left, Bing on the right. Bing has been spaced out to make comparing easier, but there’s no change in the order:

As it turns out, what Reikofski is most upset about, that top ranked site that’s apparently years out of date, it ranks exactly the same as Bing — the search engine she also commented that she would switch to. In fact, Bing lists two pages from the site, whereas Google only lists one. More correctly, Google give it one listing and some smaller “sitelinks,” but that doesn’t deprive some other page for the chance to be in the top results, as happens with Bing. Google also ranks the main national society, as best I can tell, higher than Bing does.

Postscript: Reikofski explains more about her reasoning for switching to Bing below in the comments; it’s less about Bing being more relevant than Google and more about Google being less relevant than she expects.

Google has one page that seems last updated in 2010, but Bing also has a page that seems outdated and kind of AdSense-heavy. Google lists two local organizations that might not be useful to many general searches; Bing lists one like this. Google has two decent-looking info sites that Bing lacks; Bing has a non-profit support group and a Wikipedia page that Google doesn’t have.

I don’t know that Google is really that much better or worse than Bing in this case.

For SEO, Google Edges Out Bing

How about an area that I’m a subject expert in, such as SEO? I’ve written before about how disappointing the results for that search can be, when it seems like getting a bunch of client links or inserting some links into a blog template could rank you well. Nor am I the only one. If that’s all it takes to rank for such a competitive area, those talking about doing white hat SEO best practices can be made to seem stupid by Google allowing this.

Let’s do the side-by-side, Google on the left, Bing on the right. Same as before, I’ve spaced things out on the Bing side, removed related links and a newsbox, so you can compare more easily. The order is untouched:

Several sites I’d consider offering good SEO information are listed in both places. For the first time, Search Engine Land finally even ranks for the term. I think we’re a good site, so that’s an improvement in my books.

I don’t know what moved out. I do know there’s still a lot of irrelevant garbage. An SEO firm near my home is listed. So what? Who buys SEO locally? I get some national SEO firm I’ve never heard of. Some Australian SEO firm that few, if anyone doing this search, is going to use. Another national firm that ranks primarily on the strength of its domain name — but hey, that happens for the same company at Bing, too.

I actually give Google the edge here, and not because we did better. It’s mainly because Bing lists some local SEO company from out of my area, as well as a “secrets” site with no information nor any active form to learn more. The latter is the type of thing some are mocking Google’s latest change as rewarding, as in the case of the “make money online” search. Bing deserves the same criticism, but being Bing, no one notices.

Being As “Bad” As Bing Isn’t Good

Of course, this isn’t really about whether Google is as good as Bing, or if Bing also has the same problems. I’m using Bing as a comparison because you need something to compare the Google results to, otherwise you lose all perspective. But ideally, you’d want the perfect set of results to compare with. Someone let me know where those are!

Really, this is about Google being as best as it says it wants to be. Some of the problems such as lacking the official site for Viagra? That’s pretty embarrassing. A site with no content ranking tops for “make money online?” Even if it happens to Bing, it still hits hard against Google that’s supposed by many to be the search leader.

Also remember that our results are increasingly personalized at both Google and Bing these days. The results above don’t show this, but when I’m logged in, some of the results can be radically different. Of course, some of the glaring problems pointed out above may still be reflected.

Some Perspective

I’ll conclude by trying to clear up some misconceptions and give a little more perspective.

As I said at the beginning, we really have no idea if the change has made Google’s results better or worse. There is no lack of people commenting on how terrible things are now, but like I said, that’s what always tends to dominate forums after any change.

Anyone who is an educated, veteran SEO in the space knows this. Susanna Miles, who commented on my original post, seems to be one of these people:

You know what I wish… I wish that people who weren’t hit or who saw positive change would post in these forums. It ends up looking really hopeless and horrible because it’s only the people who’ve been hurt that bother to do this.

Some people are happy, seeing a positive benefit. For anyone complaining, there’s someone who did better with no incentive to speak up.

I also saw this comment from Sean Paul that I thought was noteworthy:

As usual big corps wins, SMB’s lose out

Looking through the examples above, I don’t see that playing out. I see plenty of small sites doing perfectly fine, and I see some big sites who clearly aren’t. This update seems to have hit businesses of all types.

Over this past week, there’s also been a giant rise in the belief that “negative SEO” is somehow the lurking threat that people must run in panic over, and that this latest update makes it even worse. I’ll be coming back to a future article about that, but if you can’t wait, I’d suggest diving in first with this SEO Book post.

The short answer is this. Yes, there are things that people can do to hurt other people’s rankings, something Google has acknowledged for years, such as in this video last year. No, most people don’t have to worry about it, even with the unatural link warnings that went out recently. Again, I promise to come back to this more.

Recently, Rand Fiskin dared people to prove negative SEO doesn’t work by trying it against his personal website. Separately, someone did get his company web site at SEOmoz to rank for “snuggie dog bed.” I’ve seen this confused as proof negative SEO works.

No, the latter test proves you can still Googlebomb pages for a short period of time. We knew that. Or it proves you can redirect one site to another site to help it inherit rankings. We knew that, too. It doesn’t prove negative SEO can work, but we don’t need that proven. As I said, we already knew that could happen in some specific cases.

I think the most important thing for those who are worried about the latest changes to remember is this. Forget the rankings. Look at your traffic. Give it a day or two or three. Is your overall traffic from Google Search much better than it was before yesterday? You’ve probably gained. No change? The update had no net impact on you. A severe drop? Then yes, you got dinged.

What to do if you were dinged? I’ll be coming back with more about that, on whether a reconsideration request will really help or if there are other things you should try. Obviously, if you’ve been knowingly spamming Google, stop and try to correct that. If you haven’t, and you’re seeing really bad stuff showing up in the search results, it never hurts to do a good blog post illustrating why Google’s results aren’t helping users as well as they could.

That’s always the key. Google’s job is providing great results for searchers. Publishers complaining they’ve been hurt, that doesn’t carry much weight. Someone always does get hurt, as I said. Publishers showing that searchers aren’t getting the best stuff, that’s effective pushback.

I do suspect that Google’s twin goals of greatly increasing relevancy and not rewarding spam haven’t been met with this update. There’s just too much weirdness that I’ve been seeing, which does make me suspect that we’ll see the algorithm changed quickly again in the near future. But that’s really just speculation. I’ll try to get an official reaction from Google on this and other matters soon.

Related Articles

• Infographic: The Google Panda Update, One Year Later
• Pages With Too Many Ads “Above The Fold” Now Penalized By Google’s “Page Layout” Algorithm
• Google Sending Warnings About “Artificial” Or “Unnatural” Links
• Dropped In Rankings? Google’s Mistake Over Parked Domains Might Be To Blame
• What Is SEO / Search Engine Optimization?
• The Periodic Table Of SEO Ranking Factors
• Google Launches “Penguin Update” Targeting Webspam In Search Results
• Penguin Update Recovery Tips & Advice
• Penguin’s Reminder: Google Doesn’t Owe You A Living, So Don’t Depend On It

Searching For Santorum: A New Surprise

I did a search for santorum a few minutes ago, and this is what I got:

See the YouTube link showing up there? It helps illustrate all that I think many people are feeling is wrong with Google right now. It’s a pretty bad result, and it’s also something getting there probably because Google’s not catching some potential old-school search engine spamming.

Universal Search Picked This?

The video result is showing up as part of Google Universal Search. That’s a system that blends content from Google’s various “vertical” or specialized search engines into its regular search results. It’s only supposed to inject this type of specialized content if it’s deemed especially relevant to the search topic.

Certainly, you can imagine that there’s video content relevant to a search on “santorum” from across the web. The Daily Show and The Colbert Report alone have over ten different Santorum comedy clips that might all be relevant.

Beyond comedy, there are news reports from across the entire web. The same search at Bing gives some examples of this, of how video content from Bing Video, as well as Fox News and CNN is inserted into its own search results for “santorum,” as you can see here:

Out of 20,000 potential matches on YouTube, out of 21 million potential video matches across the web, what does Google’s supposedly sophisticated Universal Search algorithm pick out to display as the top video content to be shown within the top search results?

A cartoon created by a company pitching its SEO software on YouTube as a way for Santorum to solve his Google problem. Wow.

You Couldn’t Have Picked….

That’s the most relevant thing that Google can show? I think most people would agree it’s not. I mean seriously, it’s better than these?

• Any of the Colbert Report or Daily Show clips
• Republican presidential candidate Rick Santorum arguing with a student on gay marriage
• Dan Savage explaining how his campaign against Rick Santorum ultimately caused searches on Google and Bing to show a definition as “santorum” being related to anal sex

You Couldn’t Have Caught A 65% Like Ratio?

It’s embarrassing for Google to be doing this. And it’s worse when you look at the views the video has received: only about 2,000, at this point. That’s nothing compared to some of the other clips relevant to santorum, if you’re considering views to be one possible ranking factor. How does this video get such a boost?

Well, there’s another clue when you look at the number of likes the video has received: about 1,300, at this point. That means about 65% of people who viewed the video also liked it, a ratio that is hugely out of proportion to what you normally see.

For example, the classic Honey Badger video — which is hilarious — has a like ratio of 0.5%. How about the classic Double Rainbow video? Hey, 0.5% again. The Bedroom Intruder song? A tiny bit better, 0.6%.

Either this SEO tool video is something like 130x more likeable than any of these other videos or something abnormal is happening — something that you’d think Google’s spam detection systems would have flagged.

Can I Haz My Relevancy Back?

In this particular example, the poor relevancy isn’t caused by any of the ongoing Google+ification of Google. This result is what anyone would see, even if they are logged out of Google. It’s not caused by Search Plus Your World or anything like that.

But Google has spent so much time and energy shoving Google+ into seemingly every nook and cranny that it can find that this type of relevancy screw-up feels like another bit of evidence that Google’s original core mission, delivering awesome search results, is being forgotten.

Related Articles

• Google 2.0: Google Universal Search
• Google Universal Search Expands
• How The “Focus On First” Helps Hide Google’s Relevancy Problems
• Should Rick Santorum’s “Google Problem” Be Fixed?
• Google’s “Gold Standard” Search Results Take Big Hit In New York Times Story
• How Rick Santorum Is Making His “Google Problem” Worse
• After Santorum’s Win, The Daily Show & Colbert Report Laugh Again At His Google Problem
• An Interview With A Google Search Quality Rater
• Pages With Too Many Ads “Above The Fold” Now Penalized By Google’s “Page Layout” Algorithm
• Google Announces “Megasitelinks,” Image Search Improvements & Better Byline Dates
• FAQ: What’s The Debate About Google’s Search Plus Your World?

Page said in the earnings release:

“We had a great quarter,” said Larry Page, CEO of Google.  “Revenue was up 33% year on year and our quarterly revenue was just short of $10 billion. Google+ is now open to everyone and we just passed the 40 million user mark. People are flocking into Google+ at an incredible rate and we are just getting started!”

Last month, Facebook announced 800 million users last month during its F8 conference, and Twitter announced 100 “active” users last month.

Here are some background pieces:

• Facebook Hits 750 Million Users; Zuckerberg Yawns
• Twitter Hits 100 Million ‘Active’ Users.
• StumbleUpon: 20 Million Stumblers & Counting
• Schmidt: Hard To Beat Facebook, If Playing Exactly The Same Game
• Google+ Gains Nearly 10 Million Users In First 2 Days Of Being Open To The Public
• Google & Facebook Bicker Over Invitation Sharing & Number Of Users

Universal Search Loves Video, Images

The most revealing data from the study, which the company provided to us and which may appear on its site soon, is that having a video is one of the best ways of showing up in the top results at Google.

The chart below shows up videos are by far the most found results in Google, with image content a distant second:

After video and image content are map results, news, books, shopping, images and then blogs.

The key point here, if you want to be seen in Google Universal Search, do video!

Google Wins In Universal Search

Searchmetrics also sent the top US based properties that come up under each universal search vertical:

As you can see from this chart, content hosted by Google clearly dominate, ranking tops in four of the five areas:

• YouTube is the number one video site that shows up for video results
• Google Maps is the number one map site that shows up for map results.
• Google Product Search is the number one shopping site that shows up for shopping results
• Google’s Blogger is the number one image site that shows up for image results.

The top rankings will likely raise rumblings about Google favoring itself, an issue in the anti-trust reviews that it is currently undergoing.

How Is Universal Search Defined?

This study defines the universal search results as the sections in the Google search results that read “News for…” or “Shopping results” or “Places for” or “Images for” and so on. The reason Google News is not dominating the news category, Searchmetrics says, is because the URLs in the “News for…” are URLs to the Wall Street Journal, New York Times and other news sites. Very rarely does Google have a link to a news article on their own domain name, at least at this point in time.

In terms of local search, Searchmetrics says that the data is drawn from searches related to large metro areas in the United States and would only be valid in those types of areas.

Correction: Our initial edition of this story had mistakenly reported the video line as the blog line (IE, said that blog content was most popular).

Related Stories

• Google 2.0: Google Universal Search
• Google Universal Search Expands
• Study: Google “Favors” Itself Only 19% Of The Time
• Googleopoly: The Definitive Guide To Antitrust Investigations Against Google

The report covers the US, EU and Japanese markets at a high level. It largely depicts the growth of smartphones across markets, the competition among the various operating systems and some of the demographics, attitudes and behaviors of mobile users.

Outside the US the degree of smartphone penetration is larger in relation to the overall market. However the US smartphone market will likely surpass Europe this year in real terms.

European smartphone owners tend to be older than their counterparts in the US:

For US smartphone owners the operating system has become a very significant purchase consideration, though overall cost (not reflected here) is still a driver of purchase decisions for mobile users.

Below is a comparison of a range of mobile activities in the three regions measured by comScore. Japan leads in most areas except texting and social media.

The now-tired arguments about Europe and Japan being “so much farther along” than the US need to be put to bed. People who make these statements routinely are not really paying attention. We’ve got similar markets but they’re not identical, nor is one a clear predictor of behavior in another.

And here’s perhaps the most striking graphic of the report: the discrepancy in mobile engagement and time spent among the top branded sites in the UK market. Facebook, far and away, is the leader in total engagement with more than 3X the mobile time spent on Google, which is partly to be expected given the different natures of social media and search.

The report ends with a prediction that all the pieces are in place for mobile advertising to take off. The issue is not whether mobile advertising grows at some anticipated rate but whether marketers effectively reach mobile consumers, an increasingly important audience that is more receptive and ready to act, typically, than those using PCs.

As a result of the apparent monitoring, Bing’s relevancy is potentially improving (or getting worse) on the back of Google’s own work. Google likens it to the digital equivalent of Bing leaning over during an exam and copying off of Google’s test.

“I’ve spent my career in pursuit of a good search engine,” says Amit Singhal, a Google Fellow who oversees the search engine’s ranking algorithm. “I’ve got no problem with a competitor developing an innovative algorithm. But copying is not innovation, in my book.”

Bing doesn’t deny Google’s claim. Indeed, the statement that Stefan Weitz, director of Microsoft’s Bing search engine, emailed me yesterday as I worked on this article seems to confirm the allegation:

As you might imagine, we use multiple signals and approaches when we think about ranking, but like the rest of the players in this industry, we’re not going to go deep and detailed in how we do it. Clearly, the overarching goal is to do a better job determining the intent of the search, so we can guess at the best and most relevant answer to a given query.

Opt-in programs like the [Bing] toolbar help us with clickstream data, one of many input signals we and other search engines use to help rank sites. This “Google experiment” seems like a hack to confuse and manipulate some of these signals.

Later today, I’ll likely have a more detailed response from Bing. Microsoft wanted to talk further after a search event it is hosting today. More about that event, and how I came to be reporting on Google’s findings just before it began, comes at the end of this story. But first, here’s how Google’s investigation unfolded.

Postscript: Bing: Why Google’s Wrong In Its Accusations is the follow-up story from talking with Bing. Please be sure to read it after this. You’ll also find another link to it at the end of this article.

Hey, Does This Seem Odd To You?

Around late May of last year, Google told me it began noticing that Bing seemed to be doing exceptionally well at returning the same sites that Google would list, when someone would enter unusual misspellings.

For example, consider a search for torsoraphy, which causes Google to return this:

In the example above, Google’s searched for the correct spelling — tarsorrhaphy — even though torsoraphy was entered. Notice the top listing for the corrected spelling is a page about the medical procedure at Wikipedia.

Over at Bing, the misspelling is NOT corrected — but somehow, Bing manages to list the same Wikipedia page at the top of its results as Google does for its corrected spelling results:

Got it? Despite the word being misspelled — and the misspelling not being corrected — Bing still manages to get the right page from Wikipedia at the top of its results, one of four total pages it finds from across the web. How did it do that?

It’s a point of pride to Google that it believes it has the best spelling correction system of any search engine. Google even claims that it can even correct misspellings that have never been searched on before. Engineers on the spelling correction team closely watch to see if they’re besting competitors on unusual terms.

So when misspellings on Bing for unusual words — such as above — started generating the same results as with Google, red flags went up among the engineers.

Google: Is Bing Copying Us?

More red flags went up in October 2010, when Google told me it noticed a marked rise in two key competitive metrics. Across a wide range of searches, Bing was showing a much greater overlap with Google’s top 10 results than in preceding months. In addition, there was an increase in the percentage of times both Google and Bing listed exactly the same page in the number one spot.

By no means did Bing have exactly the same search results as Google. There were plenty of queries where the listings had major differences. However, the increases were indicative that Bing had made some change to its search algorithm which was causing its results to be more Google-like.

Now Google began to strongly suspect that Bing might be somehow copying its results, in particular by watching what people were searching for at Google. There didn’t seem to be any other way it could be coming up with such similar matches to Google, especially in cases where spelling corrections were happening.

Google thought Microsoft’s Internet Explorer browser was part of the equation. Somehow, IE users might have been sending back data of what they were doing on Google to Bing. In particular, Google told me it suspected either the Suggested Sites feature in IE or the Bing toolbar might be doing this.

To Sting A Bing

To verify its suspicions, Google set up a sting operation. For the first time in its history, Google crafted one-time code that would allow it to manually rank a page for a certain term (code that will soon be removed, as described further below). It then created about 100 of what it calls “synthetic” searches, queries that few people, if anyone, would ever enter into Google.

These searches returned no matches on Google or Bing — or a tiny number of poor quality matches, in a few cases — before the experiment went live. With the code enabled, Google placed a honeypot page to show up at the top of each synthetic search.

The only reason these pages appeared on Google was because Google forced them to be there. There was nothing that made them naturally relevant for these searches. If they started to appeared at Bing after Google, that would mean that Bing took Google’s bait and copied its results.

This all happened in December. When the experiment was ready, about 20 Google engineers were told to run the test queries from laptops at home, using Internet Explorer, with Suggested Sites and the Bing Toolbar both enabled. They were also told to click on the top results. They started on December 17. By December 31, some of the results started appearing on Bing.

Here’s an example, which is still working as I write this, hiybbprqag at Google:

and the same exact match at Bing:

Here’s another, for mbzrxpgjys at Google:

and the same match at Bing:

Here’s one more, this time for indoswiftjobinproduction, at Google:

And at Bing:

To be clear, before the test began, these queries found either nothing or a few poor quality results on Google or Bing. Then Google made a manual change, so that a specific page would appear at the top of these searches, even though the site had nothing to do with the search. Two weeks after that, some of these pages began to appear on Bing for these searches.

It strongly suggests that Bing was copying Google’s results, by watching what some people do at Google via Internet Explorer.

The Google Ranking Signal

Only a small number of the test searches produced this result, about 7 to 9 (depending on when exactly Google checked) out of the 100. Google says it doesn’t know why they didn’t all work, but even having a few appear was enough to convince the company that Bing was copying its results.

As I wrote earlier, Bing is far from identical to Google for many queries. This suggests that even if Bing is using search activity at Google to improve its results, that’s only one of many signals being considered.

Search engines all have ranking algorithms that use various signals to determine which pages should come first. What words are used on the page? How many links point at that page? How important are those links estimated to be? What words appear in the links pointing at the page? How important is the web site estimated to be? These are just some of the signals that both Bing and Google use.

Google’s test suggests that when Bing has many of the traditional signals, as is likely for popular search topics, it relies mostly on those. But in cases where Bing has fewer trustworthy signals, such as “long tail” searches that bring up fewer matches, then Bing might lean more on how Google ranks pages for those searches.

In cases where there are no signals other than how Google ranks things, such as with the synthetic queries that Google tested, then the Google “signal” may come through much more.

Do Users Know (Or Care)?

Do Internet Explorer users know that they might be helping Bing in the way Google alleges? Technically, yes — as best I can tell. Explicitly, absolutely not.

Internet Explorer makes clear (to those who bother to read its privacy policy) that by default, it’s going to capture some of your browsing data, unless you switch certain features off. It may also gather more data if you enable some features.

Suggested Sites

Suggested Sites is one of likely ways that Bing may have been gathering information about what’s happening on Google. This is a feature (shown to the right) that suggests other sites to visit, based on the site you’re viewing.

Microsoft does disclose that Suggested Sites collects information about sites you visit. From the privacy policy:

When Suggested Sites is turned on, the addresses of websites you visit are sent to Microsoft, together with standard computer information.

To help protect your privacy, the information is encrypted when sent to Microsoft. Information associated with the web address, such as search terms or data you entered in forms might be included.

For example, if you visited the Microsoft.com search website at http://search.microsoft.com and entered “Seattle” as the search term, the full address http://search.microsoft.com/results.aspx?q=Seattle&qsc0=0&FORM=QBMH1&mkt=en-US will be sent.

I’ve bolded the key parts. What you’re searching on gets sent to Microsoft. Even though the example provided involves a search on Microsoft.com, the policy doesn’t prevent any search — including those at Google — from being sent back.

It makes sense that the Suggested Sites feature needs to report the URL you’re viewing back to Microsoft. Otherwise, it doesn’t know what page to show you suggestions for. The Google Toolbar does the same thing, tells Google what page you’re viewing, if you have the PageRank feature enabled.

But to monitor what you’re clicking on in search results? There’s no reason I can see for Suggested Sites to do that — if it indeed does. But even if it does log clicks, Microsoft may feel that this is “standard computer information” that the policy allows to be collected.

The Bing Bar

There’s also the Bing Bar — a Bing toolbar — that Microsoft encourages people to install separately from Internet Explorer (IE may come with it pre-installed through some partner deals. When you install the toolbar, by default it is set to collect information to “improve” your experience, as you can see:

The install page highlights some of what will be collected and how it will be used:

“improve your online experience with personalized content by allowing us to collect additional information about your system configuration, the searches you do, websites you visit, and how you use our software. We will also use this information to help improve our products and services.”

Again, I’ve bolded the key parts. The Learn More page about the data the Bing Bar collects ironically says less than what’s directly on the install page.

It’s hard to argue that gathering information about what people search for at Google isn’t covered. Technically, there’s nothing misleading — even if Bing, for obvious reasons, isn’t making it explicit that to improve its search results, it might look at what Bing Bar users search for at Google and click on there.

What About The Google Toolbar & Chrome?

Google has its own Google Toolbar, as well as its Chrome browser. So I asked Google. Does it do the same type of monitoring that it believes Bing does, to improve Google’s search results?

“Absolutely not. The PageRank feature sends back URLs, but we’ve never used those URLs or data to put any results on Google’s results page. We do not do that, and we will not do that,” said Singhal.

Actually, Google has previously said that the toolbar does play a role in ranking. Google uses toolbar data in part to measure site speed — and site speed was a ranking signal that Google began using last year.

Instead, Singhal seems to be saying that the URLs that the toolbar sees are not used for finding pages to index (something Google’s long denied) or to somehow find new results to add to the search results.

As for Chrome, Google says the same thing — there’s no information flowing back that’s used to improve search rankings. In fact, Google stressed that the only information that flows back at all from Chrome is what people are searching for from within the browser, if they are using Google as their search engine.

Postscript: See Google On Toolbar: We Don’t Use Bing’s Searches

Is It Illegal?

Suffice to say, Google’s pretty unhappy with the whole situation, which does raise a number of issues. For one, is what Bing seems to be doing illegal? Singhal was “hesitant” to say that since Google technically hasn’t lost anything. It still has its own results, even if it feels Bing is mimicking them.

Is it Cheating?

If it’s not illegal, is what Bing may be doing unfair, somehow cheating at the search game?

On the one hand, you could say it’s incredibly clever. Why not mine what people are selecting as the top results on Google as a signal? It’s kind of smart. Indeed, I’m pretty sure we’ve had various small services in the past that have offered for people to bookmark their top choices from various search engines.

Google doesn’t see it as clever.

“It’s cheating to me because we work incredibly hard and have done so for years but they just get there based on our hard work,” said Singhal. “I don’t know how else to call it but plain and simple cheating. Another analogy is that it’s like running a marathon and carrying someone else on your back, who jumps off just before the finish line.”

In particular, Google seems most concerned that the impact of mining user data on its site potentially pays off the most for Bing on long-tail searches, unique searches where Google feels it works especially hard to distinguish itself.

Ending The Experiment

Now that Google’s test is done, it will be removing the one-time code it added to allow for the honeypot pages to be planted. Google has proudly claimed over the years that it had no such ability, as proof of letting its ranking algorithm make decisions. It has no plans to keep this new ability and wants to kill it, so things are back to “normal.”

Google also stressed to me that the code only worked for this limited set of synthetic queries — and that it had an additional failsafe. Should any of the test queries suddenly become even mildly popular for some reason, the honeypot page for that query would no longer show.

This means if you test the queries above, you may no longer see the same results at Google. However, I did see all these results myself before writing this, along with some additional ones that I’ve not done screenshots for. So did several of my other editors yesterday.

Why Open Up Now?

What prompted Google to step forward now and talk with me about its experiment? A grand plan to spoil Bing’s big search event today? A clever way to distract from current discussions about its search quality? Just a coincidence of timing? In the end, whatever you believe about why Google is talking now doesn’t really matter. The bigger issue is whether you believe what Bing is doing is fair play or not. But here’s the strange backstory.

Recall that Google got its experiment confirmed on December 31. The next day — New Year’s Day — TechCrunch ran an article called Why We Desperately Need a New (and Better) Google from guest author Vivek Wadhwa, praising Blekko for having better date search than Google and painting a generally dismal picture of Google’s relevancy overall.

I doubt Google had any idea that Wadhwa’s article was coming, and I’m virtually certain Wadhwa had no idea about Google’s testing of Bing. But his article kicked off a wave of “Google’s results suck” posts.

Trouble In the House of Google from Jeff Atwood of Coding Horror appeared on January 3; Google’s decreasingly useful, spam-filled web search from Marco Arment of Instapaper came out on January 5. Multiple people mistakenly reported Paul Kedrosky’s December 2009 article about struggling to research a dishwasher as also being part of the current wave. It wasn’t, but on January 11, Kedrosky weighed in with fresh thoughts in Curation is the New Search is the New Curation.

The wave kept going. It’s still going. Along the way, Search Engine Land itself had several pieces, with Conrad Saam’s column on January 12, Google vs. Bing: The Fallacy Of The Superior Search Engine, gaining a lot of attention. In it, he did a short survey of 20 searches and concluded that Google and Bing weren’t that different.

Time To Talk? Come To Our Event?

The day after that column appeared, I got a call from Google. Would I have time to come talk in person about something they wanted to show me, relating to relevancy? Sure. Checking my calendar, I said January 27 — a Thursday — would be a good time for me to fly up from where I work in Southern California to Google’s Mountain View campus.

The day after that, Bing contacted me. They were hosting an event on February 1 to talk about the state of search and wanted to make sure I had the date saved, in case I wanted to come up for it. I said I’d make it. I later learned that the event was being organized by Wadhwa, author of that TechCrunch article.

A change on Google’s end shifted my meeting to January 28, last Friday. As is typical when I visit Google, I had a number of different meetings to talk about various products and issues. My last meeting of the day was with Singhal and Cutts — where they shared everything I’ve described above, explaining this is one reason why Google and Bing might be looking so similar, as our columnist found.

Yes, they wanted the news to be out before the Bing event happened — an event that Google is participating in. They felt it was important for the overall discussion about search quality. But the timing of the news is being so close to the event is down to when I could make the trip to Google. If I’d have been able to go in earlier, then I might have been writing this a week ago.

Meanwhile, you have this odd timing of Wadhwa’s TechCrunch article and the Bing event he’s organizing. I have no idea if Wadhwa was booked to do the Bing event before his article went out or if he was contracted to do this after, perhaps because Bing saw the debate over Google’s quality kick off and decided it was good to ride it. I’ll try to find out.

In the end, for whatever reasons, the findings of Google’s experiment and Bing’s event are colliding, right in the middle of a renewed focus of attention on search quality. Was this all planned to happen? Gamesmanship by both Google and Bing? Just odd coincidences? I go with the coincidences, myself.

[Postscript: Wadhwa tweeted the event timing was a coincidence. And let me add, my assumption really was that this is all coincidence. I'm pointing it out mainly because there are just so many crazy things all happening at the same time, which some people will inevitably try to connect. Make no mistake. Both Google and Bing play the PR game. But I think what's happening right now is that there's a perfect storm of various developments all coming together at the same time. And if that storm gets people focused on demanding better search quality, I'm happy].

The Search Voice

In the end, I’ve got some sympathy for Google’s view that Bing is doing something it shouldn’t.

I’ve long written that every search engine has its own “search voice,” a unique set of search results it provides, based on its collection of documents and its own particular method of ranking those.

I like that search engines have each had their own voices. One of the worst things about Yahoo changing over to Bing’s results last year was that in the US (and in many countries around the world), we were suddenly down to only two search voices: Google’s and Bing’s.

For 15 years, I’ve covered search. In all that time, we’ve never had so few search voices as we do now. At one point, we had more than 10. That’s one thing I love about the launch of Blekko. It gave us a fresh, new search voice.

When Bing launched in 2009, the joke was that Bing stood for either “Because It’s Not Google” or “But It’s Not Google.” Mining Google’s searches makes me wonder if the joke should change to “Bing Is Now Google.”

I think Bing should develop its own search voice without using Google’s as a tuning fork. That just doesn’t ring true to me. But I look forward to talking with Bing more about the issue and hopefully getting more clarity from them about what they may be doing and their views on it.

Opening image from Real Genius. They were taking a test. There’s no suggestion that Google is cool Chris Knight or that Bing is dorky Kent (or vice versa). It’s a great movie. You can even watch it for free here on Crackle.

Related Stories:

• Google Launches Algorithm To Fight Content Spam
• Google Says Search Quality Improved With New Spam Detection
• Google Sets Sights On Content Farms In 2011
• The New York Times, Demand Media Edition
• Blekko Launches Spam Clock To Keep Pressure On Google
• Google May Let You Blacklist Domains To Fight Spam
• Google vs. Bing: The Fallacy Of The Superior Search Engine
• How The “Focus On First” Helps Hide Google’s Relevancy Problems
• 89% Find Search Engines Do Good Job Finding Information, But “Noise” Is Issue

Postscript: Bing: Why Google’s Wrong In Its Accusations is the follow-up story from talking with Bing. Please be sure to read it in addition to this story.

It seemed a draw between Google and Bing. Both of them listed news about the awards at the top of the page. Both of them had the official site listed first:

Looking more closely, I wanted to give Bing the nod. It listed the main nomination page from the official site in the first position:

Over at Google, it was the Oscar home page listed first, not the nomination page (and within minutes after making my initial review, the nomination page itself started showing in the second position):

However, both the home page and the nomination page give you the exact same information, the nomination list:

So, I declare it a tie. Meanwhile over at Blekko:

The service fails to connect you to the nominations at all. What looks like a link to the nominations page instead brings up an error on the official site:

While that’s partially the Academy Award’s fault, for failing to redirect what’s probably an old location for the list, that still doesn’t excuse Blekko. Google and Bing got it right — Blekko needs to as well, if it’s really going to be competitive.

The survey is from Ben Edelman, an assistant professor at the Harvard Business School. Edelman is a long time Google watcher who regularly publishes interesting and detailed looks at search and advertising-related topics. Edelman has also consulted for Microsoft and is involved in a lawsuit against Google.

Edelman reaches completely different conclusions from his survey than I do, writing at the end:

By comparing results across multiple search engine, we provide prima facie evidence of bias; especially in light of the anomalous click-through rates we describe above, we can only conclude that Google intentionally places its results first.

How have we ended up with such different conclusions? Why, I love Google, of course — and he clearly hates them! Seriously, statistics can easily be turned to whatever you want them to be. I feel like Edelman is turning his study into the most negative view possible. I’m just looking to provide some balance to that.

The Study

Edelman’s study is clever (and one I’m pretty sure has been done a few years ago by others). Search for products that Google offers, and see whether Google lists it own products over those of competitors. Searches included things like:

• mail
• email
• calendar
• chat
• maps
• video

The study did this, for 32 different searches at Google, Yahoo and Bing, back in August 2010.

The Age Issue

Immediately, the age of this testing is a problem. Back in August, Yahoo was still providing its own results. Today, it’s powered by Bing. The study provides no conclusions about what’s happening on Yahoo, right now.

In addition, results change all the time. For some of these queries, I get different results at either Google, Bing or both versus what the study reports. In short, this study says nothing about about the current state at any of these services.

I asked Edelman about this, via email. He agreed, saying:

Your reference to the age of our data is well-taken.  When we started the process, we intended to get this paper out quite a bit faster.  It’s easy enough to redo the results with the newest data; we can, and perhaps we will, depending on response to this piece. But incidentally, it’s also useful to preserve results as they were a few months ago. That will certainly be important in disputes: Folks will be discussing not just what results appear now, but also what was done in the past.

Sure, but a single historic data point still doesn’t necessarily prove anything. As I emailed back:

I agree it’s interesting to know what the results were six months ago for historic reasons, but that’s also just one data point. They might have been much “worse” in terms of Google’s supposed favoring six months before that. Or much less.

The Small Sample

Edelman’s report prominently features a chart that highlights in red how Google apparently favors itself in the top three listings for various terms:

Look at all that red, which indicates when a search engine supposedly favored itself. All that red must make it true! And for Yahoo, too!

Still, drawing a conclusion from only six queries would be foolish. The report is primarily based on a list of 32 searches, which are listed on a separate page. Using the full list, it’s easy to find a chunk that doesn’t portray Google as full of red favoritism:

Even looking at the “larger” sample of 32 queries, it’s still a relatively small dataset. More important, it doesn’t correct for the popularity of the particular services.

Probability Versus Popularity

There’s a section of the report where Edelman talks about doing some regression analysis to disprove that the favoritism that he alleges is just random.

Look, just because someone says they’ve done a regression analysis doesn’t mean that something is statistically correct — even if it sounds that way. The fundamental structure of the test itself might be flawed. If so, all the analysis you do isn’t going to correct for that.

Edelman’s study assumes that any time Google puts one of its own services first, that’s somehow favoritism rather than a reflection of how important the service is in general.

For instance, using Edelman’sown data, there were five instances I found where both Google and Bing put Google’s services in the top position:

• books
• images
• maps
• translate
• video

Crazy — Bing putting Google Maps first! Is it a plot by Microsoft to try and prove how unbiased that it is? Or is it perhaps a reflection that a lot of people use Google Maps — and thus link to it — which in turn can influence the search results at both search engines?

If you wanted to do a really scientific study of “favoritism,” you’d first calculated the relative popularity of each service. Then you might first try to determine if the service is being listed in the order you believe it “should” appear at. If Yahoo Mail is the most popular email service on the web, and Gmail the second, then do they get listed in that order?

Even then, I still think you have issues. Popularity doesn’t always equal relevancy. But I definitely know that, regression analysis aside, there are underlying problems that make these stats suspect.

Search Algorithms Don’t Agree

One of the biggest problems I have with the report is when Edelman writes:

Taking seriously the suggestion that top algorithmic links should in fact present the most relevant and useful results for a given search term, it is hard to see why results would vary in this way across search engines.

It’s not hard to see why search engine result differ at all. Search engine each use their own “algorithm” to cull through the pages they’ve collected from across the web, to decide which pages to rank first. The articles below explain more about this:

• Schmidt: Listing Google’s 200 Ranking Factors Would Reveal Business Secrets
• What Social Signals Do Google & Bing Really Count?
• Google: Now Likely Using Online Merchant Reviews As Ranking Signal
• Dear Bing, We Have 10,000 Ranking Signals To Your 1,000. Love, Google

Google has a different algorithm than Bing. In short, Google will have a different opinion than Bing. Opinions in the search world, as with the real world, don’t always agree.

Indeed, there have been many studies over the past years that have found search results often do not agree. Another reason for this is that the core collection of documents — the “index” — that search engines search through are not exactly the same.

Google Searchers Aren’t Bing Searchers

One factor that both search engines assess is clickthrough. Bing has explicitly said that the number of clicks a listing gets is one of many factors it considers. You can imagine that something listed in the top position would be expected to get a certain percentage of clicks. If it doesn’t, that can be a sign that maybe something else should be there, because searchers are bypassing it.

Google also measures clickthrough. What you click on is used to help personalize the results you see, even if you’re not logged in at Google. My assumption is that Google also uses clickthrough generally for non-personalized results, thought it hasn’t confirmed this.

In either case, clickthrough is not the most important ranking factor. It’s just one of many of them. But it can have an influence. In turn, that might help explain some of the “favoritism” that Edelman saw. If someone’s searching for “maps” on Google, they may be more likely to want Google Maps than Yahoo Maps — and vice versa.

If clickthrough is helping measure this, is that favoritism — or is that ensuring your search algorithm is working best for your particular audience?

So Why’s Yahoo Mail Second?

Edelman did try to account for the potential clickthrough factor, and it’s perhaps the most intriguing part of his report. He used two different sources to gather clickthrough rates on Google, Yahoo and Bing. For email, he found that Gmail was listed first on Google and pulled 29% of the clicks versus Yahoo coming second and getting 54%.

That’s odd. It suggests that searches on Google aren’t being well served by that search — and Edelman counts it as further proof that Google is favoring itself. Then again, it could be….

• Visitors click to Yahoo, don’t want that and immediately come back to Google’s results, something Edelman tells me he didn’t measure
• Lots of searchers were looking for Gmail and realized they could get to it another way after doing the search
• Google’s results are screwy. Google, like all search engines, has screwy results for all types of things

It is intriguing, but that’s also apparently the most damning part of the clickthrough analysis that Edelman could find.

In general, he says only “sometimes” does the second results listed get more clicks than the first. Apparently, it doesn’t happen the majority of times. It would have been useful for him to have shown how many times each service listed itself first and had clickthroughs that supported that.

Hey, Who’s Ranking For “Search Engine?”

Now if you really want to crank up the conspiracy theories, let’s talk about Google’s most important product, search.

Google offers a search engine, just like it offers email, chat and other products. That search engine is still Google’s most important and profitable product, to my knowledge. So what do I get in a search for search engine on Google?

Google doesn’t list its search engine at all. Yes, Google is finally at least getting a spot as a company by having its Google Custom Search service appearing here — but that’s a completely different product that the consumer oriented search engines that it does list like Dogpile, Bing, AltaVista, Ask.com, Yahoo and even tiny Duck Duck Go.

Seriously — Google can’t list Google.com but manages to get Duck Duck Go into its top results? It makes no sense. Having watched this results set for literally years, I’m borderline believing that rather than favoring itself, Google is deliberately downgrading itself here, as a way to show the world how it doesn’t favor itself.

But Google Admits Favoritism!

At the end of the report, Edelman tries to turn a comment by Google’s Marissa Mayer — made when she was vice president of search product and user experience — as further evidence that Google is tampering with its results, as if this further confirms his findings:

At a 2007 conference, Google’s Marissa Mayer commented:

“[When] we roll[ed] out Google Finance, we did put the Google link first. It seems only fair right, we do all the work for the search page and all these other things, so we do put it first… That has actually been our policy, since then, because of Finance. So for Google Maps again, it’s the first link.”

We credit Mayer’s frank admission, and our analysis is consistent with her description of Google’s practices. But we’re struck by the divergence between her statement and Google’s public claims in every other context.

Edelman’s report is about Google’s “algorithmic” results, the “10 blue links” as they’re sometimes called — the meat of the page. Mayer wasn’t talking about algorithmic results. She was talking about Google’s OneBox units, where Google shows results from its various vertical search engines.

Edelman disagrees. From our email exchange:

Look at that last sentence: “For Google maps again, it’s the first link.” “For Google maps again” – that must mean, when a user runs a map-related searches, like “Boston map” or maybe “Boston museum.” But “it’s the first link”? There was never going to be more than one map embedded on the page. Nor, to my knowledge, has Google ever run a map rich result box that linked to multiple different map services. (In contrast, as Marissa mentions, finance rich result boxes do link to multiple finance sites.) So what does Marissa mean when she says “It’s the first link”? I think she means all the other maps sites drop to lower algorithmic search links; Google puts its map service at the top and others below. And others are indeed ordinary algorithmic results, not any kind of special rich result or onebox result.

No, Google Doesn’t “Admit” That

It’s pretty easy to know exactly what Mayer meant. You can watch her answer yourself here at 44:51 seconds into her talk. She’s talking about the first link in the list of sites that appear when you do a stock search on Google and get a OneBox result. Google puts itself first on that list:

See the first arrow? It points to a list of sites that appeared under the current stock price for Google, when I did a search for goog, the Google stock symbol. Mayer was saying that before Google Finance was launched in 2006, links there were ordered by popularity. But after it launched its own service, Google thought it was fair to list itself first on that line.

Look at the second arrow. That’s the first “algorithmic” result that Edelman’s report covers and which he says Mayer’s comment is about. It’s not about that, at all. Indeed, that first algorithmic link goes to Yahoo Finance — not Google Finance. Google Finance comes second in the algorithmic listings.

Beyond The Algo Listings

Talking about the algorithmic listings as if they are somehow independent of the rest of the search page is kind of absurd, of course. Studies have found that they tend to be the results that attract the most clicks. But the days of search results that are only 10 blue links are long gone. People might interact with OneBox and other smart answers that ALL major search engines have long provided.

Is it unfair for Google to “favor itself” by showing me images from its own image search engine rather than Bing’s, as it does here:

Or does that just make sense — that Google also has an image search engine, and it should point people to that? Certainly, Bing does the same:

To me, it just makes sense for Google or any search engine to point to its vertical search engines if it runs them. They obviously believe they have good results for their users there. For them to not to would be like complaining that the New York Times continues to run its own entertainment section rather than including the entertainment section from the Los Angeles Times.

For more on this topic, see these past posts:

• Once Again: Should Google Be Allowed To Send Itself Traffic?
• The Incredible Stupidity Of Investigating Google For Acting Like A Search Engine
• The New York Times Algorithm & Why It Needs Government Regulation
• Mr. Cutts Goes To Washington, Testifies Google Has Integrity
• Deconstructing “Search Neutrality”

Measuring Fairness

Still, some are going to want to measure whether Google is somehow favoring itself. So what do you measure?

Do the “rich” OneBox style results count? Do you count only unpersonalized results, despite the fact that “normal” results these days on Google means personalized results (see Google’s Personalized Results: The “New Normal” That Deserves Extraordinary Attention).

Do you count the things that Edelman’s report considered? Whether Google lists any pages from itself in its top results, or just the top three listings, or whether Google itself first above all else?

To make sense of Edelman’s figures, I tried to keep it simple. I went to his full list of 32 searches that were conducted. I found that there were 11 searches in total where Google listed itself first above all others. I then looked to see in which of these cases that Bing also listed Google first. That happened five times, which I mentioned above. That left these six searches where Google — and only Google — “favored” itself

• academic article
• blog
• email
• finance
• mail
• scholarly journals

So in 6 our of 32 cases, Google would seem to have favored its own products in a way that might raise eyebrows – 19% of the time. After all, if Bing’s going to list Google first in those other cases, counting these “against” Google doesn’t seem fair.

Even in these cases, the stats still don’t reflect that competitors may be very visible. Even if Google listed itself first for “email” — however it happened — is it really anti-competitive when its competitor is prominently listed in second place? Wouldn’t it really be more of a concern if Google didn’t list its competitors at all?

Google’s Response

Shortly after I published this, I also received an unsolicited statement from Google (it will commonly send statements around to journalists, if a story or study is making the rounds. Here you go:

Mr. Edelman is a longtime paid consultant for Microsoft, so it’s no surprise that he would construct a highly biased test that his sponsor would pass and that Google would fail.  Google never artificially favors our own services in our organic web search results, and we perform extensive user testing to ensure that search results are ranked in a way that provides users with the most useful answer.

Google also sent some examples that contradict the idea that Google favors itself. Among them?

• search engine
• book flights
• directions

Hey, didn’t I mention that search engine example! Indeed, I did – along with some other things that Google is pointing out, such as taking the popularity of services into account or what searchers at particular services might prefer.

We Want Relevancy, Not Regulation

A search engine’s core algorithm results have long been seen in some quarters as “editorial” content that shouldn’t be tampered with to favor anything other than what’s best for the searcher. Not every search engine has followed this practice, of course. In my experience, Google has been the best at it, refusing to fix things by hand even when it should (see Google, Bing & Searching For The New Wikileaks Website).

I have a great fear of governments stepping in to dictate what a search engine’s listings should be. To me, that’s akin to telling a newspaper what it can report on, or trying to regulate opinions anywhere. There are no “perfect” search results, nor will you ever find a set that’s “neutral.” An algorithm, ultimately, is an opinion. Opinions are not neutral.

What we really should be concerned with isn’t whether Google (or any search engine) is being “fair” but rather if it is providing relevant answers. I can remember when Lycos favored itself so much back in the late 1990s that it was difficult to do a search that didn’t lead you back into Lycos.

Where’s Lycos today? Right. Relevancy will attract and retain users. If Google or any search engine isn’t providing relevant results, the market will likely correct things. And in fact, that’s what a recent survey found. People are far more concerned about getting more relevant results than government regulation of results. See the two articles below, for more on this:

• Survey: 77% Of Americans Oppose Search Engine Regulation
• 89% Find Search Engines Do Good Job Finding Information, But “Noise” Is Issue

Rasmussen Reports surveyed 740 adult Americans on January 4-5 about a variety of search engine related issues. The key question that caught my eye?

“In terms of finding what information you need, how do you rate today’s Internet search engines like Google, Yahoo and Bing …excellent, good, fair or poor?”

Most Rate Search Engines Well

In total, 89% found that search engines do a good or excellent job in finding information. Here’s the full breakdown:

• 47% – Excellent
• 42% – Good
• 10% – Fair
• 0% – Poor (technically between 0% and 1%, but specific figure not given)
• 1% – Not sure

Does that mean Google itself is gaining such high marks? Maybe these are all Bing users? Unlikely. The survey didn’t ask which search engine people used, which was unfortunate. It did ask if people used more than one search engine at the same time. Few did:

“Do you generally use the same Internet search engine all the time?”

• 78% – Yes
• 19% – No
• 3% – Not sure

Since Google is by far the most popular search engine in the US, it’s reasonable to assume that the overall satisfaction numbers indicated overall satisfaction with Google.

But Noise Is An Issue

If search engines are doing such a great job in general, and Google in particular, why have we seen a spate of posts recently suggesting that Google’s gotten worse? I think the answer is in another question from the poll:

“Which is a bigger problem when you use an Internet search engine – that you can’t find what you need or that your query generates too much irrelevant data?”

• 70% – That your query generates too much irrelevant data
• 13% – That you can’t find what you need
• 18% – Not sure

Only 13% say they can’t find what they’re looking for. The answers are there, the “signal” that people want to tune into. They’re just surrounded by a lot of noise.

Another Poll With Seemingly Conflicting Findings

I think you see a similar frustration in a poll that Lifehacker just ran. This gathered nearly 10,000 responses to the question:

“Have Google’s Search Results Become Less Useful To You?”

• 43.8% – Kind of/sort of, but it’s still the best way to get at the good stuff
• 33.8% – Absolutely. The spammers have gained a significant foothold
• 11.2% – I haven’t really noticed a change
• 7.1% – I’d say no, or not to the point where it matters, at least
• 3.6% – No, and actually, my results have been better and more convenient lately
• 0.6% – Other

The headline on Lifehacker’s poll results story was “Over 77 Percent of Lifehacker Readers Say Google’s Search Results are Less Useful Lately,” which combined the two most popular responses, one that is totally negative and one that can be read either way (results are less useful, but Google’s still the best way to find things).

However, “less useful” doesn’t mean “useless” or that people aren’t continuing to find things with Google on a regular basis. To better illustrate this, you could also do a headline from the same poll saying that 2/3rd agree that Google results are still useful. Consider the side-by-side charts below:

The first chart combines the negative and mixed responses into the “Yes” slice. The second chart combines the positive, neutral and mixed responses into the “Yes” slice. Same poll, seemingly different conclusions — unless you understand that growing noise (and frustration with it) doesn’t mean that Google has suddenly become useless.

Good Remains Good Enough

Last year, I did a long look at Google’s results in my story, How The “Focus On First” Helps Hide Google’s Relevancy Problems. The point of that was to illustrate what the poll numbers above are showing: good is good enough for Google to win. As I wrote:

In a few minutes, give me a query, and I can usually find at least one result that doesn’t match the quality you’d expect to be in the first page of results on Google. If it’s an area I’m an expert it, I can do it even faster — and find more outliers. And if you go to the second page of results, it can sometimes be laughable. Google survives this because for the most part, a few good answers are good enough.

If Google really were as bad as some anecdotal accounts suggest, you’d see a loss of users, in my opinion. That’s based on watching the search engine space for 15 years now, observing what’s helped players rise and fall. Instead, Google continues to remain well above its closest competitor. One measuring company, comScore, just reported Google having record usage in the US.

But Google Comes Under More Scrutiny Now

By the way, Bing and Blekko also get by on “good is good enough.” As with Google, I can easily find irrelevant results on them without breaking a sweat. But they aren’t coming under the fire that Google’s starting to take because, in my view, they’re enjoying the underdog benefits that previously Google enjoyed.

As I wrote in my Blekko Launches Spam Clock To Keep Pressure On Google story last week:

There was a press love affair when Google first came out. There continues to be a consumer love affair, in my mind, that the Google brand on search results can make them seem better. There have been several studies in the past where just putting the Google logo on someone else’s results will make a consumer think the results are superior.

I think we’re finally seeing this slip back on Google. Just as its achievements were inflated into super-greatness, now its results are blown-up into huge failures.

You can see this in Paul Kedrosky’s piece from this week, when he writes:

We could get better algorithms, which is happening to some degree, with search engines like Blekko and others

I like Paul, a lot. I like Blekko and the folks over there, a lot. But make no mistake. Blekko absolutely does not have a better search algorithm than Google. It has a different search algorithm that is used against a completely different collection of documents than Google — and one that is probably only a couple billion pages in total (if that) versus Google dealing with tens of billions of pages.

No one knows who has the better search algorithm. Blekko, for all the attention it has gained, still has tons of learning to do — and the folks at Blekko have said as much, even as they ride the latest “Google sucks” wave.

Perception Isn’t Reality

Personally, I’ve felt that Google’s search quality hasn’t been as good as in the past. But my gut feel might be wrong. I remember the negative experiences far more than I recall the many times that Google works extremely well for me. I also use Google far more than I use Bing or Blekko — which also means I’m less likely to notice things that go wrong at those search engines. I also search for things I never would have tried in the past. My expectations have grown, over time.

I don’t think I’m unusual. For those who’ve written recently of Google’s “decline” in search quality, I think they’re all regular Google users and mainly writing from their guts. None of the posts I’ve seen appear to have done any robust testing of queries on Google and compared those to Bing, much less tried to measure changes in relevancy at either search engine over time.

Testing Needed, But Testing Is Tough

I’ve written several times recently — as I’ve done years ago — that what we really need are batteries of tests runs against the major search engines, in a way that they’d both agree are fair. Nor is our column earlier this week — Google vs. Bing: The Fallacy Of The Superior Search Engine from Conrad Saam — the type of testing I’m talking about.

That column looked at only 20 different queries and found Bing slightly ahead. Pick a different 20, and Google might have “won.” I think you need to run many more queries than that.

In addition, do you discount if either Google or Bing changes the results based on your search history or your location? That might not make sense, if such systems are inherently designed to provide better results for individuals.

Do you discount “smart” answers or vertical search results like news headlines that may appear, focusing only on the “natural” results, those classic “10 blue links?” That’s not how a typical searcher interacts with results.

No One Really Knows

Relevancy testing was complicated enough 10 years ago when results were simpler. Today, figuring out a system is even more complex. Our column wasn’t trying to designate a “winner” in the search sweepstakes but rather point out the fallacy of anyone declaring that one search engine is “superior” to another. The honest answer is that we really don’t know. We don’t have the metrics to assess that.

I’ll be revisiting the relevancy metrics challenge in the future, plus talking with the major search engines about whether there’s a way to go forward with the idea, so that we’re not relying on someone’s ego search or anecdotal accounts to decide which search engine has the best results. There’s got to be a better way than that.

For more on this subject, see some of these past stories from us:

• Reviewing Some Bad Google Search Results With Sergey Brin
• The Google Sewage Factory, In Action: The Chocomize Story
• How The “Focus On First” Helps Hide Google’s Relevancy Problems
• Blekko Launches Spam Clock To Keep Pressure On Google

To understand what “great results” means, transport yourself back 10+ years ago and try to remember the search user experience – one dominated by short tailed queries and multiple paginated results until you could find what you were really looking for. We didn’t complain because it was free and much better than that old dewey decimal approach we learned in middle school. In fact, it was magical.

Enter Google and the experience and expectations changed significantly. On the strength of superior results, Google’s market share skyrocketed. Over the past decade we’ve grown accustomed to results so accurate that we rarely peek beyond the top three. Yet, I have trouble believing that Google’s dominance continues to be based on superior search results, especially given the financial and human resources thrown at improving search across multiple competitors.

Danny Sullivan questioned Google search relevancy last year:  How the “Focus on First” Helps Hide Google’s Relevancy Problems. And as far back as 2007, Chris Sherman wrote a review of PC World’s search engine shoot out called Stop the Presses! Google Bested in Search Shoot-out!

My personal search approach uses Google as the default while using other sites for specialty searches. On Bing, image search is far superior and Wikipedia for 101 style information. Is this a factor of inertia or am I really getting a better experience with Google? I set up an objective small sample size evaluation of search quality between Google and Microsoft to see for myself.

Let’s Test The Theory

Methodology: I evaluated 20 different searches split evenly between Transactional and Informational queries and the search engines’ ability to deliver quality results, admitting that quality is a very subjective term, but includes things like timeliness, 1 click access to info, volume of content and lack of spam.

To provide a useful analysis, I’ve upped the degree of difficulty, deliberately testing long tail queries that are fairly specific and potentially confusing to a computer. I avoided easy navigational searches in favor of things that could trip up a computer’s ability to understand the user’s intent with queries like “Attorney Tom Brady”.

Results, including the snippet, needed to convince me of a likelihood of finding what I was looking for by clicking through in order to be considered.

To rank the results, 5 points were awarded for a good quality result ranking first, 3 for second and 1 for third. Two bonus points were added for top 3 results being on a highly authoritative site (as determined solely by yours truly). Five points were subtracted if the entire first page didn’t contain any good results. I only considered non-traditional results (like local, news, or Answers) if it appeared above the fold and I logged out of my gmail account when running queries.

Not too surprisingly, there was not a massive disparity in the results of my little test. In fact, Bing came out on top. Some queries performed very differently than others, for example, Bing was able to tell that my query for “Attorney Tom Brady”, was looking for an attorney and not the pictures of the hunky Patriots quarterback served up by Google.

Bing also did well with date nuances, unlike Google, sending me to the 2011 page for “SMX West Agenda” and a future calendar of concert appearances for “When is Trans Siberian orchestra playing next in Seattle” instead of past shows. Google won points by sending me directly to the IMDB page for filming locations for “What town was Beautiful Girls filmed in?” All in all, the results aren’t too surprising.

Of course, this begs the question why has Google been so successful? Are they still riding their brand laurels? Has Microsoft’s brandings and rebrandings of search hurt them among consumers?

Last week, a CNN article suggested that Google’s stock potential upside is probably no more than 20% – are we seeing the end of the growth in search? While these questions are the subject of intense debate in board rooms and VC water coolers, the conversation no longer centers on superior search results.

While I spent most of this post in retrospective navel contemplation about search, what does this mean tactically for the future of the in-house SEO? Despite the fact that we live in an industry of perpetual seismic change, Google has been a constant dominant player for most SEO’s entire career. Is Facebook the new SEO (as someone at work keeps insisting)?

Is Quora, the newest shiny toy whom the technology press dumped Twitter for this year, going to change everything? Is there a revolution in Local or location based search brewing? Or does consumer behavior change so much more slowly than technology (watch Quora trip on results for “Who is Lindsay Lohan” for example) – providing Google with a long lasting competitive advantage.

Your answers to these questions should shape your long term online marketing strategy.

Postscript: See also 89% Find Search Engines Do Good Job Finding Information, But “Noise” Is Issue