Too much time is often spent about the new features the various major search engines roll out or the latest deals they cut. Here at Search Engine Land, we can be as guilty of that as anyone. To correct it, I’ll be spending more and more time highlighting poor quality search results that I encounter, in hopes of nudging the industry to improve things.

Canaries In The Search Mine

I’ve spoken and written for years that when it comes to search engines, I think there are two “canaries in the coal mine” that catch a whiff of something bad emanating from the search engines.

The first are librarians and research professionals. They’re acutely aware of when search counts don’t make sense, if something important in a field they know isn’t being listed and other issues.

The far larger group are site owners or search marketers. The common joke is that when they spot search engine spam, that stands for Someone Positioned Above Me. Thus, it’s easy to dismiss what they see as just being colored by self-interest.

Sure, there’s some of that. But these people are also often subject experts. As surely as Cypher in the Matrix could look at computer code and say, “All I see now is blonde, brunette, redhead,” a subject expert like a site owner or search marketer can look at results and know when they don’t smell right, when something’s wrong.

About two weeks ago, our Reviewing Some Bad Google Search Results With Sergey Brin article highlighted a few bad results I could see in my subject area of expertise, that of search engines. Today, I’ll bring in another example, that of “search term research.”

Benchmarking Against Expert Knowledge

Search term research is one of the core aspects to search marketing, and I’ve covered various tools out there for years. Here at Search Engine Land, we maintain a Search Term Research page devoted to the topic. It’s a good page. There are probably better ones, and maintaining these types of pages is always difficult. Still, it’s kind of a benchmark for me. If I don’t see it ranking, what’s the quality of stuff that IS ranking above it?

As it turns out, our page isn’t on Google at all. Not at all. And it’s, um, our fault. The All In One SEO Pack plug-in we use with WordPress here set all of our category pages to be excluded from Google. It wasn’t that way originally. Back in the summer, the latest version of the plug-in changed things to overwrite how you’d previously configured it. I should have known better, too, because I even retweeted a warning about this. Everything’s fixed today, and we’ll see how things go.

Still, that page as well as my own knowledge of the area gives me a good benchmark I can use against other pages that do appear in the search results. So how’s it look?

Google: How About Some China Wholesale?

Over at Google, a search for search term research leads off with the Google AdWords Keyword Tool, which is an excellent first choice. It’s a dependable tool, offered for free, with great data. Some more tools follow that, then two older articles (from 2007 & 2006, respectively) on conference presentations about the topic. Those are kind of iffy to be in the top results given their age, but certainly they’re relevant. Then I get another tool, a fresher article that’s not super-substantial, a compilation list of articles and a nice conference presentation.

Overall, it’s not bad. Not fantastic, but not bad. Where things really fall down is when you go to the second page of results.

OK, few people go past the first page of results. I know this. But still, that second page of results? It contains what Google is presenting as among the very best out of 126 million pages on the web for this topic. The very best. And we get on the second page?

For those who can’t see the image above for some reason, the rundown:

• Link to a keyword research tool, which makes sense
• Really weird local results about local companies in New York that somehow seem related to search term research
• A really bad directory listing of resources
• An OK page listing some tools
• Agenda for a session on the topic for a conference in 2006
• A press release from someone speaking on the topic in 2006
• Another keyword research tool
• The most amazing bad result, some “China Wholesale Supplier” with search term research products. More on this in a bit…
• A review of one particular tool from 2006
• An article I wrote on the topic in 2007

I think the amazingly poor quality of these results are self-evident. Let’s look at that China Wholesale page. Again, out of 126 million possible matches, this is what Google thinks is the 18th best out of all of them for the topic of search term research:

I can’t even figure out what this is! One arrow points to how the page is in the “Search Term Research” category of the hosting web site. The other two point at what’s listed in this category, oil paintings of some soccer stars.

How on earth has Google, with its supposedly awesome attention to search quality, allowed this to show so high in the results?

But Bing’s Worse!

At least Google can fall back on the “others are worse” excuse. Let’s go to Bing:

Ugh. The rundown:

• Terms of service for those looking into broadband research?
• A tool, OK
• Wikipedia article on research in general. Not search term research — just research
• An undated article with bad advice that the best way to do research isn’t to do it at all. Just write! See what terms generate visits after you write. That’s terrible advice, because if you haven’t written using important terms, you’ll probably never see the traffic for them in the first place to know you should use them.
• That tool listed in number two? This is an article about it from the company that owns the tool
• Coverage of a search term research session at a conference from 2007
• A compilation of articles on the topic
• A page for marketing terms. Not search term research, just marketing terms
• Another page with coverage of a search term research conference session in 2007
• A press release about a biospace research project.

Did I say ugh? I’ll say it again. Ugh. It’s self evident how many of these pages are clearly NOT the best on the topic out of the millions of pages that Bing could pick.

For some reason, I see completely different results than this when I use Safari, rather than Firefox:

These are generally better, but I still get weird outliers like one for a Utah History center or a place to buy essays.

Yahoo: 50% Irrelevant

How about Yahoo? Ugh again:

Pages for biotech research, autism research, research at Oregon Health & Science University plus two for Lexis/Nexis show up. That’s 50% of the top results completely off target for what I searched for. Not just 50% so-so results. They’re just totally not right. At all.

Ask’s OK, If You Can Stand The Ads

Ironically, given I’ve written it off as a serious search engine last year, Ask seems to have fairly decent results for the topic. There’s a bad press release of someone speaking on the topic, but everything else is good or at least related to the subject.

Of course, you have a giant ad unit containing five paid listings that’s shoved between the first result and the rest. Then another five at the bottom. Then one more paid listing after that, with no disclaimer as required by the Federal Trade Commission.

Things Feel Worse, But Hard To Quantify

Sadly, we still have no commonly accepted measurements of relevancy across search engines, and it’s an area that gets harder and harder to assess, as more material is blended in alongside web page results. That’s something I’d still like the search engine to collaborate on, some independent regular assessment of their quality.

To me, it feels like they’re getting worse, not better. But I can’t document that. What I can do is demonstrate without much difficulty, for areas where I have subject expertise, how bad they can be. They get by because along with the bad, there’s enough good. But they should be better than this.

A new and wide ranging report from the UK regulatory agency Ofcom about media literacy finds that 32 percent of UK kids aged 12-15 years old believe that the links/listings shown at the top of search results are the most “truthful.” Here’s how the report summarized this finding:

Among children aged 12-15 who use the internet, almost all have experience of using search engine websites (94%).

Those who use search engine sites were shown a list of options and were asked to say which, if any, apply in terms of the way results are shown on search engine sites.

There is no clear consensus among search engine users, but 12-15s are more likely to respond that results are ranked on their usefulness or relevance (37%) or their truthfulness (32%) than they are to respond that websites pay money to be at the top of the list (14%).

The question prompting the responses charted above was “Which if any of these explain the way results are shown when you use search engine websites like Google, Yahoo, MSN or Ask Jeeves?”

From 2008 to 2009 the number of those who think that top search results are based on “relevance” declined in favor of “truthfulness.” It doesn’t appear there was much detailed questioning around sponsored vs. organic results and definitions or explanations of each category. Consequently it’s not clear to me if the kids are referring to paid results in their statements about relevance or truthfulness.

uTest released a study they named “Battle of the Search Engines” which measured the relevancy of Google, Yahoo, and Bing based on “search results accuracy, page load speed, real-time relevance and usability.” The report found that there were over 600 “bugs” found in the test. By “bug” I believe they mean links that lead to dead ends, i.e. broken links. I am not exactly sure how that measures “relevancy” of the search result, but it does tell us some things.

In any event, here are the key points from the survey:

• 71% of respondents stated search results accuracy as the most important when choosing their search engine.
• In overall accuracy, Google led the way with a top two box score (those rated as “excellent” or “good”) of 90%. Caffeine followed with 83%. Yahoo & Bing trailed with 53 and 42%, respectively.
• Bing had the most reported bugs; however, it scored very high for usability and pics/vids search
• More than 30% noted that Bing surprised them favorably; and 10% said that after testing all three engines, they would make Bing their default engine.

Here is the break down by search engine:

Google

• 130 bugs reported
• 8 percent classified as “showstoppers”
• 28 percent were Technical, 46 percent were Functional, and 26 percent were GUI

Bing

• 321 bugs reported
• 14 percent classified as “showstoppers”
• 29 percent were Technical, 46 percent were Functional, and 25 percent were GUI

Yahoo

• 70 bugs reported
• 10 percent classified as “showstoppers” and 50 percent as “high severity”
• 28 percent were Technical, 57 percent were Functional, and 15 percent were GUI

For the full report, click here do download the PDF.

In a detailed report about the Canadian search landscape, Hitwise says Bing is the best search engine at producing “successful searches,” beating out both Google and Yahoo, as well as their Canada-specific search engines.

The chart above, provided specifically to Search Engine Land, shows that Bing is the 6th most popular search engine in Canada, but has the best success rate at 78.61%. That’s about 2% better than Yahoo Canada (ca.search.yahoo.com), and more than 6% better than Google Canada (www.google.ca).

It’s important to understand how Hitwise defines a “successful search”:

“A successful search is defined as one where the consumer leaves the search engine after performing a search.”

You could make the argument that some searches are successful even if the user doesn’t leave the search engine, especially considering the availability of search shortcuts and “instant answers” that sometimes show up on a search results page. But without knowing how many queries lead to shortcuts and answers, there’s no way to gauge their impact on these Hitwise numbers. Plus, it’s quite possible that searchers still leave the search engine even when a shortcut appears, which would further minimize their impact on the definition of a successful search.

Overall, Hitwise says an average of 70% of searches across all search engines in Canada were successful in the 12 weeks leading up to June 27, 2009. As you’d expect, though, when Canadian searchers add a geographic modifier to a query — such as “wedding dresses canada” instead of just “wedding dresses” — the percent of successful searches goes way up.

Oddly, though, Canadian searchers seem to lag behind U.S. searchers when it comes to the complexity of their search queries. Hitwise says more than half of Canadian search terms contain only one or two words.

The chart above shows that Canadian searchers use one- or two-word queries 51% of the time. That’s less than UK searchers, where the number is close to 60%, and more than the U.S., where 43% of searches had only one or two words.

Other interesting pieces of data from the Hitwise Canada Search Report:

• When you combine the .com, .ca, and other domains, Google powers 80% of searches in Canada. MSN is second with 9% (combining Bing and Live.com), and Yahoo third with 8%.
• The top 17 search terms used in those four weeks leading up to June 27th were all navigational — searchers looking for specific web sites like Facebook and YouTube.
• Canadians are bigtime gamers. The top non-navigational English search term was “games” and the top French search term was “jeux,” which also means “games.”
• Search engines are the largest traffic source for web sites in Canada: 33% of all web site visits in June came from a search engine.

Back to the headline of this article … it’s worth remembering Greg Sterling’s article yesterday about how U.S. searchers say Google is the best search engine. The Hitwise data reported here isn’t based on a consumer survey, but on measuring searcher behavior. You can’t help but wonder what Canadian searchers would say if they were surveyed about search satisfaction. Would the results match what we’re reporting here?

On an annual basis the University of Michigan puts out the American Customer Satisfaction Index (ACSI) E-Business report, sponsored and administered by Foresee Results. Among the things the index examines is consumer satisfaction with search engines.

The survey asks a representative sample of consumers to rate their experiences with portals and search engines according to several criteria, which produces an overall score on a 100 point scale. Google is again on top in search with a score of 86, identical to last year. The survey is supposed to be predictive of future consumer behavior, as the associated report argues:

[The ACSI] is a predictor of future success on both the micro and macro level. Google’s huge 10 point jump in satisfaction from 2007 to 2008 preceded a 7% increase in search market share from 2008 to 2009. Satisfaction with Yahoo, on the other hand, dropped 2.5% from 2007 to 2008, preceding a 17.5% decrease in search market share and a 5% drop in portal market share.

When the 2007 results saw Yahoo jump ahead of Google for the first time since the inception of the e-business survey I asked about this asserted relationship between the survey and market share:

The satisfaction data clearly don’t correlate with search market share. I asked [Foresee Results CEO Larry] Freed in this context why people should care and pay attention to the ACSI. Freed was confident that “search market share reflects past behavior. But the ACSI is predictive of future consumer behavior.” He said that historically it has been a very accurate gauge of future consumer behavior in other industries. He added that Google’s decline was a second dip in a row after a smaller decline last year.

Considered over the course of almost a decade of satisfaction rankings, the survey results would indeed appear predictive of market share (for Google), though the 2007 satisfaction win did not correctly predict corresponding market-share gains for Yahoo in 2007-2008.

Source: ACSI/Foresee Results (2009)

According to the survey, conducted before Bing, Google has a significant lead over its competitors, which has widened since 2007.

These overall results line up with a recent search loyalty report from comScore showing Google users as the most loyal.

It’s somewhat disappointing that we’ll have to wait an entire year to see what the ACSI has to say about Bing, because it would give us another opportunity to test the accuracy of the index as a predictor of future behavior.

According to comScore Bing now has an 8.9 percent share of the search market, up from 8 percent in May, before the launch. A year from now, when the next report comes out, we’ll know generally whether Bing is a success. For now, here’s what the ACSI report commentary has to say about Bing and its prospects:

[I]t seems unlikely that customers will actually leave Google in enough numbers to allow Bing to seriously challenge Google’s market dominance, given Google’s extremely high customer satisfaction. People are happy with Google, so why would they switch? They might switch if Bing is better, and that’s a tall order considering Google is the second- highest scoring ACSI service-sector company, behind Newegg.com. Bing has been called a search engine war “game changer,” but Google’s game will be very hard to change at this point. If anyone can do it, it’s the combined resources and market share of Yahoo and MSN.

The central “takeaway” from the report would seem to be that it’s not simply “habit” driving Google usage — people are actually “satisfied.”

Citigroup conducted a study comparing 200 queries on Google, Yahoo and Bing. The study found that Google is much more relevant than Bing, in returning relevant results. Google returned the most relevant result 71 percent of the time, compared with Bing at 49 percent of the time and Yahoo 30 percent of the time.

The queries picked were pulled mostly from Google Zeitgeist, Microsoft’s xRank and Yahoo Buzz in the categories of Entertainment, Health, Local, News, Retail, Sports, Travel and Other. Then after running the 200 queries, Citigroup picked the winner based on two criteria:

1) Relevancy of the organic search results; and 2) Robustness of the search experience, which included factors such as image and video inclusion, Search Assist, and Site Breakout.

Here are some of the charts from the study:

Greg covered how many searchers preferred Bing’s look over Google, but said they would stick with Google as the default engine.

For more coverage on this study, see paidContent.org and AllThingsD.

Trade magazine AdWeek and pollster Harris Interactive conducted a survey in early June among 2,521 US adults. People were asked what types of ads they found most “helpful” and what types of ads they ignored. Below are the top-line results as well as the age and geographically segmented data.

Most helpful in making purchase decisions:

• 37 percent say that television ads are most helpful
• 17 percent say newspaper ads are most helpful
• 14 percent say search engine ads are helpful
• Radio ads (3 percent) and Internet banner ads (1 percent) are not considered helpful by many people
• 28 percent say that none of these types of advertisements are helpful to them in the purchase decision making process

Ad types that people ignore:

• 46 percent say they tend to ignore Internet banner ads
• 17 percent ignore search ads
• 13 percent ignore TV ads
• 9 percent ignore radio ads
• 6 percent ignore newspaper ads

One in ten Americans say they do not ignore any of these types of ads. Interestingly, younger survey respondents tended to find search ads less helpful than older users and ignored them more. That’s a bit counter-intuitive.

Here are the data segmented by age and geography:

Google Blogoscoped reports we have a new blind search taste test in town at blindsearch.fejus.com. The way this blind search test works is that you enter a query and you vote for the best search results without knowing who the search results came from.

Unfortunately, the site was hit by a guy who “gamed the system” and the data is currently skewed by that individual. They do promise to remove the fake data in the near future. In addition, the person who created the site works for Microsoft but claims this is not a “Microsoft initiative.”

This reminds me of a test I ran back in 2005, which I called the Search Engine Relevancy Challenge. I called the white-labeled search engine RustySearch and put search results from Google, Yahoo, Ask.com and MSN Search on the same page. Searchers could have rated each result, as opposed to rating the whole ten links at once. It appears this is still somewhat working with over 20,000 queries. I cannot wait to see the results from the Blind Search test.

Satya Nadella, SVP of Microsoft’s Online Services Division, gave the morning keynote at the Microsoft Search Summit. It was an introduction to and tour of Bing and adCenter upgrades and improvements. Nadella began with a review of the search market and its growth. He was initially apologetic to the audience about Microsoft’s market share. He proceeded to outline the problems with the current state of search that Bing tries to address.

Nadella said “only 1 in 4 queries deliver successful results.” This is based on Microsoft’s observation of search user behavior from historical Live Search logs and its toolbar installs (where they get visibility on search behavior on other engines). Repeat queries or refinements and abandonments indicate current dissatisfaction or deficiencies of the current state of search.

“In the quest to find the perfect search engine, we still have a lot of room.”

Nadella explains that people engage in long search sessions. Almost 50% of time spent searching is spent during sessions longer than 30 minutes. But those sessions, according to Microsoft, represent only 5% of search sessions overall. He also showed the following consumer data focused on Microsoft’s four strategic verticals.

The slide shows 66% of people are using search more frequently as a decision-making tool; and in their strategic verticals:

• 75% product purchases
• 62% Local activity
• 45% Flight or hotel
• 43% Healthcare

Nadella explains Bing’s “task orientation” and begins a hands-on walk-through of the site. He shows the homepage and discusses its strong “emotional appeal.”  He says that among consumers it’s one of the most liked features of the site. Then he takes us on a tour of the site and concretely points out the features (e.g., Best Match, Instant Answers) that are designed to minimize clicks and respond to typical user behaviors.

He shows a local search “San Diego Events” and points out a range of information about events but also about San Diego more broadly. He discusses health search and authoritative answers, with health-related content and articles that can be read on the SERP. Nadella goes on to discuss shopping and the range of information that can be obtained on the SERP without having to click away. In general, what these and other examples collectively show is the deeper integration of verticals and related vertical content into the search result (to avoid too many clicks and the back button).

In local Nadella pointed to “one-click directions” from the SERP (showing directions from multiple starting points) as a differentiator.

Overall Nadella gave an impressive presentation. Although I don’t discuss these things, he also spoke about the forthcoming Bing marketing campaign (radio, online, TV). He showed the “Hawaii” commercial, which I thought was funny and effective. He also spoke at some length about adCenter improvements. Those changes will be addressed on this site in coming weeks and months. Nadella ended by soliciting honest feedback.

At the end of the session there were a number of questions, many dealt with social media and Twitter and how Bing was going to address that phenomenon.

Nadella said that he felt passionate about the need to incorporate the social graph with the web graph. He said that Microsoft has a lot of work to do in this area.  He discusses potential re-ranking of results based on one’s social graph. But he also says that many searchers are looking for more comprehensive information than what one’s friends have to say. He argues that the social graph works better in some categories than others (e.g., Local as opposed to Health).

In response to a question about neutrality regarding working with non-Microsoft products, Nadella says that Bing will absolutely have to work with all browsers and platforms. He says that MSN, Bing and Windows Live are three brands that have distinct objectives (presumably Bing will be the neutral or “agnostic” one).

Google Leads in Search Satisfaction from adAge.com reports that Google not only is leading in the University of Michigan and ForeSee Results in American Consumer Satisfaction Index, but is also “surging” in growth.

Google scored an 86 in the index, up 10.3% in consumer satisfaction from last year. While Yahoo was down 2.5% to 77, Microsoft’s MSN remained at 75, and Ask.com dropped 1.3% to 74. AOL did increase by 3% to 69, but lost a lot from the year prior and is still not where they were then, which was 74.

The scores don’t seem to be live yet at the ACSI web site, but when they are, you should be able to find them over here.