The Myth Of Great Search Engine Results

Too much time is often spent about the new features the various major search engines roll out or the latest deals they cut. Here at Search Engine Land, we can be as guilty of that as anyone. To correct it, I’ll be spending more and more time highlighting poor quality search results that I encounter, in hopes of nudging the industry to improve things.

Canaries In The Search Mine

I’ve spoken and written for years that when it comes to search engines, I think there are two “canaries in the coal mine” that catch a whiff of something bad emanating from the search engines.

The first are librarians and research professionals. They’re acutely aware of when search counts don’t make sense, if something important in a field they know isn’t being listed and other issues.

The far larger group are site owners or search marketers. The common joke is that when they spot search engine spam, that stands for Someone Positioned Above Me. Thus, it’s easy to dismiss what they see as just being colored by self-interest.

Sure, there’s some of that. But these people are also often subject experts. As surely as Cypher in the Matrix could look at computer code and say, “All I see now is blonde, brunette, redhead,” a subject expert like a site owner or search marketer can look at results and know when they don’t smell right, when something’s wrong.

About two weeks ago, our Reviewing Some Bad Google Search Results With Sergey Brin article highlighted a few bad results I could see in my subject area of expertise, that of search engines. Today, I’ll bring in another example, that of “search term research.”

Benchmarking Against Expert Knowledge

Search term research is one of the core aspects to search marketing, and I’ve covered various tools out there for years. Here at Search Engine Land, we maintain a Search Term Research page devoted to the topic. It’s a good page. There are probably better ones, and maintaining these types of pages is always difficult. Still, it’s kind of a benchmark for me. If I don’t see it ranking, what’s the quality of stuff that IS ranking above it?

As it turns out, our page isn’t on Google at all. Not at all. And it’s, um, our fault. The All In One SEO Pack plug-in we use with WordPress here set all of our category pages to be excluded from Google. It wasn’t that way originally. Back in the summer, the latest version of the plug-in changed things to overwrite how you’d previously configured it. I should have known better, too, because I even retweeted a warning about this. Everything’s fixed today, and we’ll see how things go.

Still, that page as well as my own knowledge of the area gives me a good benchmark I can use against other pages that do appear in the search results. So how’s it look?

Google: How About Some China Wholesale?

Over at Google, a search for search term research leads off with the Google AdWords Keyword Tool, which is an excellent first choice. It’s a dependable tool, offered for free, with great data. Some more tools follow that, then two older articles (from 2007 & 2006, respectively) on conference presentations about the topic. Those are kind of iffy to be in the top results given their age, but certainly they’re relevant. Then I get another tool, a fresher article that’s not super-substantial, a compilation list of articles and a nice conference presentation.

Overall, it’s not bad. Not fantastic, but not bad. Where things really fall down is when you go to the second page of results.

OK, few people go past the first page of results. I know this. But still, that second page of results? It contains what Google is presenting as among the very best out of 126 million pages on the web for this topic. The very best. And we get on the second page?

For those who can’t see the image above for some reason, the rundown:

• Link to a keyword research tool, which makes sense
• Really weird local results about local companies in New York that somehow seem related to search term research
• A really bad directory listing of resources
• An OK page listing some tools
• Agenda for a session on the topic for a conference in 2006
• A press release from someone speaking on the topic in 2006
• Another keyword research tool
• The most amazing bad result, some “China Wholesale Supplier” with search term research products. More on this in a bit…
• A review of one particular tool from 2006
• An article I wrote on the topic in 2007

I think the amazingly poor quality of these results are self-evident. Let’s look at that China Wholesale page. Again, out of 126 million possible matches, this is what Google thinks is the 18th best out of all of them for the topic of search term research:

I can’t even figure out what this is! One arrow points to how the page is in the “Search Term Research” category of the hosting web site. The other two point at what’s listed in this category, oil paintings of some soccer stars.

How on earth has Google, with its supposedly awesome attention to search quality, allowed this to show so high in the results?

But Bing’s Worse!

At least Google can fall back on the “others are worse” excuse. Let’s go to Bing:

Ugh. The rundown:

• Terms of service for those looking into broadband research?
• A tool, OK
• Wikipedia article on research in general. Not search term research — just research
• An undated article with bad advice that the best way to do research isn’t to do it at all. Just write! See what terms generate visits after you write. That’s terrible advice, because if you haven’t written using important terms, you’ll probably never see the traffic for them in the first place to know you should use them.
• That tool listed in number two? This is an article about it from the company that owns the tool
• Coverage of a search term research session at a conference from 2007
• A compilation of articles on the topic
• A page for marketing terms. Not search term research, just marketing terms
• Another page with coverage of a search term research conference session in 2007
• A press release about a biospace research project.

Did I say ugh? I’ll say it again. Ugh. It’s self evident how many of these pages are clearly NOT the best on the topic out of the millions of pages that Bing could pick.

For some reason, I see completely different results than this when I use Safari, rather than Firefox:

These are generally better, but I still get weird outliers like one for a Utah History center or a place to buy essays.

Yahoo: 50% Irrelevant

How about Yahoo? Ugh again:

Pages for biotech research, autism research, research at Oregon Health & Science University plus two for Lexis/Nexis show up. That’s 50% of the top results completely off target for what I searched for. Not just 50% so-so results. They’re just totally not right. At all.

Ask’s OK, If You Can Stand The Ads

Ironically, given I’ve written it off as a serious search engine last year, Ask seems to have fairly decent results for the topic. There’s a bad press release of someone speaking on the topic, but everything else is good or at least related to the subject.

Of course, you have a giant ad unit containing five paid listings that’s shoved between the first result and the rest. Then another five at the bottom. Then one more paid listing after that, with no disclaimer as required by the Federal Trade Commission.

Things Feel Worse, But Hard To Quantify

Sadly, we still have no commonly accepted measurements of relevancy across search engines, and it’s an area that gets harder and harder to assess, as more material is blended in alongside web page results. That’s something I’d still like the search engine to collaborate on, some independent regular assessment of their quality.

To me, it feels like they’re getting worse, not better. But I can’t document that. What I can do is demonstrate without much difficulty, for areas where I have subject expertise, how bad they can be. They get by because along with the bad, there’s enough good. But they should be better than this.