« New Year's Resolution: Know Your Inbound Link Potential! | Main | Microsoft Live & Yahoo Push For Firefox Users, Plus Revisiting The IE7 Search Battle »

Jan. 15, 2007 at 9:53am Eastern by Danny Sullivan

Study Says Get In Top 5 Not Top 10 & Search Engines May Need To Highlight Official Sites

On Friday, we reported briefly on a new eye tracking study from Microsoft on how users interact with search results. I spent some time doing a deep dive into the survey, which is full of interesting information. Among the findings is that search marketers may need to be more concerned about getting into the top five rather than the top ten, if they want to be seen. In addition, search engines might want to seriously experiment more with adding "official site" links at the top of their pages and possibly enlarge the size of listing descriptions or "snippets" to help searches find what they are looking for.

The study from Microsoft Research can be found here in PDF format. The survey was very small, involving only eighteen people from ages 18 to 50. The small number makes me worry a bit on how much I want to trust some of the conclusions. Still, it's a start, and many of the questions were well thought out and tested.

Those in the survey were assigned both navigational tasks (find the homepage of something like the World Cup 2006 site) or informational tasks (find when the Titanic set sail). Answers to all questions were listed within the results they received. They were all "moderately experienced" at web search.

Part of the test was to see how people interacted with results if they were given different lengths of listing descriptions or snippets, to use the Google term. People looked at short (one line), medium (two to three lines) and long (six to seven lines) snippets.

The study watched where people looked and also measured the time they spent on a task along with their click accuracy -- did they find the best answer?

Heat Maps & Eye Tracking

Where people looked provided no revelations:

First, confirming previous findings, we found that people viewed search results in a roughly linear order. Most gaze activity was directed at the first few items; items ranked lower got users’ attention last and least.

The heat map below is one example of how people scanned:

Interesting, the pattern is different that the "golden triangle" that Enquiro has long talked about in its eye tracking studies, where you see all the red along the horizontal line of the top listing (indicating a lot of reading there), then less on the second listing, then less still as you move down. That's because in this particular case, the seventh listing was the "right" answer plus, as the study explains, the longer than normal descriptions got people to read more.

Reviewing The Top Five

The study also looked at what people viewed before making a click. IE, did people review all 10 items listed? Only the first three? Did they review everything in order until they spotted a choice and clicked? The survey found people often reviewed one or two items past the item they eventually clicked on:

no matter what result they eventually clicked on, our participants usually looked at the first 3 or 4 search results. When they clicked on the first or second result, they still looked at the first 4 results. When they clicked on lower ranked results, they usually had looked at most of the items ranked above them. Finally, people go through about 8 results on a page before changing their queries without clicking on anything (indicated by “Requery�). With the exception of position 1, these results are very similar to findings reported by Joachims, et al. [12]. In their study, participants rarely looked at more than 1 or 2 items following the one that they clicked on, even when they clicked on the first item.

Consider this chart from the study:

The first long line dropping down shows that on average, participants reviewed at least eight results. That's good news for search marketers who worry that if they aren't number one, they aren't visible. You are. Get in the top eight, and you've got a decent chance of being seen.

Well, on average. The other bars tell an interesting story. Let's say the number one listing is the "perfect" listing. For example, if I look for us patent office on Google and have the intention of going to the official site, the top result is precisely what I need. Do I click immediately? The study says no.

See the bar in the #1 column? The dot shows that the first result was clicked on. But the long blue bar shows that up to four results were reviewed before the person went back up and clicked on the first result. You see a roughly similar pattern for when they click on the second, third and fourth results.

In other words, get into the top four or five results, and you greatly increase the odds that you're going to be seen and then perhaps clicked. After that -- positions six through 10, it's really going to depend on whether the top results simply aren't satisfactory. Notice how the bar gets longer as the clicks occur further down the page. That shows the first results aren't satisfying, so people will review more before clicking.

To summarize, we've long known being on the first page of ten results is crucial. But this study suggests it's about being in the top five for success. A post-survey questionnaire further confirms this:

In the post-questionnaire, several responses from participants on their expectation on the search results also speak to this effect: they highly agreed on the statement, “I expect the information I'm looking for to be in the top five results�

People Less Likely To Find The Right Answer If Buried

Another key finding was that the further down the page the "right" answer is located, the less likely people are to find it:

The click accuracy rate dropped from 84% (average of 78% and 89%) to about 11% when the target was displayed at position 8. Figure 6 shows that for navigational searches, people had the highest click accuracy rate when the target was on position 1 or 2 (78%, 83%). With the target on position 4, 5, 7, and 8, their click accuracy dropped to 33% or less. For informational search, the effect of target position on click accuracy was much more dramatic. When the target was displayed at lower positions (4, 5, 7, and 8), participants correctly selected the target for less than 20% of time. None of our participants correctly selected the target when it was at position 8.

That's a crucial issue for search engines. Giving a top ten list is in some ways a cop-out. Search engines can't know exactly what people mean when they do a search, much less what exactly is the "right" answers. As a result, you get a ranked list with the hopes that one of the choices will be satisfactory. It may be, but the further down the list it is -- even if on the first page of results -- the less likely searchers will find it.

Why does this happen? The study suggests:

1) Since participants rarely went through the whole result list, they never saw the target result when it was placed at a low position, especially for informational search. Therefore, they couldn’t find the correct target. This could be tested by looking at the number of results people fixated upon.

2) Alternatively, participants may have seen the target result for both navigational search and informational search, but they did not feel the result at lower position was as compelling as others. This could be tested by looking at the effect of task type and target position on fixation duration (an indicator of participants’ attention.)

Navigational Queries Need Short Snippets; Informational Ones Need Longer

I started out talking about the attempt to measure the impact that listing descriptions of different lengths have on how people search. As it turned out, short snippets were best for navigational queries. Long snippets slowed people down in finding what they wanted in those cases and made them less accurate. It was the opposite with informational tasks. The two charts below illustrate this:

The first shows the time to complete a navigational task (green bars) and an informational task (orange bars). The higher the bar, the longer it took. You can see how time rises for navigational queries when snippets become long, while the opposite happens for informational queries.

The second chart shows accuracy. The higher the bar, the more accurate the choices. Again, short is better for navigational while long is better for informational.

Time To Highlight Official Sites?

The study suggests that perhaps search engines should reconsider something that Yahoo does at the Overture site. Overture (and GoTo before that) was the old name of the paid listings company that Yahoo bought at the end of 2003. Back in 2000, GoTo began running "Quick Hit" listings at the top of its search pages. As I wrote about these at the time:

The new "Quick Hit" links that appear at the top of GoTo's search results are a significant change for the search engine, because they are the first real editorial content the service has provided. Quick Hits appear in response to queries involving popular brand or company names. For example, search for "microsoft," and a link that says "Quick Hit Result: The official site for microsoft" appears at the top of the page. Clicking on the link takes you straight to the Microsoft web site.

Until now, GoTo had been a pure Yellow Pages-style service. As with the Yellow Pages, those with the biggest ad budgets get the biggest ads -- or top listings, in GoTo's case. That's not necessarily a bad relevancy model for some types of searches, especially for products and services. The companies you want might be the same companies that can afford to spend on advertising. In fact, GoTo's paid placement results have outperformed some other major search engines in surveys performed by NPD.

Nevertheless, there are definitely times when you want editorial guidance, and Quick Hits is GoTo's way of providing some. They ensure that users seeking certain companies can find those companies, regardless placement was purchased.

In a similar vein, GoTo has been examining new bids since March to ensure they are relevant to particular terms and rejecting them, if not. As with the addition of Quick Hits, the changes are designed to prevent the paid placement service from becoming a free-for-all.

"Toward the end of last year, we were able to catch our breath and take a look at how we were doing on the relevancy issue. We observed a couple of things. The brand name issue and the company name issue was real, and we were probably being too loose on relevancy," said Ted Meisel, GoTo's chief executive officer.

It's been years since GoTo/Overture/Yahoo has added to the Quick Hit database, but you can still see existing links showing up like this below for nike, where it says "Quick Hit Result" at the top of the page:

AOL does something similar:

As far as I can tell, these "Official Site" links with logos only happen at AOL in cases where there's some paid relationship helping it along.

The idea of short one line "official" site navigational links is one I've long liked. This study suggests it could help improve relevancy. At the very least, it might help give further reassurance to concerned advertisers who dislike they idea they might feel they need to buy their names to be found.

Ask.com, Microsoft Live and Yahoo all do nothing special in terms of navigational queries, to my knowledge. Google does, presenting a Sitelinks display in some cases, as shown below for cisco:

See how Cisco is listed below the big blue box ad as "Cisco Systems, Inc," then there are five indented "sitelinks" to key parts of the site below the main listing?

Ironically, the survey suggests that Sitelinks might be a bad way to go. They potentially make the display longer and less relevant to users. Then again, perhaps they work better because a key element of Google Sitelinks is providing actual hyperlinks rather than more words. My gut is on them being effective. Otherwise, I wouldn't have expected Google to have rolled them out last September after over a year of testing.

By the way, notice that the listing below the first Cisco result is also from cisco.com. This is a unique bug that happens with Sitelinks. Normally, Google should indent the second result for any domain that's already listed on a page under the first result from that domain, as explained here. That can't happen with Sitelinks, since stuff is already being indented.

Personally, I'd like to see all the indented results go away, so the page has more variety on it. And stop dishing up pages from domains I've rejected if I drill into the results, as well. That was number two on my 25 Things I Hate about Google list from last year, as I wrote:

Despite results clustering, Google keeps serving up sites you've seen. You may not know the name results clustering, but you recognize Google doing it. That's when it sees there's more than one page from a web site that might match what you are looking for, so it "indents" the second best one below the first. Search for books, and you'll see this happening with Amazon. But clustering only happens on a results page-by-results page basis. In other words, look at mars landing sites, and there's a link to a page at the msss.com domain near the bottom. Say you reject this. Go to the next page, and msss.com is back again, as is the BBC. If I rejected content from these sites the first time, I want to see something new. Try whirlpool s20d, and you'll see the same thing. Lycos.co.uk and householdappliances.kelkoo.co.uk both come back. Give me the best page from a domain once, then give me some variety, not these second chances.

People Do Read & URL Strings Important

Back to snippet length, a fascinating question was raised about why searchers looked at more results for informational tasks when snippets increased in length. IE, most people seem not to want to read more things in general, yet that's what they did:

This finding presents a puzzle: it seems plausible that increasing the amount of information on the search-results page would result in looking at fewer results simply because there is more information to read: more lines of text in each result means that fewer results will be visible without scrolling. However, why would adding more information cause one to look at more results? And why would this effect be task-dependent? One explanation could be that adding more information would lead a searcher to be more thorough, spending more time with search results because the information density is higher. Therefore they just spend more time reading the results and less time reading Web documents. But if this were true, we should find that users look at more results in both informational and navigational tasks. What’s going on here?

One possibility is that because the goal of navigational tasks is locating a specific site, the information provided by contextual snippets is much less relevant for navigational than for informational tasks where details related to site content, authority, etc., are more important. In contrast, URLs may be proportionately more relevant for navigation because they are directly related to the location of target sites. If this were true, we would expect that searchers would spend proportionately more time looking at the URL in navigational than informational tasks. In our study, this was true but the difference was small: across all snippet lengths, people spent 25% of their time looking at the URL in navigational tasks vs. 22% in informational tasks. However, if we break this down by each snippet length, a pattern begins to emerge. Figure 12 shows the relative proportion of total fixation duration for each search result component (title, snippet and URL) broken down by snippet length and task type (for reference, the mean fixation duration for each condition is shown in Table 5).

Examination of Figure 12 shows that as we increased the snippet length, the relative time looking at the snippet also increased for both task types. However, while the proportion of time looking at the title stayed roughly constant, the increase in time looking at the snippet came primarily at the cost of looking at the URL. This decrease was particularly dramatic for the informational tasks, but it was true for navigational tasks as well. Figure 12 suggests that when our participants looked at search results with long snippets, they read them, whether or not the snippets were relevant to their task.

In short, people will read what they're shown to some degree even if there's more to read, but with navigational queries, this comes at the expense of closely looking at URLs. It makes you wonder if the search engines should either experiment with:

1. Putting URLs above the descriptions, where they might be more easily spotted and read if snippets get longer
    
2. Make better guesses as to when a search is navigational in nature and dynamically shrink URLs

The survey also found that URL strings were very important to searchers:

When I'm searching the Web, I usually read the snippet (text under the title) to help me decide if the page will be useful.� For these questions, the median scores were 7 and 6 respectively, and the means were 6.4 and 6.2. These answers suggest that our participants deliberately use various elements in the search results to help them find what they are looking for. We were particularly surprised to see the overwhelming endorsement of the URL because this is often characterized as a “power-user� feature that is used by only a small percentage of users.

Like The Story? Vote For It On Yahoo Buzz! Subscribe To Our Daily Search News Recap! Your Email: Send me the monthly search newsletter too! (Learn more about our newsletters and feeds)

Subscribe To Our Search Feed!

Share & Bookmark This Story!

By Danny Sullivan Permalink Jump To Comments See Related Stories In: SEO: Titles & Descriptions, Search Features: General, Stats: Search Behavior

Reader Comments