This week I caught up with Larry Cornett, the relatively new Director of User Experience Design at Yahoo and Kathryn Kelly, Director of PR for Yahoo! Search. Again, to set the stage for the interview, here are some high level findings from our eye tracking study that I’ll be discussing in more detail with Larry and Kathryn.
Emphasis on Top Sponsored Results
In the study we found Yahoo emphasized the top sponsored results more than either Microsoft or Google. They showed top sponsored results for more searches and devoted more real estate to them. This had the effect of giving Yahoo! the highest percentage of click throughs on top sponsored, but on first visits only. On subsequent visits the click through rate on these top sponsored ads dropped to a rate lower than what was found on Google or Microsoft.
Better Targeted Vertical Results
Yahoo’s vertical results, or Shortcuts, seem to be better targeted to the queries used by the participants in the survey. Especially for commercial searches, Yahoo did a good job disambiguating intent from the query and providing a researcher with relevant vertical results in the product search category.
How Searching from a Portal Impacted the Search Experience
When we look at how the search experience translated from a portal page where the query is launched to the results page, we found that Yahoo had a greater spread of entry points on the actual search results page. This brings up the question of how launching a search from a portal page rather than a simple search page can impact the user experience and their interaction with the results they see.
I had the chance to ask Larry and Catherine about the Yahoo search experience and how their own internal usability testing has led to the design and the experience we see today. Further I asked them about their plans for the future and what their strategy is for differentiating themselves from the competition, namely Microsoft and Google. One difference you’ll notice from Marissa’s interview last week is the continual reference to Yahoo’s advertisers as key stakeholders in the experience. At Yahoo, whenever user experience is mentioned, it’s always balanced with the need for monetization.
Here’s the interview:
Gord: First let’s maybe just talk in broad terms about Yahoo’s approach to the user experience and how it affects your search interface. How are decisions made? What kind of research is done? Why does Yahoo’s search page looks like it does?
Larry: I can give you a little bit. I have been here about 7 months, so I’m still fairly new to Yahoo. But I can tell you a little bit about how we move forward with our decision making process and the approach we are taking with Yahoo.. We try to strike a balance between user experience and the needs of business as well as our advertising population. We want to provide the best user experience for the users who are trying to find information and give them most efficient experience they can have and then also provide a good ecosystem for our advertisers. And we do a tremendous amount of research here: we do a lot of usability testing, we do surveys. We also do eye tracking studies. All those help inform us when thing are working really well for users and one we need to work on improving things.
Kathryn: And we do bucket testing on a lot of different features, on different properties, to get feed back from our users before ever implementing any new features.
Gord: Since we conducted the study we’ve noticed some changes on the search interface. Do changes tend to be more evolutionary or do you lump them together into a major revision and roll them out together?
Larry: I’d say we do both. We have essentially two parallel tracks. One is continuous improvement, so we’re always looking to improve the experience. They would be considered the evolutionary changes based on a lot of data we are looking at. Then we also have larger things that we’re definitely interested on a more strategic direction, that we look at in a longer term window for larger changes.
Gord: One of the things that we did spend a fair amount of time on in this study is this whole idea of perceived relevancy. If we set a side the whole question of how relevant are the actual results based on the content of those results and what shows, and look more how quickly scent is picked up by the user and how the results that they see are perceived to be relevant. Does that notion coincide with your findings from the internal research and how is that idea of the appearance of relevancy rather the actual relevancy play into the results you present?
Larry: Yes, that absolutely is similar to the types of research findings that we’ve had, specifically with some of the eye tracking studies. We also continue to make efforts on actual relevance so our Yahoo search team is constantly doing improvements to everything to have real relevance improve. But you right, that perceived relevance is actually the most important thing because, at the end of the day, that’s what the users are looking at and that’s what they walk away with. In terms of: Was my search relevant? Did I find what I was looking for?
I do like the concept that you have with the information scent, the semantic mapping. I think it definitely ties into the mental model that a user has when they approach search and they are doing a query. They looking for things that come back to match what they have on their mind, what they are looking for in the results, so the more they actually see those search terms and things they are having in their mind, in terms of what they’re expecting to see, the more relevant the search is going to be for them.
Gord: It comes down to the efficiency of the user experience too, how quickly they find what they are looking for, how quickly they think they find what they are looking for and how successful that click through is. Did the promise match up with what was actually delivered on the other end?
Larry: Yes absolutely.
Gord: One of the biggest differences we saw between Yahoo and the other engines was the treatment of the top sponsored ads. I think it’s fair to say that in both the percentages of the searches that ads were presented for and the number presented, you were more aggressive than MSN (now Live Search) and Google. Obviously I understand the monetization reasoning behind that, but maybe you can speak a little bit as far as the user experience.
Larry: Sure, I mean in many cases those results are, and even in your report you showed this, those results are exactly what the users are looking for. Very often what they see in that sponsored section actually is a good fit for the type of query they are doing, especially if you look at a commercial query. So it’s always finding that balance between monetization and showing organic results. We’re just trying to get the best results for the user based on what they are looking for.
Gord: I certainly agree with you with the fact that in a lot of cases the sponsored results were what they were looking for, but we couldn’t help but notice that there was always a little bit of suspicion or skepticism on the part of the user, both in how they scan the results, and even when they do click through to a result there seems to be a hesitancy to stop there. We found a tendency to want to check out at least the top organic listing as well. One thing with Yahoo is that, with the more aggressive presentation of top sponsored, their choices on the organic tend to get pushed closer and closer to the fold. Have you done any testing on that?
Larry: Yes, those are definitely things that we are exploring as we’re trying to improve the user experience. And we’ve done our own eye tracking as well. A lot of it does come down to a big difference between what’s above the fold and what’s below the fold. So we’re always being very careful when we’re exploring that, thinking about the dominant monitor resolution, settings that we’re looking at as people start to have more advanced systems and larger monitors and really trying to understand what they seeing when they given that first load of the search page.
Gord: We did notice that of all the three engines, Yahoo has the highest percentage of click throughs on top sponsored and first visits. A little more then 30 % of the clicks happened there. But we noticed that it dropped substantially on repeat visits, much more then it did on the other engines. Combine that with the fact that we saw more pogo sticking on Yahoo then we did on the other engines; someone would click through a top sponsored ad then click back to the search results. So, my question is; does the more aggressive presentation of top sponsored ads even out when you factor in the repeat visits and does those lower repeat click through rates negatively impact the monetization opportunities on those repeat visits?
Larry: I can’t get into too many details about that, because it starts to get into some of the business logic and business rules that we have. Especially looking at CTR (click through rates) and repeat visits, so yes, that’s probably a little more information that I actually have access to, myself.
Gord: OK, we’ll put that one off limits for now. Let’s shift gears a little bit. One of the other things that we noticed, that actually works very well on Yahoo, was Yahoo Shortcuts on the vertical results. It seems like you are doing a great job at disambiguating intent based on query and really giving searchers varied options in that vertical real state. Maybe you can talk about that.
Larry: You are absolutely right that those are very effective. And you’ll probably notice over time that we’re continuing to refine our Shortcuts, try to find even more appropriate shortcuts for different types of queries. A lot of that is based on the best end result, what the user is trying to find, and the more we can give them that information the better
Kathryn: And it’s faster too, right?
Larry: Exactly. So a lot of time people just want a quick answer; they don’t want to have to dig through a lot of web pages. They just want a very simple quick answer, so if we can provide that, then that is a great experience for them. Kathryn: And we found that certain queries like movies, entertainment, weather, sports, travel, blend very nicely with those types of Shortcuts.
Gord: In talking to Google about this, they have fairly strictly monitored click throughs thresholds in both their top sponsored ads and on their vertical results; and if results aren’t getting clicked they don’t tend to show. It automatically gets turned off. What’s Yahoo’s approach to that? Are you monitoring CTRs and determining whether or not vertical results and top sponsored ads will appear for certain types of queries.
Larry: We definitely monitor that as well. We’re interested in tracking usage and so looking at the CTR, because we don’t want to be showing things that are not actually getting usage, so we do continually monitor the CTR and the Shortcuts.
Kathryn: Are you just referring to the Shortcuts or to all of the ads?
Gord: Both; top sponsored and vertical results or Shortcuts.
Kathryn: It’s the same for both.
Larry: Obviously we track CTR in both of those areas and look at that trend over time.
Gord: When we look at the visibility difference or the delta between those top sponsored ads and those side sponsored ads, when you factor in conversion rates, click through rates and everything else, is the difference as significant as it appears to be from an eye tracking study? How do you work with your advertisers to maximize their placement and to help them understand how people are interacting with that search real estate?
Larry: That is actually a separate team that works with those folks.
Gord: Is there overlap between the two departments? You would be on top of how the user is interfacing or interacting with the search results page. Do you share that information with that team and keep them up to date with how that real estate is been navigated?
Larry: Absolutely, it’s a very collaborate relationship, We are in communication constantly, they are giving us performance, we are giving them performance. So it’s always a very collaborative kind of relationship with that team. We definitely can give them recommendations and vice versa. We each have our own worlds that we own.
Gord: Maybe you can speak a little bit from the user’s perspective how Yahoo is when you position it against Microsoft Live Search and Google. What’s unique, why should a user be using Yahoo rather than the other two?
Larry: I’d say one of the key differentiators that you’ve seen released last year, and Terry Semel actually talks about this, is that we’re starting to introduce social search. If you look at Yahoo Answers, it’s one of the key examples of that. It’s a very exciting site that’s performing very well. There’s a lot of great press around it, and we starting to integrate that within the search experience itself, so you can do certain types of queries within search and at the bottom of the page you’ll see relevant best answers that are brought from Yahoo’s Answers. In many cases people look at that and say that it actually adds value. That’s one of the key differentiators here; there is definitely a social aspect to Yahoo Search.
Gord: As Yahoo Answers and that whole social aspect gains traction , is that something that either be moved up as far as visibility on the real state page, moved up into that Golden Triangle real estate or would it be rolled in almost transparently in what the results being shown are?
Larry: Anything is possible, but it’s something that we’re evaluating, so we’re constantly looking at data that comes from user studies and the live site performance, and so we’ll be making a decision about that as the year goes on.
Gord: One other thing that I think somewhat distinguishes Yahoo, especially from Google, is where the searches are launched from. When you look at user’s experience, obviously you are taking into consideration where those Yahoo searches are being launched from; a tool bar versus a portal versus the search page. How does that factor into the user experience?
Larry: You’re right. We definitely look at that type of data and really try to understand how those users might be different and their expectations might be different. So we’re constantly looking at that whole ecosystem because Yahoo is a very large network, with a lot of wonderful properties, so you have to understand how we all play together and what the relationship is between the properties back and forth.
Kathryn: Another thing is knowing where to put a search box on what property and what is going to work with the right mix of users for that property, because not every property is conducive to having a search box prominently displayed, and that’s something that we look at very closely.
Gord: Which brings up another question. One of the things that we speculated on the study is; does the intent of the user get colored based on the context that shows around that search box? If it gets launched from a very clean minimalist search page, there is little influence on intent, but if it gets launched from a portal, where there is a lot of content surrounding it, the intent can then be altered in between the click on the search box and ending up on the search page . Do you have any insight on that? Have you done your own studies on that impact?
Larry: We are definitely doing research within that area to understand the affect of the context and I don’t really have anything I can share at this point but I would say that there’s probably a lot of very interesting information to be derived from looking at that.
Gord: Ok. I’ll go out on the limb once more and say: Obviously if you can take the contextual messaging or what is surrounding that search box, and if it obviously correlates to the search, then I suppose that will help you in potentially targeting the advertising messaging that can go with that, right?
Larry: I think that’s fair to say.
Gord: Ok. I’ll leave it there. One question I have to ask comes down to the user interface. It seems as changes are made the differences between the three engines are getting fewer and fewer and it does seem like everyone is moving more to the standards that Google has defined. Is Google’s interface as it sits the de facto standard for a search results page now and if so, then in what areas does Yahoo differentiate itself? I’m talking more about the design of the page, white space, font usage, where the query bolding is…that type of thing.
Larry: There are a lot of really smart people in each of those companies. They also do their own user studies and they look at their metrics, so I think everyone is realizing over time that they’ve refined their search experience, what is working and what is not. So it’s not surprising to see some convergence in terms of the design and what seems to be most effective. I can’t really speak to whether Google is the de facto standard, but definitely they have a lot of eyeballs, so I think people do get use to seeing things in a certain way. I know from the Yahoo perspective that we want to do what’s best for our users. And I think we do have a user population that has certain expectations of us. I think a big part of that is the social search component, because people do think of Yahoo as a distinct company with its own brand, so there’s a lot that we want to do on that page that is completely independent of what other people might be doing because we want to do what’s best for our users.
Kathryn: And our users tend to be different than Google users. There is obviously overlap but we also have a distinct type of user than Google. We have to take that into consideration.
Gord: Can we go a little bit further down that road? Can we paint a picture of the Yahoo user and then explain how your interface is catering to their specific needs?
Larry: I can’t speak too much about it but one difference is that Yahoo is a lot of different things. We’re not just a search company, not just a mail company, not just a portal company; we serve a lot of needs. And we have a lot of tremendously popular, very effective properties that people use. Millions and millions come to the Yahoo network every day for a whole variety of reasons, so I think that’s one thing that’s different about the Yahoo user. They’re not coming to Yahoo for one purpose. There is often many, many purposes, so that’s something we definitely we have to take into consideration. And I think that’s one reason of many that we’re looking at social search. We know that our users are doing a lot of things in our network and it’s really effective if we’re aware of that.
Gord: So, rather then the task oriented approach with Google where their whole job is to get people in and out as quickly as possible, Yahoo Search supports that community approach where search is just one aspect of several things that people might be doing when they are engaged with the various properties?
Larry: We want to support whatever the user’s task is; and I think search is actually a very simple term and it encompasses a lot. People use search for a whole variety of reasons, millions and millions of reasons, so you have to be aware of what their intent is and, you talk a lot about that in your report, you support that. If they want to get in and out, that’s one task flow. If they want to have a place where they have access to data and information that is coming from their community, all the social information that they think is valuable, that’s another task flow. So, I think just being aware of the fact that search is multi-faceted, it’s not just a simple single type of task flow.
Kathryn: And another thing we talk about a lot is that Google is really about getting people off of their networks as fast as possible; we tend to want to keep people in our network and introduce them to other properties and experiences. So I think that’s also something that we take a look at.
Gord: So, what’s the challenge for Yahoo for search in the future, if you were looking at your whiteboard of the things that you’re tackling in 2007? We talked a little bit about social search, but as far as the user’s experience, what is the biggest challenge that has to be cracked over the next year or two?
Larry: We’ve been touching on that and I think the biggest challenge is really disambiguating intent. Really trying to understand what does the user want when they enter a few words into the search box. It’s not a lot to work with, obviously. So the biggest challenge is understanding the intent and giving them what they’re looking for, and doing that in the most effective way we can. Yes, probably not anything new but I’d say that is the biggest challenge.
Gord: And in dealing with that challenge, I would suspect that moving beyond the current paradigm is imperative in doing that. We’re used to interacting with search in a certain way, but to do what you’re saying we have to move quickly beyond the idea of a query and getting results back on a fairly static page.
Larry: There are certain expectations that users have, because search is search, and it’s been that way for many years, but I think you can see with our strategy with social search and what we’ve been doing with the integration of Yahoo Answers that it is a shift. And it’s showing that we believe for certain types of queries and for certain information that it’s very useful to bring it up, not just purely algorithmic results.
Gord: I’m just going to wrap up by asking one question, and I guess…somewhat of a self serving question, but with our eye tracking report, are there parts where we align with what you have found?
Larry: No…I found the report fascinating. I think you guys have done a wonderful job. It’s a very interesting read. There is a lot of great information there. And I think there is a lot that is in sync with some of our findings as well. So I think you definitely found some themes that make a lot of sense.
Gord: Thanks very much.
Next week, I talk with Justin Osmer, Senior Product Manager at Microsoft about the new Windows Live Search experience, how MSN Search fared in the eye tracking study, and how MSN Search evolved into the Live Search experience.
Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.