Exploring The Shift In Search Behaviors With Microsoft’s Jacquelyn Krones

Jacquelyn Krones (Photo: Annie Laurie Malarkey)

I first met Jacquelyn Krones, a Senior Product Manager from Microsoft, at a search show. A mutual friend on the Bing team, Product Manager Stefan Weitz, introduced her to me and said, “You have to meet Jacquelyn. You speak the same language.”

Stefan was right. Jacquelyn started talking about the research project she was then actively engaged in. She explained about the ethnographic approach Microsoft was taking to understanding search behavior in a broader context. Since then, we’ve been able to continue the conversation at various opportunities in the search conference schedule. It’s always been fascinating.

In today’s column, I wanted to share parts of that conversation with you. I had a chance to talk to Jacquelyn recently about some of the findings that have come out of the study. But before that, I want to talk a little bit about the methodology she chose to use.

Qualitative research explores gray, murky areas to uncover insights impossible through more quantitative methodologies. I believe it’s fair to say that qualitative research discovers, where quantitative research either proves or disproves.

As another fan of qualitative research, Ball State’s Michael Holmes, Director of Insight and Research at the Center for Media Design, once said, “Quantitative is essential for refinement and optimization of what you’re currently doing, but it won’t drive reinvention.”

It was the qualitative angle that Krones took with the research that fascinated me. The other thing that was of note was the fact that this was an ongoing project with Microsoft. This is the third qualitative study looking at search user behavior, with the first conducted in 2004, the second in 2007 and the third having just wrapped up, in 2010.

Given the degree of change in not only the core search experience, but also in rapidly emerging areas such as mobile, I was anxious to hear the results from the latest study. But first, I’ll let Jacquelyn explain a little more about the approach they took:

Jacquelyn Krones:

“We needed to be able to look at the search space with new eyes. To me that says qualitative research, specifically ethnography. That would be one of the techniques I would choose first because it allows you to get so deep and so rich because you’re spending time with people in their homes and in the contexts that matter to them.

In addition to being in their homes, we spent time out in the world with them in the places that matter to them. You just pick up a lot more of what’s important to them and what they actually do. You get to find little bits of insights based on seeing them in a way that you won’t see in a quantitative survey or even in a lab setting with a focus group.

We started with a technique called ZMET, which is a technique that’s based more on psychology. The way that ZMET works is that people bring in pictures that represent the topic that you’re talking about. Then there’s a very deep and structured interview process to get at why that picture represents that topic for them. At the end of it, they create a collage and then tell a story about that topic based on their interview and the pictures that they brought in. So that allowed us to create some basic insights about the space before we were even into ethnography.”

So, did the investment in qualitative research pay off for Microsoft? Did they find the big insights they were looking for? According to Krones, the answer was yes.

Krones:

“We actually did come out with pretty big insights for us, the first one being around how we can organize our thinking about the user activities that they’re involved in when they come to search. I like to tell people that nobody wakes up in the morning saying, “I want to go use a search engine.” We really should be looking beyond what they actually do on the search engine if we want to understand what’s important to them when they get to the search engine.

If you look at the breakdown of the number of minutes that people spend online, I think it’s about 5% of minutes are spent with search engines, and we want to understand what’s happening in the other 95% of the time. People are using search engines to make choices about things they do in the real world, it’s not just an online world. Also, we recognize that the definition of what search is has actually expanded –  a lot of what people are using apps for really could be considered search. Even what people are using their social network for could sometimes be considered search.”

The notion of search behavior being dependent on the context of the real world requirements the user brings with them makes perfect sense, but for some reason, it’s not something we typically spend much time discussing when we draft our search marketing strategies. Krones goes on to explain more about the different search “modalities” the research uncovered…

Krones:

“Another big finding was that there are three kinds of activities that are bringing the people to search.

The first one is a mission where people have a specific, well-defined goal and they want to complete a task efficiently.

The second one is excavation where they have a goal that likely doesn’t have a single right answer and they spend more time with it because they want to get the optimal result. They want the best option, the best price, or the most accurate understanding of an issue.

And then there are explorations, which really come in two types – spending time on something that you care about –  continuing to explore that topic, or passing time while you’re waiting in line, in public transportation or waiting for somebody. And in both cases for exploration, people are optimizing for novelty – they want to see things they haven’t done before, they want to be engaged that way.”

The third finding uncovered in the study was particularly interesting because it charted a change in attitudes amongst search users over the span of the three studies. From 2004 to 2007, and on to 2010, the way we looked at search and its ability to answer our questions changed in both subtle and fundamental ways.

Krones:

“I would say the other big insight for us was around knowledge creation, which is also called “sense-making” in the information science academic discipline. We have really seen a shift over the past several years that we’ve been doing ethnography.

In 2004 people really said that knowledge lives with experts and the experts help them make decisions.

In 2007,  people said that search engines actually had all of the knowledge in the world and it was just there for them to go out and pull it out. And now, in 2010, people told us that they created their own knowledge, that even though the search engine never really had all the knowledge in the world, it was linked to information.

People are much more sophisticated now in how they think about that. They say “The search engine’s a great tool for getting access to information, but I need to look at that information and contrast and compare it, and come to my own conclusion about what the right answer is for me. And when I do that, that’s knowledge, but before that, it isn’t knowledge.” People have a sense that knowledge is something that they are actively creating and that is very personal to them.”

If search behaviors can shift that much, depending on the type of search mission we’re undertaking, then it stands to reason that the type of result should also be able to shift to match our intent. Jacquelyn admits that this is a fairly new insight for search in general:

Krones:

“We were really overly focused on always getting the right response in position 1 on the SERP.  That’s still very important, but sometimes that position 1 is an information scent (a cue to information that lies beyond the link) and sometimes it’s actually an answer.

If you’re on a mission search, we might even be able to answer the question for you by either using universal search or in the snippet in position 1 or position 2 or 3, but if you’re in excavation mode, you’re not actually going to find the answer on the SERP and nothing that we can do for you can actually allow you to say that we nailed it on the SERP because people feel like they need to go through this process.

Our biggest insight in 2007 was that in addition to people looking for what one user called “close doors,” which means “Get a definite answer, move on,” people come to search to open doors –  to open their mind and learn about a topic or to explore. Now in 2010 we went even further beyond that, even though we didn’t start out saying, “Okay, what is there beyond exploring and getting an answer?”

What came back, however, was that we found that there were these three activities and that exploration is its own special thing. If you look at the things that are goal-directed, there really is a pretty big difference between searches where you’re looking for a specific answer and then be done and searches where you really want to get the right answer and you’re going to invest a lot of time. There’s a big difference between optimizing for being efficient and optimizing for being thorough.”

I’ve written before in this column about the “Google Habit”, and I had to ask Jacquelyn if this shift in behavior opened the door to breaking this habit. If our habitual search behaviors weren’t yielding the expected results, might we change those behaviors?

Krones:

I don’t think people actually even are aware that they’re changing their behaviour. What they would tell you is that it is more that the environment has changed and that there’s more information available to them, more of what they need available to them in more contexts. They don’t really see it as “I would have done this on my search engine before,” – they see it as a new behaviour.

The second thing is I would say that is that because people are more sophisticated, they’re becoming more demanding about what they want from a search engine. There’s less glow around the power of a search engine than there was in 2007, which is just a natural part of the product category maturing.

If we really want to delight people and surprise them, which are the things that would cause them to break their habit in a conscious way, we have to recognize that the user’s demanding more, they have higher expectations. I think that’s really exciting actually, because we used to feel like any change to the current search model was just going to be immediately rejected by everyone. I don’t think we feel that way anymore.

In my next Just Behave column, I’ll continue my conversation with Jacquelyn Krones, where we explored more about the three types of search activity, how it plays out across different types of devices, searching through apps and what this all might mean for interfaces and advertising in the future.