Chinese Eye Tracking Study: Baidu Vs Google

I thought the easiest way to share the results of the Chinese Eye Tracking study might be in a Q&A format, as that’s what most research programs tend to be anyway. I’ll cover off some of the highlight findings in this column, but the full report will be available on our site next week.

On to the questions:

Q: Why the Chinese market?

Every time I present results from our eye tracking study (as a side note, I just returned from SES Toronto and saw at least 4 or 5 presentations that showed our heat maps or referred to the research. I should have started a royalty program!) the question I most often receive is, “Have you done this in any other cultures?” And the most common request is China. As Westerners, we have a curiosity about the world’s largest market but very little insight into the actual nature of their online interactions. When the opportunity came to present in China, we realized we could actually do a eye tracking study fairly easily.

Our local University has a popular English as a Second Language program for Chinese students so we had a sample available that had only been in North America for a few weeks. We set up our Tobii eye tracking station to be as native an experience as possible for them, with a Chinese keyboard, Chinese character sets and all prompts in simplified Chinese. We also were fortunate to be able to work with Piewen Li from Microsoft, a certified Chinese translator and research manager for the Microsoft Digital Advertising Network. Piewen helped with study design, translation and interpretation of results.

But it was more than just the fact that we could do the study. It was that we should do the study. I cannot overstate the importance of the Chinese Market in the global marketplace. One number that I found mind boggling was the fact that Chinese is the world’s 2nd largest online market, with 150 million users. That puts China only slightly behind the US at 154 million users. But that represents 68% penetration in the US market, and only slightly over 10% in the Chinese market. Obviously, the potential in China is huge and barely untapped.

China presented another interesting question. Why has a home grown engine, Baidu, captured 62% of the market, compared to Google’s 20% share? Baidu’s claim is that it understands the Chinese market better. Was it true? Did Baidu offer Chinese users a more native experience?

The blah blah blah stuff

So, let’s cover off a little housekeeping first. We brought 50 Chinese students between the ages of 18 and 25 into the Enquiro lab. They were given a number of tasks, including free searches and some that we used in our North American study, including looking for online information about a digital camera. We kept this task because we wanted to see if scan patterns were significantly different in China than North America. We also had them look for information on the 2008 Beijing Olympics. After the tasks, we asked them a few questions about where in China they lived, their internet usage and some other basic demographic information. The data collection happened in April.

Q: Do Chinese searchers interact differently than North American searchers?

Yes. A lot differently, in a number of ways. Let’s look at a heat map comparison for a similar task on Google in North America and China.

Notice that there doesn’t appear to be a Golden Triangle or an F Shape scan patterns in the Chinese example. While Google China’s heatmap has the typical upper left orientation hot spot, it doesn’t have the vertical scan down that creates the “leg” of the F. And horizontal scanning is much more spread out. In North America, scan patterns have largely standardized into this F shape. Variations to the norm are minimal.

But in China, other than the upper left orientation and some predisposal to the first listing, we identified little in the way of standardized behavior. We did see (especially on Google) the tendency to scan results in groups of 4 or 5 at a time. This “consideration set” scanning was consistent with what we’ve seen in North America. But the interaction in North America is pretty predictable. A scan down the left side, looking for information scent, then, if scent is picked up, a horizontal scan on the title. In China, after the initial orientation, the eye bounces around the total area of the listing consideration set. There appear to be no consistent patterns.

Another difference is the way we interact with the information in the listings themselves. In North America, we scan and pick up word patterns. We recognize words quickly and determine if they fit in our “semantic maps” (another term covered in our eye tracking studies), but we don’t read the listings.

Because Chinese is presented as symbols, where concepts take their final meaning from a group of combined symbols, it’s much more difficult to scan this information quickly. To try to put in a Western conceptual framework, imagine how difficult it would be to scan meaning from this paragraph if our alphabet was extended to 2000 characters, presented in block letters and all the spaces between words were removed. I can’t do anything about extending the alphabet, but I can change it to block letters and remove the spaces:

TOTRYTOPUTINAWESTERNCONCEPTUALFRAMEWORK,IMAGINEHOW
DIFFICULTITWOULDBETOSCANMEANINGFROMTHISPARAGRAPHIF
OURALPHABETWASEXTENDEDTO2000CHARACTERS,PRESENTEDIN
BLOCKLETTERSANDALLTHESPACESBETWEENWORDSWEREREMOVED

.

One can begin to understand why it might be difficult to scan and pick up individual concepts quickly.

One other amazing comparison was the amount of time spent on the page before the first click. In our North American studies, we saw an average interaction with Google lasted about 8 to 10 seconds. In our study, an average interaction with Google.cn lasted about 30 seconds, and with Baidu over 55 seconds. In interactions with Google, it looks like a the same number of listings are scanned, but in North America, where a quick vertical scan will suffice to determine the degree of relevance, in China, it needs a scan of the entire listing. Using the example again, notice the difference of scanning concentration, particularly on the left side, in the North American example on the left, compared to the Chinese example on the right.

Q: What’s the difference in user experience between Baidu and Google?

When you compare a 62% market share (Baidu) against a 20% market share (Google China) you would think that there should be a significant difference in user experience behind those numbers, right? Well, there is, but not in the way you would expect. Based on North American standards, Google should present a better user experience.

In the comparison above, you can see the difference in degree of scanning for the same task on Baidu and Google. On Google, users found what they were looking for in literally half the time (30 seconds vs 55) and with half the real estate. In Google, most users didn’t scan behind the fourth listing, whereas on Baidu, they scanned right to the bottom of the page. It should also be noted in the Baidu example on the left, the large number of red X’s (indicating mouse clicks) on the top search query box and the suggested searches at the bottom. This indicates that there wasn’t a very high success level on Baidu. So,why the difference?

One of the reasons has to do with the quality of results themselves. The Baidu results page is a pretty murky prospect. There’s virtually no transparency on what’s sponsored and what’s not. There are “preferred listings” that are paid listings, pushing true organic listings down the page. And the preferred listings are cluttered with affiliates and spam. By North American standards, Baidu would be a horrible search experience. But the fact remains, they’re still the preferred choice for the vast majority of Chinese users.

Q: Why choose Baidu?

The answers to this question are complicated. There are a number of aspects that could be part of it.

Home grown vs foreign

There’s a strong preference for a true Chinese search experience. Baidu’s media advertising positions them as a suave Chinese native vs the clueless foreigner (Google). The message is clear, East is Best, and what the ads lack in subtlety, they seem to make up for in impact and have to be considered when looking at Baidu vs. Google factors. I noticed in a blog that picked up our very early findings that several Chinese users commented that they use Baidu because it is Chinese.

The MP3 factor

Baidu is the primary vehicle to locate and download free MP3 files. This generates a huge amount of traffic, as this is one of China’s most popular online activities. There’s no way of knowing what Baidu’s search share might be if you separate the MP3 traffic out, but my suspicion, backed by a conversation with Piewen at Microsoft, is that it would drop dramatically. In fact, Piewen ventured that if the Chinese government decided to crack down on the downloads (something they’re being pressured very heavily to do) it could be an irrecoverable blow to Baidu. Imagine combining all the activity on Napster in its heyday with all the search traffic on Google and you start to understand why the market share is so heavily biased towards Baidu.

Just a last word on this topic. As part of our study, we gave our participants a completely free task to conduct on their engine to kick off the session. They can search for whatever they want to. We typically do this to get them comfortable with the equipment and also to get a baseline heat map of what typical scanning activity should look like. In the Google group, we saw a pretty even distribution of tasks, including information searches and product searches. With the Baidu group, almost every one of them used their free search to look for an MP3 file. A result of conditioning? We suspect so.

Chinese multitasking

Another factor could be found in the nature of a Chinese user’s typical online activities. This was put forward as a possible reason by someone from Baidu for the discrepancies found in the study, although not to myself, so I pass this along second-hand.

Chinese connections tend to be slower so that load times are longer than what we’re used to. They often have multiple windows open. They’re used to seeing several things, often unrelated, at the same time. Perhaps this conditions them to be more accepting of a search experience that Westerners would find irrelevant and unacceptable.

This, together with my introductory look 2 weeks ago, covers off the high level findings and some conjecture on our part about what those findings might mean. You can download a preliminary full report on the Enquiro site. Because so many questions came up that are specific to the culture, we’re still trying to connect with some experts in the Chinese search market (including Google, so far unresponsive) and hopefully Baidu to lend some more insight. We’ll continue to add these insights to the study and notify those that have downloaded previous versions that an updated one is available.

On a Totally Different Note

Over the coming months, we’re very excited to welcome a number of guest writers to Just Behave. These will include key people from the engines, along with other user behavior gurus. These guest writers will fill in for me every second week. I’m thrilled to bring them onboard and it will move us towards my personal goal of making Just Behave the place to turn to for understanding user behavior in the search space. Here are just a few of the search behavior experts who have agreed to submit a column over the next several months:

• Sep Kamvar – the man behind Google’s personalization algorithm
• Larry Cornett – Point person for Yahoo’s usability team
• Justin Osmer – Product Manager for Microsoft Live Search
• Michael Ferguson – IAC and Ask’s usability lead
• Kara Pernice Coyne – Research Director for the Nielsen Norman Group
• Shari Thurow – Well known search usability expert and fellow SEL columnist

More on this as we move into the summer!

Gord Hotchkiss is CEO of Enquiro, a search marketing firm that produces search engine user eye tracking studies and other research. The Just Behave column appears Fridays at Search Engine Land.