Study Asks, Can You Trust Google’s Personalized Search Results?

New research designed to analyze Google’s personalized search results and make the process the search engine uses more transparent has generated interesting and controversial findings. The study examined a number of commonly held notions about personalization, including the idea that personalization is subtle, that it can often surface more long-tail results, and that personalization is based exclusively on a person’s past search and web browsing behavior. Among the key conclusions: Despite personalized results, for most people search quality has been declining, results are less personal, reflecting more of a standardized Google-centric view than ever before, and that personalized search serves the interests of advertisers more than searchers—even when looking at organic results and excluding paid AdWords listings on a search result page.

The study, by researchers Martin Feuz and Matthew Fuller of the University of London and Felix Stalder of Zurich University of the Arts, focused exclusively on Google given its massive global presence, but the study offers interesting insights into search in general.

A Brief History Of Google Personalized Results

It’s important to note that “personalization” has numerous meanings when it comes to search results. Today, the personalized search results you see are heavily influenced by your own online behavior. But search engines have always aggregated and analyzed data from all users to help improve overall relevancy, using techniques such as collaborative filtering, link popularity and so on. Even PageRank is a form of aggregate analysis, counting “votes” from content creators (links) rather than looking at searcher behavior.

Google initially started tracking search history for users signed in with a Google account in April 2005. In June of that year, it introduced personalized search based on search history as a Google Labs experiment.

Personalized search graduated from Labs to become a feature available to all signed-in users in early 2007. Initially, personalization was based largely on your search history and the results you tended to click on, as well as your Google bookmarks and any content you may have added to your personal iGoogle home page. Google also asserted that personalization was subtle—”What we’ve been doing before is taking two or three results that were suited to your tastes and injecting them. That’s unchanged,” said Google vice president Marissa Mayer at the time.

In April 2007, Google dramatically expanded information it kept about users, going beyond just tracking search history and monitoring everything users did on the web. This hugely increased the data Google could use to fine-tune personalized search results. You could (and still can) opt out of Google collecting your web history and personalizing results by managing your Google account, but you must have an account in the first place to manage this.

In late 2009, Google started personalizing search results for everyone, whether logged into a Google account or not. This means there is no longer any “standard” or “typical” set of Google results for any given query. And somewhat ironically, unless you create and sign in to a Google account, there’s no way to opt out of personalized search results.

Recently, Google has experimented with adding personalization to many of its products and services, including local and product search results (using the location of your IP address or GPS coordinates), its personalized recommendations engine Hotpot and many others. Visit our member library for more on Google’s personalization efforts.

Is Google A Search Engine Or Advertising Company?

The researchers start with the position that Google isn’t really in the business of running a search engine—rather, it’s an advertising company with its “audience” (searchers) as its primary commodity. Just like TV programming, which is expensive to produce and is given away for free to attract an audience, algorithmic search results serve the same purpose, and are “paid for” by advertisers who value the opportunity to promote their goods and services alongside natural search results.

It follows that the more closely targeted advertising is, the more valuable search results are, both to Google and advertisers. Personalizing search results is one way of targeting users. According to the researchers, to personalize results Google builds three-dimensional profiles of users. First, the knowledge person, what you’re interested in based on your queries and click-stream data. Second, the social person—who you’re connected to via email, social networks and other communication tools. Third, the embodied person, your whereabouts as determined by the physical location of your computer or mobile device.

However, the researchers claim that Google is going beyond targeting individual users based on their online behavior, by aggregating personal profiles into statistically related groups. Thus your personalized results will be a combination of what likely interests you, and others who share similar characteristics determined by Google.

Personalizing Results For Immanuel Kant, Friedrich Nietzsche & Michel Foucault

With every searcher getting personalized results, even seeing different results for the same keyword on occasion, the problem was how to actually detect which results were based on user behavior and which were “unpersonalized” results. To accomplish this, the researchers created three personas in the form of famous philosophers: one each from the 18th, 19th and 20th centuries. The search terms used to generate the web history for each philosopher were based on the indexes of seven books from each philosopher.

After “training sessions” that allowed Google to build up search history for each philosopher, results were compared with an “anonymous user,” defined as someone without login credentials or past history with Google services. This allowed the researchers to compare the philosophers’ profiles to a “generic” set of results (as generic as possible, at any rate, since Google personalizes all search results).

Finally, three sets of search terms were used to compare personalized results. The first group was based on terms from the training set that all three philosophers had in common (e.g. aesthetics, knowledge, virtue, etc.). The second group was based on popular tag words from social bookmarking service Delicio.us (e.g. software, travel, blogs, etc.) The third group was based on Amazon’s “Statistically Improbable Phrases” from three books concerning surveillance, network theory and global democracy.

All told, more than 18,000 queries were submitted during the test period during the testing period in July 2009. Only page one results were considered. (Ironically, though Google has long supported many types of research by outside academics, the researchers reported that Google blocked their queries as looking “similar to automated requests from a computer virus or spyware application.”)

The Results: Testing Three Hypotheses

The researchers tested three hypotheses, ultimately rejecting all three based on the data collected.

Hypothesis 1: “Personalization is subtle – at first you may not notice any difference.”

This hypothesis is a direct quote from a 2007 Official Google blog post by Sep Kamvar, Engineering Lead for Personalization, and Marissa Mayer, VP Search & User Experience. Translation: You won’t see many personalized results, especially initially. To the contrary, the researchers found that personalized results appeared relatively quickly, and with the Foucault persona receiving on average 6.4 personalized results out of 10 results on the first page.

The researchers note that even these large number of personalized results may not be easily detectable by users, leading them to ask the question “how can they trust the results?” and conclude that most users will not be able to judge the quality of the personalization Google is performing on their search results.

Hypothesis 2: The more user search history is gathered, the more long-tail content is retrieved.

The promise of personalization is that the more Google knows about your interests, the less likely it should be that you’ll get “generic” results and more likely to get results from deeper in the index that more closely match what you’re looking for—the holy grail of finding the needle in the haystack, or what Chris Anderson has called “the long tail.” What this means practically is that some of personalized search results should be surfaced by Google in the set beyond the first 100 results for a search term.

The researchers found that this was not the case, writing “our research finds that Google personal search does not seem to be able to make long-tail content available in a substantial manner.” Why? They offer three possible reasons, none of which may be mutually exclusive:

• Personalization is limited to re-ranking already highly ranked results.
• Google’s ranking algorithms don’t work well on long tail content.
• Personalization isn’t really about search, but fine-tuning the relationship between users and advertisers.

The third conclusion is obviously the most controversial. Google has consistently maintained that it’s all about relevance for searchers, not about making money. But Google has also consistently found ways to make more and more money from advertising, and both searchers and advertisers would not be happy if long tail content was less relevant than “head” content—even if personalization algorithms suggested that the content was appropriate for a given user based on past behavior and presumed intent.

Hypothesis 3:Personalization reflects only an individual user’s past search and web interests.

During testing, the philosophers all received personalized results for some queries even if there was no relationship between search history and search terms. This led the researchers to conclude that Google has created group profiles over time that are associated not just with search terms, but other demographic information and social preferences, such as age, income, preferred vacation destinations and so on. Unlike true personalization, the researchers suggest that this grouping actually reduces diversity (and therefore quality) in search results.

But as I mentioned at the beginning, search engines have long aggregated data to compile statistical profiles of groups. Of all three hypotheses tested by the researchers, this one seems the weakest and least supported by what we know of how search engines work.

Conclusions And Further Questions

The researchers conclude that personalization is “both taking place to a surprising extent but with relatively trivial results, most likely reflecting that we are in the early stages of the process.” They also note that doing this type of research is difficult due to the dynamism with Google as it constantly changes its algorithms, together with the very nature of personalization itself making it difficult to establish any meaningful universal baselines. But it’s important to continue this type of research because as the authors conclude, “Unless we can update our research methods and tools, we cannot adequately address the social and political issues connected with personalisation and the power of search engines more widely. But we urgently need to do this, otherwise the knowledge and power differentials between those on the inside of search engines and those who are mere users of a powerful but opaque machine are bound to grow.”

Citation:

Personal Web searching in the age of semantic capitalism: Diagnosing the mechanisms of personalisation
Feuz, Martin, Fuller, Matthew, AND Stalder, Felix. “Personal Web searching in the age of semantic capitalism: Diagnosing the mechanisms of personalisation” First Monday [Online], Volume 16 Number 2 (1 February 2011)