BusinessWeek Dives Deep Into Google’s Search Quality

In a series of interviews and articles recently published online, BusinessWeek magazine tries to open up the curtains on Google’s search quality team — the ways team members evaluate Google’s search rankings and their decision-making process when changes are being made.

Perhaps the place to start is with BusinessWeek’s interview of Google CEO Eric Schmidt, where one of the main talking points about search quality comes down to one word: data. Schmidt says Google’s biggets strength is having “so much scale in terms of the data we can bring to bear,” but admits that data is a stumbling block where some real-time and social activities are concerned.

“If we can’t get the data, it’s very, very difficult for us to rank it. Facebook has chosen to keep much of its data behind a wall, that’s what it has decided to do. We favor openness, because we think that works best for the users.

Twitter is a good example of something that is very hard to rank. With real-time, we should over time find a proper way to rank them.”

The series gets more detailed in an interview with Google VP Udi Manber, head of the search quality group. He reveals some stats about the search quality team’s ongoing evaluation of search results.

“We ran over 5,000 experiments last year. Probably 10 experiments for every successful launch. We launch on the order of 100 to 120 a quarter. We have dozens of people working just on the measurement part. We have statisticians who know how to analyze data, we have engineers to build the tools. We have at least five or 10 tools where I can go and see here are five bad things that happened. Like this particular query got bad results because it didn’t find something or the pages were slow or we didn’t get some spell correction.”

Manber explains that Google’s search quality team is able to try any idea it has for improving results, and the infrastructure allows the team to learn “sometimes in a day” if the change is good or bad. He says Google will bring more tools to try solving the challenge of real-time search: “If something is written on the Web that is important, we should bring it back to you in seconds. Right now we’re in minutes. Five years ago, it was once a month.” There are several questions (and answers) on this topic, and I strongly suspect the conversation reveals much of what the Google Caffeine update is about.

In a separate interview, Amit Singhal (who runs Google’s core ranking team) reveals that he’s “dismantled” the original ranking system that Larry Page and Sergey Brin developed. And Singhal talks in depth about the regular meetings Google’s search quality team has, which often involve the discussion and evaluation of “outlandish” ideas.

Scott Huffman leads the Google team that evaluates search results, and he tells BusinessWeek about how Google uses both human evaluators and automated tools to measure how good its search results are. He goes into detail about both, and the explanation of Google’s automated measurement system is particularly interesting.

“On a continuous basis in every one of our data centers, a large set of queries are being run in the background, and we’re looking at the results, looking up our evaluations of them and making sure that all of our quality metrics are within tolerance.

These are queries that we have used as ongoing tests, sort of a sample of queries that we have scored results for; our evaluators have given scores to them. So we’re constantly running these across dozens of locales. Both broad query sets and navigational query sets, like ‘San Francisco bike shop’ to the more mundane, like: Here’s every U.S. state and they have a home page and we better get that home page in the top results, and if we don’t … then literally somebody’s pager goes off.”

Matt Cutts, Google’s head of Web spam, gets the last interview in the series. Search Engine Land readers are probably most familiar with what Matt’s team does, but he does go into detail on how Google tries to distinguish when spam is intentional versus when it’s the result of a site being hacked.

“If you’ve got a longstanding site and then all of a sudden a brand-new directory pops up and it’s got a bunch of spammy terms like online casinos and debt consolidation, pills, and you’ve seen a bunch of weird links from other sites show, then you think maybe this part of the site has been hacked. So let’s not show this directory of sites to people for a little while until we know whether it’s spam or malware—or maybe scan those other 80 pages for malware as well.”

The overall impression I’m left with after reading the whole set of interviews is that, as Schmidt says initially, Google is relentless in its use of data to manage search quality — but it’s also quite willing, especially in recent years, to use human intervention to make sure the automated systems are doing what they should.

Related Topics: Channel: Consumer | Google: Web Search


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide