The Human Side To Google: Search Evaluations

Google’s next blog post regarding web search is about search evaluation, basically how Google measures quality. Scott Huffman, an Engineering Director who is responsible for leading search evaluation, wrote the blog post.

In short, he explained that the evaluation process is comprised of two main elements:

(1) Human evaluators, actual people around the world trained to rate search results and search UI and experience. We actually covered the Google Quality Raters Handbook over a year ago, which had many details on the training materials for these human evaluators. There are estimated to be over 10,000 evaluators and if you see cart.corp.google.com in your logs, it might be one of those evaluators on your site.
(2) Live traffic experiments, which is us, you and me. Google plants tests on a small fraction of their users, collects data and sees if they improve or hurt the quality of the search results or search experience. That is why we often blog about user interface tests and changes, before everyone sees them, because Google is testing things out on a small user base to decide if they want to make that new UI or Google results available to everyone.

In any event, Scott goes through more of the search evaluation process at his blog post at the Google Blog.