13,000 Precision Evaluations: Schmidt’s Testimony Reveals How Google Tests Algorithm Changes

Google Logo - StockPrecision evaluations. Side-by-side experiments with human focus groups. Click evaluations. All of it led to more than 500 algorithm changes in 2010.

Eric Schmidt is speaking today at a Senate subcommittee hearing in Washington, DC, about Google’s business practices, and answering claims that Google’s business practices are anti-competitive. As part of his appearance, Schmidt (as all witnesses do) prepared a set of written remarks that was submitted to the subcommittee prior to his appearance today. The subcommittee has released that document (PDF download), which contains an interesting section that reveals more details about how Google tests its algorithm changes.

Schmidt explains that “potential refinements to the algorithm go through a rigorous testing process, from conception to initial testing in Google’s internal ‘sandbox’ to focused testing to final approval.” He then details how Google uses both human review and internal analysis to develop what amounted to more than 500 algorithmic changes in 2010:

To give you a sense of the scale of the changes that Google considers, in 2010 we conducted 13,311 precision evaluations to see whether proposed algorithm changes improved the quality of its search results, 8,157 side-by-side experiments where it presented two sets of search results to a panel of human testers and had the evaluators rank which set of results was better, and 2,800 click evaluations to see how a small sample of real-life Google users responded to the change.

Ultimately, the process resulted in 516 changes that were determined to be useful to users based on the data and, therefore, were made to Google’s algorithm. Most of these changes are imperceptible to users and affect a very small percentage of websites, but each one of them is implemented only if we believe the change will benefit our users.

It’s no secret that Google uses human “quality raters” — we’ve written about that in past years and others have, too. If you want to read what might be a somewhat outdated document, a Google Quality Raters Handbook was posted online in 2008. It’s since been removed, but our article recapping the handbook is still around.

Related Topics: Channel: SEO | Google: SEO | Google: Web Search


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://www.consultingcumbria.co.uk RayCassidy

    It’s pretty astounding the resource that the Big G throws at its quality control. Just wondering if they can lend me a smidgen of their manpower to teach me how to work to their hymn sheet.

  • http://www.ericgoldman.org Eric Goldman

    They have been publicly sharing these numbers since at least June. I reported on them here: http://blog.ericgoldman.org/archives/2011/06/george_mason_la.htm Unfortunately, they would not provide me with a copy of their slides, so this could be the first time these numbers have shown up in a published document originating from Google. Eric.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide