Google Running Feedback Experiment That’s Similar To Human Quality Rater Test

Google Logo - StockGoogle has long asked searchers to provide feedback on the quality of its search results, and often runs a number of tests aimed at encouraging such feedback. The latest such experiment, which seems to have been live for at least a month or so, is a bit different because it asks searchers for feedback only on certain results.

Eli Schwartz recently shared with us a screenshot after doing a search for [product synonym]. As you can see below, Google is using the open space on the right of the results page to ask “Which result do you prefer?”


What’s most interesting is that Google isn’t asking for general feedback on the search results page as a whole; it’s specifically pulling out two of the results — and not the top two. In this case, Google is specifying the third and fourth links and asking the searcher to “visit both pages before choosing.”

That’s similar to the “side-by-side” tasks that Google’s army of human search quality raters often perform — a type of task that you can learn more about in these articles:

In that second article, you’ll find a screenshot of a “basic” side-by-side task that also asks the evaluator to view two specific pages and choose the better result.

The Street recently saw this same feedback form, and pointed out that the “Learn more” link at the bottom leads to a Google help page that says user feedback “will not directly influence the ranking of any single page,” and refers to how the data is used in conjunction with its “professional search evaluators.”

In a typical year, we experiment with tens of thousands of possible changes. These changes, whether minor or major, are tested extensively by professional search evaluators, who look at results and give us feedback, and “live traffic experiments” where we turn on a change for a portion of users. Testing helps us whittle down our list of changes to about 500 improvements per year.

Earlier this summer, Google was running a similar, but less specific, feedback test that asked searchers “How Satisfied Are You With These Results?” That survey asked for feedback on the full search results page.

Related Topics: Channel: Other | Features: Analysis | Google: Web Search | Top News


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • George Bounacos

    I’m actually impressed they’re continuing to test this. I can’t think of anyone who wouldn’t answer 1 or 2 of those every day. That’s a lot of data points, especially for logged in users whose responses they can calibrate against a gajillion other data points.

  • Brandon Lobo

    Well, if they do happen to rank sites based on this data it would be really good for sites that actually offer people useful content…

  • Maurice Walshe

    I would imagine they are trying to validate there in house testers with a real world sample

  • Josh

    I’m going to hire an army of interns to simply tell Google what my competition does not want to hear.

  • daveintheuk

    I’d love to see them ask users if they actually do want Google’s content. “Did you find the big box about easily-scraped simple facts Abraham Lincon useful?”. “Are you glad that we put a blank Google+ Local listing in a huge prominent box on the right and into the listings both under the place’s own page, and with its own Google+ Local entry?”. etc….

  • Matt McGee

    Yes, I’d agree that that’s part of what they’re doing.

  • mikereginatorn

    If this goes worldwide what is stopping people to do just what Josh said?

  • Medic Tourism

    It means that Google still cant recognize quality content.. there is too much spam i guess for those raters..

  • Durant Imboden

    Wouldn’t it be expensive to have an army of paid interns doing thousands of searches on “red widgets” or “blue whatsits” in the hope of triggering an occasional Google survey box?

  • Flemming Kaasgaard

    Just for the sake of argument, let’s put Googles reach at around 1 billion users, which should be a fair assumption in light of available data. If you wan’t to make up just a tiny fraction of that user base, you still need thousands of interns. Of course, they could make many more queries than the average Joe, but that is assuming no flagging method.
    This method of direct user feedback coupled with a bit of filter for automation/unnatural behavior could be a very strong signal, as long as users are not annoyed by it.

  • Josh

    It was a cute joke. There’s no reality behind it. Just a point that it will enable a system (such as automated bots) that /could/ manipulate the system.

  • Ahmad Wali

    This doesn’t show up to everyone. Only random people or certain locations!

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide