Google Gutted Its Search Quality Rating Guidelines For Public Release

google-quality-raterAs part of today’s big “How Search Works” reveal, Google also took the big step of sharing its Search Quality Rating Guidelines for the first time.

This is the document that Google’s human search quality raters use when grading Google’s search results.

But the new, public document is actually an edited version of the old one that circulated quietly several times amongst webmasters and SEOs. In fact, “gutted” is more accurate than “edited” — where the most recent non-public version of the document was 161 pages, the public document released today is only 43 pages.

What’s Changed

The biggest change, in my opinion, is the complete removal of Parts 3 and 4 — “Page Quality Rating Guidelines” and “Rating Examples.” These sections offered extremely detailed guidance on how to rate pages, how to rate sections of pages, how to judge the reputation of a website, and specific examples of public web pages and how they should be rated. I’ll share more on this below.

Many sections, in fact, are now missing the specific URL examples that were included in the old document — URLs that matched the different labels of Google’s rating scale, for example (“vital,” “relevant,” “useless,” etc.) It’s likely that Google doesn’t want all those specific examples of pages and how they’re rated to be public. It’s also possible that Google decided web pages/URLs change regularly and it’s not efficient to have to manage a public list of URLs in that setting.

A substantial amount of material related to local search and location-based queries was removed.

There are also some sections that were expanded to add clarification — a section on “thin affiliates” is one example.

I’ve spent most of the morning comparing the most recent “underground” version of the Search Quality Rating Guidelines with the new document made public today, side-by-side, listing all the relevant (and some you might say aren’t relevant!) changes. Who loves ya, baby?

Ready? Here we go….

34 (At Least!) Changes To Google’s Search Quality Rating Guidelines

1) The old document was version 3.27. The new document is 1.0.

2) Google added this preface to the new document:

“Google relies on raters, working in countries and languages around the world, to help us measure the quality of our search results, ranking, and search experience. These raters perform a variety of different kinds of ‘rating tasks’ designed to give us information about the quality of different kinds of results in response to different kinds of queries. The data they generate is rolled up statistically to give us within the Google search team a view of the quality of our search results and search experience over time, as well as an ability to measure the effect of proposed changes to Google’s search algorithms. Raters’ judgments do not directly impact Google’s search result rankings. While a rater may give a particular URL a score, that score does not directly increase or decrease a given website’s ranking. Instead these scores are used in aggregate to evaluate search quality and make decisions about changes.

This document is a ‘Cliff’s Notes’ version of our search quality rating guidelines. By this, we mean that it is not the entire version that raters use on a daily basis; however, it is a summary of the important topics. The raters’ version includes instruction on using the rating interface, additional rating examples, etc. These guidelines are used as rating specifications for search raters, and this document in particular focuses on a core type of rating task called ‘URL rating.’ In a URL rating task, a rater is shown a search query from their locale (country + language) and a URL that could be returned by a search engine for that query. The raters ‘rate’ the quality of that result for that query, on a scale described within the document. Sounds simple, right? As you’ll see, there are many cases to think through, and this document is used to guide raters on some of those cases and how to look at them.

Our search quality rating guidelines are in constant flux as we learn and search evolves over time. We’ve created this version especially for those individuals who want to understand better how Google thinks about relevance and quality of search results.”

3) Section 1.3 – “The Purpose of Search Quality Rating”

Old text: “Your ratings will be used to evaluate search engine quality around the world. Good search engines give results that are helpful for users in their specific language and location.”

New text (change in bold): “Your ratings will be used to evaluate search engine quality around the world. Good search engines give results that are helpful for users in their specific language and location. Please note that your ratings do not directly impact Google’s search result rankings or ranking algorithms.

That’s something that Google has emphasized in the past, and obviously has reason to emphasize again.

3) Google removed Section 1.6 (“Releasing Tasks”).

This section explained that it’s okay for raters to skip (release) some rating tasks, and described some cases when that’s acceptable, i.e., “You believe that the landing page will be offensive to you.”

4) Section 2.4.1 (“Action Queries – Do”) and 2.4.2 (“Information Queries – Know”)

In these sections, Google edited the charts showing the relationship between a query and a landing page; example URLs of helpful pages were removed, as shown below.

OLD

2-4-2-old

NEW

2-4-2-new

The example URLs were left in Section 2.4.3 (Navigation Queries – “Go”) and Section 2.4.4 (Queries with Multiple User Intents (Do-Know-Go)).

5) In Section 3.0, “The Language of the Landing Page,” a chart showing examples of landing page language flags was removed.

6) In Section 4.1, “Vital,” which is about the highest rating that a page can receive, Google removed a paragraph directing the reader to a later part of the document.

7) Sections 4.1.1 through 4.1.3 were removed. These were titled:

  • 4.1.1 – Examples of English (US) Navigation Queries with Vital Pages for the Task Location
  • 4.1.2 – Examples of Entity Queries with Vital Pages
  • 4.1.3 Vital Pages for People Queries

Each section included a chart listing examples of “vital” web pages for different types of queries.

8) Section 4.1.4 (“Other Important Vital Concepts”) was changed to become Section 4.1.1 (“Important Vital Concepts”) and was edited substantially. A chart showing examples of queries that don’t have vital pages was left in, but two charts showing examples of vital pages were removed — one chart showed how there may be more than vital page for some queries, and the other showed examples of unofficial website that look official and should not be considered vital.

9) Section 4.1.5 (“Vital Pages and Geographic Location”) was removed completely. This section offered guidance for dealing with websites and web pages that have multiple versions for different languages.

10) Also removed completely are:

  • Section 4.2.1 (“Examples of Useful Pages”)
  • Section 4.3.1 (“Examples of Relevant Pages”)
  • Section 4.4.1 (“Examples of Slightly Relevant Pages”)
  • 4.5.1 (“Examples of Off-Topic or Useless Pages”)

Each of these sections showed specific web pages matching the labels from Google’s rating scale. The company clearly doesn’t want the general public — and the companies whose pages were used as examples — to see specific examples of how Google assesses their pages. And, it’s also likely that these examples were removed because web pages change and some of the ratings may not have been accurate.

One interesting example is this paragraph from Section 4.4.1 which is now gone:

Please note that not all pages with copied content are considered “low quality”. The website www.answers.com contains content copied from Wikipedia.org and other dictionary and encyclopedia sites, but is not considered to be a low quality site because the content is well-organized and intended to be helpful for users. Similarly, there are pages on medical information sites that contain copied content. If the page is well-organized and appears to be designed to be helpful for users and not just to display ads for users to click on, it should be rated based on how helpful the content would be for users.

I’m guessing that Google wouldn’t want an endorsement of Answers.com like that to be widely distributed.

11) In 4.5 (“Off-Topic or Useless”), this paragraph was added to the new document:

A rating of Off-Topic or Useless also applies when there is lack of attention to an aspect of the query that is important for satisfying user intent.

12) In Section 4.6.1 (“Unratable: Didn’t Load”), some text describing a military site that triggers a message saying its security certificate isn’t trusted was removed, and a chart showing similar examples was edited to remove the URLs of sample pages that deserve the “didn’t load” rating. Another chart showing similar web page-generated messages was removed.

13) Charts with example URLs were removed from

  • Section 5.1 (“User Intent and Page Utility”).
  • Section 5.2 (“Location is Important”)
  • Section 5.3 (“Language is Important (This section is for Non-English Task Languages)”)
  • Section 5.5 (“Specificity of Queries and Landing Pages”)

14) In the “Common Rating Problems” section of the document, Section 5.6.1 (“Dictionary or Encyclopedia Results”) was changed as you see here:

OLD

5-6-1-old

NEW

5-6-1-new

15) Section 5.6.2 (“Action vs. Information Intent”) was removed completely from the new document.

5-6-2-gone

16) Section 5.6.3 (“Queries that Ask for a List”) became 5.6.2, and a chart with example URLs was removed. Ditto for Section 5.6.4 (“Misspelled and Mistyped Queries”).

17) The very detailed Section 5.6.5 (“URL Queries”) was renamed to 5.6.4, but remains very detailed. Some example URLs were changed, and two charts with example URLs were removed.

18) Section 5.6.6 (“New and Old Pages”) becomes 5.6.5 and two charts with example URLs were removed.

19) Section 5.6.7 (“Search Engine Result Pages”) was removed entirely, and probably should’ve been a long time ago. This section tells raters that they should rate search engine result pages “just like any other landing page” … even though Google has said for years that it doesn’t want its own search results to include search result pages.

20) Section 5.6.8 (“Video Landing Pages”) was also removed completely.

21) In the “Flags” section of the document, these sections were edited substantially, usually to remove charts with example URLs:

  • Section 6.2.1 (“Clear Non-Porn Intent”)
  • Section 6.2.2 (“Possible Porn Intent”)
  • Section 6.2.2 (“Clear Porn Intent”)

22) Section 6.2.4 (“Reporting Illegal Images”), which discusses child pornography and bestiality and has specific reporting instructions for both Leapforce and Lionbridge employees (the two companies that hire and manage the Quality Raters), was removed.

23) There are significant changes to “Part 2: URL Rating Tasks with User Locations”, where several sections were removed completely:

  • Section 1.0 (“Important Definitions”)
  • Section 1.1 (“What is the User Location?”)
  • Section 1.2 (“Why are the Task Location and User Location important?”)
  • Section 1.3 (“User Location, Task Location, and Explicit Location in the query”)

All of those were replaced with a Section 1.0 called “Query Locations” that’s about half as long as the old material.

24) Section 2.0 (“Location-Specific Rating Task Screenshot”) was changed — a chart was moved to Section 1.0 (“Query Locations”).

25) Another major change in this section is the consolidation of 11 pages of local query information down to two. These sections are gone and/or were condensed:

  • Section 3.0 (“The Role of User Location in Understanding Query Interpretation/User Intent”)
  • Section 3.1 (“Queries with Local Intent”)
  • Section 3.2 (“Rating Landing Pages when the task has a User Location”) Unchanged
  • Section 3.3 (“Vital Ratings for Rating Tasks with User Locations”)
  • Section 3.4 (“Rating Examples”)

Those are replaced by two sections:

  • Section 3.0 (“Assigning a Rating When There is a Query Location”)
  • Section 3.1 (“When Does the Query Location Matter?”)

26) Parts 3 and 4 of the old document are completely missing from the new one. Part 3 was called “Page Quality Rating Guidelines” and Part 4 was “Rating Examples.”

They totaled about 50 pages of very detailed instructions and guidelines about identifying the quality of web pages, identifying “main content” and “supplemental content” on a page, how to rate the layout of a page, how to identify the reputation of a website, determining if content is high, medium, or low quality … and so forth.

Some of the material in these two sections was very “inside baseball” and not stuff that the general public would care about, but certainly anyone creating web content would.

27) “Webspam Guidelines” used to be Part 5 of the document, but becomes Part 3 with the removal of the two sections described above.

Some text has been edited to educate the reader about things like PPC ads, and some text has been removed — including a section (2.0) on browser requirements that human raters have to follow, text that linked to screenshot examples showing raters how to check for spam (i.e., “use Ctrl-A to reveal hidden text”), and text-based instructions for doing this like disabling javascript.

28) In the new document, Section 2.2 (“Keyword Stuffing”), these two bullet items were added to the list of examples:

  • Pages with a large amount of what look like gibberish or random keywords.
  • Pages that appear to be programmatically or automatically generated text that doesn’t really make sense.

29) This sentence was added to Section 2.4 (“Cloaking”): “True cloaking is somewhat rare, but spammers do use other methods to show different pages to search engines than to users.”

30) Section 3.1.5 (“Copied Message Boards”) now mentions “copies of Usenet posts” as an example of spam.

31) In the old document, a section on “Recognizing Copied Content” was broken up in the new document to two sections, but the content is the same.

32) The definition of “thin affiliate” has been clarified in the new document; the sentence in bold is new:

A thin affiliate is a site that offers little additional information and does not offer substantial value to users compared to many other sources on the Web. For example, an affiliate that has only copied content from the merchant site is considered a thin affiliate. This is a moneymaking spam technique.

Also in the new document, these two bullet points describing thin affiliates have been removed:

  • Click buttons on the page. Click on a “More Information” or “Make a Purchase” button. If you are taken to a merchant on a different domain, it is probably a thin affiliate. You will not be able to make the purchase on the affiliate webpage.
  • Check properties of images on the page. Right-click on an image on the page with your mouse and look at “Properties” to see where the image originates. Check to see if the address of the image is the same as the address of the page or if it is the address of a “real” merchant.

33) The instructions for identifying “Pages with Unhelpful Content and PPC Ads” have been expanded. Google says raters should ask themselves these questions:

  • Is the content likely to be helpful to users or is it too general, too poorly written, or gibberish?
  • Does the page provide substantial value when compared to other pages in search results?
  • Does the page have an excessive amount of ads that distract from or interfere with the main content?
  • Would you trust the content?
  • Would you be comfortable giving your credit card information to the site?

34) In the new document, “Part 6: Using EWOQ” is removed — that has instructions for using the evaluation system where tasks are performed. Also gone is “Part 7: Quick Guide to URL Rating” and “Part 8: Quick Guide to Webspam Recognition.”

Conclusion

I think it’s great that Google has publicly released this Search Quality Rating Guidelines document. But the changes above strongly suggest that this is not the document that its hired human raters will be using, but more of a watered-down, public-friendly version.

Related Topics: Channel: SEO | Features: Analysis | Google: Search Quality Raters | Google: SEO | Top News

Sponsored


About The Author: is Editor-In-Chief of Search Engine Land. His news career includes time spent in TV, radio, and print journalism. His web career continues to include a small number of SEO and social media consulting clients, as well as regular speaking engagements at marketing events around the U.S. He recently launched a site dedicated to Google Glass called Glass Almanac and also blogs at Small Business Search Marketing. Matt can be found on Twitter at @MattMcGee and/or on Google Plus. You can read Matt's disclosures on his personal blog.

Connect with the author via: Email | Twitter | Google+ | LinkedIn



SearchCap:

Get all the top search stories emailed daily!  

Share

Other ways to share:
 

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://twitter.com/ChaseSEO Chase Anderson

    Terrific post – thank you for spending the time to review these in such massive detail!

  • http://twitter.com/jimbeetle jimbeetle

    “But the changes above strongly suggest that this is not the document that its hired human raters will be using, but more of a watered-down, public-friendly version.”

    No, it’s not the document the raters will be using. Google stated up front that “This document is a ‘Cliff’s Notes’ version of our search quality rating guidelines,” and that “it is a summary of the important topics,” and that “The raters’ version includes instruction on using the rating interface, additional rating examples, etc.”

    Ain’t no smoking gun here.
    ,

  • Lyndon NA

    Me thinks someone has seen the other versions (bit of a give away really :D).
    Darn fine post (and I bet that was hard work).

    What I don’t understand is why G would provide this under the same label?
    Why not at least give it a different name, or state it was redacted/cutback?
    Instead of looking honest and interested in getting people looking at quality – they now look somewhat slimey and as if they were trying to pull a fast one :(

  • Matt McGee

    Well said, Jim. Agreed.

  • Matt McGee

    No, I don’t think it looks slimy. As Jim points out, and as I include early in the article, they do specifically say this is a Cliff’s Notes of the real thing.

  • Miklin SEO

    wow Google

  • http://www.devdigital.com/ Mitch Ballard

    Excellent article Matt and thanks for sharing such long informative post.

  • http://entertainmentbuzzz.com/ Jia

    I totally agree Jim.

  • Durant Imboden

    You went to a great deal trouble to provide exhaustive proof of what Google had already said it was doing (presenting a ‘Cliff’s Notes’ version of the rating guidelines), but that’s one of the things I like about SearchEngineLand: You folks haven’t bought into the premise that nobody on the Web has an attention span of more than 30 seconds. Good for you!

  • Matt McGee

    Thx Durant. It might be a bit of overkill, but even if they say it’s an abridged version, I think there’s value in looking at how it was abridged.

  • http://www.brickmarketing.com/ Nick Stamoulis

    “If the page is well-organized and appears to be designed to be
    helpful for users and not just to display ads for users to click on, it
    should be rated based on how helpful the content would be for users.”

    I think it’s interesting that Google added that side note. You’ll notice they said “should be rated” and not “will be”—still leaves a lot of room for interpretation and that’s how some site owners get into trouble. My thought is if you feel like you have to squeeze your site past any of these guidelines by splitting errors you’re probably missing the mark on some level.

  • cheryl511

    upto I looked at the paycheck which said $5281, I did not believe that…my… neighbours mother was realy making money part-time from there pretty old laptop.. there great aunt haz done this 4 only about seven months and by now paid the loans on their villa and bourt a great Mazda. go to, jump15.comCHECK IT OUT

  • http://twitter.com/OurMarketingGuy CallTheMarketingGuy

    Great article, These are healthy changes in my opinion ,and kudos for all your research.

  • deborah638

    If you think Jesus`s story is really great…, last pay check my mum’s best friend got paid $5984 workin a seventeen hour week from their apartment and their roomate’s aunt`s neighbour has been doing this for three months and made more than $5984 parttime on their labtop. applie the information available at this link, jump15.comCHECK IT OUT

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!

 


 

Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide