I really hate arguments over cloaking. Like really hate them. But Rand Fishkin recently did a chart outlining the degrees of "safeness" to cloaking that in turn riled up Google’s Matt Cutts in comments there. No doubt others at search engines also dislike the idea that any cloaking is "white hat." So I wanted to revisit some of the things that Rand outlined about cloaking plus the guidelines Google updated last month. Over the years, content delivery methods that were once considered cloaking have become acceptable. This is a look at what those are and how we need a new name for them.
How Does Google Define Cloaking?
First, if you want to understand how many tiring debates we’ve had over cloaking issues, I strongly urge you to read my YADAC: Yet Another Debate About Cloaking Happens Again post from last year. One of the things it discusses is how Google dropped its definition of what cloaking was in 2006. Before then, it was:
The term "cloaking" is used to describe a website that returns altered webpages to search engines crawling the site. In other words, the webserver is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they’ll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking.
We were left with this on another page:
Make pages primarily for users, not for search engines. Don’t deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking."
Then around June of last year, as best I can tell, we got this:
Cloaking refers to the practice of presenting different content or URLs to users and search engines. Serving up different results based on user agent may cause your site to be perceived as deceptive and removed from the Google index.
Some examples of cloaking include:
- Serving a page of HTML text to search engines, while showing a page of images or Flash to users.
- Serving different content to search engines than to users.
- Provide alt text that describes images for visitors with screen readers or images turned off in their browsers.
Then in June of this year, Google did a major blog post on the topic, which along with cloaking covered topics such as:
- IP delivery
- First Click Free
Much of what’s in the blog post has yet to migrate to the help pages, though I’m sure it will. But the more important point to me is that over time, things that people might have once considered cloaking have become acceptable to Google — and get named something other than cloaking in the process.
Let’s take geolocation as an example of that. Years ago when Google would campaign against cloaking, those who disagreed would argue that Google itself cloaked. For example, go to Google.com from outside the US and you’ll see a different page than someone in the US sees. Cloaking! No, Google eventually argued — that’s not cloaking. It’s geolocation and perfectly fine for anyone to do.
OK, how about showing Google content that human users only see if they register with a site and/or pay for access. Cloaking! Well, here Google took some time to clarify things for web search. Google informally said that users shouldn’t have to pay to see content if they came from Google. In Google News search, this is called First Click Free, and it’s a formalized process for news publishers to follow (that’s why you can read the Wall Street Journal for free via Google News). But for web search, we never got a formal program or formal mention that this was allowed until the June 2008 blog post.
It’s confusing. Very confusing. Now what was cloaking again?
To help, Rand put out a tactics chart with things rated from "pearly white" to "solid black." That’s nice, but it’s also debatable. In fact, it IS being debated! So I wanted to go back to what Google itself is saying and work from that.
The most important thing is that Rand takes a "tactics" approach — are you using IP detection, cookie detection, and so on — then suggests that as you add things like user agent detection or other tactics, you tip into the danger zone.
In contrast, Google is less about how you physically deliver content and more focused on the user experience — the end result of what a user sees. The bottom line in that remains quite simple — Google generally wants to be treated the same as a "typical" user. So if you typically use cookie detection to deliver content, Google’s got no real issue with that — as long as it sees what a typical user might see who doesn’t accept cookies.
Below, some content delivery methods now acceptable to Google that might have once been considered cloaking:
|Content Delivery Method||Description||How Done||OK With Google If…|
|Geolocation||Users see content tailored to their physical location||IP detection, cookies, user login||Don’t do anything special just for Googlebot|
|First Click Free||Users clicking from Google to a listed page can read page without having to pay or register with the hosting site [if they try to click past that page, it's then OK to toss up a barrier]||IP detection, cookies, user login||You let Googlebot through as if it were a paid/registered member and also allow anyone coming from Google’s search listings through|
Where am I getting the info for the chart above?
- Geolocation & First Click Free come from Google
Webmaster Central’s June 2008
blog post. The post talks about Google’s First Click Free as if it is
only for news content. That’s not correct. You can use this method for
Google web search, as was covered by Google reps speaking at SMX Advanced
2008 last month.
page on cloaking addresses this, plus reps have covered this advice in
many public forums.
- Landing Page Testing: Google Website Optimizer Now Available, But Is It Cloaking? covers the issues here, and How does Website Optimizer fit in with Google’s view of cloaking? from Google AdWords help has detailed advice on how it is acceptable (and when it is not)
- URL Rewriting: Good Cloaking, Evil Cloaking & Detection from Stephan Spencer last year talked about the ways to strip down URLs so they appear "nicer" in search results. He’d surveyed the major search engines that emphatically told him this was OK, even if their spiders were detected in order to do it. I believe this is still fine as long as the content itself isn’t changing.
And what’s not OK?
|Content Delivery Method||Description||How Done|
|Cloaking||Showing Google content that typical users do not see||IP, User Agent Detection|
|Conditional Redirects||Showing Google a redirect code (301) that is different from what someone else sees||IP, User Agent Detection|
Cloaking comes from Google Webmaster Central’s June 2008
blog post and its help
page on the topic. If you’re delivering content to users that’s
different than what Google sees — and it’s not listed on the first chart
– you’re probably cloaking.
- Conditional Redirects: AKA, cloaking redirects. Discussed at SMX Advanced 2008, especially how Amazon has been doing it. Google warned that conditional redirects might be risky.
I said I’m pretty tired over the cloaking debate, right? That’s because way back when the debate started, we had search engines that DID allow cloaking via XML feeds or other methods – or they turned blind eyes to cloaking they knew about — or they had programs like Google Scholar that sure felt like cloaking before guidelines caught up to say no, those aren’t cloaking.
These exceptions made it difficult to say cloaking was "bad" when clearly some of it was allowed. That’s why I urged the search engines — Google in particular — to update the guidelines to say that any type of "unapproved" cloaking might get you banned. My thought was that we could maybe skip past the debate over tactics (and finger-pointing — ah-ha!!!! — this major site is cloaking, ban them!) and focus more on the user experience.
Things have changed since then. The guidelines, as I’ve explained, have gotten more detailed — and exceptions have been spun off into new names. As part of this, the idea of "white hat cloaking" has come out, the "good" cloaking that’s now acceptable.
I can tell you firsthand that Google doesn’t like the phrase "white hat cloaking" at all. To Google, there’s cloaking — it’s always bad — and there are the other content delivery methods I’ve outlined above.
OK, I’ll roll with that. Personally, I’ll avoid talking about "white hat" or "good" cloaking if it helps improve relations between webmasters and the search engines AND helps newbies avoid getting into trouble. But I do think we need a term to encompass content delivery methods that do target spiders or consider them in some way.
If cloaking is a bad word to the search engines that can never be redeemed, we still need a name for the "good" things that are out there. I’ve got my thinking cap on, and if you’ve got ideas, let me know by commenting.
I’ll close by saying that my list above is not complete, in terms of the various content delivery methods that are out there. I hope to grow this over time — and you can help with your comments, as well.