YADAC: Yet Another Debate About Cloaking Happens Again

Sigh. Double sigh. Triple sigh. I guess now that the SEO industry has had the
required twice-yearly
debate about the reputation of SEO, it’s time to do the go round about
cloaking once again.
A quick
word about cloaking has Google’s Matt Cutts trying to clarify concerns that
Philipp Lenssen of Google Blogoscoped
has been raising about WebmasterWorld.
The comments are now up over 100, as people rehash things that have been hashed,
mashed, rebaked so many times before. Below, some cloaking history plus an
honest plea about trying to get past this stupid, stupid issue.

Definition Time

Let’s do the definition, first:

Cloaking is when you show a search engine content that is different than what
a human being sees.

Got it? That’s my definition, and Matt says virtually the same thing in his
post today:

Cloaking is serving different content to users than to search engines.

So simple. What’s to debate? Well, is it cloaking if…

• A spider coming from a US IP address sees a different page than a user
  from a UK IP address?
• A spider sees content that a user sees, but only if they do free
  registration
• A spider sees content that a user sees, but only if they do paid
  registration
• A spider sees content in text that represents what a users sees in Flash
• A spider sees content that’s slightly different than what a user sees when
  their browser renders Javascript

Pick one of those above — pick something else (see our
Good Cloaking, Evil
Cloaking & Detection column from last week) — and people can, will and have
pointed at something someone is doing, then yelled "cloaking" and screamed for a ban
to happen. A ban? Well, as you know, all search engines hate cloaking. Actually,
that’s always been a confused point. Here starts the lesson.

History Time: Tactics Versus Intent

Back in January 2003, Alan Perkins wrote this big

Cloaking Is Always A Bad Idea article, telling us that search engines always
said cloaking was bad. I was never a proponent of cloaking. I was, however, well
aware that NOT all the guidelines were against cloaking. In addition, with paid
inclusion, I argued some cloaking was actually allowed. All this went into my
Ending The
Debate Over Cloaking that came out in reaction to Alan’s article, in
February 2003.

Since I knew that all the search engines had allowed some types of cloaking,
my advice to marketers was this, with the stress on avoiding "unapproved
cloaking:"

Cloaking is getting a search engine to record content for a URL that is
different than what a searcher will ultimately see, often intentionally. It can
be done in many technical ways. Several search engines have explicit bans
against unapproved cloaking, of which Google is the most notable one. Some
people cloak without approval and never have problems. Some even may cloak
accidentally. However, if you cloak intentionally without approval — and if you
deliver content to a search engine that is substantially different from what a
search engine records — then you stand a much larger chance of being penalized
by search engines with penalties against unapproved cloaking. If in doubt, ask
the search engine if it has a problem with what you intend to do, assuming you
can’t get a clear answer from written guidelines that are provided. If you are
working with a third-party search engine marketer, ask them for proof that what
they intend to do is approved. Otherwise, be prepared for any adverse
consequences.

The suggestion to avoid "unapproved cloaking"

infuriated Doug Heil over at the iHelpYou forums, who could not (and to this
day still cannot) get over the idea that cloaking MUST equal spamming.

My response back then remains the same today. There’s a difference between
tactics and intent. Many of the things that might cause penalties with
search engines are tactics (hidden text, gibberish pages, cloaking) that are
closely aligned with the intent of trying to mislead or game the search
algorithms. But in some cases, what’s a bad tactic (or technical implication)
might have a good intent as agreed by the search engines. So they’ll allow it,
either turning a blind eye to it or giving it some official endorsement.

That difference is important because back then, if the search engines got
behind the "unauthorized" versus "authorized" suggestion, we wouldn’t be having
today’s wasteful argument. But let’s carry on.

NPR, Google Scholar & Approved Cloaking

In May 2004, I
looked at
how National Public Radio was, in my view, cloaking text transcripts of audio to
search engines but only letting human visitors by those. At the time, Google had
a guideline against cloaking that read:

The term "cloaking" is used to describe a website that returns altered
webpages to search engines crawling the site. In other words, the webserver is
programmed to return different content to Google than it returns to regular
users, usually in an attempt to distort search engine rankings. This can mislead
users about what they’ll find when they click on a search result. To preserve
the accuracy and quality of our search results, Google may permanently ban from
our index any sites or site authors that engage in cloaking to distort their
search rankings.

I argued that this was an example of "good cloaking" and that the real issue
I had with it was that other marketers were supposedly banned from doing it:

As a searcher, I’m actually glad the method is being used. It does mean I’m
more likely to find audio content of interest. Moreover, I can listen to that
for free via the NPR site.

As a search engine marketer, I’m not so thrilled. I’m well aware that many
other companies would like the ability to feed Google content in this manner.
In addition, they have just as compelling arguments as NPR about having good
content that isn’t adequately indexed by the Google crawler. Unfortunately,
they’re denied the privilege of feeding relevant material just to Google’s
crawler.

What about Yahoo? Anyone can enjoy the same benefits that NPR has, the
ability to cloak content when relevant, through Yahoo’s content acquisition
program. Non-profit organizations are offered this for free. Commercial
organizations have to pay, making use of Yahoo’s trusted feed program.

By November 2004, I was writing about cloaking again. Now Google had an
officially approved program that, in my view, allowed cloaking. This was
Google Scholar. As I
wrote:

This system may lead to problems for some searchers. In the example above,
not only could I NOT read the paper, as I didn’t have a subscription, but I
also could not read even an abstract. Instead, a password-prompt continued to
appear, even when I cancelled it, making it extremely difficult to finally
close the window (and that’s why I haven’t linked to the actual paper, to save
other people the problem).

This situation is probably unusual, however. One of Google’s requirements
for inclusion in Google Scholar is that publishers at least show abstracts to
searchers.

The special access for publishers flies in the face of Google’s
anti-cloaking policy. Google is being shown material that regular users
wouldn’t normally see, its own definition of cloaking. This is a GOOD thing
for searchers, but the company needs to amend its cloaking policy so as not to
be hypocritical.

Indeed, that’s long overdue. This has been a problem since I first reported
about a similar issue earlier this year. A sidebar piece … looks at the
latest case and suggests some fixes for Google, including finally moving
forward with formalizing such programs for ALL publishers.

Plea For Google To Change The Cloaking Guidelines

In the sidebar to that article, and
again in a
follow-up piece a few days later, I urged Google to alter its cloaking policy to
something stressing that cloaking was bad only if not approved:

The term "cloaking" is used to describe a website that returns altered
webpages to search engines crawling the site without permission. In
other words, the webserver is programmed to return different content to Google
than it returns to regular users, usually in an attempt to distort search
engine rankings. This can mislead users about what they’ll find when they
click on a search result. To preserve the accuracy and quality of our search
results, Google may permanently ban from our index any sites or site authors
that engage in cloaking without our permission, if we feel it is harmful to
our search rankings.

BMW, WebmasterWorld & New York Times Cloaking Accusations

In March 2005, there was great amusement in some quarters when Google’s
policy against cloaking caused it to ban itself when pages apparently with text
designed to help internal Google searching made also onto external versions seen
only by Google spiders, rather than humans (see
here,
here and
here for more).

By the end of 2005, WebmasterWorld came under
accusations
of cloaking. Since it is one of the most important forums about search engines
around — frequented by official Google reps — it sort of became a poster child
of "why can they do it but others can’t" for some.

Far bigger news came in 2006. In February, BMW got

banned on Google for using hidden text — in particular, a "poor-man’s"
version of cloaking that used JavaScript to show different content to users. It
got back in a few days later. Then in June, an
article
about how the New York Times optimizes content for search engines sparked a new
cloaking debate when it seemed that the major search engines were allowing
cloaked content. In particular, the New York Times was allowing search spiders
to read content that was only accessible to humans if you registered for free
or, in some cases, paid for access.

Marshall
Simmonds, the NYTimes & Acceptable Cloaking was a giant discussion that came
out of that article, which had me originally arguing that this was cloaking,
since search spiders were seeing something different than most humans could see.
But I was convinced to change my mind. Since the spiders were indeed shown what
anyone could ultimately see, this wasn’t cloaking. I commented:

Until now, I would have considered feeding a search engine a page that people
couldn’t see unless they registered to have been unapproved cloaking, since most
users were seeing something different that then spider saw.

But sure, I’ll buy into the "eventually you’ll see the same thing" argument as
this not being cloaking. Why not? Google’s allowed this in approved cases for
about two years now and never wants to go on the record as this being approved
cloaking. So don’t call it cloaking and everyone’s happy. Google’s not allowing
something officially they say not to do, and content owners can do this without
fear.

In fact, I expect Fantomaster, Beyond Engineering and anyone with IP delivery
lists now can have some new-found respect from people who previously slammed
them as helping cloakers. Here they’ve been saying its all about content
delivery systems and now they’re right, at least in some situations. Because
after all, some people aren’t going to want to depend on user agent delivery to
feed content this way.

Of course, I’m still not going to do this yet with my own members-only content.
Despite agreeing with you, Phil — despite seeing others allowed to do this —
I’m still fearful Google might arbitrarily decide to call it cloaking anyway
when they choose. But maybe I’ll be braver down the line, and why not? It’s like
a whole new world.

I did have several off-the-record conversations with Google about this. The
main thing that came out that I can report was that Google really felt most
users should see what their spiders saw WITHOUT having to register or pay for
access. That’s similar to what Matt wrote about WebmasterWorld content today,
when covering the latest criticisms that it might be cloaked:

I consider the issue in a much better state now, in that most (all?) Google
searchers get the identical page to what Googlebot saw. But I still consider
Philipp’s February posts open for investigation, and I will get to them, in the
same way that I tackled Philipp’s first two posts about this.

FYI, to understand a bit more about the WebmasterWorld situation, see Matt’s
post from March 2006,
How
to sign up for WebmasterWorld. It explains how in some cases, trying to
access a thread that you might come across from a search result can trigger a
registration results. In today’s post at Matt’s, WebmasterWorld’s Brett Tabke
explains and
points at more information on how this has changed.

The comments in Matt’s post also get into an apparent return of some New York
Times content requiring registration, as well as more complaints about Google
Scholar content (added with the full cooperation of Google) also being annoying,
in that you can’t read it without paying when clicking from search results.

Solution 1: Allow Registered Content

Enough history. Time for some solutions. Both Google and Yahoo have programs
that allow people with free registration or fee-based content to show up in
search results — and I mean mixed in with regular search results, not
segregated out like the Yahoo
Subscriptions product launched in June 2005. Yahoo’s will happen mainly
through paid inclusion, and not that much. Google’s will happen primarily
through the aforementioned Google Scholar plus the

First Click Free system used for Google News but

which also may happen with some web search content.

None of this content is labeled in any way. Back to that discussion on
cloaking and the New York Times, I commented:

It is annoying to hit one of these links and not know that payment or
registration is required, however. That problem’s going to get worse as more
and more people decide this isn’t cloaking and give it a go. Google and the
others should look to establish a way for site owners to better flag premium
or registration only content….

I don’t agree having paid content in regular search results is bad. I have
a Wall Street Journal paid subscription. They have lots of great content. If
I’m doing a search, and they’ve got a good match, I want to know that.

And over at Yahoo, as I explained way way back above, I can do that. I can
choose specifically to have this content revealed to me. It doesn’t make my
results bad at all.

It is a bad user experience if you constantly get back results that you
can’t actually view, of course, without paying. We simply aren’t going to
subscribe to everything.

The solution is easy. Give users the option. Let me choose to see content
that requires payment or not. Or similarly, let me choose to see content that
might require free registration. We just need Google and the others to
graduate from 1999 mentality and better accommodate web sites with this type
of content. It’s easily done, if they want to do it. And getting more
formalized program for publishers, as well as options for searchers, will
help.

So enough already. Enough with the special programs that only some publishers
get to do. I want Google — which leads the charge in scaring people about
cloaking — to fast track a system to let anyone with registration-based or
fee-based content to be in their search results.

As for usability, either flag the URLs so users know to expect a charge or
registration request or make it possible for users to exclude this information.
Or experiment with both. But do something so that publishers don’t take matters
into their own hands and the SEO industry has to have yet another one of these
debates over whether it’s cloaking.

Solution 2: Allow For Approved Cloaking

Remember that suggested revision for Google’s cloaking policy I gave above?
Well sometime in 2006, Google dropped its definition of cloaking entirely.
Instead, we were just left with shorter definition of cloaking

here:

Make pages for users, not for search engines. Don’t deceive your users or
present different content to search engines than you display to users, which is
commonly referred to as "cloaking."

And a warning against it

here:

However, certain actions such as cloaking, writing text in such a way that it
can be seen by search engines but not by users, or setting up pages/links with
the sole purpose of fooling search engines may result in removal from our index.

That warning, like the old warning, uses the word "may" in terms of removal
for cloaking. Will cloaking get you banned? Maybe. Maybe if it is noticed. Maybe
if it is deemed harmful to the users. Maybe if after a closer look, it can’t be
pigeonholed in some other definition.

For me, it would be clearer to go back to the old definition and stress that
unless approved, cloaking might result in a ban. Roll that out along with a
program making it easier for people to feed in registered content. That gives
Google flexibility, helps publishers plus stops this insane focus on
technical/tactical implementations and refocuses concern where it belongs — the
user experience.

Did what you do help or harm the search results, in Google’s opinion? If you
were harming search results, Google’s always reserved the right to boot you out.
And if you were technically violating guidelines but not in a harmful way,
Google’s always reserved the right to turn a blind eye. Or rather, an approving
eye, an eye knowing that it’s intent that matters, not some technicality.