Google Webmaster Tools has just launched a “labs” section, where you’ll find new features that may be early in the development cycle and not quite as robust as the rest of the tools. The features available so far are Fetch as Googlebot, which lets you see exactly what Googlebot is served when it requests a URL from your server and Malware Details, which shows you malicious code snippets from your site if it’s been flagged as containing malware.

Fetch as Googlebot

Of most interest to webmasters, SEOs, and web developers is likely the Fetch as Googlebot feature. You can specify any URL on your site and see the HTTP response (header and contents) that the server returns. Simply  indicate the URL and click the Fetch button. It may take a few moments for Googlebot to access the page and return the results, since it fetches the page in real time. (Refresh the page to see the progress.)

Click the Success link once it’s been processed to see the results.

How is this different from simply looking at the source code of the page?

• You see the HTTP header information at the top. This information is generally easily available through tools such as Live HTTP Headers, but isn’t contained in the source code itself (since that information is coming from the server, not the page).
• You can see if the server is returning any of the page information differently than the page has been coded.
• You can see if the server is returning something different to Googlebot than what other users see. This tool uses the same user-agent and IP range as Googlebot when it crawls the web, so if the server is configured conditionally for user agent or IP address (typically known as “cloaking“), you’ll see  what’s being conditionally served to Google.
• You can use the tool to test changes (particularly things like redirects) in real-time.

Note that this tool won’t necessarily show you the content that Google is able to extract from the page. If the page contains JavaScript, for instance, you’ll see the raw JavaScript code contained on the page, not the rendered view visible in the browser. Which, unfortunately means you can’t use this tool to determine if Google is able to access content contained in rich markup.

What’s this about cloaking?

This tool can help you determine if the pages are being cloaked to Google. This may be useful if you’re coming into a project late and aren’t sure what’s been previously done. It can also help uncover if your site has been hacked. Back in 2006, Googler Matt Cutts and I did a show on Webmaster Radio during which we talked about how in some cases, a hacker might add links to a site and then cloak those pages so that the site owner never sees them. Only Google does. At the time, Matt suggested using Google Translate (and choosing English to English) to see what Googlebot was being served, but this tool can now more easily serve that purpose. Matt confirmed this to me this morning: “The biggest use case is just debugging site issues. Of those, the biggest case will be hacked sites. Some attacks will hide content until search engines fetch the page (and some attackers add a noarchive tag so that the search result doesn’t have a “Cached” link), so a site could look clean to the website owner. Using this feature will site owners verify that there are no hidden links in the page that Google actually fetches.”

How do I test redirects?

If you’ve implemented redirects, you can use this tool to test how Googlebot will interpret those redirects without waiting for those pages to be crawled. For instance, when I fetch www.searchengineland.com, I see that the redirect is correctly implemented as a 301 and points to searchengineland.com:

You can also use the tool to troubleshoot URLs listed in the Crawl Errors > Not Followed report. You can also test these URLs using something like Live HTTP Headers or by trying to access the URLs in a browser, but if neither of those methods uncover the problem, this tool can help determine that the issue is specific to Googlebot. You can also use this tool to verify that fixes you’ve made to redirect errors uncovered by the Not Followed report have really solved the problem.

(Note that the tool currently has a limit of 100kb per page. However, this is for the tool only and doesn’t apply to Googlebot’s normal crawl of the site. Google is monitoring feedback to see if many site owners find this size to be limiting.)

Malware details

The Google Online Security Blog has more information on the malware details tool. Previously, webmaster tools reported when the site was flagged has having malware and listed sample URLs. This new tool will also show samples of the malicious content, and in some cases, the underlying cause. This should help those site owners whose sites have been hacked to include malware find the problem and fix it. If your site does contain malware and you’ve fixed it, you can request a review to have the malware alert removed in search results.

Google’s long had a “First Click Free” program that allows news publishers to make their content accessible to search spiders but requires human visitors to login if they’ve already viewed one page on the site for free — hence the “first click free” name. Earlier this year, Google said this program was OK for web publishers to use — not just news publishers. Today, they’ve done a post more formally making that point.

Let me take you through some of the history of this program as it relates to web search.

Back in March 2007, Danny Sullivan’s Yet Another Debate About Cloaking Happens article talked about how First Click Free was offered to news publishers but not general web publishers, something he wanted to see changed

In September 2007, First Click Free: Accessing Subscription-Based Articles For Free Via Google News covers how Google did a blog post to help promote the idea of First Click Free. It still was a news publisher-only program, but many news publishers weren’t aware of it.

In June 2008, Google did a blog post saying that Google News’s First Click Free policy could be applied to “to include your premium or subscription-based content in Google’s websearch index without violating our quality guidelines.”

Many people missed this important change. Danny highlighted it in a post that came out of SMX Advanced. Over at Search Engine Roundtable, Matt Cutts also commented to clarify things, after Google elsewhere has mistakenly said First Click Free was something “reserved” for Google News. Matt said:

First Click Free originated with Google News, but you can use the same way of handling content in web search (show the same page to users and Googlebot, then if the user clicks to read a different article, then you can show them the registration or pay page).

Because the same page is presented to users and to Googlebot, it’s not cloaking. So First Click Free is a great way if you have premium content to surface it in Google’s web index without cloaking. Hope that makes sense.

Today’s post should help end any further confusion. First Click Free isn’t just for Google News — anyone can use it.

I really hate arguments over cloaking. Like really hate them. But Rand Fishkin recently did a chart outlining the degrees of "safeness" to cloaking that in turn riled up Google’s Matt Cutts in comments there. No doubt others at search engines also dislike the idea that any cloaking is "white hat." So I wanted to revisit some of the things that Rand outlined about cloaking plus the guidelines Google updated last month. Over the years, content delivery methods that were once considered cloaking have become acceptable. This is a look at what those are and how we need a new name for them.

How Does Google Define Cloaking?

First, if you want to understand how many tiring debates we’ve had over cloaking issues, I strongly urge you to read my YADAC: Yet Another Debate About Cloaking Happens Again post from last year. One of the things it discusses is how Google dropped its definition of what cloaking was in 2006. Before then, it was:

The term "cloaking" is used to describe a website that returns altered webpages to search engines crawling the site. In other words, the webserver is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they’ll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking.

We were left with this on another page:

Make pages primarily for users, not for search engines. Don’t deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking."

Then around June of last year, as best I can tell, we got this:

Cloaking refers to the practice of presenting different content or URLs to users and search engines. Serving up different results based on user agent may cause your site to be perceived as deceptive and removed from the Google index.

Some examples of cloaking include:

• Serving a page of HTML text to search engines, while showing a page of images or Flash to users.
• Serving different content to search engines than to users.

If your site contains elements that aren’t crawlable by search engines (such as Flash, Javascript, or images), you shouldn’t provide cloaked content to search engines. Rather, you should consider visitors to your site who are unable to view these elements as well. For instance:

• Provide alt text that describes images for visitors with screen readers or images turned off in their browsers.
• Provide the textual contents of Javascript in a noscript tag.

Ensure that you provide the same content in both elements (for instance, provide the same text in the Javascript as in the noscript tag). Including substantially different content in the alternate element may cause Google to take action on the site.

Then in June of this year, Google did a major blog post on the topic, which along with cloaking covered topics such as:

• Geolocation
• IP delivery
• First Click Free

Much of what’s in the blog post has yet to migrate to the help pages, though I’m sure it will. But the more important point to me is that over time, things that people might have once considered cloaking have become acceptable to Google — and get named something other than cloaking in the process.

Renaming Confusion

Let’s take geolocation as an example of that. Years ago when Google would campaign against cloaking, those who disagreed would argue that Google itself cloaked. For example, go to Google.com from outside the US and you’ll see a different page than someone in the US sees. Cloaking! No, Google eventually argued — that’s not cloaking. It’s geolocation and perfectly fine for anyone to do.

OK, how about showing Google content that human users only see if they register with a site and/or pay for access. Cloaking! Well, here Google took some time to clarify things for web search. Google informally said that users shouldn’t have to pay to see content if they came from Google. In Google News search, this is called First Click Free, and it’s a formalized process for news publishers to follow (that’s why you can read the Wall Street Journal for free via Google News). But for web search, we never got a formal program or formal mention that this was allowed until the June 2008 blog post.

Along the way, we’ve also had more advice about doing things to make Flash content visible through JavaScript detection and replacement (see today’s excellent post, Google Now Crawling And Indexing Flash Content, from Vanessa Fox on those methods plus the new Flash reading that Google’s doing). We’ve also had advice about rewriting URLs to make them more search engine friendly — good advice – but also a situation that in some cases can be URL cloaking. We’ve also had situations where the search engines have absolutely known someone was cloaking — no ifs, ands or buts — but decided the intent wasn’t bad so let it go through (which is fine for me — I care more about intent than kneejerk reaction to a bad "tactic").

It’s confusing. Very confusing. Now what was cloaking again?

Charting Acceptability

To help, Rand put out a tactics chart with things rated from "pearly white" to "solid black." That’s nice, but it’s also debatable. In fact, it IS being debated! So I wanted to go back to what Google itself is saying and work from that.

The most important thing is that Rand takes a "tactics" approach — are you using IP detection, cookie detection, and so on — then suggests that as you add things like user agent detection or other tactics, you tip into the danger zone.

In contrast, Google is less about how you physically deliver content and more focused on the user experience — the end result of what a user sees. The bottom line in that remains quite simple — Google generally wants to be treated the same as a "typical" user. So if you typically use cookie detection to deliver content, Google’s got no real issue with that — as long as it sees what a typical user might see who doesn’t accept cookies.

Below, some content delivery methods now acceptable to Google that might have once been considered cloaking:

Where am I getting the info for the chart above?

• Geolocation & First Click Free come from Google Webmaster Central’s June 2008 blog post. The post talks about Google’s First Click Free as if it is only for news content. That’s not correct. You can use this method for Google web search, as was covered by Google reps speaking at SMX Advanced 2008 last month.
   
• JavaScript Replacement: Google’s page on cloaking addresses this, plus reps have covered this advice in many public forums.
   
• Landing Page Testing: Google Website Optimizer Now Available, But Is It Cloaking? covers the issues here, and How does Website Optimizer fit in with Google’s view of cloaking? from Google AdWords help has detailed advice on how it is acceptable (and when it is not)

• URL Rewriting: Good Cloaking, Evil Cloaking & Detection from Stephan Spencer last year talked about the ways to strip down URLs so they appear "nicer" in search results. He’d surveyed the major search engines that emphatically told him this was OK, even if their spiders were detected in order to do it. I believe this is still fine as long as the content itself isn’t changing.

And what’s not OK?

• Cloaking comes from Google Webmaster Central’s June 2008 blog post and its help page on the topic. If you’re delivering content to users that’s different than what Google sees — and it’s not listed on the first chart – you’re probably cloaking.
   
• Conditional Redirects: AKA, cloaking redirects. Discussed at SMX Advanced 2008, especially how Amazon has been doing it. Google warned that conditional redirects might be risky.

"Whitehat" Cloaking

I said I’m pretty tired over the cloaking debate, right? That’s because way back when the debate started, we had search engines that DID allow cloaking via XML feeds or other methods – or they turned blind eyes to cloaking they knew about — or they had programs like Google Scholar that sure felt like cloaking before guidelines caught up to say no, those aren’t cloaking.

These exceptions made it difficult to say cloaking was "bad" when clearly some of it was allowed. That’s why I urged the search engines — Google in particular — to update the guidelines to say that any type of "unapproved" cloaking might get you banned. My thought was that we could maybe skip past the debate over tactics (and finger-pointing — ah-ha!!!! — this major site is cloaking, ban them!) and focus more on the user experience.

Things have changed since then. The guidelines, as I’ve explained, have gotten more detailed — and exceptions have been spun off into new names. As part of this, the idea of "white hat cloaking" has come out, the "good" cloaking that’s now acceptable.

I can tell you firsthand that Google doesn’t like the phrase "white hat cloaking" at all. To Google, there’s cloaking — it’s always bad — and there are the other content delivery methods I’ve outlined above.

OK, I’ll roll with that. Personally, I’ll avoid talking about "white hat" or "good" cloaking if it helps improve relations between webmasters and the search engines AND helps newbies avoid getting into trouble. But I do think we need a term to encompass content delivery methods that do target spiders or consider them in some way.

First Click Free, JavaScript replacement, even geolocation — while maybe you don’t do something special for the search engines as part of them – you’re still considering them. Indeed, part of what you consider is to ensure that you might NOT do something special.

If cloaking is a bad word to the search engines that can never be redeemed, we still need a name for the "good" things that are out there. I’ve got my thinking cap on, and if you’ve got ideas, let me know by commenting.

I’ll close by saying that my list above is not complete, in terms of the various content delivery methods that are out there. I hope to grow this over time — and you can help with your comments, as well.

Maile Ohye at the Google Webmaster Central Blog has “defined” what Google considers to be IP delivery, geolocation, and cloaking.

On the geolocation front, Google recommends you treat “Googlebot as you would a typical user from a similar location.” So, if Googlebot’s IP is coming from California, then serve up the same page you would serve a web user from California. The same rule applies to IP delivery: serve the “same content a typical user from the same IP address would see” to Googlebot. Clearly, if you serve different content to Googlebot then you would a normal user, that would be cloaking and would go against Google’s guidelines. Finally, Google discusses the first click free program from Google News, where if you visit an article a second time, you would need to enter a password. When serving this content, you need to make sure the content is the same as when a normal user goes to the page.

Did Google really “define” what they think cloaking or IP delivery is? Well, if you consider a definition to remove any doubt or questions, then no. If you look at the related Google Groups thread, you can clearly see there is a lot of confusion and “what-ifs” in the thread. So although the post did clarify some issues to those who are unaware of the issues with IP delivery and its various forms, it did not necessarily cover every “what-if” scenario. Will it ever? I doubt it.

Here is an awesome video from Maile on IP delivery:

Today, Microsoft has posted to the Live Search Webmaster Center blog that the cloaking detection system they have been running for the past few months has, for some sites, skewed site statistics and ads reporting, as well as caused a high traffic load, and they have made some adjustments to the process to correct these errors. Below, details on the problems this process caused, what Live Search is doing about them, and a brief look at how the engines have historically dealt with cloaking.

For the last few months, there have been slight rumblings in the places webmasters frequent about odd referrers and traffic from MSNbot, as well as from a bot not identified as such, but coming from an IP range identified as belonging to Microsoft.

In September, MSNDude posted on Webmaster World, confirming that the traffic was coming from Microsoft as a result of “quality check” and that webmasters who blocked this bot faced being removed from the Live Search index.

Now Microsoft is further clarifying that the “quality check” referred to cloaking detection, although they allow that some cloaking may be valid.

In the blog post, Nathan Buggia of the Live Search Webmaster Team notes that over the last eight months, the team has been making good progress on spam fighting tools. One aspect of spam that they have been working on is cloaking. Cloaking is showing a different page to search engines than to users and the use of it has been the subject of many a debate over the years. Google has generally sent the message that all cloaking violates its guidelines (although in practice, some instances seem to be accepted). Microsoft’s post notes that “not all cloaking is spam related and we do our best to take this into account.” When I talked to Nathan about this, he said that they consider cloaking to be situations when site owners are trying to deceive search engines. However, they don’t recommend cloaking in any situation. In part, this recommendation stems from the fact that it’s difficult to gauge intent with automated cloaking detection tools and some innocent sites could suffer collateral damage.

If you show the same content to users and to search engines, but use a method that automated checks may flag as cloaking, or if you have cloaked your site in the past and you feel your site is being penalized in the Live Search index, you can request manual review of your site using the feedback at the Live Search Webmaster Center.

Live Search has been using a cloaking detection process that has used a bot not identified as MSNbot to crawl pages and compare them to the versions served to MSNbot. Microsoft says that this non-identified bot has adhered to robots.txt rules (this bot hasn’t retrieved the robots.txt file separately, but has followed the one retrieved by MSNbot), but has caused a few other problems for some site owners.

• Skewed ads reporting: The bot initially had a bug that caused it to download ad blocks, which made it appear as a user for ads reporting purposes. This may have overinflated impression counts and lowered click-through rate reporting.
• Skewed site statistics: In some cases, the bot caused high traffic loads on sites, which not only consumed extra bandwidth, but also distorted visitor counts in analytics reports. Because this bot didn’t identify itself as a bot, some traffic reporting software reported the visits as coming from users.
• Added unrelated search terms to HTTP logs: The bot often crawled a page as a user clicking through a search result. The search queries the bot used were often not related to the site, which caused incorrect search query reports and perplexed many a webmaster.

In addition, when webmasters began reporting these issues to Microsoft, they were unresponsive and, in fact, MSNDude posted on the Webmaster World forums that site owners who blocked this bot risked being removed from the Live Search index:

“We would request that you do not actively block the IP addresses used by this quality check; blocking these IP addresses could prevent your site from being included in the Live Search index.”

The blog post today says that these issues have been resolved. The bot will no longer download ads blocks, will no longer crawl sites enough to place noticeable traffic loads, and won’t use non-related search strings. Also, the Live Search team says it has created a forum specifically to discuss crawler issues and that webmasters can also use the feedback form and should receive a timely response.

I asked if sites that blocked this bot would indeed be removed from the Live Search index and was told that:

“The tool we’re using to detect cloaking is just one of many inputs we use to determine if a site is spam. Blocking this bot does not necessarily mean that your site will be considered spam, but it is a possibility. We recommend any webmasters concerned should log into our Webmaster Tools (http://webmaster.live.com) and check to see if their site has been blocked. If so, they should submit a reinclusion request using the form.”

In a separate release, Microsoft notes that their Webmaster Center is now live and “provides all the necessary resources to optimize a Web site for achieving the highest possible algorithmic or “organic” listing on Live Search.”

When I talked to Nathan, he said that sites that are cloaking may continue to see some amount of traffic from this bot. This tool crawls sites throughout the web — both those that cloak and those that don’t — but those not found to be cloaking won’t continue to see traffic.

Live Search isn’t the only search engine to have processes that check for cloaking. Matt Cutts has said that Google checks both algorithmically and manually. (Although they have implemented a “first-click free” program, which is a type of approved cloaking.)

Yahoo’s guidelines warn against “pages that give the search engine different content than what the end user sees (cloaking).” Search engines want the search results to accurately represent what users will see when they click through to the linked pages, and deceptive cloaking interferes with this goal.

More signals from a major search engine that they are both actively seeking out cloaking and acknowledging that it’s not always spam aren’t likely to settle the cloaking debate. And one has to wonder how effective methods like this really are. Those savvy enough to cloak may be able to cloak for this new cloaker detection bot as well. I asked Microsoft if this tool has helped them achieve their spam reduction goals. They said that:

“Like all search engines, Live Search has been vigilantly identifying and removing spam content from our index since the inception of our search engine. With the launch of the update this Fall, we have measured a significant decrease in the amount of spam.”

Matt Cutts of Google has written two posts on the topic of SEO:

• Detecting more “undetectable” webspam
• The anatomy of a search result

The first, Detecting more “undetectable” webspam, should trigger some heated comments at his blog. It shows how he was able to find what was claimed to be “undetectable” cloaking and web spam. Matt then issues his warning to webmasters:

More generally, if someone is trying to manipulate Google by deceptive cloaking, it means that a webserver is returning different content to Googlebot than to users. That’s a condition that can be checked for by algorithms or manually, and such cloaking is certainly not “undetectable.” For cloaking to be completely “undetectable,” it would have to be like that Steven Wright joke: “Last night somebody broke into my apartment and replaced everything with exact duplicates.” And a cloaking script that gave users and Googlebot exactly duplicate pages would be a bit pointless.

There are currently over thirty comments at Matt’s blog, I assume you will be able to find over a hundred by the end of today.

In Matt’s other post, The anatomy of a search result, he breaks down how the search result snippet works. He describes in the video below each of the following components: the title, the description, the plus expansion box, the URL, how Google bolds keywords, the cache link, similar pages link, and, Google notes, Site Links and the more results link. Here is the video:

The Google News Blog wrote about the First click free program that allows publishers to let readers pass registration and subscription barriers, if they come from Google News.

I am sure you have seen this in action. You see an article, you click on it from Google News and you are able to read the full article. Then you want to see the article again an hour or so later, but the publisher asks you to login to see the full article. That is the “first click free” program in action.

Sometimes publishers do not want even the first click to be free, in those cases, Google will label that news result with a “subscription” tag, so the user knows you need to be a subscriber to read the content. Here is an example of what that may look like in Google news:

The “First click free” program isn’t new, just not that well known. Danny covered it in his YADAC: Yet Another Debate About Cloaking Happens Again article. I also wrote about it at the Search Engine Roundtable in How To Allow Google News To Index Your Subscription Only Source?

In addition, we had two Sphinn discussions on this topic. The first was started by Danny and named Got Registration? Cloaking Google News Is Now Officially OK and the second was started by Sebastian and named Cloaking is alive and doing well in Google.

Of course, if Google News allows this form of cloaking, why can’t others use it in Google web search? Heck, we have Google Universal Search now, so Google News can be directly embedded in the Web search results and those results can participate in the First click free program.

Somewhat related, Academic Publishers as Spammers from BlogSci.com covers how through other program such as Google Scholar, cloaked pages are allowed into Google.

Have you ever clicked on a promising search result, only to find that the web page you land on has nothing to do with what you expected, or even worse, is a page of porn or other spam?

If so, you may have been the unwitting victim of black-hat cloaking. Today’s Search Illustrated shows how this bait-and-switch tactic is accomplished:

Graphic by Elliance, an eMarketing firm specializing in results-driven search engine marketing, web site design, and outbound eMarketing campaigns. The firm is the creator of the ennect online marketing toolkit. The Search Illustrated column appears Tuesdays at Search Engine Land.

Google Website Optimizer, a tool that allows you to easily test different page layouts, is now available to anyone with an AdWords account. Previously, it had been out in limited beta since last October. It remains unclear whether using the tool would be considered cloaking, which is against Google’s webmaster guidelines. The Good Cloaking, Evil Cloaking & Detection article from Stephan Spencer here on Search Engine Land looked at this issue last month. Threadwatch has a recent discussion also going. I’ll explain more about the situation below, and I’m checking with Google for the official word.

Fair to say, Google’s probably going to say that using the tool is NOT cloaking. Otherwise, you’ve got one part of Google putting out a tool that will cause anyone using it to potentially be banned by another part of Google. This also means questions that have hung over other page testing tools such as Offermatica should go away.

It won’t make the cloaking debate necessarily easier, however. Consider the If Your Page Ranks Well, You’d Better Be Messing With It article from Rand Fishkin at SEOmoz last month. In it, Rand picks up on a challenge that Jamie Roche (CEO of Offermatica) makes to the traditional wisdom that if a page is ranking well, you don’t want to alter it much.

Rand argues the major factors controlling rank are mostly off-the-page (importance of your domain, links to your pages) and thus not impacted by change to the page itself. I’m largely with him. If a page is doing well, altering it really shouldn’t have that much of an impact, unless you’re doing something really drastic such as replacing all the text for images. And yet — I’d still be cautious, despite knowing that’s not likely something to fear.

This leads back to Roche’s article. Roche is clearly reacting to clients that are concerned that his product, Offermatica, might suggest changes that could cause them to lose rankings. So, there’s an element of self-interest in his writing. But I still find many of his points valid:

Unfortunately, even if you don’t mess with your page, your rank could change. That’s because the rules that the search engines use to rank sites change as they discover newer, and presumably better, ways to rank results. A page that ranked well one day might drop to the third or fourth page the next, or get removed altogether.

The result is that we live in a state of fear about changing well-ranked pages, while knowing that even if we don’t change them we could lose rank anyway.

So when we realize that we want to change our natural search landing pages because they don’t provide the best user experience, we wonder whether it might make sense to experiment with changes despite our fears.

He then also gets into three tactics, warning that potentially, they could get you delisted or cause a rank drop:

• Test various pages, and keep showing spiders the original one during testing. Anyone not seeing what the spider sees is technically getting a cloaked page. Yes, as he notes, that’s what Google Website Optimizer does – so Google itself seems to be giving tacit approval to cloaking.
   
• Change just some of the page. He seems to think this is a bigger risk. I think it’s minor. The main risk is that if you change some of the page using JavaScript, potentially that could be seen as cloaking. But that’s part of the page testing opt-out already covered above.
   
• Target pages to second-time visitors. Search engines don’t read cookies. So if you cookie a first time visitor, you can then show them a different page when they return, even if they go to the same original URL. Search engines will continue to see the same page every time, since they won’t have a cookie.

Now here’s an even more complicated issue. Many who do good old-fashioned cloaking have long argued that it doesn’t matter how they get ranking, as long as they are relevant. So what if they use gibberish text or put out some highly-optimized textual content? As long as they show the visitor a relevant page, what’s the harm? Indeed, if you buy into the argument that what you do on the page largely doesn’t matter — that it’s really down to domain authority and links — then it becomes even harder to understand why cloaking is such an issue.

Moreover, what prevents someone from eternally testing a page? If you gain a ranking with some butt-ugly page, you might then keep feeding that to spiders but constantly test it with a page testing tool. Potentially, that keeps you out of the cloaking hot seat yet it does the same exact thing that cloaking does.

Overall, Google in particular has dodged its outdated guidelines on cloaking time-and-time again over the past few years. My YADAC: Yet Another Debate About Cloaking Happens Again article from last month provides the background here. They can’t keep dodging it in the case of page testing tools now, not when they themselves are offering one. So I’m pinging them for the official word.

As for the tool, here’s more from the formal announcement:

As part of our continued commitment to help advertisers make smart business decisions, we are happy to announce that the Google Website Optimizer™ application is now available to AdWords advertisers worldwide. Google Website Optimizer is designed to help website owners increase conversions such as sales, sign-ups or downloads. This multivariate landing page optimization tool enables marketers to test different ideas for web page content such as different headlines, promotional copy, or images. The application provides easy-to-read reports that enable advertisers to see which variation resonated best with their site visitor. It is a self-service application that enables website owners to set-up and run multivariate landing page experiments.

Google Website Optimizer is a beta application that is integrated with the Google AdWords™ program and free to AdWords advertisers. Advertisers can sign up immediately at www.google.com/websiteoptimizer. Over the coming weeks, the Google Website Optimizer application will become available automatically in all advertisers’ accounts. Website owners can now determine what content was most effective as indicated by the highest conversion rate.

By giving website owners the tools they need to improve their website content, Google is helping improve the user experience on the internet as a whole. Since the beta launch of the application in October 2006, many advertisers who used Google Website Optimizer have achieved major results.

"Using Website Optimizer enables us to approach our website like a living lab, where we can test and play and constantly figure out how to improve the site," says Deborah Krier, Marketing Manager, Dale and Thomas Popcorn, "Website Optimizer is a powerful tool that allows us to understand our users better, leading to increased conversions and increased business success."

In addition, we are announcing the formation of a new partner program, Google Website Optimizer Authorized Consultants. As of today, Optimost, EpikOne, Future Now, ROI Revolution, and SiteTuners.com have signed on as charter members.

“We’re delighted that Google is now offering Website Optimizer. In the past, not everyone had the tools to test regularly,” said Bryan Eisenberg, co-founder, Future Now, Inc. and author of the New York Times bestseller Call to Action, “Google Analytics had a major effect on the accessibility of data and on how website owners valued analytics; Website Optimizer will take the benefits of testing to a much broader audience and help them increase online conversion rates.”

For more information or to sign up to use Website Optimizer, please visit: www.google.com/websiteoptimizer

Sigh. Double sigh. Triple sigh. I guess now that the SEO industry has had the required twice-yearly debate about the reputation of SEO, it’s time to do the go round about cloaking once again. A quick word about cloaking has Google’s Matt Cutts trying to clarify concerns that Philipp Lenssen of Google Blogoscoped has been raising about WebmasterWorld. The comments are now up over 100, as people rehash things that have been hashed, mashed, rebaked so many times before. Below, some cloaking history plus an honest plea about trying to get past this stupid, stupid issue.

Definition Time

Let’s do the definition, first:

Cloaking is when you show a search engine content that is different than what a human being sees.

Got it? That’s my definition, and Matt says virtually the same thing in his post today:

Cloaking is serving different content to users than to search engines.

So simple. What’s to debate? Well, is it cloaking if…

• A spider coming from a US IP address sees a different page than a user from a UK IP address?
• A spider sees content that a user sees, but only if they do free registration
• A spider sees content that a user sees, but only if they do paid registration
• A spider sees content in text that represents what a users sees in Flash
• A spider sees content that’s slightly different than what a user sees when their browser renders Javascript

Pick one of those above — pick something else (see our Good Cloaking, Evil Cloaking & Detection column from last week) — and people can, will and have pointed at something someone is doing, then yelled "cloaking" and screamed for a ban to happen. A ban? Well, as you know, all search engines hate cloaking. Actually, that’s always been a confused point. Here starts the lesson.

History Time: Tactics Versus Intent

Back in January 2003, Alan Perkins wrote this big Cloaking Is Always A Bad Idea article, telling us that search engines always said cloaking was bad. I was never a proponent of cloaking. I was, however, well aware that NOT all the guidelines were against cloaking. In addition, with paid inclusion, I argued some cloaking was actually allowed. All this went into my Ending The Debate Over Cloaking that came out in reaction to Alan’s article, in February 2003.

Since I knew that all the search engines had allowed some types of cloaking, my advice to marketers was this, with the stress on avoiding "unapproved cloaking:"

Cloaking is getting a search engine to record content for a URL that is different than what a searcher will ultimately see, often intentionally. It can be done in many technical ways. Several search engines have explicit bans against unapproved cloaking, of which Google is the most notable one. Some people cloak without approval and never have problems. Some even may cloak accidentally. However, if you cloak intentionally without approval — and if you deliver content to a search engine that is substantially different from what a search engine records — then you stand a much larger chance of being penalized by search engines with penalties against unapproved cloaking. If in doubt, ask the search engine if it has a problem with what you intend to do, assuming you can’t get a clear answer from written guidelines that are provided. If you are working with a third-party search engine marketer, ask them for proof that what they intend to do is approved. Otherwise, be prepared for any adverse consequences.

The suggestion to avoid "unapproved cloaking" infuriated Doug Heil over at the iHelpYou forums, who could not (and to this day still cannot) get over the idea that cloaking MUST equal spamming.

My response back then remains the same today. There’s a difference between tactics and intent. Many of the things that might cause penalties with search engines are tactics (hidden text, gibberish pages, cloaking) that are closely aligned with the intent of trying to mislead or game the search algorithms. But in some cases, what’s a bad tactic (or technical implication) might have a good intent as agreed by the search engines. So they’ll allow it, either turning a blind eye to it or giving it some official endorsement.

That difference is important because back then, if the search engines got behind the "unauthorized" versus "authorized" suggestion, we wouldn’t be having today’s wasteful argument. But let’s carry on.

NPR, Google Scholar & Approved Cloaking

In May 2004, I looked at how National Public Radio was, in my view, cloaking text transcripts of audio to search engines but only letting human visitors by those. At the time, Google had a guideline against cloaking that read:

The term "cloaking" is used to describe a website that returns altered webpages to search engines crawling the site. In other words, the webserver is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they’ll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking to distort their search rankings.

I argued that this was an example of "good cloaking" and that the real issue I had with it was that other marketers were supposedly banned from doing it:

As a searcher, I’m actually glad the method is being used. It does mean I’m more likely to find audio content of interest. Moreover, I can listen to that for free via the NPR site.

As a search engine marketer, I’m not so thrilled. I’m well aware that many other companies would like the ability to feed Google content in this manner. In addition, they have just as compelling arguments as NPR about having good content that isn’t adequately indexed by the Google crawler. Unfortunately, they’re denied the privilege of feeding relevant material just to Google’s crawler.

What about Yahoo? Anyone can enjoy the same benefits that NPR has, the ability to cloak content when relevant, through Yahoo’s content acquisition program. Non-profit organizations are offered this for free. Commercial organizations have to pay, making use of Yahoo’s trusted feed program.

By November 2004, I was writing about cloaking again. Now Google had an officially approved program that, in my view, allowed cloaking. This was Google Scholar. As I wrote:

This system may lead to problems for some searchers. In the example above, not only could I NOT read the paper, as I didn’t have a subscription, but I also could not read even an abstract. Instead, a password-prompt continued to appear, even when I cancelled it, making it extremely difficult to finally close the window (and that’s why I haven’t linked to the actual paper, to save other people the problem).

This situation is probably unusual, however. One of Google’s requirements for inclusion in Google Scholar is that publishers at least show abstracts to searchers.

The special access for publishers flies in the face of Google’s anti-cloaking policy. Google is being shown material that regular users wouldn’t normally see, its own definition of cloaking. This is a GOOD thing for searchers, but the company needs to amend its cloaking policy so as not to be hypocritical.

Indeed, that’s long overdue. This has been a problem since I first reported about a similar issue earlier this year. A sidebar piece … looks at the latest case and suggests some fixes for Google, including finally moving forward with formalizing such programs for ALL publishers.

Plea For Google To Change The Cloaking Guidelines

In the sidebar to that article, and again in a follow-up piece a few days later, I urged Google to alter its cloaking policy to something stressing that cloaking was bad only if not approved:

The term "cloaking" is used to describe a website that returns altered webpages to search engines crawling the site without permission. In other words, the webserver is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they’ll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking without our permission, if we feel it is harmful to our search rankings.

BMW, WebmasterWorld & New York Times Cloaking Accusations

In March 2005, there was great amusement in some quarters when Google’s policy against cloaking caused it to ban itself when pages apparently with text designed to help internal Google searching made also onto external versions seen only by Google spiders, rather than humans (see here, here and here for more).

By the end of 2005, WebmasterWorld came under accusations of cloaking. Since it is one of the most important forums about search engines around — frequented by official Google reps — it sort of became a poster child of "why can they do it but others can’t" for some.

Far bigger news came in 2006. In February, BMW got banned on Google for using hidden text — in particular, a "poor-man’s" version of cloaking that used JavaScript to show different content to users. It got back in a few days later. Then in June, an article about how the New York Times optimizes content for search engines sparked a new cloaking debate when it seemed that the major search engines were allowing cloaked content. In particular, the New York Times was allowing search spiders to read content that was only accessible to humans if you registered for free or, in some cases, paid for access.

Marshall Simmonds, the NYTimes & Acceptable Cloaking was a giant discussion that came out of that article, which had me originally arguing that this was cloaking, since search spiders were seeing something different than most humans could see. But I was convinced to change my mind. Since the spiders were indeed shown what anyone could ultimately see, this wasn’t cloaking. I commented:

Until now, I would have considered feeding a search engine a page that people couldn’t see unless they registered to have been unapproved cloaking, since most users were seeing something different that then spider saw.

But sure, I’ll buy into the "eventually you’ll see the same thing" argument as this not being cloaking. Why not? Google’s allowed this in approved cases for about two years now and never wants to go on the record as this being approved cloaking. So don’t call it cloaking and everyone’s happy. Google’s not allowing something officially they say not to do, and content owners can do this without fear.

In fact, I expect Fantomaster, Beyond Engineering and anyone with IP delivery lists now can have some new-found respect from people who previously slammed them as helping cloakers. Here they’ve been saying its all about content delivery systems and now they’re right, at least in some situations. Because after all, some people aren’t going to want to depend on user agent delivery to feed content this way.

Of course, I’m still not going to do this yet with my own members-only content. Despite agreeing with you, Phil — despite seeing others allowed to do this – I’m still fearful Google might arbitrarily decide to call it cloaking anyway when they choose. But maybe I’ll be braver down the line, and why not? It’s like a whole new world.

I did have several off-the-record conversations with Google about this. The main thing that came out that I can report was that Google really felt most users should see what their spiders saw WITHOUT having to register or pay for access. That’s similar to what Matt wrote about WebmasterWorld content today, when covering the latest criticisms that it might be cloaked:

I consider the issue in a much better state now, in that most (all?) Google searchers get the identical page to what Googlebot saw. But I still consider Philipp’s February posts open for investigation, and I will get to them, in the same way that I tackled Philipp’s first two posts about this.

FYI, to understand a bit more about the WebmasterWorld situation, see Matt’s post from March 2006, How to sign up for WebmasterWorld. It explains how in some cases, trying to access a thread that you might come across from a search result can trigger a registration results. In today’s post at Matt’s, WebmasterWorld’s Brett Tabke explains and points at more information on how this has changed.

The comments in Matt’s post also get into an apparent return of some New York Times content requiring registration, as well as more complaints about Google Scholar content (added with the full cooperation of Google) also being annoying, in that you can’t read it without paying when clicking from search results.

Solution 1: Allow Registered Content

Enough history. Time for some solutions. Both Google and Yahoo have programs that allow people with free registration or fee-based content to show up in search results — and I mean mixed in with regular search results, not segregated out like the Yahoo Subscriptions product launched in June 2005. Yahoo’s will happen mainly through paid inclusion, and not that much. Google’s will happen primarily through the aforementioned Google Scholar plus the First Click Free system used for Google News but which also may happen with some web search content.

None of this content is labeled in any way. Back to that discussion on cloaking and the New York Times, I commented:

It is annoying to hit one of these links and not know that payment or registration is required, however. That problem’s going to get worse as more and more people decide this isn’t cloaking and give it a go. Google and the others should look to establish a way for site owners to better flag premium or registration only content….

I don’t agree having paid content in regular search results is bad. I have a Wall Street Journal paid subscription. They have lots of great content. If I’m doing a search, and they’ve got a good match, I want to know that.

And over at Yahoo, as I explained way way back above, I can do that. I can choose specifically to have this content revealed to me. It doesn’t make my results bad at all.

It is a bad user experience if you constantly get back results that you can’t actually view, of course, without paying. We simply aren’t going to subscribe to everything.

The solution is easy. Give users the option. Let me choose to see content that requires payment or not. Or similarly, let me choose to see content that might require free registration. We just need Google and the others to graduate from 1999 mentality and better accommodate web sites with this type of content. It’s easily done, if they want to do it. And getting more formalized program for publishers, as well as options for searchers, will help.

So enough already. Enough with the special programs that only some publishers get to do. I want Google — which leads the charge in scaring people about cloaking — to fast track a system to let anyone with registration-based or fee-based content to be in their search results.

As for usability, either flag the URLs so users know to expect a charge or registration request or make it possible for users to exclude this information. Or experiment with both. But do something so that publishers don’t take matters into their own hands and the SEO industry has to have yet another one of these debates over whether it’s cloaking.

Solution 2: Allow For Approved Cloaking

Remember that suggested revision for Google’s cloaking policy I gave above? Well sometime in 2006, Google dropped its definition of cloaking entirely. Instead, we were just left with shorter definition of cloaking here:

Make pages for users, not for search engines. Don’t deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking."

And a warning against it here:

However, certain actions such as cloaking, writing text in such a way that it can be seen by search engines but not by users, or setting up pages/links with the sole purpose of fooling search engines may result in removal from our index.

That warning, like the old warning, uses the word "may" in terms of removal for cloaking. Will cloaking get you banned? Maybe. Maybe if it is noticed. Maybe if it is deemed harmful to the users. Maybe if after a closer look, it can’t be pigeonholed in some other definition.

For me, it would be clearer to go back to the old definition and stress that unless approved, cloaking might result in a ban. Roll that out along with a program making it easier for people to feed in registered content. That gives Google flexibility, helps publishers plus stops this insane focus on technical/tactical implementations and refocuses concern where it belongs — the user experience.

Did what you do help or harm the search results, in Google’s opinion? If you were harming search results, Google’s always reserved the right to boot you out. And if you were technically violating guidelines but not in a harmful way, Google’s always reserved the right to turn a blind eye. Or rather, an approving eye, an eye knowing that it’s intent that matters, not some technicality.