Here is the video, I will post the main points below:

• Cloaking is showing different content to users than GoogleBot
• Cloaking Violation of Quality Guidelines
• Cloaking is High Risk
• It Is Often Used For Deceptive Reasons
• No Such Thing Has White Hat Cloaking
• Details on what might be perceived as cloaking is mentioned in the video
• Why geolocation and mobile detection is not cloaking

Related Stories:

• Google Vows Renewed Look At Cloaking In 2011
• Be Careful When Your Affiliates Practice Link Cloaking
• Search Illustrated: Black Hat Cloaking Explained
• Google’s Matt Cutts On Cloaking & Search Snippets
• Good Cloaking, Evil Cloaking & Detection
• YADAC: Yet Another Debate About Cloaking Happens Again
• The Long Road To The Debate Over “White Hat Cloaking”

Google’s Own Words About the Farmer/Panda Update

Google’s Matt Cutts said that while the change isn’t 100% perfect, searcher feedback has been overwhelmingly positive. He noted that the change is completely algorithmic with no manual exceptions.

Blocking “Low Quality” Content

Matt reiterated that enough low quality content on a site could reduce rankings for that site as a whole. Improving the quality of the pages or removing the pages altogether are typically good ways to fix that problem, but a few scenarios need a different solution.

For instance, a business review site might want to include a listing for each business so that visitors can leave reviews, but those pages typically have only business description information that’s duplicated across the web until visitors have reviewed it. A question/answer site will have questions without answers… until visitors answer them.

In cases like this, Google’s Maile Ohye recommended using a <meta name=robots content=noindex> on the pages until they have unique and high-quality content on them. She recommends this over blocking via robots.txt so that search engines can know the pages exist and start building history for them so that once the pages are no longer blocked, they can more quickly be ranked appropriately. I noted in the panel where we discussed this that an exception might be for a very large site, robots.txt would ensure that the search engine bots were spending the crawl time available on the pages with high-quality content.

Ad-To-Content Ratio

Matt said that having advertising on your site does not inherently reduce its quality. However, it is possible to overdo it. I had noted in my earlier articles about this change that in particular, no content and only ads above the fold, as well as pages that have so many ads, it’s difficult to find the non-advertising content often provide a poor user experience.

Slowed Crawl

Matt also noted that if Google determines a site isn’t as useful to users, they may not crawl it as frequently. My suggestion based on this is to take a look at your server logs to determine what pages Googlebot is crawling and often those pages are crawled. This can give you a sense of how long it might take before quality changes to your site take affect. If Google only crawls a page every 30 days, then you can’t expect quality improvements to change your rankings in 10 days, for instance.

International Roll Out

Matt confirmed that the algorithm change is still U S. only at this point, but is being tested internationally and would be rolling out to additional countries “within weeks”. He said that the type of low quality content targeted by the changes are more prevalent in the United States than in other countries, so the impact won’t be as strong outside the U.S.

Continued Algorithm Changes

Matt said that many more changes are queued up for the year. He said the focus this year is on making low quality content and content farms less visible in search results as well as helping original creators of content be more visible. Google is working to help original content rank better and may, for instance, experiment with swapping the position of the original source and the syndicated source when the syndicated version would ordinarily rank highest based on value signals to the page. And they are continuing to work on identifying scraped content.

Does this mean that SEO will have to continue to change? Not if your philosophy is to build value for users rather than build to search engine signals. As Matt said, “What I said five years ago is still true: don’t chase algorithm, try to make sites users love.”

Gil Reich noted my frustration at the way some people seemed to interpret this advice.

“The Ask the SEOs sessions used to be battles between Black and White. Now, with the same participants, it’s between “focus on users” and “focus on creating the footprint that sites that focus on users have.” It seemed that Vanessa found this debate far more frustrating than when her fellow panelists were simply black hats.”

It’s true. For instance, at one point, Bruce Clay said that they’d done analysis of the sites that dropped and those that didn’t and they found that those that dropped tended to have almost an identical word count across all articles. So, he said it was important to mix up article length. I told the audience that I hoped that what they got out of the session was not that they had to go back to their sites and make sure all the lengths were different but that they actually made sure each page of content was useful.

You know the best way to ensure your site has a “footprint that sites that focus on users have”? Focus on users!

“White Hat” Cloaking?

We hear a lot about “white hat”cloaking. This tends to mean anytime a site is configured to specifically show search engines different content than visitors for non-spam reasons. For instance, a search engine might see HTML content while visitors see Flash. The site might not serve ads to search engines or present only the canonical versions of URLs.

Some SEOs say that Google condones this type of cloaking. However, Matt was definitive that this is not the case. He said:

“White hat cloaking is a contradiction in terms at Google. We’ve never had to make an exception for “white hat” cloaking. If someone tells you that — that’s dangerous.”

Matt said anytime a site includes code that special cases for Googlebot by user agent or IP address, Google considers that cloaking and may take action against the site.

First-Click Free Program and Geotargeting

Danny Sullivan asserted that the “first-click free” program is white hat cloaking, but Matt disagreed, saying that the site was showing exactly the same thing to Googlebot and visitors coming to the site from a Google search. Matt also noted that showing content based on geolocation is not cloaking as it’s not special casing for the Googlebot IPs (but rather by the geographic location).

Cloaking For Search-Friendly URLs

Note that several Search Engine Land articles (including this and this) assert that cloaking to show search-engine friendly URLs is OK with the search engines, but Google in particular has been definitive that this implementation is not something they advocate.

The basis for some thinking this scenario is OK by Google seems to be a statement from a Google representative at SES Chicago in 2005. But back in 2008, Matt Cutts clarified to me that “cloaking by rewriting URLs based on user-agent is not okay, at least for Google.” (My understanding is that was the first speaking engagement for the Google representative and he wasn’t fully up on the intricacies of cloaking.)

In any case, there are now lots of workarounds for URL issues. If developers are unable to fix URLs directly or implement redirects, they can simply use the rel=canonical link attribute or the Google Webmaster Tools parameter handling feature (Bing Webmaster Tools has a similar feature).

Hiding Text for Accessibility

What about using a -999px CSS text indent for image replacement? Maile had previously done a post on her personal blog noting that hiding text in this way can be seen as violating the Google webmaster guidelines even if it’s done for accessibility, not spammy reasons. Generally, you can use a different implementation (such as ALT attributes or CSS sprites). On stage at SMX, Maile also recommended using font-face. This can be tricky to implement, but at least for the font files themselves, you can use Google Web Fonts rather then building them yourself.

Matt seconded this in a later session: “hidden text? Not a good idea. I think Maile covered that in an earlier session.”

Showing Different Content to New Visitors Vs. Returning Visitors

Someone asked about showing different content to new vs. returning users. Both Matt and Duane Forrester of Bing commented that it was best to be careful with this type of technique. Generally, this type of scenario is implemented via a cookie. Both new visitors and Google won’t have a cookie for the site, while a returning visitor will have one. Matt noted that if you treat Googlebot the same as a new user, this generally is fine.

Building Links

Links continue to be important, but how can sites acquire them using search engine-approved methods? Matt said to ask yourself “how can I make a good site that people will love?” Make and impression and build a great site with a loyal audience (not necessarily through search) that brings brand value and links will come.

Creating Value Beyond Content

Someone asked how to get links to internal pages of an ecommerce site. Product pages just aren’t that interesting and don’t have a ton of editorial content. Matt recommended looking at different ways of slicing the data and becoming an authority in the space.

How Valuable is Article Marketing?

Not very.  Both Duane and Matt said that articles syndicated hundreds of times across the web just don’t provide valuable links and in any case, they aren’t editorially given. Duane made things simple: “don’t do the article marketing stuff.”

He suggested contacting an authority site in your space to see if they would publish a guest article that you write particularly for them. If the authority site finds your content valuable enough to publish, that’s a completely different situation from article hubs that allow anyone to publish anything.

What About Links in Press Releases?

Someone noted that while paid links violated the search engine guidelines, you can pay a press release service to distribute your release to places such as Google News, so don’t those links count? Matt clarified that the links in the press releases themselves don’t count for PageRank value, but if a journalist reads the release and then writes about the site, any links in that news article will then count.

Are Retweets More Valuable Than Links?

Someone asked about the recent SEOmoz post that concluded that retweets alone could boost rankings. Matt said he had asked Amit Singhal, who heads Google’s core ranking team, if this was possible. He said that Amit confirmed links in tweets is not currently part of Google’s rankings so the conclusions drawn by the post were not correct. Rather, other indirect factors were likely at play, such as some who saw the tweet later linked to it. (Purely speculating on my part, those tweets could have been embedded in other sites that in turn were seen as links.)

Matt mentioned that signals such as retweets might help in real-time search results and then talked about a recent change that causes searchers to see pages that have been tweeted.

Some mistakenly took this to mean that the Google algorithm would give a rankings boost to pages that have been tweeted vs. those that haven’t, but Matt was talking about the change a few weeks ago that personalizes search results based on a searcher’s social network  connections. As Matt McGee explained in his Search Engine Land article about it:

In some cases, Google will simply be annotating results with a social search indicator, says Google’s Mike Cassidy, Product Management Director for Search. Google’s traditional ranking algorithms will determine where a listing should appear, but the listing may be enhanced to reflect any social element to it.

In other cases, the social search element will change a page’s ranking — making it appear higher than “normal.” This, I should add, is a personalized feature based on an individual’s relationships. The ranking impact will be different based on how strong your connections are, and different people will see different results.

Can Competitors Buy Links To Your Site and Hurt Your Site’s Rankings?

This is an age old question, but as several high profile sites have had rankings demotions lately for external link profiles that violate the search engine guidelines, it was top of some people’s minds. Matt reiterated that competitors generally can’t do anything to hurt another site. The algorithms are built to simply not value links that violate search engine guidelines. Demotions generally only occur when a larger pattern of violations is found.

What About Exact Match Domains and Incoming Anchor Text?

Several people have commented on spammy sites with exact math domains and lots of spammy incoming links with exact match anchor text ranking quite well in Google and Bing. Matt said they are looking into this.

How Google Handles Spam and Reconsideration Requests

Matt reiterated some basics about how this works:

• Google’s approach to spam: Engineers write algorithms to address spam issues at scale. In parallel, manual teams are both proactive and reactive in looking for spam and both removing it from the index and providing it to  the engineering team to help them modify their algorithms to not only find that specific spam instance, but all similar instances across the web.
• How Google handles spam reports: Spam reports have four times the weight of other spam found in terms of manual action as it’s clearly spam that a searcher has seen in results.
• Notifying site owners of guidelines violations: Google is looking at providing additional transparency around guidelines violations found on sites. (They already provide some details in the Google Webmaster Tools message center.)
• How Google handles reconsideration requests: Only reconsideration requests from sites that have a manual penalty are routed to Googlers for evaluation. Generally, the reconsideration process takes less than a week. Algorithmic penalties can’t be manually removed. Rather, as Google recrawls the pages the algorithm adjusts the rankings accordingly.

I’ve pulled out the key points from this post in bullet format:

• Google differentiates between traditional mobile phones and smartphones.
• Google has two bots: Googlebot and Googlebot-Mobile. Googlebot crawls desktop-browser type of webpages and content embedded in them and Googlebot-Mobile crawls mobile content.
• Currently only traditional phones are supported with special useragent strings within Googlebot-Mobile, not smartphones (this may change)
• Google said they “expect smartphones to handle desktop experience content so there is no real need for mobile-specific effort from webmasters.
• It does not mean you can’t serve a special style sheet to smartphones, Google said, “the decision to do so should be based on how you can best serve your users.”
• URL structure: For Googlebot and Googlebot-Mobile, it does not matter what the URL structure is as long as it returns exactly what a user sees too.
• Using the same URL “is not considered cloaking by Google.”
• Mobile sitemaps: you should include only mobile content URLs in Mobile Sitemaps, even if these URLs also return non-mobile content when accessed by a non-mobile User-agent.

This blog post was in response to a question I had in regards to a video Matt published, which is below. But make sure to understand that you do not need to use a mobile specific URL, you can serve up mobile content on the same URLs you serve up to desktop users. In addition, it is important to note that smartphones like iPhones and Android phones are not necessarily and currently considered “mobile” devices to Google or Googlebot-Mobile.

Related Stories:

• The New Mobile SEO: What You Need To Know
• Two New Mobile SEO Tips For 2011
• Has The iPhone Made Mobile SEO Obsolete?
• Mobile SEO Tip: Transcoding Services Can Dilute Link Popularity
• Don’t Penalize Yourself: Mobile Sites Are Not Duplicate Content
• Why Mobile Searchers Need Mobile-Optimized Sites
• Sorting Out The Mobile Search & SEO Mess
• Google Writes On Mobile Site SEO Concerns & Techniques

Cloaking, which is against Google’s webmaster guidelines, is defined by Google as follows:

Cloaking refers to the practice of presenting different content or URLs to users and search engines. Serving up different results based on user agent may cause your site to be perceived as deceptive and removed from the Google index.

Matt’s tweet read:

Google will more at cloaking in Q1 2011. Not just page content matters; avoid different headers/redirects to Googlebot instead of users.

Honestly, I thought Google already did a good job with detecting cloaking of different headers and redirects, and of course cloaking content. But I guess I was wrong. It is hard to tell exactly what Matt meant in the 136 characters he posted on Twitter. It is leading to a lot of concern in a WebmasterWorld and Cre8asite Forums threads.

So if you are doing any cloaking and getting away with it, I guess you may have to be on the lookout.

Postscript From Danny Sullivan: Over the years, Google has (in my view) redefined some things that would have been considered cloaking in the past to be acceptable, such as its First Click Free program. This has lead, again in my view, to some confusion. For some background on this, please see:

• YADAC: Yet Another Debate About Cloaking Happens Again
• The Long Road To The Debate Over “White Hat Cloaking”

Personally, I’ve long wanted Google to take a less technical approach to examining cloaking (and whether it should be banned) and rather more one of intention. That would help with cases where people end up innocently cloaking and getting penalized, as Google had to penalize itself last year:

• Why Google Should Ban Its Own Help Pages — But Also Shouldn’t

Meanwhile, one thing that’s been long on my follow-up list is whether the system that Perfect Market uses to list pages specifically for Google and send people from Google to those pages is cloaking. I wrote about this in more depth last year in my Search Engines + Newspapers: Perfect Market’s Delivery System Aims To Please Both story.

My view was that it was cloaking, since something special was being done just for Google (regardless of whether users saw the same thing coming from Google). Cutts was away on vacation when this came up, and we’ve just not had a chance to connect further on this since then. I’m on vacation myself now, but I’ll follow up on this after the New Year, as well as more about what else Google may be planning in terms of cracking down on cloaking.

Google’s Help Area & Cloaking

In particular, Google has a long-standing rule against “cloaking,” where a web site shows a search engine content that’s different from what a human would see.

Last week, our AdWords Cloaks Its Help Pages, Gets Banned By Google story covered how Google’s AdWords help pages were found to be cloaking. Google promptly removed those pages from its index. For how long? Google told us:

The web quality team is currently reviewing the reconsideration request and don’t have any more details to share.

Soon after our story came out, it was also noted that help pages for Gmail and Google Webmaster Central were also cloaking. Ironically, the some of the help pages in Google Webmaster Central contain Google’s guidelines against cloaking. In addition, it seemed likely that help pages for many other Google services were also cloaking. However, these have all stayed in the index.

What’s up with that? The answer, says Google, is that by the time it investigated the cloaking, it had stopped — so those pages aren’t being punished. The company told us:

Yes other support pages were were inadvertently showing different content to the Google crawler than to users….

The webspam team submitted a removal after confirming that cloaking was taking place on the AdWords support pages. By the time they finished investigating the non-AdWords support pages, the issue had already been fixed and therefore we did not remove those pages from the index.

To step back a bit, as the postscript to our “AdWords Cloaks” story explains, the issue that hit the AdWords pages and caused them to cloak seemed to be a technical problem shared by the system that served ALL of Google’s help pages. All of Google’s help pages appeared to be cloaking, though we only spot verified a few cases.

By the time we asked Google about this, the problem that was causing all of its help pages to cloak had been fixed. Therefore, despite knowing that its own pages had violated its own “law” against cloaking, Google’s looking the other way.

Since You’re No Longer Speeding…

In the real world, it would be as if a traffic cop saw a car speed past it well in excess of the speed limit. By the time the police officer catches up with the car, it is now doing the speed limit, so the officer decides it can no longer issue a ticket.

If Google’s serious about proving how even it has to follow its own rules, then all of Google’s help pages should be banned. Banning just the AdWords pages, when Google knows the “crime” was committed by other pages beyond those, just feels like an excuse to get around a really bad situation.

After all, too many people search on Google for answers to things about Google. The company simply cannot ban all of its help pages. It’s a bad user experience, not to mention an embarrassment if people have to go to Yahoo or Bing to search on how to use Google.

Big Enough? You Get A Handslap

Google’s sidestepping a bad situation, and it’s far from the first time it has done this. There’s long been a feeling in some SEO quarters that companies that are big enough, or important enough, can violate Google’s guidelines and get only a handslap.

BMW is one of the classic examples. Back in 2006, the company was found to be cloaking on its German site. Google announced that BMW’s pages were banned. See, even big companies like BMW have to follow the rules! Except a few days later, BMW was back. The cloaking had stopped, and BMW had a speedy reinclusion that ensured people trying to find the car company could do so.

Google couldn’t afford not to include BMW. Someone looking for BMW isn’t going to blame BMW for not showing up because it violated Google’s rules. The typical Google user doesn’t care about those rules. They just want relevant results. In a search for BMW, dammit, that means finding BMW!

But a small site, some mom-and-pop service or some other company that isn’t a big brand, that few people are likely to miss? Those sites are far more likely, in my view, to face being banned for a lengthy period of time and struggle with redeeming their reputation with Google’s ranking algorithms even after being restored.

The gut reaction to all this might be that Google should be fair. Ban all of its help pages, and ban them for a lengthy period of time! Punish big companies as Google would small companies. Be fair!

Look At Intent, Not Tactics

I’d argue that the “Be Fair” mantra means looking at intent, rather than tactics. Being fair means you don’t ban either a big company or a small company because they violated a technical guideline. You punish them because they intentionally worked to harm the user experience, in your opinion.

Google almost certainly didn’t intentionally mean to harm the user experience nor violate its webmaster guidelines by cloaking its help pages. It was a goof, and banning the AdWords pages is helping no one. I doubt Google’s technical team behind the help pages feels they’re being taught a better “lesson” than if they were just told to fix the problem without a penalty being applied.

As for sites other than Google, same thing. Banning them for cloaking, if there’s no clear intention to be violating Google’s guidelines, doesn’t make a lot of sense. It just engenders bad will. Send a warning, sure (and in some cases, Google does this). But an immediate penality? Hold on there.

Perhaps the worse thing that the technical guidelines do is cause a lot of wasteful finger-pointing across the SEO space. There’s no end to the parade of sites that are spotted cloaking or violating some other design guideline. A blog post is done. “What’s Google going to do?” is asked. And life moves on to the next target.

This rinse-and-repeat cycle is so very tiresome. For more about it, plus some history on the long debate over cloaking in general from me, see these past posts:

• YADAC: Yet Another Debate About Cloaking Happens Again
• The Long Road To The Debate Over “White Hat Cloaking”

Know The Rules; Publicize When You Don’t Punish

To be clear, I’m not encouraging people to break Google’s rules. Anyone involved with creating or marketing a web site should understand Google’s webmaster guidelines. If you want to succeed with Google, you play by its rules. Or, if you break the rules knowingly, don’t complain if you’re caught.

But I’m also encouraging that perhaps Google should be more vocal about when it cuts people some slack, including itself. Rather than confirming that some big (or small) site has been punished, perhaps talking about why it chooses not to punish in some cases would be helpful.

Postscript: Matt Cutts, who heads Google’s web spam team, has sent us this additional information:

The webspam team’s policy is that when we’re ready to submit an incident, if the behavior in question has already been corrected (e.g. hidden text has been removed), then we typically don’t submit the incident. In this case, the cloaking stopped after the webspam team had submitted the initial removal, but before the investigation was completed for the other support pages.

What’s going on?

The Polish blog Magiczne SEO i SEM noticed yesterday that the snippet for the Google Adwords help page in the search results seemed a little unusual. (You can read the English version via Google Translate.) Looking closer, the cache didn’t seem to match the page the visitor sees. We at Search Engine Land then took a look and found that accessing the page with the Googlebot user agent brought up substantially different content, particularly sections with headings marked as “hidden”.

We asked Google about this, and a spokesperson told us:

“I can confirm that some Google support pages were inadvertently showing different content to the Google crawler than to users. This error has since been fixed.  We will investigate how this happened and make sure that we take appropriate action.”

And it seems they have.

Google is the first to point out that those working on Google products don’t have any special knowledge of search just because they work at a company whose primary product is a search engine. The recent SEO report card they issued on Google products illustrates this well. And Google products have been found to be violating the webmaster guidelines before (and have had action taken against them).

In fact, the Google support pages were banned back in 2005 for cloaking. (In this case, it was due to a misconfiguration of the Google Search Appliance, which could very well be the problem again.) In early 2009, Google Japan was found to be buying links for PageRank, and action was taken in this case as well.

In this case, Google’s action was quick. They responded as soon as they were alerted to it. What can we learn from this? Don’t show search engines different content than you show users, even if you have good  technical reasons for it. Check that your internal infrastructure (such as the Google Search Appliance) isn’t misconfigured to serve different content to search engines. And don’t necessarily look to Google products as the shining examples of what to do for SEO. Just because those teams work at Google, doesn’t mean they’re search experts.

Postscript by Barry Schwartz: It seems like this goes well beyond just the AdWords Help area, it seems like this is how Google’s help sections work site wide.

The Google Webmaster Help area’s cache doesn’t match up with what I see on the non cache version nor does Gmail’s help forum and many others.

Postscript 2: See our follow-up piece, Why Google Should Ban Its Own Help Pages — But Also Shouldn’t.

Fetch as Googlebot

Of most interest to webmasters, SEOs, and web developers is likely the Fetch as Googlebot feature. You can specify any URL on your site and see the HTTP response (header and contents) that the server returns. Simply  indicate the URL and click the Fetch button. It may take a few moments for Googlebot to access the page and return the results, since it fetches the page in real time. (Refresh the page to see the progress.)

Click the Success link once it’s been processed to see the results.

How is this different from simply looking at the source code of the page?

• You see the HTTP header information at the top. This information is generally easily available through tools such as Live HTTP Headers, but isn’t contained in the source code itself (since that information is coming from the server, not the page).
• You can see if the server is returning any of the page information differently than the page has been coded.
• You can see if the server is returning something different to Googlebot than what other users see. This tool uses the same user-agent and IP range as Googlebot when it crawls the web, so if the server is configured conditionally for user agent or IP address (typically known as “cloaking“), you’ll see  what’s being conditionally served to Google.
• You can use the tool to test changes (particularly things like redirects) in real-time.

Note that this tool won’t necessarily show you the content that Google is able to extract from the page. If the page contains JavaScript, for instance, you’ll see the raw JavaScript code contained on the page, not the rendered view visible in the browser. Which, unfortunately means you can’t use this tool to determine if Google is able to access content contained in rich markup.

What’s this about cloaking?

This tool can help you determine if the pages are being cloaked to Google. This may be useful if you’re coming into a project late and aren’t sure what’s been previously done. It can also help uncover if your site has been hacked. Back in 2006, Googler Matt Cutts and I did a show on Webmaster Radio during which we talked about how in some cases, a hacker might add links to a site and then cloak those pages so that the site owner never sees them. Only Google does. At the time, Matt suggested using Google Translate (and choosing English to English) to see what Googlebot was being served, but this tool can now more easily serve that purpose. Matt confirmed this to me this morning: “The biggest use case is just debugging site issues. Of those, the biggest case will be hacked sites. Some attacks will hide content until search engines fetch the page (and some attackers add a noarchive tag so that the search result doesn’t have a “Cached” link), so a site could look clean to the website owner. Using this feature will site owners verify that there are no hidden links in the page that Google actually fetches.”

How do I test redirects?

If you’ve implemented redirects, you can use this tool to test how Googlebot will interpret those redirects without waiting for those pages to be crawled. For instance, when I fetch www.searchengineland.com, I see that the redirect is correctly implemented as a 301 and points to searchengineland.com:

You can also use the tool to troubleshoot URLs listed in the Crawl Errors > Not Followed report. You can also test these URLs using something like Live HTTP Headers or by trying to access the URLs in a browser, but if neither of those methods uncover the problem, this tool can help determine that the issue is specific to Googlebot. You can also use this tool to verify that fixes you’ve made to redirect errors uncovered by the Not Followed report have really solved the problem.

(Note that the tool currently has a limit of 100kb per page. However, this is for the tool only and doesn’t apply to Googlebot’s normal crawl of the site. Google is monitoring feedback to see if many site owners find this size to be limiting.)

Malware details

The Google Online Security Blog has more information on the malware details tool. Previously, webmaster tools reported when the site was flagged has having malware and listed sample URLs. This new tool will also show samples of the malicious content, and in some cases, the underlying cause. This should help those site owners whose sites have been hacked to include malware find the problem and fix it. If your site does contain malware and you’ve fixed it, you can request a review to have the malware alert removed in search results.

Let me take you through some of the history of this program as it relates to web search.

Back in March 2007, Danny Sullivan’s Yet Another Debate About Cloaking Happens article talked about how First Click Free was offered to news publishers but not general web publishers, something he wanted to see changed

In September 2007, First Click Free: Accessing Subscription-Based Articles For Free Via Google News covers how Google did a blog post to help promote the idea of First Click Free. It still was a news publisher-only program, but many news publishers weren’t aware of it.

In June 2008, Google did a blog post saying that Google News’s First Click Free policy could be applied to “to include your premium or subscription-based content in Google’s websearch index without violating our quality guidelines.”

Many people missed this important change. Danny highlighted it in a post that came out of SMX Advanced. Over at Search Engine Roundtable, Matt Cutts also commented to clarify things, after Google elsewhere has mistakenly said First Click Free was something “reserved” for Google News. Matt said:

First Click Free originated with Google News, but you can use the same way of handling content in web search (show the same page to users and Googlebot, then if the user clicks to read a different article, then you can show them the registration or pay page).

Because the same page is presented to users and to Googlebot, it’s not cloaking. So First Click Free is a great way if you have premium content to surface it in Google’s web index without cloaking. Hope that makes sense.

Today’s post should help end any further confusion. First Click Free isn’t just for Google News — anyone can use it.

How Does Google Define Cloaking?

First, if you want to understand how many tiring debates we’ve had over cloaking issues, I strongly urge you to read my YADAC: Yet Another Debate About Cloaking Happens Again post from last year. One of the things it discusses is how Google dropped its definition of what cloaking was in 2006. Before then, it was:

The term "cloaking" is used to describe a website that returns altered webpages to search engines crawling the site. In other words, the webserver is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they’ll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking.

We were left with this on another page:

Make pages primarily for users, not for search engines. Don’t deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking."

Then around June of last year, as best I can tell, we got this:

Cloaking refers to the practice of presenting different content or URLs to users and search engines. Serving up different results based on user agent may cause your site to be perceived as deceptive and removed from the Google index.

Some examples of cloaking include:

• Serving a page of HTML text to search engines, while showing a page of images or Flash to users.
• Serving different content to search engines than to users.

If your site contains elements that aren’t crawlable by search engines (such as Flash, Javascript, or images), you shouldn’t provide cloaked content to search engines. Rather, you should consider visitors to your site who are unable to view these elements as well. For instance:

• Provide alt text that describes images for visitors with screen readers or images turned off in their browsers.
• Provide the textual contents of Javascript in a noscript tag.

Ensure that you provide the same content in both elements (for instance, provide the same text in the Javascript as in the noscript tag). Including substantially different content in the alternate element may cause Google to take action on the site.

Then in June of this year, Google did a major blog post on the topic, which along with cloaking covered topics such as:

• Geolocation
• IP delivery
• First Click Free

Much of what’s in the blog post has yet to migrate to the help pages, though I’m sure it will. But the more important point to me is that over time, things that people might have once considered cloaking have become acceptable to Google — and get named something other than cloaking in the process.

Renaming Confusion

Let’s take geolocation as an example of that. Years ago when Google would campaign against cloaking, those who disagreed would argue that Google itself cloaked. For example, go to Google.com from outside the US and you’ll see a different page than someone in the US sees. Cloaking! No, Google eventually argued — that’s not cloaking. It’s geolocation and perfectly fine for anyone to do.

OK, how about showing Google content that human users only see if they register with a site and/or pay for access. Cloaking! Well, here Google took some time to clarify things for web search. Google informally said that users shouldn’t have to pay to see content if they came from Google. In Google News search, this is called First Click Free, and it’s a formalized process for news publishers to follow (that’s why you can read the Wall Street Journal for free via Google News). But for web search, we never got a formal program or formal mention that this was allowed until the June 2008 blog post.

Along the way, we’ve also had more advice about doing things to make Flash content visible through JavaScript detection and replacement (see today’s excellent post, Google Now Crawling And Indexing Flash Content, from Vanessa Fox on those methods plus the new Flash reading that Google’s doing). We’ve also had advice about rewriting URLs to make them more search engine friendly — good advice – but also a situation that in some cases can be URL cloaking. We’ve also had situations where the search engines have absolutely known someone was cloaking — no ifs, ands or buts — but decided the intent wasn’t bad so let it go through (which is fine for me — I care more about intent than kneejerk reaction to a bad "tactic").

It’s confusing. Very confusing. Now what was cloaking again?

Charting Acceptability

To help, Rand put out a tactics chart with things rated from "pearly white" to "solid black." That’s nice, but it’s also debatable. In fact, it IS being debated! So I wanted to go back to what Google itself is saying and work from that.

The most important thing is that Rand takes a "tactics" approach — are you using IP detection, cookie detection, and so on — then suggests that as you add things like user agent detection or other tactics, you tip into the danger zone.

In contrast, Google is less about how you physically deliver content and more focused on the user experience — the end result of what a user sees. The bottom line in that remains quite simple — Google generally wants to be treated the same as a "typical" user. So if you typically use cookie detection to deliver content, Google’s got no real issue with that — as long as it sees what a typical user might see who doesn’t accept cookies.

Below, some content delivery methods now acceptable to Google that might have once been considered cloaking:

Where am I getting the info for the chart above?

• Geolocation & First Click Free come from Google Webmaster Central’s June 2008 blog post. The post talks about Google’s First Click Free as if it is only for news content. That’s not correct. You can use this method for Google web search, as was covered by Google reps speaking at SMX Advanced 2008 last month.
   
• JavaScript Replacement: Google’s page on cloaking addresses this, plus reps have covered this advice in many public forums.
   
• Landing Page Testing: Google Website Optimizer Now Available, But Is It Cloaking? covers the issues here, and How does Website Optimizer fit in with Google’s view of cloaking? from Google AdWords help has detailed advice on how it is acceptable (and when it is not)

• URL Rewriting: Good Cloaking, Evil Cloaking & Detection from Stephan Spencer last year talked about the ways to strip down URLs so they appear "nicer" in search results. He’d surveyed the major search engines that emphatically told him this was OK, even if their spiders were detected in order to do it. I believe this is still fine as long as the content itself isn’t changing.

And what’s not OK?

• Cloaking comes from Google Webmaster Central’s June 2008 blog post and its help page on the topic. If you’re delivering content to users that’s different than what Google sees — and it’s not listed on the first chart – you’re probably cloaking.
   
• Conditional Redirects: AKA, cloaking redirects. Discussed at SMX Advanced 2008, especially how Amazon has been doing it. Google warned that conditional redirects might be risky.

"Whitehat" Cloaking

I said I’m pretty tired over the cloaking debate, right? That’s because way back when the debate started, we had search engines that DID allow cloaking via XML feeds or other methods – or they turned blind eyes to cloaking they knew about — or they had programs like Google Scholar that sure felt like cloaking before guidelines caught up to say no, those aren’t cloaking.

These exceptions made it difficult to say cloaking was "bad" when clearly some of it was allowed. That’s why I urged the search engines — Google in particular — to update the guidelines to say that any type of "unapproved" cloaking might get you banned. My thought was that we could maybe skip past the debate over tactics (and finger-pointing — ah-ha!!!! — this major site is cloaking, ban them!) and focus more on the user experience.

Things have changed since then. The guidelines, as I’ve explained, have gotten more detailed — and exceptions have been spun off into new names. As part of this, the idea of "white hat cloaking" has come out, the "good" cloaking that’s now acceptable.

I can tell you firsthand that Google doesn’t like the phrase "white hat cloaking" at all. To Google, there’s cloaking — it’s always bad — and there are the other content delivery methods I’ve outlined above.

OK, I’ll roll with that. Personally, I’ll avoid talking about "white hat" or "good" cloaking if it helps improve relations between webmasters and the search engines AND helps newbies avoid getting into trouble. But I do think we need a term to encompass content delivery methods that do target spiders or consider them in some way.

First Click Free, JavaScript replacement, even geolocation — while maybe you don’t do something special for the search engines as part of them – you’re still considering them. Indeed, part of what you consider is to ensure that you might NOT do something special.

If cloaking is a bad word to the search engines that can never be redeemed, we still need a name for the "good" things that are out there. I’ve got my thinking cap on, and if you’ve got ideas, let me know by commenting.

I’ll close by saying that my list above is not complete, in terms of the various content delivery methods that are out there. I hope to grow this over time — and you can help with your comments, as well.

On the geolocation front, Google recommends you treat “Googlebot as you would a typical user from a similar location.” So, if Googlebot’s IP is coming from California, then serve up the same page you would serve a web user from California. The same rule applies to IP delivery: serve the “same content a typical user from the same IP address would see” to Googlebot. Clearly, if you serve different content to Googlebot then you would a normal user, that would be cloaking and would go against Google’s guidelines. Finally, Google discusses the first click free program from Google News, where if you visit an article a second time, you would need to enter a password. When serving this content, you need to make sure the content is the same as when a normal user goes to the page.

Did Google really “define” what they think cloaking or IP delivery is? Well, if you consider a definition to remove any doubt or questions, then no. If you look at the related Google Groups thread, you can clearly see there is a lot of confusion and “what-ifs” in the thread. So although the post did clarify some issues to those who are unaware of the issues with IP delivery and its various forms, it did not necessarily cover every “what-if” scenario. Will it ever? I doubt it.

Here is an awesome video from Maile on IP delivery: