A new feature has appeared in the Site Configuration Settings Sections of Google Webmaster Tools. The setting, called Parameter Handling, enables site owners to specify up to 15 parameters that Google should ignore when crawling and indexing the site.

Google lists the parameters they’ve found in the URLs on your site, and indicates whether or not they think they those parameters are extraneous (with a suggested “Ignore” or “Don’t ignore”. You can confirm or reject those suggestions and can add parameters that aren’t listed.

So what does this mean for site owners?

The primary value of the feature is to improve the canonicalization of a site in Google’s index due to duplicate content. Canonicalization issues occur when multiple URLs load the same content. This scenario can be problematic for a number of reasons (for instance, it can skew analytics data) but from a search perspective, canonicalization issues can cause:

• Crawl efficiency problems: if search engine bots crawl the same page via multiple URLs, they may not have resources to crawl as many unique pages on the site
• PageRank dilution that can lead to lowered search rankings: if external sites link to multiple versions of a page, each page has less Page Rank value than if all links were to one version
• Display and branding problems: search engines display only one version of the URL; you ideally want the canonical version of a URL to display (mysite.com/goldfish) rather than a version with extraneous parameters (mysite.com/goldfish?adid=1205123&sid=452006&sort=high-rating&loc=sea)

A number of canonicalization solutions exist, including several that are Google-specific, so why did they launch this new feature? Yahoo! has included a similar feature as part of its Site Explorer webmaster product for some time and site owners have been asking for a similar feature from Google for a while (certainly at least since I was working on Webmaster Central).

Below a rundown of the various canonicalization options and how this one differs.

Google Webmaster Tools Parameter Handling: When URLs Can Contain Optional Parameters

This new option only helps with canonicalization issues that are caused by optional parameters that are in a standard key-value pair format and that you specify. In other words, it can only be an exclusionary list (don’t crawl parameters x,y, and z) rather than inclusionary (only crawl parameters a and b).

Wouldn’t you always know the complete list of potential parameters? Hopefully. But some canonicalization issues happen because a URL can take any parameters at all. Ideally, you want to ensure your server isn’t set up this way, but if you need this configuration (for instance another team or outside agency needs the ability to use any custom tracking code without waiting for that parameter code to be added to the server set up), then you’re better off using the meta canonical tag.

The two most common reasons for optional parameters and that this feature will work well for are:

• Tracking codes used for analytics data (in this case, you may not want to implement a 301 redirect from the long version of the URL to the canonical one since you could lose the data)
• Page layout changes, such as sort orders (in this case, the code on the page uses the parameter to change the layout of the page, but from a search engine perspective the content on each version is the same, just in a different order)

Why use this canonicalization option over the others? The biggest benefit is likely in the increase in crawl efficiency. When Google discovers a new URL, they can check the included parameters against the parameter handling list and remove any optional ones before crawling it (but still credit any found links to the page). This could substantially reduce the crawling overhead on a site and could free up considerable bandwidth for getting other pages of the site crawled.

It’s also fairly simple to use. Just scan the list of suggested parameters and click the ones that are optional. In some organizations, it can be difficult to get source code added to web pages, making the implementation of the canonical tag difficult and time consuming. With this option, if you have verified webmaster tools access, you don’t need to involve IT at all.

What are the drawbacks to this option? The most obvious issue with this option is that it only works for Google. In time past, you could use this setting and the corresponding one in Yahoo! Site Explorer and not worry about other engines. But with Microsoft Bing’s impending (likely) replacement of Yahoo’s search index, it’s quite possible that Yahoo’s feature will go the way of its index, and if Microsoft doesn’t offer something similar, then a search index with 25%+ market share could be getting your URLs wrong.

You could also shoot yourself in the foot, metaphorically speaking. You could accidentally tell Google to ignore important parameters that, if dropped from the index, could wipe out large portions of your site. As Google adds more of these types of features to webmaster tools, it becomes more important to ensure that anyone who has access to them know what they’re doing.

In reality, Google likely has safeguards in place that at least partially protect against such accidental destruction. That’s undoubtedly why they say that “While Google takes suggestions into account, we don’t guarantee that we’ll follow them in every case.” They don’t want large portions of their index disappearing either.

Unlike accidental blocking with robots.txt, which search engines follow as a directive, this feature (and many of the others) is a signal only. If the other signals already in place strongly contradict it (for instance, the content seems to be vastly different), it likely won’t be used.

But even though Google has safeguards like this one in place, you may not want to chance it if you’re not confident of which parameters are really optional (all the time, since this is a site-wide setting).

This option also won’t work if your canonicalization issues aren’t related to parameters or if the parameters aren’t in standard key-value pair format.

Meta canonical attribute

The canonical attribute is a page-level meta tag that specifies the canonical version for the page. This can be useful because no matter what optional parameters are added to the version of the URL that renders the page, search engines can always know the canonical version. You can find detailed information about this tag in my article about its launch.

Why use this canonicalization option over the others? You just specify the canonical version of a page once, and no matter what parameters are added to the URL, search engines are always provided with the canonical version.

Since this meta data is on the page itself, any search engine can read it, and in fact, Google, Yahoo!, and Microsoft have all announced support for it. As of yet though, only Google seems to be actively using it.

What are the drawbacks to this option? Unlike with the parameter handling feature, search engines have to crawl the page before they can read the tag, so some crawl efficiency is lost. This tag should promote long-term efficiency, however, since theoretically,  once the bot has crawled the non-canonical version of the URL and read the tag, it shouldn’t have to crawl that version of the URL again.

As already noted, implementation requires modification of the page source code, which isn’t always easy within some organizations.

As with parameter handling, it’s possible to implement this tag incorrectly. For instance, it’s been discovered that some sites have accidentally set the canonical version of every page to the home page. As with the parameter handling feature, search engines consider the tag a “strong hint” as a precaution against these types of mistakes and won’t use the data when it strongly contradicts their other signals. In the case of Google, the only search engines who is actively using the tag so far, this has proven to be the case.

301 redirect

It’s  universally agreed that (other than not have multiple versions of a URL at all) the best way to canonicalize URLs is to redirect all versions to the canonical one using a 301 redirect. This implementation sends all users and search engines to the canonical version and effectively consolidates all links to the page and ensures only the canonical one is indexed and ranked.

Why use this canonicalization option over the others? It’s understood and followed by all major search engines and it provides the best user experience (visitors have one URL to access, bookmark, and share). In most cases, search engines consolidate all links to the redirect target and rank the canonical one.

This option is the best choice when you are moving content (for instance, changing your URL structure or changing domains) and to indicate whether your want content indexed under the www or non-www version of the domain.

Also keep in mind that if you redirect to the canonical version you’re more likely to get links to the right version, since most visitors will simply copy and paste what they see in the address bar.

What are the drawbacks to this option? When you are using parameters for sort orders or tracking, a redirect may negate those parameters. ou can generally configure your analytics program to handle this properly, but it probably won’t work out of the box.

Redirects aren’t always properly implemented. For instance, they might inadvertently be implemented as a 302 (or worse yet, a JavaScript redirect or meta refresh). Or they may generate redirect loops or infinite redirect chains. In these cases, search engine bots eventually abandon the crawl attempt (and with Google and Microsoft, you can get a list of these URLs in their webmaster tools products).

Redirects can also slow down crawl efficiency, particularly due to redirect chains. Ideally, search engines crawl the redirect then eventually stop crawling the origination URL, but if the bot encounters links to the original URL, it will continue crawling both versions (or more, if the page has moved multiple times).

Google webmaster tools change address feature

This feature enables you to tell Google when you’re changing domains. You have to verify ownership of both the old domain and the new domain and then you can specify a move from one to the other. You can find more information about this  feature xx.

Why use this canonicalization option over the others? The best use of this feature is when you are changing domains and you aren’t able to implement a 301 redirect from the old domain to the new. (This is the case, for instance, with blogspot.com sites.) Even if you are able to implement the redirect, it can’t hurt to let Google know as well!

What are the drawbacks to this option? You can only use this option to move from one domain to the other. And as with the other Google webmaster tools features, it only works for Google.

Google webmaster tools preferred domain feature

The preferred domain feature enables you to tell Google whether your want your domain indexed with the www subdomain or without it.. Since most sites resolve either way, a complete duplicate set of content  of your site will exist if you don’t set www/non-www canonicalization. Why is this a problem? Ideally it’s not and search engines consolidate the content correctly. But often, search engines find links to both versions and end up crawling both, indexing both, and crediting the links to the versions separately.

Why use this canonicalization option over the others? You may as well always use this option, although you should implement a 301 redirect as well, if you can. Google initially implemented this feature for those sites that weren’t able to do so.

What are the drawbacks to this option? Again, this option works only for Google. And it doesn’t provide as much of a guarantee as a 301 redirect.

Blocking duplicate content with a robots directive

The traditional advice for avoiding duplicate content has been to block the duplicates with robots.txt (or a robots meta tag) to ensure the correct version is indexed. It can be important that the right version be indexed vs. the version intended for print, for instance.

Why use this canonicalization option over the others? Generally speaking, you shouldn’t now that the canonical meta tag is available. The scenarios for which you wouldn’t want to redirect (such as the print version example) can be more easily solved with the canonical tag and the scenarios for which you’re worried about crawl efficiency issues that would leave large portions of your site uncrawled (such as large-scale optional parameters) can now more easily be solved with Google’s parameter handling feature.

What are the drawbacks to this option? The primary drawback to this option is the loss of link credit. Any links to blocked pages fall into a black hole and can’t be credited to the canonical version of the page, as happens with the other options.

The parameter handling feature can also provide insight on how Google sees your site

For some time, Google has been attempting to canonicalize URLs and show the canonical version in the results, even when a site owner hasn’t implemented any of these canonicalization options. For instance, they may determine that several pages contain the same content and algorithmically consolidate them and associate them with the one Google determines is canonical. They haven’t described exactly how they determine the canonical version, but they might, for instance, choose the URL with the fewest number of parameters or the shortest version of the URL.

Last year, they started letting webmasters know when they encountered URLs that they thought were extraneous and were causing crawling problems. It’s likely that Google is using a similar source to generate the list of parameters it suggests should be ignored.

In this way, the parameter handling feature provides insight into how Google perceives the site. If you see many parameters listed that aren’t optional, take a look at the content on the URL that use those parameters.

This could signify a larger problem. It could be that Google doesn’t see enough unique content on them (this can happen, for instance, with pages that list part numbers, contain mostly images and item codes, or list little information outside of a login). You may want to look for ways to differentiate the pages a bit more.

Interestingly, the Google Webmaster Central blog has a new post about duplicate content, but no mention of this new feature. Thanks to Brian Ussery for pointing it out!

Imagine that you fired up your computer and found that a bunch of your programs no longer worked, because behind the scenes, the operating system had been upgraded without any backwards compatibility. That’s what happened this week with Google. Some things that were working just fine now are broken, because Google isn’t being backwards compatible. And that’s fairly unprecedented.

Don’t panic. One of the changes really shouldn’t hurt many sites, impacting only a “power SEO” technique commonly called PageRank sculpting that I’d say fairly few use. The other has a bigger impact and potentially means thousands of sites may now be violating Google’s rules on paid link without knowing it. But that’s not likely to have an immediate impact. I’ll explain both changes in more depth below.

The most important thing is that in both cases, the changes may require site owners to alter their web sites not because they were “chasing the algorithm” but instead because they were following Google’s own rules and instructions. They were doing what was advised, and now they may have to undo that work.

That’s the unprecedented part. Google has constantly upgraded how it deals with site content, from early advances like indexing PDF documents to later changes like showing “sitelinks” for web sites. These upgrades have been generally good and involved little to no work on the part of the site owner, until now.

PageRank Sculpting: Spending A Page’s Authority Money

Let’s take PageRank sculpting. In general, every individual web page that Google finds has some degree of importance that the page can pass on to other pages — PageRank. Links from that page to other pages are how it passes that importance along. And in its most basic, earliest form, each link on the page equally shared some of the importance.

Consider it like this. Imagine authority is money, and a particular page has $10 in “authority” to spend. It links out to 10 pages, so each of those pages gets $1 ($10 divided by 10). If it links to 20 pages, each gets 50 cents ($10 divided by 20). If it links to 5 pages, each page gets $2 (you get the math by now).

With PageRank sculpting, the idea is to effectively block some of the links on your web page (using the nofollow attribute or some other means) from getting any authority. Perhaps you have a lot of navigational links to other pages inside your web site. Rather than spend authority money on these pages, you might prefer to spend it on a smaller number of important pages that could use a boost.

PageRank Sculpting Gets Popular

This technique has been around for ages and had various names until the middle of 2007. That’s when it went more mainstream in the advanced SEO space. And in particular, it went that way I feel because Google spam fighting czar Matt Cutts talked about how Google’s YouTube was using PageRank sculpting during an open discussion at Google with a variety of advanced SEO people about techniques and issues.

I recall it being described as a means to ensure your best pages got the most PageRank. I also recall being kind of annoyed about it (and think I said so during the meeting). For years, we’d been told that site owners shouldn’t have to do extraordinary things to help search engines. Good page titles, good ability to be crawled, sure. But having to think about things on a link-by-link basis? That’s something I assumed Google was already up to snuff about. My assumption had been that Google long decided to discount how much credit it assigned to things like navigational links, when it could see the same links appearing on multiple pages within the same web site.

Now to be clear, it’s not like Matt told everyone in the room to immediately do PageRank sculpting. Many topics were discussed, and this was just one of many things covered. But it was advice that came from Google — and it turned into a genie that wouldn’t go into the bottle.

Soon after, Rand Fishkin at SEOmoz did an article about the topic, and more soon followed on the web. It was a topic at conferences. It was a hot new fashion in SEO. And while plenty in the SEO space will chase after the latest (and often useless) algorithm fad — this was a chase sparked by Google itself. Why wouldn’t advanced people do it?

PageRank Sculpting Gets Debated

Not everyone agreed it was helpful. There’s been quite a bit of debate on whether it gives boost or not. Some have argued against using it at all. And the search engines, when asked about it since it gained popularity, have generally said that there are other things that are better worth the effort. But neither had they ruled it out. As I summarized last year:

I agree with the view that sculpting is a marginal activity compared to other things that can be done. But if you’re an advanced SEO — even someone advanced in terms of working with design issues — maybe it’s not so marginal. The search engines themselves are saying it has some value. They’ve not said it’s a flat out waste of time. And if you’ve mastered all the other things that are much more important, then yes, something like this may very well be worth giving more attention to.

Or as Michael Gray explained, if you’re driving a beat-up old car of a web site, putting a PageRank sculpting “engine” in it probably isn’t worthwhile. But if you’ve got a hot new sports car, well….

PageRank Sculpting Gets Depreciated

So today at SMX Advanced, sculpting was being discussed, and then Matt Cutts dropped a bomb shell that it no longer works to help flow more PageRank to the unblocked pages. Again — and being really simplistic here — if you have $10 in authority to spend on those ten links, and you block 5 of them, the other 5 aren’t going to get $2 each. They’re still getting $1. It’s just that the other $5 you thought you were saving is now going to waste.

Further, it was explained that YouTube wasn’t doing sculpting way back in 2007 as a way to boost certain video content. Instead, it was that YouTube randomly shows some video content and didn’t want these random selections to perhaps gain more authority than they should. And even with the change announced today, that still works. In the past, the unblocked videos got more authority money and the blocked ones got none. Now, the unblocked videos still get authority money — just not as much — and the blocked ones still get none.

But while that may be how it works on YouTube, I still recall PageRank sculpting being positioned by Google as a way to also give some pages more link juice. To quote Matt when asked about this in an official Google thread:

What are some appropriate ways to use the nofollow tag? One good example is the home page of expedia.com. If you visit that page, you’ll see that the “Sign in” link is nofollow’ed. That’s a great use of the tag: Googlebot isn’t going to know how to sign into expedia.com, so why waste that PageRank on a page that wouldn’t benefit users or convert any new visitors? Likewise, the “My itineraries” link on expedia.com is nofollow’ed as well. That’s another page that wouldn’t really convert well or have any use except for signed in users, so the nofollow on Expedia’s home page means that Google won’t crawl those specific links.

Most webmasters don’t need to worry about sculpting the flow of PageRank on their site, but if you want to try advanced things with nofollow to send less PageRank to copyright pages, terms of service, privacy pages, etc., that’s your call.

I’ve bolded the key part. Matt stresses — as he’s consistently done since talking about this at the SEO meeting — that this is something most people didn’t need to worry about or do. But saying “why waste that PageRank” means that at the time of giving this advice, PageRank was something that could be “saved” and “spent” on other pages.

You can expect Matt will do a blog post to cover this topic more. You can expect lots of people to be analyzing the change, and what it might or might not mean. And you should really understand that it was never the case that links shared equally in the amount of authority money a page had. In talking with Matt during the “You & A Session” at SMX Advanced, he confirmed that Google itself makes many determinations of how exactly a page can spend that authority money. IE — while a page might have $10 to spend, Google itself largely acts as the page’s investment banker, not the page’s author.

I wouldn’t panic and immediately start removing nofollow attributes that have been done for PageRank sculpting purposes. In general, I’d never recommend changing anything to a site that seems to be performing well. Take the time to let more discussion and information come from Google and other sources.

JavaScript onClick & Paid Link Worries

Those who PageRank sculpted following Google’s advice may have spent time doing something that no longer will work, or work as effectively, but they’ve not necessarily wasted time. Maybe it was helping them some in the past (plenty believe this). And they might not have to spend time removing it, any more than there are plenty of sites that still have meta keywords tags in place even though widespread search engine support of this was dropped long ago. That’s good depreciation, or effectively backwards compatibility. No one needs to change anything because the sites still keep “working” despite the past support being gone.

It’s a different case with Google’s new handling of JavaScript’s “onClick” function. To fully understand it, read Vanessa Fox’s in-depth report from last week, Google I/O: New Advances In The Searchability of JavaScript and Flash, But Is It Enough?, which broke the news here.

Links in JavaScript that were invisible to Google before are now being read. And some people have used JavaScript as a way to deliver paid links in a way that don’t violate Google’s guidelines may not technically on the wrong side of the Google law. It’s been a long accepted practice that this was a “safe” way to deal with paid links, once that Google’s suggested itself.

It’s as if Google has suddenly passed a new safety helmet law for web sites, mandating that the old helmets they’d been using are no longer good enough. Now they need to do something different.

What about nofollow? After all, Google’s been pushing nofollow as something sites should do as a safety device for paid links long after paid links themselves had been in existence.

True, and there are plenty of sites out there that have never caught up with this new Google guideline (and still sucky for those who really still innocently don’t know better). But that’s different than sites that thought they were doing the right thing and now which have to change again.

For the record, Matt said today that there’s no immediate penalties likely to be given out. Honestly, I think the spam team is still having to digest how to handle this change that’s been brought about by Google’s crawling team. And he also said that the nofollow attribute can be applied to JavaScript links that are not otherwise being redirected through a robots.txt block.

As I said in the case of PageRank sculpting, I wouldn’t immediately panic. But unlike with PageRank sculpting, if you’re selling paid links and thought JavaScript was protecting you, I would fairly quickly ensure that redirects are blocked by using nofollow within the JavaScript itself or by going through a robots.txt block.

(For an example of this, our paid links get delivered through JavaScript generated by Google Ad Manager. The links all get redirected through this domain — http://googleads.g.doubleclick.net — and you can see from the robots.txt file there that search engines aren’t allowed to crawl it. So, the links pass no authority on to other pages)

Backwards Compatibility Is Important

Overall, I want Google to keep advancing. But it needs to ensure that the changes don’t dramatically cause more work for site owners, as a result. We need a period of backwards compatibility in terms of Google indexing, just as much as it’s helpful with computer operating systems.

For more about the discussions today out of SMX Advanced, also see these selected stories from the live blogging round-up:

• Duplicate Content Solutions & The Canonical Tag – SMX Advanced Coverage 2009, SEO Gadget
• Duplicate Content Solutions & The Canonical Tag, outspokenmedia.com
• Beyond the Usual Link Building, outspokenmedia.com
• Beyond the usual linkbuilding – SMX Advanced 2009, SEO Gadget
• Live from SMX Advanced: Beyond the Usual Link Building, Search Marketing Sage
• Nofollow Makes News at SMX Advanced, BruceClay.com
• You&A With Matt Cutts, outspokenmedia.com

AOL recently soft-launched Love.com, which it advertises as a collection of “topic blogs” that “provide a central view into what the world is loving now.”

But it’s really an SEO play. Give the site any subdomain under the sun, and Love.com will make a new site on-the-fly with content scraped from news sites, blogs, YouTube videos, Twitter messages and more. Like this:

Everybody loves cold sores, right?

Well, not everybody loves Love.com. Says Dave Naylor:

“Approximately 1 bazillion keyword specific subdomains filled with scraped content and ads? Are AOL so desperate that they’re resorting to five year old spamming tricks?”

Aaron Wall shifts some of the blame to Google and Eric Schmidt’s recent statements about brands being the way to clean up the Internet “cesspool”:

“Google’s original strategy with the authority-centric algorithm was a false belief that the emphasis on authority would make the web a deeper and richer experience. New content would need to be better than older established content to outrank it. But as media companies face sharp losses Google is quickly finding out that their authority emphasis is creating a shallower web, where most of the big networks have 2 primary roles: create garbage and recycle garbage.”

Aaron also points out on Twitter that Love.com has more than 100,000 pages in Google’s index and is offering a free month of SEO training to whomever guesses how long it’ll stay above 100k.

For now, Love.com has 350,000 of these “topic blogs” and gets 100,000 unique visitors a week, according to TechCrunch. TC also says a full launch of Love.com is coming later this year.

A month ago, Google, Yahoo and Microsoft announced they will be supporting a new canonical tag that allows you to tell search engines that page X is a duplicate page to page Z. In a way, it is a 301 redirect, without the physical redirect.

The tag is incredibly powerful, as are 301 redirects and using this tag should be done with caution and slowly. Matt Cutts posted a new video explaining how one should go about using this tag, being that it is so new. Here is the video:

The web is full of duplicate content. Search engines try to index and display the original or “canonical” version. Searchers only want to see one version in results. And site owners worry that if search engines find multiple versions of a page, their link credit will be diluted and they’ll lose ranking.

Today, Google, Yahoo and Microsoft (links are to their separate announcements) have united to offer a way to reduce duplicate content clutter and make things easier for everyone. Webmasters rejoice! Worried about duplicate content on your site? Want to know what “canonical” means? Read on for more details.

Multiple URLs, one page

Duplicate content comes in different forms, but a major scenario is multiple URLs that point to the same page. This can come up for lots of reasons. An ecommerce site might allow various sort orders for a page (by lowest price, highest rated…), the marketing department might want tracking codes added to URLs for analytics. You could end up with 100 pages, but 10 URLs for each page. Suddenly search engines have to sort  through 1,000 URLs.

This can be a problem for a couple of reasons.

• Less of the site may get crawled. Search engine crawlers use a limited amount of bandwidth on each site (based on numerous factors). If the crawler only is able to crawl 100 pages of your site in a single visit, you want it to be 100 unique pages, not 10 pages 10 times each.
• Each page may not get full link credit. If a page has 10 URLs that point to it, then other sites can link to it 10 different ways. One link to each URL dilutes the value  the page could have if all 10 links pointed to a single URL.

Using the new canonical tag

Specify the canonical version using a tag in the head section of the page as follows:

<link rel="canonical" href="http://www.example.com/product.php?item=swedish-fish"/>

That’s it!

• You can only use the tag on pages within a single site (subdomains and subfolders are fine).
• You can use relative or absolute links, but the search engines recommend absolute links.

This tag will operate in a similar way to a 301 redirect for all URLs that display the page with this tag.

• Links to all URLs will be consolidated to the one specified as canonical.
• Search engines will consider this URL a “strong hint” as to the one to crawl and index.

Canonical URL best practices

The search engines use this as a hint, not as a directive, (Google calls it a “suggestion that we honor strongly”) but are more likely to use  it if the URLs use best practices, such as:

• The  content rendered for each URL is very similar or exact
• The canonical URL is the shortest version
• The URL uses easy to understand parameter patterns (such as using ? and %)

Can this be abused by spammers? They might try, but Matt Cutts of Google told me that the same safeguards that prevent abuse by other methods (such as redirects) are in place here as well, and that Google  reserves the right to take action on sites that are using the tag to manipulate search engines and violate search engine guidelines.

For instance, this tag will only work with very similar or identical content, so you can’t use it to send all of the link value from the less important pages of your site to the more important ones.

If tags conflict (such as pages point to each other as canonical, the URL specified as canonical redirects to a non-canonical version, or the page specified as canonical doesn’t exist), search engines will sort things out just as they do now, and will determine which URL they think is the best canonical version.

The tag in action

This tag will most often be useful in the case of multiple URLs pointing at the same page, but might also be used when multiple versions of a page exist. For instance, wikia.com is using the tag for previous revisions of a page. Both http://watchmen.wikia.com/index.php?title=Comedian%27s_badge&diff=4901&oldid=4819 and http://watchmen.wikia.com/index.php?title=Comedian%27s_badge&diff=5401&oldid=4901reference the latest version of the article (http://watchmen.wikia.com/wiki/Comedian%27s_badge) as the canonical.

The search engines stress that it’s still important to build good URL structure and also note that if you aren’t able to implement this tag, they’ll still keep the processes they have now to determine the canonical. For instance, at SMX West on Tuesday, Maile Ohye of Google explained how Google can detect patterns in URLs if they use standard parameters. For instance, with these URLs:

• http://www.example.com/buffy?cat=spike
• http://www.example.com/buffy?cat=spike&sort=evil
• http://www.example.com/buffy?cat=spike&sort=good

Maile explained that Google can detect (particularly when looking at patterns across the site) that the sort parameter may order the page differently, but that the URLs with the sort parameter display the same  content as the shorter URL (http://www.example.com/buffy?cat=spike).

While it’s rare for the search engines to join forces, this isn’t the first time they’ve come together on a standard. In November 2006, they came together to support sitemaps.org. And in June 2008 they announced a standard set of robots.txt directives. Matt Cutts of Google and Nathan Buggia of Microsoft told me that they want to help reduce the clutter on the web, and make things easier for searchers as well as site owners.

This new tag won’t completely solve duplicate issues on the web, but it should help make things quite a bit easier particuarly for ecommerce sites, who likely need all the help they can get in the current economic conditions. Site owners have been asking for help with these issues for a really long time so this should be a greatly welcomed addition.

Postscript by Barry Schwartz:

The search engines will be talking about this news at the Ask the Search Engines panel at SMX West. We will be blogging this panel live at the Search Engine Roundtable.

Struggling with duplicate content woes? Google’s done a blog post today to try and provide further advice and guidance. Among the highlights:

• Including a preferred URL in sitemaps submitted to Google can increase the chances that the URL you want is used, in cases where content can be found within your own site in different ways.
   
• Consider using 301 redirects, blocking, consistent linking, and other tips covered in this and this, which are official Google blog posts.
   
• To deal with offsite content that ranks above you, ensure that your site itself is being crawled and not-penalized by Google.
   
• Consider tips like providing slightly alternate versions of your content for syndication; use absolute links back to your own content or ask partners to block your content from being crawled. These come from a post that post-Google Vanessa Fox did recently on her personal blog: Ranking As The Original Source For Content You Syndicate.

With so much good content on the web, it’s inevitable that the same information will be displayed many times over. Whether it’s a blog post pointing to good statistics, an RSS feed pulled into a complementary site, or blatantly copied material, duplicate content can be an issue.

This week’s infographic shows how search engines make distinctions between original and duplicate content:

Graphic by Elliance, an eMarketing firm specializing in results-driven search engine marketing, web site design, and outbound eMarketing campaigns. The firm is the creator of the ennect online marketing toolkit. The Search Illustrated column appears Tuesdays at Search Engine Land.

“Water, water everywhere nor any drop to drink,” laments a sailor adrift in Samuel Taylor Coleridge’s The Rime of the Ancient Mariner. The poor guy needs water to say alive—and he’s surrounded by it— yet if he drinks it, the salt content will kill him.

As a search marketer, perhaps you can relate.

Much like our sailor friend, you’re surrounded by what you need—namely, content—which, of course, is great for your search presence. Yet too much of the same content can be… well, hazardous.

A little salt can’t hurt, right?

Thumbing through a retail catalog, you can often find the same merchandise represented in multiple categories. For instance, a particular pair of hiking boots might be found in the Men’s Clothing section, and in the Footwear portion, and maybe even in the Outdoors offering as well.

Not surprisingly, the same holds true for a website. After all, users don’t mind where they find the product on a site, as long as they don’t have to look too hard to find it. However, while a multiple-category strategy may work for the glossy paper ilk, the same approach could prove problematic for a search marketer. Specifically, wouldn’t it be considered duplicate content and carry a penalty?

Yes and no.

There’s really nothing wrong with having multiple versions of a page co-exist on your website. Sure, there’s some redundancy, but it won’t break the Internet. However, the problem arises when the same page lives in different sections on your site AND it has different URLs. Then it’s considered duplicate content by the search engines.

The result? It impairs a search engine’s ability to properly determine the relevance of your site to the product. And if an engine can’t serve the best results, their audience will… um, jump ship. In addition, attempts to game the system by duplicating your pages will get your site flagged and your visibility will suffer. Instead, avoid duplicate content issues and any potential penalties by utilizing a robots.txt or no index tag to keep multiple versions of your pages from being indexed.

The problem isn’t penalties&mdash’it’s link dilution

With that said, however, keep in mind that the duplicate content issue is far less about a ranking penalty than it is about link dilution.

Let’s say you have 30 external sites lined up to provide a link to your site; you just need to tell them what page to link to. If all 30 of those links point to the hiking boots page in the Men’s Clothing section, a good deal of link value will be passed to that page.

However, what happens if those 30 links get divvied-up across three different versions of that page, and are split between Men’s Clothing, Footwear, and Outdoors? Potentially, each page would only get 33 percent of that total link value. Clearly, 100 percent would be better.

Determining which page is the authority, and then directing all of your inbound links there, provides more benefit than trying to divide and conquer. Doing so could mean the difference between being found at the top of the search results and receiving a sales windfall, or being sunk to the depths of the results and getting nothing at all.

Who’s the captain of this ship?

But how do you decide which page is the authority? Staying with our example, should it be the one in the Men’s Clothing section? Or the one in Footwear, or the one in Outdoors?

To decide, first assess the competitiveness of the term you’re targeting. If it’s not very competitive and you believe you can get two similar—but not identical—versions of a page to rank for that term, you can try to optimize both pages equally. I’m not advocating that you attempt to get as many versions of a page into the search results as possible, but there are occasions where it might make sense to go after two versions.

For instance, let’s say you have two pages about hiking boots. One is the general product page, and the other is a promotional page for a limited time offer on the boots. Obviously, you want to optimize the promo page so anyone looking for special offers on hiking boots would find the page with the limited time offer. However, the general product page about hiking boots should also be optimized for people looking for hiking boots outside of any promotional periods.

In most cases though, it’s tough enough to get one page ranking for a term. The best course of action is to decide which version provides the greatest opportunity for revenue over the long term. Then optimize that page, direct all inbound links to it, and block any other versions from being indexed. Remember, there’s no reason to head into a storm if you can sail around it.

While at times you may feel stranded amidst a sea of content that you just can’t seem to make work, don’t resort to tactics that could blow your efforts off course. Avoid creating multiple versions of the same page just to score more listings in the search results. Instead, differentiate your content by tailoring messages to speak to a particular deal or offer, and concentrate your linking efforts on sending value to the page with the power to produce revenue for the long run.

Dave Feldman is director of client services for search engine marketing firm iProspect and can be reached at dave.feldman@iprospect.com. The Brand Aid column appears Wednesdays at Search Engine Land.

The Yahoo Search Blog announced a new Site Explorer feature that enables webmasters to inform Yahoo about dynamic URLs that might be creating duplicate content issues or crawl barriers.

The new feature is named “Dynamic URL Rewriting,” and it allows webmasters to view potential duplicate content issues and have Yahoo rewrite those URLs at the index level. Yahoo allows you to define which URL is the primary, and the remaining URLs will be dynamically rewritten to that primary URL.

One of the most discussed advanced topics in SEO is duplicate content. Duplicate content can often be generated by accident by having automated dynamic URLs. How so? The URLs can append characters to the URLs such as session ids, tracking parameters, format modifiers and other parameters that do not change the content of the page but yet create a new unique looking URL. Savvy SEOs detect these duplicate URLs and use rewriting technology to inform the search engine of the primary URL. But sometimes webmasters forget.

This new tool will enable webmasters to easily identify these URLs and classify the primary versions of the URLs.

For example, say you set up a tracking URL for a banner ad such as searchengineland.com/?ad_id=1234, which is basically a duplicate URL for searchengineland.com. You can tell Yahoo that anything that conforms to the “ad_id” parameter following the ? will by a duplicate URL to the primary and Yahoo will rewrite the URL to default to the primary.

Here is a screen capture of the tool:

As you can see, Yahoo gives you two options with the tool

(1) Remove these parameters from the URLs, such as in case of session ids, you could ask to remove ’sid’ from URLs (2) Use a default value for the parameter, for example you could set the ’src’ parameter to be ‘yhoo_srch’

It currently seems you can apply up to three parameters for the rewriting or removal in this tool.

Yahoo has done a great job explaining how it works at the Yahoo Search Blog and over at Yahoo Site Explorer help.

I admit it. I am a search engine geek. Because I have a passion for understanding search usability, one of my particular interests is duplicate content filtering. If you want to really irritate searchers, present the same content to them in all or most of the top 10 positions in search results.

In the past, before search engines became effective at name results clustering, many search engine optimization (SEO) professionals, including myself, considered it quite the accomplishment to help client sites appear in the majority of the top 30 search results. I remember when one of my client sites held 24 of the top 30 positions. He thought I was the greatest invention since the light bulb. However, having analyzed the search data and Web analytics data, I realized that having all of those top positions did not necessarily mean top conversions. So I was happy to see how search engines are becoming increasingly effective at filtering out duplicate content.

At the SMX Advanced conference in June 2007, there were a few takeaways that I thought were very important for SEO professionals to keep in mind: multiple duplicate content filters, and knowing when to apply 301 redirects.

Multiple Filters

One common misconception about duplicate content filtering is that there is only one main duplicate filter. In fact, there are multiple duplicate filters, and they are applied throughout the three main parts of the search engine process:

• Spidering or crawling
• Indexing
• Query processing

Some duplicate content filters weed out content before Web pages are added to the index, meaning that some duplicate content will not be displayed in search results. A Web page cannot rank until it is in a search engine index; therefore, crawl-time filters can actually exclude URLs from being added to the search engine index.

Some duplicate content filters are applied after pages are added to the search engine index. Web pages are available to rank, but they might not display in search engine results pages (SERPs) as Web site owners might like them to appear. For example, no one wants their content to appear in the dreaded Supplemental Index.

Another common misconception is that if a listing appears in Google’s Supplemental Index, the site has been penalized. Duplicate content does not cause a site to be placed in the Supplemental Index. From Vanessa Fox’s blog:

If you have pages that are duplicates or very similar, then your backlinks are likely distributed among those pages, so your PageRank may be more diluted than if you had one consolidated page that all the backlinks pointed to. And lower PageRank may cause pages to be supplemental.

And from Matt Cutts’ blog:

Having urls in the supplemental results doesn’t mean that you have some sort of penalty at all; the main determinant of whether a url is in our main web index or in the supplemental index is PageRank. If you used to have pages in our main web index and now they’re in the supplemental results, a good hypothesis is that we might not be counting links to your pages with the same weight as we have in the past. The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit).

301 redirects vs. robots exclusion

Remember when meta-tag content used to be the "secret weapon" to getting top rankings in Infoseek? Lately, I feel that search engine optimization professionals feel that 301 redirects are the secret weapon to getting and preserving link development, especially when redundant/duplicate content is involved.

For those of you who do not know what a 301 redirect is, I like to use this analogy. Have any of you ever moved and had to fill out those change of address cards at the post office? Basically, when you fill out these change of address cards, you are telling the U.S. postal service that your address has moved permanently to a new address.  I like to think of a 301 is a change of address card for computers. The status code is telling search engines that the content at a specific URL (Web address) has permanently moved to another URL.

There are times when using 301 redirects are appropriate and times when it is not appropriate. For example, let’s use a home page. The following home page URLs typically lead to the same content:

• www.companyname.com
• companyname.com/index.htm
• www.companyname.com/default.cfm

In this situation, it is best to implement a 301 redirect so that the most appropriate URL will lead the home page content. Search engines utilize canonicalization, which is the process of selecting the most appropriate URL when there are several choices. Be pro-active. Don’t let the search engines determine the most appropriate URL to crawl and to display in search results. As the Web site owner, you should select the URL that is best for your business and target audience.

Implementing 301 redirects is not the solution for every instance of duplicate content, in spite of what many SEO professionals might claim. The robots exclusion protocol is often far more appropriate.

Here is an example. Suppose a Web site owner has purchased and implemented a new content management system (CMS), and, as a result, the URL structure changed. During the site redesign, the Web site owner has eliminated content that has not converted well or is outdated.  Should the Web site owner implement 301 redirects for the eliminated content?

Many SEO professionals often state that 301 redirects should be implemented to preserve the "link juice" to the expired content. In this situation, if a searcher clicks on a link to the expired content, he/she will typically be redirected to the home page. How does this benefit the search experience? The searcher expects to be delivered to specific content. Instead, he/she is redirected to a home page to begin searching for the desired content. It is a futile process, as the content has been removed. The result is a negative search experience and a negative user experience.

If content is removed, then delivering a custom 404 page is more appropriate, in spite of the "link juice" theory.

Conclusion

Search usability is not a term that is only applicable to Web search engines. Search usability does not only address querying behavior. It also addresses other search behaviors (browsing, scanning, etc.) Duplicate content delivery often has a negative impact on a site’s overall search usability, before site visitors arrive at your site and after they arrive. By understanding how the commercial Web search engines filter out and display duplicate content, Web site owners can obtain greater search engine visiblity and a better user experience.

Shari Thurow is SEO Director for Omni Marketing Interactive. The 100% Organic column appears Thursdays at Search Engine Land.