The power of UGC reviews

In recent years, multichannel retailers like Walmart, Best Buy and many others have embraced user-generated product reviews as an effective social media tool to increase conversion rates. The benefit is that in exchange for giving customers a forum to share their product experience (positive or negative), the merchant receives unbiased “voice of the customer” content that helps sell prospective customers. (And be honest, you have to respect a brand like Cabela’s willing to allow comments like “this thing sucks” live on the website.) All kidding aside, brands need to consider the risks with these systems and have a strategy to achieve high customer adoption. But in nearly every case we’ve encountered, the process works very well.

The SEO issue for most retailers is that their UGC product review functionality tends to rely on AJAX, iframes or sub-domains for presenting the “voice of customer” review content. These methods make it difficult or impossible for engines to match up the rich user-generated content with the actual product pages themselves—eliminating nearly all of the potential SEO value. I’ve encountered this issue frequently over the years in helping retailers increase their organic search performance through our product which optimizes such landing page content for SEO benefits.

So anecdotally, we had seen consumers adding descriptive phrases and shorthand to client pages as (crawler-friendly) UGC , which Google matched on when other consumers searched for the same phrases—apparently increasing long-tail organic search traffic at no marginal cost. For instance, one multichannel merchant’s “under armour hooded sweatshirt” landing page gets top rankings for about a half dozen derivatives of “UA Hoodie”—not because the merchant, or the Under Armour brand, use that shorthand on the page; this page gets found because end-consumers added the “UA hoodie” phrase to the page through their UGC product reviews. And that’s one phrase. Multiply by dozens of reviews per page. Across thousands of pages. Refreshed frequently… It all adds up to goodness, right?

Some providers of this UGC review functionality argue for keeping UGC reviews off-page as a benefit, possibly because their technology tends to rely on AJAX for easier implementations. Some SEOs also favor this approach, as it theoretically preserves page keyword density and prevents theme dilution. But our hypothesis was that UGC may in fact be more useful embedded on-page to grow SEO results organically over time.

We know the task of researching keywords is time-consuming, and a potentially infinite task for a large site. We also know optimizing pages for those phrases can take not only political willpower but months or years to actually prioritize and execute. So if outsourcing this content optimization to customers for free, through UGC, extends your brand’s reach up the funnel, acquiring the very people searching the web for your type of product, we would have a brilliant strategy. Unless adding this content to the page does in fact dilute keyword density and ultimately harms performance!

Research results provide some clues

We approached the answer to this difficult question using a three step process aimed at developing a data-driven point of view.

First, we studied the SEO effects of UGC across 10,000 unique client product pages over a 30 day period. Roughly 33% of these had UGC reviews embedded on the page, 66% did not. What we found was that product pages with embedded reviews were crawled as much as 200% more frequently, with as much as 250% broader keyword reach, and as much as 200% more organic traffic. That was impressive on the surface, but not necessarily a causal relationship.

To understand causality, we applied a suppression technique to the UGC reviews on a test segment of pages. Essentially we made the reviews invisible to search engine spiders. We compared the performance of these pages against a control group over 60 day period. The results after 60 days of suppression were clear: Removing the UGC reviews from the test segment pages caused significant performance degradation. The average quantity of referring search phrases to each page (”keyword reach”) decreased by 50%, and average non-brand organic traffic dropped by 31% per page during the 60 day period.

To complete the process, we reversed the suppression technique at day 60 by re-embedding the UGC reviews on the page. 30 days later (day 90 of the trial), the average keyword reach of each page had rebounded to pre-suppression levels (offsetting the 50% loss during suppression). Meanwhile non-brand organic traffic also rebounded to pre-suppression levels, and even exceeded them by an additional 12% .

Our conclusion

Based on these side-by-size page comparisons, we believe it is reasonable to conclude that product pages with UGC review content embedded as HTML (as opposed to AJAX, pop-ups, subdomains or other) do indeed cause a page to receive greater crawl frequency, nearly twice as many referring search phrases, and two times as much non-brand organic traffic, as product pages without review content embedded on the page.

Does this mean you should implement UGC purely for SEO benefits? Probably not. But the data makes a compelling case that UGC can act as a form of free keyword research, and create opportunity for large-scale advertisers to smartly outsource the “optimization” of their organic landing pages to the very people looking to find them—your customers.

Incremental traffic and revenue

What you consider incremental revenue directly impacts your ROI calculation. Trust me—this is a metric worth fighting over. You need to frame and drive the business conversation around it. This requires a strong understanding of SEO performance metrics like keyword reach, keyword placement, and landing page yield, plus understanding revenue quality metrics and awareness of any search attribution problems.

Executives require your natural search channel make more money this month than last, and on a year-over-year basis. But your landing pages are impacted by many stakeholders and factors—internal and external. So before allowing executives to judge the incremental traffic and sales alone, be sure to communicate changes in the volume of keywords, the volume of traffic and sales driven by non-brand keywords, and your average Google keyword placement.

For example, you may find yourself in the situation where traffic or revenue is flat or down on expectations, after having invested in SEO. The CFO/CMO will conclude natural search warrants no further investment. But there may be more to the story than meets the eye. Maybe a major site-wide change was introduced, like a new categorization system or a faceted navigation scheme. You may find that landing page yield, keyword reach and keyword placement all decreased dramatically against the prior period. So in actuality, it’s a minor miracle that performance was just flat—perhaps offsetting dramatic growth experienced in other areas of the site. These metrics enable you to constructively communicate both progress made, as well as any loss experienced by other stakeholders or factors.

Talk with your analytics package provider about whether your natural search sales are getting the proper attribution they deserve. Since branded keyword navigation through paid search is a key end-user technique for reaching your site, make sure your package is not just giving paid search credit for sales that natural search made possible. If it is, paid search will continue to command more resources than natural search in your organization, and you will continue getting what you’ve always got.

Last, be sure to highlight to your CMO the quality of your natural search revenue against other acquisition channels. For example, contrast your conversion rates across natural, paid and direct-load visitors. Natural search will typically compare favorably against paid search—even in spite of the branded keyword navigation bias of paid search. If your company tracks new to file customers, natural search is likely to be your biggest source—and probably your most profitable. Ideally, also compare the lifetime value of searchers against other channels as well.

Calculating natural search return on investment (ROI)

Once you’ve determined your true incremental revenue, its time to calculate return on investment. What has your cost of revenue been over the same period of time? You need to add up a few line items here:

• External costs, including SEO training, consulting, and implementation services
• Internal cost estimates of IT, marketing, copywriting, project management time

If incremental revenue is $500,000 and incremental spend was $250,000, a simple ROI is 2:1. Of course you may need to fine-tune the cost metric to account for SEO work deployed over a prior time period that’s paying dividends in the current period or vice versa. For example, you may “amortize” the cost of developing new institutional processes, such merchandisers or copywriters that now consult keyword research tools before naming products or writing copy. Conversely, you may need to account for time-to-market as a missed-opportunity cost if it takes six months let’s say, to execute tactics that you “purchased” in a prior period.

In review

The business case we calculated through this series goes like this: Our addressable market spans 200,000 unique keyword markets and totals roughly 5,000,000 searches per month. We’re capturing 1% currently. The cost of acquiring that traffic through PPC totals $1,000,000 per month. Of our 200,000 addressable keyword markets, we are reaching 1,000. We rank on Page 1 of Google in 70% of those markets. We attract 2.5 searchers per keyword. 95% of our search traffic comes from queries for our company name. And 97.5% of our pages attract no search visitors.

Trying to communicate your performance and market opportunity in this disciplined manner has probably felt torturous. But the benefits to you as a marketer and organization are enormous. You can now let the data be your guide towards growth. The scenario we have been calculating screams opportunity to executives. Let’s role play this from an executive perspective: You’re telling me that nearly all my search traffic is coming from my brand name? I’m visible in fewer than 1% of my non-brand markets? I’m getting only two searchers per keyword? Only 2% of my pages are attracting searchers? What does it take to double any of those metrics? I can connect any of those dots to revenue. I want to hire internal/external SEO who understands how to make that happen and I will incent them to do so. What does it cost me? As long as the ROI is better than my alternatives, I’m in.

This is a metrics-driven business conversation. This process will help you win the time and monetary resources to dramatically increase search performance. Having the technology and skills to execute, of course, is another story.

Through the process we’re now able to tell the CFO something like this: “Our addressable market spans 200,000 unique keyword markets and totals roughly 5,000,000 searches per month, of which we’re capturing 1% currently. The cost of acquiring that traffic through PPC totals $1,000,000 per month.”

That paints a compelling (perhaps rosy) picture. But what’s driving your current performance? Why do you believe you can do better? In this installment, I share three more metrics that can you help create strong baselines, set realistic expectations, and direct your campaign towards the most effective outcome.

Keyword reach

So you’re getting 50,000 searchers from natural search a month (1% CTR). But is that good or not? It depends. Are a handful of search phrases sending 99% of the traffic? Or is traffic distributed evenly across thousands of long-tail phrases? The Keyword reach metric helps you quantify the breadth of your presence to better manage it.

To understand keyword reach, start with the number of unique search phrases that drive natural search traffic to your site currently. What volume of traffic does each phrase drive? Your analytics package likely contains this basic data. But before we get excited, let’s apply a few filters.

First, differentiate brand phrases from non-brand phrases in your calculation. Why? Searches for your brand or company name are just table stakes. Don’t let yourself feel good about them. What you care about increasing is your share of the non-brand markets. Once you’ve separated these, determine the percentage of your traffic driven from brand vs. non-brand queries. Many retailers, for instance, find that as much as 95% of their natural search traffic comes from brand vs. just 5% from non-brand phrases. This confirms a big opportunity.

Now that you’ve distinguished between brand and non-brand traffic phrases, you can calculate your traffic per keyword yield rate on your non-brand phrases. If 5% of your search traffic (2,500 searchers) is from 1,000 non-brand phrases, your yield rate would be 2.5 searchers per keyword (2,500 searchers / 1,000 phrases). If your optimization is successful at increasing rankings in these 1,000 keyword markets, you’d expect to see the keyword yield increasing as well, perhaps doubling to 5.0 searchers per keyword.

For benchmarking purposes, you should also slice search phrase data by engine. This helps you avoid being misled by high level trends, and will show you if your optimization work is helping your Bing performance, at the expense of your Google performance, for example.

Last, make sure you can determine whether keyword reach is increasing or decreasing on a comparable basis, such as month-over-month (MoM) or year-over-year (YoY). This way you can measure the impact of your optimization work going forward in an objective manner.

Understanding keyword reach lets us augment our business case with some more powerful information: “Of our 200,000 addressable keyword markets, we are reaching only 1,000. These keyword markets drive 2.5 searchers/keyword. 95% of our search traffic comes from queries for [company name].”

Keyword placement

How highly do you rank in those 1,000 non-brand keyword markets that drive traffic? Can you do better?

We all know there are scraping tools to capture rankings. Rather than worry about ranking in markets no one searches for, and that you get no traffic for, what if you could determine where you rank for the thousands of phrases that did drive traffic? There are tools that tell you this by parsing the referral strings on engine traffic. Each major engine thankfully passes an argument in the URL which tells you what page of the SERPS (although not the actual position—yet) the searcher clicked on to land on your page. Naturally you want to filter these by engine.

Armed with this information, you can identify which phrases are ranking on Page1 (P1) of Google, Yahoo, or Bing. You can use this of course to talk up your SEO ninja skills in front of your CEO. More importantly, you can identify which phrases are ranked on P2 or P3 etc, in each engine. Perhaps you’re ranked on Google’s Page 1 in 70% of the non-brand markets. 20% of the time you’re on P2. You can begin to imagine an SEO campaign aimed at these markets, in order to get your ranking onto P1, thereby increasing CTR and traffic by many multiples.

The keyword placement metric helps you further enhance your business case: “Of our 200,000 addressable keyword markets, we are reaching only 1,000. We rank on page 1 of Google in 70% on these keyword markets, driving 2.5 searchers/keyword. 95% of our search traffic comes from queries for [company name].”

Landing page yield

Natural search is a marketing funnel: at the top are your supply of pages which get crawled. Some percentage of those get indexed. A percentage of the pages indexed get ranked. A percentage of those get clicked, etc. So an important way to baseline current performance is to map your search traffic against the pages that drive the traffic. Surprisingly, this is more difficult than it seems. In fact, it is highly likely that you do not know how many pages are on your dynamic site (don’t worry—I won’t tell).

The easiest way we find to approximate site size is to measure the number of unique pages crawled by a particular engine over a period of time. Engine crawl has slowed recently. Review your log files for unique URLs requested by Googlebot over a 60 day time frame. Or you can run a crawler over your site. Either way, you should be able to parse highly duplicated pages from the count. For our purposes, let’s assume there were 10,000 unique URLs requested by Googlebot over the past 60 days. In other words, Google thinks your site has 10,000 pages.

Naturally, you want to compare this to how many pages Google reports having indexed. If you have just 1,000 indexed, you should investigate why 90% of the available landing page “inventory” you take to market is being rejected, and then develop strategies to address the issue.

You need to understand how many of your URLs drove search traffic. Suppose 250 pages are driving your 2,500 non-brand monthly searchers. That means 250 of your 1,000 indexed pages are performing—a rate of 25% yielding 10 searchers per page. However, you have 10,000 total pages, of which 250 are performing—an actual yield rate of 2.5%. If these were your metrics, you should have many questions:

• Why are 90% of my pages not being indexed?
• Why are 75% of my indexed pages not being found at all?
• How can I identify my non-performing pages?
• How can my 25% yielding pages achieve more than 10 searchers per page?

Asking and answering these questions leads to smarter strategies and more accountable performance.

Your business case now looks like this: “Our addressable market spans 200,000 unique keyword markets and totals roughly 5,000,000 searches per month, of which we’re capturing 1% currently. The cost of acquiring that traffic through PPC totals $1,000,000 per month. Of our 200,000 addressable keyword markets, we are reaching only 1,000. We rank on Page 1 of Google’s in 70% on these keyword markets, driving 2.5 searchers/keyword. 95% of our search traffic comes from queries for [company name]. 97.5% of our pages drive no search traffic.”

Remember, you can only manage what you measure. With these metrics, you can create performance baselines and develop data-driven SEO strategies aimed at achieving specific performance outcomes. Trust in the force, Luke.

Define the market opportunity

First, what’s the total size of market you’re chasing in natural search? Yeah, I see your blank stare. This number has to be your mission; it must be top of mind in order to successfully convince executives to invest in SEO. Think of this first metric as a basic marketing feasibility study. The SEO costs and tactics are irrelevant at this stage. Tell me how many people are searching for what you sell every month. What keywords do they use—besides your brand name. This exercise is different from traditional “keyword research.” We’re looking for an estimate of market demand.

To estimate keyword market demand, first look at the phrases currently driving traffic. Export (from your site analytics) all the non-brand phrases that drove natural search traffic over the last completed month. Augment this list with all your PPC phrases that outperform your average site conversion. You can further augment the list by running a spider over your site to extract onpage phrases that may be worth exploring, but this is a more advanced step and is probably counter-productive at this stage. (Trust me: your business case will be compelling enough by the time we’re done. Resist the temptation to inflate it into unrealistic fantasy data that will only hurt your credibility later.)

Historically the tools needed to conduct our feasibility study have simply been unavailable, or we were forced to make keyword research tools do a job they weren’t meant for. Thankfully, Google made its AdWords Keyword Tool, with demand data, accessible for advertisers over a year ago. Hopefully Yahoo and Bing will follow. For now, simply feed your keyword list into the AdWords tool over the comparable time frame. You can query using broad or exact match, but I’d suggest exact match to calculate more precisely how much of that market you penetrated.

A few caveats here: First, Google’s numbers are imperfect, but better than the alternatives. We find that Google will fail to report useful data for many long-tail keywords that have small demand. This is an issue since long-tail keywords are the essence of what big natural search programs are made of (another tool limitation, for the time-being). Second, we suspect the demand numbers Google reports are overstated by a factor of between 20% – 40%. So you can discount accordingly to keep a clean, consistent Google-only calculation through your business case.

At the end of this process, you want a spreadsheet with the number of keyword phrases and the demand of those markets, with a total market size that you are competing in, across natural and paid search. You have defined your market opportunity when you can confidently tell your executives something like this: “Our addressable market spans 200,000 different keyword markets, and totals roughly 5,000,000 searches per month.”

Estimate click through rate (CTR)

The next question asked will be, “But how much of that are we getting right now?” To answer this, determine what percentage of those markets your current keyword traffic represents. This is an easy and powerful calculation. You know what non-brand traffic you received from Google during the same time period (from your site analytics). Simply divide the market opportunity by actual traffic received for each keyword to determine your CTR for each keyword. You should calculate CTR on an individual keyword-level basis. For bottom-line purposes, calculate a weighted average of the individual keyword CTRs.

You have defined your CTR when you can now tell your executives something like this: “Our addressable market spans 200,000 unique keyword markets and totals roughly 5,000,000 searches per month, of which we’re currently capturing 1%.”

(Pause the movie a second. Did you catch what just happened? With just two metrics you now have the power to compare the efficacy and potential of natural search against other forms of online advertising using familiar performance metrics—without resorting to ineffective faith-based arguments or resource-begging. It gets better. Unpause…)

For added power, insert a column in your spreadsheet showing where your site ranks for each phrase (SERP page is fine for now). As you might expect, you’ll find that phrases for which you’re ranked on Page 1 of Google (here I’ll be using shorthand like P1, P2, etc.) will have a higher CTR than phrases for which you are ranked deeper. Now compare CTR on P1-ranked phrases against P2-ranked phrases, etc. Our retail clients find Google P1 consistently achieves CTR of around 10%, while P2 achieves around half the CTR (<5%). If you see similar values, the business proposition is crystal clear: There are performance gains to be had by investing in phrases ranked deeper than P1. We all know this instinctively, but now the data can do the talking for you. Note: There are tools that report on SERP rankings automatically. Or you can parse referral strings yourself on all engine traffic to capture what page of the SERPS the searcher clicked on to land on your page.

If you let it, the data can say something even more powerful. Once you quantify the performance difference between your P1 and P2 (and deeper) phrases, you can assign and model a budget value to achieving the desired lift across the desired markets—along with some probability assumptions. Don’t be surprised if that budget is much more than you’re budgeting for all your SEO efforts right now. You’ll recall my premise: your SEO is underperforming now for this very reason.

Map out traffic acquisition cost

Now for the trifecta: Knowing your addressable market size and current penetration, your executives want to know two things: How much are you spending on that performance now, and how much does “more” cost. You’re in the game. The costs of advertising via paid search continue to increase. The cost of advertising through natural search is dirt cheap. Your executives want to increase profit by acquiring more traffic (and sales) at less cost. Natural search is an obvious way to increase traffic, decrease costs and increase profit, right?

You can easily answer the “cost of more” question by exporting the average CPC value reported by Google’s AdWords Keyword Tool above. Simply add two columns to your spreadsheet, one to reflect a growth assumption on your current natural search traffic (50%, 100%, whatever), the other to estimate what it would cost to acquire that traffic using AdWords. Total it up. You’ve just created a reality-based framework for valuing natural search that executives will appreciate – even though it may total more than what you’re currently budgeting to spend on SEO.

Your message to the CFO now looks like this: “Our addressable market spans 200,000 unique keyword markets and totals roughly 5,000,000 searches per month, of which we’re capturing 1% currently. The cost of acquiring that traffic through PPC totals $1,000,000 per month.”

More can be said on this point in future installments. But notice that you’re no longer taking Quixotic approaches towards SEO. Instead you’re making a business proposition based on markets with known performance and that have real costs associated. Your data is doing the talking for you, and your CFO’s heart is warming up to you. Powerful, the force in you is, yes…

Next time I’ll discuss the remaining metrics that help measure incremental gains in performance.

I hope to help. The development of more robust search metrics has been a passion for me over the past 8 years in the field and in working with premier online brands at Netconcepts. I love measuring things. Looking for patterns. Questioning. Concluding. Experimenting. Learning. Then doing it all over (and developing products that automate those processes). I very much view natural search as a form of direct marketing. One my professional aspirations is to help elevate natural search from that of “black art” to a genuine business discipline capable of producing sustainable results more predictably (OK, I have ambitious tendencies). And as my company and colleagues would attest, I am a firm believer in the old management truism “you can only successfully manage what you have successfully measured.”

So what I want to offer you in my regular contribution to this column is simply this: How to systematically improve your natural search performance. More specifically, how metrics can help you win the resources needed to grow your channel; how to calculate them; which metrics matter; how to shape them into powerful business conversations; how to evaluate the smorgasbord of SEO tactics available to you, and their effectiveness at driving performance for you. When your CEO knocks on your door demanding you double your search performance in 6 months, I want you to know how to assess the feasibility and requirements of achieving that objective—or at the very least, how to tell your CEO what is more attainable and why.

Let’s get started.

The natural search opportunity

Allow me to humor you with some industry facts:

• Search, as an industry, is growing from a $12B industry in 2007 to a $25B industry in 2011—100% growth in just 4 years (SEMPO)
• CPC costs are on the rise—Google’s average CPC rose 15% last year
• Consumers click on natural listings 85% of the time according to research (from Forrester)
• Meanwhile marketers spend 85% of their search budgets on paid search (according to Marketing Sherpa)
• 56% of Google queries have 0 paid ads placed (each of those queries present 10 “non-paid” or natural search results (or for the forward thinking, “ads”) on the results page) (according to comScore)

Clearly, natural search presents marketers with a much bigger pool of potential consumers—each of which can be acquired much more profitably than by any other means.

But as practitioners, how do we normally communicate this opportunity? Most search geeks (I mean that lovingly and inclusively) default to tactic talk and irrelevant metrics: begging for an hour of IT or copywriting or mechanizing to modify HTML elements on templates, H1 tags, page titles, or add links, or rewrite a URL or two. Then you get shot down because when pressed for “what’s it worth,” you start fumbling for something impressive to say (”um… thousands! Maybe millions” are gems I’ve heard over the years). And when that fails you resort to “leap-of-faith” arguments that CFO’s just don’t buy – or at least not for very much money. The standard SEO metrics you do share have no connection with business—you know what I mean. Things like PageRank flow, inbound links, keyword density, engine distribution, ego term rankings.

And that’s the nub of our issue. Winning the Darwinian resource struggle requires not that we lose our passion or faith in search, but that we realize it; and I am saying that we realize it by first talking business metrics before we talk implementation tactics.

Show me the money

Ultimately if we want resources allocated towards SEO work, we must be able to communicate the value of that work in return on investment (ROI) or return on ad spend (ROAS) language. That seems simple enough. Yet the reality of measuring natural search is complicated by the long-tail and distributed nature of natural search.

How do you define what is actually incremental traffic or sales? One month you get three visits for a long tail phrase. Next month you get just one. Did market demand decrease? Or did your rankings decrease? Or did both stay constant while other SERP page competition stole your traffic? And practically speaking, how can you possibly care about such small volumes? You don’t worry about the height or width of each blade of grass in your lawn (I hope). Yet successful search channels are comprised of just that: hundreds of thousands (or millions) of non-brand phrases each acquiring handfuls of searchers every month for various pages of your website.

In addition to the scale issue, many factors impact natural search performance, and many stakeholders are likely influencing what the engines are seeing and responding to. Can you correlate gains or losses to specific site changes? Or is your performance driven by an invisible hand of unnoticeable activities—perhaps the combined effort of user interface enhancements and keyword-rich copywriting tempered by new AJAX navigation schemes, all of which may have been deployed at various times over the past quarter or year. It’s complicated!

So our challenge is this: not only must we speak in ROI/ROAS business language, but we must also understand what’s driving performance currently, what the level of effort put forth has been, and communicate how we expect actions to contribute towards that positive ROI/ROAS we see.

That said, here is a framework of progressive metrics we have developed that take many of these factors into account in order to drive a stronger natural search business conversation:

• Market opportunity
• Clickthrough rate
• Traffic acquisition cost
• Keyword coverage
• Non-brand reach
• Landing page placement
• Landing page yield
• Incremental traffic / revenue
• Return on investment / return on ad spend

In my next few installments, I will walk through each metric, how we derive them, their relationship to each other, and how to use them in a practical manner.

In recent years, many retailers have implemented powerful attribute-based (a.k.a. faceted) navigation systems that make it easier for customers to filter, sort, navigate, and buy from a retailer’s inventory. Most of Netconcepts’ clientele who have implemented these technologies report good results, increased conversion and sales. From the natural search perspective, many such marketers are pleasantly surprised to find increases in the number of their pages indexed by Google upon implementing such systems. But in this case, what’s good for users, is actually not so good for search engines — or for your bottom line.

Recently, Google Webmaster Central took a position on this by pronouncing the “infinite filtering” and “resulting page duplication” produced by such guided navigation systems as negative for bots, and urged that the “blech” of duplicated pages be cleaned up, by saying:

“Another common scenario is websites which provide for filtering a set of search results in many ways. A shopping site might allow for finding clothing items by filtering on category, price, color, brand, style, etc. The number of possible combinations of filters can grow exponentially. This can produce thousands of URLs, all finding some subset of the items sold. This may be convenient for your users, but is not so helpful for the Googlebot, which just wants to find everything – once!

….One fix is to eliminate whole categories of dynamically generated links using your robots.txt file… Another option is to block those problematic links with a “nofollow” link attribute.”

While useful for consumers, the attribute filters also produce unique URL parameters based on elements like product colors, sizes, price, popularity, the number of results per page, and the pagination construct itself, resulting in dozens or hundreds of permutations of unique crawlable URLs. But these unique URLs contain content that is completely similar and duplicated in nearly all respects. And duplicate content of this magnitude hurts your natural search marketing performance.

Here’s an example from the Artful Home brand: Compare this page to this page. Unique URL addresses, but same content. Click on the “price range” or “color” or “medium” attributes in the left navigation and you’ll get even more URLs with the same content. In fact Google has indexed 20 URL permutations containing the exact same content for this one page. Same title tags, headings, body copy, and more.

Below are five natural search marketing performance issues introduced by attribute-based retail navigation models (as typically implemented):

1. You end up creating self-competition between your site’s own pages. (If you were Googlebot, which of these do you consider the authority page?)
2. Your pages don’t resonate with searchers. (People seldom include attributes like price in their search queries)
3. You bloat the engine index. (You don’t have as many pages indexed as you’re bragging about to your CEO.)
4. You burn your “crawl equity.” (More of your unimportant pages get crawled with each visit a bot makes.)
5. You fragment your available link popularity (PageRank) between so many different versions of the same/similar pages.

To deal with the situation, merchants with attribute-based navigation systems need more sophisticated strategies and execution capabilities.

For example, the on-page attributes that create the navigation scheme should be researched during the design phase to ensure your resulting landing page themes are consistent with searcher vocabulary. Filtering options that are not beneficial to your natural search performance should be similarly evaluated, and tactics put in place that make any “false” landing pages uncrawlable for bots. A combination of nofollows, meta noindexing and disallows strategies should be employed for this. This is easier said than done as most merchants require IT resources to modify the site architecture in this way.

(If you’re in this boat and expending any serious amount of IT effort is not an option, you may wish to consider a proxy site technology such as GravityStream to generate and apply such rules-based changes simply and automatically across your large scale site. (Disclaimer: I manage the GravityStream product at Netconcepts.) One benefit of this approach is that you can, without involving your IT department, automatically resolve the many parameter-filled URL permutations created by such navigation systems into singular, authoritative versions of each category, subcategory, and product level URL — a form of “canonicalization” via intelligent redirection. By eliminating page duplication in this fashion, you maximize the distribution of your available link popularity to the greatest number of unique landing pages.)

Google’s position on this matter may seem to contradict their Webmaster Quality Guideline around “making your site for users not search engines.” After all, attribute-based navigation systems do offer many great user benefits. But they will unquestionably dilute your natural search marketing performance if not engineered with search engines in mind. Today’s savvy merchant is already implementing sophisticated strategies to capture the best of both worlds.