Google Website Optimizer. In October 2006 Google announced the release of its website testing tool, Google Website Optimizer. While the tool is not as powerful as some of its commercial counterparts, it allowed the average smaller to mid-size website to integrate some sort of A/B and multivariate testing into their marketing initiatives. After working with hundreds of clients on many different software platforms, I can say with high confidence that Google Website Optimizer satisfies the needs of 90% of websites out there.

The bad economy. As the economy tanked in 2009, many online marketers started to pay close attention to new ways to convert their existing traffic into revenue. So, while net visitors still mattered, conversions mattered a lot more.

Of course the concept of conversion is not unique to the web. Print media has been using the terms response rate and conversion ratios for over forty years. Most snail or junk mail campaigns convert at a rate of 1% or less. And the truth is that most websites report conversion rates close to those of junk mail. Data reported by Shop.org reflects a continuous decline in online conversion rates. In June of 2007, Fireclick index reported an average ecommerce conversion rate of just 2.2%.

In 2007, the majority of Fortune 500 companies allocated less than 5% of their online marketing budgets to conversion optimization. Based on the current trends, I expect that by 2012 most companies will spend close to 15% of their online marketing budgets on conversion optimization. These numbers are not where they should be, but nonetheless, that is a significant increase.

And while it is great that many companies are finally paying attention to conversion optimization, there is a new reality many companies are starting to deal with. Based on talking to nearly 200 companies who tried conversion optimization in the last year, over 60% of them report no significant improvement in conversion rates. This data should concern anyone who is considering conversion rate optimization (CRO) as well as companies considering offering it to their clients.

There are many reasons why these types of projects do not produce the intended results.

Client expectations are not set correctly from the start. Conversion optimization projects require clients to invest time and resources to make modifications to their websites. While some optimization companies try to get around this issue by implementing changes themselves, this approach does not scale. It might work if you are a small company dealing with few clients. In the last count, we provided recommendations to clients on over 40 different platforms. Can you imagine the amount of resources we have to maintain to support all the different clients?

Starting points aren’t obvious. While attempting to encourage companies to start conversion optimization, too many consultants play down or simply do not understand that value of creating a conversion optimization roadmap. By a roadmap, I mean a plan for which pages or processes should be optimized first. Simply saying “start with your pages with highest bounce rate, or pages with the highest exit rate” oversimplifies the planning stage, and in many cases will result in failure. Creating an optimization roadmap requires careful analysis of analytics and website personas.

The guessing game. Assuming the consultant uses a good methodology to create a conversion optimization roadmap, the next big question is what changes you should make in the target areas. Give a usability or conversion consultant one screen, and they can give you hundreds of changes to that screen. But how do you know which of these changes will work? The typical answer is to test any changes. I think that misses the main point. Testing ten or twenty changes on a page is a good thing. But what about the other hundred-or-so possible changes? Are you going to test them all? Do you even have enough traffic to test these changes? At this point in the discussion, many will fall back to best practices. They will tell you that best practices will guide which elements to test. And yes, best practices might help, but it does leave you with a large vulnerability.

The fallacy of best practices. I do not discount the value of best practices in conversion optimization. However, I take issue with companies who make them the cornerstone of their practice. The reason? For as many usability best practices that exist, there are as many exceptions. I think those who disagree with my assertion simply have never run into these exceptions, for one reason or another.

Conversion optimization is harder than SEO. In SEO you deal with search engine algorithms. While you are not privy to how these algorithms work, experts deduce their logic by making changes and observing how a search engine responds. We follow the same approach in conversion optimization, by making changes and trying to observe how visitors respond. The problem is that humans are a lot more complex than search engines. Humans tend to act in a non-linear and complex fashion.

The more complex your website, the more pronounced each of the above problems will be. Some companies can guarantee a 15% percent uplift in conversion rates for a single landing page, but they would not be able to achieve the same results with an entire ecommerce website. My goal is not to scare you from considering conversion optimization, because the results can be quite profound. But before you jump into conversion optimization, you must be aware of some of the challenges you will face. I will address how to mitigate each of the above risks in later posts.

Using the wrong type of test (A/B vs. multivariate test)

A/B tests allow you to test two versions (or more) of an entire page against each other to see which one works best. If you are testing three different versions of a page, then you are conducting an A/B/C test and so on. A/B tests are especially useful because they allow you to test major design decisions by putting two or three completely different designs against each other to find out which one of these works best with your visitors.

Multivariate tests (MVT) allow you to test multiple elements of a single page at the same time. So you are able to test different headlines, different images, or different colors on a single page. Although it may seem like testing different elements on one page is less complex than testing one or more pages against each other, the opposite is true. You can think of A/B tests as simplified versions of multivariate tests.

One of the first questions you must ask yourself is whether you should start with an A/B or multivariate test for a particular page. And while multivariate tests are indeed powerful, they require considerably larger number of visitors to the test page for you to reach a decisive conclusion.

So, should you go with A/B or multivariate testing for your particular case? There is no universal answer. If you are starting out with a brand new landing page or have limited number of visitors, we usually recommend conducting an A/B experiment to start out. However, if you already have an existing page, we might conduct a small multivariate test (less than 12 or so different scenarios). The goal of this type of test is to learn which elements have the highest impact on conversion. Based on the results we learn from that MVT, the team will decide to conduct further MVT or A/B tests.

A successful test will take into account the number of visitors and number of conversions on a page (not the site). Although your site may have 30,000 visitors a month, your test page may have as few as 500 visitors and one conversion per month.

A/B tests are good for alternate designs of an entire page, whereas multivariate tests are helpful in determining the most successful elements at different locations on a page.

Testing too many elements

In the case against multivariate testing, I discussed the perils of letting multivariate testing software do the thinking instead of using an online marketer’s judgment, intuition and persuasion to guide the testing process.

There are plenty of software solutions out there which allow you to test tens of thousands of combinations on a single page—there is software that actually allows you to test millions of combinations. The problem with testing so many variations is the amount of time and resources it will take to conduct such tests.

And yes, some websites have sufficient traffic that they feel testing tens of thousands of combinations is just fine.

I disagree. There is no art in testing millions of scenarios and hoping that one of them will convert better than the current page you have. I’ve found that testing fewer than 100 scenarios can increase conversion rates from 4% to 15% in a matter of a few months. It’s not about testing only, it’s taking a common-sense, holistic approach to the process testing that leads to improved results.

Running tests that take too long

There are two kinds of factors that can have an impact on your conversion rate. There are internal factors that you can control, such as your design, messaging, copy, etc. But there are also external factors over which you have little control. If your competitor is running a 50% discount sale, you should expect your conversion rates to suffer. If the economy gets worse, you can expect your conversion rates to suffer as well.

One of the dangers of conducting testing is not accounting for the impact of these external factors. There is no real way of controlling them; however, there are few things you can do to minimize their impact:

• Keep track of external factors and understand that they might have an impact on your results. Being aware and not being sidetracked by a plummeting economy is probably a good idea.
• Minimize the time it takes to run your experiments. We do not advise conducting tests that take months to complete. I usually like to see tests completed in four weeks at the most. If that is not possible, then go back to the drawing board and rethink your test scenarios.
• Stay on top of what your competitors are doing. There are several tools that allow you to “spy” on competitor activity, including whether or not they are employing testing or some type of conversion optimization. I’ll write about those tools in a later article.

Failing to monitor your test as it takes place

Conventional wisdom states that you can set up an experiment, start the test and just sit around and wait for the results. That works well in many cases. But there are times where you might want to change an experiment in mid-stream. Closely monitor experiments to decide if you should eliminate some elements when there is enough evidence that they are not producing the results you are looking for.

This point is especially applicable when you’re using Google’s website optimizer as your testing tool. Website Optimizer is a great testing tool but running experiments on it may take too long. Now, you must be careful, seeing that one scenario is not producing any results after only 20 or so visitors does not mean anything. You also have to know when it is the right to time to interfere.

Failing to conduct follow-up experiments

Let’s say that you conducted a test, and as a result you were able to double your conversion rates. But then you stopped testing. That is a typical story.

We had a client who started with a conversion rate of around 3%. After we did the initial redesign and testing, we were able to increase their conversion rates to 9.8%. Our clients was so pleased with the results, they decided to suspend the campaign they already paid for.

It took about another month to convince them that they should continue testing and that there is the potential to increase conversion rates even further. Designing follow up experiments, and learning which elements worked and which elements did not, is at the heart of conversion optimization. To make a long story short, the client was able to increase their conversion rates to 14.9% with just two more months of testing.

Thinking that testing is a silver bullet

Testing is a great tool to increase your conversion rates. It is also an important step in any conversion rate optimization process you conduct. However, it is only one step in the process. It is a critical step that should happen at the end of conversion optimization projects. If you are looking for double digit conversion rates, then you must start with understanding your target market through the process of persona creation, site analysis, analytics assessments and design and copy creation. Trying to test random scenarios without doing the initial homework is like throwing darts in the dark.

To recap:

• Select the right type of test to conduct based on the number of visitors and other data collected from you analytics.
• Limit the number of scenarios through a holistic approach to testing by considering market segment, persona development, trust factors, fear factors, etc.
• Don’t be side-tracked by external factors that impact your conversion rate. Stay on top of how the economy is performing and what your competitors are doing so you can test and optimize accordingly.
• Stay involved in the tests you are conducting. Don’t minimize the importance of following up and tweaking test elements as results become statistically significant.
• Don’t stop once you see improvements. Get beyond single-digit improvement by continuing to test.
• Approach any project from a number of different angles, because testing alone will not maximize your results.