I have written in the past about the differences between A/B and Multivariate Testing and how to choose between them. Back then, I wrote that one of the advantages of multivariate tests is that you can analyze each and every interaction between elements in a much deeper level.

However, one of the points that I missed is that A/B Tests are easier to be analyzed using Web Analytics tools. Since every test version is a page, they are being tracked by default in your analytics reports. This is not true for Multivariate Tests since the versions are combinations of multiple elements, which are not served as a page but brought into the page by external tools.

In this article, I will go over a technique that can be used in order to make the most out of your Google Analytics implementation so that you can understand the bigger picture of your test (independently of which testing tool you are using).

Analyzing A/B Tests Results Beyond Success & Failure Metrics

Usually, when performing an A/B Test, marketers implement the codes on the original page, on the test page and on the conversion page. This means that the testing tool will track the number of visitors to each of these pages and report back the percentages of visitors viewing each page and their conversion rates. This approach misses two important issues:

1. Beyond Success And Failure Metrics: some websites (mainly ecommerce) may have completely different values to conversions, meaning that tracking success and failure as a binary decision might be misleading. As George Orwell wrote on the Animal Farm: “All animals are equal, but some animals are more equal than others.”
2. Engagement Metrics: sometimes the objective of the test may not be to reach a goal page, but to decrease the bounce rate, to increase time on page, or pages per visit.

Since very often the information above is already available in Web Analytics tools, it is logical to integrate testing results into them. So, if you are using Google Website Optimizer, Visual Website Optimizer or other tool of your choice, here is a way to improve the measurability of your tests using Google Analytics.

Setting Up Google Analytics to Track A/B Test Properly

Google Analytics content reports do not show a tab for goal conversion per page. That is understandable, since it is not possible to attribute conversions to specific content. Google’s solution was to create the $ index. Here is this metric’s description as provided by Google:

$ Index is the average value for a page that a user visited before landing on the goal page or completing an Ecommerce transaction (or both). This value is intended to give you an idea of which page in your site contributed more to your site’s revenue. If the page wasn’t involved in an ecommerce transaction for your website in any way, then the $ Index for that page will be $0 since the page was never visited in a session where a transaction occurred.

Since we are discussing A/B tests, by default you will be able to analyze the $ index of each of your test variations (if you have ecommerce this will be defined according to it; if not, be sure to set your goal values when setting your goals on Google Analytics).

But if you want more detail to be collected regarding your test variations, you can also use Custom Variables. Basically, to set a custom variable is to add a value to the user cookie that will persist across the life of the visitor’s cookie. This means that this additional segmentation capability will allow us to compare between pages, both when it comes to engagement metrics and when it comes to conversion metrics.

Here is a sample of the Google Analytics code that should be added to each test page:

body onload="_gaq.push(['_setCustomVar', 1, 'abtest', 'variation2', 1]);"

On the example above, we see the following:

1. The custom variable is set to slot 1 (out of 5): this important as you might overwrite other variables if they share the same slot.
2. The name of the variable is “abtest”
3. The value is “variation2″: each variation should have its own value. If possible, there should be an explanatory name in order to facilitate analysis, e.g. “big_picture”, “long_form” or “wonderful_graph”.
4. The scope is 1 (visitor level): this should be set to 1 as you want your returning visitors to still have the variable attached to them. If you use 2 or 3 that won’t happen.

Below is a sample of which metrics you will be able to use to analyze your A/B Tests.

Integrated A/B Tests

As Michal Nassimian wrote on A/B Testing for SEO: “Search Marketing should be viewed as a conglomerate of variables, where SEO, paid campaigns and Web Analytics are intertwined. Each effort to optimize a website should strengthen all others to increase synergy and boost revenues.”

I believe that by integrating Tests into Analytics tools we will be one step closer to an integrated optimization strategy.

Google sent out an email to Google Website Optimizer users saying:

Dear Website Optimiser user,

We are writing to inform you of a potential security issue with Website Optimiser. By exploiting a vulnerability in the Website Optimiser Control Script, an attacker might be able to execute malicious code on your site using a Cross-Site Scripting (XSS) attack. This attack can only take place if a website or browser has already been compromised by a separate attack. While the immediate probability of this attack is low, we urge you to take action to protect your site.

We have fixed the bug, and all new experiments are not susceptible. However, any experiments you are currently running need to be updated to fix the bug on your site. Additionally, if you have any Website Optimiser scripts from paused or stopped experiments created before 3 December 2010, you will need to remove or update that code as well.

There are two ways to update your code. You can either stop current experiments, remove the old scripts and create a new experiment, or you can update the code on your site directly. We strongly recommend creating a new experiment as it is the simpler method.

The email goes on to give specific examples on how to modify your current experiments to make sure you do not have malicious code on your site.

At the end of the email, Google apologized, saying:

We’re committed to keeping Website Optimiser secure, and we’re deeply sorry for this issue. We will continue to work hard to prevent future vulnerabilities.

Yours sincerely, Trevor Google Website Optimiser Team

For more information on this XSS issue, see Dave Naylor’s blog.

On the Google URL shortener issue, Twitter is taking care of it and I don’t believe Google has anything to do with it. TechCrunch has a comment from a Twitter representative that reads, “We’re aware and have sent out password resets for affected users. We’ll monitor the situation in case of further iterations.”

Postscript: Google has a new post on this issue at the Google Website Optimizer bug.

While this seems to me to be a fairly accurate description of the process, it’s also important to note what conversion optimization isn’t.

• It isn’t making a website “better/look nicer/rank better”
• It isn’t concerned with current customers. In fact some of the processes can stand at odds with best practices for retaining current customers, and we should be mindful of this.
• Last but not least, it isn’t mindless chopping up of website content and structure “for the sake of it.”

The important thing to take into account when beginning a conversion optimization project is that the planning is at least if not more important than the test itself.

Before running a conversion optimization project we need to make sure we’re barking up the right tree. This first part looks how to decide which pages to think about optimizing. Next month we’ll look at some further cheap/free techniques to get more information from the behavior of your users.

Important metrics: bounces & exits

Starting with your excellently deployed analytics package, let’s look at which reports are useful to us in determining the focus of our conversion optimization process.

First, take a look at pages with high bounce rates/exit rates. That’s an easy one—pages with high bounce rates are bad and need to be optimized right? Well, maybe. I would suggest some segmentation here wouldn’t go amiss. Remember, we’re looking predominantly at new visitors here, the ones which haven’t been to your site before and need that extra helping hand to get them further involved. Try using advanced segmentation to convert more new users.

So, how do the high bounce rate pages look now? Select pages with bounce/exit rates which are significantly higher than the site average and then ask yourself:

• Do these pages have a significant proportion of website traffic?
• Are they important in the path to conversion?
• Are the traffic sources to these pages good quality and relevant to the content?

Pick no more than two distinct pages to follow up, look at the navigation report to understand where users are going to from this page. Once you’ve answered these questions you should have a couple of pages (or groups of pages) to analyse in more depth.

Advanced funnel analysis

Regardless of the pages you identified in the previous step, it’s almost certainly worth analyzing the pages which are directly involved in the conversion process, whether they be a lead generation conversion or an ecommerce sale. This involves tracking every page of the funnel, but also creating multiple funnels to analyse users coming from different areas of the website. You can do this by making required first steps of different pages, for example product or category pages.

This requires your analytics package to allow you to create multiple goals. I often find that there are subtle differences in the way users interact with different site categories or content types. Once you’ve got your funnels set up, collect data and analyse which funnel pages are causing users to exit the process. One word of caution here—keep in mind what a true funnel “exit” is here. To understand what I mean here, take a look at my colleague’s analysis of the true Google Analytics funnel post.

Listen to your users

Anecdote time: I recently carried out a project for a website which had at least 2 years worth of data. The site was receiving 12M+ pageviews per month and over 700,000 monthly searches. Yet the company was completely unaware that the site had 150,000 searches per month for “iphones” which resulted in an empty results page (and a huge search exit rate). If your website has a site search function, hopefully you are mining this data already. If not, get it set up!

At the risk of sounding cheesy, I always say it’s like having a conversation with your customers. If you listen carefully you’ll hear them. Sometimes it’s a whisper; sometimes it’s a sign that they’re confident of using a site search function and don’t mind typing. At other times though, they’re screaming at you. They’re angry; they took time to find your site, they invested time in browsing your navigation and come up dry. But, in searching again, they’re giving you a second chance. This time show them what they want to see. First concentrate on finding the screaming, angry users.

Which pages aren’t satisfying users simply with their content and navigation?

These pages are the ones which are most searched from. Once you’ve identified them, try to understand whether or not users are sticking with the searches or leaving the site. If there is a high search exit page from these pages then they should be a focus of your efforts. Some questions to ask:

• Do I need to add content to this page?
• Is the look and feel of this page incongruous to the content?
• Am I providing an obvious “next step” to my users?
• Are certain products/content less prominent than they should be?
• Do I need to make certain information more easily available?

Finally, after studying the stats, forget the stats. Look at each page under suspicion and think critically about what you should expect from this page. What is its purpose? Does it make sense for it to have relatively poor stats? Is there a logical reason or should you be concerned?

Take a decision

By now, you should have two or three pages or types of page which you have singled out for treatment. They should have the following attributes:

• Poor bounce/exit rate
• Low contribution to the conversion process
• Other signs such as high number of searches and search exits
• No reason, intuitively, to have this poor performance

The next step is to look at which aspects of these pages need improvement. That’s the focus of my post; brining the site search back into the arena along with some other interesting technologies in order to create priorities for optimization A/B & multivariate testing.

Google Website Optimizer. In October 2006 Google announced the release of its website testing tool, Google Website Optimizer. While the tool is not as powerful as some of its commercial counterparts, it allowed the average smaller to mid-size website to integrate some sort of A/B and multivariate testing into their marketing initiatives. After working with hundreds of clients on many different software platforms, I can say with high confidence that Google Website Optimizer satisfies the needs of 90% of websites out there.

The bad economy. As the economy tanked in 2009, many online marketers started to pay close attention to new ways to convert their existing traffic into revenue. So, while net visitors still mattered, conversions mattered a lot more.

Of course the concept of conversion is not unique to the web. Print media has been using the terms response rate and conversion ratios for over forty years. Most snail or junk mail campaigns convert at a rate of 1% or less. And the truth is that most websites report conversion rates close to those of junk mail. Data reported by Shop.org reflects a continuous decline in online conversion rates. In June of 2007, Fireclick index reported an average ecommerce conversion rate of just 2.2%.

In 2007, the majority of Fortune 500 companies allocated less than 5% of their online marketing budgets to conversion optimization. Based on the current trends, I expect that by 2012 most companies will spend close to 15% of their online marketing budgets on conversion optimization. These numbers are not where they should be, but nonetheless, that is a significant increase.

And while it is great that many companies are finally paying attention to conversion optimization, there is a new reality many companies are starting to deal with. Based on talking to nearly 200 companies who tried conversion optimization in the last year, over 60% of them report no significant improvement in conversion rates. This data should concern anyone who is considering conversion rate optimization (CRO) as well as companies considering offering it to their clients.

There are many reasons why these types of projects do not produce the intended results.

Client expectations are not set correctly from the start. Conversion optimization projects require clients to invest time and resources to make modifications to their websites. While some optimization companies try to get around this issue by implementing changes themselves, this approach does not scale. It might work if you are a small company dealing with few clients. In the last count, we provided recommendations to clients on over 40 different platforms. Can you imagine the amount of resources we have to maintain to support all the different clients?

Starting points aren’t obvious. While attempting to encourage companies to start conversion optimization, too many consultants play down or simply do not understand that value of creating a conversion optimization roadmap. By a roadmap, I mean a plan for which pages or processes should be optimized first. Simply saying “start with your pages with highest bounce rate, or pages with the highest exit rate” oversimplifies the planning stage, and in many cases will result in failure. Creating an optimization roadmap requires careful analysis of analytics and website personas.

The guessing game. Assuming the consultant uses a good methodology to create a conversion optimization roadmap, the next big question is what changes you should make in the target areas. Give a usability or conversion consultant one screen, and they can give you hundreds of changes to that screen. But how do you know which of these changes will work? The typical answer is to test any changes. I think that misses the main point. Testing ten or twenty changes on a page is a good thing. But what about the other hundred-or-so possible changes? Are you going to test them all? Do you even have enough traffic to test these changes? At this point in the discussion, many will fall back to best practices. They will tell you that best practices will guide which elements to test. And yes, best practices might help, but it does leave you with a large vulnerability.

The fallacy of best practices. I do not discount the value of best practices in conversion optimization. However, I take issue with companies who make them the cornerstone of their practice. The reason? For as many usability best practices that exist, there are as many exceptions. I think those who disagree with my assertion simply have never run into these exceptions, for one reason or another.

Conversion optimization is harder than SEO. In SEO you deal with search engine algorithms. While you are not privy to how these algorithms work, experts deduce their logic by making changes and observing how a search engine responds. We follow the same approach in conversion optimization, by making changes and trying to observe how visitors respond. The problem is that humans are a lot more complex than search engines. Humans tend to act in a non-linear and complex fashion.

The more complex your website, the more pronounced each of the above problems will be. Some companies can guarantee a 15% percent uplift in conversion rates for a single landing page, but they would not be able to achieve the same results with an entire ecommerce website. My goal is not to scare you from considering conversion optimization, because the results can be quite profound. But before you jump into conversion optimization, you must be aware of some of the challenges you will face. I will address how to mitigate each of the above risks in later posts.

This post will present a fast and easy way to get your behavioral targeting (BT) process up and running. Although any discussion of BT can sometimes turn into a PhD-level conversation, it is really important to pluck some low-hanging fruit to convince executives that it is a worthwhile journey. If you do, your path will likely be paved with gold.

Recently, BTBuckets, a free behavioral targeting and segmentation tool, launched two brilliant plugins to integrate their tool with Google Website Optimizer and Google Analytics. These free tools are a boon for anyone who seeks to target the highest quality audience to a website.

Targeting users from within Google Analytics

The first plugin, for Google Analytics (GA), enables users to create behavioral targeting campaigns from within GA for a series of reports (see report details in the BTBuckets plugin page).

Say, for example, you post a press release on your corporate blog describing a very important new feature of a key product. Naturally, you get many incoming links from blogs and news websites. However, bbc.co.uk links to your blog home page instead of linking to the feature description post itself. During the time the post is at the top of the blog home everything is great, and your traffic converts pretty well.

However, after you publish a few additional posts, you note on Google Analytics that your conversion rates for visitors from the BBC (which still sends a good amount of traffic) drops drastically. This is most certainly a consequence of visitors not finding what they were looking for. One way to deal with that is to create a redirect for traffic arriving from BBC to the blog home and lead these visitors to the specific post (see explanation on video below). This can now be done from within GA with the help of this plugin.

Click here to view the embedded video.

Test and target using Google Website Optimizer

The second plugin, for Google Website Optimizer, enables users to segment visitors that will be included in a test. It is important to note that the plugin works for specific user segments including new visitors, returning visitors, search and social media.

For example, suppose you are a publisher, and display AdSense advertisements. This is a great way to earn a few bucks out of your website; however, is it a good strategy to show ads to all your visitors? Perhaps, perhaps not. If someone is a first time visitor, it is probably better to show a special promotion, or maybe a banner for an internal campaign, or a signup form for an email subscription or an RSS feed. A returning visitor is more likely to know your brand and is likely to return even if he or she clicks on a contextual ad. For these visitors, instead of showing a signup form, it’s fine to display AdSense ads.

Since you can never know if that is the right thing to do without testing, you should create two tests, one for new visitors and one for returning visitors. One test displays AdSense ads for 50% of visitors, and the other displays an internal banner to the other half. If you happen to find a significant difference, bingo! If not, keep testing and targeting.

Below is a video showing how to use the plugin, and more info about it can be found on BTBuckets plugin page.

Click here to view the embedded video.

Now we have a suite of tools to analyze, test and target for free. Are you doing behavioral targeting? If so, please share your successes and challenges in the comments section below.

The example I want to discuss today is segmenting your tests to see how they perform for different kinds of users of your site. I’m going to show you a few ways of gathering GWO data in Google Analytics (GA) to bring the full power of the GA platform to your GWO testing.

A classic example is testing the impact of changes to the call-to-action for new vs. returning users. These two segments of users often have very different requirements and we can see why they might react differently to different enticements. For someone who knows your brand and is familiar with your offer, a discount could be supremely effective while a new visitor might respond better to long copy explaining the benefits (this is pure speculation and is designed only to serve as an example). For sites that get high numbers of returning visits (e.g. eBay), returning visitors are a self-selected set of people who understand how to use the control and/or have struggled through the learning curve. They may be resistant to change and prefer the control despite there being room for improvement for new visitors who haven’t gone through that learning curve.

Other useful segmentations during conversion rate tests include:

• Traffic source (e.g. organic search vs. paid search vs. referral traffic)
• Country/location
• Whether a user has visited a specific page on the site or in the conversion funnel
• And a whole load of other things

There are three basic steps to getting the data you need:

1. Ensure that you are able to track GWO variables in GA
2. Set up appropriate filters and avoid contaminating your existing profiles and data
3. Analyse the data

#1 Track GWO Variables

Google Analytics has a (relatively little known) function called utmx to return basic information about a GWO test to analytics. By tracking pageviews that include information in the URL about the test and variant being run, you can set up a profile within GA that has an understanding of conversion rate tests. There is more information on how to do that in this quite technical guide. It may also be useful to read this alternative approach from ROI Revolution that uses a custom ga.js file.

Tracking a pageview such as /my/page.html?gwo_exp=1234567890&gwo_var=0 gives all the information needed to create a profile that stores the variant of the test in a custom variable.

#2 Avoid contaminating your main profile

I suggest implementing the tracking of GWO variables in a new profile. In your core profile, you will probably want to strip out this information so that you can see all visitors of a single page concept together (i.e. not splitting out by which variant they saw).

In this example, you would want a filter such as (from the guide referenced above):

• In “Field A -> Extract A”, select Request URI and (^.*)(.gwo_exp.*)
• In “Output To -> Constructor”, select Request URI and $A1

#3 Analyse the output

One of the nicest things about GWO is its ability to abstract away the maths behind determining winners. While that functionality is obviously still available for the aggregate test (on the whole set of users) there is an additional step required to work out whether you are seeing statistically-significant results on segments of your users.

Although learning the statistics behind these calculations is fun(warning: your experience may differ), we are primarily looking to get to the answer here. For this I recommend using an online sample size calculator. In order to analyse, for example, whether your test has reached statistical significance on new users only, you would segment the data to show only new users and find the number of users in this segment who had seen each variant and the number of each of those who had converted (for whatever conversion goal you are seeking).

With this data in hand, you can fill in the fields in your chosen calculator and, see the relative conversion rates. For most tests, you would seek a 95% confidence that a variant was better than the control. Consider the following data:

These results suggest that you would be happy declaring variant B better than the control (confidence > 97%) but even though variant A has a higher conversion rate than the control at present, you would not be confident declaring it better yet.

What I’d like to have shown is real-world examples of the ways that tests can show misleading things when you look at the average data when digging into the segmentation reveals insights about different classes of user. Unfortunately, I don’t have any public examples. If anyone has real-world examples they can share in the comments, I’m sure the SEL community would love to discuss them.

The one constant in search engine marketing is change. Things that worked a few months or even years ago don’t work today. Conversely, things that work today maybe didn’t work 2 months ago. Learning what is best for you and your site takes a lot of patience, research and testing.

If you’re a do-it-yourself website owner—or run your own company and do your online marketing in-house, using a tool like Google Website Optimizer can help you test different ideas and theories with ease. Testing your landing pages and conversion funnels can help you craft a site that is user friendly and more conversion friendly. Whether your goal is to sell an item, or generate a lead, simple steps to set up and implement a test can help you gain insight into what your audience is looking for.

Visiting Google.com/WebsiteOptimizer is an easy—and free—way to easily get started with a great tool that’s easy and intuitive to use. If you can design a new page and paste a bit of code into your website, you don’t need any help at all to get started. If you’re a design novice and have someone help you out, have them help design your test page and then it’s literally less than a half-hour of effort to install the scripts, provided there aren’t any special steps needed to make them work.

Tip: If you use WordPress, install the Easy Google Optimizer Plug-in for WordPress from ImpressionEngineeers.com. This makes setting up your campaign a snap!

I also like to recommend Website Optimizer if you’re testing out some targeting and segmentation ideas, like the ones I put out in my last column “How to Target your SEO Landing Page’s Demographic Profile.” The reality is sometimes segmentation and targeting takes many many versions to dial in the right one. Being able to A/B test your original against a new version is a great way to see if the change should be permanent without guesswork.

The nice thing about the Website Optimizer interface is the step by step setup instructions. There is a “shopping list” of what you need before you start your test, and then a step by step Q&A to help you decide what type of test you need, what your conversion goal page should be and then the code is generated… voila!

Here are some screenshots to get you familiar with what you’re doing. This will take you through to the code generation portion of your test:

Step 1: Create a new experiment

Step 2: Choose the type of experiment

Step 3: Name and identify your control, test and conversion pages

Step 4: Tell Google how your scripts will be installed

Step 5: Copy and paste the code into your site and validate your pages

The most important rule to remember is to allow enough conversions for an accurate test. Google recommends around 100 conversions for an accurate measure. Some pages can reach that goal in a day, while some may take a month or more. If you have a site that receives a few conversions a week or month, you can still test. Consider making your “goal” a clickthru to the next page in the conversion path instead of a full blown conversion.

If you’re not sure where to start with your first test, some things I would suggest are:

Conversion button style and placement. Whether you’re looking for a “book online” or “contact us” type of conversions, work on how that message appears to your visitors. If it’s a text link buried in a paragraph, separate it out and give it a colorful, eye-catching button to entice more conversions. Use your testing to design the right button for your audience.

Navigation overhaul.. Is having left-hand navigation the best or should you move it to the right, can you eliminate some navigation to encourage the conversion action? Is the order of your navigation encouraging or discouraging engagement and conversion? Do some testing to figure it out.

Page layout. The middle left side of your page is a dead spot. Are you putting important information there? Try rearranging your text and important links to encourage more time on site and more clickthrus.

Virtual tour or image conversion optimization. If you use virtual tours or image galleries, try different techniques to layover or include links to help the visitor into your conversion path. These pages can rank well, so try to use them as much as you can.

Getting started with your testing can be intimidating, but the first test is definitely a milestone you should hit in January 2010! The results you see and the potential become addicting. Remember, a key part of the overall SEO process is to take time to work on not only the rankings and traffic to your site, but the nuts and bolts that make your site work for visitors.

Below is a comparison between the testing techniques mentioned, taking into consideration the overall use of the testing technique, coding needs, design needs, granularity of results and other considerations.

Most testing tools provide these options, but since Google Website Optimizer is a free tool that provides both options, it is a good place to start and try the examples I provide below.

A/B Test

An A/B test is the most common and easiest type of landing page test to conduct. It consists of creating alternative pages for a specific page and showing each of them to a certain percentage of visitors. For example, if you create 4 different variations of a landing page, 20% of visitors to the website will see each version (4 variations + original). Cookies are used to maintain a consistent user experience—if a visitor sees one version, they will see it again and again when visiting the website as long as the cookies are not deleted. Below is a representation of how this technique works.

Image created by Yam Designs. For a high res version of the image go to A/B test scheme.

To implement the test with Google Website Optimizer, scripts need to be included on the pages to be tested. These include:

• Two JavaScript codes on the original page: one that performs a redirect to the additional variations (head of the page) and one that measures the number of times the page was seen (this can be placed anywhere below the redirect code).
• One JavaScript code on each variation page to measure the number of visitors viewing each page.
• One JavaScript code on the conversion page to measure which visitors converted; this will measure the success of each page variation.

Advantages of A/B tests

Design freedom. A/B tests are often used to experiment page design options that vary dramatically, including position of text and pictures, background colors, number of pictures on the page, use of icons and navigation structure. Implementing such tests using the multivariate technique is possible, but it is technically challenging (but if you really want to do it, and you are technically savvy, see this post on the Google Website Optimizer Tricks blog.

Less JavaScript coding. as described above, the codes necessary to implement an A/B test are very simple and can be added to the website in a matter of minutes.

Faster results. A/B tests usually involve fewer combinations with more extreme changes; multivariate tests involve many more combinations and variations. In addition, since A/Bs show significantly different designs, the expected improvement of the page is usually higher, diminishing the time the test will run.

Multivariate test

Rather than testing different versions of web pages, as we do with A/B tests, Multivariate tests experiment with elements inside one specific page (for purists, we are referring to full factorial experiments, which is the method used by most testing tools). Basically, we define elements inside a page (e.g. a picture, a text or a button) and provide different alternatives of each element. The testing tool will show each element combined with all other elements to visitors. The resulting combinations are derived from the number of elements multiplied by the number of element variations. Just as with A/B testing, however, each visitor sees only one particular combination of elements regardless of how many times they view a page. Below is a representation of how this technique works.

Image created by Yam Designs. For a high res version of the image go to Multivariate test scheme.

In terms of coding, the programming a multivariate test is slightly more complex than a simpler A/B test. A few pieces of JavaScript code need to be implemented: one opening the test, one for each tested element and one closing the test. In addition, a JavaScript will be added to the conversion page to measure combination success.

Advantages of multivariate tests

Granularity of results. Since it is a full factorial experiment, multivariate tests show which elements are the best performing separately, as well as the correlation between the elements. This can be very useful when projecting the results to other parts of the website.

No redirects required. Since all elements tested are inside the page, there is no need to redirect from the original page to the tested pages. Although redirects can be performed smoothly, I believe it is better not to use them whenever possible, as they can slow the flow and affect user experience.

Fewer design resources required. Since we will be testing different designs with existing elements on a page, this will not require too much design effort.

Concluding, both types of testing have their own advantages and disadvantages. Each can be a perfect technique, depending on the needs of the website. They should always go hand-in-hand, using one to test completely different designs and the other to optimize the current design. The important thing is to understand that testing is not a one-time effort It is an ongoing exercise that should be part of the mindset of an organization. As Avinash Kaushik once wrote in his blog, Experiment or go home!

And that’s a mistake. Good testing can not only help you get rid of stinker ads and landing pages, it can also help you optimize winning campaigns, improving their conversion rates and making them even more profitable. And testing needn’t be difficult or time-consuming, especially if you have the right tools. One of the best tools for helping you test and refine your search marketing campaigns is Google’s Website Optimizer. Even better, Website Optimizer is a free service.

Google deliberately designed Website Optimizer to be easy to use, even though the tool allows you to run complex, multivariate tests. At its most basic level, the tool lets you test different pages or combinations of elements on a single page to see how well they perform against a “conversion outcome” that you define. Google even claims that “you can launch a simple test in five minutes” in the video tour of Website Optimizer.

The problem is that to be effective, tests should be carefully planned to measure your desired outcomes. Another problem is that the results of your testing need to be interpreted properly so that you can be sure that any implementation based on the results will perform as expected. Testing is as much art as science, and it shouldn’t be done in an offhand or cavalier fashion.

Not sure where or how to start? Try Always Be Testing: The Complete Guide to Google Website Optimizer, by Bryan Eisenberg & John Quarto-vonTivadar with Lisa T. Davis. It’s a comprehensive guide to Google’s free testing tool, written for search marketers who are more interested in enhancing the performance of their search marketing campaign than in the analytic joys of multivariate testing. The authors don’t shy away from technical explanations when necessary, but the focus is always on the “whys” of testing even when looking closely at the “hows.”

The book is divided into three parts. The first part is a general overview of website testing and tuning using Google’s Website Optimizer. The first few chapters focus on the tool itself, but after that the focus changes to case studies drawn from the authors’ own search marketing campaigns. These are valuable not only as examples of how to test, but because the authors also share the processes and best practices they have developed to maximize the impact of the testing process.

Part two “What You Should Test” looks at more than 30 key factors that can impact conversion rates. These factors differ based on how conversion is defined, which also differs for publishers, retailers and lead generation web sites. Most of the chapters in this part follow a similar format, with sections on “questions to ask,” an exercise, “what to test” and “apply this to your site” tips. This is one of the best—and most thorough—guides to the overall website testing and optimization process I’ve read.

Part three, “Diving Deep For the Technically Challenged” goes into even more detail about the testing process, how things work, behind the scenes details of how Website Optimizer works and more. Meaty stuff for the more advanced search marketer who wants to know everything to gain a competitive edge.

The nice thing about Google’s Website Optimizer, apart from being free, is that it can be used to test campaigns running on Google or any other search engine. And the general practices discussed in the book can be applied to any testing tool, not just Website Optimizer.

Always Be Testing is an excellent book that tackles one of the most important aspects of successful search marketing. If you aren’t currently testing and want to know how to start, buy the book. Even if you are already testing and tuning your sites as part of your search marketing efforts, the case studies and best practices discussed in this book make it worth far more than the cover price when you apply the principles to your own campaigns.

Always Be Testing: The Complete Guide to Google Website Optimizer
by Bryan Eisenberg & John Quarto-vonTivadar with Lisa T. Davis
ISBN-13: 978-0470290637
Sybex, $29.99

Now, you can use Website Optimizer to not just A/B test your AdWords landing pages, but also test your Yahoo landing pages, your banner landing pages, and different elements throughout your web site to improve conversions and desired actions from your users.

If you already use Website Optimizer within your AdWords account, you can continue to do so. The data and tool will remain in your AdWords account and can continue to be used there for the foreseeable future. The new Website Optimizer will have its own user interface but will function a lot like the old Website Optimizer.

In addition, Google has launched a blog for Website Optimizer named the Website Optimizer Blog. The blog will contain “up-to-date product news, industry insights, testing strategies, and other educational information from the Google Website Optimizer team and our partners,” said Google.

Google Website Optimizer is available in 27 languages including Czech, Danish, Dutch, English (Australia, UK & U.S.), Finnish, French, German, Hebrew, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese (Brazil & Portugal), Russian, Spanish, Swedish, Tagalog, Thai, and Turkish.

Update: In other Google web tracking news, Google updated us that Urchin 6 is now out of beta. Urchin is the paid version of Google Analytics and runs directly on your own server. Also, Actual Metrics is offering free demonstrations of Urchin 6 if you are interested.