In a previous post, I wrote about how to get started with website testing, both choosing which pages to test and how to define which elements will contribute the most to profits. However, there are quite a few testing methods to choose from. In this post I will delve into the two most common testing methods: A/B tests and multivariate tests (MVT). What is the difference between them? How can you choose which one best fits your needs?

Below is a comparison between the testing techniques mentioned, taking into consideration the overall use of the testing technique, coding needs, design needs, granularity of results and other considerations.

Most testing tools provide these options, but since Google Website Optimizer is a free tool that provides both options, it is a good place to start and try the examples I provide below.

A/B Test

An A/B test is the most common and easiest type of landing page test to conduct. It consists of creating alternative pages for a specific page and showing each of them to a certain percentage of visitors. For example, if you create 4 different variations of a landing page, 20% of visitors to the website will see each version (4 variations + original). Cookies are used to maintain a consistent user experience—if a visitor sees one version, they will see it again and again when visiting the website as long as the cookies are not deleted. Below is a representation of how this technique works.

Image created by Yam Designs. For a high res version of the image go to A/B test scheme.

To implement the test with Google Website Optimizer, scripts need to be included on the pages to be tested. These include:

• Two JavaScript codes on the original page: one that performs a redirect to the additional variations (head of the page) and one that measures the number of times the page was seen (this can be placed anywhere below the redirect code).
• One JavaScript code on each variation page to measure the number of visitors viewing each page.
• One JavaScript code on the conversion page to measure which visitors converted; this will measure the success of each page variation.

Advantages of A/B tests

Design freedom. A/B tests are often used to experiment page design options that vary dramatically, including position of text and pictures, background colors, number of pictures on the page, use of icons and navigation structure. Implementing such tests using the multivariate technique is possible, but it is technically challenging (but if you really want to do it, and you are technically savvy, see this post on the Google Website Optimizer Tricks blog.

Less JavaScript coding. as described above, the codes necessary to implement an A/B test are very simple and can be added to the website in a matter of minutes.

Faster results. A/B tests usually involve fewer combinations with more extreme changes; multivariate tests involve many more combinations and variations. In addition, since A/Bs show significantly different designs, the expected improvement of the page is usually higher, diminishing the time the test will run.

Multivariate test

Rather than testing different versions of web pages, as we do with A/B tests, Multivariate tests experiment with elements inside one specific page (for purists, we are referring to full factorial experiments, which is the method used by most testing tools). Basically, we define elements inside a page (e.g. a picture, a text or a button) and provide different alternatives of each element. The testing tool will show each element combined with all other elements to visitors. The resulting combinations are derived from the number of elements multiplied by the number of element variations. Just as with A/B testing, however, each visitor sees only one particular combination of elements regardless of how many times they view a page. Below is a representation of how this technique works.

Image created by Yam Designs. For a high res version of the image go to Multivariate test scheme.

In terms of coding, the programming a multivariate test is slightly more complex than a simpler A/B test. A few pieces of JavaScript code need to be implemented: one opening the test, one for each tested element and one closing the test. In addition, a JavaScript will be added to the conversion page to measure combination success.

Advantages of multivariate tests

Granularity of results. Since it is a full factorial experiment, multivariate tests show which elements are the best performing separately, as well as the correlation between the elements. This can be very useful when projecting the results to other parts of the website.

No redirects required. Since all elements tested are inside the page, there is no need to redirect from the original page to the tested pages. Although redirects can be performed smoothly, I believe it is better not to use them whenever possible, as they can slow the flow and affect user experience.

Fewer design resources required. Since we will be testing different designs with existing elements on a page, this will not require too much design effort.

Concluding, both types of testing have their own advantages and disadvantages. Each can be a perfect technique, depending on the needs of the website. They should always go hand-in-hand, using one to test completely different designs and the other to optimize the current design. The important thing is to understand that testing is not a one-time effort It is an ongoing exercise that should be part of the mindset of an organization. As Avinash Kaushik once wrote in his blog, Experiment or go home!

Why is it that some web sites are golden, compelling visitors to buy while similar web sites, perhaps even offering the same products at similar prices, fail to make the grade? It may be dumb luck on the owner’s part, but a more likely explanation is that the golden site is the result of careful design, testing, tweaking and testing yet again. Testing has grown into a core search marketing activity, but strangely, it’s also an activity that many search marketers either choose to avoid or don’t even consider at all.

And that’s a mistake. Good testing can not only help you get rid of stinker ads and landing pages, it can also help you optimize winning campaigns, improving their conversion rates and making them even more profitable. And testing needn’t be difficult or time-consuming, especially if you have the right tools. One of the best tools for helping you test and refine your search marketing campaigns is Google’s Website Optimizer. Even better, Website Optimizer is a free service.

Google deliberately designed Website Optimizer to be easy to use, even though the tool allows you to run complex, multivariate tests. At its most basic level, the tool lets you test different pages or combinations of elements on a single page to see how well they perform against a “conversion outcome” that you define. Google even claims that “you can launch a simple test in five minutes” in the video tour of Website Optimizer.

The problem is that to be effective, tests should be carefully planned to measure your desired outcomes. Another problem is that the results of your testing need to be interpreted properly so that you can be sure that any implementation based on the results will perform as expected. Testing is as much art as science, and it shouldn’t be done in an offhand or cavalier fashion.

Not sure where or how to start? Try Always Be Testing: The Complete Guide to Google Website Optimizer, by Bryan Eisenberg & John Quarto-vonTivadar with Lisa T. Davis. It’s a comprehensive guide to Google’s free testing tool, written for search marketers who are more interested in enhancing the performance of their search marketing campaign than in the analytic joys of multivariate testing. The authors don’t shy away from technical explanations when necessary, but the focus is always on the “whys” of testing even when looking closely at the “hows.”

The book is divided into three parts. The first part is a general overview of website testing and tuning using Google’s Website Optimizer. The first few chapters focus on the tool itself, but after that the focus changes to case studies drawn from the authors’ own search marketing campaigns. These are valuable not only as examples of how to test, but because the authors also share the processes and best practices they have developed to maximize the impact of the testing process.

Part two “What You Should Test” looks at more than 30 key factors that can impact conversion rates. These factors differ based on how conversion is defined, which also differs for publishers, retailers and lead generation web sites. Most of the chapters in this part follow a similar format, with sections on “questions to ask,” an exercise, “what to test” and “apply this to your site” tips. This is one of the best—and most thorough—guides to the overall website testing and optimization process I’ve read.

Part three, “Diving Deep For the Technically Challenged” goes into even more detail about the testing process, how things work, behind the scenes details of how Website Optimizer works and more. Meaty stuff for the more advanced search marketer who wants to know everything to gain a competitive edge.

The nice thing about Google’s Website Optimizer, apart from being free, is that it can be used to test campaigns running on Google or any other search engine. And the general practices discussed in the book can be applied to any testing tool, not just Website Optimizer.

Always Be Testing is an excellent book that tackles one of the most important aspects of successful search marketing. If you aren’t currently testing and want to know how to start, buy the book. Even if you are already testing and tuning your sites as part of your search marketing efforts, the case studies and best practices discussed in this book make it worth far more than the cover price when you apply the principles to your own campaigns.

Always Be Testing: The Complete Guide to Google Website Optimizer
by Bryan Eisenberg & John Quarto-vonTivadar with Lisa T. Davis
ISBN-13: 978-0470290637
Sybex, $29.99

Google has announced that AdWords is no longer a requirement for Website Optimizer. In fact, you can now go to google.com/websiteoptimizer and register for free to use Website Optimizer, even if you do not have an AdWords account.

Now, you can use Website Optimizer to not just A/B test your AdWords landing pages, but also test your Yahoo landing pages, your banner landing pages, and different elements throughout your web site to improve conversions and desired actions from your users.

If you already use Website Optimizer within your AdWords account, you can continue to do so. The data and tool will remain in your AdWords account and can continue to be used there for the foreseeable future. The new Website Optimizer will have its own user interface but will function a lot like the old Website Optimizer.

In addition, Google has launched a blog for Website Optimizer named the Website Optimizer Blog. The blog will contain “up-to-date product news, industry insights, testing strategies, and other educational information from the Google Website Optimizer team and our partners,” said Google.

Google Website Optimizer is available in 27 languages including Czech, Danish, Dutch, English (Australia, UK & U.S.), Finnish, French, German, Hebrew, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese (Brazil & Portugal), Russian, Spanish, Swedish, Tagalog, Thai, and Turkish.

Update: In other Google web tracking news, Google updated us that Urchin 6 is now out of beta. Urchin is the paid version of Google Analytics and runs directly on your own server. Also, Actual Metrics is offering free demonstrations of Urchin 6 if you are interested.

Google Website Optimizer, a tool that allows you to easily test different page layouts, is now available to anyone with an AdWords account. Previously, it had been out in limited beta since last October. It remains unclear whether using the tool would be considered cloaking, which is against Google’s webmaster guidelines. The Good Cloaking, Evil Cloaking & Detection article from Stephan Spencer here on Search Engine Land looked at this issue last month. Threadwatch has a recent discussion also going. I’ll explain more about the situation below, and I’m checking with Google for the official word.

Fair to say, Google’s probably going to say that using the tool is NOT cloaking. Otherwise, you’ve got one part of Google putting out a tool that will cause anyone using it to potentially be banned by another part of Google. This also means questions that have hung over other page testing tools such as Offermatica should go away.

It won’t make the cloaking debate necessarily easier, however. Consider the If Your Page Ranks Well, You’d Better Be Messing With It article from Rand Fishkin at SEOmoz last month. In it, Rand picks up on a challenge that Jamie Roche (CEO of Offermatica) makes to the traditional wisdom that if a page is ranking well, you don’t want to alter it much.

Rand argues the major factors controlling rank are mostly off-the-page (importance of your domain, links to your pages) and thus not impacted by change to the page itself. I’m largely with him. If a page is doing well, altering it really shouldn’t have that much of an impact, unless you’re doing something really drastic such as replacing all the text for images. And yet — I’d still be cautious, despite knowing that’s not likely something to fear.

This leads back to Roche’s article. Roche is clearly reacting to clients that are concerned that his product, Offermatica, might suggest changes that could cause them to lose rankings. So, there’s an element of self-interest in his writing. But I still find many of his points valid:

Unfortunately, even if you don’t mess with your page, your rank could change. That’s because the rules that the search engines use to rank sites change as they discover newer, and presumably better, ways to rank results. A page that ranked well one day might drop to the third or fourth page the next, or get removed altogether.

The result is that we live in a state of fear about changing well-ranked pages, while knowing that even if we don’t change them we could lose rank anyway.

So when we realize that we want to change our natural search landing pages because they don’t provide the best user experience, we wonder whether it might make sense to experiment with changes despite our fears.

He then also gets into three tactics, warning that potentially, they could get you delisted or cause a rank drop:

• Test various pages, and keep showing spiders the original one during testing. Anyone not seeing what the spider sees is technically getting a cloaked page. Yes, as he notes, that’s what Google Website Optimizer does – so Google itself seems to be giving tacit approval to cloaking.
   
• Change just some of the page. He seems to think this is a bigger risk. I think it’s minor. The main risk is that if you change some of the page using JavaScript, potentially that could be seen as cloaking. But that’s part of the page testing opt-out already covered above.
   
• Target pages to second-time visitors. Search engines don’t read cookies. So if you cookie a first time visitor, you can then show them a different page when they return, even if they go to the same original URL. Search engines will continue to see the same page every time, since they won’t have a cookie.

Now here’s an even more complicated issue. Many who do good old-fashioned cloaking have long argued that it doesn’t matter how they get ranking, as long as they are relevant. So what if they use gibberish text or put out some highly-optimized textual content? As long as they show the visitor a relevant page, what’s the harm? Indeed, if you buy into the argument that what you do on the page largely doesn’t matter — that it’s really down to domain authority and links — then it becomes even harder to understand why cloaking is such an issue.

Moreover, what prevents someone from eternally testing a page? If you gain a ranking with some butt-ugly page, you might then keep feeding that to spiders but constantly test it with a page testing tool. Potentially, that keeps you out of the cloaking hot seat yet it does the same exact thing that cloaking does.

Overall, Google in particular has dodged its outdated guidelines on cloaking time-and-time again over the past few years. My YADAC: Yet Another Debate About Cloaking Happens Again article from last month provides the background here. They can’t keep dodging it in the case of page testing tools now, not when they themselves are offering one. So I’m pinging them for the official word.

As for the tool, here’s more from the formal announcement:

As part of our continued commitment to help advertisers make smart business decisions, we are happy to announce that the Google Website Optimizer™ application is now available to AdWords advertisers worldwide. Google Website Optimizer is designed to help website owners increase conversions such as sales, sign-ups or downloads. This multivariate landing page optimization tool enables marketers to test different ideas for web page content such as different headlines, promotional copy, or images. The application provides easy-to-read reports that enable advertisers to see which variation resonated best with their site visitor. It is a self-service application that enables website owners to set-up and run multivariate landing page experiments.

Google Website Optimizer is a beta application that is integrated with the Google AdWords™ program and free to AdWords advertisers. Advertisers can sign up immediately at www.google.com/websiteoptimizer. Over the coming weeks, the Google Website Optimizer application will become available automatically in all advertisers’ accounts. Website owners can now determine what content was most effective as indicated by the highest conversion rate.

By giving website owners the tools they need to improve their website content, Google is helping improve the user experience on the internet as a whole. Since the beta launch of the application in October 2006, many advertisers who used Google Website Optimizer have achieved major results.

"Using Website Optimizer enables us to approach our website like a living lab, where we can test and play and constantly figure out how to improve the site," says Deborah Krier, Marketing Manager, Dale and Thomas Popcorn, "Website Optimizer is a powerful tool that allows us to understand our users better, leading to increased conversions and increased business success."

In addition, we are announcing the formation of a new partner program, Google Website Optimizer Authorized Consultants. As of today, Optimost, EpikOne, Future Now, ROI Revolution, and SiteTuners.com have signed on as charter members.

“We’re delighted that Google is now offering Website Optimizer. In the past, not everyone had the tools to test regularly,” said Bryan Eisenberg, co-founder, Future Now, Inc. and author of the New York Times bestseller Call to Action, “Google Analytics had a major effect on the accessibility of data and on how website owners valued analytics; Website Optimizer will take the benefits of testing to a much broader audience and help them increase online conversion rates.”

For more information or to sign up to use Website Optimizer, please visit: www.google.com/websiteoptimizer