Landing Page Testing: Choosing Between A/B Or Multivariate Approaches

In a previous post, I wrote about how to get started with website testing, both choosing which pages to test and how to define which elements will contribute the most to profits. However, there are quite a few testing methods to choose from. In this post I will delve into the two most common testing methods: A/B tests and multivariate tests (MVT). What is the difference between them? How can you choose which one best fits your needs?

Below is a comparison between the testing techniques mentioned, taking into consideration the overall use of the testing technique, coding needs, design needs, granularity of results and other considerations.

Most testing tools provide these options, but since Google Website Optimizer is a free tool that provides both options, it is a good place to start and try the examples I provide below.

A/B Test

An A/B test is the most common and easiest type of landing page test to conduct. It consists of creating alternative pages for a specific page and showing each of them to a certain percentage of visitors. For example, if you create 4 different variations of a landing page, 20% of visitors to the website will see each version (4 variations + original). Cookies are used to maintain a consistent user experience—if a visitor sees one version, they will see it again and again when visiting the website as long as the cookies are not deleted. Below is a representation of how this technique works.

AB test scheme

Image created by Yam Designs. For a high res version of the image go to A/B test scheme.

To implement the test with Google Website Optimizer, scripts need to be included on the pages to be tested. These include:

  • Two JavaScript codes on the original page: one that performs a redirect to the additional variations (head of the page) and one that measures the number of times the page was seen (this can be placed anywhere below the redirect code).
  • One JavaScript code on each variation page to measure the number of visitors viewing each page.
  • One JavaScript code on the conversion page to measure which visitors converted; this will measure the success of each page variation.

Advantages of A/B tests

Design freedom. A/B tests are often used to experiment page design options that vary dramatically, including position of text and pictures, background colors, number of pictures on the page, use of icons and navigation structure. Implementing such tests using the multivariate technique is possible, but it is technically challenging (but if you really want to do it, and you are technically savvy, see this post on the Google Website Optimizer Tricks blog.

Less JavaScript coding. as described above, the codes necessary to implement an A/B test are very simple and can be added to the website in a matter of minutes.

Faster results. A/B tests usually involve fewer combinations with more extreme changes; multivariate tests involve many more combinations and variations. In addition, since A/Bs show significantly different designs, the expected improvement of the page is usually higher, diminishing the time the test will run.

Multivariate test

Rather than testing different versions of web pages, as we do with A/B tests, Multivariate tests experiment with elements inside one specific page (for purists, we are referring to full factorial experiments, which is the method used by most testing tools). Basically, we define elements inside a page (e.g. a picture, a text or a button) and provide different alternatives of each element. The testing tool will show each element combined with all other elements to visitors. The resulting combinations are derived from the number of elements multiplied by the number of element variations. Just as with A/B testing, however, each visitor sees only one particular combination of elements regardless of how many times they view a page. Below is a representation of how this technique works.

Multivariate test scheme

Image created by Yam Designs. For a high res version of the image go to Multivariate test scheme.

In terms of coding, the programming a multivariate test is slightly more complex than a simpler A/B test. A few pieces of JavaScript code need to be implemented: one opening the test, one for each tested element and one closing the test. In addition, a JavaScript will be added to the conversion page to measure combination success.

Advantages of multivariate tests

Granularity of results. Since it is a full factorial experiment, multivariate tests show which elements are the best performing separately, as well as the correlation between the elements. This can be very useful when projecting the results to other parts of the website.

No redirects required. Since all elements tested are inside the page, there is no need to redirect from the original page to the tested pages. Although redirects can be performed smoothly, I believe it is better not to use them whenever possible, as they can slow the flow and affect user experience.

Fewer design resources required. Since we will be testing different designs with existing elements on a page, this will not require too much design effort.

Concluding, both types of testing have their own advantages and disadvantages. Each can be a perfect technique, depending on the needs of the website. They should always go hand-in-hand, using one to test completely different designs and the other to optimize the current design. The important thing is to understand that testing is not a one-time effort It is an ongoing exercise that should be part of the mindset of an organization. As Avinash Kaushik once wrote in his blog, Experiment or go home!

Related Topics: Channel: Analytics | Google: Website Optimizer | How To: SEM | Search Marketing: Landing Pages


About The Author: is the Founder of Conversion Journey, a Google Analytics Certified Partner. He is also the founder of Online Behavior, a Marketing Measurement & Optimization website. You can follow him on Google+ or Twitter.

Connect with the author via: Email | Twitter | Google+ | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Comments are closed.

Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide