• Search Engine Land
  • Sections
    • SEO
    • SEM
    • Local
    • Retail
    • Google
    • Bing
    • Social
    • Resources
    • More
    • Home
  • Follow Us
    • Follow
  • Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • Live
  • More
  • Events
    • Follow
  • SUBSCRIBE

Search Engine Land

Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • More
  • Newsletters
  • Home
Analytics & Conversion

Landing Page Testing: Choosing Between A/B Or Multivariate Approaches

There are quite a few testing techniques available in the market. In this post I will dwelve into the two commonest testing methods: A/B tests and Multivariate tests. What is the difference between them? How can you choose which one best fits your needs?

Daniel Waisberg on October 14, 2009 at 3:21 pm
  • More

In a previous post, I wrote about how to get started with website testing, both choosing which pages to test and how to define which elements will contribute the most to profits. However, there are quite a few testing methods to choose from. In this post I will delve into the two most common testing methods: A/B tests and multivariate tests (MVT). What is the difference between them? How can you choose which one best fits your needs?

Below is a comparison between the testing techniques mentioned, taking into consideration the overall use of the testing technique, coding needs, design needs, granularity of results and other considerations.

Most testing tools provide these options, but since Google Website Optimizer is a free tool that provides both options, it is a good place to start and try the examples I provide below.

A/B Test

An A/B test is the most common and easiest type of landing page test to conduct. It consists of creating alternative pages for a specific page and showing each of them to a certain percentage of visitors. For example, if you create 4 different variations of a landing page, 20% of visitors to the website will see each version (4 variations + original). Cookies are used to maintain a consistent user experience—if a visitor sees one version, they will see it again and again when visiting the website as long as the cookies are not deleted. Below is a representation of how this technique works.

AB test scheme

Image created by Yam Designs. For a high res version of the image go to A/B test scheme.

To implement the test with Google Website Optimizer, scripts need to be included on the pages to be tested. These include:

  • Two JavaScript codes on the original page: one that performs a redirect to the additional variations (head of the page) and one that measures the number of times the page was seen (this can be placed anywhere below the redirect code).
  • One JavaScript code on each variation page to measure the number of visitors viewing each page.
  • One JavaScript code on the conversion page to measure which visitors converted; this will measure the success of each page variation.

Advantages of A/B tests

Design freedom. A/B tests are often used to experiment page design options that vary dramatically, including position of text and pictures, background colors, number of pictures on the page, use of icons and navigation structure. Implementing such tests using the multivariate technique is possible, but it is technically challenging (but if you really want to do it, and you are technically savvy, see this post on the Google Website Optimizer Tricks blog.

Less JavaScript coding. as described above, the codes necessary to implement an A/B test are very simple and can be added to the website in a matter of minutes.

Faster results. A/B tests usually involve fewer combinations with more extreme changes; multivariate tests involve many more combinations and variations. In addition, since A/Bs show significantly different designs, the expected improvement of the page is usually higher, diminishing the time the test will run.

Multivariate test

Rather than testing different versions of web pages, as we do with A/B tests, Multivariate tests experiment with elements inside one specific page (for purists, we are referring to full factorial experiments, which is the method used by most testing tools). Basically, we define elements inside a page (e.g. a picture, a text or a button) and provide different alternatives of each element. The testing tool will show each element combined with all other elements to visitors. The resulting combinations are derived from the number of elements multiplied by the number of element variations. Just as with A/B testing, however, each visitor sees only one particular combination of elements regardless of how many times they view a page. Below is a representation of how this technique works.

Multivariate test scheme

Image created by Yam Designs. For a high res version of the image go to Multivariate test scheme.

In terms of coding, the programming a multivariate test is slightly more complex than a simpler A/B test. A few pieces of JavaScript code need to be implemented: one opening the test, one for each tested element and one closing the test. In addition, a JavaScript will be added to the conversion page to measure combination success.

Advantages of multivariate tests

Granularity of results. Since it is a full factorial experiment, multivariate tests show which elements are the best performing separately, as well as the correlation between the elements. This can be very useful when projecting the results to other parts of the website.

No redirects required. Since all elements tested are inside the page, there is no need to redirect from the original page to the tested pages. Although redirects can be performed smoothly, I believe it is better not to use them whenever possible, as they can slow the flow and affect user experience.

Fewer design resources required. Since we will be testing different designs with existing elements on a page, this will not require too much design effort.

Concluding, both types of testing have their own advantages and disadvantages. Each can be a perfect technique, depending on the needs of the website. They should always go hand-in-hand, using one to test completely different designs and the other to optimize the current design. The important thing is to understand that testing is not a one-time effort It is an ongoing exercise that should be part of the mindset of an organization. As Avinash Kaushik once wrote in his blog, Experiment or go home!


Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.



About The Author

Daniel Waisberg
Daniel Waisberg is the Principal og Conversion Journeyand the founder of Online Behavior, a Marketing Measurement & Optimization website. He holds a M.Sc. in Operations Research and Decisions from Tel Aviv University, where he developed a statistical model that helps to optimize websites using Markov Chains. Daniel is a frequent speaker & member of the Advisory Council of the eMetrics Marketing Optimization Summit. You can follow him on Twitter or Google+.

Related Topics

Channel: Analytics & ConversionGoogle: Website OptimizerHow To: PPCSearch Marketing: Landing Pages

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter search marketers rely on.
See terms.

ATTEND OUR EVENTS

Lorem ipsum doler this is promo text about SMX events.

February 23, 2021: SMX Report

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

October 2021: SMX Advanced Europe

December 17, 2021: SMX Code

Available On-Demand: SMX

×


Learn More About Our SMX Events

Discover actionable tactics that can help you overcome crucial marketing challenges. Our next conference will be held:

MarTech 2021: March 16-17

MarTech 2021: Sept. 14-15

MarTech 2020: Watch On-Demand

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • The State of Local Marketing Report 2020-2021
  • Quality CRM Data: The Key to Delivering Great Customer Experiences
  • How the Microsoft Search Network Can Maximize Your Search Campaigns
  • The Marketer’s Playbook for Customer Acquisition
  • How To Optimize SEO With UGC
See More Whitepapers

Webinars

  • How to Avoid the Digital Transformation Trap
  • How to Build a Marketing System of Record
  • Meet BIMI: The brand-boosting email security marketers must have for 2021
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

h
Receive daily search news and analysis.
Search Engine Land
Download the Search Engine Land App on iTunes Download the Search Engine Land App on Google Play

Channels

  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social

Our Events

  • SMX
  • MarTech

Resources

  • White Papers
  • Research
  • Webinars
  • Search Marketing Expo
  • MarTech Conference

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • Youtube
  • iOS App
  • Google Play

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok