Landing Page Testing: How Much Is Too Much?

Your landing page is a key part of any successful paid search campaign, and by now, the value of testing and optimizing your landing pages has become accepted wisdom. However, many testing methodologies—especially multivariate testing—are complex and are only appropriate for sites that get hundreds of conversions a day. Small- and mid-sized companies simply do not have the volume required to achieve statistical significance in a timely fashion except using simpler A/B testing approaches.

In this article, I will share a new formula and Landing Page Test Calculator that will tell you how many test versions your landing page can support, and therefore which test methodology is right for you.

Test everything

MarketingSherpa research shows that testing your landing pages can improve conversion rates by 40% or more. Think about it: getting 40% more conversions for the same spending is a lot smarter than spending 40% more!

Michael Egan of Yahoo! Search Marketing further emphasizes the value of testing, even for the best marketers:

It’s hard to argue that Tiger Woods is pretty darn good at what he does. But even he is not perfect. Imagine if he were allowed to hit four balls each time and then choose the shot that worked the best. Scary good.

That’s the value of testing.

Too many tests?

There are some great high-end software tools on the market to make it easier for marketers to create landing page tests and measure results. Perhaps the best known are Offermatica and Google Website Optimizer. My company, Marketo, provides software that helps marketers both create and test landing pages.

The high-end tools promote the ability to test multiple elements at one time. Google Website Optimizer creates a different test "recipe" for each and every possible combination, while Offermatica uses the Taguchi method to reduce the number of recipes. The argument in favor of multivariate testing is that testing multiple things at once results in faster overall learning since each conversion yields insight into more than one test element. Also, it can provide insight into how different elements interact to improve conversion.

For example, say you are testing the headline, image, caption, call to action, and form (five elements) and each element has four test variants. Using Google Website Optimizer, this would create a total of 5 x 5 x 5 x 5 = 625 test variants. Using a Taguchi test array might reduce the number of recipes down to 10 or 20 versions.

Reducing your test variants to ten is great, especially if you have a high-volume website that generates 100s of conversions a day. However, it is still too many for any site that would not be considered "high volume".

How many tests can you run?

To understand this, we need something that tells us how many tests a given landing page can support. The Landing Page Test Calculator lets you enter how many conversions your landing page gets per day and returns the number of test versions you can support.

The calculator uses the following formula (see Landing Page Testing – The Ultimate Guide to Test Statistics for a derivation and more):

Where:

• T = number of tests you can have
• D = number of days to get results (e.g. 14 days)
• a = resolution of test, as a % of p (e.g. 25%, for example)
• p = your average expected response rate (e.g. 10% conversion rate)
• R = responses / conversions per day for this landing page (e.g. 20 per day)
• z_a = confidence level z-value (use 1.2816 for 80% confidence)
• z_ß = confidence of detecting a real result (use 0.84 for 80%)

We can also calculate a "rule of thumb" by making some simplifications. Using typical values such as a=25%, p=10%, z_a =1.28, and z_ß =0.841, the formula gives T = 0.0077 x D x R. Divide D by 7 to turn it into W (weeks). This gives T = 0.052 x W x R. Now, 0.052 is almost the same as 1/20, so simplifying gives the following rough formula:

# Test Versions = (Responses per Day / 20) x (# Weeks In Test)

In other words, to get the number of landing page versions you can confidently test, take the number of conversions your page gets per day and divide it by 20. Then take your testing period in weeks. Multiply the two results together, and you’ll estimate the number of test versions that your page can support.

For example, assume your landing page gets R=20 conversions each day. How many landing page test versions can you run if you want significant results within 2 weeks (W=2)? Plugging this into the rough formula gives 2 x 20 / 20 = 2, meaning you can test two versions and get significant results within two weeks.

Be careful not to over-test

So far, so good. But how long will it take to get valid results if you have 8 test versions and get 20 conversions per day? Entering this into the Landing Page Test Calculator gives 52 days (which is probably longer than most marketers are willing to wait for results).

Worse, even 20 conversions per landing page per day is higher than most small- and mid-size companies get. At Marketo, I spend about $100 a day on one of my ad groups. The landing page has a great conversion rate (17%), but even this page only gets about five conversions a day (e.g. $20 per conversion). If I wanted to test 10 versions, I’d have to wait 239 days (or at least 87 if I’m willing to accept Type II errors). Instead, I test only two versions—one champion and one challenger—at a time and act on the results every 18 days.

Summary

Don’t get me wrong: I’m a fan of multivariate approaches to testing for sites that can support those volumes. Further, doing any testing is more important than arguing about what kind of testing to use. The trick is to pick the right approach for the volumes your landing pages receive. To help, here’s a table that tells you how many test versions you can support, given your test period and your conversions per day.

What this means is that high-volume pages near the bottom right-hard corner of this table can and should use multivariate testing to test 5 or more variants at a time. However, landing page tests that don’t support more than 5 variants at a time will be better off with a more straightforward approach such as A/B testing.

Jon Miller is VP of Marketing for Marketo, a provider of marketing automation software that helps B2B marketing professionals drive revenue and improve marketing accountability.The Paid Search column appears Tuesdays at Search Engine Land.