How To Test A Home Page For Conversion Effectiveness

Conversion rate improvement, for many of us, is as much a part of the overall process of optimizing a paid search campaign as, say, “off-page factors” are to SEO.

Recall the holistic view taken in last month’s column, Your Paid Search Performance Is Relative. I spoke about the potential for widening the gap between your company and a competitor by looking beyond just simple bid management, to improving conversion rates, working on the business model, and other larger issues of business process. The end result of doing these things can be a virtuous circle, allowing you to bid higher, to appear prominently in high ad positions, and to return a strong ROI on those ads but on significantly higher volumes. Your momentum and market share continue to build, while competitors acquire customers at a lower rate.

Describing this ideal snowball effect that leaves a business potentially pulling away from a rival like Secretariat in the 1973 Belmont Stakes, I wrote: “…Wendy is making so much money, meanwhile, that she can afford to invest money back into improving various elements of her business; as a result of this overall re-engineering, her target cost-per-order numbers are relaxed because a customer is worth more. Luke continues to nibble away, out of sight, out of mind, while Wendy continues to gain market share.”

I also noted, in a related vein, that I was testing 24 variants of a home page for a client right now, and I think I even promised to talk about the results when finished. The purpose of the test is to find out which page recipe generates the highest conversion rate to sales (from all sources, paid and unpaid as well as direct navigation). Related to that, it’s to find out which page elements (and changes in them) contribute most to improved user response.

So here’s a run-through of what the test looks like. I hope the reasons for doing the test, and for writing about it in a column on Paid Search, are more or less self-evident.

Methodology basics:

• This will be a multivariate test using all potential permutations of the page based on our identifying four strong potential influencers of performance, and generating creative and/or scientific alternatives for each page element to be tested. A test like this typically comes at an advanced stage, and would ideally have been preceded by professional web architecture and design efforts, or arbitrary best-practices audits.
• We chose four page elements, three of which have two values, one of which has three, for 2X2X2X3=24 total page combinations. You can tell we understand the principles but are not math majors just based on the lack of scientific notation in this bullet point.
• We’re using Google Website Optimizer. This product helps with the rotation of combinations on the fly, and measurement and reporting on conversion rates. It also doesn’t require you to have a math PhD. Technical installation and overall preparation effort is nontrivial however!

Assumptions:

Among other things, we are going to assume:

• The home page is an important page on this site, even on the paid search side. A high proportion of the company’s sales still come from “core” paid search words, and while it’s important to discuss the whole keyword portfolio, economic performance is definitely dependent on making such core words work. The home page is currently a good landing page for this core keyword inventory, but it must improve.
• Low volume of overall traffic will require a long test period, but related to the above, being high volume, the home page is an excellent candidate for a test.
• We’ll have to live with a variety of extraneous influences on the test, such as seasonality, shifting proportions of paid and unpaid clicks, etc.
• If we can increase conversion rates significantly, we’ll unleash a virtuous circle that will allow the business to grow, reinvest, bid higher, become more visible, etc.
• Based on my distillation of usability, credibility, and persuasion literature, I assume that we as conversion rate improvement scientists will be acting as both economists and ideologues; or to put it more bluntly, plumbers and persuaders. Conversion rates improve if you get the plumbing right. They improve if you persuade better, in a sense broadly defined. They get worse if you do the opposite of those things.
• Hunches as to what to test come from what seems plainly to suck on the site or page. They also come from thinking more deeply about customers and their mindset. The latter is harder. We can do a lot by fixing what sucks.

Page elements chosen

After arbitrarily eliminating some page clutter and moving some other things to reorient priorities, we selected four page elements to test. Depending on your site’s traffic volumes, you can’t test everything, unfortunately. I alter my examples slightly so you can’t find them using a search engine.

Headline. If you have any kind of core message, headline, tagline, etc. at the top of an offer page or home page, this seems to be a perennial influencer of results. It’s telling users what you are, but it’s here that you can also introduce doubts in the user’s mind. You can unwittingly plant negative or misleading thoughts that lead to an aversion to doing business with you. The three headlines tested were: (a) Original: Rita’s Hanging Basket Wholesale Offers 30-60% Off Always and Free Shipping On Any Order Over $50; (b) 30-60% Off A Wide Selection of Hanging Baskets; (c) Trusted Online Hanging Basket Wholesaler Since 1998. The original was seen as too long. We simply wanted to gauge if other ways of encapsulating the value proposition and/or positioning, without duplicating other messages on the same page, would perform better or worse. Moreover, we wanted the three test items to be different enough that they might lead to different outcomes. By and large, because this is a message, this is in the realm of better persuasion or communications.

Body Copy. This is a company proud of its dedication to its niche and its long tenure online. But we simply wondered about the time-honored long-copy vs. short-copy conundrum. I rewrote the fairly compelling long copy to reduce the number of benefits and information points, while hopefully conveying the same message. At this stage, I feel like the test isn’t quite right because the short copy doesn’t get a fair shot; it seems to unduly change the whole feel of the page by being so short. Body copy, too, falls squarely within the range of persuasion and getting a message sharply across to the target customer, but it’s potentially a navigational issue also. Do people scroll and read? Do they scan? You might not need to know or care exactly what they do if you have solid statistics showing which version of the body copy (long or short) performs better, in combination with which other page elements.

Clutter-Muck #1: We thought the whole site was too cluttered. There were four small boxes with different product promotions and other info at the top of the body copy. Small is still annoying if there is too much stuff. This element was tested as a simple present or absent element. It’s mainly about plumbing/navigation, in the sense that we’re looking for the statistical rationale to go ahead and declutter what we felt was a decluttered interface. Test, don’t guess.

Clutter-Muck #2: There is a box for a rotating featured special that pushes other info boxes such as the shipping offer farther down the page on the right nav, and in general adds to clutter. Could this element alone contribute to lower conversion rates? We’ll go with the present or absent test here, too, and we’re doing it for the exact same reason as the other test. Less cluttered home pages ‘r’ us, but we need proof.

Early feedback: Bad plumbing hurts, and better persuasion doesn’t help

This is early days, and I won’t give you my full comments on the result for a couple of months. But it’s so rewarding when the test begins showing you stuff right away, and conversion numbers for all 24 page elements are shown so handily in the Website Optimizer interface. What we’re getting so far is that both clutter tests are showing the expected result, perhaps a bit more significantly than expected. Page versions without the info boxes and special box are doing better than the cluttered versions.

The other elements aren’t yet conclusive, but it’s quite early yet. One page combination, Combination 23, has pulled out into a commanding early lead, about 4X the current average conversion rate. If the result holds up even anywhere close to that—even if it’s only 2X—the client’s bottom line improves significantly… and without a major website overhaul. On paid search, that would be equivalent to cutting the average cost per order from $36 to $18. Previous paid campaign optimization had cut it to $36 from $80+.

I actually expect the main trend to hold, and I expect it may translate well to other experiments. The persuasive copy elements here so far all come out about equal, because someone who is fairly passionate about the business, and/or a good writer, wrote all of them. Users responded reasonably well to all versions.

By contrast, users with some buying predisposition showed a marked dislike for extraneous distractions and offers, visual distractions, and the overall feeling of a lack of focus. Plumbing, then, was a bigger influencer than persuasion. What’s “broken” from a navigation or attention standpoint often won’t sell. A slightly better way of “telling,” on the other hand, had not too much influence on skeptical consumer minds. Seth Godin’s The Big Red Fez, that short book on usability, stands vindicated (thus far). But it’s probably also the case that the persuasion process requires segmentation and deep-seated scene-setting that go beyond a few tweaks. To cite more books and authors, Bryan Eisenberg et al., Waiting for Your Cat to Bark; and Seth Godin, All Marketers Are Liars, come to mind).

Big bang for few bucks

The leverage offered by a proper analysis and test conducted by a small committee of technically and creatively qualified professionals (including the client) is powerful. So is the leverage offered by a properly-designed testing protocol and the software that runs it. Above all, the process moves us another step towards optimal marketing communications, and one step farther out of the realm of noble savagery and superstition in the marketing profession.

Won’t this testing potentially harm organic rankings?

For a small company to so ruthlessly test its homepage, especially when it’s responsible for more than 50% of the company’s overall traffic due to favorable organic rankings, seems reckless. Jamie Roche has advocated such testing, and Website Optimizer spokespeople have stated that such tests won’t harm search rankings. But where’s the proof? Is it a question of appetite for risk? The conversion rate improvement offsetting any disruption to rankings that leads to a decline in organic traffic? Or are rankings left largely unaffected? Are there cases where they might improve, as in the decluttering effort that leads to potentially quicker page load times and more user time spent on site, which are quality signals that might increase organic and paid search standing? I’m afraid this elephant in the room is so complicated that I’ll have to mix that metaphor in a future column, perhaps titled The Complicated Elephant on the Home Page. But clearly there is enough doubt here to suggest that no one should just plunge into arbitrarily testing important pages, without understanding the risks and rewards.

Andrew Goodman is the founder and principal of Page Zero Media and author of Winning Results with Google AdWords. The Paid Search column appears Tuesdays at Search Engine Land.