• Search Engine Land
  • Sections
    • SEO
    • SEM
    • Local
    • Retail
    • Google
    • Bing
    • Social
    • Resources
    • More
    • Home
  • Follow Us
    • Follow
  • Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • Live
  • More
  • Events
    • Follow
  • SUBSCRIBE

Search Engine Land

Search Engine Land
  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social
  • Resources
  • More
  • Newsletters
  • Home
SEM

Advanced testing: What 4,000 A/B tests can teach you

Contributor Jason Puckett summarizes a presentation from SMX Advanced 2017 on lessons learned from 11 years of conversion optimization. In it, speaker Ayat Shukairy explains what's critical for the success of any A/B testing program.

Jason Puckett on June 20, 2017 at 1:54 pm
  • More

SMX Advanced has always featured the industry’s top talent discussing what they’re passionate about: search marketing. Last week’s conference in Seattle was no different.

Ayat Shukairy, co-founder and vice president of client solutions at Invesp, shared nine lessons from 11 years of A/B testing experience. The goal of the presentation was to present the many nuances which go into a successful A/B testing plan. After 4,000 successful conversion rate optimization (CRO) implementations, Shukairy has developed quite a knowledge base.

Ayat Shukairy

Below I have listed Shukairy’s nine key takeaways, along with some additional insights and commentary:

1. We often focus on the wrong thing

Shukairy advised us to stop focusing on the short term. We need to keep our long-term testing plans in mind when executing our strategies.

Additionally, Shukairy reminded the audience to be skeptical of giant lifts in performance; if you see a huge lift in conversion rates (CVR), then check to make sure that (a) you’ve run the test long enough to produce meaningful results, and (b) this test is not an outlier.

2. Create a compelling website narrative

There are three types of A/B testing: element-level testing, page-level testing (layout) and visitor flow testing. Shukairy notes that page-level testing can yield between 7-9 percent improvement in CVR, while visitor flow testing is something that’s going give you larger improvements (successful implementations can show 16-18 percent improvement).

However, even the best testers will reach a threshold without a compelling narrative. Define your narrative and “wow” your visitors.

3. Don’t assume you live in a vacuum

Your website experience is only a single touch point for customers, so it’s important to consider your website within the context of your entire brand. Consider your brand from the perspective of your potential customers — ask yourself, “How does my brand look to my audience?” Understand every touch point and its impact on the consumer, as each touch point can impact conversion rates.

4. Don’t assume you know everything about customers

Listen to customers through qualitative research. Polls and surveys to uncover motivations should be conducted prior to launching A/B tests on your web experience. Well thought-out market research is essential to development of a good hypothesis.  

5. Obtain buy-in from everyone

Make sure you can get buy-in from all client stakeholders regarding the philosophies of CRO. You need everyone in different departments to be vested in order to succeed. The more information, discussion and engagement you receive from different departments, the more successful your CRO efforts will be.

6. Embrace failure

Shukairy noted that only 13 percent of A/B tests report significant uplifts — which means that 87 percent do not. Let this fact change the way you think about CRO. More often than not, your tests will fail to drive meaningful growth. That means you must push through and innovate on your failure, using failed A/B tests to generate research opportunities.

7. Align your website objectives with overall business KPIs

A testing tool is only as useful as the actionable insights obtained from it. So, when measuring the results of your A/B tests, ask yourself, “What is the dollar impact?” If you can’t answer this question, you may need to update your reports to ensure that you are measuring KPIs (key performance indicators) that most closely align with business objectives.

Use data to uncover problems that are hindering growth, and make sure the problems you solve have a direct impact on the KPIs that matter most.  

8. Understand A/B testing statistics

There is a key statistic that most testers underutilize: statistical power. Statistical power is the probability that a statistical test will detect an effect when there is, in fact, an effect to be detected. A high statistical power reduces the likelihood of making a Type II error (concluding there is no effect when there actually is one).

Tests with a high statistical power and a high confidence threshold will reduce the likelihood of a false positive, so always collect a significant sample size before you determine results.

9. Be careful with pollution of your A/B testing results

Run tests longer, and don’t make changes while the test is running. Any change you make to the test environment during a test will pollute your results. This includes trimming off under-performing variations, changing the allocation percentage of traffic and so on.

If you absolutely need to make a change to the environment, you will need to restart your tests and begin with a clean sample.

Thinking differently about CRO

All in all, I thought Ayat Shukairy gave a wonderful presentation. After it was done, I asked her, “What overarching piece of advice would you give to A/B testers?”

She replied, “Change the way you think about conversion rate optimization.”

Advanced Testing: What 4,000 Successful A/B Tests Can Teach You By Ayat Shukairy from Search Marketing Expo – SMX

Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.



About The Author

Jason Puckett
Jason Puckett is the CEO & Founder of AdBasis. Ad creative optimization is the name of Jason's game. AdBasis is a creative optimization technology company built for today's most advanced advertisers. AdBasis is designed for dynamic & scaled ad implementations, creative measurement and scaled optimization. Jason is a digital marketing strategist, ad optimization expert and conversion rate enthusiast. AdBasis has helped over 1,000 companies improve their paid media through creative testing.

Related Topics

Channel: SEMPaid Search ColumnSearch MarketingSearch Marketing: Landing PagesSEM IndustrySEM Industry: ConferencesSMX® - Search Marketing Expo & Conference Series

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter search marketers rely on.
See terms.

ATTEND OUR EVENTS

Lorem ipsum doler this is promo text about SMX events.

February 23, 2021: SMX Report

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

October 2021: SMX Advanced Europe

December 17, 2021: SMX Code

Available On-Demand: SMX

×


Learn More About Our SMX Events

Discover actionable tactics that can help you overcome crucial marketing challenges. Our next conference will be held:

MarTech 2021: March 16-17

MarTech 2021: Sept. 14-15

MarTech 2020: Watch On-Demand

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

White Papers

  • 6 Powerful Ways Experience Analytics Can Help Your Business Now
  • Email Tune-Up: A 5-Point Inspection to Get Your Program in Gear
  • Digital Marketing Report Q4 2020: Benchmarks and Insights for 2021
  • Data SEO – The Next Big Adventure
  • Getting Started with Email Marketing Automation
See More Whitepapers

Webinars

  • How to Avoid the Digital Transformation Trap
  • How to Build a Marketing System of Record
  • Meet BIMI: The brand-boosting email security marketers must have for 2021
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

h
Receive daily search news and analysis.
Search Engine Land
Download the Search Engine Land App on iTunes Download the Search Engine Land App on Google Play

Channels

  • SEO
  • SEM
  • Local
  • Retail
  • Google
  • Bing
  • Social

Our Events

  • SMX
  • MarTech

Resources

  • White Papers
  • Research
  • Webinars
  • Search Marketing Expo
  • MarTech Conference

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • Youtube
  • iOS App
  • Google Play

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok