Why Big Testing Will Be Bigger Than Big Data

Big data is a big topic these days, one that has made its way up to the C-suite. The CMO may not yet fully understand what big data is, exactly. But the CMO knows he or she needs a plan for how to use it.

Big Data, Big Testing, Big Experience

In fact, three of IDC’s Top 10 predictions for CMOs in 2013 revolve around mastering this data explosion in marketing.

In many ways, this attention on big data is a breakthrough moment for marketing. Sure, data has always been present in certain corners of the marketing department — especially with you astute search marketers and conversion optimization pros. But marketing management and culture have thrived more generally around gut instincts, creative concepts, and compelling communications.

Marketing has been about big ideas. Big thinking. Big budgets.

But big data? That’s something new. What’s qualitatively different — and somewhat ironic — is that big data actually promises more visibility into ever smaller circles of customer segments, asymptotically approaching a fully personalized “segment of one.”

With the new tools that are emerging, marketers can crunch their petabytes of disparate data, pulled from owned, earned, and paid media, mingled with transaction histories, mixed with profiles from third-party exchanges, and combined with public and industry statistics to divine all kinds of interesting correlations.

The goal: uncover connections that appear to influence different subsets of your audience to take action more successfully.

The Big Hypothesis Generator

I choose the words “appear to influence” with care. Because correlation is not causation. The insights generated by big data are usually tentative at best — possible relationships, in the past, between our behaviors as marketers and the behaviors of our prospects and customers.

In other words, most of these insights are the seeds of hypotheses. There’s no guarantee that the correlations unearthed in big data can directly influence customer behavior.

There are many reasons for this. For one, while big data naturally implies a large amount data, it’s far from complete. There always remains a huge number of “confounding factors” out there in the world, not captured in our data, influencing customers in ways not represented in our closed big data models.

(In a way, this is good news for marketers: the CEO is unlikely to be able to replace you with a robot, at least, any time soon.)

But it does mean that we have work to do to harness the insights that big data nominates as possible opportunities. We should look to big data for inspiration — and combine it with our ability to distill those revelations into testable customer experiences.

That’s the powerhouse human-machine combination that lets us move from interesting correlations to actionable marketing.

Big Testing: Big Ideas, Big Team, Big Deal

However, to take advantage of this, marketing’s culture must shift towards testing.

For years, testing and optimization have been niche practices in the marketing department. A/B testing with a few direct mail pieces in the past. A/B testing with a few landing pages in the present. But, most marketing programs have been run on intuition.

But now, big data is opening the door to the executive suite for a more hybrid analytical-creative method. The questions big data raises — okay, how do we use this data to grow our business? — have an answer: broadly embrace testing and controlled experimentation as the new “operating system” of marketing.

The answer is big testing.

Big Testing

What exactly does “big testing” mean? Its bigness is a function of three things:

First, big testing is about experimenting with big ideas. This is best captured in an article by Eric Ries, author of The Lean Startup: learning is better than optimization (the local maximum problem). It actually points out that most landing page and website optimization programs, while useful in some ways, are not very helpful at learning how to build a better business.

“The right split-tests to run are ones that put big ideas to the test,” he writes. “For example, we could split-test what color to make the ‘Register Now’ button. But how much do we learn from that? Let’s say that customers prefer one color over another? Then what? Instead, how about a test where we completely change the value proposition on the landing page?”

Embracing big ideas in big testing is about fearlessly answering the question — Who’s Afraid Of The Big Bold Test? — with a resounding, “Not us!”

Second, big testing is about empowering many people in the marketing organization to do testing. It’s about giving them the training, the tools, and — most importantly — the mandate to test new ideas.

Historically, testing has been restricted to a small number of gatekeepers. But now that we have such a fragmented and fractured marketing landscape — and with big data helping us identify ever more granular opportunities within it — we need to tap more marketers on the team to run controlled experiments.

Hal Varian, the chief economist at Google, has said that Google runs about 10,000 experiments each year. A large number of different people throughout the company are engaged in all kinds of different tests in parallel. It’s not the cult of a few; it’s the culture of the many.

Finally, big testing is about making a big deal about testing from the top down, fostering a culture of experimentation.

This last point will probably be the most challenging, as culture is not something that changes quickly. Executives need to make a conscious effort to encourage real testing — starting with the acknowledgement that good experiments prove or disprove hypotheses. Not every test will be a winner, but if the test was executed well, a negative result shouldn’t reflect poorly on the tester.

Marketing leaders need to make their teams feel good — not scared — about testing those big ideas.

Big data is like fuel. Big testing will be the engine that turns it into forward momentum.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: Analytics | Search & Conversion


About The Author: is the president and CTO of ion interactive, a leading provider of landing page management and conversion optimization software. He also writes a blog on marketing technology, Chief Marketing Technologist. Follow him on twitter via @chiefmartec.

Connect with the author via: Email | Twitter


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • http://twitter.com/soloportfolio Clare McDermott

    Hi Scott… Really interesting article. I’d love to know more about what this looks like: “Historically, testing has been restricted to a small number of
    gatekeepers. But now that we have such a fragmented and fractured
    marketing landscape — and with big data helping us identify ever more
    granular opportunities within it — we need to tap more marketers on the
    team to run controlled experiments.”

    I love the idea in concept, but some examples of how this plays out would help me understand it better.

    By the way, we are talking about Big Data in our August issue of CCO… would love to be in touch to find out how we can access your brain!

  • Pat Grady

    You need to add a “Big Mistakes” blog to that graphic. It includes report blindness, faith errors, bad assumptions, and attribution stupidity.

  • http://twitter.com/chiefmartec Scott Brinker

    I was going to, but I was afraid that might generate Big Laughs. Where would I fit that on the diagram then without creating a Big Mess?

    Seriously though, I agree with your point. All of this boils down to people and organizations being better at using data well. It’s a big change in mindset, culture, and behavior. But when I think of “big testing” that’s part of the transformation in data sensibility that is necessary to make it useful.

  • http://twitter.com/chiefmartec Scott Brinker

    Thanks, Clare.

    Happy to chat with you anytime. A couple of books that capture this phenomenon of many small tests include:

    Little Bets by Peter Sims
    Lean Startup by Eric Ries
    Uncontrolled by Jim Manzi

    Lots of good examples in those books that cross many domains (including marketing).

  • Jaume Clotet

    Hi Scott, thanks for your article, I am 100% with you. I would like to place you a question. As we can observe demand (users) are evolving faster and deeper than the offer (advertisers/companies), there is a big gap, and is getting bigger and faster everday. I see “big data” like the rope that will allow me to first stop the bull, and finally control it. Well, first of all I need to tie the beast. Now, How do you suggest put the noose around his neck, convincing the boss? Cheers!

  • http://twitter.com/TylerHakes Tyler Hakes

    I love this in concept – but I also see that it gets messy quickly.

    For one, having multiple people running multiple tests in parallel makes it hard to generate any meaningful conclusions unless you have a PERFECT data set (my experience is that almost almost no company has flawless, complete data about everything they’re doing).

    More importantly, each test that answers one question generates an infinite number of next questions — specifically if you already are working from a highly-segmented customer data set. That means that having tons of tests occurring simultaneously may just muddy the waters to the point that it becomes worthless.

    Interested to hear your thoughts on how to deal with these problems.

  • http://www.facebook.com/justin.fogarty Justin Fogarty

    Great article Scott. Testing is key if we want to use big data to create “personalized segment of one” for customers based on their intent.

    The only thing I would add is that in order to do conduct those tests, analyze the results and act on the insights at at a big data scale and at the speed needed to delight customers, machine learning will need to play a role.

    I expanded on this more on our blog today – http://www.bloomreach.com/2013/02/big-data-needs-big-testing/


Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide