Note: By submitting this form, you agree to Third Door Media's terms. We respect your privacy.

Why Big Testing Will Be Bigger Than Big Data

Scott Brinker on January 30, 2013 at 9:42 am

Big data is a big topic these days, one that has made its way up to the C-suite. The CMO may not yet fully understand what big data is, exactly. But the CMO knows he or she needs a plan for how to use it.

In fact, three of IDC’s Top 10 predictions for CMOs in 2013 revolve around mastering this data explosion in marketing.

In many ways, this attention on big data is a breakthrough moment for marketing. Sure, data has always been present in certain corners of the marketing department — especially with you astute search marketers and conversion optimization pros. But marketing management and culture have thrived more generally around gut instincts, creative concepts, and compelling communications.

Marketing has been about big ideas. Big thinking. Big budgets.

But big data? That’s something new. What’s qualitatively different — and somewhat ironic — is that big data actually promises more visibility into ever smaller circles of customer segments, asymptotically approaching a fully personalized “segment of one.”

With the new tools that are emerging, marketers can crunch their petabytes of disparate data, pulled from owned, earned, and paid media, mingled with transaction histories, mixed with profiles from third-party exchanges, and combined with public and industry statistics to divine all kinds of interesting correlations.

The goal: uncover connections that appear to influence different subsets of your audience to take action more successfully.

The Big Hypothesis Generator

I choose the words “appear to influence” with care. Because correlation is not causation. The insights generated by big data are usually tentative at best — possible relationships, in the past, between our behaviors as marketers and the behaviors of our prospects and customers.

In other words, most of these insights are the seeds of hypotheses. There’s no guarantee that the correlations unearthed in big data can directly influence customer behavior.

There are many reasons for this. For one, while big data naturally implies a large amount data, it’s far from complete. There always remains a huge number of “confounding factors” out there in the world, not captured in our data, influencing customers in ways not represented in our closed big data models.

(In a way, this is good news for marketers: the CEO is unlikely to be able to replace you with a robot, at least, any time soon.)

But it does mean that we have work to do to harness the insights that big data nominates as possible opportunities. We should look to big data for inspiration — and combine it with our ability to distill those revelations into testable customer experiences.

That’s the powerhouse human-machine combination that lets us move from interesting correlations to actionable marketing.

Big Testing: Big Ideas, Big Team, Big Deal

However, to take advantage of this, marketing’s culture must shift towards testing.

For years, testing and optimization have been niche practices in the marketing department. A/B testing with a few direct mail pieces in the past. A/B testing with a few landing pages in the present. But, most marketing programs have been run on intuition.

But now, big data is opening the door to the executive suite for a more hybrid analytical-creative method. The questions big data raises — okay, how do we use this data to grow our business? — have an answer: broadly embrace testing and controlled experimentation as the new “operating system” of marketing.

The answer is big testing.

What exactly does “big testing” mean? Its bigness is a function of three things:

First, big testing is about experimenting with big ideas. This is best captured in an article by Eric Ries, author of The Lean Startup: learning is better than optimization (the local maximum problem). It actually points out that most landing page and website optimization programs, while useful in some ways, are not very helpful at learning how to build a better business.

“The right split-tests to run are ones that put big ideas to the test,” he writes. “For example, we could split-test what color to make the ‘Register Now’ button. But how much do we learn from that? Let’s say that customers prefer one color over another? Then what? Instead, how about a test where we completely change the value proposition on the landing page?”

Embracing big ideas in big testing is about fearlessly answering the question — Who’s Afraid Of The Big Bold Test? — with a resounding, “Not us!”

Second, big testing is about empowering many people in the marketing organization to do testing. It’s about giving them the training, the tools, and — most importantly — the mandate to test new ideas.

Historically, testing has been restricted to a small number of gatekeepers. But now that we have such a fragmented and fractured marketing landscape — and with big data helping us identify ever more granular opportunities within it — we need to tap more marketers on the team to run controlled experiments.

Hal Varian, the chief economist at Google, has said that Google runs about 10,000 experiments each year. A large number of different people throughout the company are engaged in all kinds of different tests in parallel. It’s not the cult of a few; it’s the culture of the many.

Finally, big testing is about making a big deal about testing from the top down, fostering a culture of experimentation.

This last point will probably be the most challenging, as culture is not something that changes quickly. Executives need to make a conscious effort to encourage real testing — starting with the acknowledgement that good experiments prove or disprove hypotheses. Not every test will be a winner, but if the test was executed well, a negative result shouldn’t reflect poorly on the tester.

Marketing leaders need to make their teams feel good — not scared — about testing those big ideas.

Big data is like fuel. Big testing will be the engine that turns it into forward momentum.

Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.

About The Author

Scott Brinker

Scott Brinker is the conference chair of the MarTech® Conference, a vendor-agnostic marketing technology conference and trade show series produced by MarTech Today's parent company, Third Door Media. The MarTech event grew out of Brinker's blog, chiefmartec.com, which has chronicled the rise of marketing technology and its changing marketing strategy, management and culture since 2008. In addition to his work on MarTech, Scott serves as the VP platform ecosystem at HubSpot. Previously, he was the co-founder and CTO of ion interactive.