# The Right Time To Start Analyzing Data

You’re a product person. You’ve bootstrapped together your website, dropped relevant metadata, setup an e-commerce platform, installed analytics, and are generating solid original content on your site and/or blog. You’re doing everything by the book.

But the question remains: when it is the right time to dig in and try to understand what people are doing once they get to your site?

In the early stages of a website, analytics are a double-edged sword. Dig in too early and you don’t have enough data, wait too long and you potentially lose out on opportunities for growth and revenue. As a result, I believe it’s never too soon to look at the data but you need to respect statistical significance or else you’ll be making emotional rather than data driven decisions.

The right time to dig into traffic patterns is when you have statistically significant insights from visitor behavior. With significant data, you can differentiate between the noise (unexplained variation within a data set) and the true variation.

There are a number of tools available to help calculate statistical significance, usually based in excel. I’m a fan of Rags Srinivasan’s calculator because it’s simple and to the point. (Note the link is a downloadable ZIP file from Avinash Kaushik’s site.)

It’s worth noting that the calculator has two tabs – a T test and Chi Squared test. Chi-Squared tests are more stringent than T tests and as a result I prefer to use the Chi-Squared test. The math behind these tests is pretty heavy. If you’re interested in fully understanding the methods of calculation, start here.

The logic behind statistical significance stems from standard deviation. The standard deviation represents how much variance exists within a data set. To calculate the standard deviation (σ) we consider the average variance between a random point in the data set and the mean.

To make sense out of this, consider a normally distributed bell curve:

By definition of normal distribution, 95% of the data exists within two standard deviations of the mean. If we test a data point for significance and it falls outside of this 95% interval then we can confidently say it’s statistically significant. If the test generates a data point within the 95% confidence interval then we consider it to be statistically insignificant.

Check out this post by Avinash Kaushik if you want to read more about the calculation process of statistical significance.

Getting back to analyzing website traffic, we can leverage statistical significance formulas to determine the right time to begin making decisions based on visitor data. I refer to this point the action threshold – the number of actions that make a particular traffic source viable and significant.

It’s worth noting that this is an unconventional application of a significance test – a traditional use case is to compare two data sets for correlation – in this case I’m not asking how similar two data sets are, I’m asking at what point is a data set significantly different than a null case.

Consider a very simple example where we have 100 visitors and our goal is to get visitors to sign up for a free trial. In the calculator, set the control to 100 visits and 0 sign ups. Then in the test, set visits to 100 and sign ups to 1:

The result is not significant because the Chi-Squared value is below the 95% confidence value. Keep increasing the number of sign ups until the Chi-Squared value is larger than the 95% confidence value and the test will deem the results significant:

In this example, 4 sign ups represents the action threshold. This means that making decisions on less than 4 actions is premature. We run this test to prevent ourselves from making rash decisions caused by impatience and gut reactions. I suggest you run a similar test to determine your action threshold then sit back and collect traffic, wait until there are enough actions to deem the results statistically significant before hunkering down to dig through the data.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: Analytics | Search & Analytics

About The Author: is in charge of client strategy at Fusion Tree, a performance marketing and analytics consulting group, and is based in the San Francisco bay area.

# Like This Story? Please Share!

Other ways to share:

# Like Our Site? Follow Us!

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.

Quick follow up: this same test can be used to determine the effectiveness of an A/B test. Rather than setting the control to 0 sign ups and 100 visitors, just enter your actual conversion rate (ex: 8 sign ups and 1352 visits) and then the data for the test in progress (ex: 13 sign ups and 1191 visits). The calculator will tell you if the variance is statistically significant.

When dealing with uneven proportions, statistical power can be very elusive and lead to inaccurate T, Z or Chisquare results.

Good writeup.

Jeff

# Get Our News, Everywhere!

North America

EMEA

APAC

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.

## Search Engine Land Webcasts, Whitepapers

Learn more about internet and search marketing with our free webinars, whitepapers and research reports at Digital Marketing Depot.

## Internet Marketing News & Strategies

News From Marketing Land:

See more at Marketing Land