The Right Time To Start Analyzing Data

You’re a product person. You’ve bootstrapped together your website, dropped relevant metadata, setup an e-commerce platform, installed analytics, and are generating solid original content on your site and/or blog. You’re doing everything by the book.

But the question remains: when it is the right time to dig in and try to understand what people are doing once they get to your site?

In the early stages of a website, analytics are a double-edged sword. Dig in too early and you don’t have enough data, wait too long and you potentially lose out on opportunities for growth and revenue. As a result, I believe it’s never too soon to look at the data but you need to respect statistical significance or else you’ll be making emotional rather than data driven decisions.

The right time to dig into traffic patterns is when you have statistically significant insights from visitor behavior. With significant data, you can differentiate between the noise (unexplained variation within a data set) and the true variation.

There are a number of tools available to help calculate statistical significance, usually based in excel. I’m a fan of Rags Srinivasan’s calculator because it’s simple and to the point. (Note the link is a downloadable ZIP file from Avinash Kaushik’s site.)

It’s worth noting that the calculator has two tabs – a T test and Chi Squared test. Chi-Squared tests are more stringent than T tests and as a result I prefer to use the Chi-Squared test. The math behind these tests is pretty heavy. If you’re interested in fully understanding the methods of calculation, start here.

The logic behind statistical significance stems from standard deviation. The standard deviation represents how much variance exists within a data set. To calculate the standard deviation (σ) we consider the average variance between a random point in the data set and the mean.

To make sense out of this, consider a normally distributed bell curve:

By definition of normal distribution, 95% of the data exists within two standard deviations of the mean. If we test a data point for significance and it falls outside of this 95% interval then we can confidently say it’s statistically significant. If the test generates a data point within the 95% confidence interval then we consider it to be statistically insignificant.

Check out this post by Avinash Kaushik if you want to read more about the calculation process of statistical significance.

Getting back to analyzing website traffic, we can leverage statistical significance formulas to determine the right time to begin making decisions based on visitor data. I refer to this point the action threshold – the number of actions that make a particular traffic source viable and significant.

It’s worth noting that this is an unconventional application of a significance test – a traditional use case is to compare two data sets for correlation – in this case I’m not asking how similar two data sets are, I’m asking at what point is a data set significantly different than a null case.

Consider a very simple example where we have 100 visitors and our goal is to get visitors to sign up for a free trial. In the calculator, set the control to 100 visits and 0 sign ups. Then in the test, set visits to 100 and sign ups to 1:



The result is not significant because the Chi-Squared value is below the 95% confidence value. Keep increasing the number of sign ups until the Chi-Squared value is larger than the 95% confidence value and the test will deem the results significant:


In this example, 4 sign ups represents the action threshold. This means that making decisions on less than 4 actions is premature. We run this test to prevent ourselves from making rash decisions caused by impatience and gut reactions. I suggest you run a similar test to determine your action threshold then sit back and collect traffic, wait until there are enough actions to deem the results statistically significant before hunkering down to dig through the data.

Opinions expressed in the article are those of the guest author and not necessarily Search Engine Land.

Related Topics: Channel: Analytics | Search & Analytics


About The Author: is the Vice President of Performance Marketing and Analytics at SellPoints and is based in the San Francisco Bay Area.

Connect with the author via: Email | Twitter | LinkedIn


Get all the top search stories emailed daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. Comments may also be removed if they are posted from anonymous accounts. You can read more about our comments policy here.
  • Benny Blum

    Quick follow up: this same test can be used to determine the effectiveness of an A/B test. Rather than setting the control to 0 sign ups and 100 visitors, just enter your actual conversion rate (ex: 8 sign ups and 1352 visits) and then the data for the test in progress (ex: 13 sign ups and 1191 visits). The calculator will tell you if the variance is statistically significant.

  • Delve Partners

    When dealing with uneven proportions, statistical power can be very elusive and lead to inaccurate T, Z or Chisquare results.

    Good writeup.



Get Our News, Everywhere!

Daily Email:

Follow Search Engine Land on Twitter @sengineland Like Search Engine Land on Facebook Follow Search Engine Land on Google+ Get the Search Engine Land Feed Connect with Search Engine Land on LinkedIn Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Search News Recap!

SearchCap is a once-per-day newsletter update - sign up below and get the news delivered to you!



Search Engine Land Periodic Table of SEO Success Factors

Get Your Copy
Read The Full SEO Guide