• http://docsheldon.com Doc Sheldon

    Interesting, Siddharth. Analytics not being my strong point, I would never have predicted such a scatter as shown in your second example. Plotting out projections in Excel definitely sounds like a worthwhile exercise, before starting the experiment.

  • http://www.epiphanysolutions.co.uk SteveBaker

    Hi Siddharth,

    Unforseen events have wreaked havoc on many tests that we’ve run in the past.

    Most recently, we were running a test to improve the conversion rate on a client’s home page, but at the end of February, the agency managing their banner advertising found that they had some spare budget, and ramped up the traffic dramatically. Inevitably, this crashed the conversion rate, making any kind of significance test meaningless.

    Fortunately, as we’ve been caught out like this before, we tracked the performance of the various test versions for each major traffic source, so we were able to deal with it (this is a good idea if possible for website optimisation tests, since the results may be different traffic sources anyway).

    Advert tests can also be a problem for a PPC campaign. Clearly, changing your bids changes your positions, and hence your click through rates. As a result, we very rarely run advert tests in the formative stages of account optimisation, when changes to the bids are likely to be greater and more frequent…

    The assumption of stable means is obviously critical to an effective significance test, but in the ‘real world’, it tends to be the first thing to go…

  • http://bit.ly/VS_Blog Wilson Kanaday

    Siddharth –

    Would there be a way to use multiple regression analysis to look at the different variables affecting the test?

  • http://braddlibby.wordpress.com Bradd Libby

    “the statistical confidence measure (R squared) is 5.25% indicating that we are not confident that the regression is meaningful.”

    R-squared is not a measure of statistical confidence. It only says what portion of the variability in a given data set is explainable by the regression model. To assert anything else is appallingly ill-informed.

  • http://searchengineland.com Siddharth Shah


    Nice catch. R^2 being 5.25% means that only that much variability is explained by the independent variable. Actually the p values for that regression coefficient are well over 0.3. So my point about the confidence is still valid.

    Nice to see that you read my blog posts carefully. Very flattering !