• Brian Lam

    Hi Bradd. For the Teasley calculator at least, it looks like it uses t-values assuming infinite degrees of freedom, which would definitely not apply in the example you described with only 4 impressions. As was mentioned in the first part of the article, you should have a fair amount of responses before evaluating the results or you could get misled into thinking one ad/homepage/headline is awesome when it really isn’t.

  • http://www.websharedesign.com JulieFB

    Hi Bradd. This is a great article. I am a statistician and I couldn’t agree more that there is a fundamental flaw with the tools you mentioned. I’ve written a PPC ad split testing tool that does take low sample size into consideration. You should check it out. In your scenario with only four clicks this tool will not provide any recommendations and will tell you to collect more data. http://j.mp/9zUeor

  • smec

    Hi Bradd,
    full ack – there are a lot of naïve implementations out there – inappropriate use of statistical tests, which require more samples (e.g. chi square test would be inappropriate here, but even a chi sq. test would not pause any ad in your example).
    Your example CTRs: our own ecommerce optimization software would spit out a p-value of 0.4 => far away from the common 0.05 (or 95% prob.) threshold.

  • http://searchengineland.com Bradd Libby

    JulieFB, your ‘PPC Ad Split Testing Tool’ looks and seems to work great. The FAQ is very informative, the ‘Show me details…’ link provides useful information when a test is successful, and the explanation when the test fails is informative.

    One recommendation: You should set up your testing tool (and your company’s other tools) on a separate domain with a simple, catchy name. ‘smec’ is right; there are a lot of low-value tools available and yours look like they are are worth being seen by a lot more people in the industry.

  • http://www.cpcsearch.com Terry Whalen

    Brad, great article – I’m going to link to this from my blog. Thank you.

    Now, it’s time to check out JulieFB’s ad testing tool(!)

  • http://www.cpcsearch.com Terry Whalen

    Julie, the tool is great. The FAQ is very thorough – I wonder if you may want to add the importance that ads being evaluated should have been run over the same period of time, with equal impression rotation. If folks take 2 ads where 1 of the ads had most of the action in week 1 and the other ad had most of the action in week 2, statistical differences may lead to wrong conclusions based on other things going on with bids and keywords during the 2 different time frames.

    Also, as a side note, in order to make sure ads get equal rotation for accurate testing, remember to set ad-serving to rotate (settings tab in adwords)!