Paid And Organic Search: Understanding The Math (And The Truth)

In last month’s column I talked about buying brand keywords and some of the great (somewhat) new ad products available to advertisers. All that talk about buying brand keywords brought me back to a familiar topic about which, if you read my post with any frequency, you just might be getting a tiny bit tired of hearing me rant. In spite of that, I’m back for one more round of discussion about the relationship between paid and organic search traffic, this time with some real math to show for myself. Bear with me, this one’s worth it.

I mentioned in my previous post about paid and organic search traffic that the goal in any analysis is to determine if the paid ad is a source of lift, cannibalization, or both. At the time I was looking at isolating the traffic from the organic listing and then quantifying the incremental effect of the paid search ad. Since then, thanks to Matthias Blume, I have seen the light, statistically speaking, and have thus found a much more elegant way to conduct this analysis. Let’s put on our statistical hats and go on a journey to the heart of the issue.

Let’s reset the situation: You rank #1 organically for your brand keyword search, and you’re trying to determine if and how much you are willing to pay for a paid ad on your brand keyword search. The goal then is to understand how the clickthrough rate (CTR) of the organic listing is affected by the presence of the paid ad. You’ll do this by looking at how the CTR of the organic listing changes with and without the presence of the paid ad. The CTR rates I am referring to need to be normalized for search volume. I’ll explain what that means below, but that was the tricky part that I didn’t get before, the elusive reality about which Matthias enlightened me.

To calculate this normalized CTR, you’ll first need to try to figure out what the search query volume is for the keyword you’re targeting. This isn’t always explicitly defined, but if you’re clever you can find some ways to approximate it with a good deal of accuracy. Some search engines provide approximate monthly search query volume for keywords, and I believe there are third party providers of approximate search query volume data as well. Ideally you will want to have search volume by day for whichever keyword you’re testing on a particular search engine. If you can’t get to this level of resolution that’s OK—just go with weekly or monthly search volume. This will mean that it will take longer to get your results.

I’m going to explain how we did our analysis, and you can make any variations you need based on your own constraints. First, we chose a few brand keywords to track. At Yahoo!, we have many brand terms to choose from: [yahoo], [yahoo mail], [yahoo finance], etc. At your company you may have a similar portfolio, or you may have only one brand keyword that really matters to your business.

Here’s how the analysis works: on day 1 of our research period, we had the organic listing in #1 position, with no paid search ads (at all) on the page. We began to collect referral data from the organic listing. After a week we started buying paid search ads for our brand terms, and began tracking referrals from the paid ad as well. From then until day 30 we bought paid search at different budget levels and turned it off for periods of time as well, just to get a variety of data points.

At the end of the research period we compiled our data and began to plot it out on a chart. The metrics that we calculated were fairly simple. On any given day, organic CTR is [referrals from the organic listing divided by search query volume], and paid search CTR is [referrals from the paid search ad divided by search query volume]. We calculated these two metrics for each day 1-30, and plotted them on a scatter graph. It looked like this:

Here the way to read this graph: On the X (horizontal) axis is the paid search CTR. On the Y (vertical) axis is the organic CTR. Each of the data points represents a day where we gathered data, and each point is plotted where the two CTRs meet on the graph (I’ve stripped the values out of the chart to protect the innocent). The line on the graph is a linear regression, basically a trend line that represents a summary of the scattered data points.

Here’s the key: If the slope of the line is positive (the line goes up and to the right) as it is here, there is positive synergy between the organic listing and the paid ad. This means that on days where we bought the paid ad, the CTR of the organic listing actually increased. If the line had a negative slope (went down and to the right), there would be negative synergy between the organic listing and the paid ad. I referred to this as cannibalism in my previous column on this topic, and it means that your paid ad is stealing traffic from your organic listing.

So in our case I ran around the building, loudly declaring victory. I mean, what could be better than buying paid search traffic knowing that you’re driving more clicks to your organic listings? Believe me, I’m not saying that it will turn out this way in every case. Do your own analysis and come to your own conclusions, because naturally it’s going to work differently for everyone. I’m just here to share my story with you, and mine happens to have a very, very happy ending.