The Clickthrough Rate Equation In Organic Search

When I was in middle school, my favorite book and my favorite TV show were both Cosmos by Carl Sagan. I must have read the book at least 10 times, and I watched the series every time it was on the local PBS station.

One of the most interesting parts of Cosmos that has stuck with me is the Drake equation:

The Drake equation is an attempt to estimate the current number of intelligent civilizations in the Milky Way by breaking it down into component parts (such as “f(p), the fraction of stars that have planets” and “f(l), the fraction of planets capable of sustaining life”) and then multiplying them all together.

You can watch Dr. Sagan explain the Drake equation on YouTube. He pessimistically puts N at 10 (the early 80’s were a bummer, man) but then upgrades it to “millions” a less than a minute later (short term memory is also a bummer, man).

As I was talking to someone at SMX East a few weeks ago, it occurred to me that measuring conversions from organic search could be expressed similarly to the Drake equation like this:

In this version, C is the number of conversions, N(k) is the number of people searching for a keyword (or a group of keywords), f(I) is the fraction of searches where one or more links from your site show up (also called an impression), f(CTR) is the clickthrough rate from the search engine results, and f(conv) is the fraction of people who convert after clicking through.*

Then it occurred to me that a lot of attention is paid to three of these terms. Roughly speaking, N(k) is covered by keyword research, f(I) is a major goal of SEO, and f(conv) is in the realm of usability and graphic design.

Relative to the other three terms, clickthrough rate doesn’t get a lot of attention or optimization even though gains in clickthrough rate multiplies the effectiveness of these other factors. This is odd considering that most of the factors influencing CTR are within our direct control and won’t affect usability of the website at all.

So if we consider CTR as a highly leveraged but undervalued factor in converting users from organic search and one that is largely within our control, it is probably worth a column or two to take a high-level look at the various ways we can influence it.

The rest of this article only covers the title and snippet in search results and how they affect clickthrough rate. Next month in this column, I’ll cover many more.

The basic components of the search result is covered in an article by Vanessa Fox, so go check that out if you need a refresher or if you find some of the terminology is unclear to you.

Title & Meta Description

The most visible and largest components of the typical search engine result are the title and snippet. The title is generally taken from the HTML title tag of the page. The snippet can be taken from several sources, but ideally it comes from a well-written meta description tag.

Note that both the title tag and meta description aren’t generally visible when viewing the page in a browser (especially with the number of tabs I usually have open). This gives a lot of latitude in influencing the search engine results display, but it also gives enough rope to hang yourself if you aren’t careful.

Search Engines Overriding Titles & Meta Descriptions

In the example above, the snippet is pretty good. It’s descriptive, and ultimately was the result that I clicked on to refresh my memory on the topic.

However, when I checked the source code of the page to see if the snippet was pulled from the meta description this is what I found:

So while the title is coming directly from the page, the meta description clearly is not. This is some boilerplate text left in the page template. Because this text probably appears in many pages on the site and because it is clearly unrelated to the content of the page and because it’s too short, Google generated the snippet for this result from text on the page.

Usually the results aren’t that good, which is why it is important to pay attention to the meta descriptions of each page. Here are some of the other results for the same query, none of which gives me a good sense of the page:

Look at it this way: If you wouldn’t let a computer write your AdWords ads, then you shouldn’t allow a computer to write snippets for your site.

From the sites I’ve evaluated for clients, duplication of titles and meta descriptions are the main reason that they are ignored by Google or Bing, so it’s important to take care to make these unique for each page.

In the SMX East session about rich snippets, Jack Menzel from Google listed some additional reasons that Google might overwrite the title in a search result:

• The title is “unclear based on the query.” (I’m taking this to mean that important keywords are missing in the title.)
• If the title is missing the company or site name, Google may tack it on the end.
• If the title is “overoptimized” with keywords, Google may remove a few of them.

Jack was careful to point out that Google will only modify the title when they believe it is beneficial to users, but again, I think it’s important to retain as much control as possible over the way your pages are displayed in search results.

Another issue with duplication is the special case when the title and snippet generated are both identical. When this happens, Google will only show one result, suppress the rest, and show this message at the bottom of the search results:

This is a depressing message because it means that there are pages from your site that ranked for the query but won’t be shown because Google couldn’t differentiate it from the other page that ranked. (This message could also be an issue that your site has pagination issues, which should be dealt with accordingly.)

Placement Of Keywords Within Titles

When people are reading through the search results and deciding which one to click on, they are acting more like a monkey scanning a tree for fruit than someone sitting down with a glass of wine and a copy of Ulysses to ponder the classics.

This means people are scanning for the keywords that are already in their working memory (the current search), or — according to some theories — scanning for the general shapes of these keywords.

If you combine this observation with eye-tracking studies that show how people’s eyes trace around on a typical search engine result page, like this one and this one and this one, then logically follows that important keywords should be put at the beginning of the title where they are more likely to be seen by the monkey-scanners.

(I have heard arguments against putting keywords on the left, but I’ll leave this discussion for people who are more interested in human psychology than I am.)

Thoughts On Scale For Larger Sites

For sites with hundreds of thousands of pages, obviously it’s not possible to write unique and meaningful titles and meta descriptions by hand.

It’s okay to automatically generate these in a way that strongly encourages click through using metadata for the item(s) the page is about.

Here is an example I came across recently:

If I were looking for a home in Willow Glen, it would be hard for me not to click on this result. It’s clearly generated automatically from an application database but in a way that’s unique and designed to encourage clickthrough.

In a future article, I’ll cover other factors that can affect click through rate, like URLs, breadcrumbs, structured metadata, anchors, social signals, character encoding, the phase of the moon, etc…

*After writing this post, I realized that this is similar to the searcher persona workflow as described by Vanessa Fox in her book Marketing In The Age Of Google, so check out that excellent book to explore this concept further .