How To Analyze A/B Tests Using Google Analytics

Much has been said about A/B testing techniques and the value that Website Testing can bring to website owners. In this article, I will not discuss what to test or how to do it, but how to analyze what you have already done.

I have written in the past about the differences between A/B and Multivariate Testing and how to choose between them. Back then, I wrote that one of the advantages of multivariate tests is that you can analyze each and every interaction between elements in a much deeper level.

However, one of the points that I missed is that A/B Tests are easier to be analyzed using Web Analytics tools. Since every test version is a page, they are being tracked by default in your analytics reports. This is not true for Multivariate Tests since the versions are combinations of multiple elements, which are not served as a page but brought into the page by external tools.

In this article, I will go over a technique that can be used in order to make the most out of your Google Analytics implementation so that you can understand the bigger picture of your test (independently of which testing tool you are using).

Analyzing A/B Tests Results Beyond Success & Failure Metrics

Usually, when performing an A/B Test, marketers implement the codes on the original page, on the test page and on the conversion page. This means that the testing tool will track the number of visitors to each of these pages and report back the percentages of visitors viewing each page and their conversion rates. This approach misses two important issues:

1. Beyond Success And Failure Metrics: some websites (mainly ecommerce) may have completely different values to conversions, meaning that tracking success and failure as a binary decision might be misleading. As George Orwell wrote on the Animal Farm: “All animals are equal, but some animals are more equal than others.”
2. Engagement Metrics: sometimes the objective of the test may not be to reach a goal page, but to decrease the bounce rate, to increase time on page, or pages per visit.

Since very often the information above is already available in Web Analytics tools, it is logical to integrate testing results into them. So, if you are using Google Website Optimizer, Visual Website Optimizer or other tool of your choice, here is a way to improve the measurability of your tests using Google Analytics.

Setting Up Google Analytics to Track A/B Test Properly

Google Analytics content reports do not show a tab for goal conversion per page. That is understandable, since it is not possible to attribute conversions to specific content. Google’s solution was to create the $ index. Here is this metric’s description as provided by Google:

$ Index is the average value for a page that a user visited before landing on the goal page or completing an Ecommerce transaction (or both). This value is intended to give you an idea of which page in your site contributed more to your site’s revenue. If the page wasn’t involved in an ecommerce transaction for your website in any way, then the $ Index for that page will be $0 since the page was never visited in a session where a transaction occurred.

Since we are discussing A/B tests, by default you will be able to analyze the $ index of each of your test variations (if you have ecommerce this will be defined according to it; if not, be sure to set your goal values when setting your goals on Google Analytics).

But if you want more detail to be collected regarding your test variations, you can also use Custom Variables. Basically, to set a custom variable is to add a value to the user cookie that will persist across the life of the visitor’s cookie. This means that this additional segmentation capability will allow us to compare between pages, both when it comes to engagement metrics and when it comes to conversion metrics.

Here is a sample of the Google Analytics code that should be added to each test page:

body onload="_gaq.push(['_setCustomVar', 1, 'abtest', 'variation2', 1]);"

On the example above, we see the following:

1. The custom variable is set to slot 1 (out of 5): this important as you might overwrite other variables if they share the same slot.
2. The name of the variable is “abtest”
3. The value is “variation2”: each variation should have its own value. If possible, there should be an explanatory name in order to facilitate analysis, e.g. “big_picture”, “long_form” or “wonderful_graph”.
4. The scope is 1 (visitor level): this should be set to 1 as you want your returning visitors to still have the variable attached to them. If you use 2 or 3 that won’t happen.

Below is a sample of which metrics you will be able to use to analyze your A/B Tests.

Integrated A/B Tests

As Michal Nassimian wrote on A/B Testing for SEO: “Search Marketing should be viewed as a conglomerate of variables, where SEO, paid campaigns and Web Analytics are intertwined. Each effort to optimize a website should strengthen all others to increase synergy and boost revenues.”

I believe that by integrating Tests into Analytics tools we will be one step closer to an integrated optimization strategy.