Mobilegeddon A Month Later: Small Business Study Shows…. Nothing Happened
Contributor Conrad Saam shares the results of a small-scale study looking at the impact of Google's mobile friendly update on local law firms.
We’re now a month out from the launch of Google’s purportedly “apocalyptic” mobile friendly update on April 21, 2015. The result? Across the industry, many are coming to the conclusion that the hype surrounding Mobilegeddon was overblown.
This post-Mobilegeddon yawn is echoed within the local results as well. We did a concentrated study of small to mid-sized legal firms, and after carefully sifting through data — 69 law firm websites, tens of thousands of sessions, 16 days, and even a two-tailed statistical significance model — we’ve come to the very painful conclusion that:
- Little, if anything, happened to these small businesses.
- This algorithm update was less interesting than a replay of the Mayweather-Pacquiao bout or the release of another Greatest Hits Album from Queen.
With the noted exception of all of those law firms who collectively spent a small fortune to get their websites mobile friendly, it seems that the mobile friendly update was a big bellyflop in the local small business market, at least where this type of business is concerned. (Mockingbird staff worked nights and weekends in May and April pulling some legal luddites into the mobile age. Given the lack of results, we now suspect that Mobilegeddon is a hoax cooked up by a bunch of out-of-work WordPress developers!)
Details Of The Study
We analyzed natural search traffic from mobile devices across 69 different law firms — 12 not mobile optimized and the remainder mobile friendly. We didn’t include the first week of results post launch (which also showed nothing happened) due to the fact that the algorithm wasn’t fully rolled out at that point.
We ignored that week one data to account for a slow algorithm roll-out as well as the erratic traffic fluctuations that frequently occur when Google tests their algorithm changes in real time. We pulled data for week two and compared it to a benchmark of average mobile weekly search traffic across 8 weeks of pre-April 21 data.
First Attempt To Demonstrate the Mobile Change Actually Exists
We first ran a test comparing the changes in traffic before and after the 21st across the two groups. Note that we have a reasonably small sample size — and some of these small business sites get very little traffic — so small variations can yield a large percentage change. In statistical terms, this means a large standard deviation. Not exactly an ideal data set.
We dusted off our stats text and ran a t-test to assess if there was a statistical difference in the performance between the two groups. How this works: Essentially, it calculates the expected actual distribution from two samples and then determines if the actual average of the two groups is different.
The result? By conventional criteria, this difference between the two data sets would not be considered statistically significant. (If you want to geek out with the stats, the two-tailed P value was 0.7889.)
Second Attempt To Demonstrate The Mobile Change Actually Exists
Every statistician knows that if you datamine hard enough, eventually you can find something that supports your hypothesis. (This is why I caution against marketers evaluating the efficacy of their own marketing, but I digress.) So, back to the Google Analytics we went…
Perhaps if we combined all of the traffic in each group and then compared the aggregate traffic change between mobile friendly and non-mobile friendly sites, we might see that the engineers at Google have actually been doing more than playing Segway Quiddich.
But, alas — not only was there just a 2% difference in performance between the groups, the non-mobile friendly group actually outperformed the mobile friendly group! And, if you go back to the original test above, we were just looking for a statistical difference between the two samples (not that one was higher than the other).
We have a PPC client with a non-mobile friendly site. I spent a significant amount of time in February and March aggressively pushing him to upgrade his WordPress site to a mobile friendly theme. I hate selling aggressively, but I was pretty sure this was going to be pretty devastating to his business. David, if you are reading this, there’s a bottle of scotch in the mail to you.
Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.