How Twitter Might Send Far More Traffic Than You Think

Over the past year, I’ve seen many people report that Twitter can send tons of traffic to a web site. Certainly I’ve seen first-hand how Twitter has become one of the top non-search referrer sources for Search Engine Land and for some other sites I oversee. But as much as you think Twitter is driving […]

Chat with SearchBot

Over the past year, I’ve seen many people report that Twitter can send tons of traffic to a web site. Certainly I’ve seen first-hand how Twitter has become one of the top non-search referrer sources for Search Engine Land and for some other sites I oversee. But as much as you think Twitter is driving traffic, it might be sending even more that you’re unable to measure.

One key culprit may be that a large number people view web pages using mobile Twitter applications such as Tweetie. Click on a link from a tweet in some of these applications, and they load web pages using their own browsers. Those browser may not run JavaScript. And if JavaScript doesn’t run, then some hosted metrics tools like Google Analytics never realize that your page was viewed.

Consider this recent example. I wrote an article on my personal blog about the collapse of the Clear airport security program last night and how it seems likely that alternatives to it will also go away (TSA Stays Silent As Its Registered Traveler Program Melts Down).

I tweeted that article using Bit.ly, a URL shortener that allows me to track how many people clicked on the link I tweeted:

Bit.ly Stats

The screenshot shows that Bit.ly had seen 343 total clicks to that URL through its system (when I did the screenshot), 339 of them coming from the shortened version I created. So Google Analytics ought to show me roughly that same number of people coming to the page from Twitter, right?

Nope. Google Analytics reports that Twitter sent 63 visits:

Google Analytics Stats

I know some visits from today have yet to be reflected in Google Analytics. That’s why you see the line drop. But I also know that those visits, based on my Bit.ly data, should be about the same as yesterday. In short, at best, Google Analytics is going to report I had about 130 visitors to the article from Twitter. But Bit.ly’s saying I had about 340 visits. Where are the missing 200 or so people? How come Bit.ly’s showing more than three times the visits overall to the page than Google Analytics does. I’ve seen this time-and-time again for various stories, sometimes with gaps of several thousand visits.

This is where the JavaScript stuff I’ve mentioned comes in. It seems to be the best candidate for explaining the discrepancy. Apparently, a lot of people on Twitter use mobile applications, or view from mobile devices, that aren’t processing the Google Analytics tracking code. There are also a few other reasons, as Bit.ly’s general manager Andrew Cohen explained to me in email:

Good question. You’re right about embedded browsers that don’t run JavaScript. Clients that decode bit.ly links but don’t click through to the underlying content are another challenge. If you were to download a browser plug-in to automatically expand short urls, for instance, it would look a lot like a human user. We’d see a GET request from an API, return the underlying long URL, and increment the click count … but we wouldn’t be able to see the actual CTR [clickthrough rate].

Absent JavaScript on the page, it can be tough to distinguish between a decode event and an actual click-through. That’s why we think that bit.ly complements rather than replaces JavaScript-based packages like Google Analytics or Chartbeat. One precaution that we’ve taken, though, is to filter out HEAD requests:

https://blog.bit.ly/post/89178273/talking-heads

We’re also working on some predictive modeling techniques to screen out more bots and scrapers from our click totals.

You might be interested in this article from the MIT Technology Review — https://bit.ly/qrjrf — which shows that these kinds of metrics issues are an industry-wide challenge.

I plan to look at the issue more in the future, but I wanted to get a post out there now with my initial findings and see what others have to say (please comment below!).

Somewhat related, also consider reading this post from Bit.ly: Registered Applications And Better Click Referrer Data.

Just as some referrers might not be seen if JavaScript isn’t loaded, some of the ways people are following tweeted links outside of Twitter aren’t registering, because the applications don’t send any referrer information. So, you’ll see these as “direct” visits as if someone typed in a URL, rather than clicking on a link. But by working with vendors, Bit.ly is trying to improve the picture (other tracking providers may also be doing the same, too).


Opinions expressed in this article are those of the guest author and not necessarily Search Engine Land. Staff authors are listed here.


About the author

Danny Sullivan
Contributor
Danny Sullivan was a journalist and analyst who covered the digital and search marketing space from 1996 through 2017. He was also a cofounder of Third Door Media, which publishes Search Engine Land and MarTech, and produces the SMX: Search Marketing Expo and MarTech events. He retired from journalism and Third Door Media in June 2017. You can learn more about him on his personal site & blog He can also be found on Facebook and Twitter.

Get the must-read newsletter for search marketers.