Libyan Registrar Takes Back Domain

Earlier this week, Nic.ly — a domain registrar in Libya that handles Libyans domains — took back the vb.ly domain from its American owners, after it was in use for a year.

Reason? The owners, Ben Metcalfe and noted sex blogger Violet Blue, were told the domain was violated Nic.ly’s terms as it was being used to point at adult content. From Blue’s blog post, citing what Nic.ly told her:

Pornography and adult material aren’t allowed under Libyan Law, therefore we removed the domain, and before doing so we warned you thru our Resellers and gave you a relatively long grace period to rectify your situation. Being that you didn’t receive/ ignored our warnings is your problem not ours

When we have an out cry from within our Community and even from places as remote as Morocco (a sister Muslim and Arab state) asking us how such a ‘scandalous’ domain is allowed to exist under our National extension we are left with no option but to apply the rules. I invite you to conduct a simple search to see if domains such as (what was) yours are allowed to exist under the ccTLD of other Arab and Muslim Countries.

They don’t. Why should Libya be the exception?

Metcalfe also blogged about the situation here.

Nic.ly: It Was A Porn Site, Not A Shortener Service

Nic.ly has also put up its own post on the situation today to suggest that where vb.ly went wrong was being “different” that other shorteners by focusing on adult content:

In reference to the vb.ly incident: the domain’s purpose (proclaimed by its registrants themselves) was to serve as a ‘sex friendly URL shortener’, mainly for adult uses. This means that vb.ly had a policy different than the other URL shorteners, not using filters and encouraging the use of this service for creating links to adult sites and other “NSFW” links, thus placing vb.ly by definition in the porn/adult site category.

This use was deemed as unacceptable by our local internet community, regardless of whether or not the site hosts adult material or redirects traffic to 3rd party sites

No Porn.ly, No Problem?

Effectively, Nic.ly calls vb.ly a porn site. That’s important, because it implies that the biggest shortener using the .ly domain, Bit.ly, won’t have similar problems. Since it (or other domains using .ly) aren’t trying to promote the shortening of adult content, it is seemingly in the “safe” category.

Moreover, looking at Bit.ly’s terms, the service does have rules against “pornographic” or “obscene” content. So technically, if Bit.ly found itself in the same situation as vb.ly, it could point to its terms to say it complies with the spirit of Libyan law and might look to do a better job in filtering (to my understanding, it currently doesn’t try to filter out adult content at all, only malware or similarly harmful links).

Time To Say Goodbye To .Ly Shortening, Hello J.mp

Still, it all leaves me unsettled. What’s to prevent a Libyan registrar down the road from deciding that just pointing at any adult content through your domain, even if that’s not your main purpose, is a violation worthy of pulling your domain?

The reality is that I can’t recommend anyone using the .ly extension any longer. In the case of Bit.ly, they also run the J.mp service. It works just like Bit.ly does, but the domain name is out of the Northern Mariana Islands. I’m going to trust them more than Libya, at this point.

Twitter Clients: Support J.mp & Goo.gl!

That also means I’d like to see Twitter clients ensure they are supporting J.mp as well as Bit.ly. In my Twhirl client from Seesmic, I’ve got no option to use J.mp. The same is true for TweetDeck. Adding in the new Google shortener also makes sense. Goo.gl makes use of the GL domain from Greenland.

Personally, when I had to pick a short, custom domain for Search Engine Land, I went with .com. Even though selnd.com meant I gave up a few characters, I had the security gained from my own name, using the long-standing .com out of the US.

I did ask Bit.ly about these issues, by the way, but the company wouldn’t comment on the record.

For more about URL shorteners, custom domains and more, see these previous articles from me:

• URL Shorteners: Which Shortening Service Should You Use?
• Google URL Shortener Opened To The Public; Comparing To Bit.ly & Twitter

Also see related stories here on Techmeme.

Just what the world needs, another URL shortener, right? Google seems to think so, and it’s now making its own Goo.gl service widely available to anyone — complete with tracking and statistics — for free.

Below, more about the new entry, how it compares to the more established Bit.ly plus a look at Twitter’s “wrapping” service.

Dashboard To View What You’ve Shortened

Google began offering its own shortening service last December, saying it was something they especially thought Google Toolbar users and those using its Feedburner service would want. Now it’s being offered to anyone much more easily through a dedicated web site. How’s it measure up?

I’ll compare it to the shortener I’m most familiar with, Bit.ly, one that I also consider the market leader. As with both, you can keep a running record of all URLs you’ve created.

Here’s my dashboard from Google’s URL shortener:

Here’s my Bit.ly dashboard:

In both cases, I have a record of what I’ve shortened. That requires having a Bit.ly account at Bit.ly. With Google, it requires signing in using your Google account. If you don’t sign-in, you can still shorten URLs, but Google won’t keep track of what you’ve shortened and provide easy access to statistics.

Drill Down For Stats

In both cases, you can click into a shortened URL to see who has been clicking. For example, here are stats for this story that you’re reading, which I shortened and tweeted:

It’s a nice collection of basic information, showing me things like:

• Countries of those clicking
• Browsers they used
• Operating system they use
• Where people were referred from (such as seeing the shortened URL at Twitter.com or in a Twitter client like HootSuite)

You can also see the number of clicks generated from the URL you’ve put out and over various time ranges, from two hours, through the past day, week, month and “all time.”

Bit.ly provides the same type of stats. I like that Google presents them in an at-a-glance view versus Bit.ly, where you have to click tabs in the “Traffic” section to drill in:

But still, the same data is there. Google gives you a “past day” time range option that Bit.ly doesn’t have. But Bit.ly gives you a “Conversations” view that I find helpful, a list of anyone who has tweeted your shortened URL and what they said:

Same Page, Different URLs

A great thing that both Google and Bit.ly do is allow you to shorten a URL and track the clicks you generate, using your unique short URL, versus the total number of clicks to the long “destination” URL that have gone through each system.

That make your head hurt? Consider this. Here’s the URL of Google’s blog post today on the new shortening service:

http://googlesocialweb.blogspot.com/2010/09/google-url-shortener-gets-website.html

When I shortened that URL and tweeted it, this is the short URL I was given:

http://goo.gl/dWpW

But Google also tweeted the blog post, with this URL:

http://goo.gl/k4Gj

That’s at least two short URLs leading back to the same page. How many clicks did my tweet drive, versus Google’s own shortened URL plus other shortened URLs that might be out there? Here you go:

In the “Clicks” section, you can see that I’m told my shortened URL drove 60 clicks out of the total 3,347 clicks to the destination URL (at the time I made this screenshot).

As I said, this is something that Bit.ly already has offered, too:

No Custom URLs Or Custom Domains

What’s missing? The ability to create custom short URLs. Google offers nothing like this, while Bit.ly offers three different options.

The first method, which is free, allows anyone to add a custom ending to the bit.ly domain (or to the j.mp domain that Bit.ly also operates). For example:

In the example above, I gave Bit.ly the address of today’s blog post from Google. It generated one of those usual URLs that are a mishmash of letters and numbers. But then by using “Customize, I was able to select a custom ending as shown:

http://bit.ly/shortgoog

Of course, if anyone else had already used the “shortgoog” ending, it wouldn’t have been available to me. But it’s a nice option to have.

Bit.ly offers an even nicer option, also for free, where anyone can have their own custom domain. Those who take our Twitter feed may have seen how we shorten URLs for Search Engine Land in this way. All our URLs use the selnd.com domain, like this:

http://selnd.com/aVOcLh

That URL leads to this page, this story that you’re reading now. As a publisher, I like it, in that it allows me to maintain my own brand. Other publications do similar things (either through Bit.ly, through other services or on their own), such as:

• Los Angeles Timees (lat.ms)
• Foursquare (4sq.com)
• PBS (to.pbs.org)
• New York Times (nyti.ms)
• TechCrunch (tcrn.ch)

The custom domain service above, while free, is offered through Bit.ly Pro. There’s also an enterprise version of Bit.ly Pro that offers what’s called “end-to-end branding.” This means that if you have a custom domain, if anyone else tries to shorten your URL using Bit.ly, they’ll be given a shortened URL using your domain.

You can try this with any of our stories or any of TechCrunch’s stories, to see it in action. Both sites have end-to-end branding. Make a short URL to a story, and you’ll get either selnd.com or tcrn.ch the domain.

The enterprise version is pricey, $995 per month, but Bit.ly has provided it to us so far for free (we were one of the original beta testers). If we had to pay, I doubt we’d spend that much (other benefits also come with it). But for bigger publishers, I can see it being well worth the price.

What About Twitter’s Shortener, Er, Wrapper?

At this point, some readers may be wondering about Twitter. Doesn’t it have its own shortening service? Doesn’t that make whatever Google or Bit.ly or anyone else does sort of superfluous? No.

Technically, Twitter offers a “wrapper,” not a shortener. That’s because in some cases, it might actually make a URL slightly longer (such as if you try to shorten an already short domain name, like cnn.com). More important, it’s because Twitter tells me it sees one of the service’s main functions as putting a protective wrapper around URLs rather than trying to shorten them.

In March, Twitter rolled out its wrapper, which used the twt.tl domain. This was originally implemented to change URLs that were in direct messages or in email notifications and done, Twitter said, for security reasons.

Twitter Gets Its Own URL Shortener To Stop Scams; Good Marketers Need Not Fear explains more about what happened. Any branding, as with the custom domain situation above, was lost. But any link credit for search engine purposes was kept (more about this below).

In June, Twitter announced that a different domain would be used (t.co) and that all links would eventually be “wrapped” with this. That was supposed to happen by the end of summer but to date, it still hasn’t been fully implemented. Two reasons were given for this change: security and better tracking for advertising purposes.

What Wrapping Means

To understand wrapping, let’s say you take an article from Search Engine Land, like this long one for the story that you’re reading:

http://searchengineland.com/google-url-shortener-opened-to-the-public-51964

You shorten it through Bit.ly, which applies our custom domain like this:

http://selnd.com/9DTLtc

Then you tweet that URL. When Twitter sees the tweet, when wrapping is fully implemented, the original URL will be rerouted through the t.co domain, like this:

http://t.co/oOsb16s

What you see in your tweet stream will depend. At Twitter.com, Twitter’s goal is to still show some sense of the original URL, and that’s what supposed to happen to those using the new Twitter interface that is rolling out.

Wrapping In Action

Consider this tweet that I did:

There are three URLs in that tweet, all of them leading to the same page — this article that you’re reading. The first I made using the Twitter wrapper, the second using Bit.ly and the third was the original URL to this page. The tweet was 138 characters, in all.

If you go to the tweet directly, regardless of whether you have the new or old Twitter interface, that’s what you’ll see. But if you view it using the new Twitter, as it appears in your tweetstream, it looks like this:

Look at what happened to the t.co URL. It’s been expanded. It changed from:

http://t.co/oOsb16s

to:

searchengineland.com/google-url-sho…

Even though I tweeted a t.co URL, that URL was “unwrapped” to show me some of the original URL. Twitter says it is doing this so that people will have a better idea of where they are heading to (and Bit.ly tells me Twitter is encouraging developers of Twitter applications to do the same).

Meanwhile, the click itself will still go through the t.co domain, like this:

http://t.co/oOsb16s > http://searchengineland.com/google-url-shortener-opened-to-the-public-51964

Also notice that by unwrapping the URL, Twitter actually shows a tweet that becomes 155 characters in all. The original tweet was still 140 characters — tweets can’t be entered that are longer than this. But what’s displayed can go longer, in this unique situation.

Again, things are still in a bit of flux right now. You’ll still see t.co domains “displaying” at Twitter.com, but that’s supposed to change in the near future.

Advertising, Tracking & Safety

Back to the advertising and safety aspects. On the advertising and tracking front, what Twitter’s doing make a lot of sense. It something other companies like Google have long done. Click on any search result in Google, and it registers that click and redirects you through a system that records it:

In the example above, you can see that our “searchengineland.com” domain is displayed when our site is listed at Google. But if you click on the blue hyperlink, you’re actually routed through Google’s URL processor, which tracks the click and then sends you on your way.

This tracking gives the company valuable information about how people are interacting with its service. Heck, many “mere mortal” web sites use outbound link tracking. To me, it’s hard to begrudge Twitter doing the same.

On the safety front, this is a nobrainer. If a malware URL gets tweeted, by using wrapping, Twitter can stop that dead in its tracks. It can simply turn off the redirection and protect many people. Sure, there’s the possibility that Twitter might abuse this feature. I think that’s a very slim worry.

Back To Google: Why?

Now let’s head back to Google. Why do this? Google gives three reasons in its blog post:

• Stability: People want a stable URL shortening service
• Security: People want a safe service
• Speed: People want a fast service

On stability, my initial thought was that we had plenty of good services out there already. But looking through my review from last year of URL shorteners, two of the three top choices (tr.im and cl.igs) have died.

Bit.ly still seems to thrive, of course. And Twitter offers its wrapper, that can also be used to shorten. But Twitter’s not offering tracking nor an easy standalone service. Ow.ly is backed by Hootsuite, and I like the stats within HootSuite that it provides — but as a standalone service, I don’t see stats and find it clunky to use. As for Awe.sm, it seems to offer nice features, but only for those who pay.

So yes, I can see where Google is filling a need here. I have little reason to fear that a URL shortened through Google is going to fail to work in a year, something that was a real concern when I did my review of URL shorteners last year (and phew, my top choice of Bit.ly was the one that survived!).

On the security front, Twitter’s plans mean that Google’s not bringing anything to the table. Twitter will secure links shortened through its own service, thank you very much. Of course, people do shorten URLs for use elsewhere.

As for speed, I’ve just not heard many complaints that Bit.ly was somehow resolving slowly. Google did email me a link to a TechCrunch article about how a third party firm found that its service was much faster than others. OK, but anecdotally, I just don’t hear many complaints on this issue in general.

Google’s said in the past that it doesn’t want to launch “me too” products. I’d say this is 50% “me too.” It offers basically what Bit.ly is already providing, not bringing anything particularly new or unique to the table. But as I’ve said, with all the consolidation on the standalone URL shortener front, I’d say it’s nice to have this additional option.

But Really, Why?

I talked with the head of Google’s web spam team, Matt Cutts, a bit about the reasons behind the service. Sound odd to be talking to him about Google’s URL shortener? Well, the service began as a “20% time” project by engineers in his group, time engineers at Google are allowed to spend working on whatever they’d like. So Cutts is knowledgeable about the project and serving as one of the de facto spokespeople for it.

Why do this, in his words?

“If people are that excited about it, we figured why not open it up to the world,” Cutts said, explaining that about 70,000 people were making use of an extension in the Chrome browser to shorten URLs. Cutts also reiterated the safe, speedy and secure mantra of the official blog post.

And is it taking aim at any particular product?

“By just showing up at the table and providing a really nice service, you can motivate everyone to improve in the space,” he said. “We definitely don’t want to kill anyone.”

And benefits in terms of helping Google better track links? Cutts wouldn’t go into that. But the benefits are obvious. Seeing what people are clicking on in tweets is useful information for improving real time results, as well as search results in general. Google can’t see much of that data now. But if more people use its own URL shortener, suddenly, it does. Offering one definitely puts Google at the table for any Twitter client maker that’s looking for a shortener option.

Bit.ly’s Response

Meanwhile, how’s the reaction from Bit.ly, which has watched Google suddenly roll into a space it dominated? John Borthwork, CEO of Betaworks, which owns Bit.ly sent me this:

Like I said to Marshall K. the imitation is flattering, sincerely. Google are a great company who understand workflow and user needs very well. The team at bit.ly work very very hard to understand what users want. The fact that Google copied the basics of bit.ly says they think we got most of it right. A complement indeed.

We have come a long way since Goog first launched Goo.gl. We now have 4000 pro customers, and over 20k in the pipeline, from Yahoo, to Amazon to NPR to Search Engine Land.  All use bit.ly’s white label service. 36 billion links clicked on so far this year, 4 billion+ URLs in total have been shortened.

That’s a lot of data to manage in real time, and we have done all of that for years now, without a hiccup. Users and publishers — everyone should have a choice of what service to use, and who they want to have access to their data.

More Info & Link Juice

Looking for more information? Techmeme has a compilation of related stories from others you might want to check out.

Cutts has done his own write-up on Google’s service, which you’ll find here. Google’s official announcement, in case you missed me mentioning it several times above, is here.

TechCrunch highlights how you can also use the service to make those incredibly ugly QR codes for your pages, like this one for this page:

Don’t know what to do with these codes? See these articles from us:

• What Is A QR Code And Why Do You Need One?
• QR Codes: Are You Ready For Paper-Based Hyperlinks?

Also see URL Shorteners: Which Shortening Service Should You Use?, a story I wrote last year. While it’s becoming dated in terms of services, and I plan a newer version in the future, it answers a lot of basic questions about what to seek in a shortener. It goes into more depth especially about the importance of using a service that does “301″ redirects, to ensure your get credit for links that are tweeted. Google’s service does this.

Finally, these articles explain more about how tracking services can help you better see how much traffic Twitter is driving to your site:

• How Twitter Might Send Far More Traffic Than You Think
• Is Twitter Sending You 500% To 1600% More Traffic Than You Might Think?

How Twitter Might Send Far More Traffic Than You Think is my earlier article that explains how I’d often seen big gaps in how many people apparently clicked on a tweeted link as measured by Bit.ly versus how many page views that Google Analytics was showing.

To test this further, I tweeted a particular page on my personal blog along with tracking code designed to especially help ensure it appeared in Google Analytics. I’m going to toss out a bunch of numbers as part of this analysis. If they get confusing, skip to the end for the conclusion.

The Numbers Bit

For July 7, Bit.ly reported that the page had registered 58 clicks. Were there 58 corresponding page views? No. Google Analytics only reported 17 page views from 11 unique users. That meant a gap of 41 views.

Was the gap due to clicks from non-human robots that don’t process Google Analytics JavaScript tracking code? Visits from people using mobile browsers that didn’t get tracked, because they might not process the code? To explore further, I went to the raw log files, the records that the server itself keeps. These shows any request made for the page, regardless of any JavaScript issues.

I found that there had been 57 total requests — practically the same as Bit.ly reported. However, 14 of these were for the page without the tracking codes I’d used when tweeting the page through Bit.ly.

In other words, this is the URL I put out through Bit.ly, which was reported to receive 58 visits:

http:///mothers-cookies-closes-the-sadness-for-products-i-no-longer-have-389?utm_source=twitter&utm_medium=micro-blog&utm_campaign=twitter

See the part in bold? Those are tracking codes or parameters. From the log files, I found that URL above (with the codes) had 43 visits (not 58) and the same page without tracking codes like this received 14 further visits:

http:///mothers-cookies-closes-the-sadness-for-products-i-no-longer-have-389

Those 14 visits without tracking codes almost all came from robots (Google: 5; Yanga: 4; Microsoft: 3). Two other visits seemed to be from humans. These robotic visits all likely had nothing to do with my tweet. The requests were from spiders doing their regular crawls of the web, it seems. The few human visits to the page without tracking codes were probably people who came to my article for reasons unconnected with the tweet.

What about those other 43 visits to the page that did have the tracking code? Well, 11 visits were from what appeared to be robots (OneRiot: 1; PycURL: 2; Ginxbot: 2; WebShot: 1; Google: 2; Tweetmeme: 1; Python-urllib: 1; LongURL API: 1).

That left 32 visits that appear to be from humans. That’s almost between the 58 views Bit.ly reported and the 17 page views Google Analytics reflected. Why still such a gap with GA?

One leading argument has been that some Twitter applications on mobile devices load pages within the application, rather than using an external browser, and so aren’t getting registered by Google Analytics. Also, some mobile browsers might not process JavaScript. I could see at least four iPhone-based requests like this. But there were plenty of other requests that appear to be from full-fledged desktop-based browsers. Why weren’t they showing up?

One clue is that of the 34 requests, only 5 of them contained “referrer” data, information that some browsers pass on that indicate how they found the page in the first place. For Google Analytics (or ANY analytics program) to properly indicate how much traffic a particular site is driving, it needs as much referrer data as it can get.

Of those referrers, only 2 of them were from the twitter.com domain (1 more was from my own blog’s domain, 1 from iconfactory.com/twitterific, probably indicating a Twitterific users, and one from powertwitter.me, probably indicating a Twitter-visit via a Firefox plug-in).

In short, based on referrer traffic alone, ANY analysis program would have reported that at best, Twitter sent my page only 2 visits. Yet, both Google Analytics and Bit.ly reported that it received far more than that.

Remember, Google Analytics said the page had received 17 views in all, 11 from unique users. How many of those 11 unique users came to the page via Twitter? Google Analytics said 9. One more came directly, it said; another person did a search to find it (mother’s cookie site:daggle.com was the search, which was me locating the article. Oddly, this request does NOT appear in the raw log files).

The Big Conclusion

All those earlier numbers hurt your head? Here are the most meaningful ones. Thanks for hanging in there!

Based only on referrers, at best, Google or any analytics program would have said Twitter sent 2 visits. But because I used tracking codes, I was able to overcome the lack of referring data and see that Twitter (itself or via applications or web sites using Twitter data) sent 9 visits. That means analytics packages might be undercounting Twitter visits by nearly 500%.

Meanwhile, Bit.ly was showing those 58 clicks to the page. Let’s say it wasn’t filtering out some of the robots. I can still see that there are 32 visits that the log files recorded, all with the tracking codes that never existed until I tweeted the link with them. So those are all Twitter-derived visits. That means an undercount by a standard analytics tool depending on referrer data by 1600%.

And The Analytics Companies Say?

I sent my logs to both Bit.ly and Google, along with a draft of this article, for any reaction.

Google said they’re aware that activity on mobile devices can cause issues with tracking and that they’re looking for ways to improve their product.

Bit.ly said they filter out robotic clicks such as Ginxbot, Google, and Python-urllib, through PycURL. When I asked further about the gap, they emailed back:

It looks like three types of events make up the delta.

First, browser plug-ins and automated url-lengthener applications, which make requests to the bit.ly URL, but don’t follow the redirect to the destination site.

One example is the “eventBox” at http://thecosmicmachine.com. Here’s how it appears in the logs:

(eventBox) : – - [07/Jul/2009:20:41:31 -0400] “GET /cHXSP HTTP/1.1″ 301 410 “-” “EventBox567 CFNetwork/438.12 Darwin/9.7.0 (i386) (iMac9%2C1)” 301

Second, small bots that make their way through our screening system:

(slicehost): – - [07/Jul/2009:21:05:43 -0400] “GET /cHXSP HTTP/1.1″ 301 410 “-” “-” 301

Third, browsers which don’t support JavaScript, as well as browsers with JavaScript settings turned off and browsers running JavaScript-blocking extensions like noscript.

And Some Related Reading

Last week, Fred Wilson posted Does This Blog Get More Traffic From Google or Twitter?, finding that for his personal blog, Twitter traffic has risen past Google search traffic. Fred suspected that the Twitter traffic was even more than being shown, due to undercounting. I think he’s right. While I think Google search traffic still remains a major traffic driver for many sites, those who have lots of Twitter followers or have a story go “hot” through retweets certainly may discover Twitter is a new major traffic resource –and one that’s likely undercounted.

Over at the Zebu Blog, Link Tracking – (lies, damn lies &) Statistics? also looks at the issue, questioning whether Bit.ly is overcounting. In a follow up comment, Mayank Sharma did his own small scale experiment and found:

We created a bit.ly url for this post, and posted it on Twitter. The next instant we saw, that bit.ly’s count was already 4. This only means that some twitter crawler/indexer received the tweet and de-referenced the url mentioned in it. After that I hovered my mouse over the link shown in Twitterfox. Sure enough bit.ly’s count increased by one. We did this repeatedly from multiple desktop’s of several friends and the count just kept on increasing. Not one of these folks during this time had actually clicked on the link.

I agree — Bit.ly seems to be overestimating views. But Google Analytics seems to be underestimating them, perhaps severely based on my small scale log analysis program. Using tracking codes occasionally is one way to get a reality check.

Finally, if you want to add tracking parameters for URLs you tweet, consider the Snip-n-Tag add-on for Firefox. I’ve been using it, and it makes adding these to URLs super easy.

One key culprit may be that a large number people view web pages using mobile Twitter applications such as Tweetie. Click on a link from a tweet in some of these applications, and they load web pages using their own browsers. Those browser may not run JavaScript. And if JavaScript doesn’t run, then some hosted metrics tools like Google Analytics never realize that your page was viewed.

Consider this recent example. I wrote an article on my personal blog about the collapse of the Clear airport security program last night and how it seems likely that alternatives to it will also go away (TSA Stays Silent As Its Registered Traveler Program Melts Down).

I tweeted that article using Bit.ly, a URL shortener that allows me to track how many people clicked on the link I tweeted:

The screenshot shows that Bit.ly had seen 343 total clicks to that URL through its system (when I did the screenshot), 339 of them coming from the shortened version I created. So Google Analytics ought to show me roughly that same number of people coming to the page from Twitter, right?

Nope. Google Analytics reports that Twitter sent 63 visits:

I know some visits from today have yet to be reflected in Google Analytics. That’s why you see the line drop. But I also know that those visits, based on my Bit.ly data, should be about the same as yesterday. In short, at best, Google Analytics is going to report I had about 130 visitors to the article from Twitter. But Bit.ly’s saying I had about 340 visits. Where are the missing 200 or so people? How come Bit.ly’s showing more than three times the visits overall to the page than Google Analytics does. I’ve seen this time-and-time again for various stories, sometimes with gaps of several thousand visits.

This is where the JavaScript stuff I’ve mentioned comes in. It seems to be the best candidate for explaining the discrepancy. Apparently, a lot of people on Twitter use mobile applications, or view from mobile devices, that aren’t processing the Google Analytics tracking code. There are also a few other reasons, as Bit.ly’s general manager Andrew Cohen explained to me in email:

Good question. You’re right about embedded browsers that don’t run JavaScript. Clients that decode bit.ly links but don’t click through to the underlying content are another challenge. If you were to download a browser plug-in to automatically expand short urls, for instance, it would look a lot like a human user. We’d see a GET request from an API, return the underlying long URL, and increment the click count … but we wouldn’t be able to see the actual CTR [clickthrough rate].

Absent JavaScript on the page, it can be tough to distinguish between a decode event and an actual click-through. That’s why we think that bit.ly complements rather than replaces JavaScript-based packages like Google Analytics or Chartbeat. One precaution that we’ve taken, though, is to filter out HEAD requests:

http://blog.bit.ly/post/89178273/talking-heads

We’re also working on some predictive modeling techniques to screen out more bots and scrapers from our click totals.

You might be interested in this article from the MIT Technology Review — http://bit.ly/qrjrf — which shows that these kinds of metrics issues are an industry-wide challenge.

I plan to look at the issue more in the future, but I wanted to get a post out there now with my initial findings and see what others have to say (please comment below!).

Somewhat related, also consider reading this post from Bit.ly: Registered Applications And Better Click Referrer Data.

Just as some referrers might not be seen if JavaScript isn’t loaded, some of the ways people are following tweeted links outside of Twitter aren’t registering, because the applications don’t send any referrer information. So, you’ll see these as “direct” visits as if someone typed in a URL, rather than clicking on a link. But by working with vendors, Bit.ly is trying to improve the picture (other tracking providers may also be doing the same, too).