It’s no secret that newspapers have been struggling with the disruptive innovations introduced by the internet, and this has resulted in some level of accusation towards Google and a circling-of-the-wagons mentality by the industry. But, what if they were to go in the opposite direction, with fuller engagement? Here’s one tactic for how to go about it via SEO.

In the last few years, I’ve visited a lot of newspaper websites for various projects. These sites are most frequently the online arms of what were once strictly printed local newspapers. When visiting these sites, I’ve been struck by the technical clunkiness of most—they’re typified by poor usability, layouts still closely influenced by traditional print newspaper layouts, dysfunctional on-site search engines, and content management systems hamstrung with badly-formed page templates.

Naturally, these sites are not optimized for search engines nor to make their content readily findable via search. It’s unsurprising that the sites are search-unfriendly. The newspapers probably feel highly conflicted in regards to search—the nostalgic desire for successes experienced in the past have made them grow unhappy with the internet paradigm, and they’ve worked each other up into a frenzy to hold Google responsible for their troubles. It’s hard to expressly invite a perceived enemy into your house on one hand while issuing invective against him on the other.

(I have also encountered newspaper sites which have optimized by some degree. But, these seem fairly few, and even some of them have only taken faltering steps in that direction. The exceptions are some of the biggest players such as the New York Times, Chicago Tribune, Wall Street Journal, Washington Post and other juggernaut newspapers—which are doing professional jobs at optimization.)

I’m very sympathetic to the dilemma newspapers are experiencing. I recall a time not long back when newspapers felt that internet yellow pages companies (”IYPs”) were as much of a threat as Google (see Local Media Face Growing Threat from Local Search Competitors Like Google, Overture and Yellow Pages, New AIM Group Study Reports and Search Engines Make Local Landgrabs, Leave Newspapers Out In the Cold), back when yellow pages companies had a considerable head-start over newspapers in online engagement and ad sales.

Since I used to work at an IYP, I also experienced firsthand what it was like to see a veteran print industry work to evolve to fit in the changing landscape while still being influenced strongly by legacy technologies. Technology wasn’t the only issue: organizational resistance toward seeing where things were headed, or even relatively insightful observations that there might be a risk in not engaging more aggressively also held the IYPs back. Since newspapers perceived the threat beginning such a long time ago, it’s disheartening to see that as a whole they have struggled to develop an effective adaptation for online—particularly the smaller, local market papers.

So, what’s to be done?

While there are a great many areas where online newspaper sites might improve and increase revenue prospects, one of the greatest untapped potentials on newspaper sites in my opinion is the news archive section. Even among poorly optimized newspaper sites, some articles may vanish into a walled-garden archive section at some point, going dark for search engines. Combined with very poor on-site search utilities, it’s as though these articles don’t exist at all for consumers.

I can’t count how many different newspaper sites I’ve visited where I’ve searched for articles which I knew existed, yet the on-site search engines could not locate them. In some cases, the “live” sites had search engines separate from archive search, yet offered no explanation to users as to which should be used and in what cases. Do articles pass into archive after one year? Two? Three? Why can’t the on-site search show them, regardless? In many other cases I’ve found articles by searching in Google, but the article is no longer available when I click through to the newspaper site, and searching within the site fails to reveal it. Did the article “expire” and pass into the archive graveyard or something? No messaging on the resulting error pages reveals this, nor suggests viable means for locating the article.

Newspaper folks: this is your main product! It’s all well and good to try to keep Google from making everything free and putting you out of business, but at this point there’s an even greater danger in locking away your content to the point where online consumers cannot even find—if a searcher doesn’t even know it exists, it’s certain they won’t be engaging with your site to try to obtain it, regardless of whether it’s provided “free” in return for ad impressions, in exchange for “free registration” or provided in return for some subscription fee.

How many articles are locked away in these old archives?!? It surely varies from newspaper to newspaper, but the potential numbers are staggering. While clicks on pay-per-click ads on newspaper sites may add up slowly, there’s no doubt in my mind that if newspapers dramatically expanded the content they have available to search engines, the clicks and associated revenue would increase. These newspapers must not realize the potential they’re sitting upon!

I acknowledge that current news is going to be the more popular content on newspaper sites, but there’s likely at least half as much traffic potential in the legacy content under the theory of the long tail. According to that theory when applied to newspaper website traffic, yes, there’s far more visits per contemporary news story than past ones, but the cumulative traffic from thousands and thousands of past news stories can equal or dwarf the traffic from the more popular stuff.

So, how should news archives be optimized for search?

Here are a few tips to get you started:

Optimize titles & headlines. This is one area where newspapers should utterly dominate! Reporters and editors often write beautiful article headlines which succinctly describe the topic and grab readers’ attention. But, the headline prose is squandered on some newspaper sites which either repeat the newspaper’s name for the TITLE text of all pages, or cram it up with the date, newspaper name and other “branding” messaging before the article title.

This amounts to almost criminal misuse of the title tag. The title is often the link text that’s displayed in search engine results when pages on your site are found to match the search term, and it’s displayed at the top of the browser window when a user visits the page. Also, in HTML there is a particular tag called the “heading” which is intended for just what it sounds like—used as way of identifying the heading and subheadings on a page, and it should be used when displaying article headlines. There are six different heading tags available (each uses different font attributes to add or decrease emphasis), but the main one you need to know is the <h1>, which is perfect for use in displaying an article’s headline on the page. The article headline should also appear at the beginning, not end, of title tags, and be displayed in H1 tags on the page for best usability and search engine friendliness.

Link to all of your content. To this day, search engines still rely heavily on links to pages to discover and index content. For good usability and crawability, I recommend designing a hierarchy of pages on your site so that users may click from the homepage to a page which provides a short list of top level links (such as links by dates or category of type of news story). Those top level links can link down to subcategory pages which link down further to each article ever published by your newspaper. Such a hierarchy of links is mainly for human site users to navigate down into all of your content, but it also helps the search engines understand the site structure, apply relative priority weighting of pages, and also to semantically categorize content.

Create and actively maintain sitemaps. Although the category pages I mentioned above are often loosely referred to as sitemaps, “official” sitemaps files (those created using a formal standard acknowledged by all of the major search engines) are lists of links to your pages that search engines use find all of your content. These should be used in conjunction with the hierarchy of links provided for human users. The sitemaps help insure that the search engines can find all of your pages.

Focus on creating evergreen URLs. Search engines tend to respect pages that have been around for a long while, and frequently changing the physical location of pages confuses both users and search engines alike. So, try to design article URLs which remain stable when an article is pubbed all the way through to when it’s archived for the long term. People also tend to link to articles, which helps search engines to decide how popular a page is. If you change the URL, then the link “votes” for popularity will no longer point to your article.

If you must change URLs, use 301 redirects. If your system is full of legacy processes which require you to change article page URLs once a current news article is moved off into the archive, then at least redirect the original URL to the final location instead of just delivering up an error page. Most users who click through won’t stop to poke around to try to find where something was shifted-to—they’ll just abandon your site to try to find info elsewhere. And, that redirection command should be a 301 “permanent” redirection in order to insure the search engines apply the original URL’s popularity weighting to the new URL.

Offer a “first click free” option. For those newspapers which require registration or subscription prior to showing archive content, read up on Google’s “First Click Free” program. This process allows Google to crawl your site content and index it, and a person who clicks through from Google can view and read the first page for free, but you can then require payment or registration for subsequent pages.

Create a subscription designation with Google. If you set it up with them, Google will allow you to have content crawled, but when users click through they must pay or register to see any of the article. This is less-preferred by Google since it’s a less satisfactory user-experience. If going this route, I’d recommend displaying a good-sized chunk of the article to users that click through, as a preview. In that way, they’ll feel a little less disappointed, and may be drawn in further to pay a subscription to see more.

Improve on-site search Just as a usability matter, consider using Google’s site search if your internal site search doesn’t work well. Once the pages have been optimized as I’ve outlined above, you could implement Google site search and perhaps improve your site’s overall usability.

There are certainly many other areas for optimization for newspapers, and this is not an exhaustive list of SEO improvements which could be done for articles. Simply exposing hidden and non-indexed archives would be a good start.

Google is trying to find additional ways in which to help the newspaper industry, such as its plan to roll out a micropayment system within a year. Micropayment is FAR better, in my opinion, than attempting to require someone to purchase a month-long or yearly subscription when they might want to access only a single article. I’d also suggest improving classified sections or partnering with many news sites for a multi-site subscription.

The local newspapers have lost a lot of marketshare to online news sites and aggregators, but evolving to improve their popularity and traffic in the internet economy could help them to take back marketshare and increase revenue. SEO helps with promotion and expansion of audience. If you’re a local newspaper in need of increased business, seriously consider beefing up your search engine optimization game.

Over the past year we have worked with a number of organizations that have chosen to relocate their sites from an existing domain to a new domain. One of the questions that always comes up early in the process is “how much traffic are we going to lose?” It is an excellent question and not an easy one to answer, but in today’s column I am going to explore that exact question.

Here are some of the types of changes that can have an impact on traffic or rankings.

Domain change. Any change in the domain, such as a move from http://www.old-domain.com to http://www.new-domain.com. The most common reason for doing this is a branding change of some sort. An existing business may be changing its branding, or one business entity may have been acquired by another one and the two sites are being merged.

Structural changes or URL changes. These are changes where the content that lives on a given URL on old-domain.com (such as about-us.html) gets moved to a different URL (such as about-us.php). URL changes can be “wholesale” (change nearly all or all of them), “heavy” (change a lot of them), “moderate” (change some of them), “light” (change only a few), or not done as all if you simply copy the exact site structure from one domain to another.

Structural changes often happen as a result of a change in the technology used to implement a site. For example, a business may have been using Cold Fusion as a content management system, and then switches to using ASP. The other major reason for structural changes is when wholesale content changes are made.

Content changes. Changes to the content on pages can happen without changing the URL structure of the site, by simply rewriting content on the pages, or something that causes structural changes to the site. As with URL changes, these can also be heavy, moderate, light, or not done at all.

Content changes may be made for many reasons. Perhaps the target audience has changed. Perhaps the basic positioning of the organization has changed. Another possible reason is to revamp the content as part of a wholesale expansion of the site.

Each of these things can happen independently. You can make content changes without changing the domain or the URLs. You can change the URLs without changing the domain or the content.

What are the true consequences?

You are going to lose traffic. That is a fact. Even if you only perform a domain change and preserve the exact same site structure and content, you will lose some traffic. In this simplest of scenarios you can minimize the amount of traffic loss by using 301 redirects from each URL on the old domain to point to the same URLs on the new domain, alerting the search engines that the new URLs are the important ones.

In principle, this simple domain change scenario sounds like one where there should be very little lost traffic. One factor to consider though is that of “trust.” Any time there is a domain change it may be reflective of an ownership change, even if the WhoIs info is not updated. For the search engines this raises the possibility that the new owner isn’t as trustworthy as the original owner.

Another factor concerns the 301 redirects themselves. In tests we have done at Stone Temple Consulting, we have seen evidence that they pass through the majority, but not all, of the link juice to the destination page. Sometimes there is a delay between the implementation of the redirect and when the search engines pass through the link juice, which can result in a significant drop in search engine traffic. In the medium to long term a simple domain change is usually not that costly (though there are exceptions). You may lose 20% to 40% of your traffic in the short term, and 10% to 20% in the medium to long term.

As you might imagine, the more complex the changes, the greater the potential negative consequences. For example, combining a domain change with URL changes will definitely be more costly. You have given the search engine more reasons to trust the site less, and your 301 redirect map just got more complicated. Assuming you completely restructure the site so all the URLs change, you can expect to see traffic loss of about 30% to 50% traffic loss in the near term, with gradual improvement on that over the longer term.

In our final scenario, if you change your domain, URL structure, and your content, you are asking for trouble. The big reason for the cost here is that the new content you create is not the content that people saw when they linked to your site in the past, even if it basically about the same subject matter. This probably results in the search engine significantly discounting the value of those links. Traffic loss in this scenario is likely to be 50% or more in both the short and long term.

To summarize:

Disclaimer! These numbers aren’t exact, and your mileage will vary. The actual impact of changes to your site will depend on many factors that are not possible to cover here. In addition, the chart assumes that you don’t do a lot of incremental link building to bolster rankings. However, savvy site owners rarely stand completely still.

How to mitigate the damage

Once you have made your changes, and assuming you have followed Google’s recommended best practices for doing so, the main damage control you can do is get new links to the site. In particular, if you can continue to get links at a pace similar to, or better than, what was happening before the move, this is a strong positive signal to the search engines that all is well.

Also make sure that you ask people who have linked to you in the past to update their links to go direct to the new site, bypass those pages where you’ve put 301 redirects in place. If a significant percentage of your past linkers do this it is also a very strong signal to the search engines that your site is still trustworthy in its new location.

The best remedy? Stop and think about the consequences of a move before committing. There will be a cost, and your business plan probably does not call for a dip in traffic to, and orders from, the web site.

The bottom line: Don’t make major changes to your site unless you really have to, and are willing to pay the price.

Google Webmaster Tools has just launched a “labs” section, where you’ll find new features that may be early in the development cycle and not quite as robust as the rest of the tools. The features available so far are Fetch as Googlebot, which lets you see exactly what Googlebot is served when it requests a URL from your server and Malware Details, which shows you malicious code snippets from your site if it’s been flagged as containing malware.

Fetch as Googlebot

Of most interest to webmasters, SEOs, and web developers is likely the Fetch as Googlebot feature. You can specify any URL on your site and see the HTTP response (header and contents) that the server returns. Simply  indicate the URL and click the Fetch button. It may take a few moments for Googlebot to access the page and return the results, since it fetches the page in real time. (Refresh the page to see the progress.)

Click the Success link once it’s been processed to see the results.

How is this different from simply looking at the source code of the page?

• You see the HTTP header information at the top. This information is generally easily available through tools such as Live HTTP Headers, but isn’t contained in the source code itself (since that information is coming from the server, not the page).
• You can see if the server is returning any of the page information differently than the page has been coded.
• You can see if the server is returning something different to Googlebot than what other users see. This tool uses the same user-agent and IP range as Googlebot when it crawls the web, so if the server is configured conditionally for user agent or IP address (typically known as “cloaking“), you’ll see  what’s being conditionally served to Google.
• You can use the tool to test changes (particularly things like redirects) in real-time.

Note that this tool won’t necessarily show you the content that Google is able to extract from the page. If the page contains JavaScript, for instance, you’ll see the raw JavaScript code contained on the page, not the rendered view visible in the browser. Which, unfortunately means you can’t use this tool to determine if Google is able to access content contained in rich markup.

What’s this about cloaking?

This tool can help you determine if the pages are being cloaked to Google. This may be useful if you’re coming into a project late and aren’t sure what’s been previously done. It can also help uncover if your site has been hacked. Back in 2006, Googler Matt Cutts and I did a show on Webmaster Radio during which we talked about how in some cases, a hacker might add links to a site and then cloak those pages so that the site owner never sees them. Only Google does. At the time, Matt suggested using Google Translate (and choosing English to English) to see what Googlebot was being served, but this tool can now more easily serve that purpose. Matt confirmed this to me this morning: “The biggest use case is just debugging site issues. Of those, the biggest case will be hacked sites. Some attacks will hide content until search engines fetch the page (and some attackers add a noarchive tag so that the search result doesn’t have a “Cached” link), so a site could look clean to the website owner. Using this feature will site owners verify that there are no hidden links in the page that Google actually fetches.”

How do I test redirects?

If you’ve implemented redirects, you can use this tool to test how Googlebot will interpret those redirects without waiting for those pages to be crawled. For instance, when I fetch www.searchengineland.com, I see that the redirect is correctly implemented as a 301 and points to searchengineland.com:

You can also use the tool to troubleshoot URLs listed in the Crawl Errors > Not Followed report. You can also test these URLs using something like Live HTTP Headers or by trying to access the URLs in a browser, but if neither of those methods uncover the problem, this tool can help determine that the issue is specific to Googlebot. You can also use this tool to verify that fixes you’ve made to redirect errors uncovered by the Not Followed report have really solved the problem.

(Note that the tool currently has a limit of 100kb per page. However, this is for the tool only and doesn’t apply to Googlebot’s normal crawl of the site. Google is monitoring feedback to see if many site owners find this size to be limiting.)

Malware details

The Google Online Security Blog has more information on the malware details tool. Previously, webmaster tools reported when the site was flagged has having malware and listed sample URLs. This new tool will also show samples of the malicious content, and in some cases, the underlying cause. This should help those site owners whose sites have been hacked to include malware find the problem and fix it. If your site does contain malware and you’ve fixed it, you can request a review to have the malware alert removed in search results.

The Google AdSense for Feeds (aka FeedBurner) blog announced that all of their links will be 301 redirected, as opposed to some that are 301 redirected. In summary, when you use FeedBurner to create and host your RSS feed, FeedBurner creates special links that are used to send the reader to your web site. Some users selected to use a 302 redirected link, as opposed to a 301 redirected link, to obtain better tracking.

Google has made the decision to remove the 302 redirect option and make them all 301s. Why? The main reason is Google wants these URLs to become “more compatible with search engines that crawl feeds.” A 301 redirect is the best solution for a search engine to determine where a URL lives. 302 redirects historically have confused search engines, as well as webmasters. So to be “consistent with the way that content is distributed today,” Google has made this change.

If you run FeedBurrner for your web site feeds, you don’t have to make any changes. Google has already made the 301 change for you.

In one of the SMX videos available in the member’s area of Search Engine Land, several panelists noted that a shorter URL is more likely to be clicked in the SERPs than a longer one. That is, if you have a URL like

http://www.yourdomain.com/2009/03/19/article-title/

it is less likely to get clicked than the following one, which is the same but without the dates:

http://www.yourdomain.com/article-title/

As I researched this, I noticed that Search Engine Land does not use dates in its URLs, nor does Google spam expert Matt Cutts, nor does Aaron Wall.

Since I have been using dates in my URLs on my blog for over two years, I researched how I could go about changing to not using dates. I found the following code and put it in my .htaccess file:

RedirectMatch 301 /([0-9]+)/([0-9]+)/([0-9]+)/(.*)
$ http://www.domain.com/$4

(Notes: This should be all one line, and to make this work for my blog I had to change “domain.com” to “keenerliving.com”)

Before uploading the updated .htaccess file to my site, though, I had to change the permalink structure in WordPress so it no longer generates URLs with dates in the title. Hence, I modified

/%year%/%monthnum%/%day%/%postname%/

to the following:

/%postname%/

I saved that change, saved my modified .htaccess file to the root directory of my domain, and then re-ran my plugin that generates a sitemap.xml file. Then, I immediately tested the change using several of my older links, to ensure that they redirected properly. I also made sure to activate Alex King’s 404 plugin, which emails me whenever a “not-found (404)” error is generated on my site. It is always a good to activate this plugin whenever you make any structural changes to your blog, as it will help identify any errors you may have created.

It will take a while for Google, MSN, and Yahoo! to reindex these links, but the end result should be a little bit better click-through rate for my domain.

For a while, though, one pays the price of losing page rank for individual pages. At least, that has been my experience … I am told that a 301 redirect should not drop page rank (PR), but somehow it has happened with me. In my case, these changes were made almost 4 weeks ago, and individual page ranks have still not gotten caught back up. A few of my most popular pages that had a Google Toolbar PR of 4 or 5, are currently showing a PR of 0. This may be normal, or it may be that I have done other things that are making it take longer for the PR to catch up (I’d love to hear of your experiences with this in the comments, on what is a normal catch-up time).

It turns out that I did two other things that may be impacting the catch-up time for individual page PR. One is that I went through all 700 of my blog posts and changed every internal link so that the dates were removed. That is, if I had one post that linked to another post using the format http://www.domain.com/year/month/day/linked-post-name/, I changed it link to http://www.domain.com/linked-post-name/. I thought this might speed the re-indexing process up. But I now wonder if this was stupid, and worked against me, or if it was at the very least a waste of time. It is also noteworthy that my blog has a relatively small number of external backlinks (I have not gone after them, although I know I should). There is good internal linking, but the relatively small number of external backlinks could be a factor in the PR not readjusting as one would expect. This is just speculation on my part, but I thought it should be pointed out.

Why go through 700 posts? I also wanted to clean my blog out. In looking at my Google Analytics, I noticed that several people were landing on pages that really should not even exist, and they were bouncing away immediately because the posts were worthless to them. These pages were ones in which I had noted something about a design change, or had apologized for the performance of a server I used a year ago, or some other site news. These posts were semi-relevant for my regular readers at the time the posts were written, but the posts are now worthless. In fact, they are worse than worthless because they have some negative impact on my site’s bounce rate (not a lot, but I am working on improving bounce rate, so I wanted the posts gone).

I did the proper things with these deleted posts, such as adding 410 redirects to my .htaccess file:

Redirect 410 /a-couple-of-site-changes/
Redirect 410 /admin-note/
Redirect 410 /miscellaneous-admin-information/
Redirect 410 /your-input-appreciated/
Redirect 410 /please-ensure-feeds-link-updated/
Redirect 410 /boredom-strikes-again/
Redirect 410 /revised-posting-schedule/
Redirect 410 /what-are-your-thoughts-on-this/
Redirect 410 /implemented-security-update-sorry-for-any-hiccup/
Redirect 410 /get-a-free-issue-of-smartphone-magazine-from-me/

I also updated my sitemap.xml file (so none of the deleted posts were in it), and used the Remove URLs feature in the Google Webmaster Tools to remove each of these deleted links from the Google Index.

But, did making all these changes at once work against me? I cannot say for sure at this time. However, I have learned something from this. Actually, I have re-learned, as I should have known already: when making changes, avoid making too many changes at once. Make one change, and give it time to have its impact (testing it as much as you can), before making other changes.

I hope this is a helpful case study for those of you considering changes in your site. I look forward to your comments.

Update: Two days after drafting this post, the Google Toolbar PR for each of my popular pages (all linked from my sidebar) now have a PR of 4, which is basically what they had before. I do not know if this is due to the redirects finally having their full effect, or whether it is due to the very recent Google PR update, or both.

Postscript: Almost exactly 4 weeks after implementing the redirect shown above, a friend came across a better redirect rule. He uses a slightly different URL structure, of the form site/year/month/postname/ instead of site/year/month/day/postname/. We learned that, for sites with his URL structure, the rule to redirect to site/postname/ is

RedirectMatch permanent ^/[0-9]{4}/[0-9]{2}/([a-z0-9\-/]+) http://example.com/$1

For sites with URL structure site/year/month/day/postname/, the correct rule is

RedirectMatch permanent ^/[0-9]{4}/[0-9]{2}/[0-9]{2}/([a-z0-9\-/]+) http://example.com/$1

Note that these rules apply for blogs that are installed in the site’s root directory. If WordPress is installed in a directory called blog, so that the URLs are of the form site/blog/year/month … , then the correct redirect rule is the same as above but with a change at the beginning: RedirectMatch permanent /blog/[0-9]{4}/[0-9]{2}/ …. (as in the previous two code blocks).

Also, it is important that you place the redirect rule before the redirect rules that WordPress has placed in your .htacess file. The WordPress redirect rules will typically look something like the following:

# BEGIN WordPress

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

# END WordPress

Finally, these redirect rules work on Apache-based servers. If your server is based on Microsoft’s IIS, a different approach would be likely be needed.

Over the weekend, the New York Times was once again ridiculed for how links to its former International Herald Tribune web site were not redirecting readers to new locations for IHT stories. The situation, which was first noticed at the end of March, seemed another example of a newspaper that didn’t get the online world. Well, it’s being fixed now.

The whole thing had me scratching my head in puzzlement. As I explained to IHT journalist Thomas Crampton, who led this weekend’s outrage with his Reporter to NY Times Publisher: You Erased My Career post, the New York Times has one of the most experienced SEOs out there: Marshall Simmonds. Creighton wrote up my comments in a follow up post, where I promised to look into the situation more.

Marshall himself reached out to me on Monday proactively, having seen the commentary. Effectively, the change happened out of his team’s control. He’s now emailed me this official statement from the New York Times on what happened and where things are going:

I wanted to bring you up to speed on the developments around the shuttering of the International Herald Tribune site, IHT.com, and archive transfer to the New York Times, nytimes.com.

The SEO team has been involved in the project since the beginning and recommended the implementation of one-to-one 301 redirects of all IHT content to its new location at the NYT.com. This, of course, conforms to best SEO practices when moving any content and was accepted and built into the timeline. We’ve moved plenty of content at the NYTimes before without incident.

As the project progressed, and we turned our focus to other matters, the move ran into several complications and a fast approaching deadline. Rather than consult with us, the IHT project team made a last minute decision and used an interim 302 solution without understanding the SEO implications.

After launch the impact was obvious, and we moved in to correct and educate. The NYT development team’s top priority right now is one-to-one 301 mapping of all IHT content to its new location. This, unfortunately, was a perfect storm of quick fixes and lack of attention to protocol and will be fully corrected in the coming weeks.

By one-to-one changes, Marshall means that anyone following a link to a story at the former IHT site will be redirected to the same story in a new location. And it’s good news to hear that the current redirects were done using 302 “temporary” codes rather than 301 permanent ones. That should help all the old links still carry credit to the new page locations, when those are in place and 301 redirects established.

Good news, also, that, that the links will be restored for those trying to locate the articles. Let’s hope the New York Times might find a way to speed the process.

From time-to-time, issues about how domain names can impact Google rankings come up. Is it true that if you buy a name, all the “link equity” that name has gained is lost? Below, a look at this and some related questions.

Several years ago, Google sparked some concerns when it said that buying an established domain name meant that links to that domain before the purchase were effectively lost. In other words, say someone bought a domain today that was registered in 2003 and which had built up hundreds of links over the years. All those links were effectively slapped with an invisible nofollow tag, passing along no credit. But links from after the purchase date would accrue credit.

Why did Google do this, way back then? Many people were buying old domains simply for the links – in some cases, also because they were listed well in the Yahoo Directory, when that was far more important than it is today. Putting the word out that buying domains wouldn’t gain link credit was a way for Google to dash cold water on the tactic.

Since that time, there have been any number of web sites that have had domain names change hands for various reasons, such as through acquisitions. For example, Company A absorbs Company B, which causes the domains owned by Company B to transfer to Company A. Was all that link credit was really lost?

That didn’t seem to be the case, so it’s been on my list to get the current state of how domain transfers impact link credit from Google. And that is? Google’s Matt Cutts told me:

There are some domain transfers ( e.g. genuine purchases of companies) where it can make perfect sense for links to transfer. But at the same time it wouldn’t make sense to transfer the links from an expired or effectively expired domain, for example. Google (and probably all search engines) tries to handle links appropriately for domain transfers.

Adding further, he said:

The sort of stuff our systems would be designed to detect would be things like someone trying to buy expired domains or buying domains just for links.

Let’s take those statements and see how likely it is that different types of domain acquisitions will pass link credit.

Buying Expired Domains: Don’t Expect Credit

Have you picked up a domain that was once owned by someone, not through buying it directly from them but because it had expired and went back into the common pool of domains for purchase by anyone? That’s an expired domain – and chances are, the backlinks aren’t going to pass credit according to Matt’s statement.

Buying Domains & Redirecting Links: Probably No Credit

Did you see a tasty domain and think it would be nice to get it, in order to obtain its links for a different site, such a redirecting them? Sounds like there’s a good chance that Google is going to notice the purchase date, take note of the redirection as well and decide those “historic” links shouldn’t count. What about if you just paid someone to keep the domain going under their name but closed down any existing content and point to another location? Might work; then again, Google might note the change, the oddity of one site to completely point at another, and it might be that the links won’t count.

Buying Domain & Running Web Site As Usual: Credit Likely

Did you buy a web site from someone else and are maintaining the business on that site as normal? Despite the fact that your domain name registration will have changed, since the site is carrying on as usual, there seems to be a good chance that link credit will continue as normal.

Getting Domains Through Acquisition: Credit Likely

Have a domain that changes hands, due to a company acquisition – company A buys company B? You should be OK, thought it’s unclear how Google tells the difference here from an ordinary transfer. And no, Google wouldn’t share more on how they can tell the difference.

Does Domain Registration Length Matter?

Finally, from time to time I’ve seen suggestions that the length of domain registration is a factor – domains registered for longer are rumored to be more trusted by Google. In the past, I even saw one domain registrar claim this as fact. Matt said about this:

To the best of my knowledge, no search engine has ever confirmed that they use length-of-registration as a factor in scoring. If a company is asserting that as a fact, that would be troubling.

He also added:

The primary reason to renew a domain would be if it’s your main domain, you plan to keep it for a while, or you’d prefer the convenience of renewing so that you don’t need to stress about your domain expiring.”

Digg is promising a significant change to how its DiggBar framebar operates, one that should solve SEO concerns about how link credit is passed on but won’t entirely remove misgivings about the framing of content.

With the new implementation scheduled for next week, only those who are logged into Digg will see the DiggBar. So if you click on a DiggBar URL from Twitter, you won’t see the DiggBar unless you’ve already been to Digg and have a logged in status. Just having a Digg cookie won’t be enough, Digg cofounder Kevin Rose told me when we talked about the new implementation yesterday. A Digg user must be logged in for a DiggBar to show. Logged in users can also opt-out of seeing the DiggBar.

Those who are NOT logged in will simply be redirected to the destination page, via a 301 permanent redirect. This will include search engines that follow those links. Digg’s been in touch with Google’s Matt Cutts on the change, which I’ve confirmed. The consensus is that Google will register DiggBar links as 301 redirects.

Overall, this change is a good thing. It means that link credit will flow to the destination site, rather than being kept by Digg itself.

There remains the separate issue of framing, how that was unpopular in the past and remains so in many quarters given the recent resurgence. Yesterday, it was pointed out by Michael Gray how Kevin Rose himself wasn’t pleased to find his own content being framed. So why do this to others, I asked him.

Kevin’s response was that he felt the Truveo experience was different. He wasn’t a logged in user there, so wasn’t expecting that type of display. He also didn’t like that it was an all-panel framing — that his content was completely surrounded by all four sides, in a frame.

I have mixed feelings. Showing a frame to only logged-in users seems like a fair compromise. There’s no doubt that for some Digg users, the framebar is useful. It’s also easier for them to use than downloading a separate toolbar (though Kevin said Digg still plans to improve and promote its toolbar).

On the other hand, I still hate frames. I also really worry that this will just cause other services to do the same. Imagine if Google decided that it should put up a Google framebar for anyone logged into one of its services? I still think the internet would explode in fury over such a move. But Digg’s compromise would give Google the cover to do so — as it would give cover to Yahoo or Microsoft, and as it now extends cover to Facebook that already does framing.

I wish Digg had made the tough decision to drop the framebar entirely, so that pressure could then be applied to Facebook and StumbleUpon and others to drop theirs. But the compromise does seem a good one, as long as it doesn’t end up opening the gates for more framing. Certainly sites that object can continue to run framebusting scripts, as we do here. That solves the frame issue for them, and the other change deals with the link credit issue that was developing.

For more about the DiggBar, framebusting, 301 redirection and other issues, see these two past articles from me that go into them in much more depth:

• The Growth Of Framebars & Kevin Rose On The DiggBar
• URL Shorteners: Which Shortening Service Should You Use?

For more, see discussion on Techmeme.

The DiggBar has been out for about a week now. Since then, there continues to be concerns over twin issues of whether it robs sites of link love and frames their contents in a way that’s unfair to publishers. I had a good conversation with Digg cofounder Kevin Rose today about these issues and how Digg is actively looking at ways to solve worries over the tool.

For those unfamiliar with the DiggBar, it allows people to create a short URL that’s useful in services like Twitter. Anyone clicking on a shortened URL made through Digg gets to a page with a DiggBar at the top. For example, here’s how it looks for a post I recently did on my personal blog about newspapers and concerns over Google:

The red arrow points at one feature, how the DiggBar allows anyone to vote on the page they’re viewing. There are other handy features, such as the ability to see any comments people have made at Digg about the page:

There’s no doubt that if you use Digg a lot, you’ll probably love the DiggBar. But the bar does two things that aren’t making some people (including me) very happy. It doesn’t pass along link credit, and it frames web content.

Link Credit Issues

Last week, my URL Shorteners: Which Shortening Service Should You Use? article went into depth about how various URL shorteners work. A key issue is whether these shorteners tells search engines to credit the destination URLs they point at. Those issues what’s called a “301 redirect” do this correctly (my What Is Google PageRank? A Guide For Searchers & Webmasters article covers more about link credit issues and why it is important to search rankings).

The DiggBar does not do a 301 redirect (nor can it, as this would prevent the DiggBar from showing at all). If you shorten a page using DiggBar service, then Twitter the short URL you receive, any links that Google or other search engines find via that short URL will send credit to Digg, not to the destination page you shortened.

A sidenote here. Twitter automatically puts a “nofollow attribute” on any links that people tweet. That’s a method to tell search engines that the links shouldn’t be counted as “votes” as part of their ranking processes. However, tweets often appear off the Twitter.com domain. In some of these places, the nofollow attribute (or tag) doesn’t get used. So tweeted links can get counted by search engines, and it remains important that URL shorteners pass along credit to the destination pages.

Digg had a blog post out yesterday explaining that they had done some things they believed would stem concerns about link credit not flowing properly. SEO expert Greg Boser dissected that post, finding it didn’t hold up. I also looked at it today and found problems:

1) Using the noindex tag prevents the pages that Digg makes with shortened URLs from being spidered by Google and other search engines, but that does not solve the issue of them still accumulating all the link credit rather than this going to the destination URL. Also, so far despite using noindex, some of these pages are getting listed in Google. Here’s another example of this. (Looking at the source code, that page lacked a noindex tag an a canonical tag. It seems like originally, the DiggBar didn’t add these tags. Now that they are present, it will take search engines a few days to weeks to catch-up).

2) Using the canonical tag as a form of redirection doesn’t work, because that tag is still treated as a “hint” by search engines rather than an must obey instruction. It also doesn’t work across different domains (IE, Digg.com can’t point at content off Digg.com’s own domain and use the tag to tell the search engines anything).3) The “source URL” solution Digg discusses doesn’t solve anything. What this means is that if you’re on the Digg home page, stories are listed there from across the web. For example, here’s a popular one right now from the Daily Telegraph:

http://digg.com/d1oOii

Digg uses a short URL to point you at that story, which in turn brings the DiggBar up on the top of the page. However, if you can’t run JavaScript (as search engines operate), then you get the long “source URL” like this:

http://www.telegraph.co.uk/scienceandtechnology/science/evolution/5131017/Egg-collected-by-Charles-Darwin-found-at-Cambridge-University-after-200-years.html

Digg’s thought was that by showing the long “source” URL to search engines, then the long URL ultimately will get all the link credit. However, there are plenty of places where the short URL will be found across the web by search engines because it is listed with regular HTML, rather than through JavaScript.

Framing Issues

Back in the late 1990s, framing was a big issue. For those unfamiliar, frames allow a web site to pull in content from other web sites into their own pages. It was much loathed for a variety of reasons. It often led to bad user experience. It caused serious issues for search engines, making it difficult for them to spider content properly. Some felt it was a copyright violation — that the site doing the framing was effectively copying their material without permission.

Framing largely disappeared for all of these issues. But now it’s coming back, and Digg’s use with the DiggBar may have been the tipping point.

Last October, StumbleUpon added framing of sites, so that anyone starting a browsing experience from StumbleUpon’s home page gets a framebar like this, as the red arrow points at:

Back in December, Facebook added its own framing of content through a framebar that appears when you click on posted links from within the service. Again, the red arrow points to an example that you can see for yourself here:

Ask.com started framing search results in February. The red arrow below points to the framebar, which appears when you click from Ask search results to a web page that’s listed, like this:

Ask used to do this when it first started out back in the 90s, then dropped framing apparently because so many sites moved away from that model. Now with harder economic times, it apparently finds value in trying to take over the top part of your browser window.

Of course, Google also frames web sites with its Google Image Search service. I believe it has operated this way years and years ago from when image search first started:

There was actually a lawsuit over this, which Google won. Despite that, this might be a good time for Google to reconsider the framing it does.

Also, if Google were ever to frame web sites when you click to them from search results in the way Ask does, the web would almost certain erupt in anger. I don’t think this will happen, of course — but if it’s not something we’d allow Google to do, it’s not something we should be allowing any sites to do.

Framebars Versus Toolbars

Clearly Digg didn’t start the new wave of framing, but it certainly has taken much more heat than Facebook or StumbleUpon over it. I think it’s the combination of URL shortening with framing that’s tipped people over the edge. That mixed framing with the popularity Twitter, where most people these days seem to be using URL shorteners. I think it creates worry that there will be no stopping framing or framebars now.

I feel for services like Digg and Facebook and StumbleUpon. The framebars they’ve created are useful and certainly easier than having users install toolbars for their browsers. But they remain frames, and they bring with them all the negatives about frames that we had in the past.

I’d hope that perhaps there’s an industry move to develop some standards around framebars. For example, if they’re going to be used, perhaps they are less intrusive to a publisher if shown at the bottom of a browser window, rather than at the top. Perhaps there’s also a way to ensure that the URL showing in the browser window remains that of the “source” site with the framebar also displayed (it’s been a long time since I played with frames, so I’m not sure this can be done).

Other ideas might include developing a standard script that publishers can use if they want to break frame code but also inform visitors from a particular site (such as Digg or Facebook) that they can get similar functionality using software toolbars. Perhaps pop-up toolbars in a separate window could work, though there are issues with pop-up blockers.

I don’t know the right answer. Personally, I think the easiest thing would be for everyone to just say no to frames. If you want your dedicated users to have toolbar-like functionality, then have them install an actual toolbar, not a framebar.

Kevin Rose On DiggBar

How’s Digg viewing the uproar? “It’s been a crazy learning experience for us,” Rose said. “We want to follow best practices.”

Rose explained that initially, Digg wanted simply to do a toolbar to help their most active users more easily Digg content or comment on it.

“We wondered what can we create that allows people to go visit that site with a single click and still get a Digg experience. That was kind of the idea behind creating the bar in the first place,” he said.

More as an afterthought, when seeing how popular it was to shorten URLs on Twitter, Digg added on a shortening aspect to the DiggBar.

“The goal in creating this wasn’t, ‘Let’s be the universal URL shortener.’ It was ‘Let’s make a tool that can enhance the experience for Digg users’.”

Rose said someone at Digg did speak with a software engineer at Google, as mentioned in their blog post, about the best way to pass along credit to Google — but he didn’t know who that was. Fair to say, they’ll get the straight scoop shortly, as Rose is now set to speak to Matt Cutts, who heads Google’s spam fighting efforts and who also closely watches over webmaster issues.

As for the DiggBar’s future itself, Rose said the company is taking in all the feedback to determine what’s the next best step.

“I want to make it known by all means that we’re sitting down and thinking about this stuff and trying to come up with solutions that work for anyone,” he said.

What’s A Webmaster To Do?

While Digg reexamines the DiggBar, there are webmasters who will remain concerned. My original article on URL shorteners has code you can use to block framebars. Wikipedia has a page about this, too, and you can see John Gruber’s code here. By the way, we actually had that code on our site before the DiggBar came out, just as a general best practice of breaking frames.

Of course, if you like the idea that people can more easily Digg (or Stumble or Facebook) your content, then you might not have an issue with using the frames.

I’d still recommend that if you’re wanting to shorten URLs for your own sites, use a service that’s primarily built for that and which does 301 redirect.

URL shortening services are experiencing a renaissance in the age of Twitter. When every character counts, these services reduce long URLs to tiny forms. But which is the best to use, when so many are offered and new ones seem to appear each day? Below, issues to consider and a breakdown of popular services, including recommendations and services to avoid (the new DiggBar being one of these).

The URL Shortener Mega Chart

To make recommendations, I reviewed various services and how they stacked up in a variety of features. All this got dumped into a spreadsheet, below. You can also view it in full-screen here.

[AUGUST 2009: PLEASE NOTE, I'LL TRY TO UPDATE THE CHART IN THE NEAR FUTURE; EARLIER THIS YEAR, BIT.LY WAS MADE THE DEFAULT IN TWITTER].

Each column represents a feature. Want to know more about that feature and why it is important? Each is explained below the chart. Green indicates that a particular service gets a good grade for that feature. Red indicates that it is lacking. Not all features are of equal importance, however, so don’t let the occasional red mark make you think a service is lacking. But in general, the more green, the better.

To busy to read the chart or the explanations? There’s a summary of recommended tools at the end of this story. But first, the chart:

301 Redirect

The first column in the chart is for “Redirect.” This is because a top issue to me, and many others, is that a URL shortening service does a “301 redirect” to the full URL. That number stands for the code a web server issues to a browser (or search engine) when a URL is requested.

A 301 redirect says that the URL requested (the short URL) has “permanently” moved to the long address. Since it’s a permanent redirect, search engines finding links to the short URLs will credit all those links to the long URL (see the SEO: Redirects & Moving Sites section of the Search Engine Land members library for more about redirection).

In contrast, a 302 redirect is a “temporary” one. If that’s issued, search engines assume that the short URL is the “real” URL and just temporarily being pointed elsewhere. That means link credit does not get passed on to the long URL.

In short, if you’re hoping that links you tweet will generate link credit for your web site, you want a service that issues a 301 redirect. Also keep in mind that while 301s might be issued today, a shortening service could shift to 302 directs at any time (and if they do, I hope scorn gets poured upon them).

Do tweeted links really matter that much? I mean, it’s not like search engines are really finding these URLs, right?

Wrong. Consider this search on Google:

“http://snurl.com/eom0d”

That brings back 27 links to a story that I twittered last week. Lots of those links are on Twitter pages themselves, from when other people retweeted my link. But some come from FriendFeed, and still others come from places like BusinessWeek, which embeds links from Twitter for various topics.

The bottom line is that when you tweet a link, you have no idea where or how it will spread — but if it does spread, why not ensure your site gets the right credit for it? So get a 301 redirect!

On the chart, you can see which services do 301 redirects (good — thus colored green) versus 302 redirects (bad — thus colored red). One service (Twurl/Tweetburner) issues a 303 code. I have no idea how search engines treat this, but it’s so unusual that I’ve flagged it as red (Google’s Matt Cutts told me he thinks Google may treat 303s as 302s).

Finally, want to test things for yourself? Rex Swain’s long-standing HTTP Viewer is an excellent tool. Enter the short URL into the URL box, untick the “Auto-Follow” box and submit. Then look to see what code is reported.

Tracking

Have people been clicking on your URLs that you tweet? A service like Google Analytics can show you traffic that comes from Twitter, but you might want to drill down further than it allows. A number of URL shortening services provide tracking stats, which in turn range from basic to extensive reporting.

A future article will look at the type of tracking involved. But all things being equal, it makes sense to select a service that provides tracking. Why not have the data if you want it?

To me, tracking is an important feature. That’s why services that offer it get a green for good on the chart and those lacking it a red for bad.

Stability

Nothing is more annoying than tweeting a link using a URL shortener and then having people tweet back at you that the link isn’t working, because the URL shortening service has gone down.

Unfortunately, I don’t know of any regular metrics of which service is the most stable, so there’s no stability column in the mega chart above. My assumption is that those gaining funding and partnerships are more likely to solve capacity problems than others.

Aside from short-term stability issues, there’s also a long-term consideration. What happens if a service shuts down, as happened to Zi.ma? If a service goes down permanently, it takes down all those links that were passing along credit to your site with it. With Zi.ma, fortunately Kl.am was able to keep those links working. But they could have been gone forever.

TinyURL has been around since 2002, so it has some stature in the space. Newer service Bit.ly recently raised $2 million in funding, which suggests it has some people willing to sink money into its future. But neither stature or investment is a guarantee of long-term success.

Twitter Client Support

In lieu of good stability stats, my own preference is to fallback toward depending on services with a lot of use or which are built into Twitter clients. These receive lots of traffic, which should translate into someone deciding they’re valuable, even if the exact revenue models are still being worked out.

Twitstat is one service that reports client metrics (hat-tip to Mashable on spotting that resource). The top current top clients:

• Twitter [web]: 28%
• Tweetdeck: 13%
• Twitterfeed: 6%
• Twhirl: 5%
• Tweetie: 5%

So which clients provide support for which URL shorteners? The mega chart summarizes this above. Any service that’s a default in one or more clients is colored green for good. Red indicates they have no partnerships (or are losing their TweetDeck one).

Twitter itself is used as a “client” by many people to publish their Tweets. If you give it a URL of 30 characters or longer, Twitter will shorten it using TinyURL. Unfortunately, Twitter doesn’t allow for this default choice to be changed. I hope that will happen in the future. Until then, you can always use another service to manually shorten your URLs.

TweetDeck is a nice desktop client (see How To Track Keyword-Based Tweets Within Your Twitter Stream for more about it). Currently, it supports 13 different URL shorteners (oddly including zi.ma, which no longer operates). However, the company recently announced that support for most of these would be removed, with these five continuing or being added.

The mega chart shows those that are going away (flagged as “going”), those that are staying (these just say “TweetDeck”) and the one about to be added (flagged “coming.”) Bit.ly is the default choice, so it’s noted that way.

Twhirl is another desktop client that I personally use, and it’s fourth on the popularity list above. The latest version for uses the new DiggBar service by default. This is not recommended for use, as explained further below.

Twitterfeed isn’t really a client. It’s designed to automatically feed blog posts into a Twitter account. However, you can choose from a variety of URL shorteners rather than go with the default choice of TinyURL. Changing to a service that provides tracking seems a wise choice.

Tweetie is an excellent Twitter client for the iPhone. Unfortunately, it offers no URL shortening support (and since the iPhone absurdly doesn’t offer copy-and-paste yet, there would still be issues even if it were offered). However, in comments from my survey below, someone said that Bit.ly is integrated into Tweetie. If it is, I’ve yet to figure out where and how.

Postscript: Marty Shaw twittered me instructions on how to send links to Tweetie using the iPhone bookmark feature — awesome tip.

URL Size: Domain, Path & Total Characters Used

Another issue for many is just how small you can make a URL. A first step in keeping URL size down is if the domain used by the URL shortening service itself is small. For example, popular service TinyURL has the domain of tinyurl.com, 11 characters. But service is.gd comes in at only 4 characters. TinyURL will always be at a disadvantage over is.gd because there’s simply no way for it to reduce its domain name.

Of course, after the domain name comes the remaining part of the URL, the “path” portion or what on a normal web site would be the page name. The path size for services typically ranges from 3 to 6 characters. If you really want to stay small on character counts, go with a service that uses a short 3 character path.

Keep in mind that over time, even services offering short paths won’t be able maintain this. That’s because there are only so many 3 character combinations that can be used (is.gd has a nice tutorial about this). When those have all been used, a shortening service either has to recycle old combinations or expand the number of characters used.

Recycle URLs?!!! Could that happen? Potentially, yes — a service could reissue a URL that’s been used before. They don’t seem to do this, however. Unfortunately, only one of the sites I visited explicitly said that recycling was not done — is.gd. While the others almost certainly don’t recycle, they should be clear about this.

Overall, if it’s essential to save every character, go with a service that currently uses fewer characters for the path. Otherwise, seek out a service that keeps its domain as short as possible. But as I’ll get into, a short domain doesn’t take away from other issues you may wish to consider.

On the chart, domains that are 15 characters or shorter are shown in green for good; red means they have 16 or more. Characters include the http:// prefix and the trailing / before the path. Note that with Snurl, it produces a longer snurl.com URL from within Twhirl but on the site itself, it uses a shorter sn.im domain. I’ve gone with the shorter figures for the chart.

Further along the chart, you can also see the current character counts for paths generated at each service — and next to that, total characters for short URLs. If you’re after the very smallest URL possible, look at the total characters column. However, I’ve not given any “good” or “bad” credit for either of these columns because they will change (I went with orange to indicate a neutral opinion). Any service offering short 3 or 4 character paths now will, if it becomes popular, grow to using 5 or 6 character paths.

If you’re thinking long-term, especially with tracking in mind, then looking at the total count is the wrong figure, as that will change over time. Instead, the domain character count is more important.

Custom Or Vanity URLs

Typically, URL shorteners make up a random path for your URLs — those /6Q8 or /qu03e3 and other boring sounding names you’ve seen. Some of them allow you to override this, so that your path can have a “custom” or “vanity” ending. That means you can have a custom name for your URL, such as these that lead to Search Engine Land:

• http://bit.ly/seland
• http://cli.gs/seland
• http://kl.am/seland
• http://sn.im/seland
• http://tr.im/seland
• http://budurl.com/seland

I think it’s a nice feature for a service to allow custom naming, so those that provide this are marked green on the chart — those that don’t, red. And here’s a tip. Once a particular custom name is taken, it’s gone. If you want to protect your brand, you might register any vanity URLs you’d really like to have.

Sharing

Some URL shorteners share the most popular or interesting URLs that are being clicked on through their services. Potentially, this is a benefit to using a particular service. It could bring you added traffic. However, in my survey, few of the sites offering sharing did it in a way (or seemed to have enough traffic) to make this a compelling feature choice. It’s a minor benefit, not a huge one, at the moment. That’s why regardless if it’s offered, all services are colored orange.

A quick recap:

• bit.ly shows popular links on its twitter feed here
• kl.am shows most popular links on its home page
• Plurl shows most recent URLs on its home page
• POPrl shows most popular links on its home page
• Snurl shows “interesting snips” on its Browse Snips tab
• Tweetburner shows most popular URLs in last hour and ones that have gotten the most clicks in the last hour
• Short.ie shows most popular and active users
• Eweri is supposed to show popular URLs and latest ones on its home page, but this hasn’t worked for two days running, so I didn’t credit it on the chart.

Country Of Origin

Country of origin (IE, where the domain name is registered, not where the service is based) is something to consider, when it comes to URL shortening services. In order for some of these services to get such short domain names, they’ve registered names in countries off the usual .com beaten track. Armenia, Libya and South Georgia Island are just some of the places used. To date, I don’t know of any country that has a particularly bad track record when it comes to domain names. So this is likely a minor worry. I’ve indicated countries on the chart but marked them all with orange, for neutral.

Handling Parameters

Ever want to retweet a search on Google? Some URL shorteners can’t handle parameters in a URL, all that stuff with ? and & symbols. Consider this search for movie times:

http://www.google.com/search?hl=en&q=movie+times&btnG=Search

Put that into some URL shorteners, and they break — they’ll ignore everything from the ? symbol onward and just point people at the Google home page. So when selecting a service, this is another factor to consider.

In my testing, only Short.ie came up as unable to handle parameters.

What We Use

In deciding which URL shorteners to review, I focused on those that were built into the clients. As I’ve explained, this suggests they’ll have some stability.

There are also some metrics on which are most used such as from Tweetmeme (ff.im is listed there, but that’s simply FriendFeed’s internal shortener and not something that I can see you can manually use. Heavy use is another sign of stability, especially if people are actively turning to a service that doesn’t have the natural advantage of being built into a client.

I also ran a short survey this week, asking people on Twitter what they use. I received 233 responses, which might help people when making decisions among the choices summarized at the end of this article.

TinyURL was most used by 31.1% of those answering (people could choose more than one service). After that, bit.ly was at 25.2%, then is.gd at 9.9%, then cli.gs at 8.7% and tr.im at 5.3% — all others were less than 5%.

I also asked a basic question of why people use the services. Responses were:

Half (49%) said it was to save space. Only 16% said they wanted tracking, though 25% said they wanted both. Among reasons for “other” were to preserve branding or that it was easy (as these are built in).

I also let people leave open comments about why they use these services, what they’d like to see from them or any other thoughts. I’ve grouped them below by common themes:

Comments About Particular Services

• I use bit.ly by choice when I tip to techmeme so i can track. Also really like bit.ly analytics and the fact it tracks twitter/friendfeed mentions/clicks. tinyurl is twitter default – not sure why. twitter should auction url shortners and get some revenue – that will get rid of tinyurl. It is 11 chars long vs bitly’s 6. ff.im is the only choice to shorten when posting from FriendFeed to twitter. Wish they offered bit.ly so I could track clicks. Also dont like URLs that frame content and put ads.
• The only problem with bit.ly is that they often use the lower case L and the uppercase I which look very similar.
• I like the toolbar button that posts to twitter automatically in adjix. Don’t like the ad part though… need to find a different one that has the good feature without the bad.
• Is.gd has the shortest URLs I’ve seen, but sometimes name of the linked site gets garbled (which could be misleading to readers) when I use it on PowerTwitter. The redirects always seem to work though.
• Kl.am is great but I wish I had more control over deleting my urls when I don’t need them anymore. Seems like it would be an easy thing to add in there.
• I like the custom URL and tracking features for tr.im. Hear good things about bit.ly as well.
• Tinyurl is the only one I remember when I want to shorten a link for twitter.
• Tr.im works for me because it has tracking, twitter integration and a firefox plugin. it also just works, haven’t had any issues with it. Tweetie’s bitly integration in the iphone app is great!
• Twitering from tr.im is useful as it’s one less copy-and-paste step. I wish it would give the last couple of tweets and then you could easily make a traceable reply.
• While I haven’t given this decision much thought, I appreciate TinyArrows’ bizarre characters and tiny, tiny urls.

Concerns Over Malicious Links

• You have to trust the person because you can’t trust the link.
• I have concerns about malicious links being hidden in shortened links – got bit by a joke link this week. Not a big deal in this case but could it be a prob?
• Frustrating that you have to click through usually to figure out what the link is going to. Can’t get a clue from the domain name.
• I know it is unavoidable, but I do not like not being able to tell what website I am about to visit or, in addition, not seeing the link for memory purposes. I know they are needed for twitter because of space issues, but do not like them when space is not an issue. IE when people post them in blogs or general articles etc. In those cases, use the full URL. I remember sites by seeing the URL’s and when I cannot see them and all is there is the shortened URL, the link becomes meaningless to me. IE I can never remember where I read or saw something if I have not clicked on a link with the site name in it. Maybe just me, but feel they should only be used for space considerations.
• I love them for saving space, but not knowing where they will redirect me is annoying at times, when I find the time I’ll write a firefox extension or greasemonkey script to change the title attribute to contain the real url by decoding or resolving the shortened urls, like @mattcutts was on about earlier today
• It would be great if the page content was somehow reflected in the shortened URL so that people not used to seeing shortened URLs have confidence they are going to a relevant/safe site. No idea how this would work!
• The problem w/ shorteners is the reader doesn’t know where the link will go to (e.g. YouTube, Slideshare, company site, etc.).

Concerns Over Death Of Links If Service Dies

• A lot of links could die when these services die.
• I’m just afraid one of them is going to disappear, and break all my old links … but tinyurl is too long to use with Twitter, IMO.
• I’m worried the services will die and leave my shortened URLs as broken orphans. If I’m going to switch from the market leader, I want some kind of long-term commitment.
• I’ve discovered that I like using my own short url service for important links – like affiliate links on my own sites – ones that I control. If a shortening service goes down, I don’t want my important links to be at the mercy of that site, but I do want the ability to track the links and get stats. So, I’m using my own service for those important links. For everything else – just sharing of news and interesting links – I use cli.gs. It rocks.
• They’re great. Had no idea there were so many of them. Afraid of the day they will start requiring a fee. Also wonder if a URL you create with it ever expires or if it’s a permanent thing.
• What if they (the URL shortening service) go out of business? Jill Whalen brought up the good point of how an out of business service kills all of that archived traffic potential
• Will the Internet fall over if TinyURL implodes?

Wishlists & Feature Requests

• As a bonus, i especially like to capacity to deactivate/delete links.
• I don’t know why I don’t get google-analytics like tracking capability with bit.ly. Also wish I could use my own custom domain name on top of one of these. Finally I wish they would correlate with search.twitter.com re-tweets (RT) to see if one of my links is going viral.
• I wish there was a url shortener gadget for igoogle that could give you a short url without leaving the page (opening another tab). I prefer bit.ly to tinyurl.
• I’d love for del.icio.us to create a shortcut url for anything I (or anyone else) bookmarks so I could combine social bookmarking and sharing.
• Integration with Google Analytics, pass correct referrers and pass link juice
• I’m looking for good analytics, including city – level location tracking. I would also love for it to be mobile internet compatible (not iphon), so I can check the stats while I’m out. And 301 redirects is a must.
• I’m wondering about integrating them with Google Analytics. Have some ideas, not sure if already being done…
• Integration w/in web browsers.
• There’s so many to choose from, with no real differentiation so I use what’s handy. I’m waiting for something with better tracking to measure with the URL shortening, and using the Google URL builder (or other tracking code) if I really want to track something.
• They are voodoo. No longevity, fragmentation of brand, cannot trust what is behind a shortened URI.
• We need wordpress plugins!!!!!
• Wish there was a search box tool for firefox so I could just throw the long url in there and get it generated
• Wish Twitter offered the service automatically.
• Would like to have tracking from within twitter or tweetdeck

Some Other Assorted Comments

• “Had an interesting experience the other day where my AV [antivirus program] blocked a bit.ly shortened URL (cant remember what it was linking to but It was a URL linked in a tweet from someone Im following – could have been you!). This was the first time that this has happened, although I dont run my AV all the time (shameful, I know) so not seeing it before might have been a timing thing. If this is a widespread pattern I can see this as a pretty big roadblock for the adoption of twitter by the large population of only semi tech oriented internet users that are likely to become the second wave of twitter-ers. It is highly possible that they will not being willing to whitelist a url-shortening service through their firewalls.
• I use url shortening services for space saving, and tracking. I’m not going to tweet or try to IM urls that are 87597893 characters long. I was using tinyurl always until I tried bit.ly yesterday and realized how sexy its analytics were! I haven’t tried the digg bar yet, but I’m curious if they have built in analytics. I think any of these services that continue to innovate and provide functionality besides just a short url will become rather valuable.
• I will start using one with stats as soon as I decide on which service. I’m seeing where short urls with stats are becoming necessary.
• I’ll join the list of people saying “should pass link equity”. also i like to customize the URLs when posting links to my own content, but rarely do so for any reasons other than readability.
• Maybe I am old school but they seem somewhat unofficial. www.cnn.com/top-news-of-the-day has the banding a short url will never have
• Space savings very important, especially when including more than one URL and/or tags in a tweet. Don’t like the ones that frame the resulting page. Haven’t used a tracking URL shortner since zi.ma went under. Would like to find another one to use occasionally.
• Tracking analytics is a must for me, as is a 301. I really like the real-time GUI of bit.ly’s service.
• “Stats are nice but, Do we really need 50 companies developing what is basically a 301 redirect?”
• Terribly useful in a 140 character world and for tracking, and ultimately think that major services will want to offer URL shortening directly
• The minute I tweet I know the reach of my “ad.” Once I see how many clicks I get I can calculate a basic CTR.
• Thought about making my own for personal use…
• Working on making one for my own site to create branding.. why have tinyurl displayed when it could be youfoundjake.com displayed..yeah, i miss some extra characters, but worth it branding wise..
• They’ve made domain resellers and registrars very happy.

Top Choices

You made it! After reviewing the various features, here’s my rundown on recommended services:

bit.ly: It offers all the key features you’d want in a service, with nice stats that show number of clicks over time, what sites are referring traffic to your twitted URL, locations clicks are coming from and Twitter conversations using the URL. It’s also the default in three of the four major clients — or three of the major clients, if you exclude Twitter itself.

[NOTE: AUGUST 2009, BIT.LY IS THE DEFAULT NOW IN TWITTER, ANOTHER STRONG REASON TO CONSIDER IT].

tr.im: Offers all that bit.ly does other than not being built into Twirl. Tweetdeck support is coming soon. Stats are nice. Bit.ly perhaps offers more drilldown on the referrer front, from my fast review. Remember, a more in-depth review of tracking stats will come in the future.

[NOTE: AUGUST 2009, TR.IM ANNOUNCED IT WAS CLOSING, POSTING THIS NOTE ON ITS WEB SITE AS OF AUG. 9

tr.im is now in the process of discontinuing service, effective immediately.

Statistics can no longer be considered reliable, or reliably available going forward. However, all tr.im links will continue to redirect, and will do so until at least December 31, 2009. Your tweets with tr.im URLs in them will not be affected.

We regret that it came to this, but all of our efforts to avoid it failed. No business we approached wanted to purchase tr.im for even a minor amount.

There is no way for us to monetize URL shortening -- users won't pay for it -- and we just can't justify further development since Twitter has all but annointed bit.ly the market winner. There is simply no point for us to continue operating tr.im, and pay for its upkeep.

We apologize for the disruption and inconvenience this may cause you.

LATER, TR.IM SAID IT WOULD CONTINUE OPERATING AND WOULD BE COMMUNITY-OWNED. PERHAPS IT WILL THRIVE GOING FORWARD, BUT THE INITIAL AND DRAMATIC ANNOUNCEMENT OF CLOSURE GIVES ME PAUSE ABOUT ANYONE USING THE SERVICE, IF THEY'RE CONCERNED ABOUT LONG-TERM STABILITY]

[NOTE: Cli.gs announced in October that it would be closing. See also discussion on Techmeme].

cli.gs: Offers the same as the others above, though it is not built into either of the major clients TweetDeck or Twhirl. You do get TweetFeed support. Stats include showing which search engines have spidered your links. There’s a nice “social media mentions” section that show clicks coming off My Yahoo, FriendFeed, Google Blog Search and others.

More Choices

Snipurl / Snurl / Snipr / Sn.im: You kind of want them to pick a name and settle on it (like Sn.im). The service is currently in three clients, but it’s to be dropped from TweetDeck. Within Twhirl, it also uses the long Snurl.com domain. If you want sn.im URLs, you have to go outside the client to make them. Stats are simply the number of clicks — it’s not in the same class as services above. Finally, ZoneAlarm flagged it as spyware. It’s not, but that’s something to consider in case your visitors are running that software and trying to click on your links.

budURL: Being dropped from TweetDeck. Stats look extensive, if you want to pay. One of the longest domain names of any service.

Short.ie: Originally I had this down as not recommended, as it issued a 302 redirect. They got in touch with me and said not so! I tested again, and they are issuing 301s. They also provide custom URLs –I missed that. Both have been updated. Finally, they say that Twitterfeed support is coming. That with basic stats make them a nice alternative choice. However, they still can’t handle URLs with parameters (I tested that again, and it’s still an issue).

kl.am: Not built into any clients and the stats are fairly rudimentary.

POPrl: Semi-basic stats, being dropped from TweetDeck. No custom URLs.

idek.net: Very basic stats only, being dropped from TweetDeck. No custom URLs.

Choices Without Tracking

Don’t care about tracking? Then either of these are options to consider:

is.gd: Built into both major software clients.

TinyURL: The default choice for Twitter, built into TweetDeck and Twitterfeed. But the longest domain of any of the popular services. ZoneAlarm also flagged it as spyware — as with Snipurl, this could cause issues for your visitors.

Not Recommended

The services below don’t do 301 redirection. That’s a recommendation killer, to me:

• Hex.io
• Eweri
• Plurl
• Tweetburner

Avoid These Services

lin.cr: Originally, I had lin.cr on my list, because it was included with TweetDeck. But I was shocked when I tested it — so much so that I won’t even link to the service.

Rather than doing a 301 or 302 redirect, lin.cr spits out a 200. That’s server code for “OK,” or file has been found. In other words, lin.cr is telling browsers (and search engines) that your URL is actually located on their server.

But how does it redirect? It uses a meta refresh tag set to 0, which might be interpreted as either a 301 or 302 redirect, depending on the search engine. That forwards humans to a new page — but by loading up its own page, it also generates a pop-under paid ad using JavaScript. Meanwhile, search engines that can’t or don’t execute JavaScript get shown a different page, one that appears to have a paid link on it.

DiggBar: Just launched this week by social news site Digg, DiggBar seems at first glace to be a URL shortener. It does shorten URLs. You don’t even need to visit Digg to use it. You just enter a URL after the Digg.com domain to shorten it. For example, do this:

http://digg.com/http://searchengineland.com/are-our-brains-becoming-googlized-15421.php

And you get this:

http://digg.com/d1e5BK

Sounds cool, especially because it puts tools for those who are really into Digg at the top of your page, such as the ability to read comments from Digg about your page (if it has been submitted there) or the ability to Digg the page you’re on.

The fun stops there. Like lin.cr, it does a 200 code. That means the page is actually on Digg itself — they’re making a page with the DiggBar and pulling in your content without permission into a frame. That’s not illegal, but it’s a tactic that died off years ago. It also means that if you use the Digg short URLs, none of the link credit passes to your page. It’s all kept with Digg.

There’s no need for you to give Digg all your link credit. If you want to shorten your URLs, use a service that does a 301 redirect.

By the way, if you clicked on the short DiggBar example above, you would have seen the DiggBar load on top of a Search Engine Land page briefly, then disappear. This is because we use a frame busting script that says:

if (top.location!= self.location) { top.location = self.location.href }

(I can’t get the full code to display so search for this in the source code of this page, then copy from the opening JavaScript tag to the closing one, if you want to use it)

The code clears any site trying to frame our content. To see how a page looks via the DiggBar when it isn’t cleared out, try this link:

http://digg.com/d1nYVs

That puts the DiggBar on top of Google.

Postscript: See The Growth Of Framebars & Kevin Rose On The DiggBar for more about the DiggBar, framebars growing in usage and busting techniques.

Open Letter To Twitter & Twitter Clients

I think Twitter and those who build Twitter clients have some responsibility in the URL shortener choices they offer. In particular, I think they should review carefully what they build into their clients and also allow users to add their own custom choices, if they so choice. In particular:

Twitter: TinyURL is ironically the longest domain name of all the URL shorteners. Let Twitter users have an account option to use another shortener, if they so choose. It would also be nice if URLs could be twitted without counting against the Twitter character count at all. Then we wouldn’t have to worry about all this URL shortener madness.

Twhirl: I’m hugely disappointed that the client I use and love added the non-shortener of Digg within a day of it being released plus made it the default choice. Was there no review of what the DiggBar actually does? And why make it the default over better choices?

TweetDeck: Similar to Twhirl, was there no review of something like lin.cr being made a choice for TweetDeck users, the vast majority of whom would not assume it was generating pop-under ads? It’s being dropped in the near future. Let’s hope those that remain are regularly checked.

Further Resources

In working on this article, I came across an number of other articles that you might wish to check out:

• URL Shortening: Hashes In Practice – From Coding Horror, it explains how the shortening is done
• URL Redirection – How it works, according to Wikipedia
• 90+ URL Shortening Services – From Mashable in 2008, covers even more services than above
• URL shorteners reviewed – From Malcolm Coles last month, services are charted showing if they provide tracking, custom naming and URL length.
• 11 Best URL Shortening Services – Vote Your Favorite – From last month at the Online Marketing Blog, services are listed in a nice chart summarizing features such as custom URLs, 301 redirection, tracking and other features.
• on url shorteners – From Delicious founder Joshua Schachter, covers a variety of issues with URL shorteners and why they might be bad for the web, with some suggestions for improvements. Also see discussion on Techmeme. Again, if Twitter were to simply not count URLs against a tweet’s character count — or provide its own shortener — some of these concerns would go away.