Twitter’s New SEO Play: A Profile Directory Of All User Accounts

Twitter has quietly launched a user profile directory that it hopes will bring it more search engine traffic.

It’s Twitter’s second SEO play in recent weeks, coming on the heels of a change in its robots.txt file. With that change, Twitter made its content-based search results — i.e., search results for tweets and hashtags, but not search results for users — crawlable.

With the new directory, Twitter is finishing that process by making its user profiles more discoverable by Google, Bing and other search engines. When asked why Twitter launched the user directory, a company spokesperson said it’s “to help people find the accounts they’re looking for with various search engines.”

Is The Directory Helping Twitter’s Search Visibility?

Twitter’s user directory has been online for a couple weeks. It’s essentially a master site map of Twitter accounts listed alphabetically and, from an SEO perspective, follows best practices by showing no more than 100 profile links per page.

As I write this, a Google search for site:twitter.com/i/directory/profiles gives a superficial display of 718,000 matching results. If I click through to reach the end of the 1,000 results that Google will show, I can reach page 47 before getting the “we have omitted some entries very similar to the 469 already displayed” message. And if I click to include the omitted results, I hit the end at page 57.

The 718,000 results isn’t right because Twitter’s directory doesn’t have that many pages. If you estimate that Twitter has about five million total accounts, the directory — with 100 accounts per page — would be about 50,000 pages, far less than 718,000.

UPDATE: I incorrectly typed and used five million accounts in the paragraph above, when I should’ve used 500 million accounts. My apologies. Based on 500 million accounts, the Twitter directory would have as many as five million pages, which means Google’s count of 718,000 could be an accurate reflection of how many pages it knows about.

But even if Google has only indexed about 57 pages (and those numbers aren’t perfectly reliable since I could be hitting a certain datacenter that has fewer results), it’s still a lot more than Bing. My searches show that Bing has so far only indexed the Twitter directory home page.

I have to assume that Bing will eventually crawl much deeper into the directory; Bing has gone on record admitting that its bot is slow, and that’s certainly the case here.

Of course, many Twitter profile pages have been crawled and indexed by Google and Bing for a long time now, so it’s difficult to say if the new directory is helping (or will help) the search engines find new, unindexed profiles.

The Checkered Twitter/Google Search History

What’s interesting is how Twitter’s recent search- and SEO-friendly moves fit into its checkered relationship with Google.

Google and Twitter used to be on good terms. Google had access to the full “firehose” of tweets and was primarily using that to power its Google Realtime Search product. But that ended in July 2011, forcing Realtime Search offline. Google promised that it would resurrect Realtime Search, but that never happened.

The end of that agreement meant Google no longer had access to the firehose — a special API feed of tweets — but Google was still able to do its regular crawl of Twitter profiles and content.

Except there was one temporary problem: Around the same time, in the summer of 2011, Twitter wasn’t very SEO-friendly. As we reported, Twitter was having technical infrastructure problems that hindered Google’s ability to crawl and index its site. Whoops.

In fact, Google says that it was cut entirely off from Twitter for a time. Here’s the head of Google’s search efforts Amit Singhal talking about the issue (at 1:35 in) earlier this year at our SMX London conference, as well as the challenge in crawling Twitter without a deal:

[youtube width=”560″ height=”315″]https://www.youtube.com/watch?v=PVzM4n1I-vM&t=1m38s[/youtube]

Twitter eventually fixed those issues, but its relationship with Google has been strained ever since the end of the companies’ agreement. In January, Twitter came out firing when Google launched its Search Plus Your World product, accusing Google of favoring its own social content ahead of Twitter (and other social content).

Google’s response then — and many times since — has been to mention how Twitter turned off the firehose and isn’t giving Google direct access to its content anymore. But Twitter engineers joined with peers at Facebook and MySpace to create a “Don’t Be Evil” tool that showed how easily Google could surface Twitter (and other social) content without firehose access. For more on that, see this recap of Danny Sullivan’s conversation with Google’s Eric Schmidt from January.

So today, while the war of words between Google and Twitter has died down, we now see Twitter taking a much more proactive approach to search engines — first with the robots.txt change and now with the SEO-friendly user directory.

For more on the directory itself and how it works, see my companion article on Marketing Land, Twitter Quietly Launches A User Directory.